SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Figure 5 we present the architectures and hyperparameters discovered by SMAC. Particularly we observe that conv 3x3 followed batch norm is a preferred operation and CosFace is the preferred head/loss choice.
C.4 Analysis of the Pareto front of different Fairness Metrics
In this section, we include additional plots that support and expand on the main paper. Primarily, we provide further context of the Figures in the main body in two ways. First, we provide replication plots of the figures in the main body but for all models. Recall, the plots in the main body only show models with Error  $< 0.3$ , since high performing models are the most of interest to the community.
Figure 5: SMAC discovers the above building blocks with (a) corresponding to architecture with CosFace, with SGD optimizer and learning rate of 0.2813 as hyperparameters (b) corresponding to CosFace, with SGD as optimizer and learning rate of 0.32348 and (c) corresponding to CosFace, with AdamW as optimizer and learning rate of 0.0006
Table 6: Searchable hyperparameter choices.
Hyperparameter	Choices
Architecture Head/Loss	MagFace, ArcFace, CosFace
Optimizer Type	Adam, AdamW, SGD
Learning rate (conditional)	Adam/AdamW → [1e-4, 1e-2], SGD → [0.09, 0.8]
Table 7: Operation choices and definitions.
Operation Index	Operation	Definition
0	BnConv1x1	Batch Normalization → Convolution with 1x1 kernel
1	Conv1x1Bn	Convolution with 1x1 kernel → Batch Normalization
2	Conv1x1	Convolution with 1x1 kernel
3	BnConv3x3	Batch Normalization → Convolution with 3x3 kernel
4	Conv3x3Bn	Convolution with 3x3 kernel → Batch Normalization
5	Conv3x3	Convolution with 3x3 kernel
6	BnConv5x5	Batch Normalization → Convolution with 5x5 kernel
7	Conv5x5Bn	Convolution with 5x5 kernel → Batch Normalization
8	Conv5x5	Convolution with 5x5 kernel
Figure 6: DPN block (left) vs. our searchable block (right).
Figure 7: Replication of CelebA Figure 2 with all data points. Error-Rank Disparity Pareto front of the architectures with any non-trivial error. Models in the lower left corner are better. The Pareto front is notated with a dashed line. Other points are architecture and hyperparameter combinations which are not Pareto-dominant.
Figure 8: Replication of VGGFace2 Figure 2 with all data points. Error-Rank Disparity Pareto front of the architectures with any non-trivial error. Models in the lower left corner are better. The Pareto front is notated with a dashed line. Other points are architecture and hyperparameter combinations which are not Pareto-dominant.
Second, we also show figures which depict other fairness metrics used in facial identification. The formulas for these additional fairness metrics can be found in Table 5.
We replicate Figure 2 in Figure 7 and Figure 8. We add additional metrics for CelebA in Figure 9-Figure 11 and for VGGFace2 in Figure 12-Figure 15.
C.5 Correlation between fairness and model statistics
We examine the correlation of each fairness-metric with model statistics. We compute statistics like number of parameters, model latency, number of convolutions, number of linear layers, and number of batch-norms in a model's definition. Interestingly, we observe very low and non-significant
Figure 9: Replication of Figure 7 on the CelebA validation dataset with Ratio of Ranks (left) and Ratio of Errors (right) metrics.
Figure 10: Replication of Figure 7 on the CelebA validation dataset with the Disparity in accuracy metric.
correlations between parameter sizes and different fairness metrics Figure 16. This observation supports the claim that increases in accuracy and decreases in disparity are very closely tied to the architectures and feature representations of the model, irrespective of the parameter size of the model. Hence, we do not constraint model parameter sizes to help our NAS+HPO approach search in a much richer search space.
C.6 Comparison to other Bias Mitigation Techniques on all Fairness Metrics
We have shown that our bias mitigation approach Pareto-dominates the existing bias mitigation techniques in face identification on the Rank Disparity metric. Here, we perform the same experiments but evaluate on the four other metrics discussed in the face identification literature: Disparity, Rank Ratio, Ratio, and Error Ratio.
Figure 11: Replication of Figure 7 on the CelebA validation dataset with the Ratio in accuracy metric.
Figure 12: Replication of Figure 8 on the VGGFace2 validation dataset with Ratio of Ranks metric.
Recall, we take top performing, Pareto-optimal models from Section 4 and apply the three bias mitigation techniques: Flipped, Angular, and SensitiveNets. We also apply these same techniques to the novel architectures that we found. We report results in Table 1.
In Table 8, we see that in every metric, the SMAC_301 architecture is Pareto-dominant and that the SMAC_301, demonstrating the robustness of our approach.
C.7 Transferability to other Sensitive Attributes
The superiority of our novel architectures even goes beyond accuracy when transferring to other datasets — our novel architectures have superior fairness property compared to the existing architectures even on datasets which have completely different protected attributes than were used in the architecture search. To inspect the generalizability of our approach to other protected attributes, we transferred our models pre-trained on CelebA and VGGFace2 (which have a gender presentation category) to the RFW dataset [111] which includes a protected attribute for race. We see that our novel architectures always outperforms the existing architectures across all five fairness metrics studied in this work. See Table 10 for more details on each metric. We see example Pareto fronts for
VGGFace2 Validation Set
Figure 13: Replication of Figure 8 on the VGGFace2 validation dataset with Ratio of Errors metric.
VGGFace2 Validation Set
Figure 14: Replication of Figure 8 on the VGGFace2 validation dataset with the Disparity in accuracy metric.
these transfers in Figure 17. They are always on the Pareto front for all fairness metrics considered, and mostly Pareto-dominate all other architectures on this task. In this setting, since the race label in RFW is not binary, the Rank Disparity metric considered in Table 10 and Figure 17 is computed as the maximum rank disparity between pairs of race labels.
Furthermore, we also evaluate the transfer of fair properties of our models across different age groups on the AgeDB dataset [77]. In this case we use age as a protected attribute. We group the faces into 4 age groups 1-25, 26-50, 51-75 and 76-110. Then we compute the max disparity across age groups. As observed in Table 11 the models discovered by NAS and HPO, Pareto-dominate other competitive hand-crafted models. This further emphasise the generalizability of the fair features learnt by these models.
VGGFace2 Validation Set
Figure 15: Replication of Figure 8 on the VGGFace2 validation dataset with the Ratio in accuracy metric.
Figure 16: Correlation map between different fairness metrics and architecture statistics. We find no significant correlation between these objectives, e.g. between fairness metrics and parameter count.
Table 11: Taking the highest performing models from the Pareto front of both VGGFace2 and CelebA, we transfer their evaluation onto a dataset with a different protected attribute - age - on the AgeDB dataset [77]. The novel architectures which we found with our bias mitigation strategy are Pareto-dominant with respect to the Accuracy and Disparity metrics
Architecture (trained on VGGFace2)	Overall Accuracy ↑	Disparity ↓
Rexnet_200	59.18	28.9150
DPN_SGD	71.87	22.4633
DPN_AdamW	61.32	21.1437
SMAC_301	79.97	18.827
Architecture (trained on CelebA)	Accuracy	Disparity
DPN_CosFace	65.55	27.2434
DPN_MagFace	68.17	31.2903
SMAC_000	80.23	19.6481
SMAC_010	80.37	26.5103
SMAC_680	79.88	20.0586
Table 8: Comparison bias mitigation techniques where the SMAC models were found on VGGFace2 with NAS bias mitigation technique and the other three techniques are standard in facial recognition: Flipped [9], Angular [76], and Discriminator [110]. Items in bold are Pareto-optimal. The values show (Error;metric).
Model	Baseline	Rank Disparity	SensitiveNets	Baseline	Disparity	SensitiveNets
Flipped	Angular	Flipped	Angular
SMAC_301	(3.66;0.23)	(4.95;0.18)	(4.14;0.25)	(6.20;0.41)	(3.66;0.03)	(4.95;0.02)	(4.14;0.04)	(6.14;0.04)
DPN	(3.56;0.27)	(5.87;0.32)	(6.06;0.36)	(4.76;0.34)	(3.98;0.04)	(5.87;0.05)	(6.06;0.05)	(4.78;0.05)
ReXNet	(4.09;0.27)	(5.73;0.45)	(5.47;0.26)	(4.75;0.25)	(4.09;0.03)	(5.73;0.05)	(5.47;0.05)	(4.75;0.04)
Swin	(5.47;0.38)	(5.75;0.44)	(5.23;0.25)	(5.03;0.30)	(5.47;0.05)	(5.75;0.04)	(5.23;0.04)	(5.03;0.04)
Model	Baseline	Rank Ratio	SensitiveNets	Baseline	Ratio	SensitiveNets
Flipped	Angular	Flipped	Angular
SMAC_301	(3.66;0.37)	(4.95;0.21)	(4.14;0.39)	(6.14;0.41)	(3.66;0.03)	(4.95;0.02)	(4.14;0.04)	(6.14;0.05)
DPN	(3.98;0.49)	(5.87;0.49)	(6.06;0.54)	(4.78;0.49)	(3.98;0.04)	(5.87;0.06)	(6.06;0.06)	(4.78;0.05)
ReXNet	(4.09;0.41)	(5.73;0.53)	(5.47;0.38)	(4.75;0.34)	(4.09;0.04)	(5.73;0.05)	(5.47;0.05)	(4.75;0.04)
Swin	(5.47;0.47)	(5.75;0.47)	(5.23;0.42)	(5.03;0.43)	(5.47;0.05)	(5.75;0.05)	(5.23;0.05)	(5.03;0.05)
Model	Error Ratio
Baseline	Flipped	Angular	SensitiveNets
SMAC_301	(3.66;0.58)	(4.95;0.29)	(4.14;0.60)	(6.14;0.52)
DPN	(3.98;0.65)	(5.87;0.62)	(6.06;0.62)	(4.78;0.69)
ReXNet	(4.09;0.60)	(5.73;0.57)	(5.47;0.59)	(4.75;0.58)
Swin	(5.47;0.60)	(5.75;0.56)	(5.23;0.60)	(5.03;0.60)
Table 9: Taking the highest performing models from the Pareto front of both VGGFace2 and CelebA, we transfer their evaluation onto six other common face recognition datasets: LFW [53], CFP_FF [100], CFP_FP [100], AgeDB [77], CALFW [128], CPLPW [127]. The novel architectures which we found with our bias mitigation strategy significantly out perform all other models.
Architecture (trained on VGGFace2)	LFW	CFP_FF	CFP_FP	AgeDB	CALFW	CPLPW
Rexnet_100	82.60	80.91	65.51	59.18	68.23	62.15
DPN_SGD	93.00	91.81	78.96	71.87	78.27	72.97
DPN_AdamW	78.66	77.17	64.35	61.32	64.78	60.30
SMAC_301	96.63	95.10	86.63	79.97	86.07	81.43
Architecture (trained on CelebA)	LFW	CFP_FF	CFP_FP	AgeDB	CALFW	CPLFW
Rexnet_200	71.533	73.528	54.30	55.933	60.517	52.25
DPN_CosFace	87.78	90.73	69.97	65.55	75.50	62.77
DPN_MagFace	91.13	92.16	70.58	68.17	76.98	60.80
DenseNet161	82.60	80.16	64.07	56.93	66.36	59.35
Ese_Vovnet39	77.00	76.50	64.68	49.38	61.283	61.016
SMAC_000	94.98	95.60	74.24	80.23	84.73	64.22
SMAC_010	94.30	94.63	73.83	80.37	84.73	65.48
SMAC_680	94.16	95.68	72.67	79.88	84.78	63.96
Table 10: Taking the highest performing models from the Pareto front of both VGGFace2 and CelebA, we transfer their evaluation onto a dataset with a different protected attribute - race - on the RFW dataset [111]. The novel architectures which we found with our bias mitigation strategy are always on the Pareto front, and mostly Pareto-dominant of the traditional architectures.
Fairness Metric	Transfer from CelebA	Transfer from VGGFace2
Rank Disparity	Pareto-dominant	Pareto-optimal
Disparity	Pareto-dominant	Pareto-dominant
Rank Ratio	Pareto-optimal	Pareto-optimal
Ratio	Pareto-dominant	Pareto-dominant
Error Ratio	Pareto-optimal	Pareto-optimal
Figure 17: Models trained on CelebA (left) and VGGFace2 (right) evaluated on a dataset with a different protected attribute, specifically on RFW with the racial attribute, and with the Rank Disparity metric. The novel architectures out perform the existing architectures in both settings.
(d)
Figure 18: TSNE plots for models pretrained on VGGFace2 on the test-set (a) SMAC model last layer (b) DPN MagFace on the last layer (b) SMAC model second last layer (b) DPN MagFace on the second last layer. Note the better linear separability for DPN MagFace in comparison with the SMAC model