HyperFace: Generating Synthetic Face Recognition Datasets by Exploring Face Embedding Hypersphere
Authors: Hatef Otroshi Shahreza
Sébastien Marcel
Venue: ICLR 2025 Poster (2025)
Keywords: Face Recognition
Hypersphere Optimization
Privacy
Synthetic Data
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We formulate the dataset generation as a packing problem on the embedding space (represented on a hypersphere) of a face recognition model and propose a new synthetic dataset generation approach.

Abstract: Face recognition datasets are often collected by crawling Internet and without individuals' consents, raising  ethical and privacy concerns. Generating synthetic datasets for training face recognition models has emerged as a promising alternative. However, the generation of synthetic datasets remains  challenging as it entails adequate inter-class and intra-class variations. While advances in generative models have made it easier to increase intra-class variations in face datasets (such as pose, illumination, etc.), generating sufficient inter-class variation is still a difficult task. In this paper, we formulate the dataset generation as a packing problem on the embedding space (represented on a hypersphere) of a face recognition model and propose a new synthetic dataset generation approach, called HyperFace. We formalize our packing problem as an optimization problem and solve it with a gradient descent-based approach. Then, we use a conditional face generator model to synthesize face images from the optimized embeddings. We use our generated datasets to train face recognition models and evaluate the trained models on several benchmarking real datasets. Our experimental results show that models trained with HyperFace achieve state-of-the-art performance in training face recognition using synthetic datasets. Project page: https://www.idiap.ch/paper/hyperface

Published as a conference paper at ICLR 2025
HYPERFACE: GENERATING S SYNTHETIC FACE RECOG- NITION DATASETS BY EXPLORING FACE EMBEDDING HYPERSPHERE
Hatef Otroshi Shahreza' and Sébastien Marcel
2 Idiap Research Institute, Martigny, Switzerland
? École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland 3 Université de Lausanne (UNIL), Lausanne, Switzerland fhatef. Statef.otrohi. sebastien sebaattie...mmrreeeed...........
ABSTRACT
Face recognition datasets are often collected by crawling Internet and without in- dividuals' consents, raising ethical and privacy concerns. Generating synthetic datasets for training face recognition models has emerged as a promising al- ternative. However, the generation of synthetic datasets remains challenging as it entails adequate inter-class and intra-class variations. While advances in generative models have made it easier increase intra-class variations in face datasets (such as pose, illumination, etc.), generating sufficient inter-class vari- ation is still difficult task. In this paper, we formulate the dataset generation as packing problem on the embedding space (represented on a hypersphere) of face recognition model and propose a new synthetic dataset generation ap- optimization
proach, called HyperFace. We formalize problem and solve with gradient descent-based approach. Then, we use a conditional face generator model to synthesize face images from the optimized embeddings. We use our generated datasets to train face recognition models and evaluate the trained models several benchmarking real datasets. Our experi- mental results show that models trained with HyperPace achieve state-of-the-art performance in training face recognition using synthetic datasets. Project page: https///www..diiip.ccppppppppppp
packing problem
INTRODUCTION
Recent advances in the development of face recognition models are mainly driven by the deep neural
networks (He et al., 2016), the angular loss functions (Deng et al.) 2019 Kim et al. (2022). and the availability of large-scale training datasets (Guo et al.) 2016 Cao et al. 2018 Zhu et al.) 2021) The large-scale training face recognition datasets collected by crawling the Internet and without the individual's consent, raising privacy concerns. This has created important ethical and legal challenges regarding the collecting, distribution, and use of such large-scale datasets (Nat 2022) Considering such concerns, some popular face recognition datasets, such as MS-Celeb (Guo et al. 2016) and VGGFace2 (Cao et al. 2018). have been retracted.
With the development of generative models, generating synthetic datasets has become promising solution address privacy concerns large-scale datasets (Melzi al., 2024 Shahreza et al. 2024). In spite of several face generator models in the literature (Deng et al. 2020 Karras et al.
2019 2020. Rombach et al. 2022] Chan et al., 2022), generating a synthetic face recognition model that replace real face recognition datasets and be used to train a new face recognition model from scratch is a challenging task. particular, the generated synthetic face recognition datasets require adequate inter-class and intra-class variations. While conditioning the generator models on different attributes can help increasing intra-class variations, increasing inter-class variations remains a difficult task.
In this paper, we focus on the generation of synthetic face recognition datasets and formulate the dataset generation process as packing problem on the embedding space (represented on the surface

---

Published as a conference paper at ICLR 2025
Figure 1: Sample face images from the HyperFace dataset
of hypersphere) of pretrained face recognition model. We investigate different packing strategies and show that with a simple optimization, we can find a set reference embeddings for synthetic subjects that has a high inter-class variation. We also propose regularization term in our optimiza- tion to keep the optimized embedding on the manifold face embeddings. After finding optimized embeddings,
a face generative model that generate face images from embeddings on the hypersphere, and generate synthetic face recognition datasets. We use our generated synthetic face recognition datasets, called HyperFace, train face recognition models. We evaluate the recog- nition performance of models trained using synthetic datasets, and show that our optimization and packing approach can lead to new synthetic datasets that can be used train face recognition mod- els. We also compare trained models with our generated dataset to models trained with previous synthetic datasets, where our generated datasets achieve competitive performance with state-of-the- art synthetic datasets in the literature. Figure illustrates sample face images from our synthetic dataset.
The remainder of this paper is organized as follows. In Section 2 we present our problem formula- tion and describe our proposed method generate synthetic face datasets. In Section we provide our experimental results and evaluate our synthetic datasets. In Section we review related work in literature. Finally, we conclude the paper in Section
2
PROBLEM FORMULATION AND PROPOSED METHOD
2.1 PROBLEM FORMULATION
Let us assume that we have a pretrained face recognition model F X, which can extract identity features (a.k.a. embedding) æ E X' C Rar from each face image I. Without loss of generality, we can assume that the extracted identity features cover unit hypersphere i.c., ||0||2 = 1, Vx E X.
Identity Hypersphere:
Representing Synthetic Dataset the Identity Hypersphere: We represent synthetic face recognition dataset D on this hypersphere by finding the embeddings for each image the dataset. For simplicity, let assume that for subject i in the synthetic face dataset, face image Tut, and reference embedding & = F (Tert.)) Note that the reference face image T may already exist in the synthetic dataset D, otherwise we can assume the reference embedding of as the average of embeddings of all images for subject i in the dataset D. Therefore, the synthetic face recognition dataset D with nia number of subjects can be represented embeddings {Ieer.
have reference
of reference
If the identity embedding extracted by F(.) not normalized, normalize such that ||>||
2

---

remind that Published as a conference paper at ICLR 2025
Face embedding by persphere
Face embedding hypersphere
Randonly Syntbesize Imuges for initlivation
HyperFace Optimization
0
Face
Pretmamed Face Recognition
Generator
HyperFace Darasct
Figure 2: Block diagram of HyperFace Dataset Generation: We start from randomly synthesized face images and extract their embeddings using pretrained face recognition model F. The extracted embeddings are normalised and used as initial points { tia in our HyperFace optmization. The HyperFace optimization tries increase the intru-class variation for synthetic identities on the manifold of the face recognition model over the hypersphere using regularization term. The resulting points are then used by face generator model G. which generate synthetic face images from the embeddings.
2.2 HYPERFACE SYNTHETIC FACE DATASET
HyperFace Optimization Problem: By representing synthetic dataset 'D the identity hyper- sphere as set of reference embeddings { 182 we can raise the question that "How should reference embeddings cover the identity hypersphere?" To answer this question, the distances between reference embeddings indicate the inter-class variation in the synthetic face recognition dataset D. Therefore, since we would like have high inter-class variation in the gen- erated dataset D, we can say that we need to maximize the distances between reference embeddings other words, we need solve the following optimization problem:
{Zret, J Pia
==
max
min d(Dref,is Frefa) [Kne)
subject to
|| = 1, Vk E {I,.... naa
(I)
where d(.,-) :) a distance function.
The optimization problem stated in Eq. is well-known optimization problem, which is known as spherical code optimization (J. H. Conway) 1998) or the Tammes problem (Tammes I930), where the goal is to pack given number of points (c.g., unit sphere such that the minimum distance be- studied for small dimensions and the number of points (Böröczky) 1983 Hárs 1986 Musin & Tarasov 2012: 2015). However, for a large dimension and a high number of points, there ture (Tokarchuk et al., 2024). For a large dimension and a high number of points, there are different approaches for solving this optimization problem (such geometric optimization, numerical opti- mization, etc.). However, for a large dimension of hypersphere (i.e., 71x) and a very large number of points (i.e., the number of subjects пia- such as 10,000 identities in our problem), solving this optimization can be computationally expensive. To address this issue, we solve the optimization problem with an iterative approach based on gradient descent. To this end, we can randomly ini- tialize the reference embeddings and find the optimised reference embeddings {arer i=1 using the Adam optimizer (Kingma & Ba 2015). This allows us to solve the optimization with a reasonable computation resource. For example, on a system equipped with a single NVIDIA 3090 GPU in 6 hours.
Solving the HyperFace Optimization:
particles, pores, electrons, etc.) on the surface
tween points maximized. The optimal solutions for this problem
closed-form solution in the litera-
solve the optimization for nx 512 and ni 10, 000
Regularization: While we solve the optimization problem in Eq. on the surface of a hyper- sphere, we should note that the manifold of embeddings X' does not necessarily cover the whole surface of the hypersphere. This means if we get out of the distribution of embeddings X, may not be able to generate face images from such embeddings. Therefore, we need to add a regulariza- tion term optimization problem that tends to keep the reference embeddings the manifold
3

---

Published as a conference paper at ICLR 2025
Algorithm HyperFace Optimization for Finding Reference Embeddings
1: Inputs: À learning rate, Fur number of itcrations, { ">lli embeddings of a gallery of face images,
a hyperparameter (contribution of regularization).
3: Output:
Xwt = a 1= optimized reference embeddings.
4: Procedure:
5: Initialize reference embeddingg (#ret, )=1
n I,.., Maar do
7: Find Liet,i; 2iet,j E Xiet which have minimum distance d( Bet..s)
8: Reg + 2k (ig d( d( Ryy)
> Calculate the regularization term
9:
cost + -d -d(xeet. Fzet.s)
10:
Xnd f- X Adam( Adam(Vcost, Vcost, 2)
11:
Xnr normalize (X(r))
To ensure that resulting embeddings xnt remain the hypersphere.
End For
13: End Procedure
of erembeddings X' To this end, we consider a set of face images (T, Then, we try to minimize the distance of our reference embeddings { J fi=1 to the set of embeddings {a:; ber)"" "odiin which approx- imates the manifold of embeddings X. To this end, for each reference embedding { 1nd
[ Raaler as gallery of image and extract their embeddings to have set of valid embeddings H "asley
find the closest emhedding in { "lylley and minimize their distance. We can write the optimization in Eq. T as regularized min-max optimization as follows:
f==
1 ny d( (ref.) 1hid
min
max (
min d(ziet,k Dg);
Pgiley
regularization
subject to | 1, K {I,..., maa
where a is a hyperparameter that controls the contribution of the regularization term in the opti- mization. To provide more flexibility in our optimization, we consider the size of gallery Rggallrry to be greater or equal to the number of identities nid in the synthetic dataset (i.c., Rgalllry 2 (id)).
To solve the HyperFace optimization problem in Eq. using Algorithm I we need to initialize the reference embeddings { )"= To this end,
Initialization:
generate ny number random i=1 " using a face generator model, such as StyleGAN (Karras et al. 2020),
synthetic images Latent Diffusion Model (LDM) (Rombach al. 2022). These models use noise vector as the input and can generate synthetic face images in an unconditional setting. Then, after generating (T, Aie
;=1
images, we can extract their embeddings using the face recognition model F-(-) and use the extracted
embeddings as initialization values for the reference embeddings { } 1id
in Algorithm
Image Generation: After we find the reference embeddings (wiet, i=1 using Algorithm we ddentity-oonditiooeed image generator model generate face images from reference beddings. To this end, we use a recent face generator network (Papantoniou et al. 2024), which is based probabilistic diffusion models. The diffusion face generator model G(
The
generate
a face image I = G( 2) from reference embedding &ref and random noise z ~ NNO,, [PM) )... Therefore, by changing the random noise and sampling different noise vectors, can generate different samples for the reference embedding Eeet. In addition, to increase intra-class variation, we add Gaussian noise v (/ 1 to the reference embedding Teef, and then normalize locate it on the hypersphere. In summary, we can generate different samples for each reference embedding Tee by changing and e noise vectors as follows:
G( De + Br
2),
(.00 NN PMM),
(3)
Frel + "00112
TThe gallery of face images (I can be generated using unconditional face generator network such StyleGAN (Karras et al. 2020), Latent Diffusion Model (I.M) (Rombach 2022.
4

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of recognition performance of face recognition models trained with different synthetic datasets and a real dataset (i.e., dataset in terms of accuracy and best value for each benchmark emboldened.
The performance reported for each
Dataset name
IDs
# Images
LFW CPLFW
CALFW
CFP
AgeDB
SynFace (Qiu et 2021
100000
999'994
86.57
65.10
70.08
66.79
59.13
SFace (Routros al
10'572
877
93.65
74.90
80.97
75.36
70.32
Syn-Multi-PP PIE [Cobois et 2021
100000
180'000
78.72
60.22
61.83
60.84
54.05
IDnet (Kolf 2023
10'577
1'057'200
84.48
68.12
71.42
68.93
62.63
ExFaceGAN [Boutoos al. 20236
10'000
599'944
85.98
66.97
70.00
66.96
57.37
GANDiffFace (Meizi etal. 2023
10'080
543'893
94.35
76.15
79.90
78.99
69.82
Langevin-Dispersien (Geisbuuller 2024
100000
650'000
94.38
65.75
86.03
65.51
77.30
Langevin-DisCo (Geissbohler et 2024
10'000
650'000
97.07
76.73
89.05
79.56
83.38
DigiFace- 1M et 2023
109'999
1'219'995
90.68
72.55
73.75
79.43
68.43
IDiff-Face (Uniform) Boutros al. 2025a
10'049
502'450
98.18
80.87
90.82
82.96
85.50
IDifi-Face (Two-Stage) Bentros et M023n
10'050
502'500
98.00
77.77
88.55
82.57
82.35
DCFace (Kim al. 2023
10'000
500' 000
98.35
83.12
91.70
88.43
89.50
HyperFace [ours]
100000
500'000
98.50
84.23
89.40
88.83
86.53
CASIA-WebFace et al 2014
10'572
490'623
99.42
90.02
93.43
94.97
94.32
where B is hyperparamter that controls the variations to the reference embedding. Figure depicts
the block diagram of our synthetic dataset generation process. Algorithm 3 in Appendix also present poc of dataset generation process.
EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Dataset Generation: For solving the HyperFace optimization in Algorithm 1 use initial learning rate of 0.01 and reduce the learning rate by power 0.75 every 5, 000 iterations for a total number of iterations far 100, 000. We also consider cosine distance, which is commonly used in face recognition systems for the comparison of face embeddings, as our distance function d(ss-). +)- For the hyperparameterrs and B8 consider default values of 0.5 and 0.01, respectively, in our experiments. We also consider the size of gallery to be the same as the number of identities, and explore other cases where Rgaalory& nij our ablation study. We generate 64 images, by default, per each identity in our generated datasets and explore other numbers of images in our ablation study.
We use ArcFace (Deng al. 2019) as the pretrained face recognition model F(-) with the embed- ding dimension of = 512 and use pretrained generator model (Papantoniou et al. 2024) to generate face images from ArcFace embeddings. After generating face images, we align all face images using a pretrained MTCNN (Zhang et al. (2016) face detector model. For our regularization, we randomly generate images with StyleGAN (Karras al. 2020) as default, and investigate other generator models in our ablation study.
Evaluation: To evaluate the generated synthetic datasets, we use each generated datasets as training dataset for training face recognition model. To this end, we use the iResNet50 backbone and train the model with AdaFace loss function (Kim et al. (022) using the Stochastic Gradient De- scent (SGD) optimizer with the initial learning rate and weight decay of 5 x 10-4 for 30 epochs with the batch size of 256. After training the face recognition model with the synthetic dataset, we benchmark the performance of the trained face recognition models on different benchmarking datasets of real images, including Labeled Faces in the Wild (LFW) (Huang et al. 20088) Cross-age LFW (CA-LFW) (Zheng et al.) 2017). CrossPose LFW (CP-LFW) (Zheng & Deng 2018). Celebri- ties in Prontal-Profile in the Wild (CFP-FP) (Sengupta et al.) 2016), and AgeDB 30 (Moschoglou et al. 2017) datasets. For consistency with prior works, we report recognition accuracy calculated using 10-fold cross-validation for each of benchmarking datasets. The source code of our experi- ments and generated datasets are publicly available
Project page: http:://wwwiidaaacchhhpppptttyte
5

---

Published as a conference paper at ICLR 2025
3.2 ANALYSIS
Comparison with Previous Synthetic Datasets: We compare the recognition performance of face recognition models trained with our synthetic dataset and previous synthetic datasets in the literature. We use the published dataset for each method and train all models with the same con- figuration for different datasets to prevent the effect of other hyperparameters (such as number of epochs, batch size, etc.). For fair comparison, we consider the versions of datasets with similar number of identities" if there are different datasets available for each method. Table | compares the recognition performance of face recognition models trained with different synthetic datasets. As the results in this table show, our method achieves state-of-the-art performance in training face recognition using synthetic data. Figure I illustraaees sample face images from our synthetic dataset. Figure Soureeoor of appendix also presents more sample images from HyperFace dataset.
Ablation Study; In our dataset generation method, there are differ- ent hyperparameters which can affect the HyperFace optimization and the generated synthetic datasets. Table 2 reports the ablation study on the num- ber of images generated per each syn- thetic identity in our experiments. As the results in Table 2 show, increas-
Table 2: Ablation study the effect of number of images
Image/ID
LFW
CPLFW
CALFW
CFP
AgeDB
32
98.70
84.17
88.83
88.74
86.33
50
98.50
84.23
89.40
88.83
86.53
64
98.67
84.68
89.82
89.14
87.07
96
98.42
84.15
89.00
89.51
87.45
128
98.20
83.63
89.82
89.31
87.62
ing the number of images per identity improves the recognition performance of trained face recognition model, but tends to saturate after 64 images per identity.
TableB]alss also compares the number of identities in the generated dataset, in- cluding 10k, 20k, and 50k identities. As the results in Table 3 show, in- creasing the number of identities im-
Table 3: Ablation study on the effect of number of identities
IDs
LFW
CPLFW
CALFW
CFP
AgeDB
10k
98.67
84.68
89.82
89.14
87.07
proves the recognition performance of trained face recognition model on the benchmarking datasets. The re- sults in this table demonstrates that we can still increase the number of identities and scale our dataset generation without saturating the performance. The main issue for increasing the size of dataset is computation resource, which is discussed in detail in Appendix A We can also reduce the complexity of our optimization for large number of identities, which discussed in detail in
30K
98.82
85.23
91.12
91.74
89.42
50k
98.27
85.60
91.48
92.24
90.40
Appendix B
Table reports the recognition per- formance achieved for face recogni- tion model trained with datasets with
Table 4: Ablation study on the effect of Rgallery
10k identity and optimized with dif-
Asgallery
LFW
CPLFW
CALFW
CFP
AgeDB
ferent numbers of gallery images. As the results in this table shows, in- creasing the size of gallery improves the performance of the trained model.
10K
98.53
84,00
88.92
89.34
85.9
20K
98.50
84.32
89.28
89.17
86.00
50K
98.72
84.23
88.72
89.19
86.85
However, with 10,000 images we can still approximate the manifold of face embeddings on the hy- persphere.
As another ablation study, we use different source of images for the gallery set to use in our regularization and solve the HyperFace optimization. We use pretrained StyleGAN (Karras et al.] 2020) as GAN-based generator model and pretrained latent diffusion model (Rom- bach et al. 2022) as a diffusion-based generator model. We use these generator models and randomly generate some synthetic face images. In addition, for our ablation study, we con- sider some real images from BUPT dataset (Wang et al., 2019) as dataset of real face images.
'Only in the dataset used for DigiFace (Bae et al. 2023) there are more identities, because there is only one version available for this dataset, which has greater number of identities compared to other existing synthetic datasets.

---

Published as a conference paper at ICLR 2025
As the results in Table 5 show, op- timization with images from Style- GAN and LDM lead to comparable performance for the generated face recognition dataset. However, the real images in the BUPT dataset lead superior performance. This sug- gests that the synthesized images can- completely cover the manifold of embeddings and if we use real images nition performance of our face recognition model.
Table 5: Ablation study on the type of data in gallery
Gallery
LFW
CPLFW
CALFW
CFP
AgeDB
StyleGAN
98.67
84.68
89.82
89.14
87.07
LDM
98.65
84.35
89.17
89.09
86.35
BUPT
98.70
84.77
90.03
89.16
87.13
gallery it can improve the generated dataset and recog-
We also study the effect of hyper-
Table 6: Ablation study on the effect of cr
parameters a and | on the gener- ated face recognition dataset. Ta-
ble reports the ablation study for the contribution of regularization in our optimization (a). As the results in this table shows, the regulariza- tion enhances the quality of generated dataset and improves the recognition performance of face recognition model. In fact, our regularization term helps our optimization to keep the points on the manifold of face recognition over the hypersphere, and therefore improves
O
LFW
CPLFW
CALFW
CFP
AgeDB
0
98.40
84.15
88.87
89.31
86.48
0.50
98.67
84.68
89.82
89.14
87.07
0.75
98.62
84.32
89.48
89.67
86.72
1.0
98.55
84.72
89.10
89.76
86.63
the quality of our synthetic dataset.
Similarly, Table 7 reports the ab- lation study for the effect of noise in data generation and augmentation (i.e., hyperparamter 9 in in Eq. 33. As can be seen, the added noise increases the variation for images of each sub- ject and increases the performance of face recognition models trained with the generated datasets. With a larger value of 3, the generated images for each identity have more variations, which increases the perfor- mance of the face recognition model trained with our synthetic dataset.
Table Ablation study on the effect of B
LFW
CPLFW
CALFW
CFP
AgeDB
98.53
84.00
88.92
89.34
85.90
0.005
98.67
84.68
89.82
99.14
87.07
0.010
98.7
84.72
90.05
89.54
88.42
0.020
98.4
84.05
91.32
90.13
89.83
As another experiment, sider different backbones and train face recognition models with differ- ent number of layers. As the results Table show, increasing the num- ber of layers improve the recognition performance of trained face recogni- tion model. While this is expected and has been observed for training us- ing large-scale face recognition datasets, it sheds light more potentials in the generated synthetic datasets.
Table 8: Ablation study on the network structure
Network
LFW
CPLFW CALFW
CFP
AgeDB
ResNet18
98.33
81.38
88.53
86.03
85.27
ResNet34
98.5
83.47
88.88
88.29
86.42
ResNet50
98.67
84.68
89.82
89.14
87.07
ResNet101
98.73
85.43
90.05
89.54
87.52
3.3 DISCUSSION
Scaling Dataset Generation: To increase the size of the synthetic face recognition dataset, we can increase the number of images per identity and also the number samples per identity. In our ablation study, we investigated the effect of the number of images (Table and the number of iden- tities (Table 3 on the recognition performance of the face recognition model. However, increasing the size of the dataset requires more computation. Increasing the number of images in the dataset has linear complexity in our image generation step (i.e., OT ), where Temages is the number of images in the generated dataset). However, the complexity of solving the HyperFace optimization problem with iterative optimization Algorithm has quadratic complexity (i.e., ((n There- fore, solving this optimization for larger number of identities requires much more computation

---

Published as a conference paper at ICLR 2025
Synthesized
Synthesizod
Real
Real
(a) children images
(b) adult images
Figure 3: Sample pairs of images with the highest similarity between face embeddings of images in synthesized dataset and training dataset of StyleGAN, which was used to generate random images for initialization and regularization the HyperFace optimization.
resources. Meanwhile, most existing synthetic datasets in the literature have comparable num- ber of identities to our experiments. We should note that in our optimization, we considered all points in each iteration of optimization which introduces quadratic complexity to our optimization. However, we can solve the optimization with stochastic mini-batches of points on the embedding hypersphere, which can reduce the complexity in each iteration (i.e., Obb)), where b is the size of batch and < Tia). We further discuss the complexity of our optimization and dataset generation in Appendix A and present further analyses for stochastic optimization, that reduces the complexity of our optimization in Appendix B
Leakage of Identity: In our dataset generation method, used images synthesized by StyleGAN for initialization and regularization. Therefore, is important if there is any leakage of privacy data in the images generated from StyleGAN in the final generated dataset. To this end, extract and compare embeddings from all the generated images to embeddings of all face images in the training dataset of StyleGAN. The highest similarity score between generated images and training dataset correspond to children images (as shown in Figure Ba) which are difficult to compare visually and conclude potential leakage. Figure Bbiluustilsss images of highest scores excluding children. While there are some visual similarities in the images, it is difficult to conclude leakage of identity in the generated synthetic dataset. We further study the effect of identity leakage on the recognition performance of face recognition models in Appendix [
Ethical Considerations: State-of-the-art face recognition models trained with large-scale face recognition datasets, which are crawled from the Internet, raising ethical and privacy concerns. To address the ethical and privacy concerns with web-crawled data, we can use synthetic data train face recognition models. However, generating synthetic face recognition datasets also requires face generator models which are trained from set of real face images. Therefore, we still rely on real face images in the generation pipeline. In our experiments, we investigated if we have leakage of identity in the generated synthetic dataset based on images used for initialization and regularization. However, there are other wriviiy-sensitive components used in our method. For example, we defined and solved our optimization problem on the embedding hypersphere of pretrained face recognition model. Therefore, for generating fully privacy-friendly datasets, the leakage of information by other components needs to be investigated.
We should also note that while we tried to increase the inter-class variations in method, there might be still a potential lack of diversity in different demography groups, stemming from implicit biases of the datasets used for training in our pipeline (such as the pretrained face recognition model, the gallery of images used for regularization, etc.). It is also noteworthy that the project on which the work has been conducted has passed Institutional Ethical Review Board (IRB).
8

---

Published as a conference paper at ICLR 2025
4 RELATED WORK
With the advances in generative models, several synthetic face recognition datasets have been pro- posed in the literature. lae et al. (2023) proposed DigiFace dataset where they used computer- graphic pipeline to render different identities and also generate different images for each identity by introducing different variations based on face attributes ge.g., variation facial pose, acces- sories, and textures). In contrast to (Bae et al. (023) other papers in the literature used Generative Adversarial Networks (GANs) or probabilistic Diffusson Models (PDMs) to generate synthetic face datasets. Qiu et al. (2021) proposed SynFace and utilised DiscoFaceGAN (Deng et al. 2020) to gen- crate their dataset. They generated different synthetic identities using identity mixup by exploring the latent space of DiscoFaceGAN to increase intra-class variation and then used DiscoFaceGAN to generate different images for each identity.
Boutros et al. (0222) proposed SFace by training identity conditione StyleGAN (Karras al. 2020) on the WASTA- (Yi et al.) 2014) and then generating the SFace dataset using the
trained model. Kolf al. (2023) also trained identity-conditioned StyleGAN (Karras al. 2020)
in three-player GAN framework to integrate the identity information into the generation process and proposed the IDnet dataset. Colbois al. (2021) proposed the Syn-Multi-PIE dataset using a pretrained StyleGAN (Karras et al. 2020). They trained a support vector machine (SVM) to find di- rections for different variations (such as pose, illuminations, etc.) in the intermediate latent space of a pretrained StyleGAN. Then, they used StyleGAN to generate different identities and synthesized different images for each identity by exploring the intermediate latent space of StyleGAN using linear combinations of calculated directions. [Boutros et al. (2023b) proposed ExFaceGAN, where they used SVM to disentangle the identity information in the latent space of pretrained GANs, and then generated different identities with several images within the corresponding identity boundaries. Geissbühler et al. (2024) used stochastic Brownian forces to sample different identities in the inter- mediate latent space of pretrained StyleGAN (Karras et al. 2020) and generate different identities (named Langavien). Then they solved a similar dynamical equation in the latent space of StyleGAN generate different images for each identity (named Langavien-Disperssiin the intermediate latent space of StyleGAN (named Sengavvien-DssCoo
and further explored
Melzi et al. 2023) proposed GANDiffFace, hybrid dataset generation framework, where they used StyleGAN to generate face images with different identities, and then used DreamBooth (Ruiz et al., 2023 as a diffusion-basee
generator, generate different samples for each identity. [oottros et al. (202a) trained an dientity-oonditiooedd diffusion model to generate synthetic face images and pro- posed IDiffFace datasets. They generated different samples using an unconditional model, and then generated different samples using their conditional diffusion model (named IDiff-Faace Two-Stage). Alternatively, they uniformly sampled different identities and generated different samples for each identity using their identity-conditioned diffusion model (named IDiff-Face Face Uniform). Kim et al. (2023) proposed DCFace, where they trained a dual condition (style and identity conditions) face generator model based on diffusion models on the -ebbaee dataset. They used their trained diffusion model to generate different identities and different styles for each identity by varying iden- tity and style conditions.
5 CONCLUSION
In this paper, we formalized the dataset generation as a packing problem on the hypersphere of a pre- trained face recognition model. We focused on inter-class variation and designed our packing prob- lem to increase the distance between synthetic identities. Then, we considered our packing problem as a regularized optimization and solved it with an iterative gradient-descent based approach. Since the manifold of face embeddings does not cover the whole hypersphere, the regularization allows us to approximate the manifold of face embeddings and enhance the quality of generated face images. We used the generated datasets by our method (called HyperFace) to train face recognition models, and evaluated the trained models on several real benchmarking datasets. Our experiments demon- strate the effectiveness of approach, which achieves state-of state-of-the-arr performance for training face recognition using synthetic data. We also presented an extensive ablation study to investigate the effect of each hyperparameter in our dataset generation method.

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research is based upon work supported by the Hasler foundation through the "Responsible Face Recognition" (SAFER) project.
REFERENCES
The rise and fall (and rise) of datasets. Nature Machine Intelligence, 4(1):1-2, 2022. doi: 10.1038/ >42256-022-00442--
022-00442-
Gwangbin Bae, Martin de La Gorce, Tadas Baltrušaitis, Charlie Hewitt, Dong Chen, Julien Valentin, Roberto Cipolla, and Jingjing Shen. Digiface-Ims million digital face images for face recog- nition. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3526-52666 3535, 2023.
K Böröozky. The problem of tammes for 11. Studia Sci. Math. Hungar, (8(2-4):155-111 1983.
Fadi Boutros, Marco Huber, Patrick Siebke, Tim Rieber, and Naser Damer. Sface: frivac--friendd and accurate face recognition using synthetic data. In 2022 IEEE International Joint Conference on Biometrics (IJCB) pp. 1-11. IEEE, 2022.
Fadi Boutros, Jonas Henry Grebe, Arjan Kuijper, and Naser Damer. Idiff-face: Synthetic-based face recognition through fizzy identiry-conditiondd diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19650-19661, 2023a.
Fadi Boutros, Marcel Klemt, Meiling Fang, Arjan Kuijper, and Naser Damer. Exfacegan: Exploring identity directions in gan's 's learned latent space for synthetic identity generation. In 2023 IEEE International Joint Conference on Biometrics (IJCB), pp. 1-10. IEEE, 2023b.
Qiong Cao, Li Shen, Weidi Xie, Omkar Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 20/8 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pp. 67-74. IEEE, 2018.
Eric Chan, Connor Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-awee 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16123-16133, 2022.
Laurent Colbois, Tiago de Freitas Percira, and Sébastien Marcel. On the use of automatically gen- erated synthetic image datasets for benchmarking face recognition. In 202/ IEEE International Joint Conference on Biometrics (IJCB), pp. 1-8. IEEE, 2021.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 4690-4699. 2019.
Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3d conference on computer vision and pattern recognition, pp. $154-5163. 2020.
learning. In Proceedings of the IEEE/CVF
David Geisshühler, Hatef Otroshi Shahreza, and Sébastien Marcel. Synthetic face datasets generation via latent space exploration from brownian identity diffusion. arXiv preprint arXiv:2405.00228, 2024.
Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb- Im: dataset and benchmark for large-scale face recognition. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pp. 87- 102. Springer, 2016.
L Hárs. The tammes problem for 10. Studia Sci. Math. Hungar, 21(3-4):433 451, 1986.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Published as a conference paper at ICLR 2025
Gary B Huang. Marwan Mattar, Tamara Berg, and Eric Leamed-Miller. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in Real-Life Images: detection, alignment, and recognition, 2008
N. J. A. Sloane J. H. Conway. Sphere Packings, Lattices and Groups. Springer New York, NY, 1998. ISBN 778-0-387-98885-5 doi: https:///oi..oo
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4401 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110-8119, 2020.
Minchul Kim, Anil K Jain, and Xiaoming Liu. Adaface: Quality adaptive margin for face recog- nition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18750-18759, 2022.
Minchul Kim, Feng Liu, Anil Jain, and Xiaoming Liu. Deface: Synthetic face generation with dual condition diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22715-12721 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), San Diego, California,, USA, May 2015.
Jan Niklas Kolf, Tim Rieber, Jurek Elliesen, Fadi Boutros, Arjan Kuijper, and Naser Damer. Identity- driven three-player generative adversarial network for synthetic-based face recognition. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 806-816, 2023.
Pietro Melzi, Christian Rathgeb, Ruben Tolosana, Ruben Vera-Rodriguez, Dominik Lawatsch, Flo- rian Domin, and Maxim Schaubert. Gandiffface: Controllable generation of synthetic datasets for face recognition with realistic variations. arXiv preprint arXiv:2305. 199.2292........... 2023.
Pietro Melzi, Ruben Tolosana, Ruben Vera-Rzd,, Minchul Kim, Christian Rathgeb, Xiaoming Liu, Ivan DeAndres-Tanme Aythami Morales, Julian Fierrez, Javier Satega-Garcia, et al. Fresyn challenge at wacv 2024: Face recognition challenge in the era synthetic data. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 892 901, 2024.
Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafciriou. Agedb: the first manually collected, will age database. In proceed- ings of the IEEE conference on computer vision and pattern recognition workshops, pp. 51-59, 2017.
Oleg R Musin and Alexey S Tarasov. The strong thirteen spheres problem. Discrete & Computa- tional Geometry, 48: 121-141, 2012.
Oleg R Musin and Alexey S Tarasov. The tammes problem for n= 14. Experimental Mathematics, 24(4):460-46. 2015.
Foivos Paraperas Papantoniou. Alexandros Lattas, Stylianos Moschoglou, Jiankang Deng, Bernhard Kainz, and Stefanos Zafeiriou. Are2face: A foundation model of human faces. arXiv preprint arXiv:2403.1 1641, 2024.
Haibo Qiu, Baosheng Yu, Dihong Gong, Zhifeng Li, Wei Liu, and Dacheng Tao. Synface: Face recognition with synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10880-10890, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695,000 2022.

---

Published as a conference paper at ICLR 2025
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuming text-to-in eext-to-image diffusion models for subject-rriven generation. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. Frontal profile face verification in the wild. In 2016 IEEE winter confer- ence on applications of computer vision (WACV), pp. 1-9. IEEE, 2016.
Hatef Otroshi Shahreza, Christophe Ecabert, Anjith George, Alexander Unnervik, Sébastien Marcel, Nicolò Di Domenico, Guido Borghi, Davide Maltoni, Fadi Boutros, Julia Vogel, et al. Sdfr: Synthetic data for face recognition competition. In 2024 IEEE 18th International Conference Automatic Face and Gesture Recognition (FG), pp. 1-9. IEEE, 2024.
Pieter Merkus Lambertus Tammes. On the origin of number and arrangement of the places of exit the surface of pollen-grains. Recueil des travaux botaniques néerlandais, 27(1):1-84, 1930.
Evgeniia Tokarchuk, Hua Chang Bakker, and Vlad Niculae. On the matter of embeddings dispersion on hyperspheres. In ICML 2024 Workshop on Geometry-groune grouuddd Representation Learning and Generative Modeling, 2024.
Mei Wang, Weihong Deng, Jiani Hu, Xunqiang Tao, and Yaohai Huang. Racial faces the wild: Reducing racial bias by information maximization adaptation network. In Proceedings of the ieee/cvf international conference on computer vision, pp. 692-702, 2019.
Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li. Learning face representation from scratch. arXiv preprint arXiv:1411.7923. 2014.
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters, 23(10): (3(10):1499911 450-, 2016.
Tianyue Zheng and Weihong Deng. Cross-poss Ifw: database for studying cross poss--poe face recog- nition in unconstrained environments. Beijing University of Posts and Telecommunicationn, Tech. Rep, 5(7), 2018.
Tianyue Zheng, Weihong Deng, and Jiani Hu. Cross-age Ifw: A database for studying cross-age face recognition unconstrained environments. arXiv preprint XXX::1708.08197, 2017.
Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang. Jiwen Lu, Dalong Du, et al. Webface260m: A benchmark unveiling the power of million- scale deep face recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattemn Recognition, pp. 10492- 10502, 2021.
12

---

Published as a conference paper at ICLR 2025
A COMPLEXITY AND REQUIRED COMPUTATION RESOURCE
The computation required to generate the synthetic datasets in our approach has two main parts:
1. Hyperface Optimization: The HyperFace optimization (Algorithm I considers all refer- ence points + in the hypersphere and maximizes their distances. Therefore, this optimization considers all pairs points and has quadratic complexity (i.e., (m))))
rablort the runtime for solving the HyperFace optimization for different numbers of identities (i.e., aad) on a system equipped with a single NVIDIA 3090 GPU. Note that this optimization process cannot be parallelized.
Table 9: Runtime for solving the HyperFace optimization (Algorithm for different numbers of identities on system equipped with a single NVIDIA 3090 GPU.
nid
HyperFace Optimization Runtime
10k
hours
20k
11 hours
30k
23 hours
50k
84 hours
We should note that instead of solving the HyperFace optimization on all pairs of points, we can solve the optimization stochastically in which in each iteration mini-batch of points is considered and optimized. Therefore the complexity will become O(b²), where b is size mini-batch and < Rid This way the complexity of our method can be independent of the number of identities and significantly reduced (especially for b < maa). Our stochastic optimization is further studied and discussed Section of this Appendix.
2. Image Generation: After solving the HyperFace optimization, we need to use the genera- tor network in inference mode and generate the required number of images. Therefore, the generation of dataset has linear complexity with respect the number of images (i.e., O (Himages) where Timuges is the number of images in the generated dataset). The average runtime for generating a single synthetic face image on a system equipped with a single NVIDIA 3090 GPU is 1.25 seconds. For example, generating dataset with 500,000 im- ages takes about 174 hours on a single NVIDIA 3090 GPU. Note that this optimization process can be parallelized, and therefore image generation can be deployed on a cluster or a farm GPUs.
B HYPERFACE STOCHASTIC OPTIMIZATION
As discussed Appendix A HyperFace optimization (Algorithm I considers all reference points {Kret, Mim and has quadratic complexity (nr)). To reduce this complexity, in each iteration, we can randomly select a mini-batch of b points and only optimize the selected b reference points instead of all ráj reference points. This way in each iteration we can compare only (2) pairs instead of (fx)) pairs, and therefore the complexity of our optimization will become (b²). In the following, we first theoretically prove that the expected mini- batch gradient approximates the full gradient, and then validate it with experimental analyses.
Theorem 1. Let X = Di represent nig points on a dix-dimensional hypersphere S. Con- sider objective function:
Rid
Rett
(((r))
(E Drg..))
Pad
where ((.,)) denotes a pairwise function. The goal is to minimize L(Xre) ) for Xief = (Z 1hu Suppose in each iteration, instead of computing V((X,r)) over all (mwl) pairs, we approximate it us- ing a random mini- batch B Xxe of size b < nia- Then, the expected batch gradient approximates the full gradient:
( v(())))
13

---

Published as a conference paper at ICLR 2025
Proof. For a batch B of size b, the batch objective is:
(x(X
1
Σ Σ
Ed Fiet,))
(5)
(2)
JENJERJDi
The full gradient of L ((xrer) is:
vel(
V Vi Cret.j)
(6)
i=1 1=
Similarly, the batch gradient is:
(L ( iEB JEB.j>i
Σ
Vilwie Fref..)
(7)
The expectation over all possible batches is:
E VL
P((i,) E B (irefs),
(8)
where p)(i.)) B] is the probability of selecting the pair (i,j) in random batch. For uniformly sampled random batches:
P((.,) E B]
(9)
By substituting P(i..) B] into the expectation in Eq. we will have:
1id
E V
Vi( Feets) = VC((
(10)
j=
Thus, the batch gradient an unbiased estimator of the full gradient.
Corollary 1. A special case for Theorem D when function E(Zng Rnd.s) is defined as follows:
dd( d( Rrga)
(i,j) argmaxx,aa.... X.yifj d( Brefs) otherwise.
URnefs Rmfs)
(11)
Therefore, we can rewrite Algorithm with stochastic optimization as presented in Algorithm 2
Algorithm HyperFace Stochastic Optimization for Finding Reference Embeddings
1: Inputs:
À learning rate, var number of iterations, (æy "gilley 9=1 embeddingg of a gallery of face images, a : hyperparameter (contribution of regularization), 6 size of mini-batch.
2:
3: Output: X = ( 1a optimized reference embeddings.
Procedure:
Initialize reference embeddings X = (are,, Ty
6: For n = 1,.., farr do
7: Sample random mini-batch B C X of size b
Sampling random mini-batch
8:
Find Beer.i, &iet, B which have minimum distance d(widts, Ber..)
9:
Reg + E (iglong d( d(Zrretks 2Pg)
b Calculate the regularization term
10:
cost +- -d d((eer, #ntt,))
11:
B + B Adam( Vcost,
12:
B + normalize(B))
To ensure that resulting embeddings B remain on the hypersphere.
13:
Update X
14:
End For
15:
End Procedure
To validate our theoretical analyses, we implement the HyperFace stochastic optimization (Algo- rithm 2 and the optimized embeddings to generate synthetic face recognition datasets. We
14

---

Published as a conference paper at ICLR 2025
consider 30,000 synthetic identities and solve HyperFace stochastic optimization (Algorithm 2 for different batch sizes. In each case, after solving the stochastic optimization, we generate 50 syn- thetic images per identity as described in (oction[ datasets to train face recognition models and evaluate the performance of trained face recognition models. Table T reports the performance of trained face recognition models. As the results in this table show face recognition models trained with datasets that are generated with stochastic mini- batch optimization achieve comparable performance the face recognition model trained with the dataset that is generated based on full- bull-batch optimization. Therefore, our experimental results meet our theoretical prediction in Theorem !
Generation). Then, use the generated
Table 10: Ablation study the effect of number of batch size in HyperFace stochastic optimization (Algorithm P
Batch Size
LFW
CPLFW
CALFW
CFP
AgeDB
1,000 (mini-batch)
98.28
85.23
91.05
91.86
89.37
5,000 (mini-batch)
98.62
84.98
90.73
90.41
88.97
30,000 (full-batch)
98.38
85.07
90.88
91.57
89.60
In terms of complexity, the HyperFace stochastic optimization (Algorithm 2 requires significantly less computation resources for solving the optimization. Table 11 reports the runtime for solving the HyperFace stochastic optimization (Algorithm 2 for different batch sizes and different numbers of identities and fixed size of gallery system equipped with single NVIDIA 3090 GPU, As the results in this table show, the complexity is independent of the number of identities (i.e., ria) and depends the size of mini-batch b. Comparing the results in Table TT and Table that our stochastic optimization significantly reduced the complexity.
we can conclude
Table 11: Runtime for solving the HyperFace stochastic optimization (Algorithm 2 for different
numbers of identities on system equipped with single NVIDIA 3090 GPU.
Batch Size (b)
# ID (na)
HyperFace Stochastic Optimization Runtime
30k
0.4 hours
1,000
50k
0.5 hours
100k
0.5 hours
30k
2.2 hours
5,000
50k
2.2 hours
100k
2.2 hours
30k
8.8 hours
10,000
50k
8.9 hours
100k
8.9 hours
C SYNTHETIC DATASETS AT SCALE
In To the paper, compared our face recognition models trained with our generated dataset and synthetic datasets in the literature. For previous datasets, we considered the available version of each dataset which has similar number of identities (10k). In Table 3 we studied the effect of the number of identities in our dataset generation, where the results showed that we can scale our synthetic dataset and achieve higher recognition performance. In Table 12, we compare the performance of face recognition models trained with our generated datasets and with all publicly available versions (particularly larger scale) of synthetic datasets the literature. As the results in this table show, our generated datasets achieve competitive performance with synthetic datasets in the literature at scale. Comparing different datasets in the literature, DCFace, which outperformed previous datasets in Table I does not achieve the best performance larger version. In contrast, Langevin-Diss -DiCCo achieves a significant improvement for its larger version with 30k identities compared to its smaller version with 10k identities. However, Geissbühler et al. (2024) reported a lower performance for their dataset with 50k identities compared to 30k identities, indicating limitations in further scaling the Langevin-Diste DisCoo dataset for more than 30k identities.
any of the benchmarks for its
15

---

Published as a conference paper at ICLR 2025
Table 12: Comparison of recognition performance of face recognition models trained with the largest available versions of different synthetic datasets as well as a real dataset (i.e., CASIA- WebFace). The performance reported for each dataset is in terms of accuracy and best value for each benchmark is emboldened.
Dataset name
IDs
Images
LFW CPLFW
CALFW
CFP
AgeDB
SynFace (Qiu et al. 202T
10'000
999'994
86.57
65.10
70.08
66.79
59.13
SFace (Boutros
10'572
1'885'877 877
93.65
74.90
80.97
75.36
70.32
Syn-Multi- PIE Colbois at 2021
10'000
180'000
78.72
60.22
61.83
60.84
54.05
IDnet (Kalf et 2023
107577
1'057'200
84.48
68.12
71.42
68.93
62.63
ExFaceGAN Bouiros al. 2023b
10'000
599'944
85.98
66.97
70.00
66.96
57.37
GANDiffFace Melzi etal 2023
10'080
543'893
94.35
76.15
79.90
78.99
69.82
Langevin-Diispersion (Geisshühler al.
10'000
650'000
94.38
65.75
86.03
65.51
77.30
Langevin-DisCo (Geissbuhler et al. 2024
10'000
650'000
97.07
76.73
89.05
79.56
83.38
Langevin-DisCo (Giisbobler 2024
30'000
1'950'000
98.97
81.52
93.95
83.77
93.32
DigiFace- MM ctal 2023
109'999
1'219'995
90.68
72.55
73.75
79.43
68.43
IDiff-Face (Uniform)
Boulrus 20255
10"049
502'450
98.18
80.87
90.82
82.96
85.50
IDiff-Face (Two-Stage) Boutros et al. 2023a
10'050
502'500
98.00
77.77
88.55
82.57
82.35
DCFace (Kim et al. 2023
10'000
500'000
98.35
83.12
91.70
88.43
89.50
DCFace (Kim et 2023
60'000
1'200'000
98.90
84.97
92.80
89.04
91.52
HyperFace [ours]
10'000
640'000
98.67
84.68
89.82
89.14
87.07
HyperFace [ours]
50'000
3'200'000
98.27
85.60
91.48
92.24
90.40
CASTA-WebFace at 2014
10'
490'623
99.42
90.02
93.43
94.97
94.32
Nevertheless, our method achieves improvement by scaling the number of identities (Table 3 In particular, our dataset with 50k identities and 3.2M images achieves competitive performance with large-scale synthetic datasets in the literature.
D IDENTITY LEAKAGE AND RECOGNITION PERFORMANCE
In Section we discussed identity leakage in the generated face datasets. While the leakage of identity is not evident in the generated dataset, it is important to see if identity leakage may affect the recognition performance of face recognition models. To this end, AesbA- datasets as two real face datasets and compare all possible pairs from our syn- thetic dataset with images in the real datasets. Then, for each of these real datasets, find the top-200 pairs (synthetic-real) and exclude the corresponding synthetic image from our generated dataset. This ensures that images which may contain identity leakage are excluded from the final synthetic datasets. We use the resulting cleaned datasets to train face recognition models and com- pare them with the face recognition model trained our original synthetic dataset. Table 13 reports the recognition performance of face recognition models trained with original and cleaned synthetic datasets.
consider the FFHQ and
Table 13: Evaluation of potential identity leakage on the recognition performance.
Synthetic Dataset
LFW
CPLFW
CALFW
CFP
AgeDB
cleaned (Ref.: CASIA- WebFace)
98.52
84.80
89.52
89.43
87.00
cleaned (Ref.: FFHQ)
98.77
84.53
89.35
89.36
86.67
original
98.67
84.68
89.82
89.14
87.07
As the results in Table 13 removing images with similar identities does not impact the recognition performance of the trained face recognition model. However, would like to highlight that while identity leakage may not affect recognition performance on benchmark datasets, it is an important privacy concern.
16

---

Published as a conference paper at ICLR 2025
E ADDITIONAL ABLATION STUDY
In Section 3 we reported ablation studies on different hyperparameters in our dataset generation. As new experiment, consider different optimizers for solving HyperFace optimization (full batch). We consider RMSprop, Adam, and AdamW W optimizers. Table [4 compares the performance of the face recognition model trained with datasets that are generated based on different optimizers in HyperFace optimization. As the results in this table show, solving HyperFace optimization with different optimizers leads comparable performance.
Table 14: Ablation study on optimizer
Optimizer
LFW
CPLFW
CALFW
CFP
AgeDB
RMSprop
98.47
84.73
89.18
89.27
86.83
AdamW 98.70
84.20
89.02
89.41
86.38
Adam 98.67
84.68
89.82
89.14
87.07
As another experiment, we generate random points on the hypersphere and use random points as reference embeddings without HyperFace optimization to generate synthetic dataset. We ensure that selected points have at least 0.3 cosine distance. Table 15 compares the performance of face recognition model trained with the dataset based random embeddings and HyperFace op- timization. As the results this table show, solving HyperFace optimization achieves superior performance all benchmarks. Note that with random selection of points the hypersphere, there is no guarantee to be on the manifold of embeddings of the face recognition model. However, with HyperFace optimization, we try to keep points the face recognition manifold, which results in dataset that leads to better performance.
Table 15: Ablation study on using random points vs HyperFace optimization.
Reference Embeddings
LFW
CPLFW
CALFW
CFP
AgeDB
Random
98.17 98.12
83.67
86.67
88.79
84.25
HyperFace Optimization
98.67
84.68
89.82
89.14
87.07
F HYPERFACE DATASET GENERATION
We described HyperFace dataset generation process in our method.
Algorithm summarizes the dataset generation
Algorithm HyperFace Dataset Generation
1: Inputs: Mia number of synthetic identities, nssmmll number sample images per identity,
2:
G face generator model, / hyperparameter (controls variations in embeddings) 3: Output: Diiyputfae = (7) generated dataset.
4: Procedure:
Solve HyperFace optimization find reference embeddings X..r
Algorithm
D
6: Initialize Dispotwwe []
7: For æset { 1 do
8: For n = 1,.., Rsaapic do
9:
Sample Gaussian noise ~ (O 1PM)
D For diffusion model C D For variations in the embedding arat Store the generated image the dataset Dtype
10:
Sample Gaussian noise ~ N/( 1"x
11:
Generate synthetic image = G(
Fa + | (oo
12:
Dtsyerxe..apendL1
13:
End For
14:
End For
15:
End Procedure
17

---

Published as a conference paper at ICLR 2025
G
VISUALIZATION
Figure illustrates sample face images from the HyperFace dataset. In addition, Figure and
Figure also show intra-class variations for two synthetic identities in the HyperFace dataset.
Figure 4: Sample face images of different synthetic identities from the HyperFace dataset.
18

---

Published as a conference paper at ICLR 2025
Figure 5: Sample face images of one subject from the HyperFace dataset (intra-class variations).
Figure 6: Sample face images of one subject from the HyperFace dataset (intra-class variations).
19


Reviews:
[{"review_id": "GmY9TcCU5h", "reviewer": "Reviewer_khYx", "summary": "The paper presents an approach to generate a synthetic face dataset for face recognition problem by formulating it as an optimization problem and solve it iteratively via gradient descent method to obtain optimized embeddings. Then the synthesize face images can be generated using pre-trained face generator models from those embeddings.\nThe experiment shows that the models trained with the proposed synthetic datasets can achieve SOTA performance.", "questions": "1. What is the computation resource and time needed to generate larger scale datasets, e.g. n_id = 30k or more?\n2. It would be interesting to see if we can use the FR model trained by the proposed synthetic dataset to build another good synthetic dataset", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Well-formulated optimization problem: The paper effectively defines the optimization problem for generating high-quality synthetic face recognition datasets.\n2. Efficient solution: The proposed solution is not only effective but also computationally efficient.\n3. Extensive experiments: The paper presents comprehensive experiments on various synthetic datasets to validate the approach.\n4. Ethical considerations: The authors acknowledge potential ethical concerns, such as identity leakage, demonstrating a responsible approach to AI development.", "weaknesses": "1. Limited scale: While the experiments on datasets with up to 30K identities provide valuable insights, evaluating the method's performance on significantly larger datasets is crucial.  Datasets with 100K and 300K identities would be particularly informative as they would reveal how the method scales and whether performance degrades with increased identity count and potential data noise. This would provide a more comprehensive understanding of the method's robustness and real-world applicability.\n2. Narrow focus:  The paper's focus on improving inter-class variation is important, but expanding the scope to address intra-class variability would significantly enhance its impact. Specifically, exploring how the method handles variations in pose, illumination, and expression within the same identity would be valuable. Additionally, investigating the method's robustness to occlusions or image quality degradation would further strengthen the evaluation and demonstrate its potential for real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wGudiOHYkG", "reviewer": "Reviewer_UATk", "summary": "Interesting paper that proposes some embedding optimisation in the latent space of a pretrained face recognition model. The optimised embeddings are used for generating facial images using a recently proposed generative model, and then the generated images for training a face recognition model. I liked the novelty and simplicity of the proposed approach yet there are a few issues that possibly limit the impact of the proposed work. See my questions below.", "questions": "1. Are both reference embeddings and gallery embeddings generated from StyleGan?  In this case the only difference between them is that the Gallery embeddings are not updated during optimisation?\n2. Are all methods in table 1 trained in the same way using the same face recognition model and training pipeline?\n3. Fair comparison in table 1 should use 50 images per identity for your method\n4. It’s important to compare against SOTA (e.g. DCFace) at scale (i.e. increasing the number of identities). Specifically, table 3 should not be just an ablation study but you need to show that your method scales favorably and/or outperforms SOTA as the number of training images increases. \n5. In general, how the method scales has been poorly studied (there’s only 1 result in table 3). Scaling Dataset Generation section discusses computational issues that may arise from scaling the dataset but does not provide concrete numbers (e.g. a figure showing training time vs dataset size), conclusions or practical solutions (i.e. a solution is proposed but not put in practice)\n6. Baselines: what about direct comparison with arc2face? Since they don’t have to solve your optimisation problem, they can generate a lot more images for training", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Interesting idea to perform the optimization in the latent space of a discriminatively train face recognition model\n- Well written paper, easy to read and understand\n- Decent experiments although lacking in some respects", "weaknesses": "It could be that the method has significant limitations in terms of scaling the number of images that can be generated. The impact of the work has not be fully demonstrated. See also my questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qkSSfbqeSO", "reviewer": "Reviewer_YfZL", "summary": "This paper propose an interesting solution for synthetic face recognition, i.e. optimizing the hyperspace for generation. The solution is to treat the face generation as a packing problem on the embedding space. The optimization goal is to find feature embedding that is of high inter-class variation. Finally this paper adopt a pretrained generation method to generate the dataset.", "questions": "Please see the weakness. \n\nIf the authors address the concerns well, I am happy to raise my score.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "1. This paper is good in writing and has a solid mathematical formulation.", "weaknesses": "1. [Major] I don't see the motivation to convert the face generation problem as a packing problem on the embedding space from the storytelling, Please provide some related empirical/ theoretical works regarding why the packing problem could be of use for SFR.\n\n2. [Major] This paper adopts Arc2Face for final image generation. However, (1) The ablation study doesn't show the advance of directly sending random feature embedding (each embedding is ensured by restricting the similarity below a certain threshold, e.g. 0.3) to Arc2Face; (2) The comparison with Arc2Face is missing in Table 1, additionally the experiment is marginal better than DCFace. The average performance is 89.876 which is similar to DCFace and dramatically lower than Arc2Face. \n\n3. [Major] In Section 'Solving the HyperFace Optimization', the authors choose AdamW for the optimization solution. However, the other alternative optimization methods are not specified and compared in this paper.\n\n4. Another concern is that the proposed method generates more images(640k) to produce similar performance to DCFace (500k).\n\n5. large intra-class variations can not be observed in the visualization section.\n\n6. [Minor] Notation is not specified in fig 1.2. Please provide more description for the reader to understand the mathematical formulation and the whole generation process. For example, what does reference embedding stand for, I understood it only when I saw the 'Image Generation' section. And what is X_{g}?\n\n7. Please give some detailed pseudo-code for the entire process(training/ generation) for the reader to understand the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D0JkhImCQ3", "reviewer": "Reviewer_RHfp", "summary": "This paper proposes a synthetic data generation method for FR that aims to improve inter-class variation compared to existing methods. The approach utilizes the embedding space of a pretrained FR model to create an initial gallery, then optimizes these embeddings to uniformly position identities on a unit hypersphere. A conditional diffusion generator is subsequently used to synthesize faces.", "questions": "My main concerns are threefold: (1) the lack of theoretical and empirical analysis on HyperFace optimization (Equation 1), (2) missing ablations and detailed analysis of results, and (3) understanding the limitations of the method in generating novel identities. I am hopeful these concerns can be addressed during the discussion period, and I am open to increasing my score based on the responses. Please also see the weaknesses", "limitations": "", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The empirical study shows that the proposed method generalizes well across different validation sets.\n3. Experimental results are promising, with satisfactory improvements on benchmark datasets.", "weaknesses": "1. HyperFace Optimization: Figure 2 shows that HyperFace optimization results in a uniform distribution of points on the hypersphere. However, the paper lacks evidence, analysis, or experiments demonstrating that Equation 1 is minimized by this uniform distribution. See [1] for more details.\n\n2. Use of Hyperspherical Points: One key aspect of the method is the use of uniformly positioned points on the hypersphere. While using a pre-trained FR model for generating the identity gallery is reasonable, it would be helpful to see results without applying HyperFace (using pure FR embedding as condition). How would this affect the results?\n\n3. Experimental Section: The experiments section needs major revisions. Most of the subsections only report table results without in-depth discussion or analysis. For a conference of ICLR’s quality, it’s important to explain specific behaviors and insights from the proposed method.\n\n4. Additional Experiments: Additional experiments could improve clarity on the benefits of the method. For example, while Table 3 presents an ablation study on the number of identities on FR performance, it would also be valuable to show how many novel identities the method can generate (what is the saturation point for the number of novel identities). \n\n5. Identity Leakage: The paper mentions identity leakage but lacks in-depth experiments on the synthesized data. What would the performance look like if synthesized images with high similarity to real datasets (e.g., CASIA) were excluded?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)