HyperFace: Generating Synthetic Face Recognition Datasets by Exploring Face Embedding Hypersphere

Authors: Hatef Otroshi Shahreza, Sébastien Marcel
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Face Recognition, Hypersphere Optimization, Privacy, Synthetic Data

Abstract:
Face recognition datasets are often collected by crawling Internet and without individuals' consents, raising  ethical and privacy concerns. Generating synthetic datasets for training face recognition models has emerged as a promising alternative. However, the generation of synthetic datasets remains  challenging as it entails adequate inter-class and intra-class variations. While advances in generative models have made it easier to increase intra-class variations in face datasets (such as pose, illumination, etc.), generating sufficient inter-class variation is still a difficult task. In this paper, we formulate the dataset generation as a packing problem on the embedding space (represented on a hypersphere) of a face recognition model and propose a new synthetic dataset generation approach, called HyperFace. We formalize our packing problem as an optimization problem and solve it with a gradient descent-based approach. Then, we use a conditional face generator model to synthesize face images from the optimized embeddings. We use our generated datasets to train face recognition models and evaluate the trained models on several benchmarking real datasets. Our experimental results show that models trained with HyperFace achieve state-of-the-art performance in training face recognition using synthetic datasets. Project page: https://www.idiap.ch/paper/hyperface

Decision: accept (poster)
Reviewer scores: 6, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730042826918

thread_type:
official_review

<thread_object>
creator: reviewer_rhfp

<content>
summary:
This paper proposes a synthetic data generation method for FR that aims to improve inter-class variation compared to existing methods. The approach utilizes the embedding space of a pretrained FR model to create an initial gallery, then optimizes these embeddings to uniformly position identities on a unit hypersphere. A conditional diffusion generator is subsequently used to synthesize faces.

strengths:
1. The paper is well-written and easy to follow.
2. The empirical study shows that the proposed method generalizes well across different validation sets.
3. Experimental results are promising, with satisfactory improvements on benchmark datasets.

weaknesses:
1. HyperFace Optimization: Figure 2 shows that HyperFace optimization results in a uniform distribution of points on the hypersphere. However, the paper lacks evidence, analysis, or experiments demonstrating that Equation 1 is minimized by this uniform distribution. See [1] for more details.

2. Use of Hyperspherical Points: One key aspect of the method is the use of uniformly positioned points on the hypersphere. While using a pre-trained FR model for generating the identity gallery is reasonable, it would be helpful to see results without applying HyperFace (using pure FR embedding as condition). How would this affect the results?

3. Experimental Section: The experiments section needs major revisions. Most of the subsections only report table results without in-depth discussion or analysis. For a conference of ICLR’s quality, it’s important to explain specific behaviors and insights from the proposed method.

4. Additional Experiments: Additional experiments could improve clarity on the benefits of the method. For example, while Table 3 presents an ablation study on the number of identities on FR performance, it would also be valuable to show how many novel identities the method can generate (what is the saturation point for the number of novel identities). 

5. Identity Leakage: The paper mentions identity leakage but lacks in-depth experiments on the synthesized data. What would the performance look like if synthesized images with high similarity to real datasets (e.g., CASIA) were excluded?

questions:
My main concerns are threefold: (1) the lack of theoretical and empirical analysis on HyperFace optimization (Equation 1), (2) missing ablations and detailed analysis of results, and (3) understanding the limitations of the method in generating novel identities. I am hopeful these concerns can be addressed during the discussion period, and I am open to increasing my score based on the responses. Please also see the weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730301358138

thread_type:
official_review

<thread_object>
creator: reviewer_yfzl

<content>
summary:
This paper propose an interesting solution for synthetic face recognition, i.e. optimizing the hyperspace for generation. The solution is to treat the face generation as a packing problem on the embedding space. The optimization goal is to find feature embedding that is of high inter-class variation. Finally this paper adopt a pretrained generation method to generate the dataset.

strengths:
1. This paper is good in writing and has a solid mathematical formulation.

weaknesses:
1. [Major] I don't see the motivation to convert the face generation problem as a packing problem on the embedding space from the storytelling, Please provide some related empirical/ theoretical works regarding why the packing problem could be of use for SFR.

2. [Major] This paper adopts Arc2Face for final image generation. However, (1) The ablation study doesn't show the advance of directly sending random feature embedding (each embedding is ensured by restricting the similarity below a certain threshold, e.g. 0.3) to Arc2Face; (2) The comparison with Arc2Face is missing in Table 1, additionally the experiment is marginal better than DCFace. The average performance is 89.876 which is similar to DCFace and dramatically lower than Arc2Face. 

3. [Major] In Section 'Solving the HyperFace Optimization', the authors choose AdamW for the optimization solution. However, the other alternative optimization methods are not specified and compared in this paper.

4. Another concern is that the proposed method generates more images(640k) to produce similar performance to DCFace (500k).

5. large intra-class variations can not be observed in the visualization section.

6. [Minor] Notation is not specified in fig 1.2. Please provide more description for the reader to understand the mathematical formulation and the whole generation process. For example, what does reference embedding stand for, I understood it only when I saw the 'Image Generation' section. And what is X_{g}?

7. Please give some detailed pseudo-code for the entire process(training/ generation) for the reader to understand the method.

questions:
Please see the weakness. 

If the authors address the concerns well, I am happy to raise my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730636981488

thread_type:
official_review

<thread_object>
creator: reviewer_uatk

<content>
summary:
Interesting paper that proposes some embedding optimisation in the latent space of a pretrained face recognition model. The optimised embeddings are used for generating facial images using a recently proposed generative model, and then the generated images for training a face recognition model. I liked the novelty and simplicity of the proposed approach yet there are a few issues that possibly limit the impact of the proposed work. See my questions below.

strengths:
- Interesting idea to perform the optimization in the latent space of a discriminatively train face recognition model
- Well written paper, easy to read and understand
- Decent experiments although lacking in some respects

weaknesses:
It could be that the method has significant limitations in terms of scaling the number of images that can be generated. The impact of the work has not be fully demonstrated. See also my questions below.

questions:
1. Are both reference embeddings and gallery embeddings generated from StyleGan?  In this case the only difference between them is that the Gallery embeddings are not updated during optimisation?
2. Are all methods in table 1 trained in the same way using the same face recognition model and training pipeline?
3. Fair comparison in table 1 should use 50 images per identity for your method
4. It’s important to compare against SOTA (e.g. DCFace) at scale (i.e. increasing the number of identities). Specifically, table 3 should not be just an ablation study but you need to show that your method scales favorably and/or outperforms SOTA as the number of training images increases. 
5. In general, how the method scales has been poorly studied (there’s only 1 result in table 3). Scaling Dataset Generation section discusses computational issues that may arise from scaling the dataset but does not provide concrete numbers (e.g. a figure showing training time vs dataset size), conclusions or practical solutions (i.e. a solution is proposed but not put in practice)
6. Baselines: what about direct comparison with arc2face? Since they don’t have to solve your optimisation problem, they can generate a lot more images for training
</content>
</thread_object>

</thread>
<thread>
creation date:
1730726925527

thread_type:
official_review

<thread_object>
creator: reviewer_khyx

<content>
summary:
The paper presents an approach to generate a synthetic face dataset for face recognition problem by formulating it as an optimization problem and solve it iteratively via gradient descent method to obtain optimized embeddings. Then the synthesize face images can be generated using pre-trained face generator models from those embeddings.
The experiment shows that the models trained with the proposed synthetic datasets can achieve SOTA performance.

strengths:
1. Well-formulated optimization problem: The paper effectively defines the optimization problem for generating high-quality synthetic face recognition datasets.
2. Efficient solution: The proposed solution is not only effective but also computationally efficient.
3. Extensive experiments: The paper presents comprehensive experiments on various synthetic datasets to validate the approach.
4. Ethical considerations: The authors acknowledge potential ethical concerns, such as identity leakage, demonstrating a responsible approach to AI development.

weaknesses:
1. Limited scale: While the experiments on datasets with up to 30K identities provide valuable insights, evaluating the method's performance on significantly larger datasets is crucial.  Datasets with 100K and 300K identities would be particularly informative as they would reveal how the method scales and whether performance degrades with increased identity count and potential data noise. This would provide a more comprehensive understanding of the method's robustness and real-world applicability.
2. Narrow focus:  The paper's focus on improving inter-class variation is important, but expanding the scope to address intra-class variability would significantly enhance its impact. Specifically, exploring how the method handles variations in pose, illumination, and expression within the same identity would be valuable. Additionally, investigating the method's robustness to occlusions or image quality degradation would further strengthen the evaluation and demonstrate its potential for real-world scenarios.

questions:
1. What is the computation resource and time needed to generate larger scale datasets, e.g. n_id = 30k or more?
2. It would be interesting to see if we can use the FR model trained by the proposed synthetic dataset to build another good synthetic dataset
</content>
</thread_object>

</thread>
<thread>
creation date:
1732182884994

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Authors General Response to Reviewers

comment:
We thank all reviewers for their time and valuable comments which helped us improve the quality of our paper. We carefully considered every point raised by the reviewers and revised the paper accordingly and provided point-by-point responses in our reply to each reviewer. 


The summary of important changes in the revised version of the paper is as follows:

- We generated a larger dataset with 50k identities and 3.2M images. The new dataset achieves better performance and shows the scalability of our dataset generation. We added the results to Table 3 of the paper.
- We provide an evaluation of the complexity and computation requirement for generating synthetic datasets with our method in Appendix A of the paper. As described in our initial submission, the complexity of HyperFace optimization is quadratic with respect to the number of identities. However, in our initial submission, we suggested that it can be significantly reduced by stochastic optimization. This aspect of our work received the attention of reviewers who requested more in-depth analyses. Therefore, in Appendix B of the paper, we first theoretically show that stochastic optimization leads to similar results but with less complexity, and then evaluate it with experiments. Our experimental results meet our theoretical prediction and stochastic optimization significantly reduces the complexity while maintaining performance.
- We provided a comparison of synthetic datasets in the literature at scale in Appendix C, where our dataset acheives competetive performance with largest synthetic datasets in the literature.
- We investigate the effect of identity leakage on the recognition performance of face recognition models trained with synthetic datasets in Appendix D.
- We also provided additional ablation studies to investigate the effectiveness of our method in Appendix E.
- We provided pseudo-code for the entire data generation process in Appendix F and provided more sample images in Appendix G.
- We expanded our discussions in the experiments section.


We hope we could adequately address the reviewers' concerns. 
We kindly invite all reviewers to further discussions if any part remains unclear.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a method for generating synthetic face datasets by optimizing identity embeddings on a hypersphere using gradient descent, followed by image synthesis with a generative model. It demonstrates improved inter-class variation and competitive performance on face recognition benchmarks. A well-defined optimization framework, extensive experiments, and scalability analysis are the main strengths of this paper. There are also several limitations of this paper, including limited exploration of intra-class variation, insufficient empirical/theoretical insights into optimization, and weaker performance on age-related benchmarks. Despite these gaps, the paper makes a meaningful contribution to synthetic data generation for face recognition, supporting its merit for acceptance with further refinements.