UIFace: Unleashing Inherent Model Capabilities to Enhance Intra-Class Diversity in Synthetic Face Recognition

Authors: Xiao Lin, Yuge Huang, Jianqing Xu, Yuxi Mi, Shuigeng Zhou, Shouhong Ding
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: face recognition, face image synthesis, diffusion model

Abstract:
Face recognition (FR) stands as one of the most crucial applications in computer vision. The accuracy of FR models has significantly improved in recent years due to the availability of large-scale human face datasets. However, directly using these datasets can inevitably lead to privacy and legal problems. Generating synthetic data to train FR models is a feasible solution to circumvent these issues. While existing synthetic-based face recognition methods have made significant progress in generating identity-preserving images, they are severely plagued by context overfitting, resulting in a lack of intra-class diversity of generated images and poor face recognition performance. In this paper, we propose a framework to $\textbf{U}$nleash model $\textbf{I}$nherent capabilities to enhance intra-class diversity for synthetic face recognition, shorted as $\textbf{UIFace}$. Our framework first train a diffusion model that can perform denoising conditioned on either identity contexts or a learnable empty context. The former generates identity-preserving images but lacks variations, while the latter exploits the model's intrinsic ability to synthesize intra-class-diversified images but with random identities. Then we adopt a novel two-stage denoising strategy to fully leverage the strengths of both type of contexts, resulting in images that are diverse as well as identity-preserving. Moreover, an attention injection module is introduced to further augment the intra-class variations by utilizing attention maps from the empty context to guide the denoising process in ID-conditioned generation. Experiments show that our method significantly surpasses previous approaches with even less training data and half the size of synthetic dataset. More surprisingly, the proposed $\textbf{UIFace}$ even achieves comparable performance of FR models trained on real datasets when we increase the number of synthetic identities.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730645490465

thread_type:
official_review

<thread_object>
creator: reviewer_zdsw

<content>
summary:
The paper addresses the challenge of generating diverse synthetic face images for the purpose of training face recognition models. The authors introduce a novel framework named UIFace, which employs a two-stage denoising approach alongside an attention injection module. This combination is designed to increase intra-class diversity while preserving identity consistency. As a result, UIFace achieves performance that is competitive with models trained on real-world datasets.

strengths:
1. The two-stage denoising strategy and the attention injection module work together to generate diverse images while maintaining identity preservation, effectively addressing the issue of context overfitting.
2. The adaptive partitioning strategy dynamically identifies the optimal moment to transition between the two denoising stages, thereby enhancing both the quality and diversity of the generated images.
3. UIFace demonstrates its effectiveness by achieving performance on par with models trained on real datasets.

weaknesses:
1. What about the scalability of the model? Are there any potential benefits to further increasing the number of identities beyond the 20K shown in Table 1?
2. While this paper focuses on enhancing the intra-class diversity for a specific identity, what about the inter-class discrepancies, which are also critical for training an effective face recognition model? How do the authors ensure this aspect is adequately addressed in their work?
3. The details of the overall training pipeline could be more clearly explained. For instance, in Stage 2, how are the losses from the unconditional and conditional branches calculated and combined?

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682698649

thread_type:
official_review

<thread_object>
creator: reviewer_jbls

<content>
summary:
This paper tries to solve the problem of intra-class diversity with the generation of synthetic images in face recognition, and proposes a two-stage pipeline framework of DDPM with two improvements, including adaptive stage partition strategy and attention injection module. Solid results are achieved.

strengths:
Originality: The method was proposed upon the previous DDPM with two tricks raised by the observation that early stage and later stage of denoising step can capture different semantic meanings of images.  
Quality: The methodology is conceptually simple with massive experiments supported. The solid results have achieved on public benchmarks. 
Clarify: The idea, albeit simple, has been clearly presented. It is better to re-structure introduce and related work slightly by minimizing the depiction of FR while maximizing the work related to data synthesis by generation methods.  
Significance: The authors declared that the proposed methods achieved sota performance of face recognition by training models with synthetic images. There is no future work being declared to improve the work.

weaknesses:
1. The method is proposed upon the previous DDPM work by using identity context maps as conditions. Two improvements, including adaptive partitioning strategy and attention injection module, have been added to enhance the performance of the methods. 
2. The authors declared they proposed a framework of face image synthesis by generating methods. Can authors clarify if the framework is suitable for other image generation tech, besides DDPM.
3. The paper is well-structured. However, there are certain grammar and typo errors.
4.  For introduction and related work sections, the description of background can go more specific to synthetic data-based face recognition rather than face recognition. 
5. More recent face image synthetic methods should be investigated. 
6. The authors have implied that synthetic images can solve privacy and legal issues. Please clarify how your methods can avoid privacy and legal issues, since synthetic images are generated on real-world images. 
7. Extra experiments should be conducted to support the methods raised by this work. For example, different settings of fixed t_{0} should be examined, also "baseline + 2-stage-fixed + attn" should be conducted in the ablation experiment.

questions:
1. In this abstract section, can authors explain how the methods raised in this paper preserve the privacy and legality. If this is not related to the research purpose, it is better to exclude it in case of misleading readers.  
2. There were more methods proposed upon IDiff and had better performance, for example, “SDFR: Synthetic Data for Face Recognition Competition”.
3. Line 308, ... by leveraging... Carefully check the grammar and typo errors.
4. It is not very clear about timestep interval T analysis in Figure. The interval upperbound of the x-axis is incrementally increased or resets as well the interval lowerbound.
5. For experimentation, different settings of fixed t_{0} should be examined, also "baseline + 2-stage-fixed + attn" should be conducted.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730725606424

thread_type:
official_review

<thread_object>
creator: reviewer_trrb

<content>
summary:
This paper addresses the challenge of improving synthetic data generation for face recognition (FR) by tackling the issue of context overfitting, which leads to low intra-class diversity and diminished performance. The authors introduce a novel framework named UIFace, designed to enhance intra-class diversity while maintaining identity preservation. The approach begins by training a diffusion model capable of denoising under identity-conditioned contexts or a learnable empty context. The identity context generates images that maintain identity but lack variation, while the empty context leverages the model's intrinsic ability to produce diverse images, albeit with random identities.

strengths:
1. UIFace seems to be closer to the dataset diversity of real datasets than the previous method IDiff-face.
2. better results can be achieved using less amount of synthetic data, and with increasing amount of data even approaching the performance of direct datasets

weaknesses:
1. The main contribution of the paper lies in the observation that LDM learns different aspects during both pre-training and post-training. However, only the analysis based on Figure 2 is currently insufficient, and there is a lack of theoretical discussion. 

2. The training and testing times for the two-stage LDM should be provided for comparison with the one-stage methods.

3. The font size of the axis labels in Figure 2 is too small, and it appears that the author has not used \citep and \citet correctly.

questions:
The authors are encouraged to include a theoretical analysis explaining how LDM learns different subjects at different training time steps.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732281255260

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We are very grateful for the time and valuable suggestions provided by all reviewers and we have provided an individual response to each reviewer. Here we want to highlight some general issues.

**Contributions**: The contributions of our paper compared to previous works are as follows:
- **motivation**: Our method is motivated by the diffusion model's different focus of identity-relevant and irrelevant attributes of different timesteps during sampling, which, to the best of our knowledge, has not been revealed or discussed in previous works.
-  **techniques**: We propose a novel two-stage sampling strategy and an attention injection method, which can effectively address the diversity gap between synthetic and real datasets by fully leveraging the model’s inherent capabilities. Such a gap caused by context overfitting has significantly hampered the accuracy of previous synthetic-based face recognition methods compared to FR models trained on real datasets.
-  **Experimental contributions**: We quantitatively validate that our method can enhance the diversity of synthetic face datasets. Moreover, by generating more diverse datasets, our method can achieve state-of-the-art performance with even half the number of identities compared to previous methods. When further increasing the number of identities, our method can even match the performance of FR models trained on real datasets, demonstrating the scalability and effectiveness of our method.



**Rebuttal Revision**: Based on the suggestions of reviewers, we have made the following modifications in our rebuttal revision (marked in blue):

- Increase the font size of Figure 2 to make the presentation clearer (Reviewer TRrB)
- Revise all citations to correctly use \citet and \citep (Reviewer TRrB)
- Correct spelling and grammar errors as much as possible (Reviewer JbLs)
- Emphasize the explanation about generating non-existent identities to prevent misunderstandings about privacy issue (Reviewer JbLs)
- Emphasize how our method ensures inter-class discrepancy (Reviewer zDSW)

**More Analysis and Experiments**: We greatly appreciate the reviewers' valuable suggestions to improve the analysis and experiments of our paper including

- More theoretical discussion about our motivation (Reviewer TRrB)
- Analysis about training and testing times (Reviewer TRrB)
- Experiments on more generative methods, comparison with other recent methods and more ablation studies (Reviewer JbLs)
- Involving experiments with a broader range of identities to validate the scalability of our method (Reviewer zDSW)

We have completed the analysis and experiments above and individually responded to each reviewer. These suggestions really enhance the quality of our paper, and we will incorporate them into the main body or appendix in later version.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a technique aimed at improving the diversity of synthetic datasets used for training face recognition (FR) models by presenting an innovative framework that employs a two-stage denoising process. This approach utilizes both identity-specific and non-identity-specific contexts to generate synthetic images, effectively addressing the common issue of context overfitting found in existing synthetic data methods. As a result, the method enhances intra-class diversity and boosts performance. Reviewers appreciate the motivation behind the paper but raise concerns regarding the clarity of the method’s description. The paper suggests that the training pipeline uses a two-stage strategy during the sampling phase but fails to provide a clear explanation of how the losses from the unconditional and conditional branches are calculated and integrated into each stage. After reading the paper and discussion, AC is also confused about the two-stage sampling process. Does each update need Eq(5) to decide (t0) which stage it is? To enhance the paper’s comprehensibility and reproducibility, a more detailed description of this process is essential in the final revision.