$\text{ID}^3$: Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 05:36:45

Abstract: Synthetic face recognition (SFR) aims to generate synthetic face datasets that mimic the distribution of real face data, which allows for training face recognition models in a privacy-preserving manner. Despite the remarkable potential of diffusion models in image generation, current diffusion-based SFR models struggle with generalization to real-world faces. To address this limitation, we outline three key objectives for SFR: (1) promoting diversity across identities (inter-class diversity), (2) ensuring diversity within each identity by injecting various facial attributes (intra-class diversity), and (3) maintaining identity consistency within each identity group (intra-class identity preservation). Inspired by these goals, we introduce a diffusion-fueled SFR model termed $\text{ID}^3$. $\text{ID}^3$ employs an ID-preserving loss to generate diverse yet identity-consistent facial appearances. Theoretically, we show that minimizing this loss is equivalent to maximizing the lower bound of an adjusted conditional log-likelihood over ID-preserving data. This equivalence motivates an ID-preserving sampling algorithm, which operates over an adjusted gradient vector field, enabling the generation of fake face recognition datasets that approximate the distribution of real-world faces. Extensive experiments across five challenging benchmarks validate the advantages of $\text{ID}^3$.

Main paper: 
  
 Appendix:  


Decision: To avoid privacy problem, this paper proposes ID3, an identity-preserving-yet-diversified diffusion model for generating synthetic face data for face recognition. The proposed method leverages identity embeddings and facial attributes to control inter-class and intra-class diversity of generated faces while preserving intra-class identity consistency, demonstrating promising performance on multiple synthetic face recognition benchmarks. Three out of four reviewers are/become positive to this work, while reviewer 37yP is not persuaded to become more positive. The main concern of 37yP lies in the evaluations, especially comparisons with other methods. It is not perfect, but enough to validate the effectiveness of the proposed method. So, a positive decision is reached. The authors should improve the manuscript by considering the review comments in the camera-ready version of the paper.

Ethics review comment: The authors should be mindful of the fact that using the FFHQ dataset for "development of improvement of facial recognition technologies" is discouraged and if possible, the authors should reconsider its usage in the camera-ready version.

Reviews: review: paper_summary: paper_summary: The paper "ID3: Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition" introduces a novel synthetic face recognition (SFR) approach using diffusion models. It focuses on maintaining identity consistency while providing high diversity in generated face images. The proposed ID3 model leverages identity-preserving losses and a structured sampling algorithm that respects identity characteristics. This effectively addresses the common pitfalls of existing SFR approaches that lead to poor generalization on real-world data.
strength_weakness: strength_weakness: *   **Generalization**: Additional tests on further diversified real-world datasets could strengthen the generalization claims.
*   **Complexity**: It would be beneficial to have details on the computational demands and scalability of the model when deployed in practical, real-world scenarios.
questions: questions: 1.  What measures have been taken to ensure the model's robustness against diverse ethnic and age groups, given the model's reliance on identity embeddings?
2.  Are there potential improvements or variations in the diffusion model that could further enhance identity preservation without sacrificing diversity?
3.  How does the model perform under constrained computational resources, and are there any strategies for optimizing its efficiency?
limitations: limitations: The paper discusses potential limitations, including the need for extensive computational resources and the model's performance dependency on the quality of input identity embeddings. It also mentions the ongoing challenge of bridging the gap between synthetic and real-world face recognition performance.
paper_summary: paper_summary: This paper proposes ID3, an identity-preserving-yet-diversified diffusion model for generating synthetic face data for face recognition. ID3 leverages identity embeddings and facial attributes to control inter-class and intra-class diversity of generated faces while preserving intra-class identity consistency, demonstrating state-of-the-art performance on multiple synthetic face recognition benchmarks.
strength_weakness: strength_weakness: See questions section in detail.
questions: questions: The paper addresses a well-motivated and important problem in the field of synthetic face recognition. The proposed ID3 model demonstrates significant innovation in leveraging diffusion models conditioned on identity embeddings and facial attributes to generate diverse yet identity-consistent synthetic faces. Moreover, the theoretical analysis provided in the paper, which proves the equivalence between minimizing the proposed loss function and maximizing a lower bound on an adjusted data likelihood, lends credibility and rigor to the proposed approach.

However, there is room for improvement in the presentation and writing of the manuscript. One area that could benefit from further clarification is the explanation of notations and symbols used in the mathematical formulas. For instance, the meaning of the variable d in S
d−1 is not clearly defined, which may lead to confusion for readers. Additionally, the formatting and typesetting of some equations, such as Equation 3, could be enhanced to improve readability and aesthetic appeal.
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: This paper focuses on synthetic face recognition and proposes to concentrate on three aspects: inter-class diversity, intra-class diversity, and intra-class identity preservation. Based on those, an ID-preserving loss is employed to generate diverse but identity-preserving facial images. This paper also demonstrates the proposed loss is equal to lower bound of an adjusted conditional log-likelihood over ID-preserving data.
strength_weakness: strength_weakness: Here are several concerns regarding this work:
1. The idea of Attribute Conditioning Signal is not fit for synthetic face recognition tasks, because factors contributing to solid FR training cannot be determined by simply adjusting face attributes. One reason is that the attribute network (e.g., pose, age) is not generalized enough, as the pre-trained models are obtained from relatively small-scale datasets compared to FR datasets. Additionally, the authors have not addressed which attributes are effective for FR, leaving this important question unanswered.

2. The performance trained on FFHQ dataset appears good; however FFHQ dataset has explicitly banned its use for face recognition applications. Furthermore, FFHQ is relatively small(210k images) which doesn’t contain enough diversity, that’s the reason facial attributes can be of improvement in this experiment. For more details on FFHQ please refer to: https://github.com/NVlabs/ffhq-dataset

3. When it comes to the relatively large dataset CASIA-WebFace, the improvement over DCFace is marginal. One problem is that DCFace is trained with CASIA-WebFace only, not the FFHQ+CASIA mentioned by the author.

4. Experiments are not sufficient. For example, DCFace provides experiment results on 3 data volumes: 500k, 1M and 1.2M. These are not included in this paper.

5. There are some typos, for example, Y_i should be given in line 194
questions: questions: Based on Algorithm 3, the pipeline would be: firstly, generate multiple embeddings close to the anchor; Then use the diffusion model to synthesize the images. The questions are:
1. Given the unpack(Yi) and generate different attributes, the generated identity image would be affected by the attributes. Do the authors test whether the identities change given the different face attributes on the test? How to make sure the generated identity aligns with the input embedding in the generation phase? 
2. How to make sure the diffusion model can generalize(recognize) this specific input embedding, considering the training embedding only covers a small range of the available embedding(training set)?
limitations: limitations: This paper shows an attempt to generate diverse facial images of each identity. However, what makes solid FR training is not studied. Furthermore, domain GAP exists when adopting the trained diffusion model for generation, the input embedding might differ from the embedding of the synthesized image. Consequently, the focus is on changing facial attributes but preserving identity is interesting but the overall improvement is marginal, and FFHQ has license issues related to the application of Face Recognition. I think it will be more convincing if the results of CASIA-WebFace under 1M and 1.2 M settings are presented.
paper_summary: paper_summary: This paper presents a method called $\text{ID}^3$ for the task of synthetic face recognition. The authors highlight that the accuracy of face recognition using generated data still lags behind that of training directly on real face data. They propose optimizing the generation process from the perspectives of diversity and consistency.
strength_weakness: strength_weakness: - There has been extensive research on ID preserving, and recent models based on LDM (e.g., Face0, PhotoMaker, FaceStudio, InstantID) can also be used for synthetic face recognition. The paper lacks analysis and comparative experiments on these models.
- The Face Attribute Conditioning Signal includes age and pose (pose angle range: [-90°, 90°]). However, the visual results in the paper do not reflect these attributes. The variation in pose is minimal, and there is no demonstration of different levels of age (which you mentioned as [0-100]).
- The paper devotes too much space to mathematical derivations and lacks intuitive visual results. For example, using different attributes and ID information to guide the model could be visualized by showing how the various layers of the Unet perceive this information.
questions: questions: - What is the resolution of the training and generated images? 
- How long does the training process take using 8 NVIDIA Tesla V100 GPUs? 
- What is the image generation speed during inference? 
- How much GPU memory is required for inference?
- How do the ID information and attribute information affect the Unet in the network structure? Is it through cross-attention or along with the timestep information?
limitations: limitations: Using ID attributes to assist in the generation results is already common in diffusion-based tasks. This method is essentially a conditional guided generation, and its technical contribution is limited.

Comments: title: Further discussion with the authors
comment: We thank REVIEWER 37yP for the response. Here we would like to further make a few clarifications:

First, we were in a rush to submit the rebuttal in the very last minute and were unable to put all other results into the PDF file before the deadline. And we did not omit the LFW results on purpose. Moreover, the average values we reported in the first place already took into account the LFW results. Now that it has been clarified (see response to C3), we think this issue can be closed.

Second, regarding the results of IDiff-Face, as we have clarified in the submitted paper, in Line 251, "The “IDiff-Face” results in Table 1 are obtained by using the pretrained model provided in its official repository https://github.com/fdbtrs/IDiff-Face." Using their pretrained model downloaded from the official IDiff-Face repository, we obtained the result of 84.09 rather than 87.00 as reported in the IDiff-Face paper. According to the research replication policy, we reported 84.09 in our submission and clarified it in Line 251. Thanks for the suggestion. We will point it out in Table 1 in the revised manuscript.

Finally, we think it is ***unfair*** to say the improvements are marginal. As we mentioned above (see responses to C2 and C3), DCFace requires manual filtering to post-process the generated data and resorts to additional data, whereas ID$^3$ operates as a unified automated algorithm without any form of manual filtering and additional data. And yet, ID$^3$ can still yield SoTA or even better performance.
Thank the authors for the clarification.

I can't agree with 'We were unable to show the LFW result because of the character limit (6000) in the rebuttal.' Why other numbers are not discarded? You can adjust the text to present the full results, additionally, it can also be provided in the pdf file.

Furthermore, the results of IDiff-Face reported in this manuscript(average of 84.09) are significantly lower than those reported in the original IDiff-Face (average of 87.00, without data augmentation). The authors state that both methods used the FFHQ dataset, yet the results achieved by the proposed method (average of 86.50) are still worse than the originally reported IDiff-Face results (87.00). The difference between the original IDiffFace and the implemented IDiffFace should be pointed out in Table 1. 

Additionally, the results of training the model on the CAISA-WebFace dataset are of marginal improvement( 0.24% improvement on 0.5M and 0.29% on 1.2M).
Dear Reviewers,

Thank you for engaging in discussions with the authors. As the deadline for the discussion period is approaching, please provide any additional comments or feedback if you have any, to facilitate further discussion with the authors. Your input is valuable and appreciated.

Kind regards,

Your SAC
**Response to C1:**

Thanks for the suggestion. Yes, we agree that care shall be taken when using FFHQ.

----------------------------------------------------------------------------------------------------------
**Response to C2:**

We apologize for the confusion caused by '15%'. The number '15%' was due to our false memory. We mistakenly cited this result from somewhere else. After having carefully examined the DCFace paper again, however, it suddenly occurred to us again that, to achieve promising results, DCFace requires manual data-filtering on the generated data. This can be seen from Sec 3.3 in the original paper of DCFace (https://arxiv.org/pdf/2304.07060):

*"ID Image Sampling. For sampling ID images, we generate 200,000 facial images from $G_{id}$, from which we remove faces that are wearing sunglasses or too similar to the subjects in CASIA-WebFace with the Cosine Similarity threshold of 0.3 using $F_{eval}$. We are left with 105,446 images."*

Moreover, in Style Image Sampling (see the 2nd paragraph of Sec. 3.3), the authors explored the manual sampling $X_{sty}$ from the pool of images whose gender/ethnicity matches that of $X_{id}$.

We also have noted that the authors included *the same $X_{id}$ for 5 additional times for each label* and that *$X_{id}$ is generated by $G_{id}$ trained on FFHQ using DDPM* (see the beginning of Sec. 5 in the DCFace paper). This suggests that to generate ID embeddings, the authors made use of additional data, FFHQ, while using CASIA as training data. ID$^3$, on the other hand, uses CASIA only.

In general, the authors of DCFace established several manual-filtering standards and used them to clean the data generated by their model. This practice is not in line with the standards of the SFR community. As we have claimed in our paper, in Line 40-41, *"Also note that, critically, the SFR dataset generation process should be fully automated without manual filtering or introducing auxiliary real face samples."*

Furthermore, the manual-filtering used in DCFace is a time-consuming process. When scaling up to larger data volumes, DCFace is less likely to attain promising results whereas ID$^3$ is fully automated without any form of manual filtering and can linearly scale up to any data volume (please see our response to REVIEWER qiUe on Weakness 2 where we have analysed the time complexity of ID$^3$ which scales linearly with $m$ and $N$).

The above were the points we would like to have made in the table. Here we clarify it as follows:

|Models | Data |	Requires Manual Filtering	|LFW| CFP-FP|	CPLFW|	AgeDB	|CALFW	|average|
|----------|----------|----------|----------|----------|----------|----------|----------|----------|
|DCFace	| CASIA+FFHQ|Yes |98.83 |88.40	|84.22	|90.45|	92.38|	90.86|
ID$^3$  |CASIA|    No |	97.73 |89.89	|84.98	|91.15	|92.02	|91.15 |

----------------------------------------------------------------------------------------------------------

**Response to C3**

Thanks for pointing it out. We would like to clarify that the average result has taken the LFW result into account already. We were unable to show the LFW result because of the character limit (6000) in the rebuttal. Here we show the full evaluation result. You may check whether the average value took LFW into account: 90.86 = (98.83 + 88.40 + 84.22 + 90.45 + 92.38) / 5; 91.15 = (97.73 + 89.89 + 84.98 + 91.15 + 92.02) / 5.

|Models | Data |	Requires Manual Filtering	|LFW| CFP-FP|	CPLFW|	AgeDB	|CALFW	|average|
|----------|----------|----------|----------|----------|----------|----------|----------|----------|
|DCFace	| CASIA+FFHQ|Yes |98.83 |88.40	|84.22	|90.45|	92.38|	90.86|
ID$^3$  |CASIA|    No |	97.73 |89.89	|84.98	|91.15	|92.02	|91.15 |

From these results, we observe that ID$^3$ outperforms DCFace on three test sets out of five with a higher averaged performance (91.15>90.86). And considering that DCFace **requires manual-filtering to refine the generated data** and **uses additional data for generation** (see the response to C2), our proposed model ID$^3$ (operating as a unified automated system without any form of manual filtering and using CASIA only) is more promising and efficient, especially in generating larger and larger synthetic face datasets. 

On the theoretical side, ID$^3$ provides theoretical insights and advantages over existing works such as DCFace. Our work suggests that inductive biases regarding face manifold and distribution shall be introduced into an SFR generator as a whole.

Therefore, we believe the existence of ID$^3$ will inspire more interesting works to appear in this community which further advances the field of SFR. Code will be made publicly available upon the revision of the manuscript.
Thank the authors for their rebuttal, which addresses part of my concerns. It impresses me in terms of directly using numerical facial embedding for SFR.

However, I still have the following concerns:

---
**C1[minor]. Weakness 2:'SFR community all use FFHQ for model training':** 

I partly agree. I am aware that some of these SFR methods use pretrained generative models on FFHQ to provide Identity image (DCFace), and pre-mixing up facial image (SynFace). However, DigiFace does not involve training on FFHQ, and DCFace employs a pretrained model (on FFHQ) to provide style images while training a diffusion model on CASIA-WebFace. Given that the FFHQ dataset has restrictions concerning its application to FR, I think we should be careful when directly training a synthesizing model on FFHQ before the SFR community obtains permission to use this dataset. 

I agree that FFHQ can be adopted for reference, but I think it is better to present the SFR model trained on CASIA-WebFace at different data volumes.  Since CASIA-WebFace is a larger dataset, experiments conducted on it would be more convincing and general.

---
**C2[minor]. Weakness 4: 'Add Real Faces To FR Training'**

 DCFace doesn't add real face sto FR training, instead, they add the synthesized image into the generated dataset 'we include the same $X_{id}$ for 5 additional times for each label' in Section 5.2. The $X_{id}$  is generated by the $G_{ID}$
, 'For sampling ID images, we generate 200, 000 facial images from $G_{ID}$' in Section 3.3. This means that the images used in the dataset are entirely synthetic. It’s unclear how '15%' is calculated in the rebuttal.

---
**C3[Major] . Weakness 4 : 'results of 1.2M'**

 (1) I have recommended that the authors include experiments of 1M and 1.2M to make the paper more convincing in the original review opinion. However, only 1.2M results are given. Considering the limited rebuttal time it is understandable.

(2) Normally results of 5 evaluation datasets are provided: LFW, CFP-FP, CPLFW, AgeDB, CALFW. The LFW results of 0.5M given by the manuscript are vastly worse than the competitor DCFace in Table 1. However, the LFW result is missing in the rebuttal, making the average of the rest marginally higher than DCFace. I recommend putting all the evaluation datasets especially **LFW** results for a fair comparison.

I will keep my current score unless the full evaluation results are provided for a larger data volume.
Sorry for the delayed response. We just have finished refactoring the entire project for better readibility, and here goes the anonymous link to the project: [https://anonymous.4open.science/r/id3_sfr-FB8B/](https://anonymous.4open.science/r/id3_sfr-FB8B/). To check the details about how the Attribute Conditioning Signal works, you may refer to the class *IdPoseAgeFeaEmbedder* defined in ./ldm/modules/encoders/modules.py; to check the details about how identity loss works, you may refer to the function *id_preserving_loss* defined in the class DDPM in ./ldm/models/diffusion/ddpm.py. Please let us know if you have any further questions or concerns.
Thank you for your detailed response. After careful consideration of the points raised in your rebuttal, I have decided to maintain my original rating for the paper.
The authors have partially addressed my concerns, and I will give them a higher score. If you can provide me with an anonymous code link to check details about how the Attribute Conditioning Signal and Identity Loss work, I can further increase the score.