Title: $\text{ID}^3$: Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition
Venue: NeurIPS 2024
Accepted: yes
Keywords: synthetic face recognition
Abstract: Synthetic face recognition (SFR) aims to generate synthetic face datasets that mimic the distribution of real face data, which allows for training face recognition models in a privacy-preserving manner. Despite the remarkable potential of diffusion models in image generation, current diffusion-based SFR models struggle with generalization to real-world faces. To address this limitation, we outline three key objectives for SFR: (1) promoting diversity across identities (inter-class diversity), (2) ensuring diversity within each identity by injecting various facial attributes (intra-class diversity), and (3) maintaining identity consistency within each identity group (intra-class identity preservation). Inspired by these goals, we introduce a diffusion-fueled SFR model termed $\text{ID}^3$. $\text{ID}^3$ employs an ID-preserving loss to generate diverse yet identity-consistent facial appearances. Theoretically, we show that minimizing this loss is equivalent to maximizing the lower bound of an adjusted conditional log-likelihood over ID-preserving data. This equivalence motivates an ID-preserving sampling algorithm, which operates over an adjusted gradient vector field, enabling the generation of fake face recognition datasets that approximate the distribution of real-world faces. Extensive experiments across five challenging benchmarks validate the advantages of $\text{ID}^3$.

ID 3 : Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition

Abstract

Synthetic face recognition (SFR) aims to generate synthetic face datasets that mimic the distribution of real face data, which allows for training face recognition models in a privacy-preserving manner. Despite the remarkable potential of diffusion models in image generation, current diffusion-based SFR models struggle with generalization to real-world faces. To address this limitation, we outline three key objectives for SFR: (1) promoting diversity across identities (interclass diversity), (2) ensuring diversity within each identity by injecting various facial attributes (intra-class diversity), and (3) maintaining identity consistency within each identity group (intra-class identity preservation). Inspired by these goals, we introduce a diffusion-fueled SFR model termed ID 3 . ID 3 employs an ID-preserving loss to generate diverse yet identity-consistent facial appearances. Theoretically, we show that minimizing this loss is equivalent to maximizing the lower bound of an adjusted conditional log-likelihood over ID-preserving data. This equivalence motivates an ID-preserving sampling algorithm, which operates over an adjusted gradient vector field, enabling the generation of fake face recognition datasets that approximate the distribution of real-world faces. Extensive experiments across five challenging benchmarks validate the advantages of ID 3 . Code is released at: https://github.com/hitspring2015/ID3-SFR .

1 Introduction

With the introduction of various regulations restricting the use of large-scale facial data in recent years, such as GDPR, synthetic-based face recognition (SFR) (Boutros et al., 2023) has received widespread attention from the academic community (Qiu et al., 2021; Wood et al., 2021; Wang et al., 2023). The goal of SFR is to generate synthetic face datasets that mimic the distribution of real face images, and use it to train a face recognition (FR) model such that the model can recognize real face images as effectively as possible.

There exist numerous efforts to address SFR, which can be categorized into GAN -based models and diffusion models. GAN-based models utilize adversarial training to learn to generate synthetic data for FR training. Recently, with the empirical advantages of diffusion models over GANs, many works have attempted to use diffusion models to generate synthetic face data in place of authentic data. However, the reported results by these state-of-the-art (SoTA) SFR generative models (Bae et al., 2023; Boutros et al., 2022; Kolf et al., 2023; Qiu et al., 2021; Boutros et al., 2023) show

significant degradation in the verification accuracy in comparison to FR models trained by authentic data. We deduce the degradation might be due to two reasons. First, while previous works adopt diffusion models, they operate in the original score vector field without injecting the direction with regards to identity information, which makes them unable to guarantee identity-preserving sampling. Second, they fail to consider the structure of face manifold in terms of diversity during sampling.

We thus argue that the crux of SFR is to automatically generate a training dataset that has the following characteristics: (i) inter-class diversity : the training dataset covers sufficiently many distinct identities; (ii) intra-class diversity : each identity has diverse face samples with various facial attributes such as poses, ages, etc; (iii) intra-class identity preservation : samples within each class should be identity-consistent. Also note that, critically, the SFR dataset generation process should be fully automated without manual filtering or introducing auxiliary real face samples.

To this end, in this paper, we propose a novel ID entity-preserving-yetD iversified D iffusion generative model termed ID 3 and a sampling algorithm for inference. Jointly leveraging identity and face attributes as conditioning signals, ID 3 can synthesize diversified face images that conform to desired attributes while preserving intra-class identity. Specifically, ID 3 generates a new sample based upon two conditioning signals: a target face embedding and a specific set of face attributes. The target face embedding enforces identity preservation while face attributes enrich intra-class diversity. To optimize ID 3 , we propose a new loss function that involves an explicit term to preserve identity. Theoretically, we show that with the addition of this term, minimizing the proposed loss function is equivalent to maximizing the lower bound of the likelihood of an adjusted conditional data log-likelihood. Consequently, this theoretical analysis motivates a new ID-preserving sampling algorithm that generates desired synthetic face images. To generate an SFR dataset, we further propose a new dataset-generating algorithm. This algorithm ensures inter-class diversity by solving the Tammes problem (Tammes, 1930), which maximally separates identity embeddings on the face manifold. In the meantime, it encourages intra-class diversity by perturbing identity embeddings randomly within prescribed areas. It works in conjunction with identity embeddings and diverse attributes to ensure inter-/intra-class diversity while preserving identity. Extensive experiments show that ID 3 outperforms other existing methods in multiple challenging benchmarks.

To sum up, our major contributions are listed as follows:

Model with Theoretical Guarantees : We propose ID 3 , an identity-preserving-yetdiversified diffusion model for SFR. Theoretically, optimizing ID 3 is equivalent to shifting the original data likelihood to cover ID-preserving data.

Algorithm Design : Motivated by this theoretical equivalence, we design a novel sampling algorithm for face image generation, together with a face dataset-generating algorithm, which effectively generates fake face datasets that approximate real-world faces.

Effectiveness : Compared with SoTA SFR approaches, ID 3 improves SFR performance by ∼ 2 . 4% on average across five challenging benchmarks.

2 Problem Formulation

The scope of this paper is synthetic-based face recognition (SFR), which focuses on generating high-quality training data (i.e., face images) for FR models. Generally, we aim to address SFR by generating face images that conform to diverse facial attributes while preserving identity within each class, in an automated manner. Technically, we break down this objective into the following two research questions (RQs) to be answered:

RQ1 : How can we effectively train a SFR generative model that preserves identity within each class, while boosting inter-class and intra-class diversity?

RQ2 : Once the generative model is trained, what sampling strategy can be employed to generate a synthetic face dataset that enables state-of-the-art face recognition models to perform well on real face benchmarks?

The rest of the paper aims to answer these two questions, respectively, in order to improve synthetic face recognition performance.

Inner-Product Term

+ noise

Denoising Pass

Denoising Term + One-Step Reconstruction Term

Diffusion

Model

Identity Conditioning Signal

FR Model

Attribute Conditioning Signal

…

FR Model

ˆ

x

(

t

)

0

x

0

y

∈

S

d

-

1

s

f

φ

(

·

)

f

φ

(

·

)

Trainable

Frozen

ˆ

x

θ

(

x

t

, t,

y

,

s

)

x

t

Attribute Estimation Model

3 Methodology

We propose ID 3 , a conditional diffusion model that generates diverse yet identity-preserving face images. ID 3 solves RQ1 by introducing two conditioning signals (identity embeddings and face attributes) into a diffusion model which is trained using a novel loss function. The loss function, together with identity embeddings, ensures intra-class identity preservation, while generation upon various face attributes give rise to intra-/inter-class diversity of face appearances. Our theoretical result regarding this loss function leads to an ID-preserving sampling algorithm and, further, an effective dataset-generating algorithm.

Notations. Throughout the rest of the the paper, we let D denote a real face dataset that contains face images x 0 ∈ R H × W × 3 . Let y denote a desired identity embedding and s be face attributes.

3.1 Diffusion Models

We build up our generative model, ID 3 , upon denoising diffusion probabilistic models (diffusion models for short) (Ho et al., 2020; Song et al., 2022; Rombach et al., 2022) as they empirically exhibit SoTA performance in the field of image generation. Diffusion models can be seen as a hierarchical V AE whose optimization objective is to minimize the KL divergence between the true data distribution and the model distribution p θ , which is equivalent to minimizing the expected negative log-likelihood (NLL), E x ∼D [ -log p θ ( x )] . However, directly minimizing the expected NLL is intractable, therefore diffusion models instead maximize its evidence lower bound (ELBO), where the ELBO term can further simply to a denoising task with several model assumptions:

where µ t := T -1 2 σ 2 q ( t ) · ¯ α t -1 (1 -α t ) 2 (1 -¯ α t ) 2 , ¯ α t = ∏ t τ =1 α τ . Specifically, given a sample x 0 (or interchangeably, x ) from the image distribution, a sequence x 1 , x 2 , ..., x T of noisy images is produced by

progressively adding Gaussian noise according to a variance schedule α 1 , ..., α T . This process is called the forward diffusion process q ( x t | x t -1 ) . At the final time step, x T is assumed to be pure Gaussian noise: x T ∼ N (0 , I ) . The objective is to train a denoising network ˆ x θ that is able to predict the original image from the noisy image x t and the time step t . To sample a new image, we sample x T ∼ N (0 , I ) and iteratively denoise it, producing a sequence x T , x T -1 , ..., x 1 , x 0 . The final image, x 0 , should resemble the training data.

Although the naive diffusion models are powerful in generating images, they do not deliver the promise of generating face images of the same identity (i.e. identity preservation) without direct corresponding information; nor are they aware of diverse desired facial attributes during inference. To achieve intra-class diversity and intra-class identity preservation , we would like to gain control of generating desired identities, each of which exhibits various attributes, including poses, ages and background variations. Hence, our aim is to design a diffusion model that conditions on specific identities and attributes throughout the generation of face images.

3.2 ID 3 as Conditional Diffusion Models

We propose a conditional diffusion model, ID 3 (see Figure 1 for details). Specifically, we extend the denoising network by conditioning it on two sources of signals: identity signals y and face attribute signals s . The identity signals capture discernible faces in generated images, whereas face attribute signals specify the identity-irrelevant attributes, including poses, ages, etc. We introduce how to obtain these two conditioning signals, respectively, in the next two subsections.

3.2.1 Identity Conditioning Signal

To obtain identity conditioning signals, we assume access to a pretrained face recognition model f ϕ : R H × W × 3 ↦→ S d -1 , which maps the domain of face images to a feature space S d -1 . This mapping f ϕ is parameterized by the learnable parameter ϕ , which is obtained by training the model on a real face dataset in the face recognition task. We follow the latest advancement of face recognition by setting the output space to be a unit hypersphere S d -1 . Then, given a face image x 0 drawn from the dataset D , we obtain its identity embedding y ∈ S d -1 by feeding it into a face recognition model f ϕ : y = f ϕ ( x 0 ) , which serves as the identity conditioning signals for ID 3 .

3.2.2 Face Attribute Conditioning Signal

Face attributes capture identity-irrelevant information about face images, such as age, face poses, etc. To obtain face attribute as conditioning signals, we employ pretrained attribute predictors (Serengil and Ozpinar, 2021) which output these attributes when given a face image as input. The pretrained attribute predictors are a collection of ad-hoc domain experts in age estimation and pose estimation. After obtaining each of these attribute values, s age ∈ [0 , 100] , s pose ∈ [ -90 ◦ , 90 ◦ ] 3 , we concatenate them as the overall attribute s = [ s age , s pose ] which is then fed into the diffusion model as conditioning signals.

3.3 Optimization Objective

Now the denoising network in Eq. (1) becomes ˆ x θ ( x t , t, y , s ) that takes as input the noised x t , the time step t , and the conditioning signals y and s . To optimize ID 3 , we construct a training objective upon the ELBO of log p ( x | y , s ) , ensuring that ID 3 generates identity-preserving yet diversified faces:

Here, θ is the learnable parameter of the denoising network and the datapoint-wise loss is given by

where ˆ x ( t ) 0 is the output of the denoising network that takes as input the conditioning signals y , s , the time t and the t -step noisified image x t :

Symbolically, ˆ x ( t ) 0 denotes the denoised image predicted by the denoising network when given the t -step noisified x t , the time t and the associated conditioning signals y , s . The coefficients, κ x 0 and λ t are scalars depending on x 0 and t , respectively, and C is a constant that does not depend on the learnable parameters θ . The specific value of C will be elaborated in Appendix A.

To summarize, our proposed loss function consists of four terms: the one-step reconstruction term, the denoising term, the inner-product term, and a constant. Intuitively, the denoising term, along with the one-step reconstruction term, aims to improve the generative quality by denoising the t -step noisified face images while the inner-product term encourages the face embedding of the denoisified images to get close to the groundtruth identity embedding. To understand this loss function systematically, we theoretically find that minimizing this proposed loss function is equivalent to the maximization of the lower bound of an adjusted conditional log-likelihood over identity-preserving face images, which further leads us to an ID-preserving sampling algorithm.

Theorem 3.1. Minimizing L with regard to θ is equivalent to minimizing the upper bound of an adjusted conditional data negative log-likelihood -log ˜ p ( x | y , s ) , i.e.:

where

Proof. The proof can be found in Appendix A.

Remark. We have just shown that our proposed loss is the upper bound of an adjusted conditional negative data log-likelihood. This adjusted likelihood ˜ p ( x | y , s ) can be factorized into the original likelihood p ( x | y , s ) and a reversed likelihood p ( y , s | x ) with some positive power. We term it as 'adjusted' since the original likelihood is discounted by the reversed likelihood. Intuitively, the reversed likelihood shifts the original likelihood such that the adjusted likelihood covers ID-preserving data, which is attributed to the inner-product term we introduce into the loss function in Eq. (2).

3.4 ID-Preserving Sampling

Theorem 3.1 provides insights for designing a novel sampling algorithm in the spirit of Langevin dynamics applied on the adjusted conditional likelihood ˜ p ( x t | y , s ) . Wenote that Langevin dynamics can generate new samples from a probability density p by virtue of its score function (i.e., the gradient of the logarithm of the probability density w.r.t. the sample, ∇ x log p ). Motivated by this observation, we aim to find the score function of the adjusted likelihood for sample generation. Specifically, taking the logarithm and the gradient w.r.t. x on both sides of Eq. (6) yields

Algorithm 3: Synthetic Dataset Generation

V log p(x|y,s)

V log p(xly,s)

V log p(y, s|x)

T-1

Then, our ID-preserving sampling algorithm first draws a Gaussian sample x T ∼ N (0 , I ) . Afterwards, sequentially, the algorithm performs the following update for t iterating from T backwards to 1 :

where

Note that the original likelihood score in Eq. (8) can be evaluated by

and the reversed likelihood score is given by a scaled inner product:

See Appendix B for the derivation of the above equations. As such, our ID-preserving sampling algorithm performs sampling by searching a trajectory in the vector field ∇ log ˜ p ( x t | y , s ) that can maximize the adjusted conditional likelihood ˜ p ( x t | y , s ) . See Algorithm 2 for the specific procedure.

Remark. Our proposed adjusted likelihood score differs from the original score by adding an extra scaled reversed likelihood score in Eq. (8). Consequently, as shown in Figure 2 , the resulting vector field differs from the original vector field, which leads to different Langevin sampling trajectories and thus different sampling quality.

3.5 Synthetic Dataset Generation

In terms of the second question ( RQ2 ): after training ID 3 , with what sampling strategy is it possible to generate a synthetic face dataset on which SoTA face recognition models can be trained and perform well on challenging benchmarks?

Our proposed dataset-generating algorithm goes as follows: given N target identities, we generate N anchor embeddings distributed on the sphere: w 1 , w 2 , ..., w N ∈ S d -1 as uniformly as possible in the sense that each pair of the embeddings are maximally separated on the unit sphere # . For each anchor w i , we would like to generate m identity embeddings perturbed around w i while ensuring

that these m identity embeddings get close to but different than w i . Specifically, to find these m identity embeddings, we solve the following optimization problem P i :

where the operator norm is column-wise normalization which normalizes each column of Y i into a unit vector, and the desired similarity scores ν i 1 , ν i 2 , ..., ν im are randomly generated from a continuous uniform distribution U [ lb, ub ) . After solving Eq. (9), we are able to retrieve the m optimal unnormalized vector y ∗ i 1 , ..., y ∗ im . These m vectors are then normalized, yielding m identity embeddings: norm( y ∗ ij ) , for j = 1 , ..., m . Then, the resulting identity embeddings, along with face attributes, are fed into our generative models to generate face images. Finally, the entire dataset is generated by solving each P i , i = 1 , ..., N , which yields N identities, each with m face images. The entire algorithm is summarized in Algorithm 3.

4 Experiments

In this section, we verify the effectiveness of ID 3 through empirical evaluation of the face dataset that ID 3 generates, and verify the performance of the SoTA face recognition model trained on this dataset in comparison with other baseline methods.

4.1 Dataset

Training Dataset: We train our proposed ID 3 on FFHQ (Karras et al., 2019) dataset. The FFHQ (FaceForensics++) dataset is a large-scale dataset used for benchmarking and evaluating the performance of deep learning models in the field of face forensics. It is an extension of the original FaceForensics dataset, which was designed to facilitate the development and comparison of methods for detecting and preventing face manipulation and deepfakes. In order to compare with DCFace (Kim et al., 2023), we also train ID 3 on CASIA-WebFace (Yi et al., 2014). The CASIA-WebFace dataset is used for face verification and face recognition tasks. This dataset contains 494,414 face images of 10,575 real identities collected from the web.

Benchmarks: The performance of face recognition models is evaluated on various benchmark datasets: LFW (Huang et al., 2008), CFP-FP (Sengupta et al., 2016), CPLFW (Zheng and Deng, 2018), AgeDB (Moschoglou et al., 2017) and CALFW (Zheng et al., 2017). They are used to measure the impact of different factors on face image, such as pose changes and age variations.

4.2 Implementation Details

For our ID 3 , we implement the denoising network with a U-net architecture and the projection module with a three-layer perceptron (hidden-layer size (512 , 256 , 768) ) with ReLU activation. All

models are implemented with PyTorch and trained from scratch using 8 NVIDIA Tesla V100 GPUs. Specifically, we set λ t κ x t = 0 . 5 · (1 -1 / (1 + exp ( -t/T )) for the loss coefficients in Eq. (3), and use T = 1 , 000 for the diffusion model; training batch size is set to 16 and the total training steps 500 , 000 . We directly use a pre-trained face recognition (FR) model sourced from pSp (Richardson et al., 2021) as the identity feature extractor. Throughout the entire training process, these pre-trained models are frozen. In addition, we set # of identity embeddings m = 25 in Eq. (9) for each ID and match their embeddings with randomly selected attributes as conditioning signals for the diffusion model. For face recognition, we use LResNet50-IR (Deng et al., 2019), a variant of ResNet (He et al., 2016), as the backbone framework and follow the original configurations.

4.3 Performance Evaluation

We test the performance of the face recognition model trained on synthetic face data generated by ID 3 and compare against SoTA SFR generative models, including IDiff-Face (Boutros et al., 2023), ID-Net (Kolf et al., 2023), DigiFace (Bae et al., 2023), SFace (Boutros et al., 2022), SynFace (Qiu et al., 2021) and DCFace (Kim et al., 2023).

4.3.1 Qualitative Results

Here, we illustrate a collection of face images generated by ID 3 as qualitative evaluation. Figure 3 shows the results for randomly sampled identities (IDs) under various attribute conditions; Obviously, when comparing different identities (inter-class), the essential intrinsic key information of each identity is still retained and can be easily identified. Also, different samples of each identity (intra-class) exhibit distinct diversity, stemming from variations in similarity scores ( ν ij 's) and differences in face attributes as conditioning signals. In terms of the effect of our proposed adjusted score and the original score on the sampling algorithm, we observe that the face images generated by our proposed ID 3 exhibits much better quality and identity preservation than those generated by the original score function, as shown in Figure 2.

4.3.2 Quantitative Results

Wecompare the accuracies of FR models trained on the synthetic face datasets generated by different generative models and demonstrate the results in Table 1.

As shown in Table 1, ID 3 demonstrates consistent superior performance, achieving the highest average accuracy of 86 . 50% , and outperforms other baselines in all benchmarks, notably scoring 83 . 78% in AgeDB and 85 . 00% in CFP-FP. This demonstrates the effectiveness of ID 3 in gaining pose and age control. Other methods, while effective to varying degrees, attain average scores below 86 . 50% and are inferior to ID 3 .

It is worth mentioning that ID 3 , apart from using real data during training, does not introduce any real images as auxiliary data during the sampling phase. The synthetic data is directly used in the training of the face recognition model without undergoing any secondary or manual filtering. Additionally, when training the face recognition model using the synthetic data, no real images are introduced as auxiliary data. On the other hand, DCFace, as described and reported in (Kim et al., 2023), introduces real face images as auxiliary data during the training phase for face recognition.

This helps enhance the diversity of the training data and leads to slightly better results than ID 3 in the two benchmarks.

Ablation study. We further investigate the impact of each contributing component of ID 3 in generating a synthetic face dataset on SFR. This includes three ablation studies shown in Table 2: the effect of the reversed likelihood score in Eq. (8) on ID-preserving sampling algorithm (ID 3 vs. ID 3 -w/o-reversed), the effect of using anchors in ID 3 (ID 3 vs. ID 3 -random), and the effect of lower- and upper-bound of Uniform distribution for sampling ν ij 's.

In the first study, we compare ID 3 with ID 3 -w/o-reversed, which removes the reversed likelihood score from Eq. (8) in the proposed ID-preserving sampling algorithm. We observed ID 3 consistently outperforms ID 3 -w/o-reversed with large margins. This suggests the necessity of the inner-product term in the proposed loss function Eq. (3) and the reversed likelihood score in the adjusted likelihood score Eq. (8).

In the second study (cf. Appendix E), an appropriate smaller value of lb , if not exceeding a certain range, can increase the intra-class diversity, resulting in more diverse intra-class face images. This aligns with our objective of increasing intra-class diversity in the generated data to enhance the effectiveness of SFR. As per the constraints of Eq. (9), each generated identity embedding y ij maintains the same identity as the anchor w i . This, along with our proposed inner-product term in Eq. (3), ensures consistent intra-class identities while introducing a significant amount of diversity.

3

In the third study, we demonstrate how effective it is to use maximally-separated anchors in ID as compared to ID 3 -random that randomly sets anchors on the unit hypersphere for sample generation. Clearly, ID 3 -random does not yield as good results as ID 3 . This is because the random sampling method only introduces one identity signal per ID, while the model requires a combination of attributes and identity signals. Attribute signals can only control explicit attributes, whereas identity signals control implicit properties. Introducing only one identity signal per identity implies insufficient intra-class diversity. Additionally, ID 3 -random fails to regularize the relationship among different identities, leading to inadequate diversity among classes or aliasing issues with different identity signals. The identity signals obtained using ID 3 resolves the problem of aliasing between identity signals across classes, effectively improving intra-/inter-class diversity.

5 Conclusion

We have proposed ID 3 , an identity-preserving-yet-diversified diffusion generative model for SFR. Our theoretical analysis regarding the training of ID 3 induces a new ID-preserving sampling algorithm and further, a dataset-generating algorithm that generates identity-preserving face images with inter-/intra-class diversity. Extensive experiments show that ID 3 outperforms existing methods in challenging multiple benchmarks.

Limitations. While ID 3 , designed for the sake of privacy protection, achieves SoTA performance in SFR, there remains clear margins as compared to the FR performance when training with real-world face datasets such as MS1M. This suggests that the fake face dataset generated by ID 3 does not fully approximate the real-world faces. Future work might include closing this gap.

References

Gwangbin Bae, Martin de La Gorce, Tadas Baltruˇ saitis, Charlie Hewitt, Dong Chen, Julien Valentin, Roberto Cipolla, and Jingjing Shen. Digiface-1m: 1 million digital face images for face recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 3526-3535, 2023.

Fadi Boutros, Marco Huber, Patrick Siebke, Tim Rieber, and Naser Damer. Sface: Privacy-friendly and accurate face recognition using synthetic data. In 2022 IEEE International Joint Conference on Biometrics (IJCB) , pages 1-11. IEEE, 2022.

Fadi Boutros, Jonas Henry Grebe, Arjan Kuijper, and Naser Damer. Idiff-face: Synthetic-based face recognition through fizzy identity-conditioned diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19650-19661, 2023.

Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4690-4699, 2019.

Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3d imitative-contrastive learning. In IEEE Computer Vision and Pattern Recognition , 2020.

Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14 , pages 87-102. Springer, 2016.

Md Abul Hasnat, Julien Bohn´ e, Jonathan Milgram, St´ ephane Gentric, and Liming Chen. von mises-fisher mixture model-based deep learning: Application to face verification. arXiv preprint arXiv:1706.04264 , 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in'Real-Life'Images: detection, alignment, and recognition , 2008.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4396-4405, 2019.

Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems , 33:1210412114, 2020.

Minchul Kim, Feng Liu, Anil Jain, and Xiaoming Liu. Dcface: Synthetic face generation with dual condition diffusion model, 2023.

Jan Niklas Kolf, Tim Rieber, Jurek Elliesen, Fadi Boutros, Arjan Kuijper, and Naser Damer. Identity-driven three-player generative adversarial network for synthetic-based face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 806-816, 2023.

Shen Li, Jianqing Xu, Xiaqing Xu, Pengcheng Shen, Shaoxin Li, and Bryan Hooi. Spherical confidence learning for face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15629-15637, 2021.

Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Photomaker: Customizing realistic human photos via stacked id embedding-supplementary materials-.

Jianxin Lin, Yingce Xia, Tao Qin, Zhibo Chen, and Tie-Yan Liu. Conditional image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5524-5532, 2018.

Calvin Luo. Understanding diffusion models: A unified perspective, 2022.

Pascal Mettes, Elise van der Pol, and Cees G M Snoek. Hyperspherical prototype networks. In Advances in Neural Information Processing Systems , 2019.

Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages 51-59, 2017.

Haibo Qiu, Baosheng Yu, Dihong Gong, Zhifeng Li, Wei Liu, and Dacheng Tao. Synface: Face recognition with synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10880-10890, 2021.

Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel CohenOr. Encoding in style: a stylegan encoder for image-to-image translation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨ orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, 2022.

Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. Frontal to profile face verification in the wild. In 2016 IEEE winter conference on applications of computer vision (WACV) , pages 1-9. IEEE, 2016.

Sefik Ilkin Serengil and Alper Ozpinar. Hyperextended lightface: A facial attribute analysis framework. In 2021 International Conference on Engineering and Emerging Technologies (ICEET) , pages 1-4. IEEE, 2021.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022.

Pieter Merkus Lambertus Tammes. On the origin of number and arrangement of the places of exit on the surface of pollen-grains. Recueil des travaux botaniques n´ eerlandais , 27(1):1-84, 1930.

Dani Valevski, Danny Lumen, Yossi Matias, and Yaniv Leviathan. Face0: Instantaneously conditioning a text-to-image model on a face. In SIGGRAPH Asia 2023 Conference Papers , pages 1-10, 2023.

Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5265-5274, 2018.

Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity-preserving generation in seconds. arXiv preprint arXiv:2401.07519 , 2024.

Wenqing Wang, Lingqing Zhang, Chi-Man Pun, and Jiu-Cheng Xie. Boosting face recognition performance with synthetic data and limited real data. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5, 2023.

Erroll Wood, Tadas Baltruˇ saitis, Charlie Hewitt, Sebastian Dziadzio, Thomas J Cashman, and Jamie Shotton. Fake it till you make it: face analysis in the wild using synthetic data alone. In Proceedings of the IEEE/CVF international conference on computer vision , pages 3681-3691, 2021.

Jianqing Xu, Shen Li, Ailin Deng, Miao Xiong, Jiaying Wu, Jiaxiang Wu, Shouhong Ding, and Bryan Hooi. Probabilistic knowledge distillation of face ensembles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3489-3498, 2023.

Yuxuan Yan, Chi Zhang, Rui Wang, Yichao Zhou, Gege Zhang, Pei Cheng, Gang Yu, and Bin Fu. Facestudio: Put your face everywhere in seconds. arXiv preprint arXiv:2312.02663 , 2023.

Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z. Li. Learning face representation from scratch, 2014.

Tianyue Zheng and Weihong Deng. Cross-pose lfw: A database for studying cross-pose face recognition in unconstrained environments. Beijing University of Posts and Telecommunications, Tech. Rep , 5(7), 2018.

Tianyue Zheng, Weihong Deng, and Jiani Hu. Cross-age lfw: A database for studying cross-age face recognition in unconstrained environments. arXiv preprint arXiv:1708.08197 , 2017.

Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Jiwen Lu, Dalong Du, et al. Webface260m: A benchmark unveiling the power of million-scale deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10492-10502, 2021.

ID 3 : Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition

Supplementary Material

A Appendix A

We first present a lemma (Lemma A.1) that will be further used to prove Theorem 3 . 1 .

Lemma A.1. Given any two conditional probability density functions, p ( a | b ) and p ( b | a ) , and a positive scalar w > 0 , there exists another conditional probability density function ˜ p such that

Proof. To show this result is equivalent to showing ˜ p ( a | b ) ∝ p ( a | b ) · p ( b | a ) w , w > 0 , which is equivalent to showing that, for any b = b 0 ,

Note that ˜ p ( a | b = b 0 ) is a function of a , i.e. there exists a function f b 0 ,w such that p ( b = b 0 | a ) w := f b 0 ,w ( a ) . Let

Then, is a proper probability density function of a with b = b 0 given. Therefore, Eq. (A.4) can be written as ˜ p ( a | b = b 0 ) . The above proof holds true for any b 0 , which concludes the proof for ˜ p ( a | b ) ∝ p ( a | b ) · p ( b | a ) w , w > 0 . Note that this result can be trivially extended to multivariate random variables.

Theorem 3.1. Minimizing L with regard to θ is equivalent to minimizing the upper bound of an adjusted conditional data negative log-likelihood -log ˜ p ( x | y , s ) , i.e.:

where and x 0 and x both refer to a raw image interchangeably.

Proof. Recall that

It can be shown that the reversed likelihood p ( y , s | x ) is a joint vMF density (Xu et al., 2023; Hasnat et al., 2017):

where J κ x is the normalizing constant and F a is the pretrained attribute predictor. Note that Equality (1) is obtained by the product rule of probability; Equality (2) is obtained by observing that y and s are conditionally independent when x is given; and Equality (3) is obtained by assuming that

Note that these reasonable assumptions are also held in (Xu et al., 2023; Li et al., 2021; Hasnat et al., 2017). Now we can specify the value of the scalar C :

where n = 3 HW is the dimensionality of x , and

Note that Z only depends on y , s and ∑ T t =2 λ t T -1 and hence C is a scalar that does not depend on the learnable parameter θ . Therefore, L can be rewritten into a sum of two parts: L = L 1 + L 2 , where

and

We recognize -L 1 is the evidence lower bound (ELBO) of log p ( x | y , s ) (Luo, 2022), i.e.

As for L 2 :

Here we assume access to a perfect denoising module such that ˆ x ( t ) 0 = x 0 = x , for all t 's. Hence, L 2 can be written as

Then,

where Equality (A.17c) is obtained by applying Lemma A.1. This completes the proof.

B Appendix B

In this section, we show the derivations of the following equations:

Recall that in the main text, we showed that the adjusted likelihood score is a summation of the original likelihood score and the scaled reversed likelihood score:

For the original likelihood score, we note that our proposed ID 3 itself is a conditional diffusion model. By virtue of the relation between the score and the denoising module (i.e. the Tweedie's Formula) in diffusion models (cf. Equation (133) in (Luo, 2022)), we are able to show that

Here, x 0 can be approximated by denoising x t via the trained denoising module

C An Illustration of the Dataset Generating Algorithm

We illustrate the dataset generating algorithm in Figure A.1. First, N anchor embeddings are generated on the sphere as uniformly as possible. Then, for each anchor, m identity embeddings are generated around the anchor. This strategy ensures inter-class diversity while intra-class identity preservation is guaranteed. Colors show the correspondence between the generation procedure on the left and the generated samples on the right.

Inter-class

Intra-class

4000

Mean: 0.02

Mean: 0.52

Median: 0.02

400

Median: 0.52

3500

3000

300

2500

sunon

2000

200

1500

1000

100

500

1.00

0+

0.75

-0.50

0.25

0.00

0.25

0.50

0.75

1.00

1.00

0+

0.75

-0.50

0.25

0.50

0.75

1.00

Cosine Similarity

Cosine Similarity

[1b, ub] = [0.5, 0.7]

Inter-class

Intra-class

1400

Mean: 0.02

250

Mean: 0.58

Median: 0.02

Median: 0.58

1200

200

1000

800

600

100

400

50

200

+0

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

o+

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Cosine Similarity

Cosine Similarity

[lb, ub] = [0.7, 0.9]

D Related Work

Face Recognition. Face Recognition (FR) is the task of matching query imagery to an enrolled identity database. SoTA FR models are trained using margin-based softmax losses (Wang et al., 2018; Deng et al., 2019) on large-scale web-crawled datasets (Guo et al., 2016; Zhu et al., 2021). These datasets encompasses three characteristics in common (as mentioned in the introduction): (i) sufficient inter-class diversity; (ii) intra-class diversity; (iii) intra-class identity preservation. However, due to the introduction of various regulations restricing the use of authentic face data, researchers switch their attention to synthetic face recognition (SFR). We argue that the crux of SFR is to generate a training dataset that inherits the three characteristics above.

GAN-based SFR models. Most of the deep generative models for synthetic faces generation are based on GANs. DigiFace (Bae et al., 2023) utilizes a digital rendering pipeline to generate synthetic images based on a learned model of facial geometry and attributes. SFace (Boutros et al., 2022) and ID-Net (Kolf et al., 2023) train a StyleGAN-ADA (Karras et al., 2020) under a classconditional setting. SynFace (Qiu et al., 2021) extends DiscoFaceGAN (Deng et al., 2020) using synthetic identity mix-up to enhance the intra-class diversity. However, the reported results shown by these models show significant performance degradation in comparison to FR trained on real data. This performance gap is mainly due to inter-class discrimination and small intra-class diversity in their generated synthetic training datasets. Diffusion models for SFR. Recently, Diffusion Models (DMs) (Ho et al., 2020; Lin et al., 2018; Song et al., 2020) gained attention for both research and industry due to their potential to rival GANs on image synthesis, as they are easier to train without stability issues, and stem from a solid theoretical foundation. Among SFR diffusion models, IDiff-Face (Boutros et al., 2023) achieves SoTA performance. On the basis of a diffusion model, it incorporates Contextual Partial Dropout to generate diverse intra-class images. However, IDiff-Face fails to regularize the relationship among different identities.

Latent Diffusion Models. There exist many diffusion-based models (LDMs) (e.g., Face0 (Valevski et al., 2023), PhotoMaker (Li et al.), FaceStudio (Yan et al., 2023), InstantID (Wang et al., 2024)) which use ID attributes to assist in generating images. However, we note that these LDMs are designed for image generation but not for SFR. Our empirical findings further suggests these LDMs do not perform reasonably well even when applied to SFR. We found that although these LDMs claim to be ID-preserving in the pixel space, their feature embeddings are not discriminative enough

Distribution of Pose Angle on Dataset

Distribution of Pose Angle on Dataset

Distribution of Pose Angle on Dataset

Distribution of Pose Angle on Dataset

Distribution of Pose Angle on FFHQ

Generated byIDiffFace

Generated by ID3 PoseOnly 1

Generated by ID3 PoseOnly 2

Generated by ID3PoseOnly3

Frequency

0.04

0.03

Frequency

0.030

0.025

0.020

Frequency

0.06

0.05

0.04

guency

0.06

0.05

0.04

0.07

Frequency

0.030

0.025

0.020

0.02

0.015

0.03

0.03

0.015

0.01

0.010

0.005

0.02

0.01

0.01

0.02

0.010

5000

0.00

60

40

20

20

0.000

-60

20

0

20

0.00

=0

-40

-20

20

40

60

4020

20

40

60

0.000

40

-20

20

40

00

DistributionofPitch onFFHQ

Distribution ofYaw onFFHQ

Distribution ofRoll onFFHQ

Distribution of Age on FFHQ

0.06

0.04

0.30

0.040

Frequency

0.05

Frequency

Frequency

0.25

quency

0.035

0.04

0.03

0.20

0.030

Frequ

0.025

0.03

0.02

0.15

0.020

0.02

0.10

0.015

0.01

0.01 -

0.05

0.010

0.005

0.00

30

-20

10

0

10

20

30

40

0.00

60

40

20

0

20

40

66

0.00

20

10

10

20

30

0.000

0

10

20

30

40

50

60

70

Pitch

Roll

Age

for face recognition, since there is no inductive bias (neither loss functions nor architectures) to achieve face discriminativeness.

E Ablation Study (ii)

We show the inter-class and intra-class similarity in Figure A.2when using [0 . 5 , 0 . 7] and [0 . 7 , 0 . 9] as the lower- and upper-bound [ lb, ub ] for sampling ν ij 's in our proposed dataset-generating algorithm.

F Attribute Analysis of Generated Datasets

In this section, we perform an attribute analysis of the generated face datasets by ID 3 . As shown in Figure A.3 and Figure A.4, from the distribution, we observe that the highest age in our training set is 70; the largest pose is 60 in degree and the smallest -60 in degree. Our model can interpolate within these ranges but is less likely to extrapolate outside of these ranges. To generate face images of large pose and high age, one can collect more such data and add them to the training dataset, which increases their occurrences during training.

To examine whether the attributes we used in our paper are fit for SFR tasks, we perform the following ablation study: as we increase intra-class pose variation, the SFR performances on cross-pose test sets (including CFPFP and CPLFW) are boosted whereas the performances on cross-age test sets (including AgeDB and CALFW) remain almost unchanged. Results and distribution plots are shown in Table A.1, Figure A.3 and Figure A.4. From these results, we observe that the distribution of pose angle on FFHQ, Dataset by IDiffFace, Dataset by ID PoseOnly 1 and 2 are all unimodal. And their performances are inferior to Dataset by ID PoseOnly 3 which exhibits multimodal distribution of pose angle.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: As shown in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: As shown in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: As shown in the Appendix A and B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: As shown in the experiment and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: see Abstract.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: As shown in the experiments and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: As shown in the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: NA

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited. The license and terms of use are explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: NA.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method called $\\text{ID}^3$ for the task of synthetic face recognition. The authors highlight that the accuracy of face recognition using generated data still lags behind that of training directly on real face data. They propose optimizing the generation process from the perspectives of diversity and consistency.\n\nSTRENGTHS:\n- Clear explanation of formulas and algorithm flow.\n- Achieved SOTA results compared to methods from the past two years.\n\nWEAKNESSES:\n- There has been extensive research on ID preserving, and recent models based on LDM (e.g., Face0, PhotoMaker, FaceStudio, InstantID) can also be used for synthetic face recognition. The paper lacks analysis and comparative experiments on these models.\n- The Face Attribute Conditioning Signal includes age and pose (pose angle range: [-90°, 90°]). However, the visual results in the paper do not reflect these attributes. The variation in pose is minimal, and there is no demonstration of different levels of age (which you mentioned as [0-100]).\n- The paper devotes too much space to mathematical derivations and lacks intuitive visual results. For example, using different attributes and ID information to guide the model could be visualized by showing how the various layers of the Unet perceive this information.\n\nQUESTIONS:\n- What is the resolution of the training and generated images? \n- How long does the training process take using 8 NVIDIA Tesla V100 GPUs? \n- What is the image generation speed during inference? \n- How much GPU memory is required for inference?\n- How do the ID information and attribute information affect the Unet in the network structure? Is it through cross-attention or along with the timestep information?\n\nLIMITATIONS:\nUsing ID attributes to assist in the generation results is already common in diffusion-based tasks. This method is essentially a conditional guided generation, and its technical contribution is limited.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on synthetic face recognition and proposes to concentrate on three aspects: inter-class diversity, intra-class diversity, and intra-class identity preservation. Based on those, an ID-preserving loss is employed to generate diverse but identity-preserving facial images. This paper also demonstrates the proposed loss is equal to lower bound of an adjusted conditional log-likelihood over ID-preserving data.\n\nSTRENGTHS:\n1. This work is well-written and well-organized. It brings some insights for SFR.\n2. The idea of 3 aspects is good, and quite general for SFR\n3. The proposed method shows advances when using the FFHQ dataset\n\nWEAKNESSES:\nHere are several concerns regarding this work:\n1. The idea of Attribute Conditioning Signal is not fit for synthetic face recognition tasks, because factors contributing to solid FR training cannot be determined by simply adjusting face attributes. One reason is that the attribute network (e.g., pose, age) is not generalized enough, as the pre-trained models are obtained from relatively small-scale datasets compared to FR datasets. Additionally, the authors have not addressed which attributes are effective for FR, leaving this important question unanswered.\n\n2. The performance trained on FFHQ dataset appears good; however FFHQ dataset has explicitly banned its use for face recognition applications. Furthermore, FFHQ is relatively small(210k images) which doesn’t contain enough diversity, that’s the reason facial attributes can be of improvement in this experiment. For more details on FFHQ please refer to: https://github.com/NVlabs/ffhq-dataset\n\n3. When it comes to the relatively large dataset CASIA-WebFace, the improvement over DCFace is marginal. One problem is that DCFace is trained with CASIA-WebFace only, not the FFHQ+CASIA mentioned by the author.\n\n4. Experiments are not sufficient. For example, DCFace provides experiment results on 3 data volumes: 500k, 1M and 1.2M. These are not included in this paper.\n\n5. There are some typos, for example, Y_i should be given in line 194\n\nQUESTIONS:\nBased on Algorithm 3, the pipeline would be: firstly, generate multiple embeddings close to the anchor; Then use the diffusion model to synthesize the images. The questions are:\n1. Given the unpack(Yi) and generate different attributes, the generated identity image would be affected by the attributes. Do the authors test whether the identities change given the different face attributes on the test? How to make sure the generated identity aligns with the input embedding in the generation phase? \n2. How to make sure the diffusion model can generalize(recognize) this specific input embedding, considering the training embedding only covers a small range of the available embedding(training set)?\n\nLIMITATIONS:\nThis paper shows an attempt to generate diverse facial images of each identity. However, what makes solid FR training is not studied. Furthermore, domain GAP exists when adopting the trained diffusion model for generation, the input embedding might differ from the embedding of the synthesized image. Consequently, the focus is on changing facial attributes but preserving identity is interesting but the overall improvement is marginal, and FFHQ has license issues related to the application of Face Recognition. I think it will be more convincing if the results of CASIA-WebFace under 1M and 1.2 M settings are presented.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-30"}

comments: SUMMARY:
This paper proposes ID3, an identity-preserving-yet-diversified diffusion model for generating synthetic face data for face recognition. ID3 leverages identity embeddings and facial attributes to control inter-class and intra-class diversity of generated faces while preserving intra-class identity consistency, demonstrating state-of-the-art performance on multiple synthetic face recognition benchmarks.

STRENGTHS:
See questions section in detail.

WEAKNESSES:
See questions section in detail.

QUESTIONS:
The paper addresses a well-motivated and important problem in the field of synthetic face recognition. The proposed ID3 model demonstrates significant innovation in leveraging diffusion models conditioned on identity embeddings and facial attributes to generate diverse yet identity-consistent synthetic faces. Moreover, the theoretical analysis provided in the paper, which proves the equivalence between minimizing the proposed loss function and maximizing a lower bound on an adjusted data likelihood, lends credibility and rigor to the proposed approach.

However, there is room for improvement in the presentation and writing of the manuscript. One area that could benefit from further clarification is the explanation of notations and symbols used in the mathematical formulas. For instance, the meaning of the variable d in S
d−1 is not clearly defined, which may lead to confusion for readers. Additionally, the formatting and typesetting of some equations, such as Equation 3, could be enhanced to improve readability and aesthetic appeal.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"ID3: Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition\" introduces a novel synthetic face recognition (SFR) approach using diffusion models. It focuses on maintaining identity consistency while providing high diversity in generated face images. The proposed ID3 model leverages identity-preserving losses and a structured sampling algorithm that respects identity characteristics. This effectively addresses the common pitfalls of existing SFR approaches that lead to poor generalization on real-world data.\n\nSTRENGTHS:\n*   **Originality**: The paper presents an innovative use of diffusion models tailored to synthetic face recognition, emphasizing identity preservation.\n*   **Quality**: Demonstrated improvement over state-of-the-art models through extensive benchmarking.\n*   **Clarity**: Exceptionally clear presentation and thorough explanation of the methodology and results.\n*   **Significance**: This paper addresses significant challenges in synthetic data generation and offers substantial benefits for training more robust and generalizable face recognition systems.\n\nWEAKNESSES:\n*   **Generalization**: Additional tests on further diversified real-world datasets could strengthen the generalization claims.\n*   **Complexity**: It would be beneficial to have details on the computational demands and scalability of the model when deployed in practical, real-world scenarios.\n\nQUESTIONS:\n1.  What measures have been taken to ensure the model's robustness against diverse ethnic and age groups, given the model's reliance on identity embeddings?\n2.  Are there potential improvements or variations in the diffusion model that could further enhance identity preservation without sacrificing diversity?\n3.  How does the model perform under constrained computational resources, and are there any strategies for optimizing its efficiency?\n\nLIMITATIONS:\nThe paper discusses potential limitations, including the need for extensive computational resources and the model's performance dependency on the quality of input identity embeddings. It also mentions the ongoing challenge of bridging the gap between synthetic and real-world face recognition performance.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-21"}