$\text{ID}^3$: Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition
Authors: Jianqing Xu
Shen Li
Jiaying Wu
Miao Xiong
Ailin Deng
Jiazhen Ji
Yuge Huang
Guodong Mu
Wenjie Feng
Shouhong Ding
Bryan Hooi
Venue: NeurIPS 2024 poster (2024)
Keywords: synthetic face recognition
Primary area: diffusion_based_models

Abstract: Synthetic face recognition (SFR) aims to generate synthetic face datasets that mimic the distribution of real face data, which allows for training face recognition models in a privacy-preserving manner. Despite the remarkable potential of diffusion models in image generation, current diffusion-based SFR models struggle with generalization to real-world faces. To address this limitation, we outline three key objectives for SFR: (1) promoting diversity across identities (inter-class diversity), (2) ensuring diversity within each identity by injecting various facial attributes (intra-class diversity), and (3) maintaining identity consistency within each identity group (intra-class identity preservation). Inspired by these goals, we introduce a diffusion-fueled SFR model termed $\text{ID}^3$. $\text{ID}^3$ employs an ID-preserving loss to generate diverse yet identity-consistent facial appearances. Theoretically, we show that minimizing this loss is equivalent to maximizing the lower bound of an adjusted conditional log-likelihood over ID-preserving data. This equivalence motivates an ID-preserving sampling algorithm, which operates over an adjusted gradient vector field, enabling the generation of fake face recognition datasets that approximate the distribution of real-world faces. Extensive experiments across five challenging benchmarks validate the advantages of $\text{ID}^3$.

ID 3: Leenity-Preerringyyee--eeveesee Diffusion Models for Synthetic Face Recognition
Jianqing Xu* Shen Li+2
Jiaying Wu?, Miao Xiong", Ailin Deng", Jiazhen Ji', Guodong Mu', Wenjie Feng", Shouhong Ding', and Bryan Hooi
Yuge Huang
'Tencent Youtu Lab
National University of Singapore
Abstract
Synthetic face recognition (SFR) aims to generate synthetic face datasets that mimic the distribution of real face data, which allows for training face recogni- tion models privacy-preserving manner. Despite the remarkable potential of diffusion models in image generation, current diffusion- based SFR models strug- gle with generalization line three key objectives for SFR: (1) promoting diversity across identities (inter- class diversity), (2) ensuring diversity within each identity by injecting various facial attributes (intra-class diversity), and (3) maintaining identity consistency within each identity group (intra-class identity preservation). Inspired by these goals, introduce diffusion-fucled SFR model termed ID3, ID employs ID- ID-preserving loss to generate diverse yet identity-consisten facial appearances. Theoretically, show that minimizing this loss is equivalent maximizing the lower bound of an adjusted conditional log-likelihood over D-preserving data. This equivalence motivates ID-preserving sampling algorithm, which operates over an adjusted gradient vector field, enabling the generation of fake face recog- nition datasets that approximate the distribution of real-world faces. Extensive experiments across five challenging benchmarks validate the advantages of ID Code is released https: http::///ggttoo..m//tttrggggggg
real-world faces. To address this limitation, we out-
com/hitaarring2015/1SS
Introduction
With the introduction of various regulations restricting the use of large-scale facial data in recent years, such GDPR, synthetic-based face recognition (SFR) (Boutros al. 2023) has received widespread attention from the academic community (Qiu et al. 2021 Wood et al. 2021 Wang al. 2023). The goal of SFR real face images, and use it to train face recognition (FR) model such that the model can recognize real face images as effectively as possible.
generate synthetic face datasets that mimic the distribution of
There exist numerous efforts to address SFR, which can be categorized into GAN- models and diffusion models. GAN-based models utilize adversarial training to learn generate synthetic data for FR training. Recently, with the empirical advantages of diffusion models over GANs, many works have attempted to diffusion models generate synthetic face data in place of authentic data. However, the reported results by these state-of-the-an (SoTA) SFR generative models (Bac)
al. 2023 Boutros et al. 2022 Kolf et al. 2023 Qiu al., 2021 Boutros al. 2023 show
"Equal authors. The order was determined by numpy. www.om. ().
*Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

significant degradation in the verification accuracy in comparison to FR models trained by authentic data. We deduce the degradation might be due to two reasons. First, while previous works adopt diffusion models, they operate in the original score vector field without injecting the direction with regards to identity information, which makes them unable to guarantee dientity-preservinn sampling- Second, they fail to consider the structure of face manifold in terms of diversity during sampling.
We thus argue that the crux of SFR is to automatically generate training dataset that has the fol- lowing characteristics: (i) inter-class diversity: the training dataset covers sufficiently many distinct identities; (ii) intra-class diversity: each identity has diverse face samples with various facial at- tributes such as poses, ages, etc; (iii) infra-class identity preservation: samples within each class should be identity-consiste be fully automated without manual filtering or introducing auxiliary real face samples.
Also note that, critically, the SFR dataset generation process should
To this end, in this paper, propose novel Deentit-preservinn--eee Diveessiieeeieeedd Diffusion gen- erative model termed and a sampling algorithm for inference. Jointly leveraging identity and face attributes as conditioning signals, IDx can synthesize diversified face images that conform to desired attributes while preserving intra-class identity. Specifically, ID generates a new sample based upon two conditioning signals: a target face embedding and a specific set of face attributes. The target face embedding enforces identity preservation while face attributes enrich intra-class di- versity. To optimize ID², we propose new loss function that involves an explicit term preserve identity. Theoretically, we show that with the addition of this term, minimizing the proposed loss function is equivalent to maximizing the lower bound of the likelihood of an adjusted conditional data log-likee oogliikllhood. Consequently, this theoretical analysis motivates new ID>-rrserving sampling algorithm that generates desired synthetic face images. To generate an SFR dataset, we further pro- pose new cataset-generating algorithm. This algorithm ensures inter-class diversity by solving the Tammes problem [Tammes) 1930), which maximally separates identity embeddings on the face manifold. In the meantime, it encourages intra-class diversity by perturbing identity embeddings randomly within prescribed areas. works in conjunction with identity embeddings and diverse at- tributes to ensure inter-/intra-class diversity while preserving identity. Extensive experiments show that IDd outperforms other existing methods in multiple challenging benchmarks.
To sum up. our major contributions are listed as follows:
Model with Theoretical Guarantees: We propose IDx, an diversified diffusion model for SFR. Theoretically, optimizing ID3 is equivalent shifting the original data likelihood to cover ID- pr-preseg data.
yet-
Algorithm Design: Motivated by this theoretical equivalence, we design a novel sampling algorithm for face image generation, together with a face dataset-generating algorithm, which effectively generates fake face datasets that approximate real-world faces.
Effectiveness: Compared with SoTA SFR approaches, ID improves SFR performance by 2.4% on average across five challenging benchmarks.
2 Problem Formulation
The scope of this paper synthetic-based face recognition (SFR), which focuses on generating high-quality training data (i.e., face images) for FR models. Generally, we aim to address SFR by generating face images that conform to diverse facial attributes while preserving identity within each class, in automated manner. Technically, we break down this objective into the following two research questions (RQs) to be answered:
RQ1: How can we effectively train SFR generative model that preserves identity within each class, while boosting inter-class and intra-class diversity?
RQQ: Once the generative model is trained, what sampling strategy can be employed to generate a synthetic face dataset that enables state-of-the-art face recognition models to perform well on real face benchmarks?
The rest of the paper aims to answer these two questions, respectively, order to improve synthetic face recognition performance.

---

Trainable
Identity Conditioning Signal
Frozen
FR Model
unnerPrrodutt Term
(al-)
Attribute Conditioning Signal
Attribute Estimation Model
Sal
FR Model
Diffusion Model
Xx s,
noise
Diflusion Process
Denoising Pass
%
Denoising Term + One-Step Reconstruction Term
Figure 1:
The forward pass of ID in terms of loss computation. Given an image, its face attributes. and its face embedding. ID2 obtains the image's noised version after t diffusion steps and employs denoising network to denoise it. This denoising process is conditioned on the predicted attributes and the ID embedding. Optimization proceeds by minimizing loss function comprised of denoising term, one-step reconstruction term, inner-product term, and constant.
3 Methodology
We propose ID conditional diffusion model that generates diverse yet images. ID3 solves RQ1 by introducing two conditioning signals (identity embeddings and face attributes) into diffusion model which is trained using novel loss function. The loss function, together with identity embeddings, ensures intra-class identity preservation, while generation upon various face attributes give rise to intra-/inter-clas diversity of face appearances. Our theoretical result regarding this loss function leads to an ID-preserving sampling algorithm and, further, an effective dataset-generating algorithm.
face
Notations. Throughout the rest of the the paper, we denote a real face dataset that contains face images Xo E R Hx x3 Let denote desired identity embedding and s be face attributes.
3.1 Diffusion Models
We build up our generative model, ID3 upon denoising diffusion probabilistic models (diffusion models for short) (Ho et al. 2020 Song et al., 2022 Rombach et al. 2022) as they empirically exhibit SoTA performance in the field of image generation. Diffusion models be seen hierarchical VAE whose optimization objective to minimize the KIL divergence between the true data distribution and the model distribution pe, which is equivalent minimizing the expected negative log-likelihood (NLL), Ex-D[ [- log However, directly minimizing the expected NLL intractable. therefore diffusion models instead maximize its evidence lower bound (ELBO), where the ELBO term can further simply to denoising task with several model assumptions:
p(Xor)
log pxx) 2 Ex(xxrr/xon (x))
log qx11:T) (xo)
ELBO
1
Ey(xi (x0)
xx 1)12
Mc I|xo xo t)ll
2
where Pr : T-1 âe )
203(0) (1-6,)² , IT==1 ar,, Specifically, given a sample Xy (or interchange-
ably, x) from the image distribution, sequence X1, X2, ... XT of noisy images is produced by

---

progressively adding Gaussian noise according to a variance schedule a...... or. This process is called the forward diffusion process q(x, x,- ). At the final time step, X is assumed be pure Gaussian noise:
(0, f). The objective is to train denoising network &o that is able to predict the original image from the noisy image x; and the time step To sample new image, we sample xT ~ (0,, 7) and iteratively denoise it, producing a sequence XT, XT-1 ... X1, Xg. The final image, Xp. should resemble the training data.
Although the naive diffusion models are powerful in generating images, they do not deliver the promise of generating face images of the same identity (i.e. identity preservation) without direct corresponding information; nor are they aware of diverse desired facial attributes during inference. To achieve mntra-class diversity and intra-clast identity preservation, we would like to gain control of generating desired identities, each of which exhibits various attributes, including poses, ages and background variations. Hence, our aim to design diffusion model that conditions on specific identities and attributes throughout the generation of face images.
3.2 ID as Conditional Diffusion Models
We propose a conditional diffusion model, ID (see Figure for details). Specifically, we extend the denoising network by conditioning it on two sources of signals: identity signals y and face attribute signals s. The identity signals capture discernible faces in generated images, whereas face attribute signals specify the identity-irrelevy attributes, including poses, ages, etc. We introduce how to obtain these two conditioning signals, respectively, in the next two subsections.
3.2.1 Identity Conditioning Signal
To obtain identity conditioning signals, assume access pretrained face recognition model fø x x3 + gd-1, which maps the domain of face images to feature space gd-1, This mapping fo is parameterized by the learnable parameter Φ. which is obtained by training the model on a real face dataset in the face recognition task. We follow the latest advancement of face recognition by setting the output space to be a unit hypersphere gd-1, Then, given a face image Xg drawn from the dataset D, obtain its identity embedding y s by feeding it into face recognition model fø = f (x)), which serves as the identity conditioning signals for ID3
3.2.2 Face Attribute Conditioning Signal
Face attributes capture identity-irrelevnt information about face images, such age, face poses, etc. To obtain face attribute as conditioning signals, we employ pretrained attribute predictors (Serengil and Ozpinar 2021) which output these attributes when given face image input. The pretrained attribute predictors are a collection of ad-hoc domain experts in age estimation and pose estimation. After obtaining each of these attribute values, Sugo E [0, 100). Spose E [-900, 90%]3, we concatenate them as the overall attribute [Sgge Sposo] which is then fed into the diffusion model as condition-
ing signals.
3.3 Optimization Objective
Now the denoising network in Eq. becomes Xe (x,, that takes as input the noised x;, the time step 1, and the conditioning signals y and s. To optimize ID=3, we construct a training objec- tive upon the ELBO of log p(xly, ,ylyy,,, s), ensuring that ID3 generates cenntit-ppresevingg yet diversified faces:
min E (xxo (xo.
Here, 0 is the learnable parameter of the denoising network and the datapoint- loss is given by Lo (xo, y,8)
He Xo
X Axxx T fo denoising
x((1) 1
( + Eggisr (xo)
1 2 [|x0 - - $
+
inner-product term
step reconstruction
(3)

---

Algorithm 1: Training Algorithm
Algorithm 2: ID-Preserving Sampling Alg.
Input: The training face images ~ D; The pretrained face recognition model f-)-)
Input: Denoising network Xo; recognition
model føi conditioning signals y and
Output: The denoising network &g.
Output: A generated face Xo XT + sample from No I);
Initialize D <-
for ~ D do
for t <- T to 1 do
y fo(Xo);
Compute the score function V as in Eq. (
S Attributterredicto((os}}
D' < D' U {(xo,,,));
Draw a Gaussian sample e (); Perform the update:
end
Solve Eq. 2 using batched Backpropagation algorithm with D'; return Xe
xt-1 + + V log p( + 24e;
end return Xo
where 86 is the output of the denoising network that takes as input the conditioning signals y, the time and the t-step noisified image X₁:
x (X, t,y.).
(4)
Symbolically, x0 denotes the denoised image predicted by the denoising network when given the 1-step noisified Xt, the time L and the associated conditioning signals y, The coefficients, Kix, and d are scalars depending Xo and I, respectively, and C' is constant that does not depend on the learnable parameters 0. The specific value of C' will be elaborated in Appendix A.
To summarize, our proposed loss function consists of four terms: the denoising term, the inner-product term, and constant. Intuitively, the denoising term, along with the one-step reconstruction term, aims to improve the generative quality by denoising the t-step noisified face images while the inner-produe fied images to get close to the groundtruth identity embedding. To understand this loss function systematically, we theoretically find that minimizing this proposed loss function is equivalent to the maximization of the lower bound of an adjusted conditional log-like aigllikelihood over identity-preserviie preserving face images, which further leads us to an ID-preserving sampling algorithm.
the one-step reconstruction term,
term encourages the face embedding of the denoisi-
Theorem 3.1. Minimizing with regard to 0 is equivalent to minimizing the upper bound of an adjusted conditional data negative log- oilliikel - log p(xly, s), i.e.:
min L(xu, y, 2 - log p (x
(5)
0
where
p(xly, x p(x s|x, E
(6)
Proof.
The proof can be found in Appendix A.
Remark. We have just shown that our proposed loss is the upper bound of an adjusted conditional negative data log likelihood. This adjusted likelihood p(xly,s) can be factorized into the original likelihood p(xly,s) s) and a reversed likelihood p(y, s|x) with some positive power. We term it as "adjusted" since the original likelihood is discounted by the reversed likelihood. Intuitively, the re- versed likelihood shifts the original likelihood such that the adjusted likelihood covers ID) pr-peseeriing data, which is attributed to the inner-product term we introduce into the loss function in Eq.
3.4 DD-Preeervig Sampling
provides insights for designing novel sampling algorithm in the spirit of Langevin dynamics applied on the adjusted conditional likelihood p(x, s). We note that Langevin dynamics can generate new samples from a probability density by virtue of its score function (i.e., the gradient of the logarithm of the probability density w.r.t. the sample, Vxlog Motivated by this observation, we aim to find the score function of the adjusted likelihood for sample generation. Specifically, taking the logarithm and the gradient w.r.t. x on both sides of Eq. 6 yields
Theorem 3.1
Σ d V p(gp((ss)))
V p(xly,s Vlogp(xly,s) +
T-1
5

---

Algorithm 3: Synthetic Dataset Generation
los
Input: Denoising network Xe; recognition model fa: the number of identities N.
Output: A synthetic dataset Diyn.
Day + 0;
Generate Wi..., WN E gt- by solving the Tammes
los pllly s)
problem:
for i to N do
Generate 811: 8ms [bb, ub);
Calculate Y; by solving the optimization problem P;
Eq. !
Yú.... Yim +- umpack(Y.):
Sa.... 8.m generate different attributes;
ply. six)
D. +
for j f to m do
No f Alg.
xo. fa. norm(y?)).ss)) D. + D; U 1):
Dyn Dan
D.:
Qualitative comparison of face images generated by the ad- justed score function V log, ly, s) and the original log p(x:ly,ss.
Figure 2:
end
end
return Dan
function
Then, our ID-preserving sampling algorithm first draws a Gaussian sample xy ~ N I). After- wards, sequentially, the algorithm performs the following update for & iterating from T backwards to 1:
X t x, + V log ((x 2
where
Σ d
v p(x y.s)
v log lox
log boggpyyssan
(8)
T
original likelihnod score
reversed likellhood score
Note that the original likelihood score in Eq. (8) can be evaluated by
V Vlog
Var
(,
X Vā
VI- dz.
and the reversed likelihood score is given by scaled inner product:
V Plyg
Kx. f
See Appendix B for the derivation the above equations. As such, our ID ID-preserving sampling algorithm performs sampling by searching trajectory in the vector field V log p(x:ly,s) that can maximize the adjusted conditional likelihood p(x: (y, See Algorithm@ for the specific procedure.
Our proposed adjusted likelihood score differs from the original score by adding extra scaled reversed likelihood score in Eq. (8). Consequently, as shown in Figure 2, the resulting vector field differs from the original vector field, which leads to different Langevin sampling trajectories and thus different sampling quality.
Remark.
3.5 Synthetic Dataset Generation
In terms of the second question (RQ2): after training ID3, with what sampling strategy is it possible generate a synthetic face dataset on which SoTA face recognition models can be trained and perform well on challenging benchmarks?
Our proposed dataset-generating algorithm goes follows: given N target identities, generate N anchor embeddings distributed on the sphere: W1, W2,... wx E gd-1 as uniformly as possible in the sense that each pair of the embeddings are maximally separated on the unit sphere For each anchor w,, we would like to generate m identity embeddings perturbed around w; while ensuring
"This is known as the Tammes problem Taames 1930) for which there exists no exact solution for hyper- sphere gd-1 1 d 3. However,
the optimization technique introduced in (Mettes et al. 2019

---

that these m identity embeddings get close to but different than w, Specifically, to find these m identity embeddings, we solve the following optimization problem P,:
Nim ) [1s V/2, .. Veme
min
[- morm
Yal
Y12
where the operator norm is column-wiss normalization which normalizes each column of Y; into unit vector, and the desired similarity scores V/1. P/2,, .. Pon are randomly generated from a con- tinuous uniform distribution [[bb, ub). After solving Eq. ( we are able to retrieve the optimal unnormalized vector y:r.... Vim- These m vectors are then normalized, yielding m identity em- beddings: norm(y;)) ()): for j = ,.... m. Then, the resulting identity embeddings, along with face attributes, are fed into our generative models to generate face images. Finally, the entire dataset is generated by solving each p, 1, N, which yields N identities, each with face images. The entire algorithm is summarized Algorithm
4 Experiments
In this section, we verify the effectiveness of IDd through empirical evaluation of the face dataset that IDx generates, and verify the performance of the SoTA face recognition model trained on this dataset in comparison with other baseline methods.
4.1 Dataset
Training Dataset: We train our proposed ID on FFHQ (Karras et al. 2019) dataset. The FFHQ (FaceForensicc++)) dataset is a large-scale dataset used for benchmarking and evaluating the perfor- mance of deep learning models in the field of face forensies. It is extension of the original Face- Forensics dataset, which was designed to facilitate the development and comparison of methods for detecting and preventing face manipulation and deepfakes. In order to compare with DCFace (Kim jet al. 2023), we also train ID3 on CASIA-WA -ebbaee (Yi et al. 2014). The CASIA-WebFace WebFace dataset used for face verification and face recognition tasks. This dataset contains 494,414 face images of 10,575 real identities collected from the web.
Benchmarks: The performance of face recognition models is evaluated various benchmark datasets: LFW (Huang et al. 2008), CFP-FP (Sengupta et al., 2016), CPLFW (Zheng and Deng 2018), AgeDB (Moschoglou al. 2017) and CALFW (Zheng al. 2017). They are used to mea- sure the impact of different factors on face image, such as pose changes and age variations.
4.2 Implementation Details
For our ID', we implement the denoising network with a U-net architecture and the projection module with three-layer perceptron (hidden-layer size (512, 256, 768)) with ReLU activation. All
Figure Uncurated samples generated by I (Top) and those by IDiff-Face (Bottom).

---

Table I: SoTA Comparison. Face verification accuracy (%) of LResNet5O-IR different benchmarks when
trained on synthetic datascts from ID' and state-of the art SFR generative models. For fairness, all methods generate face datasets of 10K identities each of which has 50 face images.
Method
Training set
LFW
CFP-FP
CP-LFW
AgeDB
CA-LFW
Average
ID-Net
FFHQ
84.83
70.43
67.35
63.58
71.50
71.53
DigiFace
FFHQ
88.07
70.99
66.73
60.92
69.23
71.19
SFace
FFHQ
91.43
73.10
73.42
69.87
76.92
76.95
SynFace
FFHQ
91.93
75.03
70.43
61.63
74.73
74.75
IDiff-Face
FFHQ
97.10
82.00
76.65
78.40
86.32
84.09
IDd (Ours)
FFHQ
97.28
85.00
77.13
83.78
89.30
86.50
DCFace
FFHQ+CASIA
98.55
85,33
82.62
89.7
91.6
89.56
ID (Ours)
CASIA
97.68
86.84
82.77
91.00
90.73
89.80
models are implemented with PyTTorch and trained from scratch using 8 NVIDIA Tesla V100 GPUs. Specifically, we Ax 0.5 (1 1/(1 exp(T) for the loss coefficients in Eq. (3) and use T = 1, 000 for the diffusion model; training batch size is set to 16 and the total training steps 500, 000. We directly use a pre-trained face recognition (FR) model sourced from pSp (Richardson et al. 2021) as the identity feature extractor. Throughout the entire training process, these pre-trained models are frozen. In addition, we set # of identity embeddings m 25 in Eq. ( for each ID and match their embeddings with randomly selected attributes as conditioning signals for the diffusion model. For face recognition, we use LResNet50-IK (Deng et al.] 2019). variant of ResNet (He) et al. 2016 as the backbone framework and follow the original configurationss
4.3 Performance Evaluation
We test the performance of the face recognition model trained on synthetic face data generated by ID's and compare against SoTA SFR generative models, including IDiff-Face (Boutros et al. 2023).
ID-Net (Kolf al., 2023). DigilFace (Bae al. 2023), SFace (Boutros al. 2022). Synl-cce (Qu
et al. 2021) and DCFace (Kim et al. 2023).
4.3.1 Qualitative Results
Here, we illustrate a collection of face images generated by ID qualitative evaluation. Figure shows the results for randomly sampled identities (IDs) under various attribute conditions; Obvi- ously. when comparing different identities (inter-class), the essential intrinsic key information of each identity is still retained and can be easily identified. Also, different samples of each identity (intra-class) exhibit distinct diversity, stemming from variations in similarity scores (Vij 's) and dif- ferences in face attributes as conditioning signals. In terms of the effect of our proposed adjusted score and the original score on the sampling algorithm, we observe that the face images generated by our proposed ID' exhibits much better quality and identity preservation than those generated by the original score function, as shown in Figure 2.
4.3.2 Quantitative Results
We compare the accuracies of FR models trained on the synthetic face datasets generated by different generative models and demonstrate the results in Table
As shown in Table I ID demonstrates consistent superior performance, achieving the highest aver- age accuracy of 86.50% and outperforms other baselines in all benchmarks, notably scoring 83.78 73% in AgeDB and 55.00% in CFP-FP. This demonstrates the effectiveness of ID3 in gaining pose and age control. Other methods, while effective to varying degrees, attain average scores below 86.50% and are inferior to ID3
It worth mentioning that ID3, apart from using real data during training, does not introduce any real images as auxiliary data during the sampling phase. The synthetic data is directly used in the training of the face recognition model without undergoing any secondary or manual filtering- Additionally, when training the face recognition model using the synthetic data, no real images are introduced as auxiliary data. the other hand, DCFace, as described and reported (Kim al.) 2023), introduces real face images auxiliary data during the training phase for face recognition.
8

---

Table 2: Ablation Study. Face verification accuracy (%) of LResNeL5O-IR when trained synthetic datasets from ID' and other model variants, ID"-[1b, ubj represents an ID' variant using db and ub as lower- and upper- bound for sampling Vij s. ID' -random denotes a model variant that randomly sets anchors on the unit hypersphere for sample generation. ID hoo-wwattribute denotes one that does not use attributes as conditioning signals. IDD w/ooreeerrse denotes one that removes the reversed likelihood score from Eq. (8) in the proposed ID-preserving sampling algorithm.
Method
LFW
CFP-FP
CP-LFW
AgeDB
CA-LFW
Average
DDd w/o-reversed
78.82
62.68
61.83
58.20
63.71
65.05
ID -/-ooaattribute
97.12
85.57
81.70
87.50
89,48
88.27
ID ID*-[0.7, 0.9]
97.28
84,26
81,48
86.25
89.63
87.78
ID [0.5, 0.7]
97.38
85.00
81.10
86.63
90.13
88.05
ID rraddom
96.00
80.81
78.05
85.53
87.57
85.59
ID (Ours)
97.68
86.84
82.77
91.00
90.37
89.80
This helps enhance the diversity of the training data and leads to slightly better results than ID$ in the two benchmarks.
Ablation study. We further investigate the impact of cach contributing component of ID3 in gen- erating a synthetic face dataset on SFR. This includes three ablation studies shown in Table 2 the effect of the reversed likelihood score in Eq. (8) on -D-reserying sampling algorithm (ID vs. ID w/o-reversed), the effect of using anchors in IDx (ID3 vs. ID 1D*-random), and the effect of lower- and upper-bound of Uniform distribution for sampling Vij's.
In the first study, we compare ID with ID w/o-reeereeeedd which removes the reversed likelihood score from Eq. (8) in the proposed ID- ID-preserving sampling algorithm. We observed ID3 consistently outperforms ID w/o-reverse with large margins. This suggests the necessity of the inner- inner-product term in the proposed loss function Eq. (3) and the reversed likelihood score in the adjusted likelihood score Eq.
In the second study (ef. Appendix E), appropriate smaller value of (b, if not exceeding cer- tain range, can increase the intra-class diversity, resulting in more diverse intra-class face images. This aligns with our objective of increasing intra-class diversity in the generated data enhance the effectiveness of SFR. As per the constraints of Eq. (9), each generated identity embedding Yij maintains the same identity as the anchor w,, This, along with our proposed inner-product term in Eq. (3) ensures consistent intra-class identities while introducing a significant amount of diversity. In the third study, we demonstrate how effective it is to maximally-separatte separated anchors in ID as compared to 3D3 IDa-random that randomly sets anchors on the unit hypersphere for sample gen- eration. Clearly, ID²-random does not yield good results as ID This is because the random sampling method only introduces one identity signal per ID, while the model requires combina- tion of attributes and identity signals. Attribute signals can only control explicit attributes, whereas identity signals control implicit properties. Introducing only one identity signal per identity implies insufficient intra-class diversity. Additionally, -D-rranomm fails regularize the relationship among different identities, leading to inadequate diversity among classes or aliasing issues with different identity signals. The identity signals obtained using ID resolves the problem of aliasing between identity signals across classes, effectively improving intra-finter-class diversity.
Conclusion
We have proposed ID identity-preserviine -ewtrrvvinnyyt---ivrsiiiiiee diffusion generative model for SFR. Our theoretical analysis regarding the training of ID' induces a new ID-preserving sampling al- gorithm and further, dataset-generating algorithm that generates with inter-/intra-clle diversity. Extensive experiments show that ID outperforms existing methods in challenging multiple benchmarks.
face images
Limitations. While ID3 designed for the sake of privacy protection, achieves SoTA performance in SFR, there remains clear margins as compared to the FR performance when training with real-world face datasets such as MSIM. This suggests that the fake face dataset generated by ID does not fully approximate the real-world faces. Future work might include closing this gap.

---

References
Gwangbin Bae, Martin de La Gorce, Tadas Baltrušaitis, Charlie Hewitt, Dong Chen, Julien Valentin. Roberto Cipolla, and Jingjing Shen. Digiface- Im: million digital face images for face recognition. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 3526-3535, 2023.
Fadi Boutros, Marco Huber, Patrick Siebke, Tim Rieber, and Naser Damer. Sface: Privacy-friendly and accurate face recognition using synthetic data. In 2022 IEEE International Joint Conference on Biometrics (IJCB), pages 1-11. IEEE, 2022.
Fadi Boutros, Jonas Henry Grebe, Arjan Kuijper, and Naser Damer. Idiff-face: Synthetic-based face recog- nition through fizzy identiy-conditioned diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19650-19661, 2023.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690 4699, 2019.
Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3d imitative-contrastive learning. In IEEE Computer Vision and Pattern Recognition, 2020.
Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celet- Im: A dataset and henchmark for large-scale face recognition. In Computer Vision-ECCV 2016: 14th Europaa Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IIl 14, pages 87-102. Springer, 2016.
Abul Hasnat, Julien Bohné, Jonathan Milgram, Stéphane Gentric, and Liming Chen. von mises-fisher mixture model-based deep learning: Application to face verification. arXiv preprint 1rXiv:1706.04266 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. Advances in neural
information processing systems, 33:6840-6851, 2020.
Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Leamed-Miller. Labeled faces the wild: A database forstudying face recognition unconstrained environments. Workshop on faces Real-Life' Images: detection, alignment, and recognition, 2008
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4396 4405, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems, 33:12104- 12114, 2020.
Minchul Kim, Feng Liu, Anil Jain, and Xiaoming Liu. Deface: Synthetic face generation with dual condition diffusion model, 2023.
Jan Niklas Kolf, Tim Rieber, Jurek Elliesen, Fadi Boutros, Arjan Kuijper, and Naser Damer: Identity-driven three-player generative adversarial network for synthetic-based face recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 806 816, 2023.
Shen Li. Jianqing Xu, Xiaqing Xu, Pengcheng Shen, Shaoxin Li, and Bryan Hooi. Spherical contidence learning for face recognition. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 15629-15637, 2021.
Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Photomaker: Cus- tomizing realistic human photos via stacked id embedding-suupleeenntay materials-.
Jianxin Lin, Yingce Xia, Tao Qin, Zhibo Chen, and Tie- Yan Liu. Conditional image-to to to-image translation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5524-5532, 2018.
Calvin Luo. Understanding diffusion models: A unified perspective, 2022.
Pascal Mettes, Elise van der Pol. and Cees G M Snoek. Hyperspherical prototype networks. In Advances in Neural Information Processing Systems, 2019.
10

---

Stylianos Moschoglou, Athanasios Papaioannou. Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, tu-the-wild age database. In proceedings of the IEEE confer- ence on computer vision and pattern recognition workshops, pages 51-59, 2017.
Haibo Qiu, Baosheng Yu, Dihong Gong, Zhifeng Li. Wei Liu, and Dacheng Tao. Synface: Face recognition with synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10880-10890, 2021.
Elad Richardson, Yuval Alaluf, Or Patashmik, Yotam Nitzan, Yaniv Azar, Stav Shapiro. and Daniel Cohen- Or. Encoding in style: a stylegan encoder for image-to to to-image translation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages (0684-10695, 2022.
Soumyadip Sengupta. Jun- Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. Frontal to profile face verification in the wild. In 2016 IEEE winter conference on applications of computer vision (WACV), pages 1-9. IEEE, 2016.
Sefik Ilkin Serengil and Alper Ozpinar. Hyperextended lightface: A facial attribute analysis framework. In 2021 International Conference on Engineering and Emerging Technologies (ICEET), pages 1 1-4. IEEE, 2021.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusson implicit models, 2022.
Pieter Merkus Lambertus Tammes. On the origin of number and arrangement of the places of exit on the surface of pollen-grains. Recueil des travaux botaniques néerlandais, 27(1):1-8 84, 1930.
Dani Valevski, Danny Lumen, Yossi Matias, and Yaniv Leviathan. FaceO: Instantaneously conditioning a text-to-image model face. In SIGGRAPH Asia 2023 Conference Papers, pages 1-10, 2023.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5265 5274, 2018.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot adentity-preservvvv generation in seconds, arXiv preprint arXiv:2401.00 07519, 2024.
Wenging Wang. Lingqing Zhang, Chi-Man Pun, and Jiu- Cheng Xie. Boosting face recognition performance with synthetic data and limited real data. In ICASSP 2023 2023 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). pages 1-5, 2023.
Erroll Wood, Tadas Baltrušaitis, Charlie Hewitt, Sebastian Dziadzio, Thomas Cashman, and Jamie Shotton. Fake it till you make it: face analysis in the wild using synthetic data alone. In Proceedings the IEEE/CVF international conference on computer vision, pages
2021.
Jianqing Xu, Shen Li. Ailin Deng. Miao Xiong, Jiaying Wu, Jiaxiang Wu, Shouhong Ding, and Bryan Hooi. Probabilistic knowledge distillation of face ensembles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3489- 1499, 2023.
Yuxuan Yan, Chi Zhang. Rui Wang, Yichao Zhou, Gege Zhang, Pei Cheng, Gang Yu. and Bin Fu, Facestudio: Put your face everywhere in seconds. arXiv preprint (tt::2332.02663:3 2023.
Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z. Li. Learning face representation from scratch, 2014.
Tianyue Zheng and Weihong Deng. Cross-pose Ifw: A database for studying cross-pose face recognition in unconstrained environments. Beijing University of Posts and Telecommunications Tech. Rep, 5(7), 2018.
Tianyue Zheng, Weihong Deng, and Jiani Hu. Cross-age Ifw: A database for studying cross-age face recogni- tion in unconstrained environments. arXiv preprint 1rXiv:170..08199 2017.
Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Jiwen Lu, Dalong Du, et al. Webface260m: A benchmark unveiling the power of million-scale deep face recog- nition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10492- 10502, 2021.
11

---

ID: Reentt--Pessrrinnyye--ivveeeeeee Diffusion Models for Synthetic Face Recognition
Supplementary Material
A Appendix A
We first present a lemma (Lemma A.1) that will further used to prove Theorem 3.1.
Lemma A.1. Given any conditional probability density functions, p(a(b) and p(bla), and positive scalar w > O, there exists another conditional probability density function such that
1
p(alb)p(b(a)) where Za, = p(a.b) Zb,
p(alb))(blaa) da.
(A.1)
Proof. To show this result is equivalent to showing p(alb) & p(alb) p(bla)", w > 0, which is equivalent to showing that, for any bo,
p(alb bo) x p(a|b bu) p(b by a)) w 0
(A.2)
Note that p(a(b = by) is a function of a, i.c. there exists a function foo. such that p(b = bala)" :=
(b Let
Zho p(calb bo) Joo (a)
(A.3)
Then,
p(alb bo) p(b bola) p(alb bo) fo
(A.4)
Zb,
Zbo
is proper probability density function of a with b = by given. Therefore, Eq. (A.4) can be written as p(a.b = by). The above proof holds true for any by. which concludes the proof for p(a(b) x p(alb) p(bla)", O. Note that this result can be trivially extended to multivariate random variables.
Theorem 3.1. Minimizing with regard to equivalent to minimizing the upper bound of an adjusted conditional data negative log-likeldhood - log p(x(y, 8), i.e.:
min Lo (Xo. v, log 5l s)
(A.5)
where
s(xly,s) & p(xxy, s) ply,s/x) and Xy and x both refer raw image interchangeably.
(A.6)
Proof. Recall that
A T fo (x)")
Xo x (1)
Lo y.s) E
Pe Xo
+ E Byy a (co)
+
demoising Lerm
inner-procucct torm
one slep reconstruction lerm
Me Xo x 1011 E2-
+ EE [xc])x))))
x 1|
1
+ Emaup,
Aetix fo (xg)
(2(11)
demoising term
one-step reconstruction term
inner-product term
(A.7)
can be shown that the reversed likelihood p(y, s/x) is a joint vMF density Xu ct al.
2023 Hasnat
al. 2017):
ply. s|x) x)pss() p(ylx)p(s|x) J2 exp fo (x) + Fx()))
(A.8)

---

where JR, is the normalizing constant and Fa is the pretrained attribute predictor. Note that Equality (1) is obtained by the product rule of probability; Equality (2) obtained by observing that y and are conditionally independent when x is given; and Equality (3) is obtained by assuming that
ply Ja. exp (Ni y fø (x)), p(six) Juz exp (Kx s Fx(x))
(A.9)
Note that these reasonable assumptions are also held in (Xu al. 2023 Li ct al. 2021 2017 Now we can specify the value the scalar C::
Hasnat ct al.
C (--BA. d (log Jog 22 + Fa(x))
log(2 DKL (g(xT (o)xxr/x))lp(x)))) log Z
(A.10)
where n 3HW is the dimensionality of x, and
Z f p(xly.s) ply. s/x)
Σ de dx y,s, T-1
(A.11)
> Note that Z only depends on y, 8 and T-1 and hence C' is a scalar that does not depend on the learnable parameter 0. Therefore, L can rewritten into a sum of two parts: L = L1 + L2, where
C1 Eu2r2
Pt Xo - x(1)
+Bg(xi
X
(1)
-
log((7)+DKi. (a(XT (x0)
denoising
Lone step reconstruction
(A.12)
and
de S2 + AeNxss Pa(x) + A So (x)+²)
L2 EEwue,,
log Z
(A.13)
inner-product term
We recognize -1 is the evidence lower bound (ELBO) of (o
Luo 2022 i.c.
L1 2 - loggopplly
(A.14)
As for L2:
L2 =-Easigp.T
log J2 + KxY" So (x00)
+xxs Fa(x)
log
inner product term.
(A.15)
X
(1) log J2 RxY fo (x) + TraxSFFF Fx(x)
+ log Z
T
Enner-product term
Here assume access to perfect denoising module such that x! Xo for all l's. Hence, L2 can be written as
L2
De (log Jhy + kxy (x) KxS Fa + log 7
(A.16a)
D) log ply, s|x)] + log Z
(A.16b)
T
T
X1
log + log Z
(A.16c)
T-1
1=2
- --logply, s|x,
+log Z
(A.16d)

---

1W
W.....
W.
Figure A.I: An illustration of the dataset-generatie algorithm.
Then,
L L1 + L2
(A.17a)
> Ae p-1
logp(xly,s) + logply, s(x)
log
(A.17b)
log (Z (xx)y)))s))))) log Z
(A.17c)
- -logp(xxyy,s
(A.17d)
where Equality (A.17c) is obtained by applying Lemma A.1. This completes the proof.
B Appendix B
In this section, we show the derivations of the following equations:
Vor
V Vlogp(x(ly,s= VT aie
(xo(xx (x
(A.18)
Vār
Vlogply, Hix, fx)( + S
(A.19)
Recall that in the main text, we showed that the adjusted likelihood score summation of the original likelihood score and the scaled reversed likelihood score:
El T-1
v log p(x: s)
V llogp(xelye s)
logply, s(x,)
(A.20)
adjusted likelihood score
original likelihood score
reversed likelihood score
For the original likelihood score, we note that our proposed ID itself is conditional diffusion model. By virtue of the relation between the score and the denoising module (i.e. the Tweedie's Formula) diffusion models (cf. Equation (133) in (Luo 2022)), we able show that
V logp(x(ly.s) Vār
X
(A.21)
Va
Here, Xo can be approximated by denoising X₁ via the trained denoising module
Xg X (X,, (,
(A.22)
C An Illustration of the Dataset Generating Algorithm
We illustrate the dataset generating algorithm in Figure A. First, N anchor embeddings are gen- erated on the sphere as uniformly as possible. Then, for each anchor, m identity embeddings are generated around the anchor. This strategy ensures inter-class diversity while intra-class identity preservation guaranteed. Colors show the correspondence between the generation procedure on the left and the generated samples the right.
3

---

Irtra-class
3503
3000
300
3000
300
Coalna Simelartt
Coann Sinierty
[lb. ub] [0.5, 0.7]
Middan
12220
400
300
-1.33
-6.75
-3.50
-6.3
0.0
0.25
030
0.5
1.0
-3.03
-0.5
-3.50
-0.5
0d0
3.55
6.50
0.55
100
[lb, ub] - [0.7, 0.9]
Figure A.2: The distribution of inter-class and intra-class similarities.
D
Related Work
Face Recognition. Face Recognition (FR) is the task of matching query imagery to an enrolled
identity database. SoTA FR models are trained using margin-basee baaeed softmax losses (Wang et al.,
2018 Deng et 2019) on large-scale web-crawled datasets (Guo et al. 2016 Zhu et al. 1021). These datasets encompasses three characteristics in common mentioned the introduction): (1) sufficient inter-class diversity; (ii) intra-class diversity; (iii) intra-class identity preservation. How- ever, due to the introduction of various regulations restricing the use of authentic face data, re- searchers switch their attention to synthetic face recognition (SFR). We argue that the crux of SFR to generate a training dataset that inherits the three characteristics above.
GAN-based SFR models. Most of the deep generative models for synthetic faces generation are based on GANs. DigiFace (Bae et al. (223 utilizes digital rendering pipeline generate syn- thetic images based on learned model of facial geometry and attributes. SFace (Boutros et al.) 2022) and ID-Net (Kolf et al.) (023) train StyleGAN-ADA (Karras al., 2020) under class- conditional setting. SynFace (Qiu et al. 202T) extends DiscoFaceGAN (Deng et al. (020) using synthetic identity mix-up to enhance the intra-class diversity. However, the reported results shown by these models show significant performance degradation in comparison to FR trained on real data. This performance gap is mainly due to inter-class discrimination and small intra-class diversity in their generated synthetic training datasets. Diffusion models for SFR. Recently, Diffusion Models (DMs) (Ho et al. 2020 Lin et al.) 2018 Song et al., 2020) gained attention both research and industry due to their potential to rival GANs on image synthesis, they are easier to train with- out stability issues, and stem from a solid thcoretical foundation. Among SFR diffusion models, IDiff-Face (Boutros et al. 2023) achieves SoTA performance. On the basis of diffusion model, incorporates Contextual Partial Dropout to generate diverse intra-class images. However, IDiff-Face fails to regularize the relationship among different identities.
Latent Diffusion Models. There exist many diffusion- based models (LDMs) (e.g., FaceO Valevski et al., 2023), PhotoMaker (Li et al.). FaceStudio (aan et al. (023). InstantID (Wang et al. 2024) which use ID attributes to assist in generating images. However, we note that these LDMs are designed for image generation but not for SFR. Our empirical findings further suggests these LDMss
do not perform reasonably well even when applied to SFR. We found that although these L.Mss claim to be ID-preserving in the pixel space, their feature embeddings are not discriminative enough

---

Blatitionne Anglare Demet Geamaet by
Dirrkaaie ofheu Kepk Dase: Gonzmadity Pradaty
DerhaskknnTho Angli Dasse Gonereed by fruoddy
Dirthaaieaaaon Aegli Daaset Gonzmed by Tradddyy
Woe Ampl--mthHQ
Pry Dmyy
Poc Anglc
Dooc Angi
Peie Angle
Pire
Pare
Figure A.3: Pose distribution of datascts generated by different models.
DitnibatinnoPPhiso FFRQ
DitritatioonSYee
Disirbutiin ofRallen
Disa of
Fraquancy
Figure A.4: Pose distribution (pitch, yaw, roll) and age distribution on FFHQ
for face recognition, since there is inductive bias (neither loss functions nor architectures) achieve face discriminativeness.
E
Ablation Study (ii)
We show the inter-class and intra-clase similarity in Figure A.2 when using [0.5, 0.7] and [0.7, 0.9] as the lower- and ppunr- [tb, ub) for sampling Viy" 's in our proposed dataset-generatin algorithm.
F Attribute Analysis of Generated Datasets
In this section, we perform an attribute analysis of the generated face datasets by ID As shown in Figure AA.3 and Figure A.4 from the distribution, observe that the highest age in training set is 70; the largest pose is in degree and the smallest -60 in degree. Our model can interpolate within these ranges but is less likely to extrapolate outside of these ranges. To generate face images of large pose and high age, one can collect more such data and add them to the training dataset, which increases their occurrences during training.
To examine whether the attributes used in our paper are fit for SFR tasks, we perform the follow- ing ablation study: we increase intra-class pose variation, the SFR performances on cross-post test sets (including CFPFP and CPLFW) are boosted whereas the performances on cross-age test sets (including AgeDB and CALFW) remain almost unchanged. Results and distribution plots are shown in Table A.I Figure A.3 and Figure A.4 From these results, we observe that the distribution of pose angle FFHQ, Dataset by IDifiFace, Dataset by ID PoseOnly and 2 are all unimodal. And their performances are inferior to Dataset by ID PoseOnly which exhibits multimodal distribution of pose angle.
Table A.l: I: Datasets generated by different models (Column attribute statisties for each dataset (Column 2, 3), and the FR performance of FR models trained on them, respectively (Column 4-8).
Pose Mean
Pose Var
LFW
CFP-FP
CPLFW
AgeDB
CALFW
FFHQ IDifTFace
11.44
233.97
11.62
222.62
97.10
82.00
76.65
78.40
86.32
ID PoscOnly 1
8.21
109.76
95.33
78.41
73.48
79.76
86.03
t PoseOnly 2
9.02
110.18
95.58
80.91
73.60
79.45
85.93
ID PoseOnly 3
14.14
247.05
95.83
82.87
75.77
79.45
86,90
5

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
As shown in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
As shown in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Ys]]

---

Justification:
As shown in the Appendix A and B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
As shown in the experiment and appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
7

---

Answer: [Yes]
Justification: see Abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips...
for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips... ctc hwtppppllbuulpeeeeeeppppppppppae /edeeeeeeeoeuebaeeeea ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification:
As shown in the experiments and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.

---

is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
As shown in the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://wwrrg..cooppo/o///ccccaaa
Answer: [Yes]
Justification: The research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: NA
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited. The license and terms of are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, age should be provided. For popular datasets, haperswithcode com/ hom/aatseets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
license, copyright information, and terms of use the pack-
10

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, applicable, well as details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "vXzxZxa68p", "reviewer": "Reviewer_qiUe", "summary": "The paper \"ID3: Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition\" introduces a novel synthetic face recognition (SFR) approach using diffusion models. It focuses on maintaining identity consistency while providing high diversity in generated face images. The proposed ID3 model leverages identity-preserving losses and a structured sampling algorithm that respects identity characteristics. This effectively addresses the common pitfalls of existing SFR approaches that lead to poor generalization on real-world data.", "questions": "1.  What measures have been taken to ensure the model's robustness against diverse ethnic and age groups, given the model's reliance on identity embeddings?\n2.  Are there potential improvements or variations in the diffusion model that could further enhance identity preservation without sacrificing diversity?\n3.  How does the model perform under constrained computational resources, and are there any strategies for optimizing its efficiency?", "limitations": "The paper discusses potential limitations, including the need for extensive computational resources and the model's performance dependency on the quality of input identity embeddings. It also mentions the ongoing challenge of bridging the gap between synthetic and real-world face recognition performance.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "*   **Originality**: The paper presents an innovative use of diffusion models tailored to synthetic face recognition, emphasizing identity preservation.\n*   **Quality**: Demonstrated improvement over state-of-the-art models through extensive benchmarking.\n*   **Clarity**: Exceptionally clear presentation and thorough explanation of the methodology and results.\n*   **Significance**: This paper addresses significant challenges in synthetic data generation and offers substantial benefits for training more robust and generalizable face recognition systems.", "weaknesses": "*   **Generalization**: Additional tests on further diversified real-world datasets could strengthen the generalization claims.\n*   **Complexity**: It would be beneficial to have details on the computational demands and scalability of the model when deployed in practical, real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that Reviewer qiUe appreciates our work in terms of originality, quality, clarity and significance. Here we respond to your questions as follows. Hopefully it will address your concerns.\n\n**Weakness 1: Additional tests on further diversified real-world datasets could strengthen the generalization claims.**\n\n**Response:** Thanks for the suggestion. To show the generalization of ID$^3$, we further run tests on RFW and obtain the following comparison results against the best SoTA model (IDiffFace) other than ID$^3$:\n\n| Models | African | Asian  | Indian | Caucasian | average |  std |\n|----------|----------|----------|----------|----------|----------| ----------|\n| IDiffFace | 71.10 | 75.83 | 78.68   | 82.05   | 76.91    | 4.64   | \n| ID$^3$ | 78.25| 79.23 | 84.05   | 85.28   | 81.70   | 3.48  |\n\nNote that RFW (Racial Faces in the Wild) is a comprehensive diverse benchmark for face recognition. It is notable for its diversity, capturing the facial variations within and across different racial groups, including but not limited to Caucasian, Black, Asian, and Indian individuals. It includes faces from different age groups, genders, and a variety of lighting and pose conditions, ensuring a realistic and challenging test bed for face recognition algorithms.\n\nFrom the table above, we observe that ID$^3$ outperforms IDiffFace by clear margins, which suggests its effectiveness and generalization to different groups of age, pose, ethnics, etc.\n\n**Weakness 2: It would be beneficial to have details on the computational demands and scalability of the model when deployed in practical, real-world scenarios.**\n\n**Response:** Thanks for the suggestion. The time complexity of our proposed Algorithm 3 is $O(mNT)$, where $N$ is the number of identities to be generated, $m$ is the number of images per identity and $T$ is the diffusion steps. The execution runtime of Algorithm 3 (for generating a face dataset that contains 500,000 face images with 10,000 identities) is 17.36 hours. This is a lot more efficient than manual effort which usually takes months to establish a dataset. The scalability of ID$^3$ is linear with $m$ and $N$, which suggests that ID$^3$ is promising in generating larger-scale datasets for synthetic face recognition when deployed in practical, real-world scenarios.\n\n**Question 1: What measures have been taken to ensure the model's robustness against diverse ethnic and age groups, given the model's reliance on identity embeddings?**\n\n**Response:** ID$^3$ models attributes in two ways. For those attributes that can be explicitly disentangled from identity, such as age and pose, ID$^3$ take explicit measures by treating age as one of conditioning signals, which renders ID$^3$ aware of age variations when generating ID-preserving faces accordingly. This makes ID$^3$ robust to diverse age groups. For those attributes that intertwine with identity, such as ethnics, ID$^3$ take implicit measures by automatically and flexibly capturing them through model training. Specifically, recall that ID$^3$ assumes access to the pretrained ArcFace, on the manifold of which dissimilar groups are separated and similar groups are gathered in terms of ethnics. Through end-to-end training, this characteristic acts as supervisory signals that are propagated back to the diffusion model of ID$^3$ (see Figure 1 in the main text). This makes ID$^3$ robust to diverse ethnic groups.\n\n**Question 2: Are there potential improvements or variations in the diffusion model that could further enhance identity preservation without sacrificing diversity?**\n\n**Response:** The proposed model ID$^3$ is inherently a conditional diffusion model conditioned on face attributes. Face attributes may include many aspects. Some can be explicitly disentangled from identity while others are implicitly intertwined with identity. Therefore, ID$^3$ models the former as explicit conditioning signals and models the latter as part of face image features. In this paper, for simplicity, we only show the effect of two major face attributes that are disentangled from identity: age and pose.  More such attributes can be added into our generation framework, such as expression. When more such attributes are incorporated into ID$^3$ explicitly, ID$^3$ becomes aware of these attributes upon which ID$^3$ utilizes the remaining information from face image features for better ID preservation enhancement.\n\n**Question 3: How does the model perform under constrained computational resources, and are there any strategies for optimizing its efficiency?**\n\n**Response:** The proposed model ID$^3$ is inherently a conditional diffusion model (DDPM) that normally costs much computational resources (e.g. ID$^3$ takes 21G GPU memory for inference; iDiffFace takes 24G GPU memory for inference). When deployed under constrained resources, one can use many efficient methods that aim to optimize DDPM inference efficiency such as DDIM [1], PLMSSampler [2] and SD (Spectral Diffusion) [3] while maintaining SFR performance.\n\n\n**References**\n\n[1] Song, Jiaming, Chenlin Meng, and Stefano Ermon. \"Denoising diffusion implicit models.\" arXiv preprint arXiv:2010.02502 (2020).\n\n[2] Liu, Luping, et al. \"Pseudo numerical methods for diffusion models on manifolds.\" arXiv preprint arXiv:2202.09778 (2022).\n\n[3] Yang, Xingyi, et al. \"Diffusion probabilistic model made slim.\" Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1wixPqUuNZ", "reviewer": "Reviewer_ThNX", "summary": "This paper proposes ID3, an identity-preserving-yet-diversified diffusion model for generating synthetic face data for face recognition. ID3 leverages identity embeddings and facial attributes to control inter-class and intra-class diversity of generated faces while preserving intra-class identity consistency, demonstrating state-of-the-art performance on multiple synthetic face recognition benchmarks.", "questions": "The paper addresses a well-motivated and important problem in the field of synthetic face recognition. The proposed ID3 model demonstrates significant innovation in leveraging diffusion models conditioned on identity embeddings and facial attributes to generate diverse yet identity-consistent synthetic faces. Moreover, the theoretical analysis provided in the paper, which proves the equivalence between minimizing the proposed loss function and maximizing a lower bound on an adjusted data likelihood, lends credibility and rigor to the proposed approach.\n\nHowever, there is room for improvement in the presentation and writing of the manuscript. One area that could benefit from further clarification is the explanation of notations and symbols used in the mathematical formulas. For instance, the meaning of the variable d in S\nd−1 is not clearly defined, which may lead to confusion for readers. Additionally, the formatting and typesetting of some equations, such as Equation 3, could be enhanced to improve readability and aesthetic appeal.", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "See questions section in detail.", "weaknesses": "See questions section in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question: One area that could benefit from further clarification is the explanation of notations and symbols used in the mathematical formulas. Additionally, the formatting and typesetting of some equations, such as Equation 3, could be enhanced to improve readability and aesthetic appeal.**\n\n**Response:** We are glad that Reviewer ThNX finds our proposed model innovative. And thanks for pointing out these issues regarding the presentation and writing. The quantity $d$ in $S^{d-1}$ is the dimensionality of the face embedding space. We will clarify them in the manuscript and rewrite some equations (e.g. Equation 3) for better readability and aesthetic appeal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dqrDEbXjU1", "reviewer": "Reviewer_37yP", "summary": "This paper focuses on synthetic face recognition and proposes to concentrate on three aspects: inter-class diversity, intra-class diversity, and intra-class identity preservation. Based on those, an ID-preserving loss is employed to generate diverse but identity-preserving facial images. This paper also demonstrates the proposed loss is equal to lower bound of an adjusted conditional log-likelihood over ID-preserving data.", "questions": "Based on Algorithm 3, the pipeline would be: firstly, generate multiple embeddings close to the anchor; Then use the diffusion model to synthesize the images. The questions are:\n1. Given the unpack(Yi) and generate different attributes, the generated identity image would be affected by the attributes. Do the authors test whether the identities change given the different face attributes on the test? How to make sure the generated identity aligns with the input embedding in the generation phase? \n2. How to make sure the diffusion model can generalize(recognize) this specific input embedding, considering the training embedding only covers a small range of the available embedding(training set)?", "limitations": "This paper shows an attempt to generate diverse facial images of each identity. However, what makes solid FR training is not studied. Furthermore, domain GAP exists when adopting the trained diffusion model for generation, the input embedding might differ from the embedding of the synthesized image. Consequently, the focus is on changing facial attributes but preserving identity is interesting but the overall improvement is marginal, and FFHQ has license issues related to the application of Face Recognition. I think it will be more convincing if the results of CASIA-WebFace under 1M and 1.2 M settings are presented.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This work is well-written and well-organized. It brings some insights for SFR.\n2. The idea of 3 aspects is good, and quite general for SFR\n3. The proposed method shows advances when using the FFHQ dataset", "weaknesses": "Here are several concerns regarding this work:\n1. The idea of Attribute Conditioning Signal is not fit for synthetic face recognition tasks, because factors contributing to solid FR training cannot be determined by simply adjusting face attributes. One reason is that the attribute network (e.g., pose, age) is not generalized enough, as the pre-trained models are obtained from relatively small-scale datasets compared to FR datasets. Additionally, the authors have not addressed which attributes are effective for FR, leaving this important question unanswered.\n\n2. The performance trained on FFHQ dataset appears good; however FFHQ dataset has explicitly banned its use for face recognition applications. Furthermore, FFHQ is relatively small(210k images) which doesn’t contain enough diversity, that’s the reason facial attributes can be of improvement in this experiment. For more details on FFHQ please refer to: https://github.com/NVlabs/ffhq-dataset\n\n3. When it comes to the relatively large dataset CASIA-WebFace, the improvement over DCFace is marginal. One problem is that DCFace is trained with CASIA-WebFace only, not the FFHQ+CASIA mentioned by the author.\n\n4. Experiments are not sufficient. For example, DCFace provides experiment results on 3 data volumes: 500k, 1M and 1.2M. These are not included in this paper.\n\n5. There are some typos, for example, Y_i should be given in line 194", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that Reviewer 37yP appreciates our work for its insight, generality and effectiveness. Here we respond to your questions as follows. Hopefully it will address your concerns.\n\n**Response to Weakness 1:**\n\nThanks for pointing it out. Factors contributing to solid FR training are complicated, and it is impossible to disentangle them all from identity and to quantify them all explicitly and accurately. But it does not suggest that this problem is totally unsolvable. Some attributes can be easily quantified and disentangled from identity such as age and pose, while others intertwine with identity such as gender and micro-features. In this paper, for now we explicitly explore those quantifiable attributes (e.g. age and pose) by injecting them into ID$^3$ as conditioning signals and leave the unquantifiable attributes automatically and flexibly captured by the U-Net of ID$^3$ via model training. We find this practice leads to sufficiently good performance on our SFR task. Our proposed ID$^3$ serves as a framework into which more quantifiable attributes can be introduced, which we leave for future exploration. We will add more discussions to the manuscript.\n\nTo examine whether the attributes we used in our paper are fit for SFR tasks, we perform the following ablation study: as we increase intra-class pose variation, the SFR performances on cross-pose test sets (including CFPFP and CPLFW) are boosted whereas the performances on cross-age test sets (including AgeDB and CALFW) remain almost unchanged. Results and distribution plots are shown in Tab. 1 and Fig. 1 in the **one-page PDF**. From these results, we observe that the distribution of pose angle on FFHQ, Dataset by IDiffFace, Dataset by ID$^3$ PoseOnly 1 and 2 are all unimodal. And their performances are inferior to Dataset by ID$^3$ PoseOnly 3 which exhibits multimodal distribution of pose angle. When comparing the results of ID$^3$ (Tab. 1 in the main text) and the results of ID$^3$ PoseOnly 3, we conclude that age is another essential attribute that should be injected into ID$^3$ as conditioning signals. We also would like to divert your attention to Table 2 in the main paper, which shows that without using attributes as conditioning signals, the performance is inferior to our original ID$^3$. These empirical findings suggests that the attributes as conditioning signals are fit for SFR tasks.\n\nRegarding the generalization of the attribute network, for simplicity and efficiency, the attribute network we used in our paper is a lightweight model, which we find performs efficiently and reasonably well in our task. Using a more powerful attribute predictor would further benefit our SFR task.\n\n**Response to Weakness 2:**\n\nThe use of FFHQ as training data is for the sake of fairness, as SFR community all use FFHQ for model training. To have a fair comparison, we make sure in our paper that all competing models are trained on FFHQ and evaluated on standard benchmarks. Besides, we further use a larger dataset, CASIA, for training, as in Tab. 1.\n\n**Response to Weakness 3:** \n\nAccording to the original paper of DCFace (see the beginning of Section 5), the authors of DCFace claim that \n\n*\"For $G_{id}$ which generates ID images, we adopt the publicly released unconditional DDPM [25] trained on FFHQ [36]. For $G_{mix}$, we train it on CASIA-WebFace [29] after initializing weights from $G_{id}$.\"* \n\nThis suggests the final performance of DCFace depends on FFHQ and CASIA. Furthermore, we successfully replicated its performance reported in their paper using their official implementation: https://github.com/mk-minchul/dcface.git, which suggests DCFace is indeed trained on FFHQ+CASIA. \n\nBack to the argument that \"the improvement over DCFace is marginal\", note that ID$^3$ trained on CASIA only and DCFace is trained on FFHQ+CASIA, and yet ID$^3$ outperforms DCFace. \n\n**Response to Weakness 4:** \n\nThanks for the suggestion. Here we show more results on the larger data volume with the base scale of 1.2M (>=):\n|Models|Add Real Faces To FR Training |CFP-FP | CPLFW | AgeDB | CALFW | average |\n|----------|----------|----------|----------|----------|----------|----------|\n|DCFace|Yes (15% Real Faces)|88.40| 84.22|90.45| 92.38|90.86|\n|ID$^3$|No (0% Real Faces)|89.89|84.98|91.15| 92.02|91.15|\n\n**Response to Question 1, 2:**\n\nYes, we did run the relevant tests, as shown in Fig. A.2 in the supplement. Given different face attributes and an $\\mathbf{Y}_i$, the intra-class cosine similarity ranges from 0.25 to 0.9 whereas the inter-class cosine similarity ranges from -0.25 to 0.25. This suggests that, given $\\mathbf{Y}_i$, an identity do not change as we vary attributes only. For visualization, in Fig. 3 in the main paper, we also demonstrate the invariance of identity when attributes vary.\n\nThe alignment between the generated identity and the input embedding stems from two factors: the manifold manifestation of the pretrained FR model $f_{\\phi}$ and the generalizability of ID-preserving loss. First, the pretrained FR model $f_{\\phi}$ is trained using ArcFace loss, which forces similar face embeddings to have large cosine values on manifold ($S^{d-1}$) and dissimilar face embeddings to have small values. Our proposed Alg. 3 (which solves Eq. (9) using $lb=0.7$ and $ub=0.9$) is based upon this manifold manifestation and ensures that $\\mathbf{y}_i$ is similar to $\\mathbf{w}_i$ with some inherent variations (intra-class diversity). Second, the ID-preserving loss maximizes the inner product between an input embedding and the embedding of the generated face image; we empirically find this loss function generalizes well to the embeddings that are close to (but not equal to) an anchor embedding on the manifold ($S^{d-1}$). We also experiment with $ub=1.0$, but it does not yield better SFR performance. We believe the choice of $lb$ and $ub$ is critical to this generalization of recognizing $\\mathbf{y}_i$. We investigated this effect in Ablation (ii) (see Appendix E).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w1NbBGyF67", "reviewer": "Reviewer_enp6", "summary": "This paper presents a method called $\\text{ID}^3$ for the task of synthetic face recognition. The authors highlight that the accuracy of face recognition using generated data still lags behind that of training directly on real face data. They propose optimizing the generation process from the perspectives of diversity and consistency.", "questions": "- What is the resolution of the training and generated images? \n- How long does the training process take using 8 NVIDIA Tesla V100 GPUs? \n- What is the image generation speed during inference? \n- How much GPU memory is required for inference?\n- How do the ID information and attribute information affect the Unet in the network structure? Is it through cross-attention or along with the timestep information?", "limitations": "Using ID attributes to assist in the generation results is already common in diffusion-based tasks. This method is essentially a conditional guided generation, and its technical contribution is limited.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Clear explanation of formulas and algorithm flow.\n- Achieved SOTA results compared to methods from the past two years.", "weaknesses": "- There has been extensive research on ID preserving, and recent models based on LDM (e.g., Face0, PhotoMaker, FaceStudio, InstantID) can also be used for synthetic face recognition. The paper lacks analysis and comparative experiments on these models.\n- The Face Attribute Conditioning Signal includes age and pose (pose angle range: [-90°, 90°]). However, the visual results in the paper do not reflect these attributes. The variation in pose is minimal, and there is no demonstration of different levels of age (which you mentioned as [0-100]).\n- The paper devotes too much space to mathematical derivations and lacks intuitive visual results. For example, using different attributes and ID information to guide the model could be visualized by showing how the various layers of the Unet perceive this information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that Reviewer enp6 finds our formulas and algorithm flow clear and that ID$^3$ has advantages over existing models over the past two years. Here we respond to your questions as follows. Hopefully it will address your concerns.\n\n**Response to Weakness 1**\n\nThanks for pointing out these interesting works on LDMs. However, we would like to clarify that the primary goal of LDMs differs fundamentally from SFR. These LDMs aim to change styles or expressions given a base face image whereas SFR is to generate a fake face dataset **all from scratch** in place of the real one to achieve face recognition. Yet, we were curious about LDMs' performance on SFR as well before starting this research, although they were not designed for SFR. Among LDMs, we chose the best performing model InstantID and ran multiple tests on it. We empirically found that the best test result is no better than: 81.5 on LFW, 68.9 on CFP-FP, 62.9 on CP-LFW, 62.9 on AgeDB, 63.8 on CA-LFW, with an average of 68.01. These results are far behind the results of the SFR generative model family: SynFace, SFace, ID-Net, DigiFace, iDiffFace, DCFace and ID$^3$. (Please compare these results with those in Table 1.) \n\nWe investigated the reasons: we found that while these LDMs claimed to be ID-preserving in the pixel space, their feature embeddings are not discriminative enough for face recognition, since there was no inductive bias (neither loss functions nor architectures) to achieve face discriminativeness.  Hence, there is no reason for them to work reasonably well in SFR. In our submitted work, we followed the prevailing practice in SFR by benchmarking ID$^3$ against SFR generative models other than these LDMs. Thanks for the suggestion, though. We will add these comparisons and discussions to the manuscript.\n\n**Response to Weakness 2**\n\nThe pose angle range and the age range in Line 134 are for theoretical purpose and are the domains of definition only. The extreme values (e.g. -90 in degree, 90 in degree, 100 in age) have low occurrences even in the training dataset and thus has low prior probability to be captured from ID$^3$. We demonstrate the distribution of age and pose in the training set shown in Fig. 2 in the **one-page PDF**.\n\nFrom the distribution, we observe that the highest age in our training set is 70; the largest pose is 60 in degree and the smallest -60 in degree. Our model can interpolate within these ranges but is less likely to extrapolate outside of these ranges. To generate face images of large pose and high age, one can collect more such data and add them to the training dataset, which increases their occurrences during training.\n\n**Response to Weakness 3**\n\nThanks for the suggestion. For intuitive understanding of our work, we will add more visual results to the manuscript which demonstrate how ID$^3$ works while generating different identities of different poses and ages. Now that the ID embeddings and face attributes affect UNet through the cross-attention mechanism, we visualize the attention map in the UNet given different identities and face attributes, as shown in the **one-page PDF**.\n\nFrom the visualization result, we observe that ID$^3$ first determines the general pose and the outline of a face, and then fill in the facial details and background noise. This provides an interesting insight for understanding how the diffusion model works in ID$^3$.\n\nBesides, we would like to divert your attention to Fig. 2 in the main paper, which gives insights for how our proposed ID-preserving sampling algorithm (Alg. 2) works. We observe that in the middle of Fig. 2, the original score function $\\nabla \\log p(\\mathbf{x}_t | \\mathbf{y}, s)$ generates low-quality face images with cluttered facial details whereas the adjusted score function $\\nabla \\log \\tilde{p}(\\mathbf{x}_t | \\mathbf{y}, s)$ produces high-quality results of various attributes. One can find the correspondence between the adjustment of the vector field and the resulting images.\n\n**Response to Q1:**\nThe resolution of the training and generated images is 64x64.\n\n**Response to Q2:**\nIt takes roughly one week to train ID$^3$ when using 8 NVIDIA Tesla V100 GPUs.\n\n**Response to Q3:**\nThe image generation speed is 0.5 images/second.\n\n**Response to Q4:**\nFor inference, it takes 21G GPU memory.\n\n**Response to Q5:**\nIt is through cross-attention. The ID information and attribute information both acts as keys and values in the cross-attention mechanism while $\\mathbf{z}_t$ acts as queries.\n\n**Response to Limitation:**\n\nYes, there indeed exist many diffusion-based models (LDMs) that use ID attributes to assist in generating images. But again they are designed for image generation but not for SFR. We argue that image generation and SFR are two different tasks with two different goals. Face image generation is to change styles or expressions given a base face image whereas SFR is to generate a fake face dataset **all from scratch** in place of the real one to achieve face recognition. Moreover, these LDMs do not perform reasonably well even when applied to SFR. It is worth noting that our technical contribution goes beyond image generation and identify three important characteristics for successful SFR: intra-class diversity, inter-class diversity and ID preservation. Various technical contributions are proposed in this paper to achieve these three characteristics, respectively. For example: ID-preserving loss, Theorem 3.1 and Algorithm 2 are proposed to achieve ID preservation; anchor generation (solving the Tammes problem) (see Line 191-193) is to achieve inter-class diversity; solving Eq. (9) is to achieve intra-class diversity. \n\nTherefore, our technical contribution consists of not only deriving a conditional diffusion model but also a new ID-preserving sampling algorithm as well as a face dataset generation algorithm that respects face distribution on the face manifold to achieve intra-class diversity, inter-class diversity and ID preservation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their time and effort in reviewing our work and their valuable comments about the paper. Attached is the one-page PDF that contains some figures and results for the rebuttal.

Meta-review: To avoid privacy problem, this paper proposes ID3, an identity-preserving-yet-diversified diffusion model for generating synthetic face data for face recognition. The proposed method leverages identity embeddings and facial attributes to control inter-class and intra-class diversity of generated faces while preserving intra-class identity consistency, demonstrating promising performance on multiple synthetic face recognition benchmarks. Three out of four reviewers are/become positive to this work, while reviewer 37yP is not persuaded to become more positive. The main concern of 37yP lies in the evaluations, especially comparisons with other methods. It is not perfect, but enough to validate the effectiveness of the proposed method. So, a positive decision is reached. The authors should improve the manuscript by considering the review comments in the camera-ready version of the paper.

Ethics review comment: The authors should be mindful of the fact that using the FFHQ dataset for "development of improvement of facial recognition technologies" is discouraged and if possible, the authors should reconsider its usage in the camera-ready version.
Decision: Accept (poster)