CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition
Authors: Zhonglin Sun
Siyang Song
Ioannis Patras
Georgios Tzimiropoulos
Venue: NeurIPS 2024 poster (2024)
Keywords: synthetic face recognition
diffusion models
center-based semi-hard
Primary area: machine_vision
TL;DR: diffusion model to generate semi-hard samples for synthetic face recognition

Abstract: Privacy issue is a main concern in developing face recognition techniques. Although synthetic face images can partially mitigate potential legal risks while maintaining effective face recognition (FR) performance, FR models trained by face images synthesized by existing generative approaches frequently suffer from performance degradation problems due to the insufficient discriminative quality of these synthesized samples. In this paper, we systematically investigate what contributes to solid face recognition model training, and reveal that face images with certain degree of similarities to their identity centers show great effectiveness in the performance of trained FR models. Inspired by this, we propose a novel diffusion-based approach (namely **Ce**nter-based Se**mi**-hard Synthetic Face
Generation (**CemiFace**) which produces facial samples with various levels of similarity to the subject center, thus allowing to generate face datasets containing effective discriminative samples for training face recognition. Experimental results show that with a modest degree of similarity, training on the generated dataset can produce competitive performance compared to previous generation methods. The code will be available at:https://github.com/szlbiubiubiu/CemiFace

CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition
Zhonglin Sun
Siyang Song"
Queen Mary University of London zhonglin.sume sun@qmul......um ac. uk
University of Exeter ss27960cam.. aww.caamacuuuu.
Ioannis Patras
Georgios Tzimiropoulos
Queen Mary University of London i patrassemul..aaa ac. uk
Queen Mary University of London g ttzimiropoulos@@mul.. ac. uk
Abstract
Privacy issue main concern in developing face recognition techniques. Al- though synthetic face images partially mitigate potential legal risks while maintaining effective face recognition (FR) performance, FR models trained by face images synthesized by existing generative approaches frequently suffer from performance degradation problems due to the insufficient discriminative quality of these synthesized samples. In this paper, systematically investigate what contributes to solid face recognition model training, and reveal that face images with certain degree of similarities to their identity centers show great effective- ness in the performance of trained FR models. Inspired by this, we propose a novel diffusion-based approach (namely Center-based Semi-hard Synthetic Face Generation (CemiFace)) which produces facial samples with various levels similarity the subject center, thus allowing to generate face datasets containing effective discriminative samples for training face recognition. Experimental results show that with modest degree of similarity, training on the generated dataset can produce competitive performance compared to previous generation methods. The code will be available at http:://gthuub.oo////bbbbbbbbbbu
Introduction
Face Recognition (FR) has gained significant achievement recent years owing to the combination of discriminative loss function [015], proprietary backbones [+-ll and large-scale face datasets (12H55.
For example, with a 4M training set, existing FR models can achieve over 99% accuracy on various
academic datasets 3 6 [6]. However, in real-world industrial face recognition applications,
collecting large-scale face datasets not always available due to the related licence agreements, ethical issues and privacy policies [I7]
To expand limited training samples real-world scenarios, generative models 18 23
are widely adopted owing to their ability generate high-quality images. However, simply adopting face images produced by those generic generative models to train face recognition models impractical as there is ambiguity about the identities of the produced images because they are derived from random noises, ie.. the identities of these generated face images cannot be obtained without a well-trained FR model [24]. To address such issues, synthetic face dataset generation-based solutions (24128 have been found to gain benefits in developing effective face recognition models. Existing synthetic face recognition (SFR) methods are frequently built upon recent advances of generative models
"Correponniigg Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

such as Style-GAN [25] 27 Diffusion methods [24] 27 and 3DMM rendering [26] For instance, ttyle-transeering diffusion model-based method namely DCface 24 is proposed, which increases the diversity of existing face recognition datasets by generating additional discriminative face images with different styles (e.g. hair, overall lighting, which can be observed in visualization Section 4.3.2) for each subject. However, domain gap issues exists as the model is trained with paired face images belonging to the same identity, while those paired images are not available at the inference stage. only take samples belonging to different identities at the inference stage, which may negatively impact on the images synthesized at the inference stage. Furthermore, the definition discriminative facial images remains unclear in this study.
To address the problems outlined above, firstly we explore the factors resulting in performance degradation for SFR and reveal that previous approaches fail to consider the properties of ef- fective FR
Inquiry Image b
Generated Samples
Sim
1.0
relationshipssimilrrry be- tween samples. Consequently, considering the facts: (a) semi-hard negative samples are cru- cial to train effective face recognition model for Triplet loss [29]; (b) samples close to the decision boundary contribute most to the train- ing gradient Bl (c) all face images belonging to the same subject can be represented by hypersphere in the latent feature space (30] (i.e., can be measured by existing FR models, e.g., AdaFace (3), whose distances (radius) to the identity center are negatively correlated to their similarities to the center. We hypothesize that face recognition performance is sensitive to the data with different levels of similarity the identity center in the hypersphere, and experimentally reveal that the optimal perfor- mance is obtained with samples of mid-level similarity, which we term center-based semi- hard samples. Inspired by this crucial finding, we propose novel diffusion-basee synthetic face recognition approach (Cemilaace) which generates center-based semi-hard face samples by regulating the similarity between the generated image and the inquiry image, through similarity controlling factor condition. Figure prresentss the overall hypothesis by showcasing samples with various similarities to the identity center. Comprehensive experiments effectiveness of our proposed CemiFace. Our method achieves promising performance in synthetic face recognition (SFR). The main contributions and novelties of this work summarized follows:
0.66
0.33
Subject:
Subject:
Figure 1: Visualization of the samples with different similarities. Given an inquiry image, can form a hypersphere based similarity to the inquiry im- age, where samples with the same similarity share the same radius. Samples with similarities between O to I with an interval of 0.33 are shown. With our proposed CemiFace, each inquiry image finally forms novel subject.
conducted illustrate the
We provide the first comprehensive analysis to illustrate how FR model performance is affected by different levels of similarity of samples, particularly center-based semi-hard samples.
We propose a novel diffusion-basse model CemiFace that can generate face images with various levels of similarity to the identity center, which can be further applied to generate infinite center-based semi-hard face images for SFR.
We demonstrate method can be extended to supervision for training which is an advantage over the previous method [24].
much the data without label
Experiments show that our CemiFace surpasses other SFR methods with a large margin, reducing the GAP-to-Real error by half.
2 Related Works and Preliminary
Synthetic Face Generation for FR: With the emergency of generative models, synthesizing fa-
cial data for various facial tasks has become a critical issue, such as applications in Face Anti-
sproofing [31] and Face Recognition [25 26 24
28
32
33 SynFace (25) aims to mix the real

---

images with the DiscoFaceGAN facial images to construct the dataset. DCFace 24 takes diffusion models to adapt style from the style bank to the identity image and result in discriminative samples with diverse styles. IDiff- Face (28) proposes the condition latent diffusion models 23 to the feature embedding and images are synthesized by pretrained decoder.
[9 samples. DigiFace 26 uses 3DMM for rendering
are generative models which denoise an image from a random noise image. The training pipeline for diffusion models consists of forward process wherein noise progressively added a given image and denoising process to predict the estimated noise for effective denoising. A single forward process is formulated as Markov Gaussian diffusion with timestep
Wrelimnnary-DDMM. Diffusion models 21 22
((x:|x(-1) = N (x:: v 111-Ba B,x 8x1)
Where NO is adding noise function. When accumulating the time step over 0 T, the final process is given as follows:
q (x)) = II ((xx(Xx-1)
(2)
Then the denoising process is conducted to predict the noise for the time step using a model ore ( typically UNet (34)) the training loss is:
IMSE -E, E II|oo( ( Vo:Xo + vI āze, () - -
where 3e the pre-set forward process variances. Then notation 2g given as: ai I[ 0₂ and
a2 = 1 - St. e is a random Gaussian noise image c ~ (0)
3 The proposed approach
In Section 3.1 we first investigate the relationship between sample similarity and their effectiveness in training FR models, presenting the finding that samples with certain similarities (i.e., center-based semiihan hard samples) to their identity centers are more effective for training FR models on a real dataset and subsequently devise toy experiment to validate it. Inspired by findings, in Section 3.2 we propose a novel CemiFace, a conditional diffusion model that produces images with various inquiry image. Specifically, Section 3.2.1 introduces how construct
levels of similarity the similarity condition which is fed to diffusion model to guide the generation, and discusses the Lat to require the generated sample to exhibit certain similarity degree the inquiry image. In Section 3.2.2 we then present how to use our diffusion model to generate a synthetic face dataset given a fixed similarity condition m and set of inquiry images.
3.1 The Relationship between Samples Similarity and Performance Degradation
Performance Degradation for Synthetic Face Recognition: Face recognition models trained on face images synthesised by existing generative models (c.g., style-transfering 24 3DMM rendering (26 and latent diffusion expansion (28)) frequently suffer from performance degradation (2) For example, with the same data volume, the model trained on the sate--of-the-an synthetic dataset DCface (24) produces 11.23% lower verification performance on CFP-FP testset than the model with the same architecture trained on the real dataset. A key reason for this issue is that these generative models only intuitively explore the properties of discriminative samples, but fail to consider the
similarity levels among synthesized face images. However, previous studies (29 2 3 empirically reveal that training effective FR models intrinsically relies on semi-hard negative samples in Triplet
Loss (291 or samples close to the decision boundary 1 35).
Hypothesis and Findings: Since face images belonging to the same identity/class can be aggregated within hypersphere [30], where the location of each face image is decided by its similarity to the identity center (the center of the hypersphere) (illustrated in Fig. []. We treat all face images of each subject N-1 (N=512 in AdaFace [3) dimensional sphere with its center representing the subject-level identity center. Then, the spheres of all subjects can be combined in an N-dimensional sphere, where each subject-level sphere is cluster.
Based on this, we hypothesize that samples of mid-level similarities to the identities center play a dominant impact on the FR performance, they exhibit discriminative style variations (c.g. age,
3

---

Sim
LFW
CFP-FP
AgeDB
CALFW
CPLFW
AVG
0.85
98.43
85.67
89.43
91.08
82.78
89.48
0.81
98.91
88.8
91,03
91.71
84,58
91.01
0.76
98.94
90.92
91.5
91.7
85.85
91.78
0.70
98.66
91.08
90.32
90.76
86.92
91.55
0.53
94.63
82.12
77.63
80.11
77.3
82.36
0.81 0.76
0.70
Figure 2: Samples with different similarity
Table 1: Accuracy of groups with different simi-
groups from CASIA-WebFace dataset. From left to right are samples with lower similarity to the identity center
larities. Sim means the average similarity to the identity center. AVG is the average accuracy on the 5 evaluation datasets
pose). To validate the hypothesis, we conduct the first comprehensive investigation for the impact of different levels of similarity to the identity center on the FR performance. We first split face images in the CASIA-WebFace (36) into various levels of groups according their similarities to their corresponding subject-level identity centers. Here, the identity center of each subject is obtained by the weight of the linear classification layer, trained using AdaFace [3. To avoid the impacts caused by different numbers of training samples, we assign around 100k face images to each group representing close similarities to their identity centers. This results in 5 distinct similarity groups. Table I reports the performance of model trained on each group and test on five standard face recognition evaluation datasets [37.4T] Table in Supplementary material Section [B.4 displays the similarity range of each group. We further validate the style variation in Visualization Sec 4.3.2] We also visualize randomly selected samples of each group in Figure 2 Results reveal that groups with middle-level similarities (0.76 and 0.70) produced similar but nop-performing average accuracy. This indicates that face images of a certain low similarity to their identity centers (which we refer to as center-based semi-hard samples) are essential for learning highly accurate face recognition models. In contrast, the group whose images have the lowest similarity (i.e., 0.53) to their identity centers obtained the worst performance, which suggests that it is difficult to train effective face recognition model with the most challenging samples (i.e., the samples are normally hard to be distinguished by human observation).
3.2
Center-based Semi-hard Face Image Generator
Inspired by the above findings, this section proposes a novel conditional diffusion model, namely CemiFace, for synthesising effective center-based semi-hard face images given the inquiry image (identity center) x and pre-defined similarity controlling factor m, based on which a new discrimi- native synthetic dataset is obtained to train effective face recognition models.
Methodology overview: As illustrated in the left side of Fig 3 the training process starts with adding a noise e ~ (0, 1) with timestep i to the clean input image x, resulting in xę. Mcanwhile, similarity conditions m and identity condition Eaa are fed to the diffusion model by cross- attention as illustrated in the lower right part of Fig. 3 Consequently, the diffusion Unet outputs the estimated noise é' = ap(2,, t, m. Eça (z)) for denoising the image as clean estimated image Co. Based on the obtained estimated image So. original x and condition Cate, the whole model is optimized by the combination LMSE and Loam defined the following Section 3.2. as well as the details of constructing condition.
At the inference stage (the upper right part of Fig. 3). random noise x; = =T = (0, 1) and the time step t = T are first fed to a CD block. This results in an estimated noise e' = ae (x,, t, Cate). Then, a denoise step is adopted to generate xt-1 from x, for efficient interface speed. This process repeatedly conducted on the obtained denoised latent images (zz-1, 22- .Fn) until = O, where To treated as the final generated face image. Here, we assign the same identity label as x to all face images generated from the inquiry image x. To ensure high inter-class variation, our inquiry images are filtered by pretrained FR ( IR-101 trained on the WebFace4M [12 dataset by AdaFace.), which enforces the similarity between each pair of query images is lower than 0.3. The number of identities fully decided by the number of inquiry face images. The pseudo-code for training and generation are given in Supplementary Material Section A.3
4

---

Training
Inference
M
T
T-1
t=0
m
CD
CD
CD
TT
TT-
T
Eia
ED
m
ID
F
Catt
F
L SimMat
Fii (fi
Bia
It
CA
Condition Diffusion (CD)
CD
Condition
Add
Eid (3))
Facial Embedding
Propangata
Step
Condition
Similarity
ID
Danaise
F
Projection
CA
Cross Attontion
Loss
Figure 3: Illustration of proposed method. The left part is the training framework for learning images with various levels of similarity. Firstly noise is added to the clean facial image before is processed by the diffusion model. Then similarity controlling condition ranging between [-1,1] with facial embedding is injected to guide the generation. Consequently, the model outputs the estimated noise, which adopted calculate the estimated image. We add similarity matching loss LsimMat between the estimated image and the input image. For generation, we gradually denoise noising image with time step scaling from T conditions for identity and similarity are left fixed. The two diffusion models in the generation part mean the same diffusion model at two different time steps. The right bottom part the details of using cross-attention inject similarity condition and facial embedding into the diffusion models
3.2.1 Training CemiFace
To facilitate our ceffusson-bbaed CemiFace can generate diverse center-based semi-hard face images, we propose novel diffusion model training strategy. During training, random Gaussian noise image e ~ N (o, 1) is firstly added to clean face image x at the time step f, before feeding it to the diffusion model to generate the noise face image Xt:
Xt vā + vI v
Then, conditions are constructed based on the similarity controlling factor m, the identity condition Cia and time step t condition. Subsequently, the diffusion model outputs the estimated noise (x
t. t, Eça, m) for denoising the image.
Constructing Similarity Controlling Condition: To address the purposes of generating images different scales of similarities, two conditions injected into the diffusion process guide the generation process. The first one is the identity condition Cja aiming to anchor the center of the generated facial images which can be formulated as:
Cja Exa
(5)
where Eia is pre-trained face recognition model (e.g.. IResnet-50 pretrained from AdaFace Cia represents the feature embedding the given image x. Then the most important part similarity controlling condition Csim which maps the scalar similarity m into feature embedding. This condition serves to regulate the similarity to the inquiry image, facilitating the generation of images spanning from the most challenging samples (m=-1) the most similar ones (m=l).
3
Csim F (m)
(6)
Where F.O is the linear projection layer. Then following DCFace [24 the two conditions combined and projected as cross-attention conditions for sending to the DDPM process. AdaGN 42 adopted embed time step condition cat() is the concatenation operation.
Catt F2(cat( Cia, Cetm))
5

---

The Catt is further processed by cross coss-attentionn operation with the intermediate latent representation of diffusion UNet oy learned from the input noisy image as:
KA, = SoftMar QW,(KK,K K.)Wa) WWIVV Vel
(8)
Vd
where Catt is treated as the key Kg and value Ve (same DCFace) to influence the generated face images. Q = K V are the query. key and value, representing the latent feature of UNet dp.
To ensure the similarity between the generated face Xo and the corresponding inquiry image (identity center) adheres the specified similarity factor m as given in the following equation:
Training Loss:
m sim(Eia (x), (xx)x)))
where sim() denotes a similarity measurement (e.g., can be computed by Cosine Similarity or Fuclidean Distance). Following DDPM [4 21]. an approximated clean sample xo can be traced from x, at the time step through the following formula:
Xo = 8o = (x: v1 - v1/) oid)/Vár
(10)
This gives a hint that the generated face image Xo can be controlled at the training phase by regularizing the estimated &o, which allows the gradient be back-propagated to the diffusion model, e.g., controlling facial attributes [43] and styles (24). Inspired by this, we propose novel similarity Matching loss LsimMat aimed at disentangling the generated face image Xo to exhibit certain similarity to the inquiry image, which is determined by the similarity controlling factor m. We employ the Time-step Dependent loss (24) with different time step Eq specifically firstly an identity loss for recovering the identity of the original inquiry image x, which will be applied to produce original facial embedding when the time step t - O:
Trec
||1 (aiaFia( (x), Esa (xt))ll2
(11)
Then, we require the estimated Xn to produce an feature embedding Exa( which matches the original x with m similarity as:
T.gim= Im sim(FFa (xo))l2
(12)
Consequently, the overall identity regularization loss at the time step i can be formulated as:
LsimMat (1 Prec 4e
(13)
where " is the scaling weight for adjusting the similarity of the generated So- At the time step t=0, the model outputs an image with the same identity as the original image x. When t scales from the maximum time step T, the generated face image gradually shifts far away from the x. When approaching T. the model will output the image with m similarity to the original image. The proposed LsimMat loss is inspired by the fact that facial images, with diverse styles but the same degree of similarity, are located at a circle of the hypersphere. This loss can regularize the model to learn this kind of pattern. Specifically, the similarity guaranteed by our proposed loss, and the diversity is facilitated by the random noise of the diffusion models, which is validated in the Visualization Sec. 4.3.2 The overall training object is:
L = LMSE + XLsimMat
(14)
where À is a hyperparameter for balance the training focuses on noise estimation or identity-related similarity regularization.
3.2.2 Face Image Generation with Appropriate Similarity
Given a random noise e and conditions(iee. identity, similarity and time step), the well-trained model progressively denoises the noisy image with varying time step (from the maximum T to 0) and a fixed similarity factor condition m to generate a clean image with a specified similarity m to the given inquiry image x, adopt DDIM [22] for efficient interface speed.
We experimentally investigate the appropriate generation similarity m for synthetic face recognition. Specifically, we first adopt fixed similarity factors to test the best similarity. We also explore mixing the similarity around the appropriate fixed m (mixing hard m) and mixing appropriate fixed m samples with easy samples (mixing easy m).

---

Experiment
4.1 Implementation Details
Evaluation Metries: We examine the 1:1 verification accuracy trained on the dataset generated
by our CemiFace on various famous testsets including LFW (37], CFP-FP [8] AgeDB-30 [3)]. CPLFW [4)], CALFW [4] and their average verification accuracy AVG. Gap-to-Real is the gap to the results trained on CeelaeWebFace with CosFace loss.
Details of CemiFace Training and Generation: The condition m is appropriately adjusted during the training phase to facilitate better generalization across various similarities. Considering the overall cosine similarity ranges from -1 to I, the model is enabled discern differences in generated images under varying similarity controlling conditions when training. Specifically, in the bini-batch, we assign a randomly selected from -1 with an interval of 0.02, allowing the model to generate corresponding images at different similarity scales. The synthetic face recognition datasets are generated in 3 volumes. Specifically in 0.5M data volume, we generate 50 images per subject and total of 10k subjects; As for 1.0M, we keep 50 images per subject but with 20k subjects; For 1 1.2M, we add 5 images per subject with 40k subjects to the 1. 1.0M settings. Oversampling method as used in DCFace is adopted which adds 5 repeated inquiry images to each subject. For more details including model, ablation studies and discussions please refer to Supplementary material A. and
Details of Training the Synthetic Dataset As the training code of DCFace (24) and DigiFace [26] for training the SFR is not re- leased. We opt for CosFace [ with some regularizations to match the performance of DCFace (24] Specifically, the margin of Cosface is 0.4, weight decay is 5e-4, learn- ing rate is le-1 and decayed by 10 the
All Curves
Curve for AVG
96
85
Acoursy
26th and 34th epoch, totally the model is
trained for 40 epochs. We add random re-
size& & crop with the scale of [0.9, 1.0), Ran- dom Erasing with the scale of 10.0 and random flip. Brightness, contrast, saturation and hue are all set to be 0.1. The backbone opted for is IR-SE50
L-W
70
ApeD8-32 30
80
GALFW
-0.5
0.0 Inquity Santarty
1.0
-1.0
-0.5
3.0
Inquiry Simbarity
4.2 Ablation Studies
Figure 4: Accuracy of samples with different similarity varying from -1 to 1. The left figure is the specific performance on each evaluation dataset. The right figure is the average accuracy of our CemiFace
4.2.1 Impact of Similarity m
Appropriate m for Generation: Herein we ablate how a scalar m influences the genera-
tion in terms of training performance. We adopt different m ranging from -1 to with the interval of 0.1 to generate face groups of 10k identities with 10 samples per identity to match the data volume in the finding for CASIA- -ebbace in Sec. [.T]] Figure 4 illustrates the accuracy curves when using those data for training face recognition (for detailed numerical results please refer to Supplementary Material B.4.2). Similarity m=0 provides the best recognition performance 89.567 in terms of the AVG, then m=0.1 has the AVG of 89.368, with m=-0.1 obtains 87.708. It can be concluded the appro- priate degree of similarity for generating discriminative samples is around 0 to 0.1, which is different from the similarity of -ebbaee where the best recognition performance is obtained with the similarity of 0.7, This may be because the model for comparing similarity on LesSaeeeeFaae is pretrained on this dataset.
Generation with Mixing m: We conduct the experiment of including mixed m when generating the dataset, as is shown in the top part of Tab Tab and Tab[]] Specifically, we opt for mixing the generation m from -0.1 1 to 0. 1, and from 0 to 0. We use training m varying from [0, and the generation interval is 0.02. The results show that mixing m with 0 to 0.1 when generating the data will bring worse performance compared to single m=0. However, mixing m with -0.1 1 to 0.1 obtains a similar performance compared to m=0. Additionally, progressively mixing with easy and semi-hare
7

---

samples are provided in Tab 3 as observed, with more easy samples included in the training dataset, the FR performance reduced more prominent. We keep the generation m be 0 for later discussion.
Experiment
Training m
Generation m
Interval
AVG
0
0.02
91.64
Generation
[0.0.1] [-0.1, 0.11
0.02
91.11
Training m
AVG
Mixing Generation
[0.1]
0.9
91.61
[0.1]
0.02
91.64
91 64
Mixing Training m
[-1,1]
0
0
0.02
92.28
91.15
[O.1]
90.36
90.12
[-1,1]
0.04
92.30
[-1.1]
0.06
92.09
89.57
Table 2: Abaltion studies for mixing m in training and generation stage. The generation m is mixed with close
Table 3: Abaltion studies for mixing m in generation stage with easy and semi- hard samples
similarity of semi-hard samples.
Training with Various m: The choice of various levels m during the training stage are ablated in
the bottom part of T where settings considered when training CemiFace:(a) single with
similarity of 0; (b) multiple discrete m ranging from 0 to with 50 steps; (c) similar to (b) but with a range of [-1,1] ] and interval 0.04. Then we synthesize the data with m=0. As observed, setting (c) yields the best performance, indicating that with a broad range of similarity across -1 to 1, covering all the available probabilities, the CemiFace model can generalize well when adapted for generating highly discriminative samples. We also include experiments of changing the interval for (c) setting
from 0.02 to 0.06 the bottom of Tab 2] the result suggests that our approach is robust to the discrete interval but sensitive to the range of training m. We do not consider continuous similarity as the trained model collapses to generate the same image when given different similarities m.
Method
Training Data
Inquiry Data
AVG
Dataset
AVG
1-shot Web
91.64
CASIA
DDPM
91 .949
-0.1
91.27
1-shot Flickr
88.97
WebFace
91.64
CemiFace
90.89
1-shot Web
90.25
Flickr
DDPM
90.19
-0.1
89.96
1-shot Flickr
88.65
DigiFace
0
90.67
1-shot Web
92.20
0.1
90.38
VGGFace2
DDPM
92.01
1 I-shot Flickr
90.586
-0.1
91.36
DDPM
91.47
DCFace
CASIA
1-shot Web
89.8
0.1
90.96
DDPM
90.18
Table 5: Accuracy of the optimal m on dif- ferent inquiry sets.
Table 4: Impact of Training and Inquiry Data. We also include results of training on DCFace for comparison
4.2.2 Ablation Study for Training and Inquiry Data
Impact of Training Data:
Since our method does not require paired images for training the diffusion model, the limitation of using unlabelled data is alleviated. Consequently, we conduct experiments to the impact of different training data. Specifically,
employ datasets for training:(a) CASIA- WebFace as used in DCFace; (b) A challenging in-the-wild dataset Flickr with 1.2M images collected by us from Flickr website; (c) VGGFace2 [13] which is large-scale dataset containing 3.3M clean images. Training m is set to vary within the range of [-1,1] while generation m is kept as O. We do not consider data from FFHQ [88] due to restrictions on being applied for face recognition.
We can see from Table using VGGFace2 as the training set produces the best performance when training model on it, indicating that training on large-scale dataset will bring more advance in generating discriminative dataset. However, to conduct a fair comparison with previous methods, we adopt CASIA-WebFace for the following studies. Additionally, although Flickr contains much more challenging conditions such as blurred, cartoon, and occluded faces, results in similar performance compared to DCFace [24). which proves the effectiveness of our proposed CemiFace.
Impact of the Inquiry Data: The choice of appropriate inquiry image x which can referred to as an initial point, is essential because regard the generated group from the given x to be an
8

---

independent identity group. DCFace employs a pre pre-trained DDPM model [21] trained on FFHQ to generate synthetic facial images. The style bank sampled from a real-world dataset, e.g. CASIA- WebFace [36]. Their process involves combination of synthetic facial data and real dataset. In contrast to DCFace, our method has fewer constraints when referring to the source data. The source data can be either synthetic or real, and we ablate the impact of using synthetic data and real data.
For taking synthetic data as the inquiry samples, use the samples from DCface to conduct fair comparison, noted as DDPM. As for adopting real-data, we consider two options: (a) 1-shot data randomly sampled from WebFace-4m [2] which provides a clean dataset. (b) 1-shot Flickr, a challenging dataset filtered from the one collected in Sec. 4.2.2 with fewer licence restriction. If inquiry images with high similarity, they result in overlapped groups of synthetic images in hypersphere space. Therefore, we follow DCface to filter out samples with a similarity higher than 0.3. We ablate the choice of the inquiry data source Table observing from changing the inquiry data, using 1-shot data of WebFace4M performs slightly better for our CemiFace. However, applying 1-shot WebFace4M to DCFace leads to a performance drop, as there are constraints for DCFace training and generation, e.g. frontal face and glasses. Then using the challenging 1-shot Flickr as inquiry data brings worse results. This indicates that clean and real inquiry images are beneficial to generate discriminative datasets. Additionally, appropriate for each inquiry dataset with 0.5M
volume is also around O which can be observed in Tab
Method
Duta Volume
LFW
CFP-FP
AgeDB
CALFW
CPLFW
AVG
GtR
CASIA-WebFuce WebFuce (AdaFace)
99.42
96.56
94.08 94.35
93,32
89.73
94.62
0.49M
CASIA WebFace (CosFace)f
99.3
94.87
93.15
89.65
94.26
SynFace
91.93
75.03
61.63
74.73
70.43
74.75
19.51
DagiFace
95.4
87.40
76.97
78.62
78.87
83.45
10.81
IDiff-Face
0.5M
98.00
85.47
86.43
90.65
80.45
88.20
6.06
DCFace
98.55
85.33
89.70
91.60
82.62
89.56
4.70
DCFacet
98.33
87.7
90.01
91.61
83.26
90.18
4.08
CemiFace, ours
99.03
91.06
91.33
92.42
87.65
92.30
1.96
DCFace
98.83
88.40
90.45
92.38
84.22
90.86
3.40
DCFacet
1.0M
98.88
89.71
91.25
92.15
85.2
91. 44
2.82
CemiFace, ours
99.18
92.75
91.97
93.01
88.42
93.07
1.19
DigiFace
96.17
89.81
81.10
82.55
82.23
86.37
7.89
DCFace
98.58
88.61
90.97
92.82
85.07
91.21
3.05
1.2M
DCFacet
99.05
89.8
91.73
92.7
86.05
91.87
2.39
CemiFace, ours
99.22
92.84
92.13
93.03
88.86
93.22
1.04
Table 6: Comparison with the previous methods, AVG is the average accuracy the evaluation datasets. GtR is the results compared to CASIA-WebFace with CosFace. Methods with f are the results reproduced by our settings
4.3
Comparison with the State-of-Art methods
4.3.1
Quantitative Results:
We compare our Cemilaaee with the previous methods demonstrate its effectiveness. The models compared are SynFace [25). DigiFace (26). IDiff-Face (28 and DCFace (24) in both 0.5M, 1M and 1.2M image volumes. The loss for training the synthetic dataset is CosFace. For the CemiFace training set, we choose -ebbFaee to have a fair comparison with DCFace, training m ranges from -1 with 50 discrete steps, and generation m The results are available in Table 6 In 0.5 M protocol, our method exceeds the previous statc-of-art method DCface in terms of all the evaluation datasets where achieve significant improvement on pose-sensitive dataset CFP-FP and CPLFW by 3.36 and 4.39 respectively. And in the average protocol, we get 92.30 while DCFace is 90.18. Our method still cannot exceed the model trained on the real dataset MAASSA-WeFFaee but we reduce the GAP-to-Redd error from 4.08 to 1.96 ( 4.08-1.96 = 51.90 11.96% relative error) compared 4.08
to DCFace. When it refers to the 1.0M and 1.2M settings, a similar phenomenon can be observed, our method surpasses DCFace on all protocols which reduces the Gap-to- Real by half, i.e. 1.59 and 1. 1.36. In general, CemiFace behaves well on all verification accuracy and improves pose-related performance by large margin.

---

4.3.2 Qualitative Results
We visualize the generated results to compare with DCFace in Figure Specifically, samples with different m scaling between [-1,1] with interval 0.2 are presented. For each row, we opt for the same noise to illustrate the variations across different similarities. We observe that when m is set to 1, the identity of the generated sample is very close to the inquiry image. When m is 0.4, gender and age change can be observed from the last two rows. With m scaling far away from the inquiry image, pose changes can be noticed for the first rows. Another interesting phenomenon appears when similarity is -1.0 where the generated samples change significantly. Additionally, when the noise changes, the generated images exhibit different styles, aligning with our hypothesis in Sec. 3.2.1 Finally, with m=0, the group looks extremely different to the inquiry image, but can deliver highly accurate face recognition performance.
best
Inquiry
-0.4
DCFace
3
3
6
4
6
6
6
4
6
6
6
Figure 5: Sample Visualization under different similarity. From left to right are inquiry images, images with m from -1 and samples generated by DCFace. Different rows in each inquiry group represent the results produced by different noises. The first column are the inquiry images. The yellow dashed box includes samples where we obtain the best accuracy. Pink dashed boxes samples that vary vastly.
5
Conclusion
This paper proposes novel method generate diseriminative dataset for training effective face recognition models with reduced privacy concerns. We investigate the factors contributing to the effective face recognition model training and re-formulate the challenge of generating discriminative samples as synthesizing center-based semi-hard samples. A similarity controlling factor condition is adopted for generating semi-hard samples. Models trained on the generated dataset with center- based semi-hard samples produce accurate face recognition performance over the previous methods. A notable advantage Cemilace is its independence from a labelled dataset for training. However, the limitations of CemiFace include relying the pretrained identity network's performance for conducting similarity comparisons, being sensitive to the quality of the inquiry image and privacy issues arising as the pretrained model derives from a dataset without user consent.
This work was in part funded by UK Research and Innovation (UKRI) under the UK government's Horizon Europe funding guarantee [grant number 10093336) and funded by the European Union [under EC Horizon Europe grant agreement number 101135800 (RAIDO)).
Acknowledgments:
10

---

References
[1] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and
Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5265-5274, 2018.
[2] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4690-4699, 2019.
[3] Minchul Kim, Anil Jain, and Xiaoming Liu. Adaface: Quality adaptive margin for face recognition. In Proceedings of the IEEE/CVF conference computer vision and puttern recognition, pages 18750-18759, 2022.
[4] Yandong Wen, Weiyang Liu, Adrian Weller, Bhiksha Raj. and Rita Singh. SphereFace2: Binary classification is all you need for deep face recognition. arXiv preprint arXiv: 100i::2008.05113, 2021.
[5] Fadi Boutros, Naser Damer. Florian Kirchbuchner, and Arjan Kuijper. Elasticface: Elastic margin loss for deep face recognition. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1578 1587, 2022.
[6] Zhonglin Sun and Georgios Tzimiropoulos. Part-based face recognition with vision transformers. In 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK. November 21-24, mpi inf .pg. de/ddee/111 pdf
2022. BMVA Press, 2022. URL
[7] Weidi Xie, Li Shen, and Andrew Zisserman. Comparator networks. In Proceedings of the European conference on computer vision (ECCV), pages 782-797, 2018.
[8] Pengyu Li. BioNet: biologically-inspired network for face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10344-10354, 2023.
[9] Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Fan-face: a simple orthogonal improve- ment to deep face recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12621-12628, 2020.
[10] Qiangchang Wang, Tianyi Wu, He Zheng, and Guodong Guo. Hierarchical pyramid diverse attention networks for face recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8326-8335, 2020.
[11] Zhonglin Sun, Chen Feng, Ioannis Patras, and Georgios Tzimiropoulos. Lafs: Landmark- facial self-supervised learning for face recognition. In Proceedings of the IEEESCVF Conference on Computer Vision and Pattern Recognition, pages 1639-1649, 2024.
[12] Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Jiwen Lu, Dalong Du, et al. Webface260m: benchmark unveiling the power of million-scale deep face recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10492-10502, 2021.
[13] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In FG, 2018
[14] Dong Yi, Zhen Lei, Shengeai Liao, and Stan Z Li. Learning face representation from scratch. 2014.
arXiv preprint arXiv:
[15] Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb- Im: A dataset and benchmark for large-scale face recognition. In European conference computer vision, pages 87-102. Springer, 2016.
[16] Jiankang Deng, Jia Guo, Jing Yang, Alexandros Lattas, and Stefanos Zafeiriou. Variational prototype learning for deep face recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 11966-19915, 2021.
[17] Protection Regulation. Regulation (eu) 2016/679 of the european parliament and of the council. Regulation (eu), 679:2016, 2016.

---

[18] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 1401-4410, 2019.
[19] Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3d imitatve-contrrstive learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1164,163, 2020.
[20] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 12873-12888 2021.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-668 (1668006551,,, 2020.
[22] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
[23] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[24] Minchul Kim, Feng Liu, Anil Jain, and Xiaoming Liu. Deface: Synthetic face generation with dual condition diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12715-12725, 2023.
[25] Haíbo Qiu, Baosheng Yu, Dihong Gong, Zhifeng Li, Wei Liu, and Dacheng Tao. Synface: Face recognition with synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10880-10890, 2021.
[26] Gwanghin Bae, Martin de La Gorce, Tadas Baltrušaitis, Charlie Hewitt, Dong Chen, Julien Valentin, Roberto Cipolla, and Jingjing Shen. Digiface- million digital face images for face recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3526-3535, 2023.
[27] Pietro Melzi, Christian Rathgeb, Ruben Tolosana, Ruben Vera-Rodriguez, Dominik Lawatsch, Florian Domin, and Maxim Schaubert. Gandiffface: Controllable generation of synthetic datasets for face recognition with realistic variations. arXiv preprint urXiv:2305.19966 2023,
[28] Fadi Boutros, Jonas Henry Grebe, Arjan Kuijper, and Naser Damer. Idiff-face: Synthetic- based face recognition through fizzy -dentity-onnditione diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19650-19661, 2023.
[29] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 81 815-823, 2015.
[30] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 212-220, 2017.
[31] Yaojie Liu, Joel Stehouwer, and Xiaoming Liu. On disentangling spoof trace for generic face anti-spoofing. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK August 23-28 2020, Proceedings, Part XVIII 16, pages 406-422. Springer, 2020.
[32] Fadi Boutros, Marco Huber, Patrick Siebke, Tim Rieber, and Naser Damer. Sface: Privacy- friendly and accurate face recognition using synthetic data. In 2022 IEEE International Joint Conference on Biometrics (IJB), pages 1-11. IEEF, 2022.
[33] Yujun Shen, Ping Luo, Junjie Yan, Xiaogang Wang, and Xiaoou Tang. Faceid-gan: Learning symmetry three-player gan for ddentity-presevve face synthesis. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 821-830 2018.
12

---

[34] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net:
Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assister Intervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241, Springer, 2015.
[35] Qiang Meng, Shichao Zhao, Zhida Huang, and Feng Zhou. Magface: A universal representation for face recognition and quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14225-14234, 2021.
[36] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12))2 2947779777777 2017.
[37] Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned- -iiller.. Labeled faces in the wild: A database for studying face recognition unconstrained environments. In Workshop on faces in RRal--Life Images: detection, alignment, and recognition, 2008.
[38]
Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David Jacobs. Frontal to profile face verification in the wild. WACV, 2016.
[39] Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first mamually collected, in-the-wild age database. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 51-59,200 2017.
[40] Tianyue Zheng and Weihong Deng. Cross-pose Ifw: A database for studying cross-poss pose face recognition in unconstrained environments. Beijing University of Posts and Telecommunications, Tech. Rep, 5(7), 2018.
[41] Tianyue Zheng, Weihong Deng, and Jiani Hu. Cross-age Ifw: A database for studying cross-age face recognition in unconstrained environments. arXiv preprint 170.:v:708.08//11 2017.
[42] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakom. Diffusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10619-10629. 2022.
[43] Bohan Zeng, Xuhui Liu, Sicheng Gao, Boyu Liu, Hong Li, Jianzhuang Liu, and Baochang Zhang. Face animation with attribute guided diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 628-637, 2023.
[44] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[45] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference machine learning, pages 8162-8177 8171. PMLR, 2021.
[46] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[47] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:2 14266696-2777, 2021.
[48] Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. arXiv preprint arXiv:2306. 13092, 2023.
[49] Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprini arXiv:
2018
[50] Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. Advances in Neural Information Processing Systems, 35: 13877-13891, 2022.
[51] Manuel Kansy, Anton Raël, Graziana Mignone, Jacek Naruniec, Christopher Schroers, Markus Gross, and Romann M Weber. Controllable inversion of black-box face recognition models via diffusion. In Proceedings of the IEFF//VV International Conference on Computer Vision, pages 3167-3177, 2023.
13

---

This is the supplementary material for the paper CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition.
A
Addition to: Implementation details
A.1 Diffusion Details
We follow most of the settings of DCFace 24 Specifically, the model is trained on CASIA-
WebFace [14] with 10 epochs. The maximum time step T for diffusion training is 1000. Then for generating the synthetic face recognition dataset, the time step for DDIM [2] is 20. The optimizer opted for is Adam 44 The batch size is 160 on A100 GPUs. CemiFace training takes 8 hours, the generation also takes hours. As a comparison, DCFace takes 10 hours for Training and hours for Generation. Both DCFace and our method need around 6-7 hours to conduct FR training.
As for the diffusion UNet, we remove the identity feature in Residual Block, for more details of the Diffusion UNet please refer to DCFace 24
A.2 High Inter-class Variations and High Intra-class Variations
(1) High Inter-class Variations: Each inquiry face image selected to highly independent on other inquiry images. Specifically, we follow DCFace to use pre-trained FR model to keep samples with a threshold of lower 0.3.
(2) High Intra-class Variations: high intra-class variations are ensured by (a) changing the similarity condition m, as small input similarity m results in the generated semi-hard images belonging to the same identity having long distances to the identity center; and (b) the face images of the same identity generated by CemiFace distributed in all directions from the identity center, which can be observed from supplementary material T-SNE Fig- 7 This is guaranteed by randomly sampled Gaussian noises input the diffusion model, which exhibits a large variation. properties would ensure the generated face images of the same identity are almost evenly distributed in sphere that has a relatively large radius, and thus they would have high intra-class variations.
a result, both
A.3 Pseudo-code
The pseudo-code is provided below.
Algorithm The training pipeline of our CemiFace
1: Initialization: Original Training Set Do, pretrained FR network Eia, Diffusion Unet de, Maxi-
mum time step T, Maximum iteration т, iteration +- 0, similarity m [-1,1]
2: repeat
3: 7. n
4: Randomly sample a batch of facial images To from (alao treated as inquiry data d), noise
images from normal distribution, similarity condition from range [-1,1], single time step t
5: construct ID & similarity condition Catt using Eq.
6: add noise _, + use Eq. | given &g&t
7: output estimated noise c' = 0g(x,, t. Catt)
8: Update 0gn+1 f- dga Voz Eq. 14
9: until converges or n
Output: output model (7
A.4 Dataset statistics
We have also calculated the number of face images belonging to different similarity groups for CemiFace and DCFace in the Tab indicating that our CemiFaaee tends to generate images showing lower similarities to their identity centers (i.e. all samples are semi-hard), while DCFace containing more easy samples.
14

---

Algorithm 2 The pipeline of CemiFace-based face dataset generation
1: Initialization: Inquiry Data Dr, pre trained Diffusion Unet dg, Maximum time step T, fixed similarity m, Maximum Number of samples in each identity K
2: n = O is the identity index, = is the sample index
3: repeat
4: T2 nl k = 0
5: Sample batch of inquiry data construct the ID similarity condition Cate using Eq.
6: repeat
7: k 1,,+,
8:
Generate noise image from normal distribution N(0, I)
9:
repeat
10:
output estimated noise e' = og (x L, Catt.)
denoise the image using following DDIM 22 x1-1 + denoise( (Zey e')
12:
13:
until t=0
14:
assign zo the same label ya of the inquiry data, [zo, ya!
15:
until k=k
16:
until n Ien((;)
Output: output the generated dataset
Number of identites
Method
avg sim
std
0-0.1
0.1-0.2
0.2-0.3
0.3-0.4
0.4-0.5
above 0.5
DCFace
36.24
9.14
14
231
2059
5899
1788
CemiFace
28.54
7.76
196
1043
3281
4539
930
Table 7: The statistics of the average similarity of each group. avg sim and std is the average/std similarity to the inquiry images the whole dataset. 0-0.1 means the number of identities has a similarity of 1-0.1. CemiFace is distributed farther away from the inquiry center with less variation than DCFace.
B Further Experiments
B.1 Impact of Identity Center and Random Center
The performance of CemiFace is highly affected by the characteristics of the inquiry samples. Herein we examine how the model behaves when subjected numerical identity conditions. Two kinds of centers are cansidered:(a) identity centers derived from the CASIA-WebFace dataset, and (b) random centers with a similarity range of [-0.1, 0.2] to (a). By observing from the Table & with random center the model results in invalid results; On the other hand, when utilizing identity centers, the model performs optimally when the similarity controlling condition m to 0 which aligns our previous finding. However, it is noteworthy that with identity center the performance is worse than dataset inquired by 1-shot WebFace, exhibiting similar results to DCFace.
Inquiry source
sim
AgeDB
Random Center
1.0
Not converge
0.7
97.22
75.03
86.90
89.93
74.47
84.71
0.5
97.50
78.96
87.12
90.38
77.62
86.32
Identity Center
0.2
98.17
86.29
89.07
91.40
83.03
89.59
98.25
87.30
89,98
91.35
83.23
90.02
0.0
98.23
87.49
89.53
91.47
83.73
90.09
I-shot DigiFace
0.0
98.28
90.04
89.68
91.23
84.12
90:67
I-shot WebFFace
0.0
99.03
91.33
92.42
87.65
92.30
DCFace
98.33
87.7
90.01
91.61
83.26
90.18
Table 8: Comparison of different inquiry centers. The results of DCFacce by setting copied for reference.
15

---

To provide deeper insights into this phenomenon, we visualize the samples generated by different inquiry centers in Figure Notably, with m=1 the random center produces images with different identities which can simply be concluded by human observation. Conversely with the identity center, given a similarity of 1.0, the generated samples appear highly similar, except for the samples circled in red. Further investigation reveals that the number of images in that subject comprises approximately 16 images while the left subject provides approximately 50 images. Intuitively. A model trained on this dataset will focus more the subjects with a large number of images which explains the suboptimal results obtained by identity center.
Random
Class
m=1
m=0
m=D
Real
m-1
m-1
m=o
m=O
Figure 6: Comparison of different inquiry center. From top to bottom images inquired by Random Center, CASIA Identity Center and 1-shot Real images. For Identity Center and 1-shot Real images, images similarity of 1 and O are shown. Different columns represent given different noise. Two examples are shown for each case. The inquiry images in the identity center are selected from the dataset. The red circles contain samples that look extreme different from the inquiry center.
We further visualize the T-SNE of the feature embedding in Figure 7 As shown the upper figure, with higher similarity, the samples tend to cluster in the central region. Subsequently, by inspecting the bottom figure, becomes apparent that with a similarity of 1.0, each subject is located a different specific area. Consequently, a similarity of -1.0 results in each image being positioned close to other subjects in the middle area.
B.2 Addition the Inquiry Data: Image Quality
The above discussion validates how CemiFace is affected by different centers in the aspect of numerical results. For a better understanding of the negative impact brought by challenge inquiry data such as 1-shot Flickr, we visualize the images generated from different image quality in Figure & Specifically, we present inquiry images subjected to blur, occlusion, extreme pose, painted and clear conditions, with similarity controlling condition set to O. By comparing the last block with the rest of the blocks, one can conclude that extreme image quality fails to generate clean images. In conclusion, unblurred, non-occluded, appropriately posed, and real-world data essential for model to generate a highly clean synthetic face recognition dataset.
B.3 Further Ablation Studies
B.3.1 Impact of Different Pretrained loss
As DCFace hasn't released its AdaFace-based SFR training code and details, we were not able to reproduce it for our model training. Thus, in Tab o compare ours with DCFace by adopting the same pre-trained AdaFace model to train our diffusion generator, and then employing the same CosFace loss for both ours and DCFace's SFR models training. Results show that our CemiFace still outperformed the SOTA DCFace. Additionally, we provide results achieved by using pre-trained model trained by CosFace. Specifically, we apply a model pre-trained by CosFace to train both our generator, and employ the same CosFace loss for their SFR models' training. The experiment shows that the model pretrained from CosFace performs better than that of AdaFace.
16

---

3XTA
26
Zahctaam1
Betroim2.
Aettrii 1.0
arddnnceree
-10
-15
-20
-20
TSMEL
DATA
1ohvt
1ahat-iimb.
Isher 1.0
75473
50
-10
Figure 7: T-SNE visualization. The bottom figure is the T-SNE generated by 1-shot data with similarity of 1.0, 0.0 and -1.0 respectively. The upper figure is different inquiry centers with two similarities 1.0 and -1.0, the random center is also given. Red circles samples worth noticing, with their order being green, red, and grey, positioned from center to outside
Method
Pretrained FR
SFR loss
AVG
CASIA-WebFaae
AdaFace
94.62
CASIA-WehFace
CosFace
94.26
CemiFace
AdaFace
CosFace
92.30
CemiFace
CosFace
CosFace
92.60
B.4 Upper/Lower Bound of Different Similarity Group in -ebbace dataset
The range of each similarity group in the Section 4.2. is given in the following Table 9
B.4.1 Impact of Different Training Backbone
the IResnet-SE-5i modified by ArcFace 2 as the default backbone. Additionally, we provide the results achieved by (Resnet-18(R18)) (Resnet-SE-50(R50) and IResneetSSE-1O00RR 100) in table 10 for reference.
Following previous WDrkseDCFace (24). DigiFace 266 SynFace (25
B.4.2 Numercial Results for Different
Here we provide the numerical results for the impact different similarity levels in Tab m provide the best performance.
17

---

Quality
Inquiry Img
lmns, neo
Blur
Occlusion
Extreme
Pose
Painted
Clear
Figure 8: Examples of samples under challenging conditions, including Blur, Occlusion, Extreme Pose, and Painted conditions, are presented. Samples generated by clear images are appended for better comparison.
Avg Sim
average largest sim
average lowest sim
AVG
0.85
0.887
0.831
89.48
0.81
0.831
0.794
91.01
0.76
0.794
0.747
91.78
0.70
0.747
0.676
91.55
0.53
0.767
0.277
82.36
Table 9: Average largest sim represents the mean value of the largest similarity values appeared in every identity; and Average lowest sim represents the mean value of the lowest similarity values appeared in every identity
B.4.3 FID Image Quality
We Fréchet Inception Distance(FID) which measures the distribution similarity of the given two datasets. Specifically, in T2b FID is reported by comparing randomly selected 10k samples with randomly selected CASIA. Need to note that our method doesn't intend to generate images similar to the distribution of CASIA- CASIA-WebFace but to construct discriminative dataset that is conducive to providing highly accurate FR performance
B.4.4 Euclidean Distance
As shown in Tab 13 using Euclidean distance leads to worse performance than cosine similarity, which might be due the FR training loss (CosPace [I]) being carried on cosine similarity.
B.4.5
Impact of X
present the results using different the left part of the Tab 14 Performance is sensitive to and large À results in performance degradation.
18

---

Backbone
R18
R50
R100
AVG
90.75
91.64
91.82
Table 10: Impact of different training backbone
Sim
LFW
CFP-FP
AgeDB-30 D0
CALPW
CPLFW
AVG
73.86
0.9
97.38
73.81
96.88
90.13
74.82
$4.644
0.5
85.75
62.42
67.8
81.85
58.41
71.25
0.6
97.52
87.
75.39
0.5
97.85
80.55
87.03
90.9
79.35
87. 316
0.4
97.88
39
$5.01
79.55
57.344
0.2
98.02
58.6
88.77
0.1
98.2
86.2 29
88.25
91.25
$2.85
89. 368
0
98.1
86.6
88.9
91.15
83.08
89.567
-0.1
97.65
84.9
86.42
89.47
80.1
$7.708
-0.2
93.15
S0..3
81.33
85.92
74.68
83 13.182
-0.3
65.17
41.
85.18
65.16
63.42
09:55
63.68
09:404
40.6
84.25
66.63
03.05
69.13
02.86
68.78
62.53
-088
4099
94.23
62.38
65:13
73,85
00.08
09:134
-1
85.75
62.42
67.8
81.85
$8.43
71,25
Table 11: Numercial results for the impact of different similarities
C Privacy Concerns
In this section, we are going to discuss the privacy issues that lie in developing synthetic face generation for face recognition. The primary aim synthetic face recognition is to mitigate concerns associated with privacy. Large-scale face recognition data are usually collected from web scrappers by searching name lists (usually celebrities), without obtaining user consent. Consequently, some of the large-scale datasets [13] [5 are abandoned by their collector to avoid Legal Risk. In addition,
IDiff-Face (28) mentions European Union (EU) has come up with the General Data Protection Regulation (GDPR) 17 to regulate the application of facial data, making it harder to use face
recognition data.
We notice that DCFace 24 incorporates labelled dataset for training style transferring solution, and when they generate the new dataset, they use samples provided by DDPM [2]] trained on FFHQ (18). However, a noteworthy concern arises as the FFHQ dataset, whose derivative model is used as pretrained model in DCFace for sample generation, explicitly bans its application in face recognition. Consequently, we are not sure whether the model and synthetic face images based on FFHQ are allowed to be used. We try to avoid privacy concerns from the aspect of collecting Flickr which contains diverse licenses with reduced privacy problems. Another potential solution to avoid privacy concerns to use samples like Digiface (26] which is rendered by 3DMM. However, DigiFace is only allowed to be adopted for non-commercial applications, but one can render images from 3DMM following the DigiFace pipeline for commercial purposes. We append the result inquired by 1-shot DigiFace in the bottom part of Table for reference and example images generated by I-shot DigiFace are shown in Figure Results reveal that I-shot DigiFace still can not surpass 1-shot WebFace but still behave better than DCFace. Finally, although 1-shot Digiface samples sometimes don't appear to be like real humans, the generated samples exhibit similar patterns to real-world images from human observation.
Our method CemiFace offers the advantage of not requiring labels during the training phase compared to DCFace. Nonetheless, both our method and DCFace adopt pre-trained face recognition model which may counter legal issues. we hope further researchers bring steps forward to avoid using this kind of pre-trained face recognition model to alleviate legal concerns in this domain.
19

---

Method
Ours
DCFace
24
DigiFace (66
FID
18.72
15.82
65.39
Table 12: Fid score to the real dataset
Base
Euclidean
Interval 0.06
91.64
90.95
91.43
Table 13: Difference between Euclidean and larger similarity interval
D Discussion
D.I
Why Semi-hard samples work
We assume the benefits of the semi-hard training face images could be attributed
easy training samples are typically images where the face is clear, well-lit, and faces the camera directly, and thus training on such easy samples would not allow the trained FR models be able to generalize for face images with large posc/age/eepreesionn variations and different lighting conditions/backgrondd that are frequently happened in real-world applications. AdaFace 3 also mentioned that easy samples could be beneficial to early-stage training, while hard sample mining is needed for achieving generalized and effective FR models;
Hard samples normally contain noise data. Specifically, FaceNet [29] demonstrates that the hardest sample mining using large batch size leads to hard convergence and produces inferior performance. This is because training with very hard samples may not allow FR models to learn effective features but focus cues apart from facial identities;
Semi-hard samples generated by CemiFace mostly contain large posed faces but fewer face-unrelated noises. We also evaluate the training epochs needed to reach the highest AVG performance for easy samples (m = 0. (.,), semi-hard samples = 0) and extreme hard samples (m -..5). Easy samples take 10 epochs reach the best AVG and 20 epochs to produce the training loss of O; Semi-hare samples take much longer (38 epochs) to provide the highest AVG while the final training loss around 3; and FR models training on extreme hard samples could not converge.
Inquiry lmg
Generated Imgs,
Figure visualization of samples inquired by 1-shot DigiFace. Different rows are results inquired by different images. Different columns are randomly selected generated samples.
20

---

0.01
0.05(default)
0.1
0.5
AVG 91.72
91.64
91.29
90.77
Table 14: Impact of different
The actual similarity to the inquiry center indicates that our CemiFace tends to generate images showing lower similarities their identity centers (i.e. all samples are semi-hard), while DCFace contains more easy samples.
D.2 Different difftusion Loss
As there are some other variation diffusion losses such as Improved-DDPM [45 which has been applied in Diffusion Transformer DIT) [6). Variational Diffusion Models (VDM) We follow
the previous SOTA SFR studies (DCFace 24 and IDifiFace (28] choose the same generic MSE
diffusion loss [21 22 our base model, ensuring the reproducibility of our approach and its fair
comparison with DCFace [4 and IDiffFace 28
D.3 Difference between Dataset Distillation
Dataset distillation methods [48 [50] are widely adopted to create a dataset that can produce high performance when training model on it. SRe2L [48] is recent state-of-the-art method for dataset distillation which trains the noise image through pretrained backbone. Their main process contains forward process get the classification label of the trainable noise inquiry image and train the noise inquiry image to produce a specific class prediction with BN alignment. The distinctions between our method with theirs are:
Embedding Classification Layer: We aim explore the feature embedding of the backbone, not the classification layer.
Consideration of Image Similarity: Our method explores the similarity of the given inquiry image, which is not considered in recent dataset distillation methods.
Their approach focuses on distilling data from existing classes, while our CemiFace distils patterns from the pretrained face recognition model. This learned pattern can be applied to unseen subjects, we utilize independent data that was not part of the pretrained model's training dataset.
Pattern Distillation:
Extra Model: We incorporate diffusion model introduce parameters for producing high-quality images.
D.4 Relationship to ID3PM
Recent work, i.e. ID3PM 51 proposes invert the Black-Box model of face recognition generate a similar image to the inquiry image. However, our method differs from theirs in several aspects:
Purpose: Their objective to invert the black-box model without full access, whereas we aim to generate discriminative dataset.
Image Similarity: They require the generated image to like the original image, while our goal is ensure the generated images encompass diverse styles.
Evaluation Approach: They evaluate by replacing the data of the evaluation dataset, whereas our approach involves training a model on the generated dataset.
Theoretical Degradation:
When is set 1, our model theoretically degrades to their
model.
Diffusion Model Structures:
We use different diffusion model structures to conduct experi- ments, specifically employing cross-attention and AdaGN 42 for inserting conditions.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Contributions of this paper are included in the abstract and introduction made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitation the conclusion section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
We only have experimental assumptions and they are proved in the main paper
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We will release the code and data upon acceptance. And we provide detailed information for reproducing.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer:
[No]
Justification:
We will release the code and data upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
wwt.Suubmisiooooiiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All experimental details are provided
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our experimental results show that the proposed method exceeds previous works by large margin. And we have run the experiments multiple times to confirm the effectiveness of the proposed method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Computational cost is included in the Supplementary Material
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:////aaaaa...cpppa/aataaaaaa
Answer: [Yes]
Justification:
We have made sure the anonymity
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the privacy issues brought by Face Recognition.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer:[Yest.
Justification: We avoid using data that is banned from being applied to Face recogmition and discussed the privacy issues.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer:[Yes]
Justification: Our models and code used in the paper are licensed
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No assets
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer:[NA]
Justification:
We do not collect data, but generate synthetic data
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not collect data, but generate synthetic data
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "1XheAy5c8A", "reviewer": "Reviewer_tciQ", "summary": "The paper proposes a new Face Recognition diffusion-based generation method. The diffusion process is completed with a semi-hard constraint on the synthetic reconstructed image: for each inquiry image of the (real) training set, the reconstructed image after the forward-backward diffusion process must have a specific cosine similarity with the inquiry image.\nAs it is usual for such methods in Face Recognition, the resulting synthetic dataset is then used for training a Face Recognition model. This model is evaluated across diverse real datasets.", "questions": "1) Could you explain the last sentence of Section 4.2.1 (lines 260-261) ?\n\n2) In Section 4.2.2, why is the range of training m equal to [0,1] while the previous subsection concludes with an optimal range [-1,1] ?", "limitations": "1) [major] The SimMat loss seems to be an interesting idea to lead towards m-similarity to the inquiry image, during training. But the derivation of the MSE loss (Eq. 3) assumes that the reconstructed image should be the inquiry image, and not a new image having a m-similarity with the inquiry image. I may be wrong here but I think that the diffusion loss of Eq. 3 is mathematically valid if the forward diffusion process is symetric to the reverse diffusion process, which is not the case here.\n\n2) [major] In Section 4.2.1, there should be a part studying the difference between the required m and the estimated m post-training. That means that, for any required m, it is easy to compute the estimated m, i.e. the  similarity between the resulting reconstructed image and the inquiry image. This estimated m must be quite different than the required m because Figure 4 shows that the best m is m=0, meaning that there is a 90 degrees angle between the inquiry image and the synthetized image. If m was truly equal to 0, the performance of the model would be very poor. So, there must have a difference between the required m and the real estimated m (post-training).\nThis is also reflected in the conclusion saying that the best setting for m is to train CemiFace with m randomly sampled from [-1,1]. A similarity truly equal to -1 would bring a model with an astonishingly poor performance.\n\n** Update **\nI have increased my score from 4 to 5.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The tackled problem is quite hard and needed at the same time. Current SOTA Face Recognition generation methods lead to a significant gap in terms of performance, compared to real Face Recognition datasets (of the same size).\nThe idea of controlling the similarity to design semi-hard samples is also interesting.", "weaknesses": "1) In Fig. 1, is the displayed similarity really the cosine similarity ? In the generated samples, the line with perfect similarity (equal to 1) seems to provide synthetic images which would not have a perfect similarity with the inquiry images displayed above the hypersphere.  \n\n2) [minor] In Eq. 3, the probability distribution of epsilon is not specified.\n\n3) The authors should cite explicitly the works that use the training loss (Eq. 2) in this precise form, as there are alternative loss functions for diffusion models. A discussion on the reasons of this particular choice of diffusion loss might be a plus (e.g. in the appendix).\n\n4) [minor] Although the lines 114-116 are accurate, they are misleading the reader. The widely known representation of Face Recognition embeddings is that they lie onto a hypersphere of dimension N, where each embedding is a point of the hypersphere. Those embeddings are clustered by identity on this sphere and the identity centers are roughly at the center of those clusters. The hypersphere mentionned in this paper is a hypersphere of dimension N-1, where the identity center is at the center of the sphere. \n\n5) In lines 120-125, the authors should detail the range of similarities to the identity center, for each of the 5 splits of the CASIA training set. Only the average similarity of each split is specified.\n\n6) [minor] Figure 3 should be a bit more explained than just its caption.\n\n7) [major] Lines 155-162 are not well written and it is hard to understand how the margin m is used to guide the diffusion process. In particular, F_1 and F_2 are not defined, while some unused F is mentionned. C_sim seems to be a vector of unknown size. Also, the temporal guidance is too briefly described.\n\n8) [minor] Some hyperparameters' values (alpha_t/beta_t, lambda) are not specified.\n\n9) The right part of Fig. 4 displays two curves that do not have the same meaning for the x-axis. For AVG, the similarity is a constrained similarity (m) for training CemiFace (i.e. a similarity between a real inquiry image and a synthetic image). For CASIA, it is the similarity between one real image and its identity center (not a real image). To sum up, for AVG it is a similarity between 2 images, while for CASIA it is between 1 image and its identity center. Thus, comparing the two curves does not seem meaningful.\n\n10) [major] The CosFace loss is used to train on synthetic datasets, while AdaFace is used to produce (identity-oriented) embeddings for the CemiFace training set generation. There should be only one model for both tasks, for fair comparisons. On Table 6, training on CASIA with AdaFace gives better results than with CosFace, so one could attribute the good performance of CemiFace to the fact that the authors used a stronger model (AdaFace) to generate the synthetic dataset than the model used to train on this dataset (CosFace). In addition, there should be a part studying the impact of this AdaFace choice (i.e. another loss), at least in the appendix.\n\n11) The ROC curve on IJB-B/IJB-C for all synthetic methods of Table 6 would be a plus, as the accuracy is easily saturated, and not really used in industrial use-cases. Previous papers (related works) provide such ROC plots.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "---\n**W1:** The displayed similarities are the input cosine similarities, based on which the displayed face images were generated. However, the actual similarities between the generated images and their inquiry images may not be exactly the same as the input cosine similarities as DL models typically cannot generate perfect/exact outputs. The actual cosine similarities between generated and inquiry images are provided in **the PDF** Tab. 2, measured by the IR-50 network pretrained using AdaFace loss (line 154).\n\n---\n**W2:** The $\\epsilon$ is a random Gaussian noise image $\\\\epsilon\\sim N(0,1)$ fed to our diffusion model (Line 145). We will explain this in the revision. \n\n\n**W3:** We assume the mentioned training loss is Eq. 3 rather than Eq. 2. We will modify and cite as: we follow the previous SOTA SFR studies (DCFace[23] and IDiffFace[27]) to choose the same generic diffusion loss [20,21,22], ensuring the reproducibility of our approach and its fair comparison with DCFace and IDiffFace.  \n\n**W4:** The theories of yours and ours are the same but described in different forms. We treat all face images of each subject as an N-1 dimensional sphere with its center representing the subject-level identity center. Then, the spheres of all subjects can be combined in an N-dimensional sphere, where each subject-level sphere is a cluster. We will follow your suggestion to rephrase these sentences\n\n**W5:** The ranges for each group are listed in the Tab. 1 of **the PDF**. Here, the splitting boundary for each group varies across different identities.\n\n**W6:** Please refer to the general response G1.\n\n**W7:** $F$ should be $F_{i}$ for representing the linear projection operation, and thus $F_1$ and $F_2$ are two stacked linear layers. Here, $F_1$ projects the input similarity m to a latent feature $C_\\text{sim}\\in \\mathbb{R}^{512} $, and then $F_2$ projects the feature concatenating $C_\\text{sim}$ and identity embedding to a condition vector $C_\\text{att} \\in \\mathbb{R}^{128} = F_{2}(cat(E_{id},F_{1}(m)))$. Subsequently, the $C_\\text{att}$ is fed to our diffusion to control its face image generation via a cross-attention (CA) as:\n\\begin{equation}\n    CA(Q,K,V,K_{c},V_{c})=SoftMax(\\frac{QW_{q}([K,K_{c}]W_{k})^{T}}{\\sqrt{d}})W_{v}[V,V_{c}]\n\\end{equation}\nwhere ${\\mathbf{C_{att}}}$ is treated as the key $K_{c}$ and value $V_{c}$ (same as DCFace) to influence the latent representation extracted from the input noisy image (treated as the query, key and value $Q=K=V$) for generating the final face image. Please also refer to the general response G1 for details.\n\n**W8:** $\\beta_{t}$ starts from 0.0001 to 0.02 controlled by time step $t$-based linear schedule, while $\\alpha_{t}=1-\\beta_{t}$ (line 101)\n\n**W9:** Sorry for the confusion. We put two curves in the same figure due to the limited space. We will separate the two curves into two smaller figures.\n\n**W10:** As DCFace hasn't released its AdaFace-based SFR training code and details, we were not able to reproduce it for our model training. Thus, lines 216-220 and lines 300-311 fairly compare ours with DCFace by adopting the same pre-trained AdaFace model to train our diffusion generator, and then employing the same CosFace loss for both ours and DCFace's SFR models training. Results show that our CemiFace still outperformed the SOTA DCFace. Based on your suggestion, Tab. 4 in **the PDF** additionally provides results achieved by using unified CosFace. Specifically, we apply a model pre-trained by CosFace to train both our generator and DCFace generator, and employ the same CosFace loss for their SFR models' training. Due to limited rebuttal time, we only include the results for the 0.5M setting, but will present results for all data volumes in the revision.\n\n**W11:** We provide ROC curves for CASIA, DCFace and CemiFace in 3 data volumes in **the PDF** Fig. 2. The FR model trained on our CemiFace generated dataset with the best curve (largest area), while the real face dataset provided inferior results than these SFR methods (CemiFace and DCFace). When FAR=1e-3,  the TAR result achieved by our self-implementation on real-dataset is around 90, which exceeds previous works (ArcFace gives around 60 in [a]]). Importantly, the ROC of the IJB-B and IJB-C are typically not provided in neither the related SFR studies [23,24,25,27] nor many of the discriminative FR methods (e.g., CosFace[1], Face Transformer[b], boundaryface[c]) when using CASIA-WebFace for training.\n\n[a]Federated Learning for Face Recognition with Gradient Correction\n\n[b]Face Transformer for Recognition\n\n[c]BoundaryFace: A mining framework with noise label self-correction for Face Recognition\n\n**Q1:** We rephrase it as: We did not provide the AVG result achieved for setting the similarity interval (Table 2) close to zero (continuous similarities), as this training setting leads our model to generate same images when inputting different $m$ values. We assume that an extremely small similarity interval prevents the model from effectively learning the differences between each level of similarity\n\n**Q2:** It is a typo and should be [-1,1]\n\n\n**L1:** Eq.3 does not directly compare the input and reconstructed images. Instead, it compares the input noise and the estimated noise (Line 98). The overall loss Eq. 13 adopts $\\lambda$ to balance between the diffusion focuses on comparing input and estimated noises for generating clean images (Eq.3 $L_{MSE}$) and the similarity between the generated samples and inquiry image (Eq. 12). Given the relationship between the estimated $\\hat{x} _{t-1}$ and real $x _{t-1}$ as follows, it is symmetric and valid if the $\\epsilon'$ is accurately estimated.\n\n\\begin{equation}\n\\hat{x} _{t-1} \\approx \\frac{x _t - \\sqrt{1 - \\alpha _t} \\epsilon'}{\\sqrt{\\alpha _t}} = x _{t-1}\n\\end{equation}\n\n**L2:** Please refer to W1, where $m=0$ produced images of an actual average similarity of 0.2854, making the generated dataset semi-hard. We will update this table in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qRCLWt4fby", "reviewer": "Reviewer_fJPL", "summary": "The paper titled \"CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition\" addresses a critical issue in face recognition (FR) related to privacy and performance degradation when using synthetic face images. The authors propose a diffusion-based approach, CemiFace, which generates facial samples with varying levels of similarity to an identity center. This method aims to enhance the discriminative quality of synthetic samples, thereby improving the performance of FR models trained on these datasets.", "questions": "While comparing the proposed work with other SOTA methods - how did you generate the results of the SOTA method?", "limitations": "Limitations were mentioned only in the last few sentences of the conclusion section but not otherwise stated separately.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Introducing Similarity controlling factor in synthetic face generation using a diffusion based approach.", "weaknesses": "(a) Due to the introduction  of this similarity control conditioning in the diffusion process there must be a  change in total sampling time ( certainly it will also depend on the number of  time steps considered in the diffusion process also)  – An illustration/analysis  on computational complexity of the proposed algorithm is needed.\n\n(b)  Seems like the overall process is dependent on how(using which method) the value of m was determined during the diffusion process! \n\n(c) A complete pseudo-code on the proposed method would have helped the reader to understand the whole process. \n\n(d) Figure 3 could have been much more elaborated   and in more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: We first define some basic calculation complexities:\n\n**Time Step $ T $**: This represents the total number of time steps required for a complete diffusion process.\n\n**UNet Complexity  $C_{\\text{UNet}}$**: The UNet model accepts the input image and outputs the estimated noise.\n\n**Pretrained Face Recognition Model $ C_{\\text{ID}} $ and Similarity Condition $C_{m}$** : These are processed by two sequential linear layers.\n\n**The backpropagation and weight updating time complexity** for the UNet is given as $B$. Since the pretrained face recognition model is fixed, there is no computational complexity associated with it during backpropagation.\n\n**Forward Diffusion Process:**\n\nThe forward diffusion process consists solely of adding noise to the clean image over $T$ time steps, resulting in a calculation complexity of $O(T)$.\n\n**Denoising Process:**\n\nThe denoising process also consists of $T$ time steps to gradually denoise a noisy image. This process involves a UNet and conditions to output the estimated noise, resulting in a calculation complexity of $O(T \\cdot (C _{\\text{UNet}} + C _{\\text{ID}} + C _{m})) $.\n\n**Training Step:**\n\nA training step includes a forward diffusion step, a denoising step, and model training (including loss calculation and backpropagation). Note that the model training only involves one time step. The overall complexity of training a CemiFace model by one step is given by:\n\n$ O(N \\cdot (C_{\\text{UNet}} + C_{\\text{ID}} + C_{m} + B)) $\n\nwhere $N$ is the number of samples, note the loss calculation involves the similarity comparison between the estimated image $\\hat{x_{0}}$ and the original image $x_{0}$, thus the pretrained FR is adopted 3 times in each training iteration. For a standard diffusion model, the complexity is:\n\n$O(N \\cdot (C_{\\text{UNet}} + B))$\n\n**Accumulated Training Iterations:**\n\nWhen accumulating all the training iterations $I $, the calculation complexity becomes:\n$O(I \\cdot N \\cdot (C_{\\text{UNet}} + C_{\\text{ID}} + C_{m} + B)) $\n\n**Generating One Sample:**\n\nTo generate one sample, the process involves denoising random noise from time step $T$ to 0. Since the forward diffusion process and loss calculation are no longer needed, the complexity is:\n$O(T \\cdot (C_{\\text{UNet}} + C_{\\text{ID}} + C_{m})) $\n\n**Generating a Dataset:**\n\nFor generating a dataset with $ L $ identities and $ S$ samples for each identity, the complete calculation complexity is:\n$O(L \\cdot S \\cdot T \\cdot (C_{\\text{UNet}} + C_{\\text{ID}} + C_{m}))$\n\nOn a single A100 GPU, (1) generating a sample with 50 time steps takes 0.9 seconds; (2) One CemiFace training step (one inquriy image) takes 0.6 seconds; (3) Training for one epoch takes 45 minutes; (4) It takes 10 epochs to converge with a total time of 7.5 hours. (5) It takes 16 hours to generate a dataset of 10,000 identities, with each having 50 images. We have mentioned the computational cost in supplementary material Sec. A.1, lines 473-480.\n\n---\n**W2**: The overall process not only depends on m but also relies on the input inquiry image. The conditions $C_{att}$ sent to diffusion models are determined by $m$ and the inquiry data using a linear projection $F_{2}$. The selection of **inquiry images** is dependent on the rule that the inquiry images should be unblurred, non-occluded, appropriately posed and independent of each other (discussed in Sec. 4.2.2, supplementary material B.1 and B.2). It is also required by SFR to ensure high inter-class variation. Then the **generation $m$** directly determines the final similarity property of the samples inside each identity group, we find the optimal $m$ is scalar number 0 and mixing the generation $m$ would bring worse performance(lines 232-243).\n\n---\n**W3**: The pseudo-code is given in **the PDF**, Algorithm 1 and Algorithm 2\n\n---\n**W4**: The revised figure is appended in **the PDF** Fig. 1, while its description is provided in the general response G1, which will be added to the main text.\n\n---\n\n**Q1**: In Tab. 6, the results of all competitors except DCFace$\\dagger$ are obtained from their original publications. Here, we additionally train a SFR model based on the **released synthetic face images/dataset** generated by the SOTA DCFace$\\dagger$ model and the CosFace loss, to facilitate a fair comparison with ours. This is because DCFace hasn't released AdaFace-based SFR training code and details, and thus we were not able to reproduce it for our SFR model training. For this SFR training, as described in lines 215-229, we use IR-SE-50 as backbone and CosFace loss for learning facial embedding, where the hyperparameters are set the same as the standard CosFace.\n\n---\n**L1**: Although most papers only discussed their limitations in their conclusion section due to page limitation, our submission has detailed limitations not only in the conclusion but also: (1) Sec. 4.2.2 (lines 276-295),  Sec. B.1 and B.2 of the supplementary material of the main submission (i.e., high-quality inquiry data is essential); (3) Sec. C of the supplementary material (i.e., privacy issues); and (4) Sec. D.1 of the supplementary material (i.e., the training of our CemiFace depends on the pre-trained FR model, and thus its performance also relies on the performance of this model).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jKp0cRqWHV", "reviewer": "Reviewer_L3p9", "summary": "The paper proposes a novel approach named C to address privacy concerns in face recognition technology. The authors propose CemiFace, a diffusion-based method that generates synthetic face images with controlled similarity to a subject's identity center, enhancing the discriminative quality of the samples. This approach allows for the creation of diverse and effective datasets for training face recognition models without the need for large-scale real face images, thus mitigating privacy risks. CemiFace outperforms existing synthetic face recognition methods, significantly reducing the performance gap compared to models trained on real datasets. The paper also discusses the potential limitations and privacy implications of the approach, highlighting the need for ethical considerations in synthetic face generation for face recognition applications.", "questions": "Refer to weakness", "limitations": "Refer to weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The use of a diffusion-based model for generating semi-hard samples is an innovative approach that has not been extensively explored in the field of face recognition.\n2. The approach can be extended to use unlabeled data for training, which is an advantage over previous methods that often require some form of supervision.", "weaknesses": "1. The paper is not well organized. This paper should be reorganized to make it easier for the reader to understand the contributions and technical details of this paper.\n2. Eq. 10 seems to be inconsistant to its description. According to the description, it is highly related to the time step.\n3. Fig. 3 is hard to understand. The training losses are not illustrated in the figure.\n4. Despite aiming to reduce privacy issues, CemiFace still uses a pre-trained model that could have been derived from datasets without user consent, raising ethical and privacy concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**:\nIn the last part of the Introduction section, we have clearly and specifically listed four contributions of our work, including: 1. a new and crucial finding; 2. a technical contribution (i.e., CemiFace face image generator) inspired by the finding; 3. an application contribution of our proposed technical approach; and 4. the effectiveness of our approach. Based on your suggestion, we will additionally add a paragraph at the beginning of the Method section to guide readers as: In Section 3.1, we first investigate the relationship between sample similarity and their effectiveness in training FR models, presenting the finding that samples with certain similarities (i.e., center-based semi-hard samples) to their identity centers are more effective for training FR models on a real dataset and subsequently devise a toy experiment to validate it. Inspired by our findings, we propose a novel CemiFace, a conditional diffusion model that produces images with various levels of similarity to an inquiry image in Section 3.2. Specifically, Section 3.2.1 introduces how we construct the similarity condition which is fed to diffusion model to guide the generation, and discusses the $L_{SimMat}$ to require the generated sample to exhibit a certain similarity degree to the inquiry image. In Section 3.2.2, we then present how to use our diffusion model to generate a synthetic face dataset given a fixed similarity condition $m$ and a set of inquiry images\n\n---\n**W2**: As clearly presented in line 174 (above the Eq. 10), Eq. 10 is not related to the time step, instead it represents the loss for reconstructing the identity of the inquiry image when $t \\rightarrow 0$, which is the first part of our Time-step Dependent loss. As presented in line 177, Eq. 12 (rather than Eq. 10) defines our $L_\\text{SimMat}$ inspired by the Time-step Dependent loss (DCFace[23]), which employs $\\gamma_{t}=\\frac{t}{\\mathbf{T}}$ to make the sample have different similarity property at different time step $t$. We will re-phases lines 173-174 as: we employ the Time-step Dependent loss [23] with different time step t at Eq. 12, specifically firstly an identity loss for recovering the identity of the original inquiry image x, which will be applied to produce original facial embedding when the time step $t\\rightarrow 0$.\n\n---\n**W3:**\nPlease refer to the general response G1, where we have rephrased the pipeline and Fig. 3.\n\n---\n**W4:** Ethical and privacy issues are the top priority of our study. While a crucial goal for developing SFR is to eliminate the privacy issue that lies in the face recognition dataset, current SFR approaches are not capable of refraining from privacy risk while providing effective recognition performance. Compared with previous SOTA approaches DCFace[23] and IDiffFace[27] approaches, our method already reduces the privacy risk by avoiding using labelled face recognition images for the model training. These are clearly presented in supplementary material Section C. More importantly, we emphasize that the model development protocol in this paper strictly follows the same protocol of previous peer-reviewed related studies (e.g., the DCFace and IDiffFace) published in top journals and conferences. Thus, we believe our study would not trigger ethical and privacy-related issues.\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ds5tnFKMDG", "reviewer": "Reviewer_5xKY", "summary": "The paper introduces an approach called CemiFace for generating synthetic face images to enhance face recognition (FR) models. The paper provides the first in-depth analysis of how FR model performance is influenced by samples with varying levels of similarity to the identity center, focusing particularly on center-based semi-hard samples. The authors propose a unique diffusion-based model that can generate face images with different levels of similarity to the identity center. This model can produce infinite center-based semi-hard face images for synthetic face recognition (SFR). The method can be extended to leverage large amounts of unlabeled data for training, providing an advantage over previous methods. Experimental results demonstrate that CemiFace significantly outperforms existing SFR methods, reducing the GAP-to-Real error by half and showcasing promising performance in synthetic face recognition.", "questions": "See above", "limitations": "No issues are found in this aspect.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Focusing on center-based semi-hard samples to enhance face recognition performance is a fresh problem formulation that addresses a notable gap in current methodologies.\n\n- The paper provides a solid experimental validation of its proposed approach. The authors investigate factors affecting performance degradation in synthetic face recognition and offer a hypothesis about the importance of mid-level similarity samples.", "weaknesses": "- The method for determining GtR remains unclear. Justification regarding how the proposed model yields a low GtR is absent. Is this low GtR attributed to the utilization of real inquiry images? If so, what measures guarantee that the synthetic facial images remain uncorrelated with the real facial images? In other words, I have a reservation that the method may not generate \"true\" synthetic data but highly relies on an inquiry image. Therefore, it is reasonable to see why a low GtR is obtained.\n\n-  Figure 5 demonstrates that different identities (such as different genders) can be obtained with different m, even with the same input query. There seems to be no way to control the \"number of identities\" generated from this model. If so, how was the supervised loss applied to train a face recognition model? \n\n- How can one ensure high inter-class and large intra-class variations as required for SFR?\n\n- B.3.3. The assertion that high-quality data is not indispensable for achieving markedly accurate facial recognition performance is somewhat counterintuitive and perplexing.\n\n- The method's reproducibility raises concerns, particularly with respect to the training of the model, which lacks clarity. Specifically, the functions F_1 and F_2 in Equations (6) and (7), as well as the role of C_att, are not explicitly defined, and these elements are absent from Figure 3.\n\nThe proposed model generally lacks controllable factors to generate true synthetic face images that favor high inter-class and intra-class variations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:** Low GtR is not attributed to real inquiry data. For instance, even with \\textbf{the same synthetic data DDPM}, our CemiFace would surpass the previous state-of-the-art method DCFace inquired by DDPM data (clearly illustrated in the upper part of Tab. 4), and DDPM provides close performance to real-data. In fact, low GtR is attributed to the way of constructing the facial dataset, i.e. containing center-based semi-hard samples.\n\nTo calculate GtR with fair comparison with other SOTA methods, we train the DCFace released data to generate close AVG performance to their paper by using CosFace loss. This is because the training SFR code of previous methods is not available which prevents us from reproducing the results. Notably, our reproduced DCFace is higher than the results reported in [23]\n\n---\n**W2:** Only images belonging to one identity can be obtained from each input query regardless of the input m value. Despite that face images generated from the same query exhibited large differences, our approach defines them with the same identity label for the later SFR model training. Consequently, the number of identities is fully decided by the number of inquiry face images, which is mentioned in Fig. 1 stating that: \"With our proposed CemiFace, each inquiry image finally forms a novel subject.\". We will further explain this in the revision.\n\n\n---\n**W3:** **(1) high inter-class variations:** Each inquiry face image is selected to be highly independent on other inquiry images. Specifically, we follow DCFace to use a pre-trained FR model to keep samples with a threshold of lower 0.3. We also elaborated this in lines 287-289. \n\n**(2) high intra-class variations:** high intra-class variations are ensured by (a) **changing the similarity condition $m$**, as a small input similarity $m$ results in the generated semi-hard images belonging to the same identity having long distances to the identity center; and (b) the face images of the same identity generated by CemiFace are **distributed in all directions from the identity center**, which can be observed from supplementary material TSNE Fig. 7. This is guaranteed by randomly sampled Gaussian noises $\\epsilon$ input to the diffusion model, which exhibit a large variation.  As a result, both properties would ensure the generated face images of the same identity are almost evenly distributed in a sphere that has a relatively large radius, and thus they would have high intra-class variations.\n\n---\n**W4:**\nDespite that our approach achieved slightly worse FID performance than the previous state-of-the-art DC face, face images generated by our approach still result in a better face recognition performance, suggesting that the semi-hard face images generated by our CemiFace compensate for their slightly worse FID compared to face images generated by DCFace with a mix of easier and hard samples. However, CemiFace still significantly outperforms DigiFace in this FiD. \nOn the other hand, high-quality data is essentially needed, as discussed in Section 4.2.2 (lines 263-282, Table 4) and supplementary material Sections B.1 \\& B.2\n\nWe rephrase the last sentence in B.3.3 as: 'Our method doesn’t intend to generate images similar to the distribution of CASIA-WebFace, but to construct a discriminative dataset that is conducive to providing highly accurate FR performance'\n\n---\n**W5**: $F_1$ and $F_2$ are two stacked linear layers. Here, $F_1$ projects the input similarity m to a latent feature $C_\\text{sim}\\in \\mathbb{R}^{512} $, and then $F_2$ projects the feature concatenating $C_\\text{sim}$ and identity embedding to a condition vector $C_\\text{att} \\in \\mathbb{R}^{128} = F_{2}(cat(E_{id},F_{1}(m)))$ including both the input similarity and identity information. We have provided more details about Figure 3 and the training pipeline in general response G1 and **the PDF**.\n\n---\n**W6:** we kindly disagree. High inter-class variation is ensured by filtering the inquiry data with a cosine similarity lower than 0.3 provided by pretrained FR model, following DCFace (mentioned in lines 288-289). As for high intra-class variations, our proposed diffusion model can generate high-variation samples belonging to each identity. Please refer to the answer of W3.\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qceehVvhz0", "reviewer": "Reviewer_Crof", "summary": "This paper proposes CemiFace, a novel diffusion-based approach for generating synthetic face images with varying levels of similarity to their identity centers. The authors argue that semi-hard negative samples, those with moderate similarity to the center, are crucial for training effective face recognition models. The core of CemiFace lies in its ability to control the similarity between generated images and the input (identity center) during the diffusion process. This is achieved by injecting a similarity controlling factor condition (m) that regulates the similarity level. The paper presents a comprehensive analysis of the relationship between sample similarity and face recognition performance, showing that semi-hard samples, generated with m close to 0, achieve the best accuracy.\nCemiFace demonstrates significant improvements over previous methods in terms of accuracy, particularly on pose-sensitive datasets. The paper further validates its effectiveness through qualitative visualizations and ablation studies that examine the impact of various factors, including training data, inquiry data, and the similarity controlling factor. Overall, this paper contributes a valuable approach to generating synthetic face datasets for face recognition with enhanced discriminative power. The method shows promise in mitigating privacy concerns associated with collecting and using real-world face data while maintaining robust recognition performance.", "questions": "- Written in weakness section.", "limitations": "- This is a well written paper with a meaningful discovery on training with synthetic dataset. The paper would be better positioned in the venue NeurIPS if it would offer more insightful analysis on why similarity control is beneficial, on top of the empirical benefits.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Discovery of the importance of similarity control in synthetic face generation: CemiFace is motivated by the discovery that face images with certain degree of similarities to their identity centers show great effectiveness in the performance of trained FR models. This is an important discovery to the community of synthetic dataset generation. \nUnique use of similarity control: CemiFace introduces a similarity controlling factor (m) within the diffusion process, enabling the generation of faces with varying levels of similarity to the input image. This provides a fine-grained control over the generated data distribution, which is a unique feature compared to existing methods.\nComprehensive analysis of similarity: The authors present a thorough analysis of the impact of different similarity levels on face recognition performance, validating their hypothesis about the importance of semi-hard samples. This analysis provides valuable insights into the relationship between data distribution and model effectiveness.\nRigorous experimental evaluation: The paper conducts comprehensive experiments across various benchmark datasets and data volumes, comparing CemiFace with other state-of-the-art synthetic face generation methods. The ablation studies provide a detailed understanding of the influence of different parameters and factors on the model's performance.\nRobustness of CemiFace: The experiments demonstrate the robustness of CemiFace to different training data, inquiry data, and similarity controlling factors. The method consistently achieves superior results, demonstrating its effectiveness and generalizability.", "weaknesses": "- An in-depth discussion on why face images with certain similarity is more beneficial as a training dataset for the face recognition model would strengthen the paper. For example, an analysis such as a similarity comparison the the real dataset and checking if the difficulty of the CemiFace synthetic dataset becomes closer to that of the real dataset would be nice. Other analysis that offers insights as to why certain similarity control is important would also be welcome.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:**  Thank you for your insightful suggestion and positive feedback. We assume the benefits of the semi-hard training face images could be attributed to: \n\n**(1) **easy training samples are typically images where the face is clear, well-lit, and faces the camera directly, and thus training on such easy samples would not allow the trained FR models to be able to generalize for face images with large pose/age/expression variations and different lighting conditions/backgrounds that are frequently happened in real-world applications. AdaFace[3] also mentioned that easy samples could be beneficial to early-stage training, while hard sample mining is needed for achieving generalized and effective FR models; \n\n**(2)** hard samples normally contain noise data. Specifically, FaceNet[28] demonstrate that the hardest sample mining using a large batch size leads to hard convergence and produces inferior performance. This is because training with very hard samples may not allow FR models to learn effective features but focus on cues apart from facial identities; \n\n**(3)** Semi-hard samples generated by CemiFace mostly contain large posed faces but less face-unrelated noises. We also evaluate the training epochs needed to reach the highest AVG performance for easy samples ($m=0.7$), semi-hard samples($m=0$) and extreme hard samples ($m=-0.5$). Easy samples take 10 epochs to reach the best AVG and 20 epochs to produce the training loss of 0; Semi-hard samples take much longer (38 epochs) to provide the highest AVG while the final training loss is around 3; and FR models training on extreme hard samples could not converge.\n\n\nDue to the real inquiry center is not available, we further calculated the average similarity (using ArcFace to fairly compute them, i.e., avoiding information from adaface and cosface) between randomly paired images, based on random selected 200 identities from CASIA-WebFace, DCFace and CemiFace, where real face images in CASIA-WebFace provide average similarity of 0.51, while face images generated by our CemiFace has a similarity of 0.48. However, face images generated by DCFace are more different from the real face images. with a much easier average similarity of 0.57.\n\nWe have also additionally added an experiment to demonstrate the actual similarity to the inquiry data in **the PDF** Tab. 2. It shows that our generated face images exhibit larger distances (lower similarity) to the inquiry centers than the DCFace. We have also calculated the number of face images belonging to different similarity groups for CemiFace and DCFace in **the PDF** Tab. 3, indicating that our CemiFace tends to generate images showing lower similarities to their identity centers (i.e. all samples are semi-hard), while DCFace containing more easy samples.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable feedback. Reviewers acknowledged that our: (i) **method** is new/innovative (tciQ, L3p9, 5xKY, Crof), interesting (tciQ), effective for SFR (fJPL, L3p9, 5xKY, Crof), and addresses privacy concerns (fJPL, L3p9, Crof); (ii) **discovery** is important (Crof); and (iii) **experiments** are solid (5xKY, Crof) and  comprehensive (tciQ) 


**General response to all reviewers:**
we denote the Author Rebuttal pdf file as **the PDF** 


**G1: Improvement and explanation of the Fig. 3  (@tciQ\& @fJPL\& @L3p9\& @5xKY):**

We have updated the Fig. 3 in **the PDF** with more detailed elaboration. Besides, we will also modify and add the following contents to the beginning of the Sec. 3 as: 


**Methodology overview:** As illustrated in the left side of Fig. 3, the **training process** starts with adding a noise $\epsilon \sim \mathcal{N}(0,1)$ to the clean input image $x$ using Eq. 4 (result in $x_{t}$). Meanwhile, similarity conditions $m$ are fed to the linear layer $\mathbf{F_{1}}$, whose output is then concatenated with the inquiry identity condition $\mathbf{E _{id}}(x)$ to generate a joint representation. This joint representation including both identity and similarity conditions is then processed by the linear projection layer $\mathbf{F} _{2}$ to output the combined condition representation $\mathbf{C _{att}}$ (the lower right part of Fig. 3).  

The $\mathbf{C_{att}}$ is further processed by a cross-attention operation with the intermediate latent representation of diffusion UNet $\sigma_{\theta}$ learned from the input noisy image as: 
\begin{equation}
    CA(Q,K,V,K_{c},V_{c})=SoftMax(\frac{QW_{q}([K,K_{c}]W_{k})^{T}}{\sqrt{d}})W_{v}[V,V_{c}]
\end{equation}

where $\mathbf{C_{att}}$ is treated as the key $K_{c}$ and value $V_{c}$ (same as DCFace) to influence the generated face images. $Q=K=V$ are the query, key and value, representing the latent feature of UNet $\sigma_{\theta}$. Consequently, the diffusion Unet $\sigma_{\theta}$ outputs the estimated noise $\epsilon'=\sigma_{\theta}(x_{t},t, \mathbf{C_{att}})$ for denoising the image as a clean estimated image $\hat{x}_{0}$ (Eq. 9). Based on the obtained estimated image $\hat{x} _{0}$, original $x$ and condition $C _\mathbf{{att}}$, the whole model is optimized by the loss defined in Eq. 13 in an end-to-end manner. 

At the **inference stage** (the upper right part of Fig. 3), random noise $x_{t}=x_{\mathbf{T}}=\epsilon \sim \mathcal{N}(0,1)$ and the time step $t=\mathbf{T}$ are first fed to a CD block, jointly with the similarity $m$ and inquiry image $x$ that are undergoing linear layers $F_1$ and $F_2$ and concatenation to produce the condition representation $\mathbf{C_{att}}$. This results in an estimated noise $\epsilon'=\sigma_{\theta}(x_{t}, t, \mathbf{C_{att}})$. Then, a denoise step is adopted to generate $x_{t-1}$ from $x_{t}$ using Eq. 12 in DDIM[21] for efficient interface speed. This process is repeatedly conducted on the obtained denoised latent images ($x_{t-1}, x_{t-2}, \cdots, x_{0}$) until $t=0$, where $x_{0}$ is treated as the final generated face image.

Here, we assign the same identity label as $x$ to all face images generated from the inquiry image $x$. To ensure high inter-class variation, our inquiry images are filtered by a pretrained FR ( IR-101 trained on the WebFace4M[11] dataset by AdaFace.), which enforces the similarity between each pair of query images is lower than 0.3.

Meta-review: Five experts in the field reviewed this paper. Their recommendations are 4 x Borderline Accepts and a Very Strong Accept. Overall, the reviewers appreciated the paper because it proposes an innovative diffusion-based generation method for face recognition that addresses privacy concerns and shows effectiveness in comprehensive experimental evaluations. The idea of controlling the similarity in synthetic face generation to generate semi-hard samples for training is interesting. Based on the reviewers’ feedback and the authors’ satisfactory rebuttal that addresses most concerns, I recommend it for acceptance. The reviewers raised some important issues and concerns in the Weaknesses that should be addressed in the final camera-ready version of the paper. In particular, the final version should significantly improve the overall structure and clarity of the paper (e.g., Figure 3). It should also provide an analysis of the computational complexity of the proposed method, and a more insightful analysis of why similarity control is beneficial. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)