AugGen: Synthetic Augmentation using Diffusion Models Can Improve Recognition
Authors: Parsa Rahimi
Damien Teney
Sébastien Marcel
Venue: NeurIPS 2025 poster ()
Keywords: Generative Models
Synthetic Data
Diffusion Models
Synthetic Augmentation
Face Recognition
Primary area: applications
TL;DR: We introduce a self-contained augmentation method that uses targeted diffusion sampling to synthesize data that boosts downstream recognition, without any external data or pretrained generators.

Abstract: The increasing reliance on large-scale datasets in machine learning poses significant privacy and ethical challenges, particularly in sensitive domains such as face recognition. Synthetic data generation offers a promising alternative; however, most existing methods depend heavily on external datasets or pre-trained models, increasing complexity and resource demands. In this paper, we introduce **AugGen**, a self-contained synthetic augmentation technique. AugGen strategically samples from a class-conditional generative model trained exclusively on the target FR dataset, eliminating the need for external resources. Evaluated across 8 FR benchmarks, including IJB-C and IJB-B, our method achieves **1–12% performance improvements**, outperforming models trained solely on real data and surpassing state-of-the-art synthetic data generation approaches, while using less real data. Notably, these gains often exceed those from architectural modifications, underscoring the value of synthetic augmentation in data-limited scenarios. Our findings demonstrate that carefully integrated synthetic data can both mitigate privacy constraints and substantially enhance discriminative performance in face recognition. Code and datasets will be made publicly available upon publication.

AugGen:
Synthetic Augmentation using Diffusion Models Can Improve Recognition
Parsa Rahimi Noshanagh
Damien Teney Idiap Switzerland damien. teeeyeidiap....
Sebastien Marcel Idiap, UNIL Switzerland marcel@idiap.cm
EPFL, Idiap Switzerland parea.rahiminoshaagggeee.
Abstract
The increasing reliance on large-scale datasets machine learning poses signifi- cant privacy and ethical challenges, particularly in sensitive domains such face recognition. Synthetic data generation offers promising alternative; however, most existing methods depend heavily on external datasets or pre-trained models, increasing complexity and resource demands. In this paper, we introduce AugGen, self-contained synthetic augmentation technique. AugGen strategically samples from a class-conditional generative model trained exclusively on the target FR dataset, climinating the need for external resources. Evaluated across 8 FR bench- marks, including IJBCC and IJB-B, our method achieves 1-12% performance improvements, outperforming models trained solely on real data and surpass- ing state-of-the-art synthetic data generation approaches, while using less real data. Notably, these gains often exceed those from architectural enhancements, underscoring the value of synthetic augmentation in lamael scenarios. Our findings demonstrate that carefully integrated synthetic data can both mitigate pri- vacy constraints and substantially enhance recognition performance. Paper website: gii thub. io/ auggen/
1 Introduction
Clas M
Baseline
Ours
Class
As machine learning increasingly relies on
application-speeife data, the demand for high-quality, accurately labeled datasets poses significant challenges. Privacy, le- gal, and ethical concerns amplify these difficulties, particularly sensitive areas like human face images. A popular solu- tion is synthetic data generation [54] 2 38).
fry.
Laver Cancentrttion
Concentration
Core idea of AugGen. AugGen boosts the model's overall discriminative capabilities without re- quiring external datasets or pre-trained networks. To achieve this, we propose novel sampling strategy us- ing a conditional diffusion model-trained exclusively the discriminator's 's original generation of synthetic "mixes" of source classes. Incor- porating these synthetic samples into the discriminator's training, results higher intra-class compactness and greater inter-class separation (fours Obaseline) than models trained solely on the original data.
Figure 1:
3), which leverages methods such as 3D- rendering graphics and generative models (e.g., GANs and diffusion models), No- tably, synthetic data can surpass real data in model performance, shown by [54), where 3D-rendered face models with pre- cise labels outperformed cell-daa-basse models in tasks like face landmark local- ization and segmentation, highlighting the advantages of data synthesis, especially for tasks requiring dense annotations. Image
enables the
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

generative models remain underutilized despite rapid advances in VAEs (27], GANs 12 22 20)
and Diffusion models 44
19
21
Comparisons of generative models often use metrics 15 which measure similarity to training data, or subjective user
17
13
like Fréchet Distance (FD)
48
preferences for text-to-image tasks 1Q]
As depicted in Figure 2 currentl, syn-
thetic data generation involves training large-scale generative models [39] on
Current Paradigm
datasets such as LAION-5B [43], then re-
Claclle
Mar
fining them via neering, or textual inversion 22 122 This
prompt engi-
Thendg
Mac
Are(M > ArtlMaae
trend also applies to Face Recognition (FR), where synthetic data aims to mitigate pri- vacy and ethical concerns. However, most methods still rely on large face datasets (which carry their own privacy issues) and auxiliary models, offering no clear advan- tage over existing real datasets. For in- stance, DCFace [25] generates diverse face images from multiple identities and uses robust FR systems and auxiliary networks to filter and balance samples. It remains unclear whether performance gains stem from the datasets, the generative models, or other fac- tors thouht larger, more diverse data typically improves results. Contrary current trends, advocate using generative models as an augmentation tool for FR training rather than replacing real datasets. Two key factors motivate this stance:
Our Approach
Muce
Trai
Nar
The
Paper
ActiMaue >. ActtMap)
Figure 2: Unlike prior methods that depend on external data or pretrained generators, our self- syn- thetic augmentation framework improves recognition purely through its own generative process.
Synthetic datasets generated by diffusion models often leak training data 29 clear benefit over existing priors 225 5 32
6 46 offering no
49 55
2. Since responsible FR datasets are scarce and difficult to collect, we aim to boost performance with limited real data, thereby narrowing the gap between small-scale and large-scale training sets.
In this paper, we focus on scenarios involving limited data, demonstrating how we can increase diseriminative power by generating synthetic samples while only using single labeled dataset. As illustrated in Figure 1, we generate mixed classes that combine features from two or more source classes while preserving their distinct identities. We choose face recognition (FR) as our primary benchmark due to its unique difficulty, as it requires distinguishing between hundreds of thousands of identities within a highly structured input space. Moreover, FR is a privacy-sensitive task where responsible labeled data scarce, making it an ideal setting for studying augmentation. Finally, it benefits from a range of well-established benchmarks that enable consistent and meaningful evaluation. To enhance the effect of margin-basee based losses used by statc-of-the systems, given a dataset of ((X, y)). where X is an image and y is its corresponding label, we train generative model, p(X y). and a discriminator, p(y X), on the same real dataset from scratch. We then introduce a simple yet novel sampling strategy to synthesize new examples. Empirically, we demonstrate that augmenting real data with these carefully generated synthetic samples leads to substantial improvements in the discriminator's 's performance. Our main contribution is to validate this hypothesis in the context of face recognition (FR):
discriminators in FR
H1: A generative model can boost the performance of a downstream discriminator with an appropriate informed sampling, and augmenting the resulting data with the original data that was used for training the generative and discriminative models.
Our contributions are summarized as follows:
We propose a P simple yet effective sampling technique that strategically conditions generative model to produce beneficial samples, enhancing the discriminator's 's training process (Subsec lion 3. without relying on any auxiliary models/data.
We show that mixing our AugGen data with real samples often surpasses even architectural- level improvements, underscoring that synthetic dataset generation can be as impactful as architectural advances (Section 4).
2

---

We demonstrate that AugGen training can be as effective as adding up to 1.7x real samples, reducing the need for more face images while preserving performance (Subsection 4.3).
We show that current generative metrics (e.g., FD, KD) are poorly correlated with downstream discriminative performance, emphasizing the need for improved proxy metrics (Appendix F).
To the best of our knowledge, this is the first demonstration of generative image models effectively enhancing augmentation at this scale without relying on auxiliary models or external datasets.
2 Related Work
Synthetic Data in Computer Vision. For a smaller number of class variations, (e-g., 2 or 3 classes for classification target), authors in [1] train separate generative models. This approach is not scalable for higher number of classes and variations of our target (e.g., we have thousands of classes for training an FR system). In (2). the authors fine-tuned pre-trained diffusion models on ImageNet classes after training on large text-image datasets, demonstrating improved performance on this benchmark through the synthesis of new samples. Authors in (54) leveraged 3D rendering engines and computer graphics. Here as they have access to the underlying 3D Morphable Face Model (3DMM) [4] and closed-form back projection to the image plane, the authors introduced a Face Dataset for landmark detection, localization and also semantic segmentation task. By design, as the method has access to accurate labels in such 3D rendered datasets authors demonstrated a slight advantage the models trained their proposed dataset when it is evaluated against real-world datasets.
Synthetic Data for Face Recognition. SynFace (37 employs DiscoFaceGAN 9 for controllable identity mixup (57), training with FR network on MS-CelebIM [14. 3DMM, keypoint matching. and other priors. DCFace [25] uses dual-condition latent diffusion models (LDMs) for style then filters generated images with auxiliary
and one for identity-trainee on demographic classifiers and a strong FR system. In 45). a StyleGAN2-ADA (33 is pre-trained on large, unlabeled, multi-ethnic dataset, and an encoder transfers space mappings to an FR network to mitigate bias. GANDifiFace [32) combines StyleGAN3 [0] and Stable Diffusion 39 (trained on LAION-5B (43 ) along with DreamBooth [40], for increased intra-class variation.
[56).
IDifffface
5 conditions a latent diffusion model on FR embeddings from network trained on
MSIMv2 g. ID$ (55) similarly conditions a diffusion model on face attributes and an FR network trained on MS Mv2, using both CASIA-WebFas and FFHQ [22) for training. Unlike DCFace's post-processing. ID incorporates identity/attribute information directly into the generation process. Note that using MSIMv2 yields higher FR performance than CASIA- -abbaeee [8]. DigiFace IM IM [3] generates diverse 3D-rendered faces with varied poses, expressions, and lighting. In [38), off off-the-shelf image-to- poree-tt-immae to-image translation [51 60 further boosts DigiFace performance despite lacking explicit identity information. Additional prior work is discussed in Appendix A.
3 Methodology
Figure 3 illlustrates our approach, where a discriminator Morig and a generator G are trained on the same dataset. By strategically sampling from G. we generate synthetic images forming new classes, augmenting the original dataset. We first define the problem for the discriminator and generator in Section 3 and [Section 3, then introduce our key contribution: generating new classes (Finding Weights, Figure 3(c) to complement real datasets with synthetic images.
Discriminatior, Assume a dataset D Dorig = (X,,, ( where each X, E l x3 and ys e (0....,(-1), 1} (1 k). The goal is to learn a discriminative model fea.. X y that estimates p(y(X) (c.g., on ImageNet [41] or CASIA- WebeFee (56)). Typically, similar images have closer features under measure (e.g., cosine distance). We train fe... via empirical risk minimization:
dis
arg min E. (xx.))))DDv [asss (X),()))))) ()),)),
(1)
0n.
where Lais typically cross-entropy, and hais denotes hyperparameters (e.g.. learning rates). The resulting model Morig = feg. is shown in Figure 3la).
Generative Model. Generative models seek to learn the data distribution, enabling the generation
of new samples. We use diffusion models [47 [], which progressively add noise to data and train
3

---

21]. S is learned in two stages, First, for a given noise level 0, we add noise N (x)e(X) (or X directly in pixel-based diffusion) and remove it via:
denoiser S. Following [19
L(S σ) (x(xs))~Dart (NV (0.((..0))
[lIS (EVAE (X)
N, X
where c(y) denotes the class condition, and E-VE(() and DvAE(.) are optional VAE encoder and decoder. In the second stage, sample different noise levels and minimize:
Ozen
arg min [o-N((u,02) DAo L(Soa..
Ogen dede
where X weights each noise scale. Latent diffusion [3) conducts denoising in a compressed latent space, reducing computational cost for high-resolution data.
3.1 Class Mixing
In our formulation, is one-hot encoded for each label in Dorig, then mapped to the denoiser's condition space. After training the conditional denoiser Spaca Figure 3] (c)) via [quation 3 we can sample from the generator two ways:
(+)
Clevsie Thereeg
(i)
Mar
0.0. 2..........
b.0.
Training
Clomph
Treving
x A/0.1
A..... Mrd,
Trideo A
Use the same one-hot vectors as in training, producing samples similar to Dorig As an example, when pass- ing the one-hot vector for the first class, the generator synthesizes sam- ples that resemble this class (Figure 3 (d)), collectively forming D reepro
I.
past.....
Inference
X X
2. Apply novel condition vectors c* dif- ferent from those used during train- ing.
Tranding
AveDtiaa Mee(Mag)
Figure 3: Overview diagram of AugGen: (a) A labeled dataset, Dorig, is used to train a class-conditional gener- ator, G(Z c). and a discriminative model. Morig- (b.d) Reproduced dataset, D)repro, closely mimics Donig under the original conditions. (c) We find new condition vectors, C to generate an augmented dataset, Daug, using the generator. (f) Augmenting Dorig with D boosts Morig performance without auxiliary datasets or models.
We explore combining known condi- tions to synthesize entirely new classes, aiming to increase inter-class separation and feature compactness
presented in Figure By leveraging the previ- ously trained Morig, these additional sam- ples can make Mmix (i.e., discriminator trained on the mix of real and generated data) better across diverse benchmarks.
Given two classes i and j with one-hot vectors et and cl, we construct a new class condition via
c" c==act Bcl,
We denote the trained denoiser's generation process by G, so X GiZ c²) uses noise Z ~ (O I) and condition e to iteratively denoise the input. To find suitable a and 3, we formulate the problem as a grid search, aiming for dissimilarity to classes i and j while preserving class coherence for repeated samples from GZZ c²). We set the a and /3 to some possible combinations in a linear space of the values between 0.1 1.1. Intuitively, the larger either a or 3, the more the generator will reflect the attributes of the corresponding class (i.e., class i and respectively). For example, possible 0.4. We denote W, the set which contains
combinations would a = 0.3, B = 0.5 or a = 1.1, possible values of a and 3. We also select some subset of L. and call it L.,, for the set to contain some specific classes. Then we randomly select two values from the T.g, namely and j. Later for each (a, 9) W apply the [qquation 4, to get the c². We generate three types of images. The first two is the reproduction datasct, Drepro as before by setting the conditions to and c², to get X = G c²) and X = G(Z, )²). Finally the third one X = G(Z, c²) ). By passing the generated images to the fe.m- (i.e., discriminator which was trained on the Dorig) we get the features, e'. e and e" respectively. We seek to maximize the dissimilarity between generated images so that

---

can treat the new sample X" as a new class. For this, we use dissimilarity measure. ma which the higher the absolute value produces the more dissimilar the inputs are. We calculate this measure for each of the reproduced images of the existing classes with respect to the new class, d; = ma(e', ee)) and d, = ma'e', e"), and we define the total dissimilarity between the reproduced classes and the newly generated class as ma ma = |d:| + Idzl. We repeat this process K times, this means that we get K different X" for the same c') and c² ). We also want each K' X" to be as similar as possible to each other so we can assign the same label/class to them for a fixed a and B. To this end, we also calculate similarity measure, m,, in which the higher the absolute output of this measure is the the more similar their input is. We define the total similarity between the K generated X as motal We hypothesize and verify later with our experiments that the good candidates for a and / are the ones total
that have a high value of the m total motal algorithm
+ motall This search for a and / is outlined in the
Algorithm 1: Grid search for a and 3
Algorithm 2: Generating Daug
Require: Search range for o, B E [0.1, 1.1],LI, C T,, K: Number iterations.
Require: a and (3" from
algorithm L, L, C:
Require: G),,-): Class-conditional Generator trained on Dorig
Number of mixed
Require: fog,. Discriminator trained Doorig
classes, N: Number of
Output: a" and (3*
samples per class.
Require: G(.,
Create set W [((,, 33 | a. 3 E 1.1]]
Class- clans-conditiona Generator trained on Dorig
Randomly select two values and j from Le, M = 1}:
for each (ax, (3 E W do ac + Be), M
0
Output: D
for k 1,.1,.... l,..., K do
Create empty set D= for 1...., C do
Get Repro Images: X' = =((Z,e'),
GiZ, c²);
Get Interpolated Images: X* c'):
Randumly select two values i and j from L,; c² =a'' + +.c'.
Get Repro Features: e', e² 80g.. (X'), foaw (X));
Get Interpolated Feature: = f0am. (x");
Add e to F;
Create empty T;
Dissimilarities: d, = ma(e', e²) ), dj = Seaaeeeee e²);
n nammlle :....;N NN do
Total dissimilarity: mootall |d. + +dgli
X = - e");
end my
Add X" to T; end
total
= 0;
Vp. q E Flp + 4 Calculate ma (e", e²) and add to m. mistal: Final measure: m total
Add T to Daug;
m, sotal
m total and add to M;;
end
Return Daug,
end
Return ax and 3 that the vn total in M is high;
After finding candidate values for a and 3, by randomly selecting classes from T., and calculating ct, we can generate images that represent hypothetically new class. The output of this process is what we call generated augmentations of the Dorig, or Daug as depicted the Figure (e) and presented in algorithm 2] As shown in Figure I] the newly generated classes are similar within themselves but distinct from their mixed classes, retaining source-class cues to aid discrimination by design. Training with the mix of Dorig and Daug (Figure 3(f) benefits the discriminator, demonstrated in Section 4
Experiments
We demonstrate the effectiveness of our proposed augmentation method for the problem of Face Recognition (FR). Large datasets are usually required for modern FR systems, improving perfor- mance with limited data is crucial.
4.1 Experimental Setup
Training Data. We evaluate our approach using two real-world datasets, Doris; CASIA- [56) and subset WebFace4M [6]]. The WebFace4M subset, referred in this work to as Web- Face1601e was selected to include approximately 10,000 identities (i.c., like CASIA- each
5

---

represented by 11 to 24 samples, resulting in a total of 160K face images. More details about the datasets are presented in the Appendixx B
Discriminative Model. To ensure fair comparison across different methods during the training of the discriminator, we adopted a standardized baseline. This baseline employed an FR system consisting of an IR50 backbone, modified according to the ArcFace's implementation [8], paired with the AdaFace head [24] to incorporate margin loss. Furthermore, when analyzing architectural improvements the network level, we explored training solely with real data versus mixed data. For this analysis, we used 101 due to its increased parameterization, which is expected to enhance its ability to generalize. Each real or mixed dataset was trained multiple times with identical hyper- parameters but different seed values. More details are outlined in [Apendixx Cj For comparisons, we repeated these procedures using several synthetic datasets from the literature: the original Digi- Face IM (3D graphics), its RealDigiface translations (38 (Hybrid, 3D, and post-processed), and two diffusion- datasets, DCFace (25] and IDiff-Face [5] Additionally, standard augmentations for face recognition tasks were applied to all models. These augmentations included photometric trans- formations, cropping, and low now-resolution adjustments to simulate common variations encountered in real-world scenarios.
Generative Model. To train our generative model, we used variant of the diffusion formulation [1 For the Dorig CASIA-WebFace we used the latent-based formulation which, as depicted in Equation 2 we employed a VAE to encode the image to a compressed space and decode it back to the image space. For WebFace 16K we used the pixel space variant for better coverage of different diffusion models. Furthermore, we set the one-hot condition vectors c=10kk have a size of ~ 10,000 corresponding to the number of classes in Dorig, We train two versions of the latent diffusion model (LDM) from scratch, labeled small and medium, to analyze the impact of network size and training iterations on the final performance, following the size presets outlined in the original papers [9]. For the pixel-space diffusion model, we mainly used the small variant. Details, including generator design choices are presented in Appendix C
Grid Search. As presented in the algoritht I we need to find an appropriate a and / for generating useful augmentations based the generator trained in the previous section. For the Dorig CASIA- WebFace which has the long-tail distribution of samples per class, we set the Lg to the classes from the generator that are presented more than the median number of samples per class. Naturally, empirically observed that these classes are better reproduced when we were generating Drepro For the case of WebFace 160K the L., is all the classes. Later we set the W to {0.1, 0.2, .... 1.0, 1.0, 1.1] for searching a and /3 to calculate the new condition vector c*. Closely related to how the FR models are being trained, especially the usage of the margin loss (i.e., AdaFace [24] or ArcFace (8), we set the measure for dissimilarity between the features of the two sample images, X and X using cosine e²
|[]e_TT Note that the es were calculated using a discriminator that was trained solely on the Dorig We treat the values of the measure in such way that the higher the output of the measure the more it reflects its functionality (i.e., the larger the measure for dissimilarity is the more dissimilar the inputs are). Accordingly, we set the similarity measure to m, |||e-] which again reflects that the inputs are more similar if the output of this measure is closer to 1. We iterate multiple choices of the and j and average our m total for each of the choices. A sample of the output of this process depicted in Figure 5) Here we observe that by increasing the a and 33 from (0.1, 0.1) to between (0.7, 0.7) and (0.8, 0.8) the measure increases and after that, it will decrease when we go toward (1.1,1.1) specifically, we are interested in the a 3 line as we do not want include any bias regarding the classes that we randomly choose. We consider three sets of values for (o,, B), (0.5, 0.5), (0.7, 0.7) and (1.0, 1.0) corresponding to the m total of 1 1.48, 1.58 and 1.53 respectively. Then the (a* , ) respectively from the algorithm 1 for CASIA-WebFace is (0.7,0.7).
similarity to ma = 1 -
e'.e²
Based on our observations, for the WebFace160K dataset, we performed a coarser parameter search with higher concentration in the range 0.5 to 0.9. The total metric value, total for WebFace 160K is illustrated in the lower part Figure 5 Using this approach, we evaluated m total for the parameter pairs (a:, 3) at specific points: (0.5, 0.5), (0.7. 0.7). (0.8, 0.8), and (1.0, 1.0). The corresponding m total values were 0.6068, 0.7256, 0.7390, and 0.7230, respectively. Based on these results, the (a",8*) pair for WebFace1600 was determined to be (0.8, 0.8), as achieved the highest m total
value of 0.7390. In Appendii G we guantitatively demonstrated the effectiveness of this measure in

---

ID. D
Dr ED"
m. D
Value of Proposed Metric
16
14
12
13
14
12
0.8
0.
0.5
0.4
bete
Wisee of Propossed Marric
Figure 4: Randomly sampled images. From left to right: The first column shows variations of randomly selected identity (ID 1) from Dorig, The second column presents the reproduction of the same ID using the generator. conditioned on the corresponding one-hot vector G(Z c1). The third and fourth columns follow the same pro- cess for different ID,, with the middle column representing a newly synthesized identity gener- ated by conditioning the generator on G(Z, c*). The samples above the red line are from CASIA- WebFace, while the lower part corresponds to WebFacel60K.
06
000
02
Figure 5: The value of the proposed measure m total for setting the candidate values of a (x axis) and B (y axis). Here for each and / and our 100 combination of L, we calculated the
m motal by setting the K in algorithm 1 to 10.
the final performance of the discriminator when we trained it on the synthetically generated dataset using various and B.
Computational Complexity. The search is computationally efficient, requiring fewer than GPU-days on single consumer-grade GPU (i.e., RTX 3090 Ti in our case), with 1000 mixes (5 samples/class) per grid point. Most compute was spent on repeated training runs for reliable mean/variance reporting. See See [Subsecttion C.6 for detailed compute complexity breakdown.
Synthetic Dataset. For generating the reproduction dataset Drepro, we the condition for each of the 100,000 classes in the original CASIA-WebFace and WebFace 160K dataset to the generator. The number of samples per class is 20 unless mentioned otherwise. For generating Daug randomly sampled 10, 000 -0 50, 000 combinations of the IL,, (Card(L=)), (samples with more than the median number of sample/class in case of CASIA WebFace as the Dorig), and fixed them for all the experiments. Later by setting the and /3 to candidate values found in the previous section, (i.e., like (0.7,0.7) for CASIA), we generated 10 to 50 sample per mixed of selected classes. In Figure 4 some samples of the generated images are shown, where the first and last columns depict examples of the two classes in the Dorig The second and 4-th columns are the reproduction of the same identities from the first and last column, respectively. Drepro, Each line is generated using the same seed (source of randomness in the generator), and finally, the middle column (3rd from left) is the Daug which is generated by X" = G(Z, c²) when we calculate the e* by optimum a and We can observe that the middle column's 's identity is slightly different from the source classes while being coherent when generate multiple examples of this new identity. By design, these classes can be considered as hard examples for the discriminator. This subtle difference is one of the reasons why our augmentation improving the final performance. In the Appendix more samples are presented.
4.2 Face Recognition Benchmarks
We show that our synthetic augmentation is boosting the performance of a model trained with the real dataset in all of the studied public FR benchmarks. For this purpose, we evaluated against two sets of FR benchmarks, The first set consists of LFW [18, CFPFP 44), CPLFW (58) CALFW [59], AgeDB
7

---

331. which includes mainly high-quality images with various lighting, poses, and ages the average of these benchmarks presented in T[Table Jas Avg-H. The second set involves benchmarks consisting of medium to low-quality images from a realistic and more challenging FR scenario (NIST IJB-B/C)
31 53 and TinyFace [7]. For evaluation, we report verification accuracy (i.e. True Acceptance Rate (TAR)), where the thresholds are set using cross-validation in the high-qualic benchmarks, and TARs at different thresholds determined by fixed False Match Rates (FMR) in IJB-B/C. Specifically for the latter, we are mainly interested in the verification accuracy for two thresholds that are usually used in real-world scenarios when the FR systems are being deployed, namely TAR @ FPR= 1--006 and TAR @ @FR=1ee-0 for both IJB-B and IJB-C. In the Table 1 the Aux column depicts that if the method under study used any auxiliary model for the generation of the dataset other than the D The ideal value for this column is N which refers to not using any auxiliary model/datasets. The n and n" depict the number of synthetic and real images used for training the discriminative model.
The final values for the benchmarks are reported as the mean and std of the observed numbers when we are changing only the seed as discussed before. Details about the benchmarks, including High-Quality benchmarks and TAR at additional thresholds, are provided in in[Appendix D TTable 1 is divided into two sections, separated by a triple horizontal line. The upper section compares AugGen, using the CASIA-WebFace dataset the source, and the lower part is when we set the Dorig to WebFace 160K. For each, we considered fully synthetic face recognition, FReyn data, and a combination of synthetic and real data, (distinguished by double horizontal line) FRauix This comparison evaluates their performance relative to the original source dataset (i.e.. fully real, Fr and relevant works, including synthetic data from three approaches: the proposed AugGen, DCFace (25). and IDiffFace [5). The triple horizontal line segmentation is primarily due to the use of MASSA-WWbFaad and among other data/models in the latter two methods' generation pipelines. For each part of the table, bold and underline numbers are presenting best and second best respectively. In the second part, in case augmentation with the real aesSe-eeeaaaeeee is performing better than solely training with the -ebbace (i.e. middle part of both tables) the cell is shaded in gray. We are observing inconsistencies in different benchmarks for other methods. For instance, for LIB-B/C DCFace is not performing better than CASTA WebFace alone and IDiffface is not outperforming FRyeal in thresholds set to low FPRs (i.e., TAR @ FPR=1c-6). In the case of FRyeal training, we additionally used the IR 101 network depicted as f. This is done to demonstrate the introduced augmentation samples can be as important as architectural-leve improvements. As in most cases the less parametrized network (i.e., IR50) trained with the AugGen samples is outperforming the more parametrized network, IR101, solely trained on the original samples, D This is in conjunction with the fact that in most cases using the IR101 FRyeal training outperforms the simpler IR50 model. Additionally, in case our augmentations also perform better than architectural improvements we shade the corresponding cell to green For the less challenging benchmarks presented by Avg-H H in Table 1 we observe that although our method consists of smaller number of samples and does not use any auxiliary model/data we are performing competitively with other state-of -oee-o--thearrr art (SOTA) methods/datasets. In the second part of this table we are observing mainly all the methods that we combined with the SeSSA-WeFFace are boosting the discriminator which is solely trained on the CASIA-WebFacd For IJB/C we demonstrate better performance being the best in most FPRs although our datasets were generated for augmentation by design. By observing the results after the augmentation (second part of the table), AugGin is the only method that consistently performs better than the baseline. One interesting finding was the performance drop of the model when it was combined with the CASIA-WebFace. But we are observing that consistently in all of the benchmarks, our augmentation methodology is boosting the baseline. We demonstrate that although we did not any auxiliary model/data our synthetic dataset performed competitively with other state-of-thee the-thr-rt methods or even outperformed them in some cases.
The lower part of the triple horizontal line reports results with AugGen samples using our Web- Face160K: as the Dorig The observations remain the same, as in most cases, we are performing even better than architectural improvements.
As shown in Figure 1, the discriminator's feature space exhibits reduced intra-class variation and increased inter-class separation, with further details in In[Appddixx H
8

---

Table I: Comparison of the FRayn training (upper part), Freaal training (middle), and FRmix training (bottom) using CASIA- WebFact/WebFace against standard FR benchmarks. Avg-H depicts the average accuracy of all high-quality benchmarks including, LFW, CFP-FP. CPLFW, AgeDB, and CALFW. Here n² and n" depict the number of Synthetic and Real Images respectively and Aux depicts whether the method for generating the dataset uses an auxiliary information network for generating their datasets (Y) or not (N), the t denotes network trained on IR101 if not the model trained using the IR50. The numbers under columns labeled like C/B-1e-6 indicate TAR for IJB-C/B at FPR of 1e-6. TRI depicts the rank-1 accuracy for the TinyFace benchmark.
when the models are evaluated in terms of accuracy
Method/Data
B-le-6
B-le-5
C-Ie-6
C-Ie:5
Avg.
DigiFaceIM IM
N/A
1.22M
15.31=0.42
29.59=0.82
26.0640.77
36. 36.3420.89
32.30=0.21
78.97=0.44
RealDigiFace
Y
1.20M
21.37
39.1420.40
36.1810.19
45.5520.55
42.64=1.70
81.34.0.02
Diff face
Y
1.2M
26.84=2.03
50.08=0.48
41,75+1 04
51.93+0.89
45.98
84.68=0.05
DCFace
Y
1.2M
22. 4.35
47.84=6.10
35,27 78
58.2247.50
45.94
91.56:10.09
DowK (Ours)
N
0.6M
1..4
04.54=0.59
45.15 04
61.52=0.47
52.3340.03
88.78=0.06
Despeo (Ours)
N
0.6M
15.71a3.12
45.9724.64
31.54416.55
58.6123.89
53.6120.47
90.6440.07
CASIA WehFace
N/A
0
0.5M
1.0240.26
5.06&1.70
0.73+0.18 19
5.37+1.41
58. 38.1240.31
94.21=0.09
CASIA- Faleee
0.5M
0.7420.31
3.94&1.62
0.3840.113
3.9221.96
59.6420.49
94.8420.07
IDiffffac
Y
1.2M
0.5M
0.8920.07
5.8020.63
0.70x0.11
7.4642.08
59.3220.34
94.8620.02
DCFace
0.5M
0.5M
0.26+0.11
1.59+0.51
0.1800.07
1.54+0.59
56.60=0 41
94.72=0.09
Dess (Ours)
N
0.6M
0.5M
2.61:0.91
15.74:3,20
4.36:1.41
18.58:3.99
59.82=0.13
94.6620.03
WebFace160K
N/A N/A
U
0.16M
12.13:1.87
72. 12.0180.18
70.37:0.75
78.8120.32
61.5120.16
92.5020.02
WcbFace160K
0.16M
34.84=0.49
74.10=0.24
72.56:0.02
81.2620.14
62.5920.01
93.32=0.12
D (Ours)
0.6M
0.16M
36
36.62=0,77
78.32=0,33
78,58#0.15
85.02+0.15
61 60.600003
94.17=0.08
Table 2: Effect of adding more real samples from WebFace4M to WebFace160k in comparison to
adding more synthetic images. The backbone for all models is IR50. Here Avg-H depicts the average accuracy of all high-quality benchmarks including, LFW, CFP-FP, CPLFW, AgeDB, and CALFW. Ratio depicts the ratio number of real samples used over the number of samples in WebFacel 10KK.
The numbers under columns labeled like C/B- 6 indicate TAR for IJB-C/B FPR of 1c-6.
Syn #Class x #Sample
n'
n²
B-1c-6
B-Ie-5
C-1c-6
C-Ie-5
Avg-H
Ratio
160K
0
32. 32.1321.87
72.18
70.37
78.81
92.502002
(10K x 20)
160K
200K
34.93.0.50
76.15
75.18
83.06
93.77a004
(20K x 20)
160K
400K
36.54a1.27
78.00+0.3
78.48-0.55
84.40=a.or
93.96s001
(25K x 20)
160K
500K
36.35-omm
77.87-0.52
78.61
84.49.0.1
10+00s
(30K x 20)
160K
600K
36.62
78.32=9.33
78.58=0.15
85.02:0.s
94, 94.17s
0
160K + 80K
0
33. 13.78
77. 29.12
77.38
83. 03.50
93. 93.85
1.5
160K+ + 110K
0
33,532147
78.26an.05
78.49
85.0220.01
94.1920.01
1.69
800K
0
32.020000
87.68
87.11
92.27
96.462000
5.0
4.3 Gains over Additional Real Data
In this section, we aim to address a critical question: How much additional real (non-generatie data would it take to achieve the same performance improvement as our synthetic augmentation? This experiment vital because the primary goal is to maximize the accuracy of the face recognition (FR) system using the existing dataset. To evaluate this, we used our WebFacel66s 160K subset as baseline and incrementally added data from the WebFace4M dataset. This process allows us to determine how the performance boost achieved through AugGen compares to the addition of real data, providing a clear measure of its effectiivnees.. In Table 2| the Ratio represents the proportion of additional real samples added to WebFace1660 160k (e.g., 160K + 110K with Ratio of 1.69). Remarkably, adding approximately 600K AugGen samples delivers performance gains comparable to including 10K real images. This highlights that AugGen achieves equivalent performance improvements with significantly fewer real images.
5 Conclusions
In this work, we introduced AugGen, a novel yet simple sampling approach that carefully conditions a generator using a discriminative model, both trained on a single real dataset, to generate augmented samples. By combining these synthetic samples with the original real dataset for training, we enhance the performance of discriminative models without relying on auxiliary data or pre-trained networks. Our proposed AugGen method significantly improves discriminative model performance across 8 FR
9

---

benchmarks, consistently outperforming baseline models and, in many cases, exceeding architectural- level enhancements highgiggnnn its potential to compete with architectural-leve improvements. We further demonstrate that training with uugGen-augmente datasets is as effective using 1.7x more real samples, emphasizing its impact on alleviating data collection challenges. Additionally, we identify inconsistencies in CASIA- A- -a--eeFFaee-aaee more reliable benchmarking on LJB-B/C. Our findings underscore the potential of augmentation-ban approaches for improving discriminative models.
evaluations and recommend alternative datasets for
Limitations. The principal limitation of our approach is its computational cost: to isolate the impact of synthetic data, we train the generator from scratch on the target datasets. Nevertheless, by conducting experiments under these controlled conditions, we establish the hypothesis that synthetic samples generated via our sampling strategy boost the discriminator's performance. Moreover, we expect our method to extend to other architectures (e.g.. other multi-step generators, autoregressive, and flows), including pre-trained generators, offering broader practical applicability.
Future work. promising research direction is reformulating margin losses in FR to be compatible with soft labels. By establishing correlation between target soft labels and c* (e.g., with a, B 0.7 increasing m total, natural choice for soft target labels would be 0.5, 0.5 for corresponding source classes), future studies can explore whether treating class as a soft-class or new one yields better performance. Also, would be interesting to see whether the selection process of IL, will have major effect on the performance of the models, like mixing some classes will deliver a better performance increase than others.
This research is based on work conducted in the SAFER project and supported by the Hasler Foundation's 's Responsible AI program.
Acknowledgment.
10

---

References
[1] Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313332, 1982.
[2] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenet classification. Transactions on Machine Learning Research, 2023.
[3] Gwangbin Bae, Martin de La Gorce, Tadas Baltrušaitis, Charlie Hewitt, Dong Chen, Julien Valentin, Roberto Cipolla, and Jingjing Shen. Digiface-lm: million digital face images for face recognition. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3526-33 3526-353,, 2023.
[4] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '99, page 187-194, USA, 1999. ACM Press/Addison-Wert Wesley Publishing Co.
[5] Fadi Boutros, Jonas Henry Grebe, Arjan Kuijper, and Naser Damer. Idiff-face: Synthetic-base based face recognition through fizzy ddentity-conditionel diffusion model. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 19650-19661, 2023.
[6] Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. 2023.
In 32nd USENIX Security Symposium (USENIX Security 23), pages
[7] Zhiyi Cheng, Xiatian Zhu, and Shaogang Gong. Low-resolution face recognition. In Computer Vision ACCon--ACVAA 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part III 14, pages 605-621. Springer, Sprrnner,, 2019.
[8] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4690-4699, 2019.
[9] Yu Deng, Jiaolong Yang. Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3d imitative-contrastive learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5154 2020.
[10] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
[11] Maayan Frid-Adar, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan. Synthetic data augmentation using gan for improved liver lesion classification. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 289-293, 2018
[12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1):119-1944 2020.
[13] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M. Susskind, and Navdeep Jaitly. Matryoshka diffusion models. In The Twelfth International Conference on Learning Representations, 2024.
[14] Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb- 1m: A dataset and benchmark for large-scale face recognition. In Computer Eetion-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October I 11-14, 2016, Proceedings, Part III 14, pages 87-102. Springer, 2016.
[15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.

---

[16] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[17] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pages 13213-
13232. PMLR, 2023.
[18] Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Searned-Milles Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop faces in RRellLLife' Images: detection, alignment, and recognition, 2008.
[19] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusson-based generative models. Advances in neural information processing systems, 03226665-265777 2022.
[20] Tero Karras, Miika Aittala, Samuli Laine, Erik Hrknen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34:852-863, 2021.
[21] Tero Karras, Miia Aittala, Jaakko Lehtinen, Janne Hellsten. Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Proc. CVPR, 2024.
[22] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401 1401-4410, 2019.
[23] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages $110-8119, 2020.
[24] Minchul Kim, Anil K Jain, and Xiaoming Liu. Adaface: Quality adaptive margin for face recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 18750-18755 2022.
[25] Minchul Kim, Feng Liu, Anil Jain, and Xiaoming Liu. Dcface: Synthetic face generation with dual condition diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12715-12725, 2023.
[26] Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems, 655166 2023.
[27] Diederik P Kingma, Auto-encoding variational bayes. arXiv preprint arXiv: 1312.611 2013.
[28] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
[29] Zhangheng Li, Junyuan Hong, Bo Li, and Zhangyang Wang. Shake leak: Fine-tuning diffusion models can amplify the generative privacy risk. In 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 18 32. IEEE, 2024.
[30] Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code. arXiv preprint arXiv:2412.0662. 2024.
[31] Brianna Maze, Jocelyn Adams, James Duncan, Nathan Kalka, Tim Miller, Charles Otto, Anil K Jain, W Tyler Niggel, Janet Anderson. Jordan Cheney, et al. Iarpa janus benchmark-c: Face dataset and protocol. In 2018 international conference on biometrics (ICB), pages 158-165. IEEE, 2018
[32] Pietro Melzi, Christian Rathgeb, Ruben Tolosana, Ruben Vera-Rodriguez, Dominik Lawatsch, Florian Domin, and Maxim Schaubert. Gandiffface: Controllable generation of synthetic datasets for face recognition with realistic variations. arXiv preprint (10i:::3035199622 2023.
12

---

[33] Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 51-59, 2017.
[34] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models, In International Conference on Machine Learning, pages 7176-7185. PMLR, 2020.
[35] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffussin probabilistic models. In International conference on machine learning, pages 8162-8171. PMLR, 2021.
[36] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El- El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07195 2023.
[37] Haibo Qiu, Baosheng Yu, Dihong Gong, Zhifeng Li, Wei Liu, and Dacheng Tao. Synface: Face recognition with synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10880-10890, 2021.
(38] Parsa Rahimi, Behrooz Razeghi, and Sehastien Marcel. Synthetic to authentic: Transferring realism to 3d face renderings for boosting face recognition. arXiv preprint (xXiv:2407.07621 2024.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[40] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-200 22510, 2023.
[41] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)221 12111,22222222 2015.
[42] Mchdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. As- sessing generative models via precision and recall. Advances in neural information processing systems, 31, 2018
[43] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25221 2022.
[44] Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. Frontal to profile face verification in the wild. In 2016 IEEE winter conference on applications of computer vision (WACV), pages 1 1-9. IEEE, 2016.
[45] Artem Sevastopolskiy. Yury Malkov, Nikita Durasov, Luisa Verdoliva, and Matthias Nießner. How to boost face recognition with stylegan? In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 00924-209344 2023.
[46] Hatef Otroshi Shahreza and Sébastien Marcel. Unveiling synthetic faces: How synthetic datasets can expose real identities. arXiv preprint Xrvi:22400. 24015,, 2024.
[47] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
[48] George Stein, Jesse C. Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Leigh Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L. Caterini, Eric Taylor, and Gabriel Loaiza-Ganem. Expos- ing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. In severth Conference Neural Information Processing Systems, 2023.
13

---

[49] Zhonglin Sun, Siyang Song, Ioannis Patras, and Georgios Tzimiropoulos. Cemiface: Center- based semi-hard synthetic face generation for face recognition. In The Thirty-eightn Annual Conference Neural Information Processing Systems, 2024.
[50] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 18182,2,,,.- 2016.
[51] Justin Theiss, Jay Leverett, Daeil Kim, and Aayush Prakash. Unpaired image translation via vector symbolic architectures. In European Conference on Computer Vision, pages 17-32. Springer, 2022.
[52] Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. In The Twelfth International Conference on Learning Representations, 2024.
[53] Cameron Whitelam, Emma Taborsky, Austin Blanton, Brianna Maze, Jocelyn Adams, Tim Miller, Nathan Kalka, Anil K Jain, James A Duncan, Kristen Allen, al. Iarpa janus benchmark- b face dataset. In proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 90-98, 2017.
[54] Erroll Wood, Tadas Baltrušaitis, Charlie Hewitt, Sebastian Dziadzio, Thomas J Cashman, and Jamie Shotton. Fake till you make it: face analysis in the wild using synthetic data alone. In Proceedings of the IEEE/CVF international conference computer vision, pages 3681-3691, 2021.
[55] Jianqing Xu, Shen Li, Jiaying Wu, Miao Xiong, Ailin Deng, Jiazhen Ji, Yuge Huang, Guodong Mu, Wenjie Feng, Shouhong Ding, al. Id preee--riiingeeeeddireeeeee diffusion models for synthetic face recognition. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[56] Dong Yi, Zhen Lei, Shengeai Liao, and Stan Z. Li. Learning face representation from scratch. arXiv preprint 1rXir:1411.7923, 2014.
[57] Hongyi Zhang. mixup: Beyond empirical risk minimization. arXiv preprint 17110:1710 2017.
[58] Tianyue Zheng and Weihong Deng. Cross-pose Ifw: A database for studying cross-pose face recognition in unconstrained environments. Beijing University of Posts and Telecoomuunications, Tech. Rep, 5(7), 2018.
[59] Tianyue Zheng, Weihong Deng, and Jiani Hu. Cross-age Ifw: A database for studying cross-age face recognition in unconstrained environments. arXiv preprint arXiv:1708.08111 08197, 2017.
[60] Shangchen Zhou, Kelvin C.K. Chan, Chongyi Li, and Chen Change Loy. Towards robust blind face restoration with codebook lookup transformer. In NeurIPS, 2022.
[61] Zheng Zhu, Guan Huang. Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Jiwen Lu, Dalong Du, et al. Webface260m: benchmark unveiling the power of million-scale deep face recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 0492-10550. 2021.
14

---

NeurIPS Paper Checklist
Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in
the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the
results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to vi- olations of these assumptions (e-g., independence assumptions, noiseless settings, model well-specificatioon asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by review- ers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't I acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The results are mainly empirical. We are reporting improvements using bench- marks. For each, we are also reporting confidence intervals.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
15

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Additionally, all the code, models and synthetic datasets will be publicly available for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
If the contribution is primarily new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]
Justification: All the code, models and synthetic datasets will become publicly available,
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (bttps::/nniss.cccppubll
guides htts://ooSuumissiooooooo
for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e-g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https!
for more details.
16

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits. hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes]
Justification: Most important hyperparameters are presented in the Appendix, also as mentioned previously, all the code and models will become publicly available upon publication.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that
is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer:[Yoe Yes]
Justification: For each experiment and other baselines, we run the experiments multiple times based on the observed variacnes
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a
library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar the standard deviation or the standard error of the mean.
It is OK report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification: Details about hardware and an estimate of the total computational capacity used are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
17

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://meurrjj: httppiacc//tiiiieuuuu.eeeeeeeel
Answer: [Yes]
Justification: We are obliging to the Neurips Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
(o0)) The authors should make sure preserve anonymity (e.g., if there is special consideration
due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, in the appendix, we highlighted the potential positive and negative societal impacts of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
18

---

Question: Are the creators or original owners of assets (e.g,, code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we acknowledge all the code, dataset, and algorithms used through this paper with their original contributors as citation or direct mention.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, if possible, include a URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use the package should be provided. For popular datasets, paperswi thcode com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, as mentioned before all the code, models, and synthetic datasets will be made publicly available upon publication.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code//modll as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: We did not perform any crowd sourced experiment with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants. whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly
state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage LLMs if it is an important, original, or non- standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration not required. Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer our LLM policy http::///aaaaeeecc//ooofeeeeeees for what should or should not be described.
20

---

Appendix
A
Summary of SOTA methods
Table 3 summmrrzedd recent methodologics for synthetic FR dataset generation. Here the Generation Methodology refers which of the main methods (i.e., Diffusion, GAN, 3DMM, ... were used to generate synthetic data. Auxiliary Networks (Aux) refers to the use of additional models (e.g.. age estimators, face parsers) datasets during synthetic data generation. The last column, FR, indicates whether a strong pre-trained FR backbone, separate from the dataset used for training, was employed
not.
Method
Year
Generation Methodology
Aux
FR
SynFace
37
2021
3DMM & GAN
Y
Y
DigiFace
M
3
2023
3D-Rendering
Y
N
DCFace
25
2023
Diffusion
Y
Y
IDiffFace
5
2023
Diffusion
N
Y
GAe
32
2023
GAN/Diffusion
Y
Y
RealDigiFace
38
2024
GAN/Diffusion
Y
NNN
[55)
2024
Diffusion
Y
Cemilaaee
49
2024
Diffusion
N
Y
Ours
Diffusion
N
N
Table 3: State-of-the-art Synthetic Face Recognition (SFR) dataset generation methods are compared based two criteria: the use of Auxiliary Networks (Aux) and External Face Recognition (FR) Systems, Aux indicates whether auxiliary networks are utilized, with Y representing "Yes" and N representing "No." Similarly, FR highlights the of external face recognition systems beyond those trained solely on the methodology' 's dataset, using the same Y/N notation.
B Original Datasets Dorig
Table summarizes the key statistics of CASIA-WehFace WebFace 160K, and the original Web- Face4M dataset. Notably, WebFace 160K was curated avoid long-tail distribution in the number of samples per identity, aligning its statistics more closely to equal presentation while differing from the CAeea.WebFace.
Name
n IDs
n²"
Min
25%
50%
75%
Max
Ceblae
10..K
490K
2
18
27
48
802
WebFace160K
10K
160K
11
13
16
19
24
WebFace4M
-066
4,,235
11
24
1497
Table 4: The middle part of the table presents the datasets used this paper as Dorig, n IDs and n" representing the number of IDs and real images. The Min and Max present the minimum and maximum number of samples per identity for the corresponding dataset. The number samples like 25%, 50%, and 75% percentiles is also provided.
C Experiment Details
C.1 Discriminator Training
In the Table the most important parameters for training our discriminative models are presented.
C.2
Generator Design Choices
Here we try to answer why we are using Diffusion Models and not different types of generators like
GANs 15 23 or VAES. Theoretically, both VAEs and Diffusion Models train a generator with a max- imum likelihood (ML) or ELBO objective; for detailed derivation, please see [26]. We chose to use a diffusion model primarily because the methodology is more mature, and there are stable empirical procedures for both training (c.g., SNR-based weighting for high-resolution images) and inference
21

---

(e.g., faster samplers like DPM-v3). The same can be said for Flow Matching [30]. More specifically. methods like Gaussian Flow Matching (used in Flux and SD3 can be directly formulated as diffusion model under v-prediction parameterization. The main difference lies with GANs, whose objective is not formulated as an ELBO or MI.. During our experimentation, we attempted to train a StyleGAN-hane from scratch on our datasets CCASTA-WebFace and WebFace I 600K), as no publicly available models were trained on these specific FR datasets, and we aimed to avoid any information leakage from external data like FFHQ. However, as it is well known, GANs are very difficult to train, and our training runs were divergent despite using the settings provided by the original authors. Furthermore, primary concern with GANs is mode collapse. This makes them unfavorable choice for our goal, which to explore out-of-distribute generation. This is especially important for long- taig-tailed datasets like by a GAN-based generator.
where modes in the tail would likely not be recovered
C.3
Why Grid Search?
C.4
Generator and Its Training
We trained two sizes of generatir, namely small and medium in (21). The training of the small- sized generator took about NVIDIA H100 GPU day for the generator to see 805M images in different noise levels with a batch size of 2048. For reaching the same number of training images for the medium-sized generator, took about 2 days with a batch size of 1024. We used an Exponential Moving Average (EMA) length of 10%. As observed literature (35], the EMA of model weights plays crucial role in the output quality of the Image Generators.
For sampling models did not employ any Classifier Free Guidance (CFG) [16]
C.5 Table Details
For the Table 11 we conditioned medium-sized generator which trained till it saw 805M images in different noise levels (~1500 Epochs). The conditions were set according to the four sets of values of the and B. This is done for fixed identity combination from the Lg for all of them. Later for each of these new conditions e* we generated 50 images. All other tables were reported from a medium-sized generator when they saw 335M training samples.
Table 5: Details of the Discriminator and its Training
Parameter Name
Discriminator Type
Discriminator Type
Network type
ResNet 50
ResNet 50 AdaFace
Marin Loss
AdaFace
Batch Size
192
512
GPU Namber
Gradient Act Step
(For every training step)
N/A
GPU Type
Nvidia RTX 3090 Ti
Nvidia 11100
Precision of Floating Point Operations
High
High
Matrix Multiplication Precision
High
High
Optimizer Type
SGD
SGD
Weight Decay
0.0005
0.0005
Learning Rate
WarmUp Epoch
Number of Epochs
26
26
LR Scheduler
Step
Step
I.R Milestanes
[12, 24, 26]
[12, 24, 26] 0.1
LR Lambda
0.1
Ingut Dimension
112 112
112 x112 RGB Images 512
Input Type
RGB images 512
Output Dimension
Seed
41,2048, 10 (In some models) 41,2048
C.6 Training time breakdown
The proposed method adds a non-trivial one-time training cost, but this is amortized as yields a model that is both more accurate and more efficient inference.
We present cost breakdown below. Augmenting the data lets the smaller IR-50 backbone outperform the much larger IR-101 model ( 1.9xFLOPs and 1. 1.7xparameters) trained on the original data Table
22

---

Crucially, our final model retains the low inference cost of the IR50 backbone while outperforming the IR101 model, which is vital for real-world deployment where cumulative inference costs quickly surpass the one-time training expense.
Table 6: Training times of IR50/IR based discriminators on Dorig or Dorig +Dang datasets next to generator's s training time
Train Generator
Train IRS0 Dork
Train IRS0 Dorks Dauk (Ours) Train IR101 Dorig
GPU type
Ix H100
4x 3090Ti
4x 3090Ti
4x 3090Ti
Wall time (h)
42.2
2.54
Average perf
N/A
27.42 + 0.92
32.63 = 2.20
27.24 + 1.07
Variances are calculated as the pooled standard deviation from the results reported in Table 1. This demonstrates a favorable trade-off: we accept a higher, fixed training cost to produce a superior model that is cheaper to deploy.
D FR Benchmark Details
The full tables are presented in this section. Detailed results for the High- Righ-Quuaityy benchmarks are presented in Table 7 Results for more thresholds set by various FPRs for IJB-B/C are presented in Table 8 and Table 8 respectively.
Table 7: Comparison of the FRayn training (upper part), FRyeal training (middle), and FRuux training (bottom) using CASIA- -ebeace and our WebFace 10K, when the models are evaluated in terms of accuracy against standard FR benchmarks, namely LFW, CFPFP, CPLFW, AgeDB and CALFW with their corresponding protocols. Here n² and n" depict the number of Synthetic and Real Images respectively and Aux depicts whether the method for generating the dataset uses an auxiliary information network for generating their datasets (Y) not (N). the denotes network trained IR101 if not the model trained using the IR50.
Method/Data
Aux
n²
LFW
CFP-FF
CPLFW
AgeDB
CALFW
Avg
DMgiFaceIM
N/A
1.22M
42.43.00
74.64.0.
82.57.0
75.7240
69.48-133
78.974044
RealDigiFace
Y
1.20M
93
76.95
85.47.0.5
77.57
72.82a059
81.34anm2
IDiff-face
Y
1.2M
97. a5
77.07
80. 28000
87. 26
81.15
84.68anns
DCFace
Y
1.2M
98.77
84. 330.35
92.52mum 91.
91.21
91.56aum
Dong (Ours)
N
0.6M
98.38 18
83.35.0.12
87. 64.6..
89.6410 29
84.88.0.5
88. 78
Dempo (Ours)
N
0.6M
98. 602002
85.26s9.14
91. 13
90.54am 16
$7.69
90.64
CASIA WebFace
NIA
0
0.5M
99. 02
88.97a0.27
96.35.006
93.07
93.34a0.1
94,2 1a.2
CASIA- Sebba-- f
N/A
0
0.5M
89.92s0.12
97.06gu
93.54
94,33am113
862u0777
Difff-face
Y
1.2M
0.5M
99.53
89, 19..92
96.91
93.6 93.64au
94,28
94.86
DCFace
Y
0.5M
0.5M
99. .0.0
89, 89.44
96.67.0.1
93.82.u04
94.24an.15
94.72.0.
D (Ours)
N
0.5M
0.5M
99 20
89.96.a.07
03.40a0
93.74xm.02
94 64.602
WebFacel 60K
N/A
0.16M
87.99
93.95.a
90.78-0.9
92.91
WebFace16008
N/A
0
0.16M
98.97
87.54.0.06
93.40
92.55sam2
90.01.004
92.50amm
Dame (Ours)
N
0.6M
0.16M
99.39
89.56.a.0e
95,8420
93.60sn
92.47
94. 17200s
E
Mixing Effect
In Table 10. by setting the original dataset to CASIA-WebFace the effect of increasing the number of samples in our augmented dataset using (a, B) (0.7, 0.7) weights is shown. On average, adding more classes (#Class) and samples per class (#Sample) improves the performance of the final discriminative model. The performance eventually decreases as more samples are added per class. We hypothesize that this is due to the similarity of images generated under the new conditions, c, when sampling GZZ c) multiple times. This reduces the intra-class variability necessary for training effective discriminator. We also observe that we should add an appropriate number of the augmentation dataset (i.e,, comparing 10k x 5 to without any augmentation) for the final performance to be better than the discriminator trained on the original dataset.
23

---

Table 8: Comparison of the FReyn training, FReeat training, and FRmix training, when the models are evaluated against IJB-B with thresholds set by various FPRs in terms of TAR. Here n² and n' depict the number of Symthetic and Real Images respectively and Aux depicts whether the method for generating the dataset uses an auxiliary information network for generating their datasets (Y) or not (N), the i denotes network trained on IRI01 if not the model trained with the IR50. The numbers under columns labeled like B- 6-6 indicate TAR for IJB- at FPR of 1e-6.
Method/Data
Aux
n²
n²'
B-le-6
B-le-5
B-le-4
B-le-3
B-0.01
B-0.1
Avg
DigiFace IM
N/A
1.22M
15.31-00m
29
47
59.8 89.89
76.62.o
91.01-0
52.66-0.7
RealDigiFace
Y
1,20M
0
21.37
39. 14,14mmm.
52.61am00
67.684071
81.30
93.15
L952
Diil-faae
1.2M
26.81=2221
50.08
64.58
77.192041
88.27
95.90aus
67.15
DCFace
Y
1.2M
0
22. 32.48
47. 17.8446 to
73.20=253
86. 1
93.55+
97.5 27
20.12:2.1
Drmn (Ours)
0.6M
29.40-134
84,54.a.0
82.95-oms 30.93-035
91.67
97. 97.05-004
71.09-a11
Dymare (Ours)
N
0.6M
15.7
45.97
23.05.n
85. 14.54
93.52
97.88 97.82.uux
68.60-1.0
CASIA-WebFace
N/A
0
0.5M
1.02:026
5.06s1.00
50,37e4.0
87. 13:03
95, 95.36+9
98. 04
56,22-0.3
CASIA WehFace
N/A
0
0.5M
0.74
3.94-182
49.30-575
88.42.00e
96.7840
98.44.n
56.10-1.m
IDiff-face
Y
1.2M
0.5M
0.89
5.80gn
54,76=231
88. 88.33
96.0290
98,59
57.40
DCFace
0.5M
0.5M
0,26
35.62-7.89
84. 84.30+3.5
95, 10
98.36-n
52.54-3.0
DJUR (Ours)
N
0.5M
0.5M
2.6
15.74.3.0
63.67-1.48
89.19.
95 78.0m
28.51
60.92.1.:
WobFace160K
N/A
0
0.16M
12.113
12.18a0.1
82.96an20
90.37a004
35.6620.1
98.75aum
78.67au4
WobFace160K
N/A
0
0.16M
74,10s0 34.84
24
84,57z0.1
91.57
96.09
98.87=nu
80.01=024
D= (Ours)
0.6M
0.16M
36.62a8m
38.32s0
87.65a0
93.34
12
99.01amm
81.97am16
Table 9: Comparison of the FRaya training. FRzeal training, and FRimix training, when the models are evaluated against IJB-C with thresholds set by various FPRs in terms of TAR. Here n and n² depict the number of Synthetic and Real Images respectively and Aux depicts whether the method for generating the dataset uses an auxiliary information network for generating their datasets (Y) or not (N). the i denotes network trained on IR101 if not the model trained with the IR50. The numbers under columns labeled like B- indicate TAR for JJB-C at FPR of le-6.
Method/Data
Aux
x²'
n
C-Ie-6
C-
C-Ie4
C-Ie33
C.0.01
C-O.1
Avg
DM
NIA
1.22M
0
26. m
36.34xo0
49.98ua.5
65. 15.
80.21
92. 1s
58.37
ReaiDigiFace
Y
1.20M
0
36. 18
45.55uass
58.63a0.9
72.06
84.77a039
94.57
65.29
iDiff-face
41.75.1.4
51.9340w
65.01
78. 75.25
89.41 19
m6
70.48.0 17
DCFace
Y
1.2M
D
35.274 1978
58.22
77.51a2
88 S6.amm
94.81amm
98.06um
75 75.46s
D (Ours)
0.6M
45.15a1.04
61.52a347
74.12a0.3
85.09
93.01
97.64aa
76.09
Dimee (Ours)
N
0.6M
31.54
58.61:3.0
1
88. S.0nn
94.79 94.79g0.m
98. 17
74.9 94.661 82
CASIA- CASIA-WehFace
N/A
0.5M
0.7340.9
5.3741
56.76-777
89 05
96. 16
98.61.am
57.84
CASIA Wehflace
N/A
0.5M
0.3840as
3.9241.
55.21
90.42xam0
96.55
98.69um m
57.53
IDiff face
Y
1.2M
0.5M
0.70aea
7.46.z1e
57.43au17
89 89.89uam1
96.63aom
98.77am
58.48a1.
DCFace D (Ours)
Y
0.5M
0.5M
0,18amm
1.54a0.5
38.17
88.182152
95 95.8822.42
98. 98.59a
53.42a2.1
0.5M
0.5M
4.36
18.5843.9
67.8522.18
92.12
96.57
62.88s135 78
WebFace160KK
NIA
~00.16M
70.37au15
78.81
86.4550.11
62.68smm
96 92.52a
99.02an.11
WebFace
N/A
16M
72.56
81.26e911
88.27
93. 33.55
97.02e0.0
99. 92.02.0
88 68.660000
D (Ours)
~0..MM
~-0.16M
78 is
85,02e011
90.87s0m
94.98sam
97.55
99.23sa.
91.04ao
Table 10: Effect of mixing different numbers of classes (#Class) and samples per class (#Sample) with the original data, CASIA- WebFace. For TinyFace Rank-1 and Rank-5 verification accuracies are presented as TR1 I and TR5 respectively. The numbers under columns labeled like C/B-le-6 indicate
TAR for IJB-C/B FPR of 1e-6.
Syn #Class x #Sample
n²
B-le-6
B-le-5
C-1e-6
C-le-5
TRI
TRS
0.5M
1.16
5.61+1.6
0.83=..10
5.86+1.11
=8.01
63.47
Ours (5k x 5)
0.5M
0.8520.06
5. 5.60aa.4
0.65.0.08
6.70.0.97
58. 19.1000
63.48s0.0
Ours (5k x 20)
0.5M
1,0820.16
5.81
0.84
6.88
57.50ao.13
+3.07
Ours (5k x 50)
0.5M
0.63a0.23
4.56s041
0.46#0.10
6.55
27.39
62.55
Ours (10K x 5)
0.5M
0. 0.7720.08
1.40xo.14
0.61
4.6910.26
38.3020.28
63. 63.2810.30
Ours (10K 20)
0.5M
1.29
8.21
1.4320.22
9.6711.01
58.01 20.50
63.00u0m1
Ours (10K 50)
0.5M
0.62
4.291027
0.6420.10
5.9820.00
62.77.0.88 10.51
F
Downstream Performance vs Metrics in Generative Models
In this section, we examine whether there is a correlation between common metrics for evaluating generative models and the discriminator's 's performance when trained on our augmented dataset. We studied the FD [5] Precision/Recall [42 28 and Coverage [3 which is usually used to quantify the performance of the Generative Models. Calculation of these metrics requires the projection
24

---

of the images into meaningful feature spaces. For feature extraction, we consider two backbones, Inception- V3 50 and DINOv2 [36] which the latter shown effective for evaluating diffusion models 48 Both these models were trained using the ImageNet [4 in supervised and semi- semi-supervised manner respectively. Experiments were performed by randomly selecting 100, 000 images of both CASIA-WebFace (as the source distribution) and our generated images by the value of a and 3 using algorithm 1 (i.e., the same settings as presented in the Seection 4). We are reporting four versions of our genenateeddd augmentation using medium-sized generator when it sees 184M, 335M, 603M, and (M for Million). For each of the classes generated from these models, we selected 20 samples, based on the previous observation in Table 10 Later by mixing the selected images with the original CASIA- -WebFcee we train FR for each of them and report the average accuracies for different thresholds in the IJB-C (i.e, similar to Avg column in the Table 9. Figure 6 and Figure 7 are showing mentioned metrics for Inception-V3 and DINOv2 feature extractor respectively. We observe clear correlation between the metrics used to evaluate generative models and the performance of a downstream task. When comparing Daug to Dorig for FD, higher FD (i.e., distinguishable Dauk images) should enhance discriminator performance, but that wasn'l It observed here. This holds when we are augmenting the dataset for training the generator and discriminator with the Dorig, This highlights the need to develop new evaluation metrics as a proxy.
805M training samples in different noise scales of the original CASIA-WebFaae
G Effectiveness of Grid Search
We also study the effectiveness of our proposed method in alggorii I which tries to find the suitable condition weights, a, and B. We compare with four sets values:
Rand: and 3 were selected randomly for 10, 000 mixture of identities from the set of
f0.1, 0.3, 0.5, 0.7, 0.9, 1.0, 1.1).
Half: a and 3 set to 0.5 for all 10, 000 random mixture of identities selected from Lg..
Full: a and 8 set to 1 for all 10, 000 random mixture of identities selected from L..
Half++: a and B set to 0.7 according to the algorithm I for the generator and discriminator trained on -eebace dataset. This done for all 10, 000 random mixture of identities
selected from Ih,
The results for this are shown in the Table 11 generated using and 3 values with higher yn total are performing better.
We observe on almost all of the benchmarks the Daug
Table 11: Effectiveness of our weighting procedure (W/ Half++) in comparison to (W/ Random) or
when putting the conditions to 0.5 (W/ Half) and when setting the condition signal to 1 (W/ Full). Best in bold, second best, underlined. TRI represents the Rank-1 accuracy for the TinyFace benchmark. The numbers under columns labeled like C/B-le-6 indicate TAR for JJB-C/B at FPR of 1c-6
C Weight Method
B-le-6
B-le-5
C-1e-6
C-1e-5
TRI
m Rotal
W/ Half
0.55
0
8.524561
27.742687
11.59
35.6915.23
46.4220.60
1.48
W/ Full
0..5
0
17.6340.08
32.47=0.7
24.30=a30
37.45H0.22
45. 45.08=0.17
1.53
W/ Random
0.55
0
24,47+1.23
39.83-1.08
30.79+1.99
44.33+0.8
49.34
N/A
W/ Half++
0.5M
0
25.44:0.19
46.20s0.12
39.6620.3
51.47
47.95
1.58
H Verifying the driving Hypothesis
As shown in Figure I. introducing a new class using algorithm 2 aims to augment the original dataset with novel mix of source classes. This approach enforces the network to improve the compactness and separability of class representations. By requiring the network to distinguish the mixed class from its source classes, we strengthen its discriminative power. To validate this approach, conducted experiments two models, BBsseline and f J
pAugGen trained before and after incorporating AugGen samples, respectively, and evaluated their performance using the following metrics:
1. Mean absolute Inter-Class Similarity of samples across all mixed classes. After applying AugGen, we expect that the average similarity of samples from different classes become lower, correspond- ing to higher fours in Figure
25

---

Correlation of average IJB-C acc and fd
Correlation of average IJB-C acc and recall
58.5
58.5
58.0
58.0
avg. jibe
57.5
57.5
56.5
56.5
M0184M
MD184M
56.0
M0335M
56.0
MD335M
MD603M
MD603M
MDB05M
300
400
450
0.36
0.38
dd_dinov2
recall, dinov2
(a) FD
(b) Recall
Correlation of average JJB-C acc and coverage
Correlation average
precision
59.0
59.0
58.5
58.5
avg. 1ibc
57.5
57.5
57.0
57.0
56.0
56.0
M0603M
MD6033M
55.5
55.5
MDB0SM
MDB0SM
0.55
0.60
0.65
0.70
0.75
0.76
0.78
0.80
0.82
0.84
0.86
coverage, dinov2
precision dinov2
(c) Coverage
(d) Precision
Figure 6: Correlation between the FD, Recall, Coverage, and Precision for the generated dataset by comparing it with the features of CASIA-WebFaae using the DINOv2 extractor.
Mean and standard deviation (i.e., std) of Intra-Class Similarity of samples of all mixed classes, (i.e., M-Intra and S-Intra in Table 12 This should indicate if the generated samples for each class, cause the model to boost its compactness.
These metrics presented in the Table 12 After adding the AugGen samples, we are observing lower M-Inter which reflects that the similarity of the samples between different classes decreased. We are also observing the M-Intra increase reflecting that the networks perceive the images of the same class as more similar.
Table 12: Comparison of models trained with and without AugGen samples: M-Inter represents interclass similarity, indicating class separation, while M-Intra and S-Intra measure the mean and standard deviation of intraclass similarity, reflecting class compactness.
Dataset/Method
M-Inter(() M-Intra(t)
S-Inrra(L)
Baseline AugGeen
0.16M
0.0672
0.49065
0.13499
0.2M 0.16M
0.0664
0.54917
0.12807
26

---

Correlation of average IJB-C acc and fd
Correlation average IJB-C acc and recall
58.5
58.5
58.0
58.0
avg. jibe
57.5
57.5
56.5
56.5
M0184M
M0184M
56.0
56.0
M0335M
M0335M
MD603M
MD603M
MD805M
22.5
25.0
0.34
fd dd_ineption
recall, inceppion
(a) FD
(b) Recall
Correlation average JBB-C acc and
Correlation average B--C
precision
59.0
59.0
58.5
58.5
avg. 1jbc
57.5
57.5
avg_jk
57.0
57.0
MO1884
M0184M
56.0
56.0
MD603M
MD60.3M
55.5
55.5
MDB0SM
MDB05M
0.600
0.625
0.650
0.675 0.700
0.725
0.750
0.600 0.625 0.650 0.675 0.700 0.725 0.750 precision inception
coverage coverage_inception
precision_incettoon
(c) Coverage
(d) Precision
Figure 7: Correlation between the FD, Recall, Coverage, and Precision for the generated dataset by comparing it with the features of CASIA-Webrac using Inception-v3 extractor.
I More Samples of Daug
In the following figures, you can find more examples of generated images for Small and Medium- sized generators and also trained for more steps. By comparing Figure 8 (generated result from small-sized generator trained when it sees 335M images (~ 700 Epochs), S335M, as the optimization of score-function, involves multiple noise levels of images). Figure 9 (M335M) and Figure 13 producing better images, but training for
(M805M) generally observe that larger generators more steps does not necessarily translate to better image quality. This is especially important as we exploring the out-of-distribution generation capabilities of image generator.
Reproducibility.
code for the discriminative and generative models, along with the generated datasets and trained models, will be publicly available for reproducibility.
27

---

THE
TH
Figure 8: Small-sized generator trained till it sees 335M images in different noise levels (~~700 Epochs). From left right, the first column is variations of random ID, in the, Dorig, the second column is the recreation of the same ID in the first column using the generator when we set the corresponding conditions The last two columns are the same but for different IDs and the middle column representing the Daug sample.
28

---

CHE
TH
Figure 9: Medium-sized generator trained till it sees 335M images in different noise levels (~~700 Epochs). From left right, the first column is variations of random ID, in the, Dorig, the second column is the recreation of the same ID in the first column using the generator when we set the corresponding conditions The last two columns are the same but for different IDs and the middle column representing the Daug sample.
29

---

CHE
TH
Figure 10: Medium-sized generator trained till it sees 805M images in different noise levels 1~1500 Epochs). From left right, the first column is variations column is the recreation of the same ID in the first column using the generator when we set the corresponding conditions The last two columns are the same but for different IDs and the middle column representing the Daug sample.
random ID, in the, Dorig, the second
30

---

Figure 11: Medium-sized generator trained for till it sees 335M images in different noise levels (~700 Epochs) for different IDs. From left to right, the first column is variations of a random ID, 1, in the, Dorig, the second column is the recreation of the same ID in the first column using the generator when we the corresponding conditions different IDs and the middle column representing the Daug sample.
The last two columns are the same but for
31

---

(a) IDs 115 and 2668
(b) IDs 760 and 1297
Figure 12: Samples from small-sized pixel space EDM generator trained on WebFacel66K for about 31M training steps (~2000 Epochs). From left to right, the first column is variations of a random ID. 1, in the, Dorig, the second column is the recreation of the same ID the first column using the generator when we set the corresponding conditions to 1. The last two columns are the same but for different IDs and the middle column represents the Daug sample.
32

---

(a) IDs 2299 and 8574
(b) IDs 7858 and 8434
Figure 13: Samples from small-sized pixel space EDM generator trained WebFacel66K for about 31M training steps (~2000 Epochs). From left to right, the first column is variations of a random ID. 1, in the, Dorig, the second column is the recreation of the same ID the first column using the generator when we set the corresponding conditions to 1. The last two columns are the same but for different IDs and the middle column representing the Daug sample.
33

---

Impact Statement
In our approach, introduce novel technique that leverages generative models to further im- prove state-of-the-an (SOTA) facial recognition (FR) systems, as demonstrated publicly available medium-sized datasets. However, these same FR systems can inadvertently facilitate unauthorized identity preservation in deepfakes and other forms of fraudulent media when attackers mimic individ- uals without their consent.
While our primary objective is to address privacy concerns and informed consent in training FR systems, the resulting performance gains could also enhance deepfake quality.
34


Reviews:
[{"review_id": "pYy2VuFf3h", "reviewer": "Reviewer_Yi5o", "summary": "This paper proposes a method that leverages an existing face recognition dataset to train a generative model, which is then used to synthesize images for training the face recognition model. The approach involves using the one-hot label of the recognition model as a conditional input to the generator, and interpolating between existing training data to create face images of intermediate identities as new data. These synthesized samples are then used to train the face recognition model, leading to improved recognition performance.", "questions": "1.\tIf the cosine similarity of two human faces is small but the difference is huge, is the interpolated face still representative?\n2.\tIf we interpolate two faces that are very similar, will the resulting faces still be distinguishable?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths :\n1.\tThe paper proposes synthesizing new identities by interpolating between two existing identities based on the available data. This approach enables the generation of novel identities, and an algorithm is introduced to evaluate whether the generated data is suitable for recognition training.\n2.\tThe method utilizes one-hot labels as input to control image generation. This relies on the training of the original recognition model (M_ori) and uses the original data to train the generative model. Additionally, the one-hot representation allows for interpolation between two identities.\n3.\tThe experimental results are promising and demonstrate the effectiveness of the proposed approach.\n\n Weaknesses:\n1.\tThe paper uses one-hot vectors as conditional input, but it is unclear how interpolated faces are generated when values between two identities (i.e., numbers not seen during training) are used. The paper does not clearly explain whether there are additional constraints during training to address this issue.\n2.\tThe experiments are conducted on a dataset with strong training data. It is unclear whether the proposed method would still be effective when the dataset contains only a single image per identity. The effectiveness of privacy protection in such cases is also not discussed. Alternatively, as a data augmentation method, experiments should be conducted on a larger dataset (such as MS1M).\n3.\tAlthough new identities are synthesized, these identities are interpolated from real identities. This approach does not demonstrate that truly new identities are generated, and thus it is unclear whether privacy is actually protected.\n4.\tWhen optimizing αα and ββ in the paper, it is only performed on a subset of the identities. The paper does not explain how these identities are selected. The search space would be larger if all identities were considered, and more identity data could potentially be synthesized.\n5.\tIt is recommended to standardize the mathematical notation throughout the paper.", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns. And after reading reviews of the other reviewers, I am willing to raise my rating.", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and suggestions, and for the time invested in reviewing and improving our manuscript.\n\nWe highlight that the paper contains **appendices** that provide additional support for our claims and address most of the reviewer's questions. We included these additional results in the final version.\n\n-------------\n## Weakness 1 ##\n\n**One-hot conditoning.** We generate interpolated faces using soft conditioning vectors (i.e. interpolation between one-hot vectors). Formally, let $\\mathbf{c}$ be the condition vector and $\\mathbf{W}$ be the **learned** condition embedding matrix. The selection of an embedding is: $\\mathbf{c}^T \\mathbf{W}$. If we set $\\mathbf{c}$ as a soft vector such as $[0.7, 0, 0.7]$, the resulting condition vector is a linear combination: $0.7 \\times \\mathbf{W}_1 + 0.7 \\times \\mathbf{W}_3$.\n\n**Constraint on condition space.** Despite being trained end-to-end using only one-hot condition vectors and without explicit regularization on condition space, the learned condition embeddings exhibit meaningful structure. Our method effectively exploits this structure to generalize to unseen conditioning vectors.\n\n**We will include this clarification in the final version.**\n\n-------------\n## Weakness 2: Training dataset ##\n**Why CASIA-WebFace and WebFace160K?** We selected CASIA-WebFace to ensure a fair comparison with prior synthetic face generation methods (e.g., DCFace, VariFace, and Vec2Face), as these methods frequently use it as a training dataset. Our approach, which relies solely on CASIA-WebFace, notably surpasses these previous methods that typically incorporate additional datasets or sources of information. However, due to identified inconsistencies within CASIA-WebFace—detailed in our paper—we introduced WebFace160K, a carefully curated subset of WebFace4M. We employed this new dataset to further validate our method's effectiveness and recommend its adoption by the community.\n\n**Will it work with a single image per identity?** To effectively capture identity-preserving features, every face recognition training requires multiple images per identity. Our method trains both the generator and discriminator from scratch, making this requirement particularly crucial for learning meaningful identity representations—especially given the margin-based discriminator loss. Similarly, the generator’s condition space benefits from multiple samples per identity. We indirectly evaluated scenarios with few images per identity using CASIA-WebFace, which has a heavy-tailed distribution (Table 4), including identities with as few as two images. However, our approach may be less effective when strictly limited to a single image per identity, due to insufficient identity-specific variation. In fact, any face recognition training, especially those using margin-based losses, would suffer under such a strict constraint, as margin losses fundamentally rely on multiple examples per identity to enforce their key design principles.\n\n**Privacy protection.** We claim to mitigate privacy concerns primarily because, using a SOTA discriminator and our method, we can reduce the dataset size required to train a SOTA discriminator (with same performance) by up to $40$%  (Table 2), thereby mitigating privacy risks associated with collecting large datasets.\n\n\n**Why not MS1M or larger datasets?**\nPlease note that large datasets like MS1M have been **retracted** due to privacy concerns, as they were often collected without consent from the identities [1R1]. Thus, for legal and reproducibility reasons, we refrained from using this dataset, despite its continued online availability. Additionally, while we validated our method on another dataset-a subset of WebFace4M- (for first time in Synthetic Dataset Generation community), training a generator directly on the full-scale dataset was beyond our available computational resources.\n\n-------------\n## Weakness 3: Novel identities or hard samples? ##\n\nThis is an excellent question. Our goal is to generate synthetic samples that are maximally beneficial for training the discriminator. While the synthesized identities have some similarities to their sources, we have demonstrated qualitatively that they represent distinct identities. By design, the generated samples are similar to their source classes. This similarity makes them effective **hard samples** for the discriminator. The discriminator must not only distinguish between different source classes but also differentiate between real images and our generated samples, which share many cues with their sources yet are distinct. \n\nOur results show that, with a limited amount of data, our approach achieves a performance boost equivalent to adding **1.7x** the original dataset's volume (Table 2). This means one would need to add 70% more real data to observe the same performance gain achieved by our method. thereby, as mentioned previously, mitigating privacy risks associated with collecting large datasets.\n\n-------------\n## Weakness 4: Selection process for finding $\\alpha$ and $\\beta$ ##\nTo determine the optimal $\\alpha$ and $\\beta$, we randomly selected 1,000 pairs of identities from the $\\binom{N}{2}$ possible combinations [L246], where $N$ is the number of classes in the source dataset (e.g., CASIA-WebFace or our WebFace160K). We tested different random sets of pairs and observed a negligible impact on the optimal values, which remained $(\\alpha=0.7, \\beta=0.7)$ with a step resolution of 0.1. We added a short ablation study for this process in the final version.\n\n-------------\n## Weakness 5: Notations ##\nThanks! We fixed the notations to be coherent throughout the paper.\n\n\n-------------\n## Question 1: Representative face images in most scenarios? ##\nQualitatively and quantitatively (please refer to the next question), we observe that the generated images are plausible both when the source identities are similar (high cosine similarity) and when they are very different, such as having distinct ethnicities or ages (low cosine similarity). We provide examples for both cases in Figures 12 and 13 of our appendix. For instance, in Figure 13a, the source identities differ significantly in gender, ethnicity, and age. Nevertheless, the AugGen samples (third column) are not only plausible faces but also maintain strong identity consistency down the column, indicating a stable, novel identity has been generated.\n\n-------------\n## Question 2: Similar and different face interpolation ##\nThis is a very interesting question. Through extensive exploration of identity selection strategies, we found that mixing identities that are **further apart** in the discriminator's embedding space yields a greater performance improvement. When we perform mixing (using `Alg. 2`) on identities that are already similar, the performance gain is lower compared to when we select identities that are highly distinguishable from each other. We have summarized these results in the following table:\n\n| Dataset | Source ID Selection Strategy | $n^r$ | $n^s$| B-1e-5 | C-1e-6 | C-1e-5 | AgeDB | CALFW | TR5 |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| WebFace160K | Baseline (No Augmentation) | 0.16M | - | 72.34 | 70.59 | 78.11 | 89.75 | 92.11 | 67.06 |\n| WebFace160K | Mix **Similar** IDs (Closer in Embedding Space) | 0.16M | 0.2M | 73.11 | 71.86 | 81.00 | 91.85 | 92.81 | 66.82 |\n| WebFace160K | Mix **Dissimilar** IDs (Distant in Embedding Space) | 0.16M | 0.2M | **77.45** | **78.60** | **84.03** | **92.95** | **93.58** | **67.46** |\n\n\n**We propose to include a more detailed analysis of this finding in the appendix.**\n\n\n-------------\n[1R1] https://www.vice.com/en/article/microsoft-deleted-a-facial-recognition-database-but-its-not-dead/"}, {"review_id": "TBBDP8lfli", "reviewer": "Reviewer_pBfk", "summary": "This paper presents AugGen, a self-contained synthetic enhancement technique. By strategically sampling from a class-conditional generative model trained exclusively on the target face recognition dataset, it generates synthetic samples without external resources, effectively boosting the model's discriminative ability. Evaluations on 8 FR benchmarks (including IJB-C and IJB-B) show that this approach achieves a 1-12% performance improvement, outperforming models trained solely on real data and state-of-the-art synthetic data generation methods. It uses less real data, and its gains often exceed those from architectural modifications, demonstrating the value of synthetic enhancement in data-constrained scenarios.", "questions": "Please see the Weakness part.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n1. The paper proposes a self-contained synthetic augmentation technique called AugGen. The mixed-class samples generated by it can significantly enhance the performance of FR benchmarks, fully demonstrating its effectiveness.\n2. The writing and structural organization of the paper are good, and the comparative experiments are comprehensive.\n\n**Weakness:**\n1. As shown in Figure 2, the paper involves multiple stages, including Generator training and data generation. The overhead brought by these processes should be considered in practical application scenarios. It is suggested to supplement information on the training time and resource consumption of different stages.\n2. State-of-the-art methods like GPT-4O and Qwen-VLo are capable of generating highly realistic face data. Could these be used to enhance the performance of FR? It would be beneficial to include some comparative analysis on this.\n3. Table 2 in the paper shows that more forged samples are needed to achieve results comparable to real samples, but there is a lack of in-depth analysis on the reasons. Is it due to the interference of mixed identities in synthetic data, or should some strategies be introduced in training to handle real and synthetic data differently?", "was_revised": true, "final_justification": "The detailed rebuttal has addressed my concerns. I maintain my original score of \"4: BA\".", "rebuttal": "We sincerely thank the reviewer for their thoughtful feedback and valuable suggestions, which have greatly helped us improve the manuscript.\n\n\n-------------\n## Weakness 1: Compute cost ##\nWe propose to add a discussion to the paper about this important aspect. The proposed method adds a non-trivial one-time training cost, but this is amortized as it yields a model that is both more accurate and more efficient at inference.\n\nWe present a cost breakdown below. Augmenting the data lets the smaller IR-50 backbone outperform the much larger IR-101 model (~1.9×FLOPs and ~1.7×parameters) trained on the original data (Table. 1.) \nCrucially, our final model retains the low inference cost of the IR50 backbone while outperforming the IR101 model, which is vital for real-world deployment where cumulative inference costs quickly surpass the one-time training expense [2R3].\n\n| | Train Generator | Train IR50 on $D^{\\mathrm{orig}}$ | Train IR50 on $D^{\\mathrm{orig}} + D^{\\mathrm{aug}}$ (Ours) | Train IR101 on $D^{\\mathrm{orig}}$ |\n| :--- | :--- | :--- | :--- | :--- |\n| **GPU type** | 1x H100 | 4x 3090Ti | 4x 3090Ti | 4x 3090Ti |\n| **Wall time (h)** | 42.2 | 2.54 | 4.1 | 5.6 |\n| **Average perf.** | N/A | 27.42 ± 0.92| **32.63 ± 2.20** | 27.24 ± 1.07 |\n\n*Variances are calculated as the pooled standard deviation from the results reported in Table 1.*\n\nThis demonstrates a favorable trade-off: we accept a higher, fixed training cost to produce a superior model that is cheaper to deploy. We will release our source code and trained models to further mitigate this cost for the community.\n\n\n-------------\n## Weakness 2: Usage of generalist models like GPT-4o, FLUX, Gemini for dataset generation ##\nThis is an important aspect that we propose to clarify in the introduction section. State-of-the-art generalist models may generate higher-quality images, but they come with significant constraints (see below). This is the key motivation for our development of a self-contained method (independent of external datasets, commercial APIs, or existing models).\n\n- **License restrictions.** Generalist models like GPT-4o and Gemini have restrictive usage policies [2R1, 2R2] that prevent their use in sensitive or commercial applications like face recognition.\n\n- **Unknown training data & consent issues.** Many generalist models are trained on private data, where subject consent cannot be guaranteed. This poses a major concern for face recognition systems, medical applications, and other sensitive use cases—an issue our work explicitly avoids. \n\n\n-------------\n## Weakness 3: Why is synthetic data less sample-efficient than real data? ##\nThis is an excellent question. Our approach **does not** claim to introduce entirely **new information**; rather, it strategically re-uses information from the original dataset to find hard samples the discriminator may have missed during initial training. We offer two primary reasons why more synthetic samples are beneficial.\n\n1.  **Recovering Missed Information.** The generator's role is to explore the data distribution and produce hard-to-classify samples. Presenting a larger volume of these generated samples naturally increases the probability of exposing the discriminator to these challenging examples, thereby improving its robustness.\n\n2.  **Generator Imperfection.** As shown by the performance of a model trained only on reproduced data ($D^{\\mathrm{repro}}$) versus the original data ($D^{\\mathrm{orig}}$) (i.e., Table.1) our generator does not perfectly replicate the real data distribution. This slight imperfection means each synthetic sample may carry less informational value than a real one, necessitating a larger quantity of augmented data to inject the same amount of useful information.\n\nUltimately, our results demonstrate that even state-of-the-art discriminator models (usually used in face recognition), which are already highly optimized, can be significantly improved without adding any new real datasets or pre-trained models.\n\n-------------\n[2R1] https://openai.com/policies/creating-images-and-videos-in-line-with-our-policies\n\n[2R2] https://ai.google.dev/gemini-api/terms#use-restrictions\n\n[2R3] https://www.breakingviews.com/columns/big-view/ai-boom-is-infrastructure-masquerading-software-2025-07-23/"}, {"review_id": "LSIB6vJhiC", "reviewer": "Reviewer_swWZ", "summary": "This paper presents AugGen, a simple yet effective framework for data augmentation using synthetic images generated from a conditional diffusion model. The method aims to enhance discriminative models, particularly in face recognition, by introducing pseudo-novel classes constructed through weighted linear combinations of one-hot class condition vectors. The generated images are filtered using a feature-space-based criterion that maximizes inter-class dissimilarity and intra-class consistency. These selected synthetic samples are then mixed with real data to form an augmented training set. The approach is validated across a wide range of face recognition benchmarks. Extensive experiments show that AugGen achieves performance improvements comparable to architectural modifications or real-data expansion, highlighting its practical value in low-resource scenarios.", "questions": "Please refer to the Weaknesses section for additional context. Below are three questions and suggestions that target specific aspects of the current design and presentation, which, if addressed, could improve the clarity and impact of the work:\n\n1. Can the heuristic search over (\\alpha, \\beta) be made learnable or end-to-end?\nThe proposed framework relies on manually enumerating mixing weights and selecting candidates based on a non-differentiable score. Have the authors considered making the weight selection process learnable, for example through a soft parameterization of the class mixing vector or meta-learning over mixing coefficients? A demonstration of a differentiable or learnable variant of the sampling mechanism could significantly enhance the method’s generality and scalability, and would strengthen the contribution.\n\t\n2. Is diffusion necessary for the framework, or could other generators be used?\nThe current method is built on conditional diffusion models, but the core idea—class mixing and sample selection—appears generator-agnostic. Could the authors comment on whether similar results would hold with GANs, VAEs, or pretrained generative models like StyleGAN? An ablation or discussion on the necessity of diffusion models versus alternative generators would clarify the scope and portability of the approach.\n\n3. Clarify presentation inconsistencies and formatting choices\nSome parts of the paper, particularly in the methodology section, suffer from overly long paragraphs and occasional typographical issues (e.g., “synthetic images.ss”). Additionally, the use of boldface for variables like \\mathbf{X} and \\mathbf{Z} is inconsistent and may confuse readers, especially since they do not appear to carry a distinct semantic meaning. Clarifying these presentation details would improve the clarity of the paper.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. Strong motivation with partially inconsistent presentation\nThe motivation behind this work is compelling and well-argued: the authors aim to enhance discriminative model performance by leveraging synthetic augmentation without relying on external models or auxiliary data. This is a relevant and timely objective, particularly in privacy-sensitive domains such as face recognition. \n\n2. Simple yet practical augmentation strategy\nThe paper proposes a lightweight, reproducible strategy for synthetic data augmentation by generating pseudo-novel classes through linear interpolation of class condition vectors in a conditional diffusion model. This approach avoids the use of external models or labelers, and instead relies solely on the internal structure of the training data and the generative model itself. While the concept of interpolating between classes is not entirely new, the authors effectively adapt it into a generative augmentation pipeline that is particularly well-suited for scenarios where acquiring or annotating real data is costly or constrained by privacy concerns. The fact that the method can be implemented without complex architecture changes or training objectives makes it attractive for deployment in real-world face recognition systems and potentially other classification settings.\n\n3. Strong empirical performance with extensive evaluation\nThe experimental results are a notable strength of the paper. The authors evaluate their method across a diverse set of face recognition benchmarks. These datasets span both high- and low-quality image domains, and the consistent improvements over baseline methods demonstrate the robustness of the proposed strategy.  Furthermore, the paper includes comprehensive ablation studies that analyze the effects of mixing ratios, pseudo-class selection, and augmentation volume.\n\nWeaknesses:\n1. Limited methodological novelty\nWhile the overall system is well-designed and thoughtfully implemented, the core components—namely, class interpolation through condition vector mixing and heuristic sample selection based on feature-space metrics—are primarily composed of existing ideas. The paper focuses on a clean and pragmatic integration of known techniques, rather than introducing new loss functions, training paradigms, or theoretical formulations. As such, its contribution lies more in empirical innovation and practical value than in conceptual novelty. \n\n2. Heuristic and non-differentiable design\nThe current pipeline depends on a manual, non-learnable search over class pairs and mixing coefficients (\\alpha, \\beta), followed by a scoring mechanism based on hand-crafted similarity and dissimilarity metrics. This makes the system less flexible for integration into end-to-end differentiable learning frameworks. Moreover, such a heuristic process may become inefficient or brittle when scaling to larger class sets or tasks requiring dynamic sample generation.\n\n3. Minor presentation issues\nAlthough the paper is generally clear in its motivation and organization, there are several presentational inconsistencies that hinder readability. Some paragraphs—especially in the methodology section—are excessively long and would benefit from better structural segmentation. In addition, there are typographical issues (e.g., “with synthetic images.ss”) and formatting inconsistencies, such as the unexplained use of boldface for variables like \\mathbf{X} and \\mathbf{Z}, despite no distinction from their standard forms. While these issues do not detract from the technical content, addressing them would improve clarity and overall polish.", "was_revised": true, "final_justification": "I appreciate the authors’ detailed reply. They clarified why they stuck with grid search and why diffusion models fit best here. It’s also good that they’ll fix the formatting issues. But I still feel the manual search limits the method’s flexibility, and there’s not a big leap in technical novelty. Overall, it’s a solid paper with useful insights, but my original concerns mean I’m keeping my initial score.", "rebuttal": "We sincerley thank the reviewer for the detailed feedback and suggestions, and also for the time and effort invested in reviewing and improving our manuscript.\n\n-------------\n## Weakness 1: Novelty ##\nWe agree that a primary contribution of our work lies in the novel system design and its empirical validation, rather than the invention of entirely new standalone components. Yet we believe that this constitutes a significant addition to the literature. Our work demonstrates that a carefully designed and pragmatic integration of existing techniques can surprisingly outperform more complex approaches. As we show in our response to Q1 (below), our simple approach surpasses a much more intricate, end-to-end guided alternative. We argue that demonstrating that simplicity can be more effective is a valuable contribution, especially for practical applications in a field like face recognition where reliability and reproducibility are paramount.\n\n\n-------------\n## Question 1/Weakness 2: Why grid search and heuristic? ##\nWe initially attempted to optimize $\\alpha$ and $\\beta$ with respect to our metric $m_{total}$ (which is differentiable via $f_{\\theta_{dis}}$), but the unstable gradients and the multi-step nature of diffusion inference made this approach diverge. We explored alternative techniques, including Classifier-Guidance [3R3] (using the discriminator to guide the generator) and GGDPM [3R2] but results were not promising.\n\nTherefore, we adopted a simple grid search, which is simple, stable, and generalizable across generator types (e.g., multi-step autoregressive models), where gradient-based guidance is often not applicable [3R6].\n\nTo demonstrate this, we present some parts of our results from experiments with classifier guidance, where we encourage the denoising process to produce samples that lie between certain classes in the discriminator's embedding space. The update rule is:\n$$ \\tilde S\\_\\theta = S\\_\\theta - \\lambda\\_t \\mathbf{g}\\_t, \\qquad \\mathbf{g}\\_t = \\nabla\\_{\\mathbf{x}\\_t} \\mathcal{L}(f\\_{\\theta\\_{dis}}(\\mathbf{x}\\_t),\\bar{\\mathbf{e}}) $$\nwhere $\\mathbf{g}\\_t$ is the gradient guiding the sample $\\mathbf{x}\\_t$ towards a target embedding $\\bar{\\mathbf{e}}$ via $f\\_{\\theta\\_{dis}}$. The rest of the notation follows Section 3.\nDespite some effort for hyperparameter tuning, we could not exceed the performance of our simpler `AugGen` approach. \n\n| Dataset | $n^r$ | $n^s$ | B-1e-5 | C-1e-6 | C-1e-5 | AgeDB | CALFW | TR5 |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| WebFace160K | 0.16M | – | 72.34 | 70.59 | 78.11 | 89.75 | 92.11 | 67.06 |\n| WebFace160K (AugGen) | 0.16M | 0.2M | **77.45** | **78.60** | **84.03** | **92.95** | **93.58** | **67.46** |\n| WebFace160K (AugGen **+ Classifier Guidance**) | 0.16M | 0.2M | 75.05 | 75.89 | 82.16 | 92.75 | 93.28 | 67.22 |\n\nAs the table shows, even with classifier guidance added, the performance did not surpass the baseline `AugGen`.\n\n### Optimizing via few-step distilled variants ###\nOne natural difficulty for end-to-end optimization is the multi-step inference process. We tested recent methods that distill generation into a few steps (e.g., Adversarial Score Identity Distillation [3R4, 3R5]). While distillation improved the performance of the discriminator trained the on reproduced data ($D^{\\mathrm{repro}}$), applying guidance or `AugGen` to the few-step distilled models resulted in a significant drop in image quality, likely due to mode collapse from the adversarial distillation loss (Please see the answer to the next question too.)\n\n**We propose to include these negative results and failure cases in our Appendix to guide future work.**\n\n\n-------------\n## Question 2: Can we use different generator types? ##\n\n### VAEs, Diffusion and Flows ###\nThis is a very good question. Theoretically, both VAEs and Diffusion Models train a generator with a maximum likelihood or ELBO objective; for a detailed derivation, please see [3R1]. We chose to use a diffusion model primarily because the methodology is more mature, and there are stable empirical procedures for both training (e.g., SNR-based weighting for high-resolution images) and inference (e.g., faster samplers like DPM-v3).\n\nThe same can be said for Flow Matching. More specifically, methods like Gaussian Flow Matching (used in Flux and SD3) can be directly formulated as a diffusion model under a v-prediction parameterization.\n\n### GANs ###\nThe main difference lies with GANs, whose objective is not formulated as an ELBO or ML. During our experimentation, we attempted to train StyleGANv3 from scratch on our datasets (CASIA-WebFace and WebFace160K), as no publicly available models were trained on these specific FR datasets, and we aimed to avoid any information leakage from external data like FFHQ. However, as it is well known, GANs are very difficult to train, and our training runs were divergent despite using the settings provided by the original authors. Furthermore, a primary concern with GANs is mode collapse. This makes them an unfavorable choice for our goal, which is to **explore out-of-distribution generation**. This is especially important for long-tailed datasets like CASIA-WebFace, where modes in the tail would likely not be recovered by a GAN-based generator.\n\n**We propose to include a discussion of these design choices in the Appendix**\n\n-------------\n## Question 3 / Weakness 3: Simplification of text ##\nThanks for spotting the typo [L126] and for the feedback. We reworked the presentation, especially Section 3, to improve readability.\n \nWe will clarify the notation in the paper to distinguish vectors, matrices, and higher-order tensors, consistent with the implementation.\n\n-------------\n[3R1] Kingma, Diederik, and Ruiqi Gao. \"Understanding diffusion objectives as the elbo with simple data augmentation.\" Advances in Neural Information Processing Systems 36 (2023): 65484-65516.\n\n[3R2] Guo, Yingqing, et al. \"Gradient guidance for diffusion models: An optimization perspective.\" Advances in Neural Information Processing Systems 37 (2024): 90736-90770.\n\n[3R3] Dhariwal, Prafulla, and Alexander Nichol. \"Diffusion models beat gans on image synthesis.\" Advances in neural information processing systems 34 (2021): 8780-8794.\n\n[3R4] Zhou, Mingyuan, et al. \"Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation.\" Forty-first International Conference on Machine Learning. 2024.\n\n[3R5] Zhou, Mingyuan, et al. \"Adversarial score identity distillation: Rapidly surpassing the teacher in one step.\" arXiv preprint arXiv:2410.14919 (2024).\n\n[3R6] Yu, Qihang, et al. \"Randomized autoregressive visual generation.\" arXiv preprint arXiv:2411.00776 (2024)"}]

Author rebuttal:

Meta-review: This paper addresses privacy and ethical challenges through synthetic data generation. The method creates new identities by interpolating between existing ones and uses one-hot labels to guide image generation. Experiments demonstrate that it outperforms models trained only on real data and other state-of-the-art synthetic data methods.

All reviewers recommend acceptance, and the paper is expected to have broad community interest. Therefore, it is recommended for acceptance.
Decision: Accept (poster)