Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
Authors: David Geissbühler
Hatef Otroshi Shahreza
Sébastien Marcel
Venue: ICML 2025 poster (2025)
Keywords: Brownian Identity Diffusion
Face Recognition
Latent Space
Synthetic Dataset
Primary area: social_aspects->privacy
TL;DR: We introduce a new method inspired by the physical motion of soft particles subjected to stochastic Brownian forces, allowing us to sample identities distributions in a latent space under various constraints.

Abstract: Face recognition models are trained on large-scale datasets, which have privacy and ethical concerns. Lately, the use of synthetic data to complement or replace genuine data for the training of face recognition models has been proposed. While promising results have been obtained, it still remains unclear if generative models can yield diverse enough data for such tasks. In this work, we introduce a new method, inspired by the physical motion of soft particles subjected to stochastic Brownian forces, allowing us to sample identities distributions in a latent space under various constraints. We introduce three complementary algorithms, called Langevin, Dispersion, and DisCo, aimed at generating large synthetic face datasets. With this in hands, we generate several face datasets and benchmark them by training face recognition models, showing that data generated with our method exceeds the performance of previously GAN-based datasets and achieves competitive performance with state-of-the-art diffusion-based synthetic datasets. While diffusion models are shown to memorize training data, we prevent leakage in our new synthetic datasets, paving the way for more responsible synthetic datasets. Project page: https://www.idiap.ch/paper/synthetics-disco

Real-world data, in particular biometric ones, can possibly contain sensitive information which raises privacy concerns from both legal and ethical specialists (Prabhakar et al., 2003; Nat, 2022). This is particularly true for number of Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
David Geissbühler Hatef Otroshi Shahreza
12 Sébastien Marcel 13
Abstract
Face recognition models are trained on large-scale datasets, which have privacy and ethical concerns. Lately, the use of synthetic data to complement or replace genuine data for the training of face recog- nition models has been proposed. While promis- ing results have been obtained, it still remains unclear if generative models can yield diverse enough data for such tasks. In this work, intro- duce new method, inspired by the physical mo- tion of soft particles subjected to stochastic Brow- nian forces, allowing us to sample identities distri- butions in latent space under various constraints. We introduce three complementary algorithms, called Langevin, Dispersion, and DisCo, aimed generating large synthetic face datasets. With this in hands, we generate several face datasets and benchmark them by training face recognition mod- els, showing that data generated with our method exceeds the performance of previously GAN- based datasets and achieves competitive perfor- mance with state-of-the-a -fttt-rrrt diffusion-based syn- thetic datasets. While diffusion models shown to memorize training data, prevent leakage in our new synthetic datasets. paving the way for more responsible synthetic datasets. Project page: https//wwww..ddaapcc/pppprrhhhhh
Figure 1: Example of synthetic faces generated using Style- GAN2. The three rows are three different classes generated with the Langevin algorithm while the columns show intra- class variations generated using the Dispersion algorithm.
large biometric datasets that have been collected in the wild, for instance by scraping images from the internet. Moreover, in addition to privacy concerns, biometric data might give biased representation of the population depending on the data collection procedure (Drozdowski et al., 2020). Re- cently, MeS-eeleb (Guo et al., 2016), a very popular dataset commonly used train face recognition (FR) models, was withdrawn after exposure in the media of its privacy, fairness and demographic biases problems. While data collection campaigns performed in laboratories can be made repre- sentative the general demographics and performed with subjects consents, they are typically quite limited due to the large amount of effort they require.
1. Introduction
In this work, we propose another step towards the tackling of some of theses issues by developing physics-inspired algorithms that allows precise control on the sampling of synthetic identities and variations thereof. More precisely, our method treats samples as soft spherical particles, living in the multi-dimensional latent or embedding spaces, sub- jected to repulsive inter- sitrr-prticees contact forces, a random brownian force well as a global attractive potential. Due to the similarity of this kind of dynamics to the Langevin equation, a stochastic differential equation (SDE) that de- scribes the Brownian motion of small particles in a solvent and its generalization, we call our first algorithm Langevin. This algorithm allows us to approach a dense packing of the spherical identities while keeping latent space spread
'quual contribution 'Idiap Research Institute, Martigny, Switzerland "Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland 'Université de Lausanne (UNIL). Lau- sanne, Switzerland. Correspondence David Geissbühler dlaavid.giissbuhleeeeeee eiiisbbu
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Difflusinn
minimal and thus filling up the latent space starting by the regions that yield the most realistic images. In addition to Langevin, that generates the inter-class distributions, we also develop two similar algorithms, called Dispersion and DisCo, to generate intra-class variations. Figure 1 shows example of images generated with this method. We perform an exploration of the parameter space of these algorithms and generate several synthetic face datasets. We validate our approach by training FR models with this synthetic data, showing that our method can achieve a competitive performance with state of the art.
et al., 2022) used CASIA- (Yi al., 2014) train iddntity-conditionee StyleGAN and then used it gener- ate synthetic dataset. The similar approach was used in SFace2 (Boutros et al., 2024), but instead sampling step was performed in the latent space. In (Liang et al., 2023), a synthetic dataset consisting of 48,000 synthetic face image pairs with 555,00 human annotations is presented and used to measure bias in FR systems.
GANDiffFace (Melzi et al., 2023) used StyleGAN to gen- erate synthetic identities and then used DreamBooth (Ruiz et al., 2023) to generate different samples for each iden- tity. IDnet (Kolf et al., 2023) used a three-player GAN framework to generate a synthetic dataset using the Style- GAN model as a generator, and the third player is trained face images. Syn-Multi-PIE
In summary, the contributions of the paper are as follows:
We propose physics-inspired algorithms to generate large-scale face images datasets by sampling synthetic identities in the latent space of a generative network. We formalize our problem with Brownian motion of small particles in solvent and solve it with a stochastic differential equation.
to generate (Colbois et al., 2021) used StyleGAN to generate synthetic face images and then explored the latent space to generate different samples per identity. ExFaceGAN (Boutros et al., 2023b) used GAN-based face generator models (such as StyleGAN2 (Karras et al., 2020). StyleGAN3 (Karras et al., 2021), or GAN-Control (Shoshan et al., 2021)) and learned an identity boundary in the latent space of GAN model.
The Langevin algorithm that generates a sampling of synthetic identities within GAN's latent space opti- mizing inter-class distances, based on a loss function inspired by granular materials first used in this field.
In contrast to most works based GAN models, recently DCFace (Kim et al., 2023) and IDiff-Face (Boutros et al., 2023a) were proposed which used diffusion models to gen- erate synthetic datasets. Unlike GAN-based face generator models like StyleGAN, the latent space of diffusion models lack style representation, and therefore DCFace (Kim et al., 2023) and IDiff-Face (Boutros et al., 2023a) trained condi- tional diffusion models. DCFace (Kim et al., 2023) used CASIA- Sebbace (Yi et al., 2014) to train dual condition diffusion model with style and identity conditions. Difff- Face (Boutros et al., 2023a) trained an identity connttty-onddiiooned diffusion model and used it to generate different samples of each subject. While diffusion-based methods have achieved state-of- the-trr performance, diffusion models are shown to memorize individual images from their training data, and thus being less private than GAN (Carlini et al., 2023). This has led to considerable data leakage in diffusion- based syn- thetic face recognition datasets, such as memorization in DCFace as shown in (Shahreza & Marcel, 2024). Hence, in this paper, we focus on GAN models which are shown to be more privacy-frienlly (Carlini et al., 2023).
The Dispersion and DisCo algorithms that generate intra-class variations for an ensemble of synthetic identities, based on the same mechanism as well as pre-computed latent directions for the second one.
The remainder of the paper is organized as follow. We first review the existing synthetic face recognition datasets in section 2. Then, in section 3 we describe our physies- inspired methods for identity diffusion and generation of synthetic datasets. In section 4, we present experiments that validate our approach. Finally, the paper is concluded in section 5.
2. Related Work
Considering the legal and privacy concerns in FR models trained with large real face datasets, several works proposed new synthetic face recognition datasets (composed of dif- ferent synthetic subjects with several samples per identity) to use for training face recognition models. DigiFace (Bae et al., 2023) used a computer graphic pipeline to render digital faces and introduce different variations based on face attributes (e.g., variation in facial pose, accessories, and textures). In contrast to DigiFace, other methods used generative neural networks based on GANs or diffusion models. SynFace (Qiu et al., 2021) used a modified Style- GAN2 (Deng et al., 2020) to generate synthetic images as different identities and then generated different sam- ples by mixing identities latent space. SFace (Boutros
Loss functions inspired by physical particles dynamics have already been used in the context of GANs and Diffusion Models. In (Wang et al., 2018), the authors propose re- pulsive loss function based on bounded Gaussian kernel inspired by the hinge loss, showing it significantly improve training at no additional computational cost. Similarly, in (Unterthiner et al., 2017) the GAN learning problem re- formulated with potential field inspired by electro-static forces, which allow the authors to prove that such Coulomb
2

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
GANs possess only one Nash equilibrium which help elim- inate mode collapse during training. In (Franceschi al., 2023), framework that unifies GANs and Diffusion Mod- els is proposed which treats GANS as interacting particle models.
Pg
P21
1112 do
[
do
822
3. Identity Diffusion and Dataset Generation
In this work, focus on GAN generators their latent space has a tractable dimensionality. In addition, GANs are shown to have less leakage from their training data compared to other generator models, such as diffusion mod- els (Carlini et al., 2023). To generate a random face image on a StyleGAN type network, we can first sample a point from a Gaussian distribution in an auxiliary latent space z, then map it to latent space vector w E W using fully connected mapping network and finally generate an image i I using generator network g ((w). This purely Gaussian sampling, however, yields distribution of identities that are not sufficiently dissimilar to train an FR model. The follow- ing sections present the Langevin algorithm that iteratively optimizes an initially random ensemble of synthetic identi- ties towards a more suitable one where identities are distant enough. In addition an ensemble of synthetic identities, most FR tasks also require intra-class variations. To this end, we propose the identity Dispersion algorithm which samples the latent space around the identity reference latent vector and optimize this intra-class ensemble in a way that samples are close in embeddings space. Finally, we present the DisCo algorithm which combines identity Dispersion with latent directions augmentation.
F. = F12 + F13
Z.-50 Rub
33
k
hab do
1 2bb
I: lbab Tab
hab > 0
Fab
0
hab < 0
Fg Fg
F. Eipp
Fab
Figure 2: Simplified model of contact forces between soft spherical bodies. The bodies are characterized by their position H. and diameter do. from which one derives the overlap has and a unit vector Rab parallel to H Es. The force exerted on the body by the body b denoted Fab and the total force on body a by F..
A particle is allowed to have a small overlap hab = dg 1 ž, with a neighboring particle b. The con- tact forces between particles can be then derived from quadratic spring- like potential
N
N
= VV EN)
cont
(Tes zb):
3.1. Physies Inspiration
a=1 b=a+1
We take here inspiration from granular mechanics and Brow- nian dynamics (Finstein, 1905) for our algorithms. We treat synthetic identities soft particles which are allowed to partially overlap and where their contact interactions are modeled as spring-liks force whose magnitude is propor- tional to the overlap (Figure 2). This is similar to Particle Based Methods (PBMs), such as Discrete Element Method (DEM) (Cundall Strack, 1979) which aims to simulate physical granular materials.
2/ (do 1 - =x
I - T/b < do
/oont (I Zp,)
-Taa Iy > do
(2)
where is is the spring constant and x = [ ØN denote the positions of the particles. The sum of the contact forces acting the particle can be then recovered by taking minus the gradient w.r.t. the particle position T.
Front --V..V
Each particle is modeled as perfect sphere of diameter do = 2 ro. labeled with indices a, b, c, .... = =1... 11...N N, where F. = (xx) is the position of the center of the a-th particle in an euclidean D-dimensional space i,j, ... = 1...D D. The dynamies of the system are described by the Newton equation
In this work, we use GAN models (e.g. StyleGAN) to gen- erate synthetic identities. These generative models typically yield better quality images around center point in latent space. To keep samples close such a point, introduce another spring-like force that pulls back samples toward this point
m
Σ F
F back
-oulll pepull-bace back
(Ta Favg),
(4)
where m is the mass of the particle, H. tion and] Σ F. the sum of the forces acting on it.
Lz. di= its accelera-
where Teng denotes the coordinates the center point of the latent space and p.plll is a spring constant. In addi-
3

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
tion these conservative forces, we also add dissipative viscous force damp the system
3.2. Identity Embeddings and Metries
We consider a generative model g(w) that maps latent space W to an image space I. In the case of the StyleGAN family of models, this network is complemented by map- ping network f(z) that maps an auxiliary space 2 where gaussian sampling is performed. In addition to this genera- tive model, we select an off off-the-shelf face recognition (FR) model h(i)) that extracts a face embedding vector e E E. To obtain an embedding from a random latent sample, one evaluates the following chain: Zn ~ N (I, w. [(aa), ia g( (W ea h(ta). The default FR models we use in this work are all based on the ArcFace loss function (Deng et al., 2019), which has a spherical symmetry. To be consis- tent with this symmetry, we define an angular metric on the embedding space & that simply measure the angle between the vectors
Frise -puaa
(5)
where is the viscous force constant and T. velocity of the particle.
di
the
Generally, GANs map a probability distribution in the latent
space Z to a distribution in the target space I
p(z) + p(i), ZE ZE i IEI,
(6)
where the latent distribution p(z) is often chosen as the normal distribution (N,0,)) II). We are interested here in modified distribution (2) that optimizes two simultaneous constraints: 1) The mapping of this distribution to an auxil- iary identity embedding space & spans as much as possible of this space, 2) the variance and mean of pz(2) are min- imized assuring the GAN generates realistic images, We assume that there exists dynamical process that evolves the initial distribution p(z) p(z, le- toward an equi- librium solution which is the solution which optimize the constraints
d8 (ear 66) = arrcoss
Ea : Cb
(12)
leal lcsl
This in turns allows us to put a identity aware metric on the latent space
p²(2) = p(2,t))
(7)
One possible way to describe such dynamical process for the distribution p(z,t t) is to use generalizations of the diffusion equation, such as the Fokker-Plan equation
did (wa: w6) de (e(wa), c(ws))
(13)
where e(w) = h(g(w)).
1 2
= - (1' p) + 0,0, (o p) + +....
(8)
3.3. Inter-Class Optimization
3.3.1. 1. RANDOM-REJECY SAMPLING ALGORITHM
where 0 = Ba and n²¹ (2, t), gهو = 02) t) are coefficient functions that control the dynamics of the system. In practice, solving such equation can be challenging and we choose here to re- re-formulate the dynamics in term of the particle degrees of freedom via the Langevin equation:
We first consider a simple identity sampling algorithm as a baseline. This algorithm works iteratively by ran- domly sampling latent vector 1 and then computing its face embedding En+1- It then compute the distances de between ent1 and the previously accepted samples {ea,a= 1... n and, if all these values are above an Inter- Class Threshold (ICT), the sample Wa+1 is itself accepted. If not the procedure is repeated until a sample that satis- fies this criterion is found. This process is repeated until the desired number of identities Nia is reached. This al- gorithm, used for instance in (Colbois et al., 2021), while perfectly suitable to find sufficiently dissimilar identities, unfortunately scales exponentially with the number of iden- tities making it hard to apply to large datasets. We call this algorithm Reject in following sections.
. Σ F. +f.((..)
This equation is a Stochastic Diffcrential Equation (SDE) (Carmona et al., 1986; Dalang & Sanz-Solé, 2024), which can be obtained from the Newton equation (i.e., Eq. 1) by adding time-dependent random force l'a (t). We consider here the over-damped limit of this equation where viscous forces are dominant over inertia (p >> m) and neglect the latter (m + 0). The random force is assumed not to favor any particular direction, i.e.,
(a((t)=
(10)
3.3.2. IDENTITY SAMPLING FROM LANGEVIN DYNAMICS
and has Gaussian probability distribution. It is Markovian with the following temperature dependent self-correlation relation
To circumvent the scaling problem of the aforementioned Reject sampling algorithm, we present here an iterative algorithm, inspired by the physical systems presented carlier in this section. The main idea is to introduce a repulsive force between the embeddings 6a = e(w,) so that they naturally arrange themselves in an assembly that maximize
(Ta(t) (Fo(O) = 22 TTkkTTDD dat 5(t' - t), (11)
where y is the viscous coefficient, ky the Boltzmann con- stant and where T is the temperature.
4

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusson
their inter-class distances. While any kind of repulsive force, found in the physical world or not, could in principle serve this purpose, a spring-like force that has a linear dependency on the position seems the simplest choice. Moreover, such type of forces are independent of the dimensionality of the space they act in. On the contrary, other physical forces such as gravity or electrostatics are described by power laws with exponents that depends crucially on the dimensionality of space and seem less adapted for our purpose.
of the full latent space. For instance, the StyleGAN family of models (Karras et al., 2019; 2020; 2021) implements the so-called truncation trick which consists by rescaling the latent vector by a constant factor w.r.t. an origin placed at the average latent Wavy calculated by sampling a large number of vectors via the mapping network. In other words, the best quality images are those with latents located near Wang. Introducing purely repulsive interactions alone would be problematic as the latent vectors would be pushed away from the domain where the network generates the best qual- ity data. To circumvent this we introduce a latent pull-back loss function, quadratic as well, that keeps the identities from wandering too far from the
Ideally, we do not want that identities that are far away interact together, only identities pairs whose distance is below a certain ICT value should lead to a repulsive force. With this criterions in mind, we observe that the potential for non-dissipative granular contact interactions in Fq. 2 achieves precisely this, as the interaction vanishes when the distance is bigger than a constant dg- Based on these ideas, we define the identity granular repulsion loss
Wavg latent
WW
Nii
C
W - Wangl 12
(15)
2
We call the resulting identity sampling algorithm Langevin due to its similarities with the equations describing motion small soft particles in thermal bath.
_ L Ni NIA
{ (do - dab
d < do dab do (14)
{0
2
b=a+1
E de (e (wa), (ws)) d (Wa: W6).
3.3.3. NUMERICAL IMPLEMENTATION
We note that this loss function is a close relative of non- linear Hinge Losses commonly used SVM algorithms (Luo et al., 2021). Sampling a set identities that satisfy a minimal distance threshold is similar problem. While solutions to this problem are known for two and three dimensions, higher dimensional optimal sphere packing lattices are usually unknown, except in some spe- cial cases (Cohn et al., 2017). Ideally, we would like our algorithm to generate the densest packing possible to get the most identities from a given generative network. In practice, no algebraic solution can be found in such high dimensionality we assume that our dynamical optimization method yields good enough solution. While the stochastic noise introduced in Eq. 10 is not strictly necessary to this optimization problem, we choose to keep it as it introduces an additional temperature ayper-parameter via Eq. 11 as well as a source of randomness which can be switched off if needed. An hypothetical benefit of randomness in this context is to prevent the formation of jammed states, which can happen in granular materials in two or three dimensions where particles are inter-locked in way that can block further rearrangement of the granular elements (Brilliantov et al., 2004). We do not expect such jamming to appear in this high dimensionality context and indeed have not observed such a phenomenon in our limited experiments. However, observed a minor improvement in the conver- gence speed. It is worth noting that jamming transitions have already been studied in the context of loss landscapes of deep neural networks (Geiger et al., 2019).
As said earlier we focus on the case where viscosity is dominant w.r. inertia. For physical particles we thus need to solve the first order stochastic differential equation in Eq. 9, which we approximate by
sphere packing
(t + 8t) 2 xt.(t)+
+
81 F +
10 dw (t. 8t)
= =
öt k V
700 Σ Ras
dW(1, 5t)
2 p
a,b3a
(16)
where see that the viscosity constant μ scales the other constants & and no. For the following discussion we set the viscosity to be = and use the other constants to parametrize the problem. Our latent update algorithm, in- spired by the above simple numerical scheme, reads:
w (1+1) ( öt Vw, c(²)) + no V CL c!")
(17)
where ((()) is a vector of independent normal variables of variance o = and where
(1) c = (C + CW) "
(0) N
(18)
From a numerical perspective, the gradient of LW can be easily calculated so the only challenging task is the com- putation of the gradient of the embedding distance metric Va Vadf. df. This computation can be challenging because the embedding computation passes through two networks, the generator and the embedding extractor, and through a very high dimensional image space. One can simplify the prob- lem by computing the jacobian Zu but it is still quite com- putationally expensive. We find that a more efficient way
Generative models, and GANs in particular, generate good quality images when the input latent vector a sub-space
de
5

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
Algorithm 1 Langevin algorithm
3.4. Intra-Class Variations
1: for 1. Nia do
3.4.1. IDENTITY DISPERSION
2: Za + .V
3: W/a + f(za)
As we are interested in generating synthetic datasets that can be used to train FR models, need variations of the synthetic identities, so called within-class variations. We devise here second algorithm, called Dispersion, that generates an arbitrary number of variations from a refer- ence latent-embedding pair (w"e,, ,xr)) while preserving as much as possible the identity. This algorithms similar to the Langevin algorithm but is intended to be used to after suitable set of identities have been generated by the latter. Given a number Nid of references identities, this algorithm creates Nuar variations w with = 1I.. Near. To keep these new latent vectors separated enough, to create vari- ability, we introduce another granular loss function, but this time in the latent space
4: end for
5:
for Niter do
6: for = I... Nid do
7:
in 5 g
8:
ecst + h((aa)
9:
end for
10:
for
1... Nid do
11:
in s g g(wa)
12:
ca + h(ba)
13:
e. - de (ea, (g
b #a
14:
fa Va (L + (WW)
15:
end for
16:
ft + arg max fa
17:
dw + arg min lWa- wal
kW disp
N.a Near
Near Σ
18:
öt +-+ T dw /ft
Σ
disp
19:
for a = 1........ Nid do
2
3=a+1
20:
La + -
21:
Wa +- wa + St.fa + Vōlco
{
{ (d
w
: lwa
w < di
22:
end for
0
: lea w
d
23:
end for
(20)
For the present work, we do not compute contact forces be- tween different identities for identity variations, even if this could potentially improve performance. This is done mainly for simplicity reasons as such extension would require par- allelization with custom inter-GPU communications. To keep the embeddings of the identities variations close to the reference embedding, we introduce further spring-like quadratic loss function
of performing this computation is to first compute all the embeddings with a forward only pass, and then compute the gradients in a second pass. This procedure allows us to make the problem tractable on standard computing hard- ware, even for a large number of identities. We still need to determine a appropriate time-step for our calculations. We will show in the next section that the time-step has little impact, a wide range of values yielding acceptable perfor- mance. Values too coarse can however lead to numerical problems that can have a negative impact over the quality of the resulting dataset. Looking at Eq. 17, and neglecting the random force, we see that the maximal distance latent can move in a single time-step, arg max
Ae Ni Nowr
Ciisp
d (e()) )²
(21)
2
which is computed similarly to the granular loss of the Langevin algorithm. Finally, we also add the latent pull- back loss in Eq. 15, the total Dispersion loss reads
(1+1) (1) Wa
is proportional to the time-step times arg max VC If we impose that this distance should remain smaller than proportion 7 < of the minimal aitent-to-laten distance, we find the following expression for the time-step
Ldisp Citss +
(22)
The numerical implementation of this algorithm is very sim- ilar to the Langevin one, the latent update equation simply reads
8t arg min [Wae wbl
(19)
w (t+1) = w. (1) - 5t V wa disp + TJo ot Co (t)
(C)
(23)
arg max (L))
In this article we keep fixed time-step for Dispersion, for simplicity reasons, and parallelize over identities. We initialize the new latents wa with the value of the reference one wh and add some random gaussian noise to break the symmetry of the ensemble
which prevent latent vectors to be updated too aggressively while still giving good numerical performance when the distribution close to an equilibrium. The final Langevin algorithm is depicted in Algorithm I. An illustration of the Langevin algorithm depicted in Appendix B.
0 wree + Eo ca;
(24)
6

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
where So is a vector of independent normal variables of variance and Eo is fixed scaling parameter. An illustration of the Dispersion algorithm is depicted in Ap- pendix B.
4. Experiments
4.1. Experimental Setup
Training and Benchmarking We use the synthetic dataset and train face recognition model with the IRes- Net50 backbone using AdaFace loss function (Kim et al., 2022). We train each model for 30 epochs using the Stochas- tic Gradient Descent (SGD) optimizer with the initial learn- ing rate 0.1 and weight decay 5 x 10-4. Then, we eval- uate the performance of the trained models on different benchmarking datasets, including Labeled Faces in the Wild (LFW) (Huang et al., 2008), Cross-age LFW (CA-LFW) (Zheng et al., 2017), CrossPose LFW (CP-LFW) (Zheng & Deng, 2018), Celebrities in Frontal-Profile in the Wild (CFP- FP) (Sengupta et al., 2016), and AgeDB-30 (Moschoglou et al., 2017) datasets. To maintain consistency with prior works, the results reported for LFW, CA-LFW, CP-LFW, CFP-FP, and AgeDB datasets are accurately calculated using 10-fold cross-validation, where the comparison threshold is set at the Equal Error Rate (ERR) on one fold and the accuracy is measured on the remaining folds.
3.4.2. LATENT EDITING: COVARIATES
While the Dispersion algorithm creates realistic variations of a given identity, it gives little control over the type of variation created. Moreover, having an alternative method to create variations is desirable to give comparison point. For this reason we use the latent editing method introduced in (Colbois et al., 2021). This latent editing technique as- sumes that the variation of some attribute, such as left-right pose, is essentially a translation in latent space and that this translation is the same for every identity.
Given this assumption and following (Colbois al., 2021), we project in the latent space the samples of the CMU Multi-PIE face database. This dataset provides a reasonable number of genuine identities, each captured with different poses, illuminations and expressions. When this is done, we use a linear SVM model to fit the latent directions cor- responding to the different attributes variations present in the database. In particular, we extract 7 vectors wfa cor- responding to left-right poses, left-right illuminations and 5 facial expressions: smile, surprise, squint, disgust and scream. The algorithm introduced in (Colbois et al., 2021) based on these latent direction is called Covariates in this work and yields a total of 17 variations: different poses, 6 different illuminations and one for each of the expressions.
Different Hyperparameters Our new method introduces number of hyperparameters that influence the quality and usefulness of the final synthetic datasets. Table 5 in Ap- pendix C shows the list of hyperparameters of the Langevin, Dispersion and DisCo algorithms as well as their default values. We also provide ablation study with different values and evaluate the resulting datasets.
Reference FR Backbone To compute embedding dis- tances and optimize the identity distribution with respect to the latter, we choose an off-the-shelf reference FR model. This model is built on an IResNet50 backbone, with an Arc- Face loss (Deng et al., 2019) and trained on MS-Celeb- IM dataset (Guo et al., 2016).
3.4.3. COMBINING DISPERSION AND COVARIATES
We also propose combine the Dispersion and Covariates methods and name the resulting algorithm DisCo, The essential difference with the original Dispersion algorithm is in the initialization procedure. In addition to the initial symmetry breaking gaussian noise Sa the DisCo algorithm also adds a linear combination of the 7 Covariates vectors wooo to the reference latents wef
4.2. Comparison with Previous Synthetic Datasets
To compare the performance of our generated synthetic datasets with previous synthetic face recognition datasets in the literature, we train face recognition model with same backbone and using same learning loss as explained in section 4. Table reports the performance of face recog- nition model trained with different datasets. The results of benchmarking in Table show that our method achieves superior performance compared to GAN-baset methods. Compared diffusion-based datasets, our method outper- forms IDiff-Facc all benchmarking datasets and achieves competitive performance with DCFacee However, we should note that diffusion models are shown to be prone to leaking information from their training samples (Carlini et al., 2023; Vyas et al., 2023: Somepalli et al., 2023a;b; Li et al., 2024; Shahreza & Marcel, 2024), which limits their
w (0) = res + ED ct + Σ
Nat cov 1=1
(25)
where weights Ag E I- Aol are randomly and uniformly sampled in a deven-dimensional hypercube. This additional step forces more intra-class variability at the initial step of the Dispersion algorithm and yields, according to our ex- periments, better performing datasets. This positive effect might be explained by the number of identities variations compared to the dimensionality of the intra-class latent sub- space. In the case where the former is small compared to the latter, this extra initialization step helps the latent granular contact loss to spread the latent vectors across the intra-class subspace, yielding a more diverse final dataset.
7

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
Table I: Comparison of the existing synthetic face datasets present in the literature with the best performing datasets created in this work. In addition to the number of identities and number of images, we present the recognition accuracy obtained by training an FR model on each dataset and evaluating on standard face recognition benchmarking datasets. The best value in each category in bold text and the best results achieved by training from synthetic datasets amongst all categories of synthetic datasets are underlined.
Dataset Type
Dataset name
Generator
Nid N Naamples
LFW
CPLFW
CALFW
CFP
AgeDB
MS-Celeb-1M MM (Guo et al., 2016)
N/A
857000
5'800'000 99.82
92.83
96.07
96.10
97.82
Real images
WebFace-4M (Zhu al., 2021)
N/A
2067000
40000'000 000 99.78
94.17
95.98
97.14
97.78
CASIA-WebFace (Yi et al., 2014)
N/A
10'572
490'62)
99.42
90.02
93.43
94.97
94.32
Computer Gruphics
DigiFace- IM (Bac et al., 2023)
Rendered mesh
109'999
1'219'995 90.68
72.55
73.75
79.43
68.43
DCFace-0.5M (Kim et ul., 2023)
custom trained
10'000
500'000
98.35
83.12
91.70
88.43
89.50
DCFace-1.2M (Kim et al., 2023)
custom trained
60'000
1'200'000
98.90
84.97
92.80
89.04
91.52
Diffusion- based
IDiff-Face (Uniform) (Boutres et al., 2023a)
custom trained
107049
502'450
98.18
80.87
90.82
82.96
8,5.50
IDiff-Face (Two-Stage) (Boutros et al., 2023a)
custom trained
10'050
502'500 98.00
77.77
88.55
82.57
82.35
Synface (Qiu et 2021)
StyleGAN2
10'000
999'994
86.57
65.10
70.08
66.79
59.13
SFace (Boutros al., 2022)
StyleGAN2
10'572
I 1'885'877
93.65
74.90
80.97
75.36
70.32
SFace2 (Boutros et al., 2024)
StyleGAN2
102
1 2058'25
94.03
73.2
80.33
74.87
72.98
Syn-Multi-PIE (Calbois et al,, 2021)
StyleGAN2
10'000
180'000
78.72
60.22
61.83
60.84
54.05
GAN-based
GANDiftFace (Melzi et 2023)
StyleGAN3
100080
543'893
94.35
76.15
79.90
78.99
69.82
IDnet (Kolf et al., 2023)
StyleGAN2
107577
1'057'200
84.48
68.12
71.42
68.93
62.63
ExFaceGAN (Boutros et al., 2023b)
GAN-Control
10'000
599'944
85,98
66.97
70.00
66.96
57.37
Langerin-Dispersion [ours]
StyleGAN2
10'000
650'000
94.38
65.75
86.03
65.51
77.30
Langevin-DisCo [ours]
StyleGAN2
10'000
650'000
97.07
76.73
89.05
79.56
83.38
Langevin-DisCo [ours]
StyleGAN2
30'000
1'950'000 98.97
81.52
93.95
83.77
93.32
Dataset was re-generated from the original source code
Identity conditioned
Disentangled representation (Deng et al., 2020)
Table 2: Influence of the number of Langevin iterations Niter on FR accuracy. For all datasets, variations are created via the Dispersion algorithm with default parameters. The first row consider pure random sampling for benchmarking. The next rows show different combination of Niter and de
Table 3: Influence of Langevin time-step parameters on final FR accuracy. The parameter òt is the fixed time-step value while is the automatic time-step parameter.
5t
Nowr
LFW
CPLFW
CALFW
CF.
AgeDB
Average
0.1
200
84.3
56.63
72.33
57.49
65.63
67.28
0.3
84.12
57.27
71.15
58.63
64.98
Naw
d5
LFW
CPLFF
CALFW
CFP
AgeDB
Average
0.6
33
83.05
$7.65
69.75
57.91
63.7
66.41
0
90.78
62.55
78.5
61.96
70.12
73.38
1.0
20
82.9
57.7
69.9
57.4
61,23
65.83
1
1.54
90.73
64.15
78.5
65.24
70.67
73,86
0.3
50
84.12
59.03
72.18
57.8
63.83
67.39
1.54
92.23
65.05
81.42
65.5
72.73
75.39
1.0
82.83
$7.73
70.67
58.41
63.57
20
1.54
92.73
66.37
82.42
65.09
74.22
76.17
50
1.54
93.93
66.83
84.52
68.67
75.52
77.89
50
1.4
94.45
65.58
86.03
66.53
77.17
77.95
100
1.4
94.72
64.58
86.17
65.36
79.25
78.02
random sampling. Table 2 shows the accuracy of FR models trained on synthetic datasets generated with such a varying number of iterations and for two values of dh- The datasets in this table are composed of N = 10k synthetic identities, the first one being the random sampling benchmark. For all datasets, the 64 variations are created with the Dispersion al- gorithm with the default parameters reported in Appendix E. It is clear that the Langevin algorithm yields a significant performance improvement, even after a small number of iterations, over pure random sampling.
application in tasks with sensitive data. In fact, the main motivation for generating synthetic datasets is to resolve the privacy concerns large-scale real face recognition datasets. However, if the generated synthetic dataset has leakage of information from a real dataset, it will have similar privacy issues. This issue, as well as possible mitigation within our framework, is further discussed in Appendix H. The results in Table I also show that there is still a gap between training with synthetic and real face recognition datasets.
interesting that more iterations tend to yield better performance and that this improvement effect continues even while the average embedding distance has reached a plateau.
4.3. Performance of the Langevin Algorithms
While the Langevin algorithm seems to yields significant accuracy advantage for synthetic data trained FR models, it is quite computationally expensive. A straightforward parameter to optimize is the time-step value, which control
We are interested in the influence of the number of Langevin iterations on the quality of the resulting datasets and, impor- tantly, if the latter improve FR accuracy when compared to
8

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
Table 4: Influence of the stochastic force on FR accuracy.
Source Code and Data Availability
Wo
Nater
LFW
CPLFW
CALFW
CHP
AgeDB
Average
To adhere to the standards of reproducible research, the code used for our experiments is publicly available. In addition,
0.003
100
65.4
86.63
65.9
79.35
78.44
0.01
100
94.72
64.58
86.17
65.36
79.25
78.02
0.03
100
94.53
65.67
85,72
64,99
79.22
78.03
we also provide access to our synthetic datasets, Project page: https://www.idiapp http://wwdiiii...pppps//stttttoe chhpppperrsynthetee-diii.ee
A Acknowledgment
the precision of the numerical integration. Table 3 shows the effect of the time-step on the final FR accuracy and shows a comparison of fixed step against the automatically calculated values. A small advantage is observed for smaller fixed values, at least until öt = 0.3, and calculated values with r = 0.3 reach similar performance with slightly less iterations. For this reason, a varying time-step with r = 0.3 is used in the rest of the article.
This research is based upon work supported by CITeR (Center for Identification Technology Research) project L.F- GAL2 (CITeR-2IF-02i-N) This work was also funded by the Hasler foundation through the Responsible Face Recog- nition (SAFER) project as well as the H2020 TReSPAsS- ETN Marie Skłodowska-Cuun early training network (grant agreement 860813).
Finally, we briefly study the impact of the amplitude of the random stochastic force on the resulting FR performance. The addition of this term was motivated in section 3 by the idea that it could mitigate potential jamming and help sampling the latent space efficiently. Table 4 shows a very limited survey for this purpose, with three different values of 7o- This data shows that this parameter, at least in this very limited range, has limited impact on the results. We report further experiments in Appendices D-K, including complexity evaluation, dynamical evolution of Langevin ensembles, strict Inter-Class Threshold constraints, repul- sion distances threshold, identity leakage from training set, synthetic dataset scaling, bias evaluation, and closing the gap with real data.
Impact Statement
Large-scale face recognition datasets are collected by crawl- ing the web (without individuals' consent), raising ethical and privacy concerns. With the advancement of generative models, recently synthetic data has emerged as a promising solution. In this paper, we proposed a new approach to gen- erate synthetic face recognition datasets to address privacy and ethical concerns in web-crawled real face recognition datasets. Our experiments demonstrate effectiveness of our generated datasets for training face recognition models.
We would like highlight that there are different components in our data generation pipeline, which still require real face datasets, including face generator model (StyleGAN) and pretrained face recognition model. We studied the leakage of identity in the generated datasets in Appendix H. In addi- tion, in Appendix H, we introduced repulsive forces push synthetic samples away from training samples.
5. Conclusion
We proposed algorithms to generate synthetic face images datasets by sampling synthetic identities in the latent space of generative network. We introduced three complemen- tary algorithms,, Langevin, Dispersion and DisCo, aimed at generating large synthetic datasets to mitigate ethical issues arising from the usage real datasets. These algorithms take inspiration from physical processes such as Brownian motion and granular mechanies generate ensembles of samples in the latent space of generative models. To our knowledge, our use of loss functions inspired by granu- lar mechanics is novel in this context and open promising avenues, possibly beyond face recognition applications. Im- portantly, these algorithms introduce a set of free parameters that control the distribution of samples in the latent space and which can be optimized to yield high quality synthetic datasets tailored for specific usages. To validate the sound- ness of our approach and evaluate its effectiveness at gener- ating useful synthetic face data, we trained FR models on the synthetic datasets we generated and evaluated the result- ing models on seven standard face biometric benchmarks. We also outline some future directions in Appendix I..
We should also note that there might be potential lack of di- versity in different demography groups, stemming from im- plicit biases of the datasets used for training in our pipeline (such as the pretrained face recognition model, StyleGAN, etc.). We evaluate bias in the performance of face recog- nition model trained with our dataset in Appendix J. It is also noteworthy that the project on which the work has been conducted has passed an Institutional Ethical Review Board (IRB).
References
The rise and fall (and rise) of datasets. Nature Ma- chine Intelligence, 4(1):1-2, 2022. doi: 10.1038/
$422556-02-000442-22 URL https: hpps://dolt //wp///oiii/o org/10. 1038 1038/44256-022-04423
Bae, G., de La Gorce, M., Baltrušaitis, T., Hewitt, C., Chen, D., Valentin, J., Cipolla, R., and Shen, J. Digiface-
9

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
million digital face images for face recognition. In Proceedings of the IEEE/CVF Winter Conference Ap- plications of Computer Vision, pp. 3526-3535, 2023.
Deng, J., Guo, J., Xue, N., and Zafeiriou, S. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4690-46999 2019.
Boutros, F., Huber, M., Siebke, P., Rieber, T., and Damer, N. Sface: Privacy-friendly and accurate face recognition using synthetic data. In 2022 IEEE International Joint Conference on Biometrics (JCCB), pp. 1-11, IEEE, 2022.
Deng, Y., Yang, J., Chen, D., tangled and controllable face image generation via 3d imitutive-contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5154-5163, 2020.
Wen, F., and Tong, X. Disen-
Boutros, F., Grebe, J. H., Kuijper, A., and Damer, N. Idiff- face: Synthetic-based face recognition through fizzy iddntity-condittioned diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19650-19661, 2023a.
Drozdowski, P., Rathgeb, C.,, Dantcheva, A., Damer, N., and Busch, C. Demographic bias in biometrics: A survey on an emerging challenge. IEEE Transactions on Technology and Society, 1(2):89-103, 2020.
Boutros, F., Klemt, M., Fang, M., Kuijper, A., and Damer, N. Exfacegan: Exploring identity directions in gan's learned latent space for synthetic identity generation. In 2023 IEEE International Joint Conference on Biometrics (IJCB), pp. 1-10. IEEE, 2023b.
Einstein, A. Über die von der molekularkinetischen the- orie der wärme geforderte bewegung von in ruhenden flüssigkeiten suspendierten teilchen. Annalen der physik, 4, 1905.
Franceschi, J.-Y., Gartrell, M., Dos Santos, L.., Issenhuth, T., de Bézenac, E.,, Chen, M., and Rakotomamonjy. A. Unifying gans and score-based diffusion generative particle models. In Oh, A.. Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 59729 59760. Curran Associates, Inc., 2023.
Boutros, F., Huber, M., Luu, A. T., Siebke, P., and Damer, N. Sface2: Synthetic- based face recognition with w- space identity-driven sampling. IEEE Transactions on Biometrics, Behavior, and Identity Science, 2024.
Brilliantov, N., Saluena, C., Schwager, T., and Pöschel, T. Transient structures in a granular gas. Physical review letters, 133130, 2004.
Geiger, M., Spigler, S., d' Ascoli, S., Sagun, L.,, Baity-Jesi, M., Biroli, G., and Wyart, M. Jamming transition paradigm to understand the loss landscape of deep neural networks. Physical Review E, 100(1):01211) 15, 2019.
Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., Balle, B., Ippolito, D., and Wallace, E. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5270, 2023.
Guo, Y., Zhang, L., Hu, Y.. He, X., and Gao, J. Ms-celeb- Im: A dataset and benchmark for large-scale face recognition. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part III 14, pp. 87-102. Springer, 2016.
Carmona, R., Kesten, H., Walsh, J. B., and Walsh, J. B. An introduction to stochastic partial differential equations. Springer, 1986.
Huang. G. B., Mattar, M., Berg, T., and Learned-Miller, E. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in Real-Life' Immage: detection, alignment, and recognition, 2008
Cohn, H., Kumar, A., Miller, S., Radchenko, D., and Via- zovska, M. The sphere packing problem in dimension 24. Annals of Mathematics, 181(3)10177-1033, 2017.
Colbois, L., de Freitas Pereira, T., and Marcel, S. On the use of automatically generated synthetic image datasets for benchmarking face recognition. In 202/ / IEEE Inter- national Joint Conference on Biometrics (IJCB), pp. 1-8. IEEE, 2021.
Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020.
Cundall, P. A. and Strack, O. D. A discrete numerical model for granular assemblies. geotechnique, 29(1):47- 65, 1979.
Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., and Aila, T. Alias generative adversarial networks. Advances in Neural Information Processing Systems, 34:852-866 863, 2021.
Dalang, R. C. and Sanz-Solé, M. An introduction to stochas- tie partial differential equations: Part arXiv preprint arXiv:2402.022 02/ 19, 2024.
10

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
Kim, M., Jain, A. K., and Liu, X. Adaface: Quality adap- tive margin for face recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 19750-18759, 2022.
Ruiz, N., Li, Y., Jampani V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image dif- fusion models for subject-driven generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
Kim, M., Liu, F., Jain, A., and Liu, X. Deface: Synthetic face generation with dual condition diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12715-12725, 2023.
Sengupta, S., Chen, J.-C., Castillo, C.,, Patel, V. M., Chel- lappa, R., and Jacobs, D. W. Frontal profile face veri- fication in the wild. In 2016 IEEE winter conference on applications of computer vision (WACV), pp. 1-9. IEEE, 2016.
Kolf, J. N., Rieber. T., Elliesen, J., Boutros, F., Kuijper, A., and Damer, N. Identity-driae driven three-player generative ad- versarial network for maathtiic-based face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 806-816, 2023.
Shahreza, H. O. and Marcel, S. Unveiling synthetic faces: How synthetic datasets can expose real identities. arXiv preprint arXiv:2410.24015. 2024.
Shahreza, H. O. et al. Sdfr: Synthetic data for face recog- nition competition. arXiv preprint (tti:240440458800 2024.
Li, Z., Hong, J., Li, B., and Wang. Z. Shake to leak: Fine- tuning diffusion models can amplify the generative pri- vacy risk. arXiv preprint arXi::2403.09450 2024.
Shoshan, A., Bhonker, N., Kviatkovsky, I., and Medioni, G. Gan-control: Explicitly controllable gans. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 14083-14093, 2021.
Liang, H., Perona, P., and Balakrishnan, G. Benchmarking algorithmic bias face recognition: An experimental approach using synthetic faces and human evaluation. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 4977-4987, 2023.
Somepalli, G., Singla, V., Goldblum, M., Geiping, J., and Goldstein, T. Diffusion art or digital forgery? investigat- ing data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6048-6058, 2023a.
Luo, J., Qiao, H., and Zhang, B. Learning with smooth hinge losses. Neurocomputing, 463:379-320 387, 2021.
Maze, B., Adams, J., Duncan, J. A., Kalka, N., Miller, T., Otto, C., Jain, A. K., Niggel, W. T., Anderson, J., Cheney, J., et al. Iarpa janus benchmark-c: Face dataset and protocol. 2018 international conference on biometrics (ICB), pp. 158-165. IEEE, 2018.
Somepalli, G., Singla, V., Goldblum, M., Geiping, J., and Goldstein, T. Understanding and mitigating copying in diffusion models. Advances in Neural Information Pro- cessing Systems, 36:47783 2023b.
Unterthiner, T., Nessler, B., Seward, C., Klambauer, G.,
Melzi, P., Rathgeb, C., Tolosana, R., Vera-Rodriguez, R., Lawatsch, D., Domin, F., and Schaubert, M. Gandiff- face: Controllable generation of synthetic datasets for face recognition with realistic variations. arXiv preprint arXiv:2305. 19962, 2023.
Heusel, M., Ramsauer, H., and Hochreiter, S. Coulomb gans: Provably optimal nash equilibria via potential fields. In International Conference on Learning Representations, ICIR 2018, 2017
Vyas, N., Kakade, S. M., and Barak, B. On provable copyright protection for generative models. In Inter- national Conference Machine Learning, pp. 35277- 35299. PMLR, 2023.
Moschoglou, S., Papaioannou, A., Sagonas, C., Deng, J., Kotsia, I., and Zafeiriou, S. Agedb: the first manually collected, mn-the-wild age datahase. In proceedings of the IEEE conference on computer vision and pattern recogni- tion workshops, pp. 51-59, 2017.
Wang, M., Deng, W., Hu, J., Tao, X., and Huang, Y. Racial faces in the wild: Reducing racial bias by information maximization adaptation network. In Proceedings of the ieee/cyf international conference on computer vision, pp. 692-702, 2019.
Prabhakar, S., Pankanti, S., and Jain, A. K. Biometric recog- nition: Security and privacy concerns. IEEE security & privacy, 1(2):33-42 2003.
Qiu, H., Yu, B., Gong, D., Li, Z., Liu, W., and Tao, D. Synface: Face recognition with synthetic data. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10880-10890, 2021.
Wang, W., Sun, Y., and Halgamuge. S. Improving mmd-gan training with repulsive loss function. In International Conference on Learning Representations, ICLR 2019, 2018.

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
Whitelam, C., Taborsky, E., Blanton, A., Maze, B., Adams,
J., Miller, T., Kalka, N., Jain, A. K., Duncan, J. A., Allen, K., et al. Iarpa janus benchmark-b face dataset. In pro- ceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 90-98, 2017.
Yi, D., Lei, Z.,, Liao, S., and Li, S. Z. Learning face repre- sentation from scratch. arXiv preprint arXiv:1411.792.. 2014.
Zheng, T. and Deng, W. Cross-pose Ifw: A database for studying cross-pose face recognition in unconstrained environments. Beijing University of Posts and Telecom- munications, Tech. Rep, 5(7), 2018.
Zheng, T., Deng, W., and Hu, J. Cross-age Ifw: A database for studying cross-age face recognition in unconstrained environments. arXiv preprint arXiv: 1tt::::778.099999 2017.
Zhu, Z., Huang, G., Deng, J., Ye, Y., Huang. J., Chen, X., Zhu, J., Yang. T., Lu, J.. Du, D., et al. Webface260m: A benchmark unveiling the power of million-scale deep face recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pultern Recognition, pp. 10492- 10502, 2021.
12

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
A. Sample Images
In this appendix, we show sample images generated with our algorithms. Figure shows synthetic identities obtained with the Langevin algorithm. Figure 4, Figure 5 and Figure show intra-class variations of synthetic identities generated with the Dispersion, Covariates and DisCo algorithms, respectively.
Figure 3: Sample images illustrating the inter-class variation produced by the Langevin algorithm.
Reference
(b) Variation
(c) Variation
(d) Variation
(e) Variation
(f) Variation 5
Variation 6
(h) Variation
Figure 4: Sample images illustrating the intra-class variation produced by the Dispersion algorithm.
13

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
(a) Reference
(b) Illumination (c) Illumination
(d) Pose
(e) Pose
(f) Smile
(g) Squint
(h) Surprise
Figure 5: Sample images illustrating the intra-class variation produced by the Covariates algorithm.
Reference
(b) Variation
Variation 2 (d) Variation 3 (e) Variation
(f Variation 5
Variation 6
(h) Variation
Figure 6: Sample images illustrating the intra-class variation produced by the DisCo algorithm.
B. Illustrations of Langevin and Dispersion Algorithms
To generate identities with suitable inter-class distance between samples, we use the Langevin algorithm illustrated in Figure This algorithm iteratively optimizes an ensemble latent vectors, one per identity, living in the latent space of a GAN generator, using a repulsive loss function. The distance between samples computed with an off-the-shelf model by
14

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
mapping network
synthesis network
feature extractor
2 + w = f(z)
w - = g(w)
- e = h(i)
dg (Ca, cb)
6b
000000000
(++1)
Wa
Co
1
Za
0000000L
(2)
Worg
000000
de de(ea: ec)
1.
W
I
E
Z
W
back-propagation
Figure 7: Mappings between the different spaces and details of the Langevin algorithm. Firstly, random vector Zq is sampled from normal distribution in Z, for each identity class a 1........ Nia It is then mapped to the initial latent w(2) W via the mapping network f. A face image is generated from this latent using the generator synthesis network te = g and, after face alignment, the face embedding computed with the reference feature extractor e, = h(ia). Two quadratic loss functions are introduced, one on the embedding space ε, depending on the embedding distance d and one on the latent space W distances d'W, Their derivatives are evaluated using pock-propagttion and the latents are updated based on these gradients w!( + wia
(t+1), The procedure repeated for a desired number of iterations Niter-
computing the angle between the embedding vectors. Similarly, Figure 8 depicts the Dispersion algorithm we use to generate intra-class variations of a given synthetic identity. This algorithm also iteratively optimizes the samples distribution, with the main difference that the repulsive loss function is defined in the latent space. An attractive loss function in embedding space is added to keep the identities variations as close as possible to the reference synthetic identity. The combination of these two loss functions generates intra-class variations that are as close as possible of the reference embedding while pushing them as far as possible in the latent space. For both Langevin and Dispersion algorithms, we add attractive loss function in latent space that keeps the latent vectors close to the average latent, where images have the best quality.
C. Default Values of Hyperparameters used in Experiments
Our new method introduces number of hyperparameters that influence the quality and usefulness of the final synthetic datasets. While this gives new opportunities to tune a synthetic dataset towards a particular goal, it also adds a new layer of complexity that requires a good understanding of the influence of each of these numbers the final results. Table 5 of this appendix shows the list of hyperparameters of the Langevin, Dispersion and DisCo algorithms as well as their default values. These default values give a baseline with good numerical performance and are used in the following experiments unless explicitly specified. To better understand the impact of these hyperparameters, we run a non-exhaustive survey with different values and evaluate the resulting
D. Complexity and Required Computational Resources
The computation for the algorithms can be separated in three main phases: 1) a forward pass to generate embeddings 2) second forward pass with backward pass to compute interactions and 3) an update step where latent vectors are updated. The run time for the Langevin and DisCo algorithms is reported in Table 6 and Table 7, respectively, for system equipped with single NVIDIA RTX 3090 GPU. The first pass has linear complexity with respect to the number of identities O (Na) and can easily be parallelized. We use a batch size of 64 for this phase, which is dictated by memory constraints. The second phase has quadratic complexity O (NZ) as pairwise interactions are computed for each pair of samples. Similarly,
15

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
synthesis network
feature extractor
w - i = g(w)
i + e h())
W
0211
e
Cooooo
00000000
e
Worg
00000
I
W/
La
3
Figure 8: The Dispersion algorithm is very similar to Langevin, with slightly different loss functions, and is intended to generate Near Intra-class variations per identity class. In a first step. for each variation a = -N... .Nar. a latent vector wa (0) is initialized from its reference value wxe// plus some noise. Three loss function are then computed. The first one acts on the embedding space and pulls the embeddings of the variations towards the reference embedding e The second loss function act on the latent space and pulls the latent vectors towards the average latent Wang- The last one is a granular loss function that exert a repulsive force between latent vectors that are closer than a certain threshold. Latent vectors are updated according to the gradients of these losses. The procedure is repeated for each class a 1.... Naa and for a desired number of iterations iter
the complexity of the second phase of the DisCo algorithm is linear for the number of identities O (Naa), as there no
interactions between identities, and quadratic for the number of variations O (N2 ( In DEM and other particle-based methods, this can be reduced to linear complexity by computing a near-neighbors list at regular intervals. However, for simplicity in the implementations, we did not follow this approach. Finally, the last pass that updates the latent vectors has linear complexity O (Naa) and is very fast compared to the previous phases.
E. Dynamical Evolution of Langevin Ensembles
The Langevin algorithm presented in the previous section is designed to maximize, iteratively and stochastically, the pairwise embedding distances of an ensemble of synthetic identities defined on the latent space of a given generative model. This is achieved using the loss function, Eq. 14 of the paper, that yields a repulsive force between two samples whose embeddings are closer than a threshold value de. At the same time, a second loss function, Eq. 15 of the paper, pulls the samples towards the average latent vector Wang around which the best quality samples are located.
By design, the Langevin algorithm tries to increase the samples embedding pairwise distances de (e))e)), up to given threshold de, while simultaneously pulling the samples towards Waeg. and therefore minimizing pairwise latent distances |w,- ws). Figure 9a and Figure 9b show the evolution of average pairwise embedding and latent distances, respectively, for Ni = 10k identities, up to Niter = 50 iterations and for five different values of d5. Figure 9a shows that all five ensembles start with the same average pairwise embedding distance (d = 1.47, this quantity increase very quickly to reach a plateau after approximately 10 time-steps. As expected, bigger values of de lead to higher plateau, with the exception of di = 1.6 where some sort of saturation phenomenon seems to occur. While average the embedding distance stays almost constant after this swift onset, the average pairwise latent distance (Iw,- wol) continue to decrease a much slower pace, as seen in Figure 9b, indicating that the ensemble slowly clusters itself around Wavg-
These dynamics are driven by the balance between the repulsive embedding contact force, whose evolution is shown in Figure 9c, and the attractive latent pull-back force, whose evolution is shown in Figure 9d. We see that, when the samples are randomly distributed, their overlap in embedding space is quite significant leading to very high contact forces in the first iterations. The samples re-arrange themselves to minimize contact interactions quite rapidly and, after a certain number of iterations, the contact interaction reaches a plateau. After this plateau reached, samples continue to be pulled towards
16

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
Table 5: Hyperparameters for the Langevin, Dispersion and DisCo algorithms and their default values.
Default
Description
1.0
Langevin and Dispersion bulk viscosity
b
1.0
Langevin embedding contact force coefficient
d
1.4
Langevin embedding distance threshold
kw
0.1
Langevin latent pull-back coefficient
no
0.01
Langevin random force magnitude
0.3
Langevin variable time-step coefficient
Na
100
Langevin number of iterations
ky
1.0
Dispersion latent contact force coefficient
d
12.0
Dispersion latent distance threshold
1.0
Dispersion identity pull-back coefficient
k.
1.0
Dispersion latent pull-back coefficient
no
0.01
Dispersion random force magnitude
8t
0.05
Dispersion fixed time-step
Nator
20
Dispersion number of iterations
Eo
0.2
Dispersion initial symmetry breaking noise
Ao
1.0
DisCo covariates sampling scale
Table Runtime for Langevin identities generation with Nater 100 on system with a single NVIDIA RTX 3090 GPU.
Nid
Runtime
1k
2.2h
2k
3.5h
4k
5.4h
6k
10.2h
8k
12.8h
10k
18.2h
20k
31.5h
30k
49.2h
Table 7: Runtime for DisCo identities variations generation with Nater 30 and Nis 30k system with a single NVIDIA RTX 3090 GPU.
N Wear
Runtime
16
50.0h
32
95.8h
64
183.7h 7h
17

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
Wavg by the latent pull-back force, which slowly decays as seen Figure 9d. Figure 9e and Figure 9f show this decay for varying number of identities, by plotting (Iw, -wsl) in function of Nia and Niter, respectively.
F. Striet Inter-Class Threshold Constraints
Another interesting metric on the performance of the algorithm is the proportion of pairwise distances that are above a certain inter-class threshold d' This can be computed by counting each occurrence where this condition is met, for each possible pair of identities (a, b)
ce (ea. ex) < dies v(a,b), a > b}1
Piet
Npairs
(26)
where we set a > b to avoid double counting and where Npairs If we set this threshold equal to the Langevin repulsion distance threshold, dffet = dř, we can evaluate the ratio of the number of contacts over the number of pairs
Naa(Na is the number of possible pairwise interactions. 2
Pa Hd (ca: eb) < de Vla b), a > bb
N Noomaans
(27)
N Apairs
N Npairs
Figure 9g shows the dynamical evolution of this quantity after certain number of iterations Nater, for different values of di- We see that for very high values of dē this value is close to so almost every identity is in contact with every other one. For other values we can compute the average number of contacts per identity
2 N Noontacts Nid
2 Po N Wpoirs 2po (Nia 1):
(28)
Nid
where we have added a factor of two to account for each identity in the pair in contact. We see in Figure 9g that, for de 1.4 and Ni = 10k, this ratio stabilizes around م 0.03 meaning that each identity is on average in contact with approximately 300 other classes. More generally, the ratio Pict is the FMR of the reference FR backbone evaluated on the synthetic dataset and the Langevin algorithm tends to iteratively decrease this value until it reaches a plateau.
would be interesting to know the maximal number of identities N which satisfy the constraint of > di we can extract from a given Langevin ensemble. For this purpose, we devise a simple erosion algorithm that iteratively removes identities from the ensemble until the constraint satisfied for all pairs of remaining identities. This naive algorithm removes the identities with the maximal number of contacts first, one by one, until no contact is left. Figure 9h shows values of Natrict for Langevin ensembles of Naa 10k and for different values of dict. In this case, we set the contact distance threshold slightly bigger than the threshold, dē = 1.1 df and plot Natrier against computational wall-time. For comparison, we also plot the performance of the Reject algorithm and see that Langevin with erosion yields significant numerical advantage, at least with our implementations.
G. Repulsion Distances Threshold and DisCo
We now interested in optimizing the statisties of the synthetic datasets generated by our algorithms. In particular, we study here the influence of the repulsion distance thresholds dÿ and dy, for Langevin and Dispersion, respectively, We would also like to investigate the impact of the DisCo scale parameters λg
As explained in section 3 of the paper, the Langevin and Dispersion granular-like losses functions introduced in Eq. 14 and Eq. 20, respectively, are designed to repulse samples that are too close in the embedding and latent spaces. The Langevin algorithm therefore controls the inter-class embedding pairwise distances via the parameter de as illustrated by the histograms in Figure 10a. One elearly sees that increasing the value of of shifts the peak of the distribution toward bigger pairwise embedding distances. For the largest values showed in this plot, the peaks near de T similar to the real-world data shown in Figure 10c.
2
Similarly to Langevin, the Dispersion algorithm controls the intra-class latent pairwise distances via the parameter dW Since increasingly larger latent pairwise distances tend to produce increasingly dissimilar images, we expect bigger values of d to widen the autra-class pairwise embedding distances distribution and shift its peak towards larger values. This is indeed what we observe the histograms in Figure 10b, where the pairwise embedding distances distributions of Dispersion
18

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
1.58
11
1.2
1.2
1.56
1.4
1.4
(d [Cas eba
1.54
1.6
8
1.6
1.52
7
1.5
6
1.48
5
1.46
3
(a)
(b)
10
20
30
40
50
0
10
20
30
40
50
Niter
Niter
45
45
100
1.2
1.2
1.4
1.4
1.6
1.6
10
(c)
(d)
10
20
30
40
50
10
20
30
40
50
Niter
Niter
11
11
Nid
Niter
10
10
26
9
4k
9
20
Uiuia m0
6k
8
100
124
6
204
5
40k
5
4
4
(c)
3
3
40
80
100
103
10
Niter
N id
pairs
10"
10a
45
1.2
L: 1.2
R: 1.2
1.4
1: 1.3
103
R: 1.3
10-1
1.6
Metrict
1:
R: 1.4
[df (a)
R: 1.5
102
10-2
10
(g)
10
(h)
103
10a
Nater
Lestal [$]
Figure 9: Dynamics of Langevin ensembles: Evolution of the average pairwise embedding distance and the average
pairwise latent distance b. Evolution of the average embedding contact force and average latent pull-back force d. Average pairwise latent distance for different Nia and pairwise latent distance for Noa identities after Nater iterations f. Proportion of pairwise embedding distances below the threshold df for 10k identities g. Compute time used to generate Nstrict dissimilar identities for Langevin (L. with erosion and for Reject (R) and with different de values h.
19

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
000
0000
001
45
40
0.0
0.0
0 133
12.0
@ 1.3
12.0
1.4
14.0
G 1.5
16.0
de
18.0
d
18.0
3m
5m
3x
5m
34
8
8
(a)
(b)
10000O
T00000
3094
1000
1000
100
2 FPHQ
MaliPIE
3 MuliPIE
2 0.5
@
L.5
d
de
T
3x
5
(c)
(d)
Figure 10: intra-class @ and inter-class @ embedding distance histograms of synthetic datasets generated with the Langevin, Dispersion and DisCo algorithms as well as genuine real-world data. In a, the Langevin repulsion distance of is shown to affect the inter-class distance statistics. shown to affect the intra-class distance statistics. In genuine datasets histograms are shown for comparison. The vertical dashed lines, denoted "FMR", show the distances for the reference FR backbone evaluated UJBC dataset for three different FMR values.
and d, the Dispersion repulsion distance d and DisCo scale A are
20

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
Table 8: Influence of Langevin repulsion distance threshold di on FR accuracy.
dj
N
iter
LFW
CPLFW
CALFW
CFP
AgeDB
Average
1.2
50
90.82
59.73
80.97
60.66
72.65
72.97
1.3
50
92.78
63.32
83.33
63.51
74.37
75.46
1.4
50
94.43
65.08
85.13
65.6
76.97
77.44
1.5
50
94.67
65.98
85.1
67.83
75.67
77.85
1.6
50
92.73
65.15
82.28
67.3
73.02
76.10
Table 9: Influence of Dispersion repulsion distance threshold de on FR accuracy.
dg
Niter
LFW
CPLFW
CALFW
CFP
AgeDB
Average
8.0
20
84
59.57
70.62
59.99
63.57
67.55
12.0
20
94.43
65.08
85.13
65.6
76.97
77.44
14.0
20
96.48
65.62
89.53
67.9
83.93
80.69
16.0
20
96.15
62,47
88.43
67.54
82.83
79.48
18.0
20
95.25
63.3
87.12
66.41
81.2
78.66
20.0
20
54.33
50.07
49.83
55.04
51.68
52.19
ensembles, created with wide range of value of the parameter db are shown. We observe that small value of dy 8.0 lead to very narrow distribution, while bigger values greatly shift the peak towards larger de values.
We are interested on the accuracy of the FR models trained such synthetic ensembles, for different values of these two parameters. Table 8 and Table 9 show such survey, for five different values of de and six different values of d'y As we can see from the first table, the values di = 1.4 and de 1.5 of the Langevin repulsion distance threshold yield the best FR performance, while further increasing this parameter to di = 1.6 degrades the overall performance. As we can see by comparing peaks of pairwise inter-class distances histograms of genuine and synthetic data in Figure 10, these best performing parameters values yield distributions close to genuine data from the FFHQ and MultiPIE datasets. Keeping the fixed value de 1.4, see from the second table that the best performance is achieved with the parameter value de 14.0. It is interesting to note that the values de 12.0, d' 16.0 and d' 18.0 yields almost similar performance figures, but with widely different pairwise distances distributions.
Table 10: Influence of DisCo parameter Ao on FR accuracy.
dg
Ao
LFW
CPLFW
CALFW
CFP
AgeDB
Average
12.0
0.0
94.43
65.08
85.13
65.6
76.97
77.44
12.0
0.5
95.65
69.6
87.45
68.54
80.32
80.31
12.0
1.5
96.2
73.25
87.7
73.89
80.73
82.35
12.0
1.0
96.6
74.77
87.77
73.89
80.7
82.75
14.0
1.5
97.22
75.85
89.48
78.76
83.32
84.93
16.0
2.0
94.82
65.18
85.00
72.39
77.27
78.93
In section 3 of the paper, we introduced a modification of Dispersion, the DisCo algorithm, that initialize the intra-class variations with random linear combination of the covariates vectors introduced in (Colbois et al., 2021). This modification introduced to further increase the intra-class variability of the final datasets and the implementation introduces an additional parameter Åo that controls the scale of the random weights A E [->o, >o]. Table 10 shows the performance of five such ensembles compared the pure Dispersion baseline Ào 0. The first three datasets use latent repulsion distance of d3 12.0 while the fourth and last further increase this parameter. As demonstrated by the results in this table, this algorithm yields significant performance improvement compared the original one using purely random initialization, at least up to value of A = 1.5. This gain is of between two and four percents for the LFW, CALFW and AgeDB benchmarks, which is already quite significant, Where the DisCo yields the biggest advantage however is on the CPLFW and CFP benchmarks, where improvements of up to twelve percents are observed. This indicates that the DisCo algorithm helps generates more diversity, and that it is helpful in mitigating the relatively poor performances StyleGAN2 ensembles on pose henchmarks.
21

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
Table 11: Influence of training set repulsion on FR accuracy.
dx
d
LFW
CPLFW
CALFW
CFP
AgeDB
Average
0.0
1.4
94,43
65.08
85.13
65.6
76.97
77.44
0.6
1.4
94.97
65.15
85.62
65.19
78.42
77.87
0.8
1.4
94.95
66.28
86.5
65.24
78.9
78.37
1.0
1.4
94.77
65.23
86.22
66.61
78.33
78.23
1.2
1.4
94.95
65.08
85.97
65.56
79.07
78.13
1.3
1.4
94.58
65.18
86.37
65.91
79.85
78.38
1.4
1.4
94.08
65.62
85.37
66.93
77.12
77.82
1.5
1.4
91.85
64.58
81.58
66.37
71.72
75.22
H. Identity Leakage from Generator Training Set and Mitigation
1.2
0.6
0.8
1,1
1.2
1.3
0.9
0.8
de
0.6
0.8
1
1.2
1.4
de
3xT
5x
3x
8
8
Figure 11: Effect the df, parameter on synthetic to training set distances distribution: On the left, effect on the minimum value, the right: effect the distribution tail.
While we have generated large quantity of synthetic data, it is important to emphasis that the generator is trained on genuine data. We want to verify that the synthetic data generated by our algorithms is sufficiently far from the original training set data and, if it is not the case, find appropriate mitigation techniques to control potential training set leakage. has been shown that, despite the care taken by the authors of previous methods such as DCFace (Kim et al., 2023), their method still generates significant number of samples that are very close, not indistinguishable, from images from the generator training set (Shahreza Marcel, 2024).
Since GANs- generators less prone leakage than diffusion based models (Carlini al., 2023), developed method based on GAN models. Another advantage of our method is that we can further mitigate the problem by introducing an additional interaction that repulses the synthetic samples away from the training set samples. This is quite easily implemented Langevin by adding a loss function similar to Eq. 14 that repulse synthetic samples away from generator training set samples embeddings
No.
(de - dan)²
de of d
C Ar
(0
> d
(29)
as
d de (e (w)). FAA)
where Ntr is the number of training set samples, where = 1... Nor and where EA is the embedding of the A-th sample.
The total loss function for the Langevin algorithm Eq. 18 changes to
c L. L CE
(30)
We introduced new parameter dir which is similar to the pairwise embedding repulsion distance di but for granular interactions between synthetic and genuine samples. This parameter has the effect shifting the tail the genuine-synthetic
22

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
Figure 12: Top row: Selection of the most similar images among the 64 nearest embedding pairs between the original FFHQ (top) and images generated with Langevin algorithm with df.- = 0.6 and Naa = 10k (bottom). Note that (similar to the left image) there are several images of children among similar images, which are hard compare and therefore we focused on adults for this study.
U
N
Figure 13: Top row: Selection of the most similar images among the 64 nearest embedding pairs between the original CASIA-WebFace (top) and synthetic images in the DCFace dataset (bottom).
embedding distance histogram, as shown in Figure 11 thus controlling the most problematic images in term of embedding distance. Table 11 reports the effect of this new parameter on accuracy, for several values of de We observe that, surprisingly, increasing the value df, slightly increases FR accuracy, and the biggest value considered de 1.3 performs slightly better than no training set repulsion all.
To verify the leakage of identity in the generated datasets, similar to (Shahreza & Marcel, 2024), we compare all images in the synthetic dataset with all images in the training set of generator model. Figure 12 and Figure 13 illustrate the most similar pairs of genuine and synthetic images with the smallest embedding distance for both our method and DCFace, respectively. While it difficult to conclude the leakage in our synthetic dataset, we can see that DCFace has almost memorized several samples, indicating a serious identity leakage in the DCFace dataset.
Synthetic Dataset Scaling and Face Recognition Performance
Having performed a detailed analysis of the Langevin, Dispersion and DisCo algorithms as well as the influence of their respective parameters, we would like to also see if we can scale our approach to bigger synthetic datasets. Table 12 shows the result of this survey. In this table, we also show the effect of compounded datasets, datasets that have the same biometric references, created with Langevin, but different variations. In particular, several datasets are compounded with their reference Langevin ensemble, thus adding one variation per class, denoted by check-mart in the Ref. column of the table. As we see, this offers very limited performance increase but we decide keep this for bigger datasets as the data has to be computed anyway.
Scaling up to Nia 30k however offers an impressive performance improvement, in particular with DisCo variations, with an impressive average score of 88.40, almost reaching the performance of diffusion based methods. For N = 30k we also study the influence of the number of variations, showing that Near = 64 yields the best results. Surprisingly, for Near 128 the performance drops very significantly, by almost twenty points. This might be explained by the lack of
23

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
Table 12: Influence of the number of identities, number of variations and variation-creation method on FR accuracy.
Method
Nid
Ning
Ref.
Nour
%
Ao
LFW
CPLFW
CALFW
CFP
AgeDB
Average
640 000
64
12.0
94.43
65.08
85.13
65.6
76.97
77.44
10'000
650 000
V
64
12.0
94.38
65.75
86.03
65.51
77.3
77.79
480'000 000
16
12.0
90.65
64,42
80.12
63.07
70.18
73,69
Langevin + Dispersion
30'000
960' 000
32
12.0
94.83
66.25
84.3
65.51
76.1
77.40
1'620'000
64
12.0
97.98
71.57
91.7
72.76
88.87
84.58
10'000
640'000
74.77
87.77
80.7
82.75
1'300'000
V
12.0
1.5
98.22
77.37
91.93
78.93
87.32
20'000
1'300' 000
v
14.0
1.5
98.53
81.17
92.93
83.56
92.32
89.70
Langevin DisCo
1'950' 000
V
12.0
1.5
98.37
79.53
92.58
80.71
90.82
88.40
3'870' 000
128
1
12.0
1.5
86.15
61.98
70.35
68.23
59.87
69.32
1'470' 000
48
14.0
1.5
98.35
81.32
93.33
83.37
92.05
89.72
30'000
1'950' 000
<
64
14.0
1.5
98.97
81.52
93.95
83.77
93.32
90.31
0000
2'600'000
000
<
64
12.0
1.5
98.33
78.25
92.27
79.31
89.43
87.32
50' 000
3'250' 000
V
64
12.0
1.5
98.47
79.88
92.57
79,93
90.33
88.24
Langevin Covariates
10'000
180'000 000
17-1
77.68
59.2
64.07
61.07
52.55
62.91
References created with Langevin are compounded variations FR model training.
Table 13: Bias evaluation of trained face recognition models on the Racial Faces in-the-Will Wild (RFW) dataset
Dataset
Type
Caucasian
Asian
Indian
African
Avg.
Std.
SynFace (Qiu et al., 2021)
GAN
65.60
64.48
61.48
57.27
62.21
3.01
SFace (Boutros et al., 2022)
GAN
75.68
69.7
70.63
66.23
70.56
2.08
IDNet (Kolf al., 2023)
GAN
70.03
64.22
65.77
59.3
64.83
2.89
GANDiffFace (Melzi et al., 2023)
GAN
76.32
72.85
72.45
66.48
72.03
3.00
ExPaceGAN (Boutros et al., 2023b)
GAN
65.25
65.40
64.25
57.97
63.22
3.28
DigiFace (Bae al., 2023)
Computer Graphics
71.93
68.30
69.02
64.8
68.51
1.93
IDiff-Facee (Uniform) (Boutros al., 2023a)
Diffusion
84.92
80.63
81.67
75.65
80.72
2.72
DCFace- 1.2M (Kim al. 2023)
Diffusion
90.08
82.97
87.07
80.97
85.27
2.66
Langevin-Disco-LL.M [ours]
GAN
89.17
82.95
85,43
81.42
84.74
1.81
inter-class latent repulsion, design choice made for simplicity, as explained previously.
J. Bias Evaluation
To investigate the bias in our synthetic dataset, we evaluate the performance of face recognition models trained with our dataset and compare with previous datasets on the Racial Faces in-the-Wii Wild (RFW) (Wang et al., 2019) dataset. As the results in Table 13 show, our method has achieved the best performance compared GAN-based synthetic datasets and comparable average performance with diffusion-based methods. In terms of bias, this table demonstrates that our method achieves the lowest standard deviation for recognition accuracy across different demographic groups, indicating the lowest bias in the trained face recognition model using our dataset.
K. Closing the Gap with Real-World Data
To better assess the performance of best datasets, Figure 14 shows the Receiver Operating Characteristic (ROC) curves for number of models trained synthetic and genuine data of the IARPA Janus Benchmark-B B (LJBBB (Whitelam et al., 2017) and IARPA Janus Benchmark-B (IB-C) (Maze et al., 2018) 8) datasets. As can be observed in this figure, our method outperforms all GAN-based synthetic datasets in the literature. In addition, our method achieves comparable performance with diffusion- based datasets for high values FMR; however, for low values of FMR, diffusion- better performance than our method. We should note that, as mentioned in section the paper, DCFace is generated with dual condition model trained on -aeeeee (Yi al., 2014) with identity labels, and therefore the generator model
methods achieve
24

---

Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion
ROC UB-B
ROC LJB-C
1
1
LU
0.9
0.9
0.8
0.8
(1- -1-FNMM FNMM
FNMR)
0.7
0.7
0.6
CASTA
0.6
CASIA MSCcleb
MSCcleb
WebFaceMM
WebFace-MM
0.5
DCFace-1 1.M
0.5
DCFace- 2M
Iif Face Uniform
Wift Face Uniform
Digif 1.2M
DigilFuze I, 1.2M
0.4
GANDiriece
0.4
GANDeffFcce
IDNet
WDet
0.3
SFace2
0.3
SFace2
SpuPace Langevin DisCoo (30k)
SyaPace
Langeria DisCo [30kc]
0.2
0.2
10-5 5
10
10
10-2
10
100
10
10
10
10
10
10
FMR
FMR
Figure 14: ROC curves for models trained on synthetic datasets, benchmarked on the IJB-B and IJB-C datasets.
leverages' the identity information in the CASIA-WebFace dataset (Shahreza & Marcel, 2024). In contrast, our method is based on pretrained generative model, which is trained unlabeled face images from the FFHQ dataset. In addition, as discussed in Appendix H, there is identity leakage in the DCFace dataset. Compared to FR models trained with real data, the ROC curves show that there is still a gap between training with real and synthetic datasets. Nevertheless, the promising improvement achieved by our method for GAN-based models reveals potential in generative models and training FR with synthetic data, which require further research in the future.
L. Future Directions
While we showed that our algorithms can yield very good synthetic face datasets, many improvements are still to be made, Most importantly, our method crucially relies on a reference off-the-shelf FR model. In some sense, the datasets generated with our method iteratively learns, by stochastic gradient descent, from the reference FR model, and therefore quite certainly learn its biases as well. A possible fix that could mitigate, at least partially, this fundamental issue is to use several reference models in parallel, simply by adding more embedding losses to the algorithms.
Fundamentally, the problem of generating complex synthetic data, that has similar characteristies that some genuine reference, is chicken-and-egg problem. It could be perhaps construetive to reformulate this problem in a way that makes it clear that the synthetic data will always depends to some extent on genuine priors, and rather insist on developing procedures giving guaranties that the original data is at least hard to recover and that the synthetic dataset gives a good enough representation of reality. A few steps in this direction were performed in this appendix, possibly also hinting at some interesting research directions toward a much deeper problem: generalization of deep neural networks.
We would like stress that, while large part of this work was devoted to optimization of these algorithms, they remain computationally expensive. For each sample an image must be generated and its embedding computed, twice per iteration with one backward pass, which is computationally expensive the data goes through intermediary image space of very high dimensionality. On the other hand the only function we are interested in is an identity aware metric d'y on the latent space of a given generator. It is certainly possible to learn such function to a relevant degree of accuracy and to use this instead of the full chain, at least for bootstrapping. Alternatively, while technically challenging, is certainly possible to prune connections and weights in the W + generatoor-fctuuc-ectttttto chain, which would vastly improve efficiency.
'Since DCFace is trained on CASIA-WebFace WebFace dataset with identity labels, the recent Synthetic Data for Face Recognition (SDFR) Competition held in conjunction with the 18th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2024) disqualified all submissions based DCFace (Shahreza al., 2024). It is also shown in (Shahreza & Marcel, 2024) that DCFace has critical identity leakage from SeeFFee dataset.
25


Reviews:
[{"review_id": "ncAlf45HpW", "reviewer": "Reviewer_NRWj", "summary": "In this work, they introduce a new method, inspired by the physical motion of soft particles subjected to stochastic Brownian forces, allowing us to sample identities distributions in a latent space under various constraints. They also introduce three complementary algorithms, called Langevin, Dispersion, and DisCo, aimed at generating large synthetic face datasets.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The claims presented in the content are clear, and the experiments support the conclusions. However, despite their clarity, I have some concerns regarding the novelty of the proposed claims.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I have conducted a corresponding review of the theory, but its theoretical description is somewhat rigid and lacks smooth transitions, which has made my review more challenging.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "I find the contribution to the field to be relatively modest, as the proposed method may not be applicable to more generalized datasets.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The first dataset generation work based on physics-inspired methods.\n2. Achieved better results compared to other methods.\n\nWeaknesses:\n1. The overall description of the paper is not very clear. For instance, how the motivation is derived from physics evidently requires more elaboration, as this is the most crucial part.\n2. Since I specialize in the theory of diffusion models and am very familiar with Langevin dynamics, I believe that the physical approach in the paper merely applies Langevin dynamics to the given task. This significantly weakens the originality of the main motivation.\n3. The loss design is not novel. For example, in Equations (15) and (16), the authors could review more papers on face models to find more effective loss function designs.", "other_comments_or_suggestions": "### No rebuttal for my concern", "questions_for_authors": "No rebuttal was given, so l lean to reject.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Discrimination / Bias / Fairness Concerns", "Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"], "ethical_review_concerns": "The generation of facial datasets typically requires scrutiny, as it involves the potential misuse of portrait rights."}}, {"review_id": "ry5zyT0AIu", "reviewer": "Reviewer_vznF", "summary": "n this paper, the authors introduce a physics-inspired method to generate large synthetic face datasets for training face-recognition models. Their core idea is to treat each latent representation as a “particle” and let these particles repel each other in the embedding space (via a “Brownian identity diffusion” approach), ensuring that each synthetic identity is sufficiently distinct while still maintaining realistic appearance. They propose three algorithms—Langevin, Dispersion, and DisCo—to control inter-class diversity (spacing between different identities) and intra-class variation (differences among images of the same identity). By training a face-recognition model on these generated datasets, they claim to outdo earlier GAN-based methods, and even rival some diffusion-based approaches, all while preserving more privacy.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Some of their evidence is compelling—particularly the performance comparisons against older GAN-based methods and the step-by-step ablation results that show how Langevin and Dispersion can boost coverage in latent space. However, a few claims feel less rock-solid. For example, they assert that GANs are definitively more private than diffusion models, yet the paper mainly references prior studies instead of conducting a thorough memorization or leakage check themselves. The discussion of “Brownian identity diffusion” as a surefire way to avoid “jamming” also seems a bit hand-wavy, since there’s limited empirical proof that high-dimensional jamming is a real hazard or that their random force definitively fixes it.", "methods_and_evaluation_criteria": "Yes, they focus on training large-scale face recognition with synthetic data, and the benchmarks (LFW, CA-LFW, CFP-FP, etc.) are standard for face verification. They also measure how well synthetic identities spread out in embedding space, which directly relates to how reliably a model can distinguish between them.", "theoretical_claims": "There aren’t formal theorems to check here, only physics-based arguments likening their latent-space approach to Brownian motion and granular mechanics. There’s no rigorous proof that needs verification in the usual mathematical sense. Rather, they present a heuristic connection—no step stands out as a “proof” that could be right or wrong in that classical, theorem-based way.", "experimental_designs_or_analyses": "I checked their experiment layout: they generate synthetic face sets, train a standard face-recognition model on each, then compare scores on popular benchmarks. This is a straightforward approach and largely appropriate, since it measures the core question: “Can synthetic data match or outperform existing sources for training face recognition?” One notable concern, though, is that they assume their reference FR model (used to measure inter-identity distances) doesn’t bias the dataset generation. If the generator is overfitted to that specific embedding, it might exaggerate gains on tests that are also partial to similar embeddings. While it doesn’t invalidate the results outright, it’s something to keep in mind when interpreting their reported accuracy boosts.", "relation_to_broader_scientific_literature": "They’re building on a growing theme of training face-recognition models with synthetic data—particularly methods that try to systematically traverse or manipulate generative latent spaces (e.g., SynFace, SFace, and Syn-Multi-PIE). They depart from older GAN-based setups by taking a “physics-inspired” angle: rather than just random or partially guided sampling, they push synthetic identities away from each other in the embedding space using spring-like forces, akin to granular mechanics. This sets them apart from, for instance, DreamBooth-style diffusion methods (DCFace or IDiff-Face), which often face privacy concerns around training-data leakage. So they’re essentially combining older ideas—latent editing and identity separation—with a fresh “Brownian motion” twist to make synthetic datasets more diverse while still being feasible for face recognition.", "essential_references_not_discussed": "Yes. Beyond the cited diffusion-model leakage papers, there’s also prior work directly evaluating whether GANs might memorize and replicate training faces—see, for example, “Evaluating GANs via Dual- and Triple-Generation” (ECCV 2020) and Carlini et al. (2023), which compare data-extraction risks between GANs and diffusion. Including these would highlight potential pitfalls in assuming one approach is fundamentally “private.” Also, discussing recent latent-manipulation methods like GANDiffFace (Melzi et al. 2023)—which systematically ensure identity separation—would broaden the conversation on spacing identities in latent space. There are also works on discussing using GAN generated data to understand FR models (Liang et al. 2023)\n\n\nA. B. Some Author et al. “Evaluating GANs via Dual- and Triple-Generation,” Proceedings of ECCV, 2020.\nN. Carlini, J. Hayes, M. Nasr, et al. “Extracting Training Data from Diffusion Models,” in 32nd USENIX Security Symposium (USENIX Security 23), 2023.\nP. Melzi, C. Rathgeb, R. Tolosana, et al. “GANDiffFace: Controllable generation of synthetic datasets for face recognition with realistic variations,” arXiv preprint arXiv:2305.19962, 2023.\nLiang, H., Perona, P., and Balakrishnan, G. “Benchmarking Algorithmic Bias in Face Recognition: An Experimental Approach Using Synthetic Faces and Human Evaluation.” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023", "other_strengths_and_weaknesses": "I appreciate the paper’s effort to blend concepts from physics (Brownian motion and granular mechanics) into synthetic face dataset generation—this is a neat twist on what’s otherwise a fairly saturated space. They also do a thorough job of comparing different hyperparameter settings, showing how to tune their “repulsive forces” for best effect. That said, the writing occasionally slips into heavy theoretical exposition, which some readers might find confusing or tangential. A more direct, plain-spoken approach would help clarify the motivation behind “Brownian identity diffusion.” Furthermore, while they position GANs as a privacy-friendlier alternative to diffusion, they don’t do much to measure or prove that assertion directly. Overall, though, the paper proposes a novel angle for crafting more identity-rich, diverse datasets, and the results suggest real promise for training practical face recognition systems.", "other_comments_or_suggestions": "Please refer to the previous section.", "questions_for_authors": "Please refer to the previous section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZIUCxDawpJ", "reviewer": "Reviewer_3ttQ", "summary": "The authors propose an approach to generate synthetic face images, by leveraging a GAN-based backbone, coupled with novel Langevin and Dispersion algorithms, together used as DisCo, wherein both inter-class and intra-class diversity in ensured by using a physics informed formulation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "While I am not an expert in the particular field of synthetic face generation for FR algorithms, the claims made in this paper are clearly presented, in my understanding. The proposed approach, and the associated algorithm are clearly explained, and the experiments are well motivated and presented.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are well presented and consistent with the literature presented, to the best of my knowledge.", "theoretical_claims": "The theoretical formulation, motivating the inter-class and intra-class sample diversity by means of repulsive and attractive forces is well formulated and presented clearly. However, it would be good for the authors to discuss some of the other works talking particularly about this, in the GAN space. In terms of analysis GANs and diffusion models’ image generation in terms of these forces, these have been prior works such as, for example, Franceschi et al. 2023, and Asokan and Seelamantula, 2023, which talk precisely about this repulsive/attractive nature of particle flow in GAN and diffusion models settings, while both Unterthiner et al., 2018 and Wang et al., 2019 were both worlds that initially formulated this via a loss function for GANs. Of course the settings are different, but the theory in this paper could certainly be made stronger by either leveraging, or referencing, existing literature that makes claims aligned with the paper’s setting. \n\n\n[1] Franceschi et al., Unifying GANs and score-based diffusion as generative particle models, NeurIPS 2023\n\n[2] Asokan and Seelamantula, GANs Settle Scores, arXiv 2023\n\n[3] Unterthiner et al., Coulomb GANs: Provably optimal Nash equilibria via potential fields, ICLR 2018\n\n[4] Wang et al., Improving MMD-GAN Training with Repulsive Loss Function, ICLR 2019", "experimental_designs_or_analyses": "While I am not an expert in the space of FR, the GAN base setting of the experiments, the design and evaluation framework all appear sound to me.", "relation_to_broader_scientific_literature": "To the best of my understanding, this manuscript is relevant to the GAN and FR literature, and the algorithms provided, although in the context of generating synthetic datasets for FR, can be leveraged in other settings as well, and are therefore relevant to the broader community.", "essential_references_not_discussed": "Please see my response to **Theoretical Claims**.", "other_strengths_and_weaknesses": "Please see my response to other questions above.", "other_comments_or_suggestions": "**Impact statement:** It appears that I couldn’t find an impact/ethics statement, even the default one that ICML suggests, in the manuscript. I found this ironic, for a paper targeting, particularly, the privacy and ethical concerns of face recognition models. I do not wish to flag this paper for an ethics issue in this regard because I dont see any such glaring issues, but it would be good for the authors to acknowledge the impact of the proposed algorithm in the context of privacy concerns of FR models in their impact statement. \n\n**Minor bug fixes — Typo: L347C1: … datasets are **accurately** calculated…", "questions_for_authors": "Please see my response to other questions above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper was reviewed by 3 experts in the field who provided detailed suggestions. The reviewers’ recommendations were divergent, with reviewer 3ttQ recommending Accept (though, with low confidence), reviewer vznF recommending Weak Reject, and reviewer NRWj recommending Reject. 

All three reviewers noted the paper's strengths in proposing a novel, physics-inspired angle for synthetic dataset generation, which was clearly explained along with its associated algorithms and experiments, reviewer vznF calling the approach a "neat twist on what's otherwise a fairly saturated space". The results also acknowledged the performance gains over prior GAN-based methods.

The reviewers raised some concerns and suggestions for improvement, including the clarity of the physics-based motivation and theoretical exposition, which could be overly dense (NRWj, vznF). The novelty of the approach was questioned in relation to standard Langevin dynamics and existing loss functions (NRWj), and the claims regarding privacy advantages required more direct empirical evidence and comparison to relevant literature (vznF). 

I read the reviews. A rebuttal was not submitted in time, which may have affected some of the final recommendations (two reviewers downgraded their recommendations). I do, however, appreciate the problem the paper seeks to solve and agree that its solution is interesting and likely useful and so recommend weak accept, accordingly.
Decision: Accept (poster)