Vec2Face: Scaling Face Dataset Generation with Loosely Constrained Vectors
Authors: Haiyu Wu
Jaskirat Singh
Sicong Tian
Liang Zheng
Kevin Bowyer
Venue: ICLR 2025 Poster (2025)
Keywords: Identity privacy
Synthetic face dataset generation
Face recognition
Image generation
Primary area: generative models
TL;DR: We proposed Vec2Face, a model for generating scalable synthetic face recognition datasets. The generated HSFace10K achieves SOTA performance and outperforms same-scale real datasets on CALFW, IJBB, and IJBC test sets.

Abstract: This paper studies how to synthesize face images of non-existent persons, to create a dataset that allows effective training of face recognition (FR) models. Besides generating realistic face images, two other important goals are: 1) the ability to generate a large number of distinct identities (inter-class separation), and 2) a proper variation in appearance of the images for each identity (intra-class variation).
However, existing works 1) are typically limited in how many well-separated identities can be generated and 2) either neglect or use an external model for attribute augmentation. We propose Vec2Face, a holistic model that uses only a sampled vector as input and can flexibly generate and control the identity of face images and their attributes. Composed of a feature masked autoencoder and an image decoder, Vec2Face is supervised by face image reconstruction and can be conveniently used in inference. Using vectors with low similarity among themselves as inputs, Vec2Face generates well-separated identities. Randomly perturbing an input identity vector within a small range allows Vec2Face to generate faces of the same identity with proper variation in face attributes. It is also possible to generate images with designated attributes by adjusting vector values with a gradient descent method. Vec2Face has efficiently synthesized as many as 300K identities, whereas 60K is the largest number of identities created in the previous works. As for performance, FR models trained with the generated HSFace datasets, from 10k to 300k identities, achieve state-of-the-art accuracy, from 92\% to 93.52\%, on five real-world test sets (\emph{i.e.}, LFW, CFP-FP, AgeDB-30, CALFW, and CPLFW). For the first time, the FR model trained using our synthetic training set achieves higher accuracy than that trained using a same-scale training set of real face images on the CALFW, IJBB, and IJBC test sets.

Published as a conference paper at ICLR 2025
VEC2FACE: SCALING FACE DATASET GENERATION WITH LOOSELY CONSTRAINED VECTORS
Haiyu Wu', Jaskirat Singh?, Sicong Tian', Liang Zheng,, Kevin W. Bowyer' l University of Notre Dame, The Australian National University." 3 Indiana University South Bend (hwu60nd. edu, jaskirat singh edu, tiansic cocciiuu.. (uu)). (liang. zhengganu.edu, kwbēnd.edul
ABSTRACT
This paper studies how to synthesize face images of non-existent persons, to create a dataset that allows effective training of face recognition (FR) models. Besides generating realistic face images, two other important goals are: 1) the ability generate a large number of distinct identities (inter-class separation), and 2) a proper variation in appearance of the images for each identity (intra-class variation). However, existing works 1) are typically limited in how many well-separated identities can be generated and either neglect attribute augmentation. We propose Vec2Face, a holistic model that uses only a sampled vector as input and can flexibly generate and control the identity of face images and their attributes. Composed of feature masked autoencoder and an image decoder, Vec2Face is supervised by face image reconstruction and can be conveniently used in inference. Using vectors with low similarity among themselves as inputs, Vec2Face generates well-separated identities. Randomly perturbing input identity vector within small range allows Vec2Face to generate faces of the same identity with proper variation face attributes. It is also possible generate images with designated attributes by adjusting vector values with gradient descent method. Vec2Face has efficiently synthesized as many as 300K identities, whereas 60K is the largest number of identities created in the previous works. As for performance, FR models trained with the generated HSFace datasets, from 10k 300k identities, achieve state-of-the-art accuracy. from 92% 93.52%, on five real-world test sets (i.e., LFW, CFP-FP, AgeDB-30, CALFW, and CPLFW). For the first time, the FR model trained using our synthetic training set achieves higher accuracy than that trained using same-scale training set of real face images on the CALFW, IJBB, and IJCC test sets.
external model for
INTRODUCTION
We aim synthesize face images in a way that enables large-scale training sets for FR models,
which have the potential to address privacy issues arising with web-scraped datasets of real face
images (DeAndres-Iee Tame et al. 2024 Melzi et al. 2024, Shahreza et al., 2024). It is generally
recognized that good training set for FR should have high inter-class separability (Kim et al. 2023
Boutros et al. 2023a) and proper intra-class variation (Qiu et al. 2021 Boutros et al.) 2022a Kim
et al] 2023 Boutros et al. 2023a).
However, existing methods lack flexibility in controlling the generation process, leading to unsatis-
factory inter-class and intra-class results. On the one hand, (Boutros et al. 2023a) points out that
identities generated in (Qiu al.) [20T [ootroo et al. 2022b a (024) have relatively low separability because identity generation controlled by either single coefficient (Deng et al. 2020) by hard class labels. On the other hand, while it is useful to employ identity features as a condition to increase
inter-class separability (Boutros et al.) 2023a Papantoniou al. 2024), it is non-trivial to increase
intra-class variation without the use of a separate model such as ControlNet (Zhang et al. 2023 or a
model for style transfer.
In light of the above, this paper proposes Vec2Face, a holistic model enabling the generation of large-scale FR datasets. In inference, the workflow of Vec2Face is similar to (Papantoniou al.

---

Published as a conference paper at ICLR 2025
ID vectors ID images
Perturbation degree to ID vectors
Figure 1: Example images generated by Vec2Face. From a random vector, we generate a face (ID image). We then perturb this vector with random values to generate diverse face images. Larger perturbation added to this vector results in larger dissimilarity to the ID images. The images in the frame are likely of the same people as the ID image.
2024: Boutros et al. 2023a), which uses random vector as input and outputs face image. But the interesting property of Vec2Face is that a small perturbation of the input vector leads to a face image of the same identity but with small changes in appearance, e.g., pose, age, and facial hair and that large perturbation of the input vector produces a face image of different identity (Fig. D. With this property, and because of the high-dimensionality (iee. 512-dim) of the input vectors, we can easily 1) sample a large number of well-separated identity vectors by controlling their similarity with each other be under certain threshold and 2) synthesize large number of face images with various attributes by perturbing the identity vectors within small range. Additionally, we can guide the image content at time of generation using gradient descent method efficiently generate faces with designated attributes, such as extreme pose angles and good image quality.
Using proper cosine similarity threshold (i.e., 0.3) to constrain the identity vector sampling, Vec2Face transfers the similarity between vectors to the similarity between generated images, resulting in large inter-class separability; intra-class variation also properly large because we control the variance of the noise for perturbation, the perturbed identity vectors have 20.5 similarity with the identity vector, exhibiting various appearances while preserving the identity on the generated verify these inter-class and intra-class variations are beneficial to the
images. Experimentally, FR model performance (in Section[4.4) Because of this, FR models trained on the synthesized 10K identities and 500K images demonstrate very competitive accuracy compared with the state-of-the-art synthetic databases of the same scale (Kim et al. 2023 Papantoniou et al. 2024). Importantly, we show that scaling up the synthetic training data 15 million images and 300K identities leads to further accuracy improvement. It is also worth noting that results on CALFW (Zheng et al. (017). UBBB WWhitelam et al.) 2017). and UBC (Maze al.) 2018) represent the first instance where a model trained with synthetic data yields higher accuracy than that trained with real data at the same scale. To our knowledge, this is the first time synthetic dataset can be this large and useful.
The training of Vec2Face aims to let a vector generate a face and let its perturbation magnitudes reflect face change magnitudes. To this end, in training we use vectors extracted from real face images by pre-trained FR model. These vectors are fed into a feature masked autoencoder architecture, after which an image decoder produces images similar the real ones at the pixel level. We summarize the main points of this paper below.
We propose Vec2Face, a face synthesis model that allows us efficiently synthesize large number of face images and well well-separated identities. Under appropriate similarity threshold, Vec2Face effectively increases inter-class separability and intra-class variation of synthesized face images.
With a generated dataset of 10K identities and 500K images, we achieve state-of-the-art FR accuracy measured the average five real-worke test sets compared with existing synthetic datasets. Our performance even surpasses the accuracy obtained by real training data on three test sets. We also report that scaling to larger synthetic training data leads further improvement.
RELATED WORK
Discrete class label conditioned image generation. VQGAN EEser et al. 202T), MAGE (Li et al.
2023). DIT (Peebles & Xie 2023), U-ViT' (Bao et al. 2022) MDT (Gao et al. 2023). and VAR (Tian
2

---

Published as a conference paper at ICLR 2025
et a 2024) control the object classes of generated images using discrete class labels condition. Therefore, these methods do not support the new class generation and are not applicable to the context of scalable face identity generation. Differently. Vec2Face uses continuous face features input and thus technically can generate infinite new identities given appropriate feature sampling.
Identity feature conditioned image generation. Using static feature as condition to guide the model to generate images for one identity is another approach. (Xiao et al.] [023]] CChn et al. 2023
Liet al. 2024b) use CLIP (Radford et al. (022)) to encode the face embedding, but this representation
is limited by CLIP's ability on facial encoding (Papantoniou et al., 2024). To improve the identity
fidelity of the generated images, (Wang et al. 2024; Valevski et al. 2023 Ye et al., 2023 Wood et al.)
202T; Papantonion et al.) 2024) use identity features extracted by FR model as condition. However,
an external model (e.g., ControlNet (Zhang et al. (023)) needed increase the attribute variation.
Different from them, Vec2Face generates images for one identity using dynamic perturbed identity
features that have high similarity value with their identity feature. This keeps a high identity fidelity
while providing way to control the attribute variation without using any external models.
Synthetic face image datasets. Existing approaches are primarily either GAN-based or diffusion- based. For the former. SynFace (Qiu et al. (02T), Usynthface (Boutros et al. 2022b), SFace (Boutros) et all 2022a), SFace2 (Boutros et al., 2024), and ExfaceGAN (Boutros et al. (223b) leverage pre- trained GAN models to generate datasets. These methods are less effective because the pre-trained
GAN models were not trained in an identity- manner. Departing from them, our GAN-base method explicitly uses face features in training, the generated images are identity-aware.
For diffusion model- -based methods, DCFace (Kim et al. 2023) uses a strong pre-trained diffusion model (Choi al. 202 and an additional style-transfer model increase the identity separability and attribute variation. Again, this pre-trained model is not identity-aware. IDiff-Face (Boutros et al.) 2023a) combines the pre-trained encoder and decoder from VQGAN (Esser et al.) 2021) with a latent diffusion model conditioned by identity features to control the separability of the generated identities. Although the latent diffusion model is identity-aware, the pre-trained VQGAN compromises such effect. Arc2Face (Papantoniou et al.) (2024) fine-tunes a pre-trained stable diffusion model (Rombach) atall 2022) on the WebFace42M dataset increase the generalizability on face. It combines the FR feature and CLIP (Radford et al. (021)) feature to control the identity of output images. The pose variation increased by adding an additional ControlNet (Zhang et al. 2023). However, neither the pre- traitrined stable diffusion model nor CLIP are optimized for face, and the slow processing speed strongly limits large scale dataset generation. Conversely, our method is specifically designed for face dataset generation and is efficient and identity- This allows our method to easily scale the dataset size to 15 million images and achieve state-of-the-art FR accuracy.
METHOD
3.1 VEC2FACE: ARCHITECTURE AND LOSS FUNCTION
As shown in Fig. Vec2Face consists of a pretrained FR model, feature masked autoencoder (fMAE), an image decoder, and basch-based discriminator (Isola et al. 2017 Yu et al. 2022).
Feature extraction and expansion. Given real-world face image, following (Boutros al. 2023a Papantoniou et al. 2024). we compute its feature using a FR model. To match the input shape of MAE,, the extracted image feature is projected and expanded to a 2-D feature map.
Feature masked auto-encoder (fMAE). Similar to MAE (He et al. 2022). the model is forced to learn better representations by masking out the input. Different from the MAE that masks an input image and introduces mask tokens to form the full-size image, the fMAE uses 2D feature map input and masks this feature map. To provide more useful information, the projected image feature is used to form the full-size feature map. Specifically, the rows in the feature map are randomly masked out by 2% before the encoding process, where 2% E NNrrrncated (max = 1, min = 0.5, mean = 0.75), and the projected image feature is filled in the masked out positions form the full-size feature map before being processed by the decoder. The structure in the Appendix.
Image decoder and patch-based discriminator. Finally, the new feature map is passed through simple image decoder to generatt/reconsstue the image. To improve image quality, a patch- based discriminator (Isola et al. 2017 Yu et al. 2022 is integrated to form GAN-type training.
3

---

Published as a conference paper at ICLR 2025
Inference
TM feature
FR Model
Feature
Feature map
Image
Ure + +ha t t CAN
fMA
map
decoder
Figure 2: Architecture of Vec2Face. Given real face image, compute its feature "IM feature" using a face recognition model. This feature is projected and expanded into a feature map, and the latter is processed by feature masked autoencoder (fMAE). Inside the fMAF, the rows in the feature map are randomly masked out before being processed by the encoder. The projected image feature then used form the full-size feature map before being processed by the decoder. Finally, the decoder outputs feature map. Outside the IMAE, a small image decoder reconstructs the pixels in the original image based the output of IMAE. During inference, Vec2Face accepts randomized vector and generates face image. This process has properties demonstrated in Fig.
Original training
Reconstructed
Original unseen
Reconstructed
Figure 3: Reconstruction results for training (left) and unseen (right) identities. Specifically, extract features of original images by using a FR model and feed them to Vec2Face for reconstruction. We observe that the reconstructed images still maintain the same identity while removing some image borders and backgro and transferring sketches into photo-realistic images.
Loss function. The training objective function includes an image reconstruction loss, an identity loss, perceptual loss and GAN loss. The reconstruction loss can be written as,
Lyee MSE Mgg)),
which compares the pixel-level difference between the reconstructed image M and the ground truth images I MMg- The identity loss is written as:
Lid
CosineSSimlrrtt((RRRLL.....)) FR(LMg)))
meaning that the features of the reconstructed image and original image extracted using the FR model should be close. In addition, the perceptual loss (Zhang al. 2018) is used to ensure the correct face structure at the early training stage and a patch-based discriminator (Isola et al. [2017] Yu et al. 2022) used to form the GAN loss to increase the sharpness of the generated images. The total loss
Liatal Lyee Lid + Lipips + LGAN.
Some findings of the loss function's effects are: 1) without Lia, Vec2Face has slightly degraded generation performance but still works reasonably well identity preservation; 2) without Lipips- the reconstructed images do not have clear face structures; 3) using LGAN at early training stage makes the model convergence difficult, so it is used only after 1,000 epochs. Examples of loss effects on image reconstruction are in Appendix A.2.
Image reconstruction results. Fig. shows that, for both seen and unseen identities, the generated images have very similar identities to the original images. Moreover, the reconstructed image would remove image border artifacts and some complex backgrounds and translate sketches into photo-realistic images.
3.2 INFERENCE: SAMPLING VECTORS TO GENERATE IDENTITIES AND THEIR FACE IMAGES
For Vec2Face, dissimilar input vectors will likely lead to images of different identities, allowing us to generate different identity images; controlling the perturbed vectors in proper similarity range with their identity vector can generate face images with different attributes while preserving the identity. Thus, controlling the sampled vectors is important for dataset generation.
4

---

Published as a conference paper at ICLR 2025
Sampling well-separated identity vectors. Suppose want to generate identity images from n identity vectors U1, ....... Un. where V E IR, ; 1.2, .... R. We ask that the similarity between any pair of identity vectors be less than threshold sim (. 5;) A i j. This ensures that the generated images are of different identities. Following (Papantoniou et al. 2024), PCA used first compute the
to sample vectors. Specifically, face features, then use these statistics to sample vectors in the PCA space via multivariate normal distribution, and finally inverse transform these vectors back to the original feature space to obtain new identity vectors. Newly obtained identity vectors that have similarity I T = 0.3 with all existing vectors are kept, otherwise they are dropped. Interestingly. we notice that, in the high- nigh-dimensional (512 in this paper) feature space, most of the randomly sampled vectors fit the condition due to sparsity. fact, only 1.7% of the sampled 300k vectors are filtered out.
and covariance of the PCA-transformed
Perturbing identity vectors for image creation. After obtaining n. identity vectors 01, U2, .. we sample m vectors for each identity vector by perturbation:
Vim, = Un + o), k ==
where Vimy is the kth perturbed vector for an identity vector Un. (N, σ)) is Gaussian distribution with 0 mean and variance. To ensure the values between perturbed vectors and their identity vectors are 2 0.5.
after perturbation, the similarity
Technically, the above steps allow us to synthesize an unlimited number of identities and their images using loosely constrained vectors via Vec2Face. Hyperparameter is selected such that identities are well-separated, i.e., high inter-class variation, although it does not act as a strong filter in practice. The selection of hyperparameter should enable the generated images for an identity to have large enough variance while staying on the same identity, i.e., high intra-class variation.
3.3 INFERENCE: EXPLICIT FACE ATTRIBUTE CONTROL
By default, Vec2Face generates images without explicit attribute control. This might lead to some attributes being under-represented, e.g., too few profile head poses. To address this, we introduce attribute operation, or AttOO,, to guide vector perturbation so that additional images of desired attributes can be generated. This work focuses on face pose and image quality.
Inspired by (Singh et al. 2023), AttrOP controls the attributes of the generated images by simply adjusting the values in the feature vectors via gradient descent, a process shown Algorithm [ Specifically, first set target image quality and pose angle P and prepare pretrained pose evaluation model Mpose. a pretrained quality evaluation model Mquaisty. and a FR model Mr R- Then, given identity vector Via and perturbed identity vector Tims generate face image from an adjusted vector o/m and compute the following loss functions:
Lattrop Lid + Cquality + Lynse where,
Lid 1 - CosSim (M vra);
(5)
Equality Q - M Mquality (1M), Cpose abs( P abs( Mpoee (TM))),
Because both Mpose and Mquatity are differentiable, gradient descent can be used to adjust v/mm to minimize Luttrop- We finally use the adjusted v'mm to generate images that exhibit the desired pose and image quality. Sample images optimized by AttrOP are shown in Fig. 4 where profile poses and various image quality levels can be seen.
3.4 DISCUSSION
Novelty statement. Vec2Face is novel in three aspects. First, is new to add small/large perturbations to input vectors to generate images of the same/different IDs. comparison, existing works typically use fixed ID vectors without an explicit mechanism to separate different IDs. Second, as minor contribution, our GAN architecture realizes the above function, where the fMAE component prevents overfitting during training, similar to VAE (Kingma & Welling 2013). Third, Vec2Face seamlessly support attribute control during inference, while existing methods have to rely on external models for this purpose.
5

---

Published as a conference paper at ICLR 2025
Algorithm 1: AttrOP
Function At rOP (Gid. Vim, Mgen. T):
Input: a) uia: sampled ID vectors,
b) Vam: initial perturbed ID vectors,
<) Mgen: Vec2Face model,
<) T: the number of iterations
Required: target quality Q, target pose P
Output: a) ''mm' adjusted perturbed ID vectors
Condition models: Mpose Mquality MPR
Initialization 8 Vim
Quality control Pose control Figure 4: Sample generated images after image quality control (left) and head pose control (right). For each ttree-image group, from left to right, image quality increases, or head poses become more profile. This improves diversity of the synthetic dataset.
for t -, 2, 0 do IM= (
Calculate Lattrop in Eq. 5 1/mm
1m AV km Lattrop
return o'mm
Why can Vec2Face generate images like in Fig. D The FR model maps images to hyperspherical space, where features of the same identity clustered, and those of different identities are farther apart (Deng et al., 2019). Using such features as input and supervised by the original images, Vec2Face learns how generate face images based on characteristics of these vectors. As such, increasing the degree of the perturbation added to the vector gradually changes the vector direction, which results in faces of the same and then different identities.
How can Vec2Face increase inter-class separability and intra-class variations? A face feature vector reflects both identity and attributes. Hence, controlling the cosine similarity between sampled vectors can enforce separate identities, and adding small perturbations to sampled vectors could change the attributes while maintaining the identity. Moreover, AttOOP allows us to easily obtain useful vectors generate images with desired attributes, further increasing intra-class variations.
Can other structures work? diffusion model conditioned on image features may achieve similar function. However, the results in Appendix A. show that the initial noise image may exert stronger influence on the final output than the conditioned image/identity features. This compromises identity- preserving capability. In comparison, the stochasticity of method mainly comes from the image feature. fMAE has some randomness in testing, but has a very minor effect on generated images. That said, it would be interesting to design effective architectures in the future.
What is an identity? An interesting philosophical question would be how to define an identity. Our research suggests that generated identities can be continuous, where threshold T is used to determine the boundary between different identities. We conducted simple experiment for this in Section 4.4 It will be interesting to deeply study how this computational definition of identities connects with and differs from philosophical understanding in the future.
4 EXPERIMENTS
4.1 DATASETS AND EVALUATION PROTOCOL
Vec2Face training set. The training data consists of 1M images and their features from 50K randomly sampled identities in WebFace4M (Zhu al. 2023), where the images features are extracted by an ArcFace-R 100 model pretrained on Glint360K (An et al.) 2021). Unless otherwise specified, this model used for feature extraction throughout this paper.
Real-world test sets. There are eleven test sets used to compare the synthetic and real datasets on FR model training. LFW (Huang et al.) 2008) tests the FR model a general case. CFP- FP (Sengupta et al. 2016) and CPLFW (Zheng & Deng 2018 test the FR model on pose variation. AgeDB (Moschoglou et al. 2017) and CALFW (Zheng et al. 2017) challenge the FR model with large age gap. These five test sets used for a general accuracy comparison actoss synthetic datasets and a real dataset. Besides these five, Hadrian (Wu et al.) 2024a) and Eclipse (Wu et al. 2024a) used for intra-class variation evaluation, where Hadrian pairs emphasize facial hair difference

---

Published as a conference paper at ICLR 2025
Training sets
# ims
LFW
CHP-FP
CPLFW
AgeDB
CALFFW
Avg.
LIB
LBCC
IDiff-Face (Boutros et al
2023a
0.5M
98.00
85,47
80.45
86.43
90.65
88.20
62.13
63.61
DCFace Kim et al. 2023)
0.5M
98.55
85.33
82.62
89.70
91.60
89.56
66.47
69.92
ID Li ct al. 2024a)
0.5M
97.68
86.84
82.77
91.00
90.73
89.80
Arc2Face Papantoniou et al. 2024
0.5M
98.81
91.87
85.16
90.18
92.63
91.73
DigiFace Bae al. 2023)
IM
95.40
87.40
78.87
76.97
78.62
83.45
5.42
6.10
SynFace QQuu ct al. 2021)
0.5M
91.93
75.03
70.43
61.63
74.73
74.75
SFace [Boutros el al. 22222
0.6M
91.87
73.86
73.20
71.68
77.93
77.71
37.90
41.94
IDnet Koll et al. 2023)
0.5M
92.58
75.40
74.25
63.88
79.90
79.13
ExFaceGAN (Boutros et al.
2023b)
0.5M
93.50
73.84
71.60
78.92
82.98
80.17
SFace2 (Boutros et al. 2024
0,6M
95.60
77.11
74.60
77.37
83,40
81.62
HSFaceIUK (Ours)*
0.5M
98.87
88.97
85.47
93.12
93.57
92.00
83.82
86.96
CASIA-WehFace (Real)
0.49M
99.38
96.91
89.78
94.50
93.35
94.79
78.71
83.44
Table 1: Comparison of existing synthetic datasets on seven real- test sets. f. *. and o represent diffusion, 3D rendering. and GAN approaches, respectively, for constructing these datasets. We also list the results of training on real-world dataset TASIA-WebFace Since previous works do not provide the IJB results, we re-train the FR models and report the accuracy on available datasets.
and Eclipse pairs differ on face exposure. The images in both datasets are indoor and high quality originating from MORPH (Ricanek& & Tesafaye, 2006). In addition, SLLFW (Deng et al. (0177) and DoppelVer (Thom et al. 2023), are used to evaluate the identity definition used in the existing works (including ours). These two test sets have Aamilar-looking pairs. Lastly, IJBB (Whitelam al. 2017) and IJBC (Maze et al.] 2018) are more challenging test sets that are closer to the world scenario.
Evaluation protocol of test sets. Besides IJBB and LJBC, the aforementioned test sets the same evaluation protocol suggested in (Huang et al.) 2008). Following the previous work (Deng et al. 2019), use TPR@FPR=1e-4 measure the model performance IJBB and IJBC.
4.2 EXPERIMENT DETAILS
Vec2Face model training: We used the default ViT-Base as the backbone to form the IMAE. Since the image size of the commonly used FR training sets is 112x 12 and the image decoder a four-layer, double-sized architecture, the feature map size is set to 49x768, so that this feature map can be reshaped to 7x7x768 for image decoder to decode 112x1 12x3 image. The optimizer is AdamW (Loshchilov & Hutter 2017), with learning rate of 4e-5 and a batch size of 32 per GPU. We use approximately 10 RTX6000 GPUs for each training run.
Synthetic dataset generation. We obtain well-separated identity vectors by PCA learned on face feature vectors of MSIMV2 (Deng et al. 2019). details are in Section 3.2 To get vectors for image generation. perturb the identity vectors with the noise sampled from three Gaussian distributions. Specifically, 50 perturbed vectors are sampled for each identity, where 40% from (0, 0.3). 40% from (0, 0.5), and 20% from (O, 0.7). But this resulting dataset suffers from the low pose variation, which causes bad performance on CFP-FP and CPLFW. To increase pose variation, we generate 30 images with large yaw angle via AttrOP for each identity and randomly replace the existing ones. In detail, the target pose P of 20 images is 60 and 10 images is 85. Meanwhile, we add the image quality control, Q 27, to mitigate the quality degradation during pose adjustment.
AttrOP details. The MagFace-RIUe (eng et al. 2021I, SixDRepNet (Hempel et al. 2024). and ArcFace-R100 are used as Mquality. Mooee, and M FR in Algorithm|] !
i' set to
Face synthesizing method comparison settings. The standard training dataset size is 0.5M images from 10K identities. Unless otherwise specified, the backbone is SE-IR50 (He al. 2016). the recognition loss is ArcFace (Deng et al. (211), and other configuration details are in the Appendix.
4.3 MAIN EVALUATION
Comparison with state-of-the-art synthetic datasets at the seale of 0.5M-0.6M images. In Table [ we compare FR accuracy of models trained with datasets synthesized by different methods, i.e., diffusion models, GANs, and 3D rendering. We have the following observations. First, HSFace10K synthesized by Vec2Face yields very competitive accuracy: 98.87%. 88.97% % 85.47 %, 93.12% 12,, 93.57% % on LFW, CFP-FP, CPLFW, AgeDB, and CALFW, respectively, and 92.00 % on average. Our method is only lower than Arc2Face on the CFP-FP dataset (88.97% vs. 91.87%) and is the
7

---

Published as a conference paper at ICLR 2025
Datasets
images
LFW
CFPP-PP
CPLFW
AgeDB
CALFW
Avg.
HSFaceIOK
0.5M
98.87
88.97
85,47
93.12
93.57
92.00
HSFace20K
IM
98.87
89.87
86.13
93.85
93.65
92,47
HSFace100K
5M
99.25
90.36
86.75
94.38
94.12
92.97
HSFace200K
10M
99.23
90.81
87.30
94.22
94.52
93.22
HSFace330K
15M
99.30
91.54
87.70
94.45
94.58
93.52
HSFace400K
20M
99.37
90.53
87.35
94.33
94.60
93.24
CASIA-WebFace (Rcal)
0.49M
99.38
96.91
89.78
94.50
93.35
94,79
CASIA-WebFace HSFacc1OK
0.99M
99.58
97.06
90.58
95.62
94.67
95.50
Table 2: Impact of scaling the proposed HSFace dataset to 1M images (20K IDs), 5M images (100K
IDs), 10M images (200K IDs), 15M images (300K IDs), 20M images (400K IDs). Continued improvement is observed until 300K IDs. We also list the performance obtained by training on the real-world dataset CASIA-WebFace and its combination with HSFace The latter combination yields even higher accuracy.
state of the art on all the other datasets and average. Second, on the CALFW, IJBB, and IJBC datasets, the accuracy of HSFace is higher than that of -ASIA-WebFace (Yi et al. 2014). To our knowledge, this the first time that a model trained with synthetic data outperforms that trained with same-scale real data. Third, generally speaking, GAN-based methods, while being prevalent, are not as competitive as diffusion-base and 3D rendering methods. Our method is GAN-based but outperforms the other methods.
Effectiveness of scaling up the proposed HSFace dataset. Vec2Face can easily generate a large- scale dataset by sampling more identity vectors which have cosine similarity S 7 with any other identity vectors. To test the efficacy of Vec2Face on scaling, we generate six datasets with increased identity numbers from 10K to 400K, where each identity has 50 images. Results of models trained on these datasets are shown in T
It is clear that scaling up synthetic dataset leads to consistent accuracy improvements until 300K IDs: from 92.00%, to 92.47%, 92.97%, 93.22%, 93.52%. Notably, our HSFace300K is 12.5 times larger than the largest synthetic dataset prior to this paper while still giving steady accuracy improvement. These results clearly demonstrate the ad advantage of Vec2Face in data scaling. Also note that Vec2Face is only trained with 1M real-world images. We speculate that a Vec2Face model trained with larger initial real data could bring further improvements.
Merging synthetic and real-world training sets. In Table after merging HSFace and CASIA- WebFace, we observe improvement over using either dataset alone for training. We obtain 95.50% average accuracy, which is higher than 92. .0% (HSFace 10K alone) and 94.79% (CASIA-WebFace alone). This indicates that HSFacel0K has good quality and is complementary to real data. Additional experiments are in the Appendix.
Effectiveness of attribute control. Images generated by Vec2Face are
Attr. control
LFW
CFP-FP
CPLFW
AgeDB
CALFW
Avg.
98.27
76.56
81.70
90.75
92.92
88.04
mostly near-frorme frontafon which can be at-
+ Quality 27
98.55
83.27
83.68
91.12
93.27
89.98
tributed to such images being most frequent in the training set. Using At- trOP, we synthesize faces with some profile poses where we use target Yaw angles 60" and 85". Results are shown in Table 3 We observe that adding faces with 60 poses leads to 3.46% improvement, while adding 85% gives further 0.5% improvement. Unlike prior works (Kim al., 2023; Papantoniou et al. 2024), this achieved by single model.
Angle 60
98.62
86.46
85.75
92.85
93.80
91.50
Angle 85
98.87
88.97
85.47
93.12
93.57
92.00
Table 3: Impact of face quality and pose control via AttrPP in FR accuracy. Each of the training sets has 0.5M images. The accuracy improvement observed when more controls are added for image quality and pose angles, especially on pose-oriented test sets, CFP-FP and CPLFW.
These results demonstrate the effectiveness of attribute control.
Image generation cost and quality of reconstruc- tion. The resource requirement for image generation is an important metric, especially in this large model era. We compare our method with a ofate-of-the-an face generation model. Experimental details: 1) both models are tested on a single Titan-Xp -x, 2) the batch size is 8, 3) 4-step scheduler (Luo et al. 2023) is
Computing cost
FID
Methods
Model size FPS
LFW
Hadrian
Arc2Face
3.4GB
1.5
43.80
53.27
Vec2Face
0.68GB
467
35.75
51.93
Table 4: Computing cost and FID measure-
ment of Arc2Face and Vec2Face.
8

---

Published as a conference paper at ICLR 2025
used for Arc2Face. Thols shows that method is 311x faster than Are2Face. We also use FID to measure the distance between the original LFW and Hadrian images, and the reconstructed images from Arc2Face and Vec2Face, where the images measured at 1 12x 12 resolution. Our model slightly outperforms the Arc2Face results, in size of model, speed of generation, and fidelity to original images, shown in Table 4 More results are in Appendix A.I.
4.4 FURTHER ANALYSIS
VeceFace generates datasets with large inter-class separability. Since only Arc2Face and Vec2Face allow us to generate a large number of identities, we directly use the datasets (including a real dataset) released in previous works calculate the identity vectors for separability evaluation. For Arc2Face and Vec2Face, we sample 200K well-separated vectors to generate identity images, where we report the results of using LCM-lora (Luo al. 2023) for Arc2Face. Fig. 5 presents the number of identities whose cosine similarity against any other identity vector is less than 0.4, as more generated identities gradually added for each dataset. Note that, reasonable threshold for separability evaluation varies when using different FR models (Kim et al. 2023 Boutros et al., 2024). We select 0.4 is because gives us a reasonable number (I83K out of 200K) on a real dataset. From the results, have three observations.
First, synthetic datasets such as SynFace and SFace generally have lower inter-class separabil- ity than real-world datasets. This is because their data synthesis methods do not utilize the face feature characteristics for image generation, as Arc2Face and IDiff-Facc do. Second, Are2Face has much larger identity separability than ex- isting methods, comparing with the number re- ported in (Kim et al.) 2023), but is still inferior to Vec2Face. This is because the initial noise
200k
100k
Number of well sepurated IDs (Sim < 0.4)
10k
1k
Vec2Face (ours)
WebFace4M (real)
image sometimes exerts stronger influence on the final output than the conditioned identity vector. Third, real-world datasets have slightly lower identity separability than our method, be- cause they have small number of twins, close relatives, or even the same person with differ- character names in TV shows (Wu et al.
Are2Face
DCFace
IDiff-Face
SFace
DigiFace
SynFace
1k
10k
100k
200k
2024b). Note that 0.4 not used to evaluate the number of actual identities in the dataset, but rather to provide a consistent measure- ment of identity separability across datasets. Overall, the special design of Vec2Face help us to generate well-separated identities.
Number IDs
Figure 5: Comparing existing synthetic FR dataset generation methods on inter-class separability.
Impact of inter-class separability FR accuracy. Table 7 summarizes how different average identity similarity or different levels of identity separability affect FR accuracy, Results indicate that high identity separability beneficial, which is consistent with previous findings (Boutros al. 2023a). A unique finding is that too large separation does not further benefit the performance.
Vec2Face generates datasets with large intra-class variation. The accuracy values in Table !
indicate the generated dataset has large variation on age and pose. To evaluate the intra-class variation on other attributes, we test the models on Hadrian and Eclipse (Wu et al.] 2024a). which have 6K image pairs for each and respectively emphasize variation on facial hair and face exposure. Table 6 shows that the datasets generated by Vec2Face have proper variation on these two attributes, as increasing the dataset size eventually surpasses the accuracy of the real datasct.
Impact of hyperparameter a.. Vec2Face, a directly controls intra-class variation, i.e., larger o means higher intra-class variation and vice versa. We test different configurations of values and its sampling %. Table 5 shows that, when increasing 0, FR accuracy first increases and then drops. Initially, increasing intra-class variation is beneficial because it creates some hard training samples. But if a is too large, the generated face may look like a different person, degrading the accuracy. Similarly, increasing the sampling % of the large may hurt the identity consistency in folder,

---

Published as a conference paper at ICLR 2025
0
Sampling % Simyain
Avg.
Datasets
Hadrian
Eclipse
SLLFW
DoppelVer
(0.3)
(100)
0.71
82.71
HSFace 10K
69.47
64.55
92.87
86.91
[0.3, 0.5)
[60. 40)
0.62
86.25
HSFace20K
75.22
67.55
94.37
88,90
[0.3, 0.5, 0.7]
(40, 40, 20)
0.53
88.04
HSFace 100K
80.00
70.35
95.58
90.39
(0.3, 0.5, 0.7)
(20, 40, 40)
0.51
87.26
HSFace200K
79.85
71.12
95.70
89.86
[0.3, 0.9] [40, 40, 20)
0.47
85.75
HSFace300K
81.55
71.35
95.95
90.49
Table 5: Impact of a values with corre- sponding sampling % for each identity on average FR accuracy (%). We use 0.5M images from 10K identities for training. AttOP is not applied and Simmin is the minimum cosine similarity between the perturbed vectors and their ID vectors.
CASLA-WebFace
77.82
68.52
96.95
95.11
Table 6: Comparing a real dataset with HSFaces on other tasks. Hadrian, Eclipse, SLLFW, and DoppelVer emphasize facial hair variation. face exposure differ- ence, similar-looking and doppelganger, respectively.
Avg. ID sim.
Avg.
# IDs x # IMs
Avg.
Architectures
Avg.
0.88
64.62
10K
75.02
SE- BE-IRRsNer18
89.60
0.70
77.65
x0Kx10
83.09
SE-IResNet50
92.00
0.3
0.56
78.67
10K x20
89.45
SE-IResNet100
92.49
0.01
88.04
10Kx50
92.00
SE-1IResNet200
92,49
0.2
0.003
86.60
x0Kx80
92.03
ViT-S
88.54
Table 7: Impact of inter-class Table 8: Impact of number of separability. Larger average images per identity. Different identity similarity, low separa- number of images are generated bility reduces the accuracy. We by the best and their sampling use 0.5M images from 10K iden- % setting. The average accuracy tities for training.
ViT-B
90.15
Table 9: HSFace 10K training on ResNets and Vision Transform- ers (ViT). The average accuracy (%) on five test sets is reported. (%) on five test sets is reported.
which also degrades the accuracy. Thus, we choose o {0.3, 0.5, 0.7) to sample (40%, 40%, 20%} images for each identity.
Does Vec2Face generate new identities compared with its training identities? We compute the feature similarity between the 300K identities from the proposed synthetic data and the 50K training identities selected from WebFace4M. We find that only 0.398%, or 1194 out of the 300K synthetic identities are similar the training identities, with similarity score greater than 0.4. To completely avoid using real identities for training FR models, we have replaced these 1194 identities with new identities, and the performance change is negligible. More analysis is in the Appendix.
More images per identity is beneficial FR accuracy. In Table increasing the number of images per identity from 5 to 80 increases the accuracy, but this improvement is saturated at 80.
Larger backbone helps the FR model performance. In Table accuracy of six models trained with HSFace and ArcFace loss increases from SE-IResNet18 SE-IResNet100 and is saturated SE- IResNer200. Accuracy increase is also observed when using a larger Vision Transformer (Dosovitskiy 2020 backbone.
Limitations. The proposed synthetic dataset is effective for general FR but would be less useful for fine-grained tasks such discriminating between doppelgangers, see Table We believe better understanding of 'identity' and stronger generation models will help solve this issue and we hope to address this future work.
5 CONCLUSIONS
This paper proposes Vec2Face, a face generation method which can elegantly create a large number of identities and face images. This method conveniently converts vectors to face images and can translate perturbations of the vectors into consistent perturbations of the resulting face images. Because of its unique design, the inter-class and intra-class variations of generated face images can be directly controlled by hyperparameters of vector sampling.. We show that the resulting dataset, HSFace 10k, has large intra- and inter-class variations and yields state-of-the-e FR accuracy. Importantly, Vec2Face allows for scaling HSFace to 300K identities and 15M images while still seeing accuracy improvement. In future work, we will study the definition of identity to solve more challenging FR tasks, try other generation structures, and extend this method generic objects.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Xiang An, Xuhan Zhu, Yuan Gao, Yang Xiao, Yongle Zhao, Ziyong Feng, Lan Wu, Bin Qin, Ming Zhang, Debing Zhang, and Ying Fu. Partial FC: training 10 million identities on single machine. In ICCVW, pp. 1445-1449, 2021.
Gwangbin Bac, Martin de La Gorce, Tadas Baltrusaitis, Charlie Hewitt, Dong Chen, Julien P. C. Valentin, Roberto Cipolla, and Jingjing Shen. Digiface-lm: million digital face images for face recognition. In WACV, pp. 3515-3524 2023.
Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu, All are worth words: vit backbone for score-based diffusion models. In NeurIPS, 2022.
Fadi Boutros, Marco Huber, Patrick Siebke, Tim Rieber, and Naser Damer. Sface: Privacy-friendly and accurate face recognition using synthetic data. In IJB,, pp. 1-11. IEFR, 2022a.
Fadi Boutros, Marcel Klemt, Meiling Fang, Arjan Kuijper, and Naser Damer. Unsupervised face recognition using unlabeled synthetic data. IEEE F&G, pp. 1-8, 2022b.
Fadi Boutros, Jonas Henry Grebe, Arjan Kuijper, and Naser Damer. Idiff-face: Synthetic- based face recognition through fizzy identity-conditooee diffusion model. In ICCV, pp. 19650-19661, 2023a.
Fadi Boutros, Marcel Klemt, Meiling Fang, Arjan Kuijper, and Naser Damer. Exfacegan; Exploring identity directions in gans learned latent space for synthetic identity generation. In IJCB, pp. 1-10. IEEE, 2023b.
Fadi Boutros, Marco Huber, Anh Thi Luu, Patrick Siebke, and Naser Damer. Sface2: Synthetic-basee face recognition with w-space identity-driven sampling. IEEE T-BIOM, 2024.
Li Chen, Mengyi Zhao, Yiheng Liu, Mingxu Ding, Yangyang Song. Shizun Wang, Xu Wang, Hao Yang, Jing Liu, Kang Du, al. Photoverse: Tuning-free image customization with text-to-image diffusion models. arXiv preprint 23t.::33000009333 2023.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. ICCV, 2021.
Ivan TrAAddrs-Taame Ruben Tolosana, Pietro Melzi, Ruben Vera-Rodriguez, Minchul Kim, Christian Rathgeb, Xiaoming Liu, Aythami Morales, Julian Fierrez, Javier Ortega-Garcia, et al. Fresyn challenge at CVPR 2024: Face recognition challenge in the era of synthetic data. CVPRW, pp. 3173-3183, 2024.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In CVPR, pp. 4690-4699, 2019.
Weihong Deng, Jiani Hu, Nanhai Zhang, Binghui Chen, and Jun Guo. Fine-grained face verification: FGLFW database, baselines, and human-demn partnership. PR, 66:63-73, 2017.
Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3d umitative-contrastive learning. CVPR, pp. 3153-5162, 2020.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXi:22010.11929, 2020.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pp. 12873-12883, 2021.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is strong image synthesizer. In ICCV, pp. 23164-23173, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, pp. 16000-16009, 2022.
11

---

Published as a conference paper at ICLR 2025
Thorsten Hempel, Ahmed A. Abdelrahman, and Ayoub Al-Hamadi. Toward robust and unconstrained full range of rotation head pose estimation. IEEE Transactions on Image Processing, 33:2377-2388) 2024. doi: 10.1 10.1109/11.2002433381183
Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in Real- Images: detection, alignment, and recognition, 2008.
Phillip Isola, Lun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. taage-to-image translation with conditional adversarial networks. In CVPR, pp.
2017.
Minchul Kim, Feng Liu, Anil K. Jain, and Xiaoming Liu. Deface: Synthetic face generation with dual condition diffusion model. In CVPR, pp. 12715-12725. IEEE, 2023.
Diederik P Kingma and Max Welling. 1312.661 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Jan Niklas Kolf, Tim Rieber, Jurek Elliesen, Fadi Boutros, Arjan Kuijper, and Naser Damer. Identity- driven three-player generative adversarial network for synthetic- based face recognition. In CVPR, pp. 806-816, 2023.
Shen Li, Jianqing Xu, Jiaying Wu, Miao Xiong, Ailin Deng, Jiazhen Yuge Huang, Wenjie Feng. Shouhong Ding, and Hooi Bryan. synthetic face recognition. arXiv preprint urXiv:2409. /7576, 2024a.
wwwpreeeeiiiiieee-ieeeeeeeeeee diffusion models for
Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang. Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In CVPR, pp. 2142-2152, 2023.
Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Photomaker: Customizing realistic human photos via stacked id embedding. In CVPR, pp. 8640-8650, 2024b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05000 2017.
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lem-lora: universal stable-diffusion acceleration module. arXiv preprint arXiv:2311.05533 2023.
Brianna Maze, Jocelyn C. Adams, James A. Duncan, Nathan D. Kalka, Tim Miller, Charles Otto, Anil K. Jain, W. Tyler Niggel, Janet Anderson, Jordan Cheney, and Patrick Grother. IARPA janus benchmark C: face dataset and protocol. In ICB, pp. 158-165, 2018.
Pietro Melzi, Ruben Tolosana, Ruben Vodrinzd,, Minchul Kim. Christian Rathgeb, Xiaoming Liu, Ivan DeAndres- Tame, Aythami Morales, Julian Fierrez, Javier Ortega- Cargia, et al. Fresyn challenge at WACV 2024: Face recognition challenge the era of synthetic data. In WACVW, pp. 892-901, 2024.
Qiang Meng, Shichao Zhao, Zhida Huang, and Feng Zhou. Magface: A universal representation for face recognition and quality assessment. In CVPR, pp. 14225-1442 14234, 2021.
Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. AgeDB: The first manually collected, in-the-wild age database. In CVPRW, pp. 1997-2005, 2017.
Foivos Paraperas Papantoniou, Alexandros Lattas, Stylianos Moschoglou, Jiankang Deng, Bernhard Kainz, and Stefanos Zafeiriou. Are2face: A foundation model of human faces. arXiv preprint 1rXiv:2403.11644 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pp. 4195-4205, 2023.
Haibo Qiu, Baosheng Yu, Dihong Gong, Zhifeng Li, Wei Liu, and Dacheng Tao. Synface: Face recognition with synthetic data. In ICCV, pp. 10860-10870, 2021.
12

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748 8763, 2021.
Karl Ricanek and Tamirat Tesafaye. Morph: A longitudinal image database of normal adult age- progression. In JEEE F&G, pp. 341-345, 2006.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Soumyadip Sengupta, Jun-Cheng Chen, Carlos Domingo Castillo, Vishal M. Patel, Rama Chellappa, and David W. Jacobs. Frontal to profile face verification in the wild. In WACV, pp. 1-9, 2016.
Hatef Otroshi Shahreza, Christophe Ecabert, Anjith George, Alexander Unnervik, Sébastien Marcel, Nicolò Di Domenico, Guido Borghi, Davide Maltoni, Fadi Boutros, Julia Vogel, et al. Sdfr: Synthetic data for face recognition competition. IEEE F&G, pp. 1-9, 2024.
Jaskirat Singh, Stephen Gould, and Liang Zheng. High-fidelity guided image synthesis with latent diffusion models. In CVPR, pp. 5997-6006, 2023.
Nathan Thom, Andrew DeBolt, Lyssie Brown, and Emily M Hand. Doppelver: A benchmark for face verification. In ISVC, pp. 431-444, 2023.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. CVPR, 2024.
Dani Valevski, Danny Lumen, Yossi Matias, and Yaniv Leviathan. FaceO: Instantancously condition- ing text-to-image model face. In SIGGRAPH, pp. 1-10, 2023.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity- preserving generation in seconds. arXiv preprint arXiv:2401.07519 2024.
Cameron Whitelam, Emma Taborsky, Austin Blanton, Brianna Maze, Jocelyn C. Adams, Tim Miller, Nathan D. Kalka, Anil K. Jain, James A. Duncan, Kristen Allen, Jordan Cheney, and Patrick Grother. IARPA Janus benchmark-b face dataset. In CVPRW, pp. 592-600, 2017.
Errlll Wood, Tadas Baltrusaitis, Charlie Hewitt, Sebastian Dziadzio, Thomas J. Cashman, and Jamie Shotton. Fake it till you make it: face analysis in the wild using synthetic data alone. In ICCV, pp. 3661-3671. IEEE, 2021.
Haiyu Wu, Sicong Tian, Aman Bhatta, Jacob Gutierrez, Grace Bezold, Genesis Argueta, Karl Ricanek Jr., Michael C. King, and Kevin W. Bowyer. What is goldilocks face verification test set? arXiv preprint arXiv:2405. 15965, 2024a.
Haiyu Wu, Sicong Tian, Jacob Gutierrez, Aman Bhatta, Kağan Öztürk, and Kevin W Bowyer. Identity overlap between face recognition train/test data: Causing optimistic bias in accuracy measurement. arXiv preprini 24X5::200500940) 2024b.
Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han. Fastcom- poser: Tuning-fres free multi-subject image generation with localized attention. arXiv preprint 13055:2300..0431, 2023.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308..677. 067211, 2023.
Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li. Learning face representation from scratch. arXiv preprint arxiv:1411. 7923, 2014.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zinui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206.1078) pp. 5, 2022.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pp. 3836-3847, 2023.
13

---

Published as a conference paper at ICLR 2025
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, pp. 586-595, 2018.
Tianyue Zheng and Weihong Deng. Cross-pose LFW: A database for studying cross-pose face recognition in unconstrained environments, Beijing University Posts and Telecommunications, Tech. Rep, 2018.
Tianyue Zheng, Weihong Deng, and Jiani Hu. Cross-age LFW: A database for studying cross-age face recognition unconstrained environments. arXiv preprint (rXx::700.0889999 2017.
Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjic Huang. Xinze Chen, Jiagang Zhu, Tian Yang, Dalong Du, Jiwen Lu, and Jie Zhou. Webface260m: A benchmark for million-scale deep face recognition. TPAMI, pp. 2627-2644 2023.
14


Reviews:
[{"review_id": "r7HiVsxxWe", "reviewer": "Reviewer_XvoX", "summary": "The paper introduces a novel Vec2Face generative framework for generating identity-specific synthetic face images suitable for training deep recognition models. To this end, the authors rely on a feature Masked Auto-Encoder (fMAE) architecture, which is trained to produce face images of a desired identity based on input features extracted from a pretrained Face Recognition (FR) model. However, these identity features are first projected and expanded to 2D feature maps to suit the nature of the fMAE architecture. Training is performed with four objectives, including the standard reconstruction and perceptual loss functions, along with an identity-based loss, which enables better identity consistency, and a GAN loss that utilizes a patch-based discriminator to ensure sharper images. During inference, the approach utilizes PCA for sampling unique identity vectors, upon which perturbation is performed to obtain diverse samples of a specific identity. Additionally, the approach also supports control over the pose and quality, through guided vector perturbation. The authors showcase the suitability of their approach through extensive experiments across multiple datasets, with a focus on training deep face recognition models with the generated synthetic data. The proposed approach outperforms existing state-of-the-art methods in terms of the accuracy achieved with a trained recognition models on standard verification benchmarks. Importantly, Vec2Face also enables better scalability than existing approaches, thus allowing the generation of diverse datasets of up to 20 million images, compared to existing datasets of 500 thousand images. Furthermore, the authors also perform sufficient ablation studies for supporting the various choices of their particular solution.", "questions": "1.\tDespite the statement provided in Section 3.4., the novelty introduced with this paper remains slightly unclear. The overall paper gives the impression that the fMAE is novel, however, Section 3.4. does not seem to point this out. Can you perhaps elaborate more on the novelty of the fMAE? Does it share similarities or draw inspirations from previous works, namely [1]? Is the novelty perhaps the projection and expansion of input features to 2D feature maps? \n\n[1] Hu, Junhao, et al. \"Features Masked Auto-Encoder-Based Anomaly Detection in Process Industry.\" 2023 IEEE 12th Data Driven Control and Learning Systems Conference (DDCLS). IEEE, 2023.\n\n2.\tWhen discussing the influence of inter-class separability on the accuracy of trained recognition models, it is observed that a too large separation does not benefit the performance. However, this might not be a unique finding, as results in [2] also showcase that well-separated synthetic identities actually result in lower verification performance. Can you comment more on this phenomenon? Does this mean that future research will have to determine the best balance between inter-class separability and intra-class variation?\n\n[2] Boutros, Fadi, et al. \"IDiff-Face: Synthetic-based face recognition through fizzy identity-conditioned diffusion model.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The submitted paper is writte well and presents the problem and proposed solution in a concise and clear manner. The proposed approach improves on the state-of-the-art with a unique fMAE architecture, differentiating itself from existing diffusion and GAN-based solutions. Differently, it also proposes to utilize guided perturbation during inference, to not only achieve higher intra-identity diversity of samples but also allow for control over the pose and quality of images, which is crucial for generating large-scale datasets suitable for training recognition models. As a result, the recognition model trained on the generated data of Arc2Face even achieves better results on the Cross-Age LFW benchmark than when training with the real CASIA-WebFace dataset. This represents a tremendous achievement as existing approaches typically achieve worse performance. The proposed approach also enables scaling the generated datasets without drastically impacting inter-class separability, in turn achieving significantly larger dataset sizes than existing approaches. Overall, the paper is suitable for the conference and will likely highly influence future research on the generation of synthetic biometric datasets.", "weaknesses": "Despite the many strengths of the paper, there are a few areas where the paper could be further improver. The evaluation of the approach mainly revolves around the accuracies achieved with trained recognition models on existing benchmarks (apart from Table 4 and Figure 5). While this focus is understandable, it would be beneficial to also analyse other aspects of the data.  \n\n1.\tFace Image Quality Assessment (FIQA) measures could provide additional insight as they are specifically designed for evaluating the quality of face images and their suitability for recognition tasks. Examples of recent FIQA measures with publicly available implementations include CR-FIQA (Boutros et al., CVPR 2023) and DifFIQA (Babnik et al., IJCB 2023). \n\n2.\tComparison with the state-of-the-art (e.g. Table 4) could also be expanded with metrics that measure fidelity and diversity separately could also be utilized to provide a deeper understanding of the generated datasets, e.g., improved precision and recall (Kynkäänniemi et al., NeurIPS 2019).   \n\n3.\tIn the supplementary material when comparing genuine and imposter distributions it would also be beneficial to provide results obtained with a real-world dataset to provide the baseline of desired distributions suitable for training face recognition models. \n\n4.\tThe utilized verification benchmarks are of a rather small scale (roughly 3000 genuine and imposter pairs). It might be beneficial to evaluate the approach on a more complex and larger-scale set, e.g. the IJB-C verification benchmark. There the difference performance improvements of the Vec2Face method could be further highlighted. \n\nLastly, Despite the detailed nature of the paper, the utilized patch-based discriminator is rather poorly explained. More information should be added to the paper or at least to the supplementary material. For example, what sort of architecture does it utilize? Is it initialized with a pretrained model? The formulation of perceptual and GAN loss functions could also be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cgk6H7c1RP", "reviewer": "Reviewer_oM4G", "summary": "In this paper by using a Masked AutoEncoding on the feature space and a separate FR system, Pose Estimator and Face Quality assessment network authors introduced a pipeline for sampling identities to generate synthetic face dataset for training FR systems.", "questions": "* There was no reproducibility section in the paper do the authors plan to publish their code/ dataset?\n* I was wondering if the authors considered using random perturbed vectors instead of features from an FR model in the training phase. \n* As the authors used the subset of WebFace4M (50K identities) for training their method, have authors considered using an FR system trained on this dataset for the ID loss in their proposed method? In this case, one of the weaknesses that I mentioned would be in some extent addressed, if authors can observe the same trend in the results. \n* The authors used an FR system trained on MS1MV2,  Glint360K, in their training objective prior, it would be informative if the authors could add the performance of such a system in Table 2.\n* In Table.2, I am curious to see what will happen if we add more synthetic images to the CASIA, let’s say CASIA+HSFace200K.\n* [392] As the improvement is on the order of 0.1% could this be also the case if authors changed the network, seed, or margin loss?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "* The paper is well written and the experiments are exhaustive and insightful both in the appendix (especially the analysis of the attributes of the synthetic data) and the main content.\n* Applying MAE on the feature space and formulating it to work with the sampling of the random vectors is quite interesting.", "weaknesses": "* As the main point of the paper is to generate a synthetic FR dataset, my main concern is that the authors have used the FR model trained on MS1MV2 and Glint360K, and a large subset of the WebFace4m for training/sampling from their method. Training an FR model on  these datasets will probably outperform the FR model trained on the CASIA-WebFace (this dataset used for comparison in Table.1 and Table .2). Later by generating the datasets in a controlled manner they still did not perform as good as a model trained on the CASIA-WebFace, although they are close in most of the benchmarks, so what is the main problem that the authors trying to solve? Why we can not use the CASIA-WebFace, Glint360K, MS1MV2, or the subset of the large WebFace4M that authors used for training their method which probably performs better? Is it trying to address the privacy issues? Are the generated identities different from the ones presented in the dataset used? \nAuthors may consider the goal of synthetic dataset generation, especially when their approach assumes the existence of a high-quality FR system or a large-scale dataset already available for training their methodologies.\nI will consider raising my score if the authors can argue a benefit of this work over the datasets and models they used for training their methodology\n\n* Maybe evaluation using more benchmarks like TinyFace might be needed.\n* There is a good amount of studies that show the InceptionV3 trained on the Imagenet is not a good candidate for reporting Frechet Distance: \n   [1] The role of Imagenet classes in Frechet inception distance: https://arxiv.org/pdf/2203.06026\n\n   [2] Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models: https://proceedings.neurips.cc/paper_files/paper/2023/hash/0bc795afae289ed465a65a3b4b1f4eb7-Abstract-Conference.html\n\n* Some minor Typos: \n  [366] surround the pose with braces to appear in the subscript.\n  \n  APP[188] five test sets -> Five Synthetic Data methods maybe?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZZehUNyO9q", "reviewer": "Reviewer_G33d", "summary": "The paper introduces a novel method called Vec2Face to synthesize large-scale, high-quality face images for training face recognition models. This method addresses the limitations of existing techniques by:\n1. Generating a vast number of distinct identities: Vec2Face can generate up to 300,000 unique identities, significantly surpassing previous methods.\n2. Controlling intra-class variation: The model can generate diverse variations of the same identity by perturbing the input vector, enabling realistic appearance changes.\n3. Manipulating attributes: Vec2Face allows for the generation of images with specific attributes by adjusting the input vector using gradient descent.", "questions": "1. Choice of Pre-trained FR Model:\n    a. You've chosen to use ArcFace-R100 pre-trained on Glint360K. Could you elaborate on the reasons behind this specific choice?\n    b. Are there any limitations to using a smaller pre-trained FR model trained on a smaller dataset? It seems that requiring a large-scale, pre-trained FR model could limit the applicability of your approach, especially in scenarios where such a model isn't readily available. This creates a potential \"chicken-and-egg\" problem where generating synthetic data requires access to large-scale real data. Could you address this potential limitation?\n\n2. Number of Identities:\n    a. Is there a specific reason for limiting the number of generated identities to 300K or 400K?\n    b. Given that your filtering process removes only 1.7% of generated images, is it possible to generate a larger number of identities (e.g., 500K or 1M)? If so, what are the potential challenges or trade-offs involved?\n\n3. Attribute Control During Inference:\n    a. You mention that your method \"seamlessly supports attribute control during inference.\" However, it seems that this relies on three external models: pre-trained pose, quality, and FR models. Could you clarify what you mean by \"seamless\" in this context?\n    b. Does relying on these external models introduce any constraints or complexities in practice?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The pipeline presented in the paper is simple and efficient.\n2. The use of a feature-masked autoencoder is somewhat novel.", "weaknesses": "1. Limited Scale of Synthetic Data: While the paper proposes a method for generating synthetic faces, the number of identities generated (300K-400K) is comparable to existing real-world datasets used to train facial recognition (FR) models. This raises concerns about the practical benefits of generating synthetic data at this scale.\n2. Lack of Comparison with Larger Datasets: The paper lacks experiments comparing the effectiveness of the generated synthetic data with larger-scale real-world datasets like Glint360K. This comparison is crucial to demonstrate the advantages of the proposed approach in scenarios where abundant real data is available.\n3. Unclear Practical Motivation: The authors do not provide a compelling reason for generating a synthetic dataset with a similar or smaller scale than commonly used real-world FR training datasets.  It is essential to clarify the specific use cases and advantages of this approach, particularly when real data is readily accessible.\n4. Missing Citation: The paper omits a key citation: \"Vec2face: Unveil human faces from their black box features in face recognition\" (CVPR 2020). This work appears relevant to the paper's methodology and should be properly acknowledged.\n\nMisleading Novelty Statements: Certain claims of novelty in the paper are potentially misleading.  (See the next questions section for more details.)  It is crucial to ensure that all claims are accurate and supported by evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "usAXFWFPta", "reviewer": "Reviewer_LGwK", "summary": "This paper proposes VEC2FACE to synthesize facial images from a sampled identity vector for synthetic face recognition. It utilizes a GAN solution to reconstruct the image at the training stage and adopt the decoder for inference. High intra-class variation is ensured by Perturbing identity vectors. comprehensive experiments are conducted to demonstrate the effectiveness of the method.", "questions": "Please see the weakness. \n\nOverall, I believe the novelty of this paper is incremental, and improvement is good but does not achieve state-of-the-art. Furthermore, privacy issues arise as this paper adopts a lot of labelled face recognition datasets.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method achieves good performance compared to the listed literature.\n2. It can generate as many novel identities as possible.\n3. The experiments are sufficient.\n4. Using GAN to achieve such results is inspiring.", "weaknesses": "1. [Major] lines 219-220, the authors state 'Following (Papantoniou et al., 2024), PCA is used to sample vectors.'. This brings a concern that the contribution of generating as many novel identities as possible is benefited from the previous work Arc2Face. Additionally, adopting pure identity embedding to generate a facial dataset is demonstrated in the related literature, ID3.\n\n2. [Major] After attribute OP, how do you make sure the generated image preserves the identity, Do you have experiments to prove it? The attribute operation is also introduced in ID3.\n\n3. [Major] line 314, Vec2Face training set is WebFace4M, the PCA is carried on MS1MV2, the feature extraction model is from Glint360K. This counters the privacy concerns, as the dataset original MS1M dataset is recalled by its creator. And the whole training requires a lot of face recognition dataset, which is a disadvantage over DCFace/ID3.\n\n4. [Major] In addition, the training dataset is webface4m (1M with 50k identities ), which is significantly larger than CASIA-WebFace (500k images with 10k identities) adopted by the competitors DCFace/ID3, making the comparison with previous methods unfair. \n\n5. [Major] The authors state they achieve state-of-the-art results. However, the SOTA result is 92.30 (on 0.5M volume) and 93.07  (on 0.5M volume), achieved by CemiFace[1].\n\n6. [Minor]How to generate the image is not clear. For example, how do you obtain the identity feature, randomly generated or from the image? \n\n7. [Minor] A study about alternative pretrained FR model should be conducted, as the proposed method uses ArcFace pretrained from Glint360k, DCFace adopted model pretrained from WebFace4M.\n\n8. [Minor] Line 243, ‘too few profile head poses’. Do you have statics results?\n\n9. [Minor] Eq. 5 is trained with the GAN?\n\n10. Long-page discussions might be better if positioned in the appendix.\n\nReference:\nCemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)