Vec2Face: Scaling Face Dataset Generation with Loosely Constrained Vectors

## Abstract

Abstract This paper studies how to synthesize face images of non-existent persons, to create a dataset that allows effective training of face recognition (FR) models.
Two important goals are (1) the ability to generate a large number of distinct identities (inter-class separation) with (2) a wide variation in appearance of each identity (intra-class variation).
However, existing works 1) are typically limited in how many well-separated identities can be generated and 2) either neglect or use a separate editing model for attribute augmentation.
We propose Vec2Face, a holistic model that uses only a sampled vector as input and can flexibly generate and control face images and their attributes. Composed of a feature masked autoencoder and a decoder, Vec2Face is supervised by face image reconstruction and can be conveniently used in inference. Using vectors with low similarity among themselves as inputs, Vec2Face generates well-separated identities. Randomly perturbing an input identity vector within a small range allows Vec2Face to generate faces of the same identity with robust variation in face attributes. It is also possible to generate images with designated attributes by adjusting vector values with a gradient descent method.
Vec2Face has efficiently synthesized as many as 300K identities with 15 million total images,
whereas 60K is the largest number of identities created in the previous works.
FR models trained with the generated HSFace datasets, from 10k to 300k identities, achieve state-of-the-art accuracy, from 92% to 93.52%, on five real-world test sets. For the first time, our model created using a synthetic training set achieves higher accuracy than the model created using a same-scale training set of real face images (on the CALFW test set).
Code link: https://haiyuwu.github.io/vec2face.github.io/

## 1 Introduction

We aim to synthesize face images in a way that enables
large-scale training sets for FR models, which have the potential to address privacy issues arising with web-scraped datasets of real face images.
It is generally recognized that a good training set for FR should have high inter-class separability (Kim et al., ; Boutros et al., ) and intra-class variation (Qiu et al., ; Boutros et al., ; Kim et al., ; Boutros et al., ).

However, existing methods lack flexibility in controlling the generation process, leading to unsatisfactory inter-class and intra-class results. On the one hand, Boutros et al. () point out that identities generated in (Qiu et al., ; Boutros et al., ; ; ) have relatively low separability because identity generation is controlled by either a single coefficient or by hard class labels. On the other hand, while it is useful to employ identity features as a condition to increase inter-class separability (Boutros et al., ; Papantoniou et al., ), it is non-trivial to increase intra-class variation without the use of a separate model such as ControlNet (Zhang et al., ) or a model for style transfer.

In light of the above, this paper proposes Vec2Face, a flexible model enabling the generation of large-scale FR datasets. In inference, the workflow of Vec2Face
is similar to (Papantoniou et al., ; Boutros et al., ), which use a random vector as input and output a face image.
But the interesting property of Vec2Face is that
a small perturbation of the input vector leads to a face image of the same identity but with small changes in appearance, e.g., pose, expression, and facial hair, while a large perturbation of the input vector produces a face image of a different identity (Fig. 1).
With this property, and because of the high-dimensionality (*i.e.*, 512-dim) of the input vectors, we can easily 1) sample a large number of identity vectors by controlling their similarity with each other to be under a certain threshold and 2) synthesize a large number of face images by perturbing the identity vectors within a small range. Additionally, we can guide the image content at time of generation using a gradient
descent method to efficiently generate faces with minority attributes, such as extreme pose angles.

Using a proper cosine similarity threshold (*i.e.*, 0.3) to sample vectors, the special design of Vec2Face ensures a high ratio of transferring the similarity between vectors to the similarity between generated images, resulting in large inter-class separability; intra-class variation is also properly large because we controlled the variance of the noise for perturbation, so the generated images have $\geq$0.5 similarity with the identity vector, exhibiting various appearances while preserving the identity.
Experimentally, we verify these inter-class and intra-class variations are beneficial (in hyper-parameter analysis). Because of this, FR models trained on the synthesized 10k identities and 500k images demonstrate very competitive accuracy compared with the state-of-the-art synthetic databases of the same scale (Kim et al., ; Papantoniou et al., ). Importantly, we show that scaling up the synthetic training data to 10 million images and 200k identities leads to further accuracy improvement. It is also worth noting that results on CALFW (Zheng et al., ) represent the first instance where a model trained with synthetic data yields higher accuracy than that trained with real data at the same scale. To our knowledge, this is the first time a synthetic dataset can be this large and more useful.

The training of Vec2Face aims to let a vector generate a face and let its perturbation magnitudes reflect face change magnitudes. To this end, in training we use vectors extracted from real face images by a pre-trained FR model. These vectors are fed into a feature masked autoencoder architecture, after which a decoder produces images similar to the real ones on the pixel level.
We summarize the main points of this paper below.

- •
We propose Vec2Face, a face synthesis model that allows us to efficiently synthesize a large number of face images and identities. Under an appropriate similarity threshold, Vec2Face effectively increases inter-class separability and intra-class variations of synthesized face images.
- •
With a generated dataset of 10k identities and 500k images, we achieve state-of-the-art FR accuracy measured as the average on five real-world test sets compared with existing synthetic datasets. Our performance even surpasses the accuracy obtained by real training data on one of the five test sets. We also report that scaling
to larger synthetic training data leads to further improvement.

## 2 Related Work

Figure: Figure 1: Sample images generated by Vec2Face. From a random vector we generate a face (ID image). We then add random values to this vector to generate diverse face images. Larger perturbation added to this vector results in larger dissimilarity to the ID images. The images in the frame are likely the same people as the ID image.
Refer to caption: x1.png

Class conditioned image generation.
VQGAN (Esser et al., ), MAGE (Li et al., ), DiT (Peebles & Xie, ), U-ViT (Bao et al., ), MDT (Gao et al., ), and VAR (Tian et al., ) control the object classes of generated images using discrete class labels as a condition, and so do not support the generation of new classes. Therefore, they are not applicable to scalable face/identity generation. Different from them, Vec2Face uses continuous face features and thus technically can generate an unlimited number of new identities given appropriate feature sampling.

Feature conditioned image generation. The models trained with conditioned features, either from a text encoder or an image encoder, perform impressively on diverse class generation. However, (Papantoniou et al., ) reports that text embedding exhibits a deficiency in face identity retention. (Chen et al., ; Li et al., ; Xiao et al., ; Wang et al., ; Valevski et al., ; Ye et al., ; Yan et al., ; Papantoniou et al., ) use an image encoder, either CLIP (Radford et al., ) or a FR model, to guide the generation process. These methods are less efficient and effective because 1) the image generation is computationally expensive on these models and 2) they use a fixed vector to generate the images from an identity, which loses the control over the attribute variation. We posit that face feature vectors contain both identity and attribute information. To this end, our method learns image generation and achieves output control purely on face features.

Synthetic face image datasets.
Existing approaches are primarily based on two methods: GAN-based models and diffusion models, with the latter generally producing datasets that enable FR models to achieve better accuracy. Here, we briefly summarize some important works.

GAN-based methods: SynFace (Qiu et al., ), Usynthface (Boutros et al., ), SFace (Boutros et al., ), SFace2 (Boutros et al., ), and ExfaceGAN (Boutros et al., ) leverage pre-trained GAN models to generate datasets. These methods are less effective because the pre-trained GAN models were not trained in an identity-aware manner. Departing from them, our method explicitly uses face features in training, so the generated images are identity-aware.

Diffusion model-based methods: DCFace (Kim et al., ) uses a strong pre-trained diffusion model (Choi et al., ) and an additional style-transfer model to increase the identity separability and attribute variation. Again, this pretrained model is also not identity-aware.
IDiff-Face (Boutros et al., ) combines the pretrained encoder and decoder from VQGAN (Esser et al., ) with a latent diffusion model conditioned by identity features to control the separability of the generated identities. Although the latent diffusion model is identity-aware, the pre-trained VQGAN reduces the sensitivity of identity on the output. Arc2Face (Papantoniou et al., ) fine-tunes a pre-trained stable diffusion model (Rombach et al., ) on the WebFace42M dataset to increase the generalizability on face. It combines the identity features and CLIP (Radford et al., ) to control the identity of output images. The pose variation is increased by adding an additional ControlNet (Zhang et al., ). Neither the pre-trained stable diffusion model nor CLIP are optimized for face and the slow processing speed strongly limits large scale dataset generation. On the contrary, our method is specifically designed for face dataset generation and is both identity-aware and fast in generation. This allows our method to achieve state-of-the-art performance while easily scaling the dataset size to 15M.

## 3 Method

### 3.1 Vec2Face

As shown in Fig. 2, Vec2Face consists of a FR model, a feature Masked Autoencoder, an image decoder, and a patch-based discriminator (Isola et al., ; Yu et al., ).

Feature extraction and expansion.
Given a real-world face image, following (Boutros et al., ; Papantoniou et al., ), we compute its feature using a FR model.
To match the input shape of $MAE_{f}$, the extracted image feature is projected and expanded to a 2-D feature map.

Feature masked auto-encoder (fMAE).
Similar to MAE (He et al., ), the model is forced to learn better information by masking out the input.
Different from the MAE that masks an input image and introduce mask tokens to form the full-size image, the fMAE uses a 2D feature map as input and masks this feature map. To provide more useful information, the image feature is used to form the full-size feature map.
Specifically, the rows in the feature map are randomly masked out by $x\%$ before the encoding process, where $x\%\in\mathcal{N}_{truncated}(max=1,min=0.5,mean=0.75)$, and the image feature is filled in the masked out positions to form the full-size feature map before being processed by the decoder. The structure is in the Appendix.

Image Decoder.
Finally, the new feature map is passed through a simple image decoder to generate/reconstruct the images. To improve image quality, a patch-based discriminator is integrated to form a GAN-type training.

Figure: Figure 2: Architecture of Vec2Face. Given a real face image, we compute its feature “IM feature” using a face recognition (FR) model. The feature is expanded into a feature map, and the latter is processed by a feature masked auto-encoder (fMAE). Inside the fMAE, the rows in the feature map are randomly masked out before being processed by the encoder. The image features are then introduced to form the full-size feature map before being processed by the decoder. Finally, the decoder outputs a feature map. Outside the fMAE, a small image decoder reconstructs the pixels in the original image based on the output of fMAE. During inference, Vec2Face accepts a randomized vector and generates a face image. This process has properties demonstrated in Fig. 1.
Refer to caption: x2.png

Loss function.
The training objective functions include an image reconstruction loss and an identity loss. The reconstruction loss can be written as,

$$ $L_{rec}=MSE(IM_{rec},IM_{gt}),$ (1) $$

which compares the pixel-level difference between the reconstructed image $IM_{rec}$ and the input ground truth images $IM_{gt}$.
The identity loss is written as:

$$ $L_{id}=1-CosineSimlarity(FR(IM_{rec}),FR(IM_{gt})),$ (2) $$

meaning that the features of the reconstructed image and original image extracted using the FR model should be close.
In addition, the perceptual loss (Zhang et al., ) is used to ensure the correct face structure at the early training stage and a patch-based discriminator (Isola et al., ; Yu et al., ) is used to form the GAN loss to increase the sharpness of the generated images. The training loss is:

$$ $L_{total}=L_{rec}+L_{id}+L_{lpips}+L_{GAN}.$ (3) $$

Without $L_{id}$, Vec2Face has slightly degraded
generation performance but still works reasonably well on identity preservation. Without $L_{lpips}$, the reconstructed images do not have clear face structures. Using $L_{GAN}$ at early training stages makes the model convergence difficult, so it is used only after 1,000 epochs. Examples of loss effect on image reconstruction are in Appendix.

Sample generation results. Image generation results are shown in Fig. 3. For both seen and unseen identities, the generated images have very similar identities to the original images.
Moreover, the reconstructed image would remove image border artifacts and some complex backgrounds and translate sketches into photo-realistic images.

Figure: Figure 3: Generation results for training (left) and unseen (right) identities. Images in the same row are of the same identity. We extract features from original images and feed them to Vec2Face. We observe that the reconstructed images still maintain the same identity while removing some image borders and backgrounds and transferring sketches into photo-realistic images.
Refer to caption: x3.png

### 3.2 Sampling identities and face images

For Vec2Face, dissimilar input vectors will likely lead to images of different identities, allowing us to generate different identity images;
controlling the perturbed identity vectors in a proper similarity range with the identity vector can generate face images with different attributes while preserving the identity.

Sampling identity images. Suppose we want to generate $n$ identity images from $n$ identity vectors $v_{1},v_{2},...,v_{n}$, where $v_{i}\in\mathbb{R}^{d},i=1,2,...,n$. We ask that the similarity between any pair of identity vectors be less than a threshold $\tau$: $sim(v_{i},v_{j})\leq\tau,\hskip 5.69054pt\forall\;i\neq j$. This ensures that the generated images are of different identities.
This paper follows (Papantoniou et al., ) in using PCA, without dimension reduction, learned on identity features of MS1MV2 (Deng et al., ) to randomly sample vectors. Specifically, we first compute the mean and covariance of the PCA-transformed identity features, then use these statistics to generate new samples in the PCA space via multivariate normal distribution, and finally inverse transform these samples back to the original feature space to obtain new identity vectors. Newly sampled vectors that have similarity $\leq\tau=0.3$ with all existing vectors are kept, otherwise they are dropped.
Interestingly, we notice that, in the high-dimensional (512 in this paper) feature space, most of the randomly sampled vectors fit the condition due to sparsity. Thus, in practice, only 1.7% of the sampled 300k vectors are filtered out.

Sampling images for each identity. After obtaining $n$ identity vectors $v_{1},v_{2},...,v_{n}$, we sample $m$ vectors for each identity vector by perturbation:

$$ $v_{im_{k}}=v_{n}+\mathcal{N}(0,\sigma),\hskip 5.69054ptk=1,2,...,m,$ (4) $$

where $v_{im_{k}}$ is the $k$th vector for an identity, $k=1,...,m$. $\mathcal{N}(0,\sigma)$ is a Gaussian distribution with 0 0 mean and $\sigma$ variance. To ensure the identity-consistency after perturbation, the similarity values between perturbed vectors and their identity vectors are $\geq 0.5$.

Technically speaking, the above steps allow us to synthesize an unlimited number of identities and images using loosely constrained vectors.
Hyperparameter $\tau$ is selected such that identities are well-separated, *i.e.*, high inter-class variation, although it does not act as a strong filter in practice;
the selection of hyperparameter $\sigma$ should enable the generated images for an identity to have large enough variance while staying on the same identity, *i.e.*, high intra-class variation.

### 3.3 Explicit face attribute control

By default, Vec2Face generates images without explicit attribute control.
This might lead to some attributes being under-represented, *e.g.*, too few profile head poses. We introduce attribute operation, or AttrOP, to guide vector perturbation so that additional images of desired attributes can be generated. This work focuses on face pose and image quality.

Inspired by (Singh et al., ), AttrOP controls the attributes of the generated images by simply adjusting the values in the feature vectors via gradient descent, a process shown in Algorithm 1. Specifically, we first set a target image quality $Q$ and pose angle $P$ and prepare a pretrained pose evaluation model $M_{pose}$ and a pretrained quality evaluation model $M_{quality}$. Then, given an identity vector $v_{id}$ and a perturbed identity vector $v_{im}$, we generate a face image from an adjusted vector $v^{\prime}_{im}$ and compute the following loss functions:

$$ $\displaystyle\mathcal{L}_{attrop}$ $\displaystyle=\mathcal{L}_{id}+\mathcal{L}_{quality}+\mathcal{L}_{pose}\text{ where},$ (5) $\displaystyle\mathcal{L}_{id}$ $\displaystyle=1-CosSim(M_{FR}(IM),v_{id}),$ $\displaystyle\mathcal{L}_{quality}$ $\displaystyle=Q-M_{quality}(IM),$ $\displaystyle\mathcal{L}_{pose}$ $\displaystyle=abs(P-abs(M_{pose}(IM))),$ $$

Because both $M_{pose}$ and $M_{quality}$ are differentiable, gradient descent can be used to adjust $v^{\prime}_{im}$ to minimize $\mathcal{L}_{attrop}$. We finally use the adjusted $v^{\prime}_{im}$ to generate images that exhibit the desired pose and image quality. Sample images optimized by AttrOP are shown in Fig. 4, where profile poses and various image quality levels can be seen.

Figure: Algorithm 1 AttrOP
Refer to caption: x4.png

### 3.4 Discussion

Novelty statement. Generating large-scale training sets presents high demands on inter-class separability and intra-class variation. Controlling the identity by either a coefficient or hard class labels is not effective. Conditioning on identity vectors or clustering generated images with a FR model could effectively generate well-separated identities but needs an additional model to address the shortage of intra-class variation, which is highly dependent on the reference images. This approach lacks flexibility in multi-attribute control. The face features contain all the information of faces, so Vec2Face is designed to learn face generation purely from the given face features. The advantage is that, once the model learns how to draw the target image with the given feature, the output can be easily controlled to add desired information to the vector (*i.e.*, adjusting the values).

Why can Vec2Face generate images like in Fig. 1? The FR model maps images to a hyperspherical space and clusters them. This results in, high similarity for image features from the same identity and low similarity for image features from different identities. Vec2Face learns how to generate face images based on the information in the vectors. Increasing the degree of the perturbation added to the vector gradually changes the vector direction, which results in faces from different identities.

How can Vec2Face increase intra-class separability and inter-class variations? A face feature vector has both identity and attribute information. Hence, controlling the cosine similarity between sampled vectors can enforce separate identities and adding small perturbations to sampled vectors could change the attributes while maintaining the identity. Moreover, AttrOP allows to more easily get the vectors to generate images with desired attributes.

Can other structures work? A diffusion model conditioned on image features may achieve similar function. However, the results in Appendix
A.1
show that the initial noise image may exert a stronger influence on the final output than the conditioned image/identity features, which fails to preserve the identity. Our structure is image-feature-only, and the image feature is used to replace the masked-out rows in the fMAE, ensuring less noise from other factors. Thus, unless there are other special designs, our image-feature-only structure is likely to be more appropriate for this task.

What is an identity? An interesting philosophical question would be how to define an identity. Our research suggests that generated identities can be continuous, where a threshold $\tau$ is used to determine the boundary between different identities. It will be interesting to study how this computational definition of identities connects with and differs from philosophical understanding in future.

## 4 Experiments

### 4.1 Datasets and evaluation protocol

Vec2Face training set. The training data consists of 1M images and their features from 50K randomly sampled identities in the WebFace4M (Zhu et al., ) dataset, where 20 images per identity provides sufficient data for Vec2Face to learn the pattern.

FR model training set. The FR model for feature extraction is ArcFace-R100 trained with Glint360K (An et al., ) dataset. The training configurations are the same as used in the insightFace repository(^1^11https://github.com/deepinsight/insightface/).

Real-world test sets. The FR models are evaluated by five commonly used test sets. LFW (Huang et al., ) tests the FR model in a general case.
CFP-FP (Sengupta et al., ) and CPLFW (Zheng & Deng, ) test the FR model on pose variation.
AgeDB (Moschoglou et al., ) and CALFW (Zheng et al., ) challenge the FR model with large age gap.
Besides these five, Hadrian (Wu et al., ) and Eclipse (Wu et al., ) are used for intra-class variation test, where Hadrian pairs emphasize facial hair difference and Eclipse pairs differ on face exposure. The images in both datasets are indoor and high quality originating from MORPH (Ricanek & Tesafaye, ).
Ability to discriminate between similar-looking persons is tested by two datasets designed for that purpose (Deng et al., ; Thom et al., ).

Evaluation protocol. The image pairs in each test set are split into 10 folds. The average value of the 10-fold cross-validation is used to evaluate the model performance, as suggested in (Huang et al., ).

### 4.2 Experiment details

Vec2Face training: We used the default ViT-Base as the backbone to form the $MAE_{f}$. Since the image size of the commonly used FR training sets is 112x112 and the image decoder is a four-layer, double-sized architecture, the feature map size is set to 49x768. After stacking the projected image features, the entire feature map becomes 50x768. The optimizer is AdamW (Loshchilov & Hutter, ), with a learning rate of 4e-5 and a batch size of 32 per GPU.
We use approximately 10 RTX6000 GPUs for each training run. The patch-based discriminator is enabled after 1000 epochs.

Dataset generation. We sample separable identity vectors by PCA learned on identity vectors of MS1MV2 (Deng et al., ).
We perturb the vectors with the noise sampled from three Gaussian distributions. Specifically, 40% from $\mathcal{N}(0,0.3)$, 40% from $\mathcal{N}(0,0.5)$, and 20% from $\mathcal{N}(0,0.7)$ for each vector.
For extreme pose image generation, we generate 30 images for each identity and randomly replace the existing ones. Specifically, the target pose $\mathcal{P}$ of 20 images is 60 and 10 images is 85. Meanwhile, we observe an image quality degradation with pure pose control, so we control the image quality with a target value 27. The iterations $T$ is initialized as 5, but we add more conditions to the vanilla AttrOP algorithm to ensure the images have the designated attributes.

AttrOP details. MagFace-R100 (Meng et al., ) is used as the quality evaluation model, SixDRepNet (Hempel et al., ) is used as the pose evaluation model, and ArcFace-R100 is used as the identity evaluation model.

FR dataset evaluation metric. The quality of the synthetic datasets is evaluated by the performance of the trained model on five test sets. The standard training dataset size is 0.5M images from 10K identities. Unless otherwise specified, the backbone is SE-IR50 (He et al., ), the recognition loss is ArcFace (Deng et al., ), other configuration details are in the Appendix.

### 4.3 Main Evaluation

**Table 1: Comparison of existing synthetic datasets on five real-world test sets. $\dagger$, $\star$, and $\diamond$ represent diffusion, 3D rendering, and GAN approaches, respectively, for constructing these datasets. We also list the results of training on a real-world dataset CASIA-WebFace.**
| Training sets | # images | LFW | CFP-FP | CPLFW | AgeDB | CALFW | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| IDiff-Face (Boutros et al., )^† | 0.5M | 98.00 | 85.47 | 80.45 | 86.43 | 90.65 | 88.20 |
| DCFace (Kim et al., )^† | 0.5M | 98.55 | 85.33 | 82.62 | 89.70 | 91.60 | 89.56 |
| Arc2Face (Papantoniou et al., )^† | 0.5M | 98.81 | 91.87 | 85.16 | 90.18 | 92.63 | 91.73 |
| DigiFace (Bae et al., )^⋆ | 1M | 95.40 | 87.40 | 78.87 | 76.97 | 78.62 | 83.45 |
| SynFace (Qiu et al., )^⋄ | 0.5M | 91.93 | 75.03 | 70.43 | 61.63 | 74.73 | 74.75 |
| SFace (Boutros et al., )^⋄ | 0.6M | 91.87 | 73.86 | 73.20 | 71.68 | 77.93 | 77.71 |
| IDnet (Kolf et al., )^⋄ | 0.5M | 92.58 | 75.40 | 74.25 | 63.88 | 79.90 | 79.13 |
| ExFaceGAN (Boutros et al., )^⋄ | 0.5M | 93.50 | 73.84 | 71.60 | 78.92 | 82.98 | 80.17 |
| SFace2 (Boutros et al., )^⋄ | 0.6M | 95.60 | 77.11 | 74.60 | 77.37 | 83.40 | 81.62 |
| Langevin-Disco (Geissbühler et al., )^⋄ | 0.6M | 96.60 | 73.89 | 74.77 | 80.70 | 87.77 | 82.75 |
| HSFace10K (Ours)^⋄ | 0.5M | 98.87 | 88.97 | 85.47 | 93.12 | 93.57 | 92.00 |
| CASIA-WebFace (Real) | 0.49M | 99.38 | 96.91 | 89.78 | 94.50 | 93.35 | 94.79 |

Comparison with state-of-the-art synthetic datasets at the scale of 0.5M-0.6M images. In Table 1, we compare FR accuracy of models trained with datasets synthesized by different methods such as diffusion models, GANs, and 3D rendering. We have the following observations.
First, HSFace10K synthesized by Vec2Face yields very competitive accuracy: 98.87%, 88.97%, 85.47%, 93.12%, 93.57% on LFW, CFP-FP, CPLFW, AgeDB, and CALFW, respectively, and 92.00% on average. Our method is only lower than Arc2Face on the CFP-FP dataset (88.97% vs. 91.87%) and is the state of the art on all the other datasets and average. Second, on the CALFW dataset, the accuracy of HSFace10K is higher than, or at least competitive with that of CASIA-WebFace (Yi et al., ) (93.57% vs. 93.32%). To our knowledge, this is the first time that a model trained with synthetic data outperforms those trained with same-size real data. Third, generally speaking, GAN-based methods, while being prevalent, are not as competitive as diffusion-based and 3D rendering methods. Our method is GAN-based but has very strong performance.

**Table 2: Impact of scaling the proposed HSFace dataset to 1M images (20K IDs), 5M images (100K IDs), 10M images (200K IDs), 15M images (300K IDs). Continued improvement is observed. We also list the performance obtained by training on the real-world dataset CASIA-WebFace and its combination with HSFace10K. The latter combination yields even higher accuracy.**
| Methods | # images | LFW | CFP-FP | CPLFW | AgeDB | CALFW | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| HSFace10K | 0.5M | 98.87 | 88.97 | 85.47 | 93.12 | 93.57 | 92.00 |
| HSFace20K | 1M | 98.87 | 89.87 | 86.13 | 93.85 | 93.65 | 92.47 |
| HSFace100K | 5M | 99.25 | 90.36 | 86.75 | 94.38 | 94.12 | 92.97 |
| HSFace200K | 10M | 99.23 | 90.81 | 87.30 | 94.22 | 94.52 | 93.22 |
| HSFace300K | 15M | 99.30 | 91.54 | 87.70 | 94.45 | 94.58 | 93.52 |
| CASIA-WebFace (Real) | 0.49M | 99.38 | 96.91 | 89.78 | 94.50 | 93.35 | 94.79 |
| CASIA-WebFace + HSFace10K | 0.99M | 99.58 | 97.06 | 90.58 | 95.62 | 94.67 | 95.50 |

Effectiveness of scaling up the proposed HSFace dataset.
Vec2Face can easily generate a large number of distinct identity images by sampling more identity vectors which have a cosine similarity $\leq\tau$ with any other identity vectors. Specifically, we generate 10K, 20K, 100K, 200K and 300K IDs, where each identity has 50 images. Results of models trained on these datasets are shown in Table 2.

It is clear that scaling up our synthetic dataset leads to consistent accuracy improvements: from 92.00%, to 92.47%, 92.97%, 93.22%, and 93.52%.
The largest synthetic face dataset prior to this paper has 60k identities and 1.2M images (Papantoniou et al., ). Our dataset reaches 12.5 times larger while still giving steady improvement. These results clearly demonstrate the advantage of Vec2Face in data scaling. Also note that Vec2Face is only trained with 50K real-world identities. We speculate that
a Vec2Face model trained with larger initial real data
could bring about further improvements.

Merging synthetic and real-world training sets. In Table 2, after merging HSFace10K and CASIA-WebFace, we observe improvement over using either dataset alone for training. We obtain 95.50% average accuracy, which is higher than 92.00% (HSFace10K alone) and 94.79% (CASIA-WebFace alone). This indicates that HSFace10K has good quality and is complementary to real data.

**Table 3: Impact of face quality and pose control via AttrOP on FR performance. Each of the training sets has 0.5M images. The accuracy improvement is observed when more controls are added for image quality and pose angles, especially on pose-oriented test sets, CFP-FP and CPLFW.**
| Attr. control | LFW | CFP-FP | CPLFW | AgeDB | CALFW | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| - | 98.27 | 76.56 | 81.70 | 90.75 | 92.92 | 88.04 |
| + Quality 27 | 98.55 | 83.27 | 83.68 | 91.12 | 93.27 | 89.98 |
| + Angle 60^∘ | 98.62 | 86.46 | 85.75 | 92.85 | 93.80 | 91.50 |
| + Angle 85^∘ | 98.87 | 88.97 | 85.47 | 93.12 | 93.57 | 92.00 |

Effectiveness of attribute control.
Images generated by Vec2Face are mostly near-frontal, which can be attributed to such images being most frequent in WebFace4M, the dataset used to train Vec2Face.
Using AttrOP, we synthesize faces with some profile poses where we use target Yaw angles 60^∘ and 85^∘. Results are shown in Table 3. We observe that adding faces with 60^∘ poses leads to 3.46% improvement, while adding 85^∘ gives further 0.5% improvement. These results demonstrate the effectiveness of attribute control. Note that this is achieved with Vec2Face only, instead of resorting to external models as in previous works (Kim et al., ; Papantoniou et al., ).

**Table 4: Computing cost and FID measurement of Arc2Face (Papantoniou et al., ) and Vec2Face.**
| Models | Computing cost | FID |  |  |
| --- | --- | --- | --- | --- |
| Model size | FPS | LFW | Hadrian |  |
| Arc2Face | 3.4GB | 1.5 | 43.80 | 53.27 |
| Vec2Face | 0.68GB | 467 | 35.75 | 51.93 |

Image generation cost and quality of reconstruction. The resource requirement for image generation is an important metric, especially in this large model era. We compare our method with the state-of-the-art face generation model. Experimental details: 1) both models are tested on a single Titan-Xp, 2) the batch size is 8, 3) a 4-step scheduler is
used for Arc2Face. Table 4 shows that our method is 311x faster than Arc2Face. To test the quality of generated images, we use FID to measure the distance between the original LFW and Hadrian images, and the reconstructed images from Arc2Face and Vec2Face. The images are measured at 112x112 resolution. Our model slightly outperforms the Arc2Face results, in size of model, speed of generation, and fidelity to original images
, as shown in Table 4. The similarity measurement and examples of reconstructed images are provided in the Appendix A.1.

### 4.4 Further analysis

Vec2Face generates datasets with large inter-class separability. Fig. 5 presents the number of identities whose cosine similarity against any other identity vectors is less than 0.4, plotted against the number of generated identities in different datasets. A reasonable threshold varies for using different FR models (Kim et al., ; Boutros et al., ). We select 0.4 is because it gives us a reasonable number (183K out of 200K) on a real dataset.
We sample 200K vectors to generate identity images by Vec2Face and Arc2Face, where we report the results of using LCM-lora (Luo et al., ) for Arc2Face. We use ArcFace-R100 to extract identity features from WebFace4M (Zhu et al., ) and the synthesized identity images. We have two observations.

Figure: Figure 5: Inter-class separability measurement. The number of well-separated identities under different identity scales in the synthetic datasets with 10k identities, 200k generated identities from Arc2Face and Vec2Face, and real identities from WebFace4M.
Refer to caption: x5.png

First, synthetic datasets such as SynFace and SFace generally have lower inter-class separability than real-world datasets. This is because their data synthesis methods do not utilize the face feature characteristics for image generation, as Vec2Face and Idiff-Face do. Second, Arc2Face has much larger identity separability than existing methods, comparing with the number reported in the (Kim et al., ), but is still inferior to Vec2Face. This is because the initial noise image sometimes exerts a stronger influence on the final output than the conditioned identity vector.
Third, real-world datasets have slightly lower identity separability than our method, because they have a small number of twins, close relatives, or even the same person with different character names in TV shows (Wu et al., ). Note that 0.4 is not used to evaluate the number of actual identities in the dataset, but rather to provide a consistent measurement of identity separability across datasets.
Overall, the special design of Vec2Face ensures a high rate of transferring the similarity of sampled vectors to the similarity of generated images.

Impact of inter-class separability on accuracy. Table 7 summarizes how different average identity similarity or different levels of identity separability affect FR accuracy. Results indicate that higher identity separability is beneficial, which is consistent with previous finding (Boutros et al., ). Moreover, we find that using a more strict $\tau=0.2$ does reduce a large number of sampled vectors, but it does not further benefit the performance.

Figure: Table 5: Impact of $\sigma$ values with corresponding sampling % for each identity on average FR accuracy (%). We use 0.5M images from 10K identities for training. AttrOP is not applied and Simmin is the minimum cosine similarity between the perturbed vectors and their ID vectors.

Vec2Face generates datasets with large intra-class variation. Table 1 indicates the generated dataset has large variation on age and pose. We also test the performance on the test sets emphasized on the variation of other attributes. Hadrian and Eclipse (Wu et al., ) have 6,000 image pairs for each and respectively control the variation on facial hair and face exposure. Table 6 shows that, while HSFace10k underperforms on accuracy, increasing the dataset size eventually surpasses the accuracy of the real dataset.

Impact of hyperparameter $\sigma$. In Vec2Face, $\sigma$ directly controls intra-class variation, *i.e.*, larger $\sigma$ means higher intra-class variation and vice versa. We test different configurations of $\sigma$ values and its sampling %. Table 5 shows that,
When increasing $\sigma$, FR accuracy first increases and then drops. Initially, increasing intra-class variation is beneficial because it creates some hard training samples. But if $\sigma$ is too large, the generated face may look like a different person, degrading performance. Same for increasing the sampling % of the large $\sigma$.
Thus we choose $\sigma=\{0.3,0.5,0.7\}$ to sample $\{40\%,40\%,20\%\}$ images for each identity.

Does Vec2Face generate new identities compared with its training identities? We compute the feature similarity between the 300k identities from the proposed synthetic data and the 50k training identities selected from WebFace4M. We find that only 0.006%, or 168 out of the 300K synthetic identities are similar to the training identities, with a similarity score greater than 0.4. To completely avoid using real identities for training FR models, we have removed these 168 identities from HSFace300K, and the performance change is negligible.

More images per identity is beneficial. In Table 8, increasing the number of images per identity from 5 to 80 increases the accuracy. The improvement is gradually saturated.

Larger backbone helps the model performance. In Table 9, accuracy of four models trained with HSFace10K and ArcFace loss increases from SE-IResNet18 to SE-IResNet100 and is saturated at SE-IResNet200. It is also useful for training with Vision Transformer (Dosovitskiy, ) backbones.

Limitations.
The proposed synthetic dataset is effective for general FR but would be much less useful for fine-grained tasks
such as discriminating between doppelgangers. See Table 6.
We believe a better understanding of ‘identity’ and stronger generation models will help solve this issue and leave this to future work.

## 5 Conclusions

This paper proposes Vec2Face, a face generation method which can elegantly create a large number of identities and face images. This method conveniently converts vectors to face images and can translate perturbations of the vectors into consistent perturbations of the resulting face images. Because of its unique design, the inter-class and intra-class variations of generated face images can be directly controlled by hyperparameters in Vec2Face. We show that the resulting dataset, HSFace10k, has large intra- and inter-class variations and yields state-of-the-art FR accuracy. Importantly, Vec2Face allows for scaling HSFace to 300K identities and 15M images while still seeing accuracy improvement. In future work, we will study the definition of identity to solve more challenging FR tasks, try other generation structures, and extend this method to generic objects.

## Appendix A Appendix

Figure: Figure 6: Similarity measurement between original and reconstructed in-the-wild images (LFW (Huang et al., )).
Refer to caption: extracted/5870616/images/open-set-generation/arc2face_vec2face_lfw.png

### A.1 Comparison with Diffusion Model

Training a diffusion model conditioned with image feature vectors is a similar approach to Vec2Face. We evaluate both architectures by measuring the performance on open-set image reconstruction. The Arc2Face (Papantoniou et al., ) model, which fine-tuned the stable-diffusion-v1-5 on WebFace42M, is used to represent the diffusion models. The images in an in-the-wild dataset, LFW (Huang et al., ), and an indoor dataset, Hadrian (Wu et al., ) are used for evaluation. The features are extracted by ArcFace-R100 and fed to both models. Since both dataset are trained/fine-tuned on the cropped and aligned dataset, no image cropping and alignment are used in this evaluation. Fig. 7 and Fig. 7 show the similarity distribution between the original images and reconstructed images. The observations are: 1) both models can preserve the identity for the in-door images, 2) Vec2Face has better performance on open-set image reconstruction on both datasets, 3) the diffusion model conditioned with image/identity features do not always preserve the identity, especially for in-the-wild images. Hence, the proposed architecture is better than the conventional diffusion model conditioned with image features. Examples are shown in Fig. 19.

### A.2 Ablation study of loss functions

This section presents the effects of identity loss $L_{id}$, perceptual loss, $L_{lpips}$ and GAN loss $L_{GAN}$ on image reconstruction. The observations in Fig. 8 are: 1) without perceptual loss, the reconstructed face edges are smoothed, 2) involving GAN loss at an early training stage causes a glitch effect on image reconstruction, 3) without identity loss, Vec2Face performs well on near-frontal images but not on images with large pose variations.

Figure: Figure 8: Examples of loss effect during training. The reconstructed examples with corresponding loss involved are on the left, otherwise on the right. For identity loss, odd rows are reconstructed images and even rows are original images.
Refer to caption: x6.png

### A.3 Dataset Noise Analysis

Figure: Figure 11: Similarity distributions of available synthetic datasets (Qiu et al., ; Boutros et al., ; Bae et al., ; Boutros et al., ; Kim et al., ) and ours. The results indicate that our dataset has the largest separability and smallest overlap / Equal Error Rate between genuine (same ID) and impostor (different ID) pairs.
Refer to caption: extracted/5870616/images/dataset_comparison/synface.jpg

Figure: (a) Intra-class Similarity Distributions
Refer to caption: extracted/5870616/images/Intra-Noise_similarity_violin_plot.png

Dataset noise could be categorized as: i) images from different identities are in the same identity folder, and ii) images from the same identity are in different identity folders. For synthetic datasets, there is no ground truth for identities, so we used the labels provided by the corresponding work.

Following the metric used in previous works (Cao et al., ; Wu & Bowyer, ; Zhu et al., ; Deng et al., ), we evaluate the noise using two hard threshold values: one for the similarity between image features and their corresponding identity features, and another for the similarity across identity features. Figure 12(a) shows the cosine similarity distributions of the available synthetic datasets in situation one. Since greater similarity means images are more likely from the same identity, the results show that our dataset has the highest identity consistency within each identity folder. Moreover, generally, if the similarity value of an image feature and its identity feature is less than 0.3, it can be regarded as an outlier/noise (Deng et al., ).
Hence, the available synthetic datasets contain greater intra-class noise than ours.
Figure 12(b) shows the similarity distributions of available synthetic datasets in situation two. WebFace260M (Zhu et al., ) is the only work discussing the details of inter-class denoising. It merged two identity folders if their identities have higher than 0.7 similarity, so if the identity similarity is larger than 0.7, we regard it as an inter-class noise. The results show that, except for DCFace and ours, the other datasets have inter-class noise to some extent.

Generally, our proposed dataset has the lowest intra-class noise and inter-class noise. Moreover, Figure 11 shows that the proposed dataset has the lowest overlapped area / Equal Error Rate (EER), showing the best dataset quality on separability between genuine (same people) and impostor (different people) pairs. It promises more reliable identity labels for the FR algorithm to learn better representations.

### A.4 Age and pose variation

Figure: Figure 15: Pose and age variation across five test sets and HSFace10k (ours).
Refer to caption: extracted/5870616/images/pose_distribution/synthetic_datasets_Pitch.png

To estimate the distribution of age and pose, we use img2pose (Albiero et al., ) and an age estimator (Albiero et al., ) to obtain the data. Fig. 15 shows the distributions of pose and age of five available synthetic datasets and HSFace10k. Since roll is controlled by face detectors, only pitch and yaw angles are estimated. The results show that all the datasets have large variations on both pose and age, suggesting the accuracy difference is not mainly caused by the attribute variation but by identity consistency. Having large variations but losing the identity consistency results in lower accuracy.

### A.5 Feature Interpolation and Feature Value Impact

Figure: Figure 16: Feature interpolation results. These show that Vec2Face can smoothly convert one image to another by simply changing the feature values.
Refer to caption: x7.png

Figure: Figure 17: Impact of feature values analysis. The first row shows images generated with feature values, from left to right, of {0, 1, 1.5, 2, 3, 10, 100}. The second row shows images generated with feature values of {-0.5, -1, -1.5, -2, -3, -10, -100}. The last row shows images generated with feature values of {1, 0.5, 0.25, 0.125, 0.0625, 0.03125, 0.015625}.
Refer to caption: x8.png

We analyze the impact of feature values on the generated images in two ways: 1) interpolating eight feature vectors between two image features and 2) proportionally changing the values in the feature vector. Figure 16 shows the results of feature interpolation between two images. It indicates that the interpolation in the feature domain can be smoothly presented in the image domain, showcasing Vec2Face’s capability of understanding the vector characteristics in the feature domain. Figure 17 shows the results of proportionally changing the values in the feature. The observations are: 1) As the values in a feature vector increase to a large extent, the image stops changing but the quality decreases, for both negative and positive directions; 2) As the values in a feature vector become close to zero, the face attributes are erased and eventually disappear.

### A.6 Impact of values at fixed dimensions

Vec2Face draws an image based on the values in the feature vector. We first change the values in one dimension. Fig. 20 shows the cases where some obvious patterns are observed. For each of the four identities, the value at index 15 is related to the hair volume, pose angle, age and expression; the value at index 26 have a high correlation with the bangs, facial hair and age, eyeglasses presence, and expression; increasing the value at index 32 changes the age, head pose, hair color, and expression. The observation is that changing values in a single dimension may change the face attribute but it is not consistent for all identities.

We then change the values in fixed-length (*i.e.*, 8) of dimension chunks and there are some noticeable patterns. Fig. 21 shows the generated images when the weight at specific dimensions increases. A general conclusion is that changing values at dimension chunks varies the facial attributes but the patterns are inconsistent across identities. Specifically, dimension [40:48] changes the age for the first identity, no obvious pattern for the second, the expression for the third, and head pose for the fourth; dimension [56:64] changes the hairstyle for the first identity, facial hair for the second, age and eyeglasses for the third, age and facial hair for the fourth; dimension [96:104] changes the gender for the first identity, age and hairstyle for the second, face exposure level for the third, and facial hair for the fourth. Therefore, it is hard to control the attributes by handcrafting the features. AttrOP is much more efficient and effective.

Figure: Figure 18: Architecture of the proposed feature masked autoencoder (fMAE).
Refer to caption: x9.png

**Table 10: Configurations used for synthetic dataset training.**
| Face Recognition Model Training Configurations |  |
| --- | --- |
| Head | ArcFace |
| Backbone | SE-IR50 |
| Input Size | 112$\times$112 |
| Batch Size | 128 |
| Learning Rate | 0.1 |
| Weight Decay | 5e-4 |
| Momentum | 0.9 |
| Epochs | 26 |
| Margin | 0.5 |
| FP16 | True |
| Sample Rate | 1.0 |
| Reduce Learning Rate | [12, 20, 24] |
| Augmentation | Random aug. and Random erase |
| Optimizer | SGD |
| Workers | 2 |
| GPU | RTX6000 |

Figure: Figure 19: Examples of reconstructed Hadrian and LFW images by Arc2Face and Vec2Face. The images in each three-image group, from left to right, are the original, from Arc2Face, and from Vec2Face.
Refer to caption: x10.png

Figure: (a) Changes happen in feature[15]
Refer to caption: extracted/5870616/images/value-change-at-single-dim/id1/0015/000.jpg

Figure: (a) Changes happen in feature[40:48]
Refer to caption: extracted/5870616/images/value-change-at-dim-chunks/id1/0005/000.jpg

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: 1.	Despite the statement provided in Section 3.4., the novelty introduced with this paper remains slightly unclear. The overall paper gives the impression that the fMAE is novel, however, Section 3.4. does not seem to point this out. Can you perhaps elaborate more on the novelty of the fMAE? Does it share similarities or draw inspirations from previous works, namely [1]? Is the novelty perhaps the projection and expansion of input features to 2D feature maps? 

[1] Hu, Junhao, et al. "Features Masked Auto-Encoder-Based Anomaly Detection in Process Industry." 2023 IEEE 12th Data Driven Control and Learning Systems Conference (DDCLS). IEEE, 2023.

2.	When discussing the influence of inter-class separability on the accuracy of trained recognition models, it is observed that a too large separation does not benefit the performance. However, this might not be a unique finding, as results in [2] also showcase that well-separated synthetic identities actually result in lower verification performance. Can you comment more on this phenomenon? Does this mean that future research will have to determine the best balance between inter-class separability and intra-class variation?

[2] Boutros, Fadi, et al. "IDiff-Face: Synthetic-based face recognition through fizzy identity-conditioned diffusion model." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
soundness: 3
strengths: The submitted paper is writte well and presents the problem and proposed solution in a concise and clear manner. The proposed approach improves on the state-of-the-art with a unique fMAE architecture, differentiating itself from existing diffusion and GAN-based solutions. Differently, it also proposes to utilize guided perturbation during inference, to not only achieve higher intra-identity diversity of samples but also allow for control over the pose and quality of images, which is crucial for generating large-scale datasets suitable for training recognition models. As a result, the recognition model trained on the generated data of Arc2Face even achieves better results on the Cross-Age LFW benchmark than when training with the real CASIA-WebFace dataset. This represents a tremendous achievement as existing approaches typically achieve worse performance. The proposed approach also enables scaling the generated datasets without drastically impacting inter-class separability, in turn achieving significantly larger dataset sizes than existing approaches. Overall, the paper is suitable for the conference and will likely highly influence future research on the generation of synthetic biometric datasets.
summary: The paper introduces a novel Vec2Face generative framework for generating identity-specific synthetic face images suitable for training deep recognition models. To this end, the authors rely on a feature Masked Auto-Encoder (fMAE) architecture, which is trained to produce face images of a desired identity based on input features extracted from a pretrained Face Recognition (FR) model. However, these identity features are first projected and expanded to 2D feature maps to suit the nature of the fMAE architecture. Training is performed with four objectives, including the standard reconstruction and perceptual loss functions, along with an identity-based loss, which enables better identity consistency, and a GAN loss that utilizes a patch-based discriminator to ensure sharper images. During inference, the approach utilizes PCA for sampling unique identity vectors, upon which perturbation is performed to obtain diverse samples of a specific identity. Additionally, the approach also supports control over the pose and quality, through guided vector perturbation. The authors showcase the suitability of their approach through extensive experiments across multiple datasets, with a focus on training deep face recognition models with the generated synthetic data. The proposed approach outperforms existing state-of-the-art methods in terms of the accuracy achieved with a trained recognition models on standard verification benchmarks. Importantly, Vec2Face also enables better scalability than existing approaches, thus allowing the generation of diverse datasets of up to 20 million images, compared to existing datasets of 500 thousand images. Furthermore, the authors also perform sufficient ablation studies for supporting the various choices of their particular solution.
weaknesses: Despite the many strengths of the paper, there are a few areas where the paper could be further improver. The evaluation of the approach mainly revolves around the accuracies achieved with trained recognition models on existing benchmarks (apart from Table 4 and Figure 5). While this focus is understandable, it would be beneficial to also analyse other aspects of the data.  

1.	Face Image Quality Assessment (FIQA) measures could provide additional insight as they are specifically designed for evaluating the quality of face images and their suitability for recognition tasks. Examples of recent FIQA measures with publicly available implementations include CR-FIQA (Boutros et al., CVPR 2023) and DifFIQA (Babnik et al., IJCB 2023). 

2.	Comparison with the state-of-the-art (e.g. Table 4) could also be expanded with metrics that measure fidelity and diversity separately could also be utilized to provide a deeper understanding of the generated datasets, e.g., improved precision and recall (Kynkäänniemi et al., NeurIPS 2019).   

3.	In the supplementary material when comparing genuine and imposter distributions it would also be beneficial to provide results obtained with a real-world dataset to provide the baseline of desired distributions suitable for training face recognition models. 

4.	The utilized verification benchmarks are of a rather small scale (roughly 3000 genuine and imposter pairs). It might be beneficial to evaluate the approach on a more complex and larger-scale set, e.g. the IJB-C verification benchmark. There the difference performance improvements of the Vec2Face method could be further highlighted. 

Lastly, Despite the detailed nature of the paper, the utilized patch-based discriminator is rather poorly explained. More information should be added to the paper or at least to the supplementary material. For example, what sort of architecture does it utilize? Is it initialized with a pretrained model? The formulation of perceptual and GAN loss functions could also be provided.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * There was no reproducibility section in the paper do the authors plan to publish their code/ dataset?
* I was wondering if the authors considered using random perturbed vectors instead of features from an FR model in the training phase. 
* As the authors used the subset of WebFace4M (50K identities) for training their method, have authors considered using an FR system trained on this dataset for the ID loss in their proposed method? In this case, one of the weaknesses that I mentioned would be in some extent addressed, if authors can observe the same trend in the results. 
* The authors used an FR system trained on MS1MV2,  Glint360K, in their training objective prior, it would be informative if the authors could add the performance of such a system in Table 2.
* In Table.2, I am curious to see what will happen if we add more synthetic images to the CASIA, let’s say CASIA+HSFace200K.
* [392] As the improvement is on the order of 0.1% could this be also the case if authors changed the network, seed, or margin loss?
soundness: 1
strengths: * The paper is well written and the experiments are exhaustive and insightful both in the appendix (especially the analysis of the attributes of the synthetic data) and the main content.
* Applying MAE on the feature space and formulating it to work with the sampling of the random vectors is quite interesting.
summary: In this paper by using a Masked AutoEncoding on the feature space and a separate FR system, Pose Estimator and Face Quality assessment network authors introduced a pipeline for sampling identities to generate synthetic face dataset for training FR systems.
weaknesses: * As the main point of the paper is to generate a synthetic FR dataset, my main concern is that the authors have used the FR model trained on MS1MV2 and Glint360K, and a large subset of the WebFace4m for training/sampling from their method. Training an FR model on  these datasets will probably outperform the FR model trained on the CASIA-WebFace (this dataset used for comparison in Table.1 and Table .2). Later by generating the datasets in a controlled manner they still did not perform as good as a model trained on the CASIA-WebFace, although they are close in most of the benchmarks, so what is the main problem that the authors trying to solve? Why we can not use the CASIA-WebFace, Glint360K, MS1MV2, or the subset of the large WebFace4M that authors used for training their method which probably performs better? Is it trying to address the privacy issues? Are the generated identities different from the ones presented in the dataset used? 
Authors may consider the goal of synthetic dataset generation, especially when their approach assumes the existence of a high-quality FR system or a large-scale dataset already available for training their methodologies.
I will consider raising my score if the authors can argue a benefit of this work over the datasets and models they used for training their methodology

* Maybe evaluation using more benchmarks like TinyFace might be needed.
* There is a good amount of studies that show the InceptionV3 trained on the Imagenet is not a good candidate for reporting Frechet Distance: 
   [1] The role of Imagenet classes in Frechet inception distance: https://arxiv.org/pdf/2203.06026

   [2] Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models: https://proceedings.neurips.cc/paper_files/paper/2023/hash/0bc795afae289ed465a65a3b4b1f4eb7-Abstract-Conference.html

* Some minor Typos: 
  [366] surround the pose with braces to appear in the subscript.
  
  APP[188] five test sets -> Five Synthetic Data methods maybe?

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Choice of Pre-trained FR Model:
    a. You've chosen to use ArcFace-R100 pre-trained on Glint360K. Could you elaborate on the reasons behind this specific choice?
    b. Are there any limitations to using a smaller pre-trained FR model trained on a smaller dataset? It seems that requiring a large-scale, pre-trained FR model could limit the applicability of your approach, especially in scenarios where such a model isn't readily available. This creates a potential "chicken-and-egg" problem where generating synthetic data requires access to large-scale real data. Could you address this potential limitation?

2. Number of Identities:
    a. Is there a specific reason for limiting the number of generated identities to 300K or 400K?
    b. Given that your filtering process removes only 1.7% of generated images, is it possible to generate a larger number of identities (e.g., 500K or 1M)? If so, what are the potential challenges or trade-offs involved?

3. Attribute Control During Inference:
    a. You mention that your method "seamlessly supports attribute control during inference." However, it seems that this relies on three external models: pre-trained pose, quality, and FR models. Could you clarify what you mean by "seamless" in this context?
    b. Does relying on these external models introduce any constraints or complexities in practice?
soundness: 2
strengths: 1. The pipeline presented in the paper is simple and efficient.
2. The use of a feature-masked autoencoder is somewhat novel.
summary: The paper introduces a novel method called Vec2Face to synthesize large-scale, high-quality face images for training face recognition models. This method addresses the limitations of existing techniques by:
1. Generating a vast number of distinct identities: Vec2Face can generate up to 300,000 unique identities, significantly surpassing previous methods.
2. Controlling intra-class variation: The model can generate diverse variations of the same identity by perturbing the input vector, enabling realistic appearance changes.
3. Manipulating attributes: Vec2Face allows for the generation of images with specific attributes by adjusting the input vector using gradient descent.
weaknesses: 1. Limited Scale of Synthetic Data: While the paper proposes a method for generating synthetic faces, the number of identities generated (300K-400K) is comparable to existing real-world datasets used to train facial recognition (FR) models. This raises concerns about the practical benefits of generating synthetic data at this scale.
2. Lack of Comparison with Larger Datasets: The paper lacks experiments comparing the effectiveness of the generated synthetic data with larger-scale real-world datasets like Glint360K. This comparison is crucial to demonstrate the advantages of the proposed approach in scenarios where abundant real data is available.
3. Unclear Practical Motivation: The authors do not provide a compelling reason for generating a synthetic dataset with a similar or smaller scale than commonly used real-world FR training datasets.  It is essential to clarify the specific use cases and advantages of this approach, particularly when real data is readily accessible.
4. Missing Citation: The paper omits a key citation: "Vec2face: Unveil human faces from their black box features in face recognition" (CVPR 2020). This work appears relevant to the paper's methodology and should be properly acknowledged.

Misleading Novelty Statements: Certain claims of novelty in the paper are potentially misleading.  (See the next questions section for more details.)  It is crucial to ensure that all claims are accurate and supported by evidence.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weakness. 

Overall, I believe the novelty of this paper is incremental, and improvement is good but does not achieve state-of-the-art. Furthermore, privacy issues arise as this paper adopts a lot of labelled face recognition datasets.
soundness: 3
strengths: 1. The proposed method achieves good performance compared to the listed literature.
2. It can generate as many novel identities as possible.
3. The experiments are sufficient.
4. Using GAN to achieve such results is inspiring.
summary: This paper proposes VEC2FACE to synthesize facial images from a sampled identity vector for synthetic face recognition. It utilizes a GAN solution to reconstruct the image at the training stage and adopt the decoder for inference. High intra-class variation is ensured by Perturbing identity vectors. comprehensive experiments are conducted to demonstrate the effectiveness of the method.
weaknesses: 1. [Major] lines 219-220, the authors state 'Following (Papantoniou et al., 2024), PCA is used to sample vectors.'. This brings a concern that the contribution of generating as many novel identities as possible is benefited from the previous work Arc2Face. Additionally, adopting pure identity embedding to generate a facial dataset is demonstrated in the related literature, ID3.

2. [Major] After attribute OP, how do you make sure the generated image preserves the identity, Do you have experiments to prove it? The attribute operation is also introduced in ID3.

3. [Major] line 314, Vec2Face training set is WebFace4M, the PCA is carried on MS1MV2, the feature extraction model is from Glint360K. This counters the privacy concerns, as the dataset original MS1M dataset is recalled by its creator. And the whole training requires a lot of face recognition dataset, which is a disadvantage over DCFace/ID3.

4. [Major] In addition, the training dataset is webface4m (1M with 50k identities ), which is significantly larger than CASIA-WebFace (500k images with 10k identities) adopted by the competitors DCFace/ID3, making the comparison with previous methods unfair. 

5. [Major] The authors state they achieve state-of-the-art results. However, the SOTA result is 92.30 (on 0.5M volume) and 93.07  (on 0.5M volume), achieved by CemiFace[1].

6. [Minor]How to generate the image is not clear. For example, how do you obtain the identity feature, randomly generated or from the image? 

7. [Minor] A study about alternative pretrained FR model should be conducted, as the proposed method uses ArcFace pretrained from Glint360k, DCFace adopted model pretrained from WebFace4M.

8. [Minor] Line 243, ‘too few profile head poses’. Do you have statics results?

9. [Minor] Eq. 5 is trained with the GAN?

10. Long-page discussions might be better if positioned in the appendix.

Reference:
CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition

Meta-review: The rebuttal provided clarifications about the proposed method and its analysis that were useful for assessing the paper's contribution and responded adequately to most reviewer concerns. After discussion, reviewer XvoX recommended acceptance, LGwK recommended marginal acceptance, G33d and oM4G recommended marginally below acceptance. The AC agreed this work is valuable to the ICLR community. The final version should include all reviewer comments, suggestions, and additional clarifications from the rebuttal.

Decision: Accept (Poster)