Title: Face Reconstruction from Facial Templates by Learning Latent Space of a Generator Network
Venue: NeurIPS 2023
Accepted: yes
Keywords: Face Recognition (FR)

Face reconstruction

Generative Adversarial Network (GAN)

Privacy

Security

Template Inversion (TI) attack

Transferability
Abstract: In this paper, we focus on the template inversion attack against face recognition systems and propose a new method to reconstruct face images from facial templates. Within a generative adversarial network (GAN)-based framework, we learn a mapping from facial templates to the intermediate latent space of a pre-trained face generation network, from which we can generate high-resolution realistic reconstructed face images. We show that our proposed method can be applied in whitebox and blackbox attacks against face recognition systems. Furthermore, we evaluate the transferability of our attack when the adversary uses the reconstructed face image to impersonate the underlying subject in an attack against another face recognition system. Considering the adversary's knowledge and the target face recognition system, we define five different attacks and evaluate the vulnerability of state-of-the-art face recognition systems. Our experiments show that our proposed method achieves high success attack rates in whitebox and blackbox scenarios. Furthermore, the reconstructed face images are transferable and can be used to enter target face recognition systems with a different feature extractor model. We also explore important areas in the reconstructed face images that can fool the target face recognition system.

Face Reconstruction from Facial Templates by Learning Latent Space of a Generator Network

Abstract

In this paper, we focus on the template inversion attack against face recognition systems and propose a new method to reconstruct face images from facial templates. Within a generative adversarial network (GAN)-based framework, we learn a mapping from facial templates to the intermediate latent space of a pre-trained face generation network, from which we can generate high-resolution realistic reconstructed face images. We show that our proposed method can be applied in whitebox and blackbox attacks against face recognition systems. Furthermore, we evaluate the transferability of our attack when the adversary uses the reconstructed face image to impersonate the underlying subject in an attack against another face recognition system. Considering the adversary's knowledge and the target face recognition system, we define five different attacks and evaluate the vulnerability of state-of-the-art face recognition systems. Our experiments show that our proposed method achieves high success attack rates in whitebox and blackbox scenarios. Furthermore, the reconstructed face images are transferable and can be used to enter target face recognition systems with a different feature extractor model. We also explore important areas in the reconstructed face images that can fool the target face recognition system.

1 Introduction

Face recognition (FR) systems tend toward ubiquity, and their applications, which range from cell phone unlock to national identity system, border control, etc., are growing rapidly. Typically, in such systems, a feature vector (called embedding or template) is extracted from each face image using a deep neural network, and is stored in the system's database during the enrollment stage. During the recognition stage, either verification or identification, the extracted feature vector is compared with the ones in the system's database to measure the similarity of identities. Among potential attacks against FR systems [Galbally et al., 2014, Biggio et al., 2015, Hadid et al., 2015, Mai et al., 2018, Marcel et al., 2023], the template inversion (TI) attack significantly jeopardizes the users' privacy. In a TI attack, the adversary gains access to templates stored in the FR system's database and aims to reconstruct the underlying face image. Then, the adversary not only achieves privacy-sensitive information (such as gender, ethnicity, etc.) of enrolled users, but also can use reconstructed face images to impersonate.

In this paper, we focus on the TI attack against FR systems and propose a novel method to reconstruct face images from facial templates (Figure 1 shows sample reconstructed face images using our proposed method). Within a generative adversarial network (GAN)-based framework, we learn a mapping from face templates to the intermediate latent space of StyleGAN3 [Karras et al., 2021], as a pre-trained face generation network. Then, using the synthesis part of StyleGAN3, we can generate high-resolution realistic face image. Our proposed method can be applied for whitebox and blackbox attacks against FR systems. In the whitebox scenario, the adversary knows the internal functioning of the feature extraction model and its parameters. However, in the blackbox scenario, the adversary does not know the internal functioning of the feature extraction model and can only use it to extract features from any arbitrary image. Instead, we assume that the adversary has a whitebox of another FR model, which can be used for training the face reconstruction network. We also evaluate the transferability of our attack by considering the case where the adversary uses the reconstructed face image to impersonate the underlying subject in an attack against another FR system (which has a different feature extraction model). Considering the adversary's knowledge and the target FR system, we define five different attacks, and evaluate the vulnerability of state-of-the-art (SOTA) FR systems. Figure 2 illustrates the general black diagram of our proposed template inversion attack.

To elaborate on the contributions of our paper, we list them hereunder:

We propose a novel method to generate high-resolution realistic face images from facial templates. Within a GAN-based framework, we learn the mapping from facial templates to the latent space of a pre-trained face generation network.

We propose our method for whitebox and blackbox scenarios. While our method is based on the whitebox knowledge of the FR model, we extend our attack blackbox scenario, using another FR model that the adversary has access to.

We define five different attacks against FR systems (based on the adversary's knowledge and the target system), and evaluate the transferability of the reconstructed face images and vulnerability of SOTA FR models to TI attacks. We also explore important areas in the reconstructed face images. To our knowledge, this is the first work which comprehensively evaluates the transferability of the reconstructed face images in TI attacks.

The remainder of the paper is organized as follows: Section 2 introduces the problem formulation and our proposed face reconstruction method. Section 3 covers the related works in the literature and

Mapping

Network

(Ours)

Synthesis

Network

Noise

(           )

Template

(           )

(                         )

Reconstructed

Face Image (    )

Face Reconstruction Network (trained using       )

Target FR system (        )

FR system from which the

template is leaked (           )

Database

Database

Feature

Extraction

(              )

Leaked Template

Feature

Extraction

(          )

Compare

Recognized ?

Injecting the Reconstructed Face Image

compares them with our proposed method. Section 4 presents our experiential results. Finally, the paper is concluded in Section 5.

2 Problem Definition and Proposed Method

In this paper, we consider a TI attack against a FR system based on the following threat model:

Adversary's goal : The adversary aims to reconstruct a face image from a template, and use the reconstructed face image to enter the same or a different face recognition system, which we call the target FR system.

Adversary's knowledge: The adversary knows a face template of a user enrolled in the FR system's database. The adversary also has either whitebox or blackbox knowledge of the feature extractor model in the same FR system.

Adversary's capability: The adversary can present the reconstructed face image to the target FR system (e.g., using a printed photograph). However, for simplicity, we consider that adversary can inject the reconstructed face image as a query to the target FR system.

Adversary's strategy: The adversary can train a face reconstruction model to invert facial templates and reconstruct underlying face images. Then, the adversary can use the reconstructed face images to inject as a query to the target FR system, to enter that system.

Let F ( . ) denote a facial feature extraction model, which gets the face image I ∈ I and extracts facial template x = F ( I ) ∈ X . According to the threat model, the adversary has access to the target facial template x database = F database ( I ) and aims to generate a reconstructed face image ˆ I . Then, the adversary can use the reconstructed face image ˆ I to impersonate the corresponding subject and attack a target FR system with F target ( . ) , which might be different from F database ( . ) .

To train a face reconstruction model, we can use a dataset of face images { I i } N i =1 with N face images (no label is required), and generate a training dataset { ( x i , I i ) } N i =1 , where x i = F database ( I i ) . Then, a face reconstruction model G ( . ) can be trained to reconstruct face image ˆ I = G ( x ) given each facial template x ∈ X . To train such a face reconstruction model, we consider a multi-term face reconstruction loss function as follows:

where L pixel and L ID indicate pixel loss and ID losses, respectively, and are defined as:

The pixel loss is used to minimize the pixel-level reconstruction error of the generated face image. The ID loss is also used to minimize the distance between facial templates extracted by F loss ( . ) from original and reconstructed face images. In Eq. 3, F loss ( . ) denotes a feature extraction model that the adversary is assumed to have complete knowledge of its parameters and internal functioning. Based on the adversary's knowledge of F database ( . ) (i.e., whitebox or blackbox scenarios), F loss ( . ) might be the same or different from F database ( . ) .

For the face reconstruction model, we consider StyleGAN3 [Karras et al., 2021], as a pre-trained face generation network 1 . The StyleGAN3 model is trained on a dataset of face images using a GAN-based framework that can generate high-resolution and realistic face images. The structure of StyleGAN3 is composed of two networks, mapping and synthesis networks. The mapping network M StyleGAN ( . ) gets a random noise z ∈ Z and generates an intermediate latent code w = M StyleGAN ( z ) ∈ W . Then, the latent code w is given to the synthesis network S StyleGAN ( . ) to generate a face image. In our training process, we fix the synthetic network S StyleGAN ( . ) and train a new mapping M rec ( . ) to generate ˆ w corresponding to the given facial template x ∈ X . Then, the generated latent code ˆ w is given to the synthesis network S StyleGAN ( . ) to generate the reconstructed face image ˆ I = S StyleGAN ( ˆ w ) . We can

train our new mapping M rec ( . ) using our reconstruction loss function as in Eq. 1. However, to obtain a realistic face image from the generated ˆ w through the pre-trained synthetic network S StyleGAN ( . ) , the generated ˆ w needs to be in the distribution W ; otherwise, the output may not look like a real human face. Hence, to generate ˆ w vectors such that they have the same distribution as StyleGAN's intermediate latent, w ∈ W , we use a GAN-based framework to learn the distribution W . To this end, we use the Wasserstein GAN (WGAN) [Arjovsky et al., 2017] algorithm to train a critic network C ( . ) which critiques the generated ˆ w vectors compared to the real StyleGAN's w ∈ W vectors, and simultaneously we optimize our mapping network to generate ˆ w vectors with the same distribution as W . Hence, we can consider our mapping network M rec ( . ) as a conditional generator in our WGAN framework, which generates ˆ w = M rec ([ n , x ]) given a facial template x ∈ X and a random noise vector n ∈ N . Then, we can train our mapping network and critic network using the following loss functions:

In a nutshell, we train a new mapping network M rec ( . ) using our reconstruction loss function in Eq. 1, and also optimize M rec ( . ) within our WGAN framework using Eq. 5. Simultaneously, we also train the critic network C ( . ) within our WGAN training using Eq. 4 to learn the distribution of StyleGAN's intermediate latent space W and help our mapping network M rec ( . ) to generate vectors with the same distribution as W . Figure 3 depicts the block diagram of the proposed method. We should note that our mapping network M rec ( . ) has 2 fully connected layers with Leaky ReLU activation function.

Mapping

Network

(Ours)

Synthesis

Network

Noise

(           )

Template

(           )

(                      )

Critic

Network

Reconstructed

Face Image (   )

Mapping

Network

Noise

(           )

real

fake

(          )

In our problem formulation, we consider three different feature extraction models, including F database ( . ) , F loss ( . ) , and F target ( . ) . Hence, based on the adversary's knowledge and the target system, we can consider five different attacks:

Attack 1: The adversary has whitebox knowledge of the system from which the template is leaked and wants to attack the same system (i.e., F database = F loss = F target ).

glyph[negationslash]

Attack 2: The adversary has whitebox knowledge of the feature extractor of the system from which the template is leaked, but aims to attack to a different FR system (i.e., F database = F loss = F target ).

glyph[negationslash]

Attack 3: The adversary wants to attack the same system from which the template is leaked, but has only blackbox access to the feature extractor of the system. Instead, we assume that the adversary has the whitebox knowledge of another FR model to use for training (i.e., F database = F target = F loss ).

glyph[negationslash]

Attack 4: The adversary aims to attack a different FR system than the one from which the template is leaked. In addition, the adversary has whitebox knowledge of the feature extractor of the target system (i.e., F database = F loss = F target ).

glyph[negationslash]

Attack 5: The adversary aims to attack a different FR system from which the template is leaked and has only blackbox knowledge of both the target system and the one from which the template is leaked. However, the adversary instead has the whitebox knowledge of another FR model to use for training (i.e., F database = F loss = F target ).

glyph[negationslash]

In the attack 1 and attack 2, the adversary has the whitebox knowledge of the system from which the template is leaked (i.e., F database ( . ) ) and uses the same model as F loss ( . ) for training the reconstruction network. However, in attacks 3-5, the adversary has the blackbox knowledge of the system from which the template is leaked, and therefore uses another FR model as F loss ( . ) . Comparing the knowledge

Adversary's Knowledge of

of the adversary in these attacks, we expect that attack 1 be the easiest attack for the adversary and attack 5 be the most difficult one. Table 1 compares adversary's knowledge and difficulty of different attacks defined in this paper.

3 Related Works

Table 2 compares our proposed method with related works in the literature. Generally, the methods for TI attack against FR systems, can be categorized based on different aspects, including the resolution of generated face images (high/low resolution), the type of attack ( whitebox / blackbox attack), and the basis of the method (optimization/learning-based).

Zhmoginov and Sandler [2016] proposed an optimization-based method and a learning-based method to generate low-resolution face images in the whitebox attack against FR systems. In their optimization-based attack, they used a gradient-descent-based approach to find an image that minimizes the distance of the face template as well as some regularization terms to generate a smooth image, including the total variation and Laplacian pyramid gradient normalization [Burt and Adelson, 1987] of the reconstructed face image. In their learning-based attack, they trained a convolutional neural network (CNN) with the same loss terms to generate face images from given facial templates.

Cole et al. [2017] proposed a learning-based attack to generate low-resolution images using a multilayer perceptron (MLP) to estimate landmark coordinates and a CNN to generate face textures, and then reconstructed face images using a differentiable warping based on estimated landmarks and face texture. They trained their networks in an end-to-end fashion, and minimized the errors for landmark estimation and texture generation as well as the distance of face template as their loss function. To extend their method from the whitebox attack to the blackbox attack, they proposed not to minimize the distance of face templates in their loss function.

Mai et al. [2018] proposed a learning-based attack to generate low-resolution images in the blackbox attack against FR systems. They proposed new convolutional blocks, called neighborly deconvolution blocks A/B (shortly, NbBlock-A and NbBlock-B), and used these blocks to reconstruct face images. They trained their proposed networks using two loss functions, including pixel loss (i.e., glyph[lscript] 2 norm of reconstruction pixel error) and perceptual loss (i.e., glyph[lscript] 2 norm of distance for intermediate features of VGG-19 [Simonyan and Zisserman, 2014] given original and reconstructed face images). They

defined two types of attacks and compared the reconstructed face images with the same (type 1) or different (type 2) image of the same subject. However, they did not evaluate the transferability of reconstructed face images.

Duong et al. [2020] and Truong et al. [2022] used a same bijection learning framework and trained a GAN with a generator with structure of PO-GAN [Karras et al., 2017] and TransGAN [Jiang et al., 2021], respectively. While their method is based on the whitebox attack, they proposed to use knowledge distillation to extend to the blackbox attack. To this end, they trained a student network that mimics the target FR model. However, they did not provide any details (nor source code) about student network training, such as the structure of the student network, etc.

Ahmad et al. [2022] used a GAN-based face reconstruction network to generate low-resolution face images in the blackbox scenario. They focus on the size of training dataset and proposed a method to learn the reconstruction network with less training data. While using GAN-based approach, sample reconstructed face images in their paper do not look realistic and suffer from many artifacts.

Dong et al. [2021] used a pre-trained StyleGAN to generate high-resolution face images in the blackbox attack against FR systems. They generated synthetic face images using pre-trained StyleGAN and extracted their templates. Then, they trained a fully connected network using mean squared error to map extracted templates to the corresponding noise in the input space Z of StyleGAN. In contrast, Vendrow and Vendrow [2021], instead of a learning-based approach, used a grid search optimization using the simulated annealing [Van Laarhoven and Aarts, 1987] approach to find the noise z ∈ Z in the input of StyleGAN, which generates an image that has the same template. As their iterative method has a large computation cost, they evaluated their method on 20 images only. Along the same lines, Dong et al. [2023] also tried to solve a similar optimization to [Vendrow and Vendrow, 2021] with a different approach. They used the genetic algorithm to find the noise z ∈ Z in the input of StyleGAN that can generate an image with the same template.

Compared to most works in the literature that generate low-resolution face images, our proposed method generates high-resolution realistic face images. While low-resolution reconstructed images can be used for evaluating the vulnerability of FR systems under some assumptions, high-resolution images can lead to different types of presentation attacks against FR systems. Previous works in the literature for reconstructing high-resolution face images [Vendrow and Vendrow, 2021, Dong et al., 2021, 2023] tried to find an appropriate noise z ∈ Z in the input of StyleGAN that can generate an image with a similar template. However, the input of StyleGAN z ∈ Z is a noise and is difficult to control and optimize. In contrast, the intermediate latent space W , which we use in our paper, is more controllable 2 . However, finding appropriate latent code w ∈ W in the intermediate space of StyleGAN that can generate an image with a similar facial template is challenging for two reasons. First, for a learning-based approach, we do not have correct values of w ∈ W for each real face image to directly use for training the mapping from face templates to the intermediate latent space W of StyleGAN. Second, as described in Section 2, the mapped latent code should be in the same the intermediate latent space W of StyleGAN, otherwise the generated image is not face-like.

We should also note that, unlike most works in the literature, we propose our method for both whitebox and blackbox scenarios and evaluate the transferability of our attack. Similar to [Cole et al., 2017, Duong et al., 2020, Truong et al., 2022], our method is based on the whitebox knowledge of FR model, however our approach for extending our method to the blackbox attack using another FR model has not been used in TI attacks. Last but not least, we define five different attacks against FR systems and evaluate the vulnerability of SOTA FR models to our attacks. To our knowledge, this is the first paper in which the transferability of reconstructed face images in TI attacks has been investigated.

4 Experiments

In this section, we present our experiments and discuss our results. First, in Section 4.1 we describe our experimental setup. Then, we present our experimental results in Section 4.2, and discuss our findings.

4.1 Experimental Setup

To evaluate the performance of our method, we consider two SOTA FR models 3 , including ArcFace [Deng et al., 2019], ElasticFace [Boutros et al., 2022], as the models from which templates are leaked (i.e., F database). For transferability evaluation, we also use three different FR models with SOTA backbones from FaceX-Zoo [Wang et al., 2021], including HRNet [Wang et al., 2020], AttentionNet [Wang et al., 2017], and Swin [Liu et al., 2021], for the target FR system (i.e., F target ). The recognition performance of these models are reported in Table 3. All these models are trained on

Table 3: Recognition performance of face recognition models used in our experiments in terms of true match rate (TMR) at the thresholds correspond to false match rates (FMRs) of 10 -2 and 10 -3 evaluated on the MOBIO and LFW datasets. The values are in percentage.

MS-Celeb1M dataset [Guo et al., 2016]. We assume that the adversary does not have access to the FR training dataset, and therefore we use another dataset for training our face reconstruction models. To this end, we use the Flickr-Faces-HQ (FFHQ) dataset [Karras et al., 2019], which consists of 70,000 high-resolution (i.e., 1024 × 1024 ) face images (without identity labels) crawled from the internet. We use 90% random portion of this dataset for training, and the remaining 10% for validation.

To evaluate different attacks against FR systems, we consider two other face image datasets with identity labels, including the MOBIO [McCool et al., 2013] and Labeled Faces in the Wild (LFW) [Huang et al., 2007] datasets. The MOBIO dataset consists of bi-modal (face and voice) data captured using mobile devices from 150 people in 12 sessions (6-11 samples in each session). The LFW dataset includes 13,233 face images of 5,749 people collected from the internet, where 1,680 people have two or more images.

For each of the attacks described in Section 2, we build one or two separate FR systems with one or two SOTA FR models based on the attack type. If the target system is the same as the system from which the template is leaked, we have only one FR system. Otherwise, if the target system is different the system from which the template is leaked, we have two FR systems with two different feature extractors. In each case, we use one of our evaluation datasets (i.e., MOBIO and LFW) to build both FR systems (so that the subject with the leaked template be enrolled in the target system too). In each evaluation, we assume that the target FR system is configured at the threshold corresponding to a false match rate (FMR) of 10 -3 , and we evaluate the adversary's success attack rate (SAR) in entering that system.

We should note that the templates extracted by the aforementioned FR models have 512 dimensions. The input noise z ∈ Z to the mapping network of StyleGAN's pre-trained network is from the standard normal distribution and has 512 dimensions. The input noise n ∈ N to our mapping network M rec ( . ) is with dimension of 8 and also from the standard normal distribution. We also use Adam [Kingma and Ba, 2015] optimizer to train our mapping network 4 .

4.2 Analysis

In this section, we consider SOTA FR models and evaluate the performance of our face reconstruction method in five different attacks described in Section 2. We also explore the effect of our WGAN training as well as effect of loss terms as our ablation study. In addition, explore important areas in the reconstructed face images that lead to success TI attack. Finally, we discuss limitations of our face reconstruction model.

Whitebox Knowledge of F database For attacks 1-2, the adversary is assumed to have whitebox knowledge of the system from which the template is leaked (i.e., F database) and use the same feature extraction model for training (i.e., F loss), thus in such cases F loss = F database. We considered ArcFace

and ElasticFace models and reconstructed face images from the templates extracted by these models in attacks against different FR systems. Table 4 reports the vulnerability of different target systems to our attacks 5 1-2 in terms of adversary's SAR at the system's FMR of 10 -3 . Similar results for the system's FMR of 10 -2 are reported in Table 7 of the appendix. According to these tables, our method achieves considerable SAR against ArcFace and ElasticFace target systems in attack 1. In attack 2, we observe that there is a degradation in SAR with respect to attack 1. However, the reconstructed face images can still be used to enter another target system. Meanwhile, the FR model with a higher recognition accuracy is generally more vulnerable to attack 2. For instance, when ArcFace is considered as F database, we observe that ElasticFace and Swin have the highest SAR as target systems, while there is the same order for their recognition performance in Table 3.

glyph[negationslash]

Blackbox Knowledge of F database For attacks 3-5, the adversary is assumed to have blackbox knowledge of the system from which the template is leaked (i.e., F database) and use another feature extraction model for training (i.e., F loss), therefore in such cases F loss = F database. Table 5 compares the performance of our method with blackbox methods 6 in the literature [Mai et al., 2018, Dong et al., 2021, Vendrow and Vendrow, 2021, Dong et al., 2023] for attacks 3-5 in terms of adversary's SAR at system's FMR of 10 -3 . Similar results for the FMR of 10 -2 are available in Table 8 of the appendix.

As these tables show, our proposed method achieves the highest SAR compared to [Mai et al., 2018, Dong et al., 2021, Vendrow and Vendrow, 2021, Dong et al., 2023] against FR systems on the MOBIO and LFW datasets. In particular, in attack 5 which is the hardest attack, where F database , F loss, and F target are different, the results show that the target FR system is still vulnerable to our attack. The results of our method for attack 5 also show transferability of our attack to different FR systems. Similar to attack 2, we can also observe that in attack 5, the FR model with a higher recognition accuracy is generally more vulnerable to our attack.

Figure 4 also shows sample face images from the LFW dataset and the reconstructed images using our proposed method from ArcFace templates in different attacks. We should highlight that as show in Figure 4, the reconstructed face images in attack 1 and attack 2 are the same, but they are used to enter different target FR system. The same holds for the reconstructed face images in attacks 3-5.

Ablation Study To evaluate the effect of WGANin training our mapping network and the effect of each term in our loss function (i.e., Eq. 1), we consider the ArcFace model in the whitebox scenario and train different face reconstruction networks with different loss functions. Then, we attack a system with the ArcFace model as a feature extractor (i.e., attack 1) and compare the SARs as reported in Table 6. According to these results, the proposed adversarial training has a significant effect on our face reconstruction method. Because we fix the synthesis network of StyleGAN, the mapped latent codes need to be of the same distribution as W . Otherwise, the generated image is not face-like and training fails to converge. The WGAN training in our method helps our mapping network to learn the distribution of StyleGAN's intermediate latent space, and thus the synthesis network generates

Original

Attacks 1-2

0.723

0.682

0.704

Attacks 3-5

0.702

0.650

0.662

face-like images. When we use the WGAN training and based on the results in Table 6, the ID loss has a high impact on the performance of the template inversion model. While the pixel loss by itself does not achieve a good performance, it improves the performance of ID loss in our reconstruction loss function in Eq. 1. This table confirms that the proposed WGAN training and our reconstruction loss function lead to a more successful attack. More experiments for ablation study on the effect of different elements in our proposed method are presented in the appendix.

Important Areas in the Reconstructed Face Images As another experiment, we explore important areas in the reconstructed face images. Finding these areas help us to investigate what features between the original template and synthetic images fool the face recognition system, and therefore we can understand what information is encoded in the facial templates. To this end, we apply the Grad-Cam [Selvaraju et al., 2017] algorithm using the face recognition model on the reconstructed face images to see which areas of the reconstructed face images are important and cause the facial templates of our reconstructed face images to be close to the original facial templates. Figure 5 shows results of applying the Grad-Cam algorithm on sample reconstructed face images using our proposed method. As the results in this figure show, important areas that cause the reconstructed face images to have similar templates to the original ones correspond to areas such as eyes, nose, lips, etc. In

0.773

0.738

0.661

0.768

0.695

particular, the area around the eyes seems to be the most important part in most of the reconstructed face images. These results also show that the general shape of the face (e.g., thin or chubby face), hairs, textures, etc., are not often necessarily important in the reconstructed face images, and thus, we can also conclude that these attributes are not well-encoded in the templates extracted by face recognition models.

Limitations Despite the significant performance of our method in terms of success attack rate in all types of attacks reported in Table 4 and Table 5, the reconstructed face images fail to enter the system in some cases. Figure 6 illustrates sample failure cases in the attack 3 against ArcFace (using ElasticFace for F loss) on the LFW dataset. From the failure cases, we can conclude that there is a bias in the face reconstruction for specific demographies, like elderly or dark skin people. Indeed, such kind of bias in the reconstructed face images is caused by inherent biases in datasets used to train FR model, the StyleGAN model, and our mapping network in our face reconstruction model 7 .

5 Conclusion

In this paper, we proposed a new method to reconstruct high-resolution realistic face images

Original

Reconstructed

0.098

0.126

0.163

from facial templates in a FR system. We used a pre-trained StyleGAN3 network and learned a mapping from facial templates to intermediate latent space of StyleGAN within a GAN-based framework. We proposed our method for whitebox and blackbox scenarios. In the whitebox scenario, the adversary can use the feature extraction model for training the face reconstruction network; however, in the blackbox scenario, we assume that the adversary has access to another feature extraction model. In addition, we consider the threat model where the adversary might impersonate in the same or another (i.e., transferable attack) FR system. Based on the adversary's knowledge of the feature extraction model and the target FR system, we defined five different attacks and evaluated the vulnerability of SOTA FR systems to our proposed method. Our experiments showed that the reconstructed face images by our proposed method not only can achieve a high SAR in whitebox and blackbox scenarios, but also are transferable and can be used to enter target FR systems with a different FR model.

Acknowledgment

This research is based upon work supported by the H2020 TReSPAsS-ETN Marie Skłodowska-Curie early training network (grant agreement 860813).

References

Javier Galbally, Sébastien Marcel, and Julian Fierrez. Biometric antispoofing methods: A survey in face recognition. IEEE Access , 2:1530-1552, 2014.

Battista Biggio, Paolo Russu, Luca Didaci, Fabio Roli, et al. Adversarial biometric recognition: A review on biometric system security from the adversarial machine-learning perspective. IEEE Signal Processing Magazine , 32(5):31-41, 2015.

Abdenour Hadid, Nicholas Evans, Sebastien Marcel, and Julian Fierrez. Biometrics systems under spoofing attack: an evaluation methodology and lessons learned. IEEE Signal Processing Magazine , 32(5):20-30, 2015.

Guangcan Mai, Kai Cao, Pong C Yuen, and Anil K Jain. On the reconstruction of face images from deep face templates. IEEE Transactions on Pattern Analysis and Machine Intelligence , 41(5): 1188-1202, 2018.

Sébastien Marcel, Julian Fierrez, and Nicholas Evans. Handbook of Biometric Anti-Spoofing: Presentation Attack Detection and Vulnerability Assessment . Springer, 2023.

Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems , 34, 2021.

Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo. Styleswin: Transformer-based gan for high-resolution image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11304-11314, 2022.

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning , pages 214-223. PMLR, 2017.

Andrey Zhmoginov and Mark Sandler. Inverting face embeddings with convolutional neural networks. arXiv preprint arXiv:1606.04189 , 2016.

Forrester Cole, David Belanger, Dilip Krishnan, Aaron Sarna, Inbar Mosseri, and William T Freeman. Synthesizing normalized faces from facial identity features. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3703-3712, 2017.

Chi Nhan Duong, Thanh-Dat Truong, Khoa Luu, Kha Gia Quach, Hung Bui, and Kaushik Roy. Vec2face: Unveil human faces from their blackbox features in face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6132-6141, 2020.

Thanh-Dat Truong, Chi Nhan Duong, Ngan Le, Marios Savvides, and Khoa Luu. Vec2face-v2: Unveil human faces from their blackbox features via attention-based network in face recognition, 2022. URL https://arxiv.org/abs/2209.04920 .

Sohaib Ahmad, Kaleel Mahmood, and Benjamin Fuller. Inverting biometric models with fewer samples: Incorporating the output of multiple models. In 2022 IEEE International Joint Conference on Biometrics (IJCB) , pages 1-11. IEEE, 2022.

Xingbo Dong, Zhe Jin, Zhenhua Guo, and Andrew Beng Jin Teoh. Towards generating high definition face images from deep templates. In 2021 International Conference of the Biometrics Special Interest Group (BIOSIG) , pages 1-11. IEEE, 2021.

Edward Vendrow and Joshua Vendrow. Realistic face reconstruction from deep embeddings. In NeurIPS 2021 Workshop Privacy in Machine Learning , 2021.

Xingbo Dong, Zhihui Miao, Lan Ma, Jiajun Shen, Zhe Jin, Zhenhua Guo, and Andrew Beng Jin Teoh. Reconstruct face from features based on genetic algorithm using gan generator as a distribution constraint. Computers & Security , 125:103026, 2023.

Peter J Burt and Edward H Adelson. The laplacian pyramid as a compact image code. In Readings in computer vision , pages 671-679. Elsevier, 1987.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 , 2017.

Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. Advances in Neural Information Processing Systems , 34: 14745-14758, 2021.

Peter JM Van Laarhoven and Emile HL Aarts. Simulated annealing. In Simulated annealing: Theory and applications , pages 7-15. Springer, 1987.

Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Transactions on Graphics (TOG) , 42(1):1-13, 2022.

Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inversion with hypernetworks for real image editing. In Proceedings of the IEEE/CVF conference on computer Vision and pattern recognition , pages 18511-18521, 2022.

Xueqi Hu, Qiusheng Huang, Zhengyi Shi, Siyuan Li, Changxin Gao, Li Sun, and Qingli Li. Style transformer for image inversion and editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11337-11346, 2022.

Jiankang Deng, Jia Guo, Xue Niannan, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Fadi Boutros, Naser Damer, Florian Kirchbuchner, and Arjan Kuijper. Elasticface: Elastic margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1578-1587, 2022.

Jun Wang, Yinglu Liu, Yibo Hu, Hailin Shi, and Tao Mei. Facex-zoo: A pytorch toolbox for face recognition. In Proceedings of the 29th ACM international conference on Multimedia , 2021.

Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence , 2020.

Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual attention network for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3156-3164, 2017.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 , 2021.

Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In European conference on computer vision , pages 87-102. Springer, 2016.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

Chris McCool, Roy Wallace, Mitchell McLaren, Laurent El Shafey, and Sébastien Marcel. Session variability modelling for face authentication. IET Biometrics , 2(3):117-129, September 2013. ISSN 2047-4938. doi: 10.1049/iet-bmt.2012.0059.

Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst, October 2007.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR) , San Diego, California., USA, May 2015.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618-626, 2017.

Tiago de Freitas Pereira and Sébastien Marcel. Fairness in biometrics: a figure of merit to assess biometric verification systems. IEEE Transactions on Biometrics, Behavior, and Identity Science , 4(1):19-29, 2021.

Cemre Efe Karakas, Alara Dirik, Eylül Yalçınkaya, and Pinar Yanardag. Fairstyle: Debiasing stylegan2 with style channel manipulations. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 570-586. Springer, 2022.

Shuhan Tan, Yujun Shen, and Bolei Zhou. Improving the fairness of deep generative models without retraining. arXiv preprint arXiv:2012.04842 , 2020.

Guha Balakrishnan, Yuanjun Xiong, Wei Xia, and Pietro Perona. Towards causal benchmarking of bias in face analysis algorithms. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 547-563. Springer, 2020.

Omkar Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In BMVC 2015Proceedings of the British Machine Vision Conference 2015 . British Machine Vision Association, 2015.

European Council. Regulation of the european parliament and of the council on the protection of individuals with regard to the processing of personal data and on the free movement of such data (general data protection regulation), April 2016.

Karthik Nandakumar and Anil K Jain. Biometric template protection: Bridging the performance gap between theory and practice. IEEE Signal Processing Magazine , 32(5):88-100, 2015.

Mulagala Sandhya and Munaga VNK Prasad. Biometric template protection: A systematic literature review of approaches and modalities. In Biometric Security and Privacy , pages 323-370. Springer, 2017.

Prabhjot Kaur, Nitin Kumar, and Maheep Singh. Biometric cryptosystems: a comprehensive survey. Multimedia Tools and Applications , pages 1-56, 2022.

Nitin Kumar et al. Cancelable biometrics: a comprehensive survey. Artificial Intelligence Review , 53 (5):3403-3446, 2020.

Hatef Otroshi Shahreza, Christian Rathgeb, Dailé Osorio-Roig, Vedrana Krivoku´ ca Hahn, Sébastien Marcel, and Christoph Busch. Hybrid protection of biometric templates by combining homomorphic encryption and cancelable biometrics. In Proceedings of the IEEE International Joint Conference on Biometrics (IJCB) , pages 1-10. IEEE, 2022.

Hatef Otroshi Shahreza, Vedrana Krivoku´ ca Hahn, and Sébastien Marcel. Mlp-hash: Protecting face templates via hashing of randomized multi-layer perceptron. In Proceedings of the 31st European Signal Processing Conference (EUSIPCO) . IEEE, 2023.

Anjos, L. El Shafey, R. Wallace, M. Günther, C. McCool, and S. Marcel. Bob: a free signal processing and machine learning toolbox for researchers. In Proceedings of the 20th ACM Conference on Multimedia Systems (ACMMM) , Oct. 2012.

Anjos, M. Günther, T. de Freitas Pereira, P. Korshunov, A. Mohammadi, and S. Marcel. Continuously reproducing toolchains in pattern recognition and machine learning experiments. In ICML 2017 Reproducibility in Machine Learning Workshop , pages 1-8, 2017. URL https: //openreview.net/forum?id=BJDDItGX-.

A More Analyses

A.1 Evaluation of Whitebox and Blackbox Attacks at FMR = 10 -2

Table 7 and Table 8 of this appendix report the evaluation of attacks with whitebox and blackbox knowledge, respectively, of the system from which the template is leaked (i.e., F loss = F database ) against SOTA FR models at FMR = 10 -2 in terms of adversary's success attack rate (SAR) using our proposed method on the MOBIO and LFW datasets. As the results in these tables show, our method outperforms previous methods in the literature.

A.2 Ablation Study

Ablation Study on the Effect of Feature Extractor in the ID loss To evaluate the effect of feature extractor in our loss function, we consider attack 3 on HRNet templates and use ArcFace and ElasticFace for F loss ( . ) in our loss function. Table 9 of this appendix reports the result of this ablation study. Comparing the results of different face recognition models used as F loss ( . ) in our loss function, we can see that the mapping which is trained using ArcFace achieves a higher SAR that the mapping that is trained with ElasticFace.

Table 9: Evaluating the effect of ID loss term in our loss function in attack 3 against HRNet in terms of SAR in the system with FMRs of 10 -2 and 10 -3 evaluated on the MOBIO and LFW datasets. The values are in percentage.

Moreover, comparing these results with the recognition performances of ArcFace and ElasticFace reported in Table 2 of the paper, we can conclude that a face recognition method with a higher recognition performance can lead to a better reconstruction when used as F loss in the blackbox attack using our proposed method.

Ablation Study on the Effect of Noise in our WGAN Training To evaluate the effect of noise used in our GAN training, we implement another ablation with the same configuration used for our ablation study in the paper (i.e., attack 1 against ArcFace), and we train two networks with and without noise in the input of the mapping network. Table 10 of this appendix reports the result of our ablation study. As this table shows, using noise in our WGAN training improves the performance of our face reconstruction method.

It is noteworthy that generally, in training GANs (even in conditional GANs) a noise (e.g., from Gaussian distribution) is used in the input of the generator network. The samples of noise in the input help the generator to learn the distribution of the output space, and therefore help the generator network to generate outputs from the same distribution of real data. The discriminator (or critic in WGAN) network tries to distinguish if the sample output is from the distribution of real

Table 10: Evaluating the effect of using noise in our method in attack 1 against ArcFace in terms of SAR in the system with FMRs of 10 -2 and 10 -3 evaluated on the MOBIO and LFW datasets. The values are in percentage.

data or not. In other words, adding random noise in the input makes the training stochastic which is suitable for learning a distribution. In our problem, it is very important that the generated latent code is from the same distribution as the intermediate latent space W of StyleGAN. In particular, if the generated latent code is not in the same distribution of W , it can easily lead to a non-face-like image at the output of StyleGAN.

Ablation Study on the Mapping Space To evaluate the effect of the mapping space in our proposed method, we consider attack 1 on against ArcFace model, and train mapping to input latent space Z and the intermediate latent space W of StyleGAN. Table 11 of this appendix reports the result of our ablation study.

As the results in this table show, mapping to the intermediate latent space W leads to a higher performance. This is because the

Table 11: Evaluating the effect of mapping space in our method in attack 1 against ArcFace in terms of SAR in the system with FMRs of 10 -2 and 10 -3 evaluated on the MOBIO and LFW datasets. The values are in percentage.

intermediate latent space has more information and is more controllable than input space Z , which is originally of Gaussian distribution for noise in StyleGAN. This ablation study highlights the importance of mapping to the intermediate latent space W of StyleGAN, which has not been proposed in the literature for template inversion.

A.3 Using a Different Face Generator Network

In our experiments, we used StyleGAN which is one of the most popular face generator models in the literature. However, our method can also be used with other face generator networks. As another experiment, we use StyleSwin [Zhang et al., 2022], which is another face generator model based on transformers. Figure 7 of this appendix shows the reconstructed face images from ArcFace templates using StyleSwin in our method instead of StyleGAN. We used a similar mapping network and learned a mapping from facial templates to the intermediate latent space of StyleSwin. As these results show, our method can also be used with other face generator networks.

A.4 Application of Our Method for Face Recognition Models with Different Inputs/Outputs

While we use three different face recognition models in our problem formulation, since these models are applied in separate stages, there is no issue if the inputs and outputs (e.g., pre-processing steps or

Original

Reconstructed

0.759

0.626

0.627

0.624

0.611

0.614

0.645

dimensions) be different in each of these face recognition models. For differences in inputs (face images), because each of these models is applied independently on the given face image, the required pre-processing can be considered within the function of the face recognition model in our problem formulation. For differences in outputs (face templates), since the facial templates extracted by each model are compared to facial templates extracted by the same model, there is no conflict in the dimensions. The only point to be noted is that the input of our mapping network should have the same dimension as the templates of .

Let us consider the complete pipeline of our problem formulation as depicted in Figure 2 of the paper. The first face recognition model (i.e., F database) uses its own pre-process and extracts facial templates from face images captured by the camera of the face recognition system (from which the template is leaked). These facial templates (extracted from F database) are then used as input to our face reconstruction model. Therefore, the input of our mapping should have the same dimension as templates of F database. In any case, the output of the face reconstruction network is a high-resolution ( 1024 × 1024 ) face image, regardless of the dimension of the input facial template. During training, the generated high-resolution face image is first pre-processed as required by F loss (i.e., normalised, resized and aligned based on coordinates required by F loss), and the extracted templates are compared with templates of the original image extracted from F loss (with the required pre-processing for F loss ). During inference (i.e., attacking the target FR system), however, the generated high-resolution face image is pre-processed as required by F target. Therefore, there is no conflict in the inputs/outputs in our pipeline.

In our experiments reported in the paper, all face recognition models except Swin take input with 112 × 112 resolution. However, the Swin model takes input with 224 × 224 resolution. The dimensions of facial templates extracted by all other face recognition models (in Table 3 of the paper) in our experiments are similar and equal to 512. To show that our method can also be used in case of different dimensions of facial templates and to showcase another face recognition model with different pre-processing, as a new experiment, we use a new model, VGGFace [Parkhi et al., 2015], with a different dimension of facial templates (2048-dimension) and different input image resolution ( 224 × 224 ) which has a different normalization as well as different

Table 12: Evaluation of success attack rate for TI attack using VGGFace templates (as F database) using ArcFace as F loss in attack against FR systems with different models (as F target). Note that pre-processing (normalization and alignment coordinates) of VGGFace is different than all target models and its input resolution is 224 × 224 . The input resolution for ArcFace (used as F loss) and ElasticFace is 112 × 112 but for Swin is 224 × 224 . The templates extracted by VGGFace has 2048 dimensions, while templates of ArcFace, ElasticFace, and VGGFace have 512 dimension.

landmark coordinates for face alignment. We use ArcFace as our F loss and evaluate the reconstructed face images in attacks against different face recognition systems (as F target) on the LFW dataset. The results in Table 12 of this appendix show that our proposed method can be applied in the case

where the inputs/outputs of face recognition models in our problem formulation ( F database , F loss, and F target) are different, and still achieves high success attack rates against face recognition systems with different inputs/outputs.

B Ethics Statement

Motivations The proposed face reconstruction method is presented with the motivation of showing vulnerability of face recognition systems to template inversion attacks. We hope this work encourages researchers of the community to investigate the next generation of safe and robust face recognition systems and to develop new algorithms to protect existing systems. In addition, we should note that the project on which the work has been conducted has passed an Institutional Ethical Review Board (IRB).

Ethics Considerations While the proposed method might pose a social threat against unprotected systems, we do not condone using our work with the intent of attacking a real face recognition system or other malicious purposes. We should, however, note that for the next generation of safe face recognition systems, any kind of potential attacks should be completely studied by the researchers; and then based upon such vulnerability studies, new protection and defense algorithms will be proposed by the research community in the future. To facilitate future studies, we publish source code of our work as described in Section C of this appendix.

Mitigation of such Attacks This paper demonstrates an important privacy and security threat to the state-of-the-art unprotected face recognition systems. Along the same lines, data protection frameworks, such as the European Union General Data Protection Regulation (EU-GDPR) [European Council, 2016], put legal obligations to protect biometric data as sensitive information. To this end and to prevent such attacks to face recognition systems, several biometric template protection algorithms are proposed in the literature [Nandakumar and Jain, 2015, Sandhya and Prasad, 2017, Kaur et al., 2022, Kumar et al., 2020, Shahreza et al., 2022, 2023].

C Reproducibility Statement

In our experiments, we use PyTorch package and the pre-trained model of StyleGAN3 8 and StyleSwin 9 to generate high-resolution face images. We train our mapping network for 16 epochs with an initial learning rate of 0 . 1 using Adam optimizer [Kingma and Ba, 2015] and divide the learning rate by 2 every three epochs. Training our mapping network using our proposed method takes around two days on a system equipped with an NVIDIA GeForce RTX TM 3090. We build face recognition pipelines using Bob [Anjos et al., 2012, 2017] toolbox 10 . The source code of our experiments is publicly available 11 to help reproduce our results.

D Licenses and Copyright Permissions

Datasets We have signed the licenses (GDPR compliance) to use from the data controller of any of the datasets used in this paper (i.e., MOBIO, LFW, and FFHQ) and followed the terms of use of these datasets in this paper. We have also cited the corresponding paper for each dataset.

Models We used pretrained models of following deep neural networks and followed the license of each one in implementing our experiments:

ArcFace, ElasticFace, and VGGFace face recognition models implemented in Bob [Anjos et al., 2012, 2017] toolbox (under BSD 3-Clause License)

HRNet, AttentionNet, and Swin face recognition models implemented in FaceX-Zoo [Wang et al., 2021] toolbox (under Apache License, Version 2.0)

StyleGAN3 (official) model published under Nvidia Source Code License 12 .

StyleSwin (official) model published under MIT License.

Peer reviews:
comments: SUMMARY:
The paper presents a method to reconstruct high resolution face images from feature vectors extracted from a face recognition (FR) system. The reconstructed face images can be used to attack FR systems to gain access under whitebox and blackbox scenarios
The paper also introduce five different scenarios that one can use to attack FR systems. The authors raised awareness to protect template feature vectors stored in FR systems' databases to avoid possible adversary attacks.

STRENGTHS:
The paper evaluates various attack scenarios with five different face recognition models. 
The reconstructed face images have high quality. Some of them have similar IDs with the original images.

WEAKNESSES:
1. The novelty in the proposed framework is limited. Most of the components are based on the existing works, e.g. StyleGAN3 and WGAN.
2. The proposed attack scenarios are straightforward as a combination of three possible feature extraction models. The authors should also rate the practical of the scenarios in addition to the level of difficulty since some scenarios cannot achieve in practice.
3. In their problem formulation, the authors consider three different feature extraction models. However, there is an implicit assumption that the input and output of those models are the same. The authors should analyze cases when the input and output are different, e.g. handling various output dimension, various pre-processing steps.
4. There are some errors in referring to tables in the paper, e.g. line 281, 287 and 300.

QUESTIONS:
See above weakness.

LIMITATIONS:
The authors addressed one limitation of the reconstruction model. However, the authors should address some other limitation mentioned in the weakness section above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-28

comments: SUMMARY:
The paper studies the template inversion attack on FR systems. The paper involves both white- and black-box attacks. Moreover, five different attacks are considered.

STRENGTHS:
This paper poses a potential threat to FR systems in that attackers can reconstruct victim's facial images with the features stored in the database. Comprehensive attack situations are considered with five different attacks.

WEAKNESSES:
- My main concern is about the limited application of the proposed method. The quality of the face images highly depends on the power of the generator, i.e. StyleGAN3. Therefore, the bias of it, such as resolution, gender, race, and head pose biases make the attack hardly generalize to other face images. For example, including the failure Fig. 5, if the features stored in the database are from low-resolution images, or non-frontal faces, or faces aligned with other alignment methods, the proposed method may also fail as these also are the biases/domain gap of the generator. It would be good if the authors analyzed more in this aspect that there is a large domain gap between the database and that in the generator. From the current setting, I have not seen any specific handle for this issue but a simple critic network.

QUESTIONS:
See the Weakness.

LIMITATIONS:
See the Weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce a new method for reconstructing high-resolution realistic face images from facial templates within a face recognition (FR) system. They employ a pre-trained StyleGAN3 network and train a mapping from facial templates to the intermediate latent space of StyleGAN using a GAN-based framework. In particular, the proposed method is designed to work in both white-box and black-box under different adversary attack scenarios.  The vulnerability of state-of-the-art (SOTA) FR systems to the proposed method is evaluated. Experimental results demonstrate that the reconstructed face images using the proposed method achieves the highest success rates in both white-box and black-box scenarios and its transferability has been validated.\n\nSTRENGTHS:\n+ Overall the paper is well organized and easy to read.\n+ The idea of training a critic network to ensure the same distribution between $w \\in W$ and ${\\hat w} \\in W$ is interesting.\n+ Deep dive into five different template inversion (TI) attacks, and evaluate the transferability of reconstructed face images in TI attacks.\n+ Solid experiment validation based on StyleGAN with better performance than the SOTA methods.\n\nWEAKNESSES:\n- The investigation on face reconstruction is insufficient. The authors didn’t mention any work related to Transformer and diffusion works, e.g, Face-Transformer (2023 Arxiv April), and VQ-DDM (CVPR’22).\n\n- The template inversion (TI) attack is a too specific adversarial attack. I would like to suggest the authors experimentally verify whether it is possible to extend the idea (e.g., critic network) to other kinds of attacks.\n\n- The current paper is heavily dependent on the StyleGAN3, which is not SOTA when considering the more advanced Transformer and diffusion models. Therefore I would like to suggest the authors to switch the current StyleGAN3 to the both Transformer and diffusion models to validate the claims made in the current paper.\n\n- Typo: miss the table number in line 287, 300.\n\nQUESTIONS:\nNA.\n\nLIMITATIONS:\nAs mentioned by the authors, there exists a bias in reconstructing faces of certain demographic groups, such as elderly individuals or people with dark skin. This bias in the reconstructed face images can be attributed to the inherent biases present in the datasets used to train the face recognition (FR) model, the StyleGAN model, and the mapping network in our face reconstruction model.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper proposes a high-resolution face reconstruction method for template inversion attacks. They make use of a GAN-based framework, StyleGAN3, by learning a mapping from facial templates to its intermediate latent space. They evaluate their method on 5 different attacks in whitebox and blackbox scenarios.

STRENGTHS:
The method is evaluated in a wide range of TI attack scenarios.

The method, its motivation and evaluation setting is described clearly and is easy to understand.

WEAKNESSES:
The novelty of the proposed method mainly lies in the introduction of the facial template mapping network which is somewhat incremental.

It would be helpful to understand the rationale behind selecting StyleGAN as the synthesis model. Additionally, exploring alternative synthesis networks could provide insights into how much of the success and limitations depend on the synthesis network. Also, it would be beneficial to explore the effect of fine-tuning the synthesis network of StyleGAN.

QUESTIONS:
Please see the weakness.

LIMITATIONS:
Yes, the limitations are addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose a new method to execute the template inversion attack against face recognition systems. By simply having access to the feature vectors (latent representations) of the original faces, the adversary can reconstruct a face like the original, therefore showing that face recognition systems can be compromised. The method used comprises a StyleGAN3 network, split such that a new Mapping network attempts to synthesize a correct embedding given random noise and the victim template vector, and a critic network discriminates between that output and StyleGAN’s own mapping network with the WGAN algorithm. \nThe authors further test their method in whitebox and blackbox scenarios, using a combination of several models for template vector leaking and transferability of reconstructing a face to attack a face recognition system. In the end, for the five attacks defined, the novel method achieves high attack success rates.\n\nSTRENGTHS:\nWhile the problem is not new, the paper takes a clever spin on existing methods and combines them to tackle five attack scenarios. Even in the most difficult attack scenario, the novel method achieves significantly higher attack success rates, but even more so, the better recognition models actually suffer the most. \n\nIt is interesting and reasonable to propose face template matching via the GAN-base framework.\n\nThe paper provides sufficient evaluation results for five attack scenarios, as the paper defined.\n\nWEAKNESSES:\nThe paper contains grammatical errors throughout, but they should be fixable. For instance, in line 108, I believe you meant to say “critique” the generated omega-hat vectors. For Section 4.2, I believe it should be titled “Analysis”. \n\nThe five attack scenarios as defined in the paper seem incremental so they do not show the broad of the method application.\n\nThe synthesized images themselves in the sense that the authors are not trying to answer the question of *what* features between the template and synthetic images fool the recognition system. I think having that aspect of the attack would both boost the novelty of the attack and provide an excellent visualization as to why (and literally where) the attack works.\n\nQUESTIONS:\n1. Is there a particular reason why a model like StyleGAN3 was chosen? (Code availability, SOTA, etc.). \n2. (Related to Weakness section) Is it reasonable to think that the template vectors would be stored “raw” and not in an encrypted form? I do not think a database concerned with security would store the raw information. \n3. It seems in many, if not all, images shown, the face subjects have smooth faces (this is somewhat related to elderly subjects reconstructing poor results). Do facial features (e.g., wrinkles, pores, moles, etc.) have any influence on the system recognition succeeding or failing? Perhaps this could be attributed to StyleGAN synthesis of images. \n4. The paper should provide an insightful comparison of the attack scenarios, i.e. what scenario A has but B does not have, and vice versa. It looks like the evaluation of the blackbox scenario is enough since it covers all of the others.\n\nLIMITATIONS:\nThe authors have addressed the limitations of the work and its biases and have stated which data they have used for experiments (and how the data influences the outcomes presented).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}