Title: Inference Attacks Against Face Recognition Model without Classification Layers
Venue: NeurIPS 2023
Accepted: no
Keywords: inference attack

face recogniton

classification

model inversion
Abstract: Face recognition (FR) has been applied to nearly every aspect of daily life, but it is always accompanied by the underlying risk of leaking private information. At present, almost all attack models against FR rely heavily on the presence of a classification layer. However, in practice, the FR model can obtain complex feature embedding of the input via the model backbone, and then compare it with the target for inference, which does not explicitly involve the outputs of the classification layer adopting logit or other losses. In this work, we advocate a novel inference attack composed of two stages for practical FR models without a classification layer. The first stage is the membership inference attack. Specifically, We analyze the distances between the intermediate features and batch normalization (BN) parameters. The results indicate that this distance is a critical metric for membership inference. We thus design a simple but effective attack model that can determine whether a face image is from the training data set or not. The second stage is the model inversion attack, where sensitive private data is reconstructed using a pre-trained generative adversarial network (GAN) guided by the attack model in the first stage. To the best of our knowledge, the proposed attack model is the very first in the literature developed for FR models without a classification layer. We illustrate the application of the proposed attack model in the establishment of privacy-preserving FR techniques.

Abstract

Face recognition (FR) has been applied to nearly every aspect of daily life, but it is always accompanied by the underlying risk of leaking private information. At present, almost all attack models against FR rely heavily on the presence of a classification layer. However, in practice, the FR model can obtain complex feature embedding of the input via the model backbone, and then compare it with the target for inference, which does not explicitly involve the outputs of the classification layer adopting logit or other losses. In this work, we advocate a novel inference attack composed of two stages for practical FR models without a classification layer. The first stage is the membership inference attack. Specifically, Weanalyze the distances between the intermediate features and batch normalization (BN) parameters. The results indicate that this distance is a critical metric for membership inference. We thus design a simple but effective attack model that can determine whether a face image is from the training data set or not. The second stage is the model inversion attack, where sensitive private data is reconstructed using a pre-trained generative adversarial network (GAN) guided by the attack model in the first stage. To the best of our knowledge, the proposed attack model is the very first in the literature developed for FR models without a classification layer. We illustrate the application of the proposed attack model in the establishment of privacy-preserving FR techniques.

1 Introduction

Face recognition (FR) [5, 31, 18, 21, 13] technology has improved steadily over the past few years. It has been widely applied in a large number of personal or commercial scenarios for enhancing user experience. Recent studies [3, 27, 36] indicated that existing FR models can remember the information from the training data, making them vulnerable to some privacy attacks such as model extraction attacks [29], model inversion attacks [6], attribute inference attacks [7] (also known as property inference attacks) and membership inference attacks [26]. As a result, malicious attackers may be able to obtain users' private information through the use of certain attacks, which could cause significant damage. In order to facilitate the development of privacy-preserving FR methods, it is essential to evaluate the leakage of the private information quantitatively. This motivates this study which leads to the establishment of novel inference attacks to quantify such privacy leakage.

Inference attacks on machine learning algorithms can be roughly categorized into membership 31 inference attacks and model inversion attacks [24]. We shall focus on both types of attacks against 32 FR models. The goal of a membership inference attack is to infer whether a record is included in 33 the training data set or not, which is usually formulated as a binary classification problem. Model 34 inversion attacks attempt to recover the input of the target model. For example, some inversion attacks 35

can recover the identity information of the training data from the FR model. 36

Backbone

Head

Classifier

Logits

Label

Similarity

metric

Train

Embedding

Inference

√

×

Traditional attack target

Our attack target

At present, both types of attacks against FR usually rely on the presence of a classification layer. In 37

other words, the performance of the attack and the corresponding defense algorithms [33] is heavily 38

dependent on utilizing the output from logits in the classification layer or the labels themselves 39

[4]. Nevertheless, FR has completely different training and inference paradigms, as shown in Fig.1. 40

Existing face recognition techniques do include a classifier after the feature extraction network 41

(backbone) in training, but the classifier is not used during inference. Instead, they extract feature 42

embeddings from the image pair using the trained backbone, then calculate their similarity based 43

on certain metrics to determine whether the two images are from the same person to achieve FR. 44

Therefore, in a realistic setup, the attacker can only acquire the feature embedding of the query 45

image and has no access to the discarded classification layer. The aforementioned difference in the 46

47

paradigms of training and inference for FR brings significant challenges to the current attack methods,

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

because there is evidence [8] indicating that compared with logits and losses, the feature embedding, as a more general representation, contains

less information about the training data. Moreover, the

training stage of an FR system is essentially a closed-set classification problem, while the inference phase becomes a more complicated open-set problem. In summary, performing inference attacks

against FR without exploiting the classification layer is non-trivial but practically important.

In this paper, we shall propose a novel two-stage attack algorithm, and its diagram is shown in Fig.2. The first stage is the membership inference attack. Compared to previous attack methods that focus on the classification layer of the target model, we explore the inherent relationship between the backbone output of the target model and the training data set. Specifically, we first analyze the distribution of the distances between the intermediate features of the member/non-member samples and statistics stored in the Batch Normalization layer in the trained backbone, and found it critical for membership inference. We then design an attack model to determine whether a sample belongs to the training data set or not, through utilizing this distance distribution. We conduct experiments under different levels of prior knowledge including partial access to training data set and no access to training data set. In both cases, the proposed membership attack is efficient and can provide state-of-the-art performance.

The second stage is the model inversion attack. Previous model inversion attacks against FR such as 63

those from [28, 1, 17] are heavily dependent on the classifier of the target model. They often guide 64

the optimization of the attack network using the logit of the classifier's output or some well-designed 65

loss functions based on the logit. For the FR models in consideration, none of these approaches are 66

67

applicable. We thus put forward a novel inversion attack approach, where our attack model in the first

68

stage guides the synthesis network, StyleGAN, to optimize the latent code such that the synthesized

face becomes close to a member sample in the training data set as much as possible. Specifically, we 69

adopt the PPA [28] paradigm, preferring to sample a random batch of samples, filtering out these 70

samples that are more likely to be considered members of the training data set by the first-stage attack 71

Conv2d

Classifier

Mean

Var

Mean

Var

Backbone

Distance

Mean

Var

Distance

Distance

Attack model

Member/Non-member

z ∈ 𝑁(0,1)

Generator

Mapping

network

𝑤 ∈ 𝒲

Synthesis

network

Stage 1

Stage 2

model, and then further optimizing the remaining samples to make them better fit the membership 72 distribution. Experiments demonstrate that the proposed attack algorithm against FR can recover 73 some sensitive private information in the training data set. 74

Our contributions can be summarized as follows: 75

76

77

78

79

We extend existing inference attacks against FR models by relaxing the assumption of the existence of a classification layer in the inference process. The proposed attack method can thus make the defense techniques in literature targeting the FR classification layer subject to the risk of private information leakage again.

80

81

82

83

84

85

86

87

88

We analyze the distance between the intermediate features of member/non-member samples of the training data set and the parameters of the BN layers. We design a simple and effective attack model accordingly, which is applicable to the trained FR models without classification layers. Experiments demonstrate that our method outperforms the state-of-the-art methods with the same prior knowledge.

We propose a model inversion attack algorithm that does not require the classification layer. It makes the synthesized samples better fit the membership distribution, and its effectiveness is verified by our experiments.

2 Related Work

2.1 Membership Inference Attack 89

Member inference attacks aim to speculate whether a record has been involved in the training of the 90

target model. In recent years, there have been significant efforts devoted to membership inference 91

attacks and defenses. Specifically, [26] first proposed the membership inference attack, where the 92

attacker achieves membership inference through multiple attack models. [25] relaxed the assumptions 93

in membership inference attacks; other works [4, 20] proposed membership inference attacks for 94

scenarios where only labels are available. In addition, [2] analyzed the membership inference 95

inference attacks, and the available algorithms can be broadly categorized into two types: overfitting 97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

reduction [30, 26, 25] and perturbation model prediction [23, 15, 33]. On the contrary, the attack method presented in this paper does not require the use of labels and model predictions. It not only achieves good attack performance, and more importantly, it renders the existing defense methods that are based on the presence of the classification layer vulnerable again.

Recently, attack techniques [19, 8] with no demand of the classification layers attracted great attention, which were developed mainly for pedestrian re-identification (Re-ID). FR is a more challenging task compared with pedestrian re-identification in the sense that the face data set contains only face-related information but has more identities and samples than the Re-ID data set. Moreover, [19] requires acquiring multiple samples of the same class, while [8] needs to obtain partial training data. These conditions are difficult to fulfill in realistic scenarios. The developed method in this paper outperforms the existing membership inference attack algorithms in terms of improved success rate under the same amount of available information.

2.2 Model Inversion Attack

The model inversion attack is a particular type of privacy attacks on machine learning models, which aims at reconstructing the data used in training. Many recent model inversion techniques [17, 28, 1, 10] have been implemented with the help of the generative power of GAN. [17] formulated model inversion as the problem of finding the maximum a posteriori (MAP) estimate given a target label, and limited the search space to the generative space of GAN. [28] proposed a more robust crossdomain attack with a pre-trained GAN, which can generate realistic training samples independent of the pre-trained data. MIRROR [1], on the other hand, explored a new space P over StyleGAN, and regularize latent vectors in P to obtain good performance. [10] cast the latent space search as a Markov decision process (MDP) and solved it using reinforcement learning.

Meanwhile, there are some works [33, 32, 22] focusing on defenses against model inversion attacks. [33] proposed a unified purification framework to resist both model inversion and membership inference attacks by reducing the dispersion of confidence score vectors. DuetFace [22] utilized a framework that converts RGB images into their frequency-domain representations and makes it difficult for attackers to recover the training data by splitting them. PPFR-FD [32] investigated the impact of visualization components on recognition networks based on the privacy-accuracy trade-off analysis and finally proposed an image masking method, which can effectively remove the visualization part of the images without affecting the FR accuracy evidently.

3 Method

In this section, we present the proposed attack method in detail, whose diagram is shown in Fig.2. We first briefly review the existing training and inference paradigms of FR in Section 3.2. Next, we introduce the difference in the distance distribution of member/non-member compared to BN layer parameters in Section 3.3, which will guide the member inference attack. In Section 3.3, we will formally introduce the first stage of the inference attack, i.e. membership inference attack algorithm. Furthermore, we will introduce our second stage - the model inversion attack method without classification layers in Section 3.4.

3.1 Background

Commonly adopted training and inference paradigms of FR are shown in Fig.1. Given a face data set D , an image sampled from it is denoted as x ∈ R c × h × w , where c , h and w represent the number of channels, height and width of the sample. Its identity label can be represented as a one-hot vector y ∈ R n × 1 , where n is the number of classes. In the training phase, a backbone network B with model parameters collected in θ b takes an input sample and extracts the corresponding highdimensional feature embedding v = B ( x ) ∈ R f × 1 , where f is the dimensionalities of the embedding. The backbone network B employs a classification layer C , whose parameters are collected in θ c , to predict the class ˆ y = C ( B ( x )) for the obtained embedding v . The cross-entropy, denoted as L CE (ˆ y, y ) , is usually used as the loss function in training, and the training process can be formulated as the following minimization problem with respect to θ b and θ c :

During inference, the goal is to decide whether the two face images come from the same person or 147 not. The trained backbone B is utilized to extract the embeddings of the two face images without 148 utilizing the classification layer anymore. The similarity of the obtained two embeddings is evaluated 149 based on certain metrics. If the computed similarity is higher than a well-designed threshold, they are 150 considered to be from the same person. 151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

3.2 Distance Distribution Comparison

Current studies [19, 8] on the distributions of feature embeddings of samples from the training data set (i.e., member samples) and other samples (i.e., non-member samples) were carried out. In particular, [19] requires a large number of inter-class samples to calculate the class centers, while ASSD [8] demands access to some training samples as reference samples to calculate their Euclidean distances. These assumptions may be too strict and are not always available to attackers. Thus, we shall look for information on the training samples stored in the trained backbone network B itself.

Specifically, we consider replacing the reference samples required in [19, 8] with the statistics stored in the Batch Normalization (BN) [14] layer of the trained backbone network. A BN layer normalizes the feature maps during the training process to mitigate covariate shifts [14], and it implicitly captures the channel-wise means and variances [35]. To gain more insights, we compare the distances between the intermediate features of the member/non-member samples and the corresponding "running mean" and "running variance (var)" in several BN layers, and the results are shown in Appendix.

There is a clear boundary separating the distance distribution of member and non-member samples in some BN layers. This indicates that the BN layer parameters may be utilized for membership inference. This observation is fundamentally different from the findings in [19, 8], as they completely ignored the information stored in the trained model itself on the training data set. The performance of the techniques from [19, 8] heavily depends on the number of samples used. If the available samples are insufficient, they will not perform as well as expected. Our observation, on the other hand, relax these constraints on the following membership inference attack.

3.3 Inference Membership Attack

The proposed membership inference attack algorithm is designed based on the observation presented in the previous subsection. It is also the first step of our attack model, as shown as Stage 1 in Fig. 2. Specifically, we consider the "running mean" parameters in certain BN layers, denoted by u i ∈ R b i × 1 , where b i represents the number of channels in the i th BN layer, and thus we obtain a set of vectors u = { u 1 , u 2 , · · · , u n } , where n represents the number of BN layers selected. It is worthwhile to point out that since the number of channels in the BN layers can be different, the vectors u i in u do not necessarily have the same dimensionality.

Given the input image x , we extract the intermediate features v i ∈ R c i × h i × w i before a particular BN layer, and then we obtain v i ∈ R c i × 1 by normalizing along both the height and width dimensions following the BN operation. We then compute the Euclidean distance between the extracted and normalized feature v i , and the reference u i using

We transmit the distance vector d = { d 1 , d 2 , · · · , d n } into the classification network A . Due to the 184 good distinguishability of the features we extract, we do not need to design a complex network for 185 membership inference. Here, we only need to use a fully connected layer and a sigmoid function to 186 compose our attack model, which predicts the probability that the sample is a member. If it is from 187 the training data set, then the attack model should output 1, otherwise we expect it to output 0. 188

More empirical experiments show that the use of both the original face image and its horizontally 189 flipped version is capable of enhancing the performance. Therefore, in the implemented membership 190 inference scheme, we first flip horizontally an image x to obtain x ′ , and then extract v ′ i by following 191 the same procedure used to find v i . The distance d i is now computed as 192

The classification network A is trained through solving 193

where θ A denotes the parameters of the classifier A in our attack model, and s is the binary label (0 or 1). The training procedure of the membership inference attack is shown in Algorithm 1 .

Algorithm 1: The training procedure of the membership inference attack (Stage 1)

Input: Member of training data x m and non-member data x n , number of training iterations M . Output: The optimal parameters of the attack model θ best .

1 Extract the parameters of Batch Normalization layers in the target FR model, denoted as u .

2 Set θ best = θ 0 and Acc best = 0 , where θ 0 denotes the initial parameters of the attack model A and Acc best records the best performance on the test set.

3 for i = 0 , 1 , 2 , · · · , M -1 do

4 Horizontally flip the member data x m and non-member data x n , and obtain x ′ m and x ′ n , respectively.

5 Feed the target model with x m , x ′ m , x n and x ′ n , and extract the corresponding intermediate features v m , v ′ m , v n and v ′ n .

6 Compute the distance using Eq.3 and obtain the distance vectors d m and d n .

7 Update the classifier parameters using the following equation:

8

9

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

where α denotes the optimization step.

Evaluate the model on the test set and obtain the accuracy

θ

i

+1

, Acc best

=

Acc

i

+1

if

3.4 Model Inversion Attack

Existing model inversion attacks require that target labels are given in order to improve the prediction score of the target identity. In this work, we consider a model inversion attack on the FR models without the classification layer. As such, our goal is not to reconstruct the original image when the target identity is known. Instead, we focus on recovering the identity of the training data set using the trained backbone model as much as possible.

The model inversion attack is Stage 2 of the proposed attack model in Fig. 2. The procedure of model inversion attack is shown in Algorithm 2 . In particular, we utilize the powerful generative model StyleGAN [16] to help restore the identity information of the training data. We first collect samples from a normal distribution z ∈ N (0 , 1) , and these samples are fed into StyleGAN's mapping network to obtain a more disentangled subspace W . All our subsequent optimizations will be performed in W space.

[28, 1] have pointed out that the selection of the initial latent vectors to optimize has a strong impact on the effectiveness of model inversion attacks and its importance should not be underestimated. As the classification layer is absent in our case, it is not feasible to obtain the confidence of the target label, and we have to utilize the attack model in Stage 1 to select samples that are closer to the distribution of the training set. For this purpose, we feed all the initial latent vectors into the StyleGAN and produce a set of initial face images. After proper resizing, these images with their horizontally flipped versions are processed following Stage 1 and we can get the prediction of the developed attack model A . Those images with high membership classification scores are retained as the 'final' set of initial vectors.

Next, we optimize the latent vectors as follows. In particular, we perform some operations on the images synthesized using the latent vectors, and propagate the transformed images through the developed membership inference attack model in Stage 1. The membership classification network predicts the probability that the generated images belong to the training set. The latent vectors will then be iteratively updated to further increase the membership prediction probability. We desire to enhance the robustness through data augmentations and finally find a good set of latent vectors.

Acc

i

+1

> Acc best

.

Acc

i

+1

, and update

θ

best

=

Algorithm 2: The procedure of the model inversion attack (Stage 2)

Input: Initial N points sampled from a normal distribution z ∈ N (0 , 1) , number of iteration M . Output: The generated candidate set q (the size of q is Int (0 . 1 N ) , Int (0 . 1 N ) is the integer part of 0 . 1 N ), and the realistic training image set r .

1 Generate N images from initial sample points z by the generator.

2 Extract the parameters of Batch Normalization layers in the target FR model, denoted as u .

3 Feed the target model with the generated images and their horizontally flipped versions, and compute the distance using Eq.3, and then predict the probability of membership. (We simplify this step as MI, i.e., the membership inference.)

4 Select Topn ( n = Int (0 . 1 N ) ) sampling points in descending probability order as the candidates

to optimize their intermediate representations w j 0 ( j = 1 , 2 , · · · , Int (0 . 1 N )) . The relationship between z and w is shown in Fig.2.

5 Set q = {} and append optimized candidates to it as follows:

7

8

9

10

11

12

13

Perform some data augmentation operations on the image x j i and get the counterparts x j i 1 , x j i 2 , · · · , x j im , where m denotes the number of the data augmentation types. Make membership inference and compute the average probability:

Optimize the sample point w j i by minimizing the loss function:

where α w denotes the optimization step size. Generate the updated images: x j i +1 = G ( w j i +1 )

Append the optimized candidate to the set q = q ∪ { x j M }

15 Select Topn ( n = Int (0 . 01 N ) ) image pairs in descending similaritiy order and obtain the realistic training image set r .

223

224

225

226

227

228

229

230

231

232

233

234

235

236

4 Experiment

In this section, we will describe our experimental setup and results in detail. First, for the membership inference attack of Stage 1, we perform two different settings based on different prior information: the partial training data D p and the auxiliary data D s . For the model inversion attack of Stage 2, we only use the auxiliary data D s .

4.1 Dataset and Target Model

We use two datasets for inference attack, CASIA-WebFace [34] and MS1M-ArcFace [5]. The CASIA-WebFace dataset is collected in a semi-automatical way from the Internet, and is usually used for face verification and face identification tasks. The dataset contains 494,414 face images of 10,575 real identities. MS1M-ArcFace is obtained by cleansing on MS1M [9], and it contains 5.8M images from 85k different celebs.

We use IR-SE-50 as the backbone, which combines an improved version of the vanilla ResNet-50 [11] with SENet [12], and use ArcFace [5] as the margin-based measurements (head). The target model is trained for 50 epochs with an initial learning rate of 0.1 and step scheduling at 10, 20, 30

Generate the image x j 0 from the selected sample point w j 0 by the generator.

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

and 40 epochs, using the SGD optimizer with a momentum of 0.9, weight decay of 0.0001. Previous researches [25, 2] have proved 'if a model is overfitted, then it is vulnerable to membership inference attack.' In this experiment, to avoid overfitting, we choose the FR model that has the best test accuracy in all epochs during training as the target model.

4.2 Membership Inference Attack

As above mentioned, we give two different cases. Case 1: access to part of the training dataset. Case 2: access to the auxiliary dataset. In the case 1, we choose CASIA-WebFace as the training data set. We use different proportions of training data for training the attack model, which uses ASSD [8] as the baseline. In the case 2, MS1M-ArcFace is used as the training data set. We train a shadow model to mimic the behavior of the target model [25]. Specifically, we first split the dataset D s by half into D shadow and D target . Then we split D shadow by half into D member shadow and D non -member shadow . D target is used for the attack evaluation, it is also split into D member target and D non -member target . All the D member serve as the members of the (shadow or target) model's training data, while the other serves as the non-member data. For evaluation, we sample 60,000 images from members and non-members separately in both cases and use the attack success rate (ASR) as the evaluation metric.

Furthermore, we perform experiments with different settings for ablation study. First, we conduct an experiment case where the images are not flipped and the "mean distance" is fed to the attack model, denoted as A mean . And then we consider replacing the "mean distance" with "mean and variance distances", which both are input to the attack model, denoted as A mean & var . Finally, we use "mean distances" of the normal image and the horizontally flipped one as fusion features to input the attack model, denoted as A mean & flip . Specially in case 2, we use different combinations of the backbones and heads as the target models to validate the generalization of our methods. All results of the membership inference attack are shown in Tab.1 and Tab.2 As Tab.1 shows, in case 1, the performance of our attack significantly outperforms the baseline. We believe this is due to the fact that our selected BN-based features characterize the membership of the training set better than the reference sample selected in [8], despite our network design is more lightweight than the latter. And we are also able to achieve relatively good results in the D s experiments in case 2, which validates the effectiveness of our method.

4.3 Model Inversion Attack

To our knowledge, this is the first time that the model inversion attack is launched against an FR model without the classification layer. Previous metrics always judge the accuracy of the generated images given a target label, which are obviously not applicable in this scenario. Therefore, we use a new metric compatible with our proposed scenario. To be specific, we will use the target model

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

Origin

Result

Origin

Result

to extract the embedding of the final generated results, and obtain the most likely class index by means of the classification layer trained for FR. Then we search the most similar image pairs as mentioned in Algorithm 2 . Some attack results are provided in Fig.3. The first and third rows show the original training images, while the second and fourth rows show the synthesis images generated by our proposed algorithm without using the classification layer. It can be seen that the reconstructed image is very similar to the original image (more results can be found in the appendix). This also confirms the effectiveness of our algorithm to some extent.

5 Limitation

Although we propose an inference attack algorithm against face recognition models without classification layers, there are actually some inherent assumptions that need to be considered. First, our membership inference attack is implemented based on the backbone's internal BN layer parameters, which means that our algorithm currently can only be applied to white-box attacks in such a scenario. Therefore, it is a future challenge to explore further attacks in a completely black-box scenario.

In addition, since we are not able to operate a delicate control on the attack, we cannot guarantee the expected results given the target class. And the identity characteristics of our final synthesis images still need to be improved, which is another major challenge in this scenario due to the fact that we cannot optimize the latent vector for a specific target label.

6 Conclusion

In this paper, we propose a new scenario where the inference attack against face recognition models can be implemented without the classification layers. This is a more realistic and more challenging attack, where the adversary cannot obtain information about the classification layer on the stage of training, and all the defenses based on the classification layer will be ineffective in this scenario. Considering the internal parameters of the model, we theoretically analyze the distance distributions between the member/non-member and the BN layer parameters, and accordingly design a simple and efficient attack model that is compatible with this novel scenario. Experiments demonstrate that our method outperforms state-of-the-art similar works under the same prior. Further, we propose a model inversion attack algorithm in this scenario. We utilize the classifier in our attack model to free model inversion attacks from dependence on the classification layer, and make the generated samples closer to the membership distribution. The final experimental results prove that our proposed method is able to recover the identities of some training members. We hope that this scenario and the algorithm we proposed will encourage more researchers to focus on face recognition attacks in real scenarios, and to attach further importance to privacy security protection.

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

References

Shengwei An, Guanhong Tao, Qiuling Xu, Yingqi Liu, Guangyu Shen, Yuan Yao, Jingwei Xu, and Xiangyu Zhang. Mirror: Model inversion for deep learning network with high fidelity. In Proceedings of the 29th Network and Distributed System Security Symposium , 2022.

Teodora Baluta, Shiqi Shen, S Hitarth, Shruti Tople, and Prateek Saxena. Membership inference attacks and generalization: A causal perspective. arXiv preprint arXiv:2209.08615 , 2022.

Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium , volume 267, 2019.

Christopher A Choquette-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. In International conference on machine learning , pages 1964-1974. PMLR, 2021.

Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4690-4699, 2019.

Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security , pages 1322-1333, 2015.

Karan Ganju, Qi Wang, Wei Yang, Carl A Gunter, and Nikita Borisov. Property inference attacks on fully connected neural networks using permutation invariant representations. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security , pages 619-633, 2018.

Junyao Gao, Xinyang Jiang, Huishuai Zhang, Yifan Yang, Shuguang Dou, Dongsheng Li, Duoqian Miao, Cheng Deng, and Cairong Zhao. Similarity distribution based membership inference attack on person re-identification. arXiv preprint arXiv:2211.15918 , 2022.

Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14 , pages 87-102. Springer, 2016.

Gyojin Han, Jaehyun Choi, Haeil Lee, and Junmo Kim. Reinforcement learning-based black-box model inversion attacks. arXiv preprint arXiv:2304.04625 , 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7132-7141, 2018.

Yuge Huang, Yuhan Wang, Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. Curricularface: adaptive curriculum learning loss for deep face recognition. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5901-5910, 2020.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong. Memguard: Defending against black-box membership inference attacks via adversarial examples. In Proceedings of the 2019 ACM SIGSAC conference on computer and communications security , pages 259-274, 2019.

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR , 2020.

Mahdi Khosravy, Kazuaki Nakamura, Yuki Hirose, Naoko Nitta, and Noboru Babaguchi. Model inversion attack by integration of deep generative models: Privacy-sensitive face generation from a face recognition system. IEEE Transactions on Information Forensics and Security , 17:357-372, 2022.

Minchul Kim, Anil K Jain, and Xiaoming Liu. Adaface: Quality adaptive margin for face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1875018759, 2022.

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

Guoyao Li, Shahbaz Rezaei, and Xin Liu. User-level membership inference attack against metric embedding learning. arXiv preprint arXiv:2203.02077 , 2022.

Zheng Li and Yang Zhang. Membership leakage in label-only exposures. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security , pages 880-895, 2021.

Qiang Meng, Shichao Zhao, Zhida Huang, and Feng Zhou. Magface: A universal representation for face recognition and quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14225-14234, 2021.

Yuxi Mi, Yuge Huang, Jiazhen Ji, Hongquan Liu, Xingkun Xu, Shouhong Ding, and Shuigeng Zhou. Duetface: Collaborative privacy-preserving face recognition via channel splitting in the frequency domain. In Proceedings of the 30th ACM International Conference on Multimedia , pages 6755-6764, 2022.

Milad Nasr, Reza Shokri, and Amir Houmansadr. Machine learning with membership privacy using adversarial regularization. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security , pages 634-646, 2018.

Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP) , pages 739-753. IEEE, 2019.

Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv:1806.01246 , 2018.

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) , pages 3-18. IEEE, 2017.

Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov. Machine learning models that remember too much. In Proceedings of the 2017 ACM SIGSAC Conference on computer and communications security , pages 587-601, 2017.

Lukas Struppek, Dominik Hintersdorf, Antonio De Almeida Correia, Antonia Adler, and Kristian Kersting. Plug & play attacks: Towards robust and flexible model inversion attacks. In Proceedings of the 39th International Conference on Machine Learning (ICML) , Proceedings of Machine Learning Research, pages 20522-20545. PMLR, 2022.

Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In USENIX security symposium , volume 16, pages 601-618, 2016.

Stacey Truex, Ling Liu, Mehmet Emre Gursoy, Lei Yu, and Wenqi Wei. Towards demystifying membership inference attacks. arXiv preprint arXiv:1807.09173 , 2018.

Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5265-5274, 2018.

Yinggui Wang, Jian Liu, Man Luo, Le Yang, and Li Wang. Privacy-preserving face recognition in the frequency domain. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 2558-2566, 2022.

Ziqi Yang, Bin Shao, Bohan Xuan, Ee-Chien Chang, and Fan Zhang. Defending model inversion and membership inference attacks via prediction purification. arXiv preprint arXiv:2005.03915 , 2020.

Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li. Learning face representation from scratch. arXiv preprint arXiv:1411.7923 , 2014.

Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8715-8724, 2020.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep 398 learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021. 399

Peer reviews:
comments: SUMMARY:
This paper introduces a novel inference attack algorithm for face recognition models that do not have classification layers. The proposed attack consists of two stages: membership inference and model inversion. The membership inference attack analyzes the distances between intermediate features and batch normalization parameters to determine if a face image belongs to the training dataset. The model inversion attack reconstructs sensitive private data using a pre-trained generative adversarial network (GAN) guided by the attack model.

STRENGTHS:
1) The paper introduces a novel two-stage attack algorithm for face recognition models without classification layers. 
2) The proposed method outperforms state-of-the-art similar works and can recover the identities of some training members. 
3) This research has implications for the development of more robust and privacy-preserving face recognition models.

WEAKNESSES:
1) This paper would be beneficial to compare the proposed method with more state-of-the-art techniques to demonstrate its superiority.
2) This paper does not provide any code implementation.
3) The model performance used in the experiment is relatively low, and we hope to conduct experiments with models at a higher level of accuracy. (For example, ResNet200/VIT-Large on WebFace260M).

QUESTIONS:
Please refer to the weaknesses section.

LIMITATIONS:
Please refer to the weaknesses section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a membership inference attack against face recognition (FR) models in the white-box scenario where membership information is known for some records and white-box model access is available, but without access to a classification layer. The attack utilizes information stored in batch-norm statistics and using a meta-classifier, the authors demonstrate the effectiveness of the proposed attack. They also extend the attack to improve model inversion attacks by utilizing their membership-classifier to \"reject\" generated samples that fall below a certain threshold.\n\nSTRENGTHS:\n- The utilization of an attack meant for one kind of privacy leakage (MI) in another scenario (model inversion) is interesting.\n\n- The paper is well written, and proposed techniques/motivation are explained properly.\n\nWEAKNESSES:\n- L16: \"..very first....without a classification layer.\" This is not the first work to explore FR models that do not use a classification layers. Please refer to [1, 2]. Similar claims appear on L39 about necessarily requiring logit access for good model performance. The authors in [1] report near-perfect detection for 3 different kinds of learning algorithms/models, none of which require classification logits.\n\n- L25: \"attribute attacks (also known as property inference attacks)\" - these two are not the same at all. Similarly, L31-32 claim that all inference attacks on ML can be categorized into membership and model inversion attacks. Please refer to [3] for a detailed explanation and to better understand these differences.\n\n- L188: \"If it is from the training data set, then the attack model should output 1, otherwise we expect it to output 0\" -how is this membership information obtained? As also outlined in Algorithm 1, the attack very clearly assumes access to not only batch-normalization parameters (which can only realistically come from full-model white-box access), but also knowledge of $m$ members and $n$ non-members. While the latter is reasonable, assuming knowledge of members is a very strong assumption (on top of an already-strong access model).\n\n- L292 says \"...theoretically analyze..\" but nowhere in the paper did I see any theoretical analysis?\n\n# Minor comments\n- Figure 2: Why is Stage 2 on the left? It seems counter-intuitive.\n\n## References\n\n[1] Chen, Min, et al. \"FACE-AUDITOR: Data Auditing in Facial Recognition Systems.\" USENIX, 2023\n\n[2] Li, G., S. Rezaei, and X. Liu. \"User-Level Membership Inference Attack against Metric Embedding Learning.\" ICLR 2022 Workshop on PAIR2Struct 2022.\n\n[3] Salem, Ahmed, et al. \"SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learning.\" 2023 IEEE S&P, 2023.\n\nQUESTIONS:\n- L100: There is no explicit mention of Differential Privacy, which is surprising. Are the authors implying that their attack can somehow break the guarantees provided by algorithms like DP-SGD?\n\n- L244: Even 1% of 494K (CASIA) or 5.8M (MS1M), which would be ~5K or ~50K images respectively, is a **lot** of images. It would be more useful to have numbers for far lower percentages- perhaps 0.1% and 0.01%. Percentage is anyway not a useful indicator in a setting where there are millions of images. 0.1% may sound very low but is still more than 5K images for MS1M.\n\n- Eq2. uses $\\bar{v}_i - u_i$, but isn't $v_i$ itself $\\frac{x - u_i}{\\sigma_i}$? From my understanding of batch-norm (and what is also the case in most implementation),  batch-norm layers default to also using a running variance estimate in addition to the mean estimates.\n\nLIMITATIONS:\n- Section 3.1 - most FR models are not trained in the way that the authors describe (and assume in their experiments) here. The norm is to focus on training good embedding models so that models can be scaled easily to enroll future participants. As expected, training the model repeatedly whenever a new user (\"class\") is added is not optimal. \n\n- Figure 3: Model inversion is supposed to recover actual records from the training data, but looking at the images in the figure that doesn't seem to be the case. Many faces (like second from left, second from right in first 2 rows) are not the same people, but rather people that \"look like\" each other.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper presents a novel method for inference attacks against face recognition method. In particular, it advocates two-stage inference attack, where the first stage identify the membership and the second stage involves model inversion attack that recover the input from embedding. Experimental evaluation shows that the proposed method can largely identify the correct membership and the model inversion sees good as well.

STRENGTHS:
1. It proposes a novel two-stage method for inference attack of face recognition systems.

2. Experimental evaluation shows the proposed method has promising results.

WEAKNESSES:
1. The evaluation is not thorough. It only conduct some ablation study with comparing to the existing attack methods.

2. The paper does not cover some import topic for inference attack of face recognition, such as the black box attacks for face recognition.

QUESTIONS:
1. Recovering input images from embedding has been widely studied and employed. What are the difference between the proposed method with the exiting decoding methods?

LIMITATIONS:
unavailable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

comments: SUMMARY:
In this submission, the authors advocate an inference attack composed of two stages for practical FR models. The first stage analyzes the distances between the intermediate features and batch normalization parameters. The second stage reconstructs data using a pre-trained generative adversarial network (GAN) guided by the attack model in the first stage.

STRENGTHS:
1.The writing and presentation are good and easy to follow.

2.The experimental results also demonstrate some effectiveness of the proposed method.

WEAKNESSES:
1.The overall technical contributions are somewhat limited, firstly, the way of using BN to perform membership inference attack has been explored for a long time. And secondly, the inversed training data are from a pretrain GAN, which is heavily depending on the strength of the pretrain GAN. And optimizing the synthesized face data is too weak only by the single supervision from the first stage. 

2.And from the Figure 3, I don’t think the recovered face data is visually close to the original data for some of the pairs. Therefore, I doubt that whether the authors really achieve the initial goal, recovering the similar enough or effective enough face training data, by their proposed method or not.

3.The experimental comparisons are too simple and rough, lacking some important state-of-art related competitors.

QUESTIONS:
Please see the details in Weaknesses.

LIMITATIONS:
This submission has adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07