Non-Adaptive Adversarial Face Generation
Authors: Sunpill Kim
Seunghun Paik
Chanwoo Hwang
Minsu Kim
Jae Hong Seo
Venue: NeurIPS 2025 poster ()
Keywords: adversarial attack
face recognition
non-adaptive attack
Primary area: applications

Abstract: Adversarial attacks on face recognition systems (FRSs) pose serious security and privacy threats, especially when these systems are used for identity verification. In this paper, we propose a novel method for generating adversarial faces—synthetic facial images that are visually distinct yet recognized as a target identity by the FRS. Unlike iterative optimization-based approaches (e.g., gradient descent or other iterative solvers), our method leverages the structural characteristics of the FRS feature space. We figure out that individuals sharing the same attribute (e.g., gender or race) form an attributed subsphere. By utilizing such subspheres, our method achieves both non-adaptiveness and a remarkably small number of queries. This eliminates the need for relying on transferability and open-source surrogate models, which have been a typical strategy when repeated adaptive queries to commercial FRSs are impossible. Despite requiring only a single non-adaptive query consisting of 100 face images, our method achieves a high success rate of over 93% against AWS’s CompareFaces API at its default threshold. Furthermore, unlike many existing attacks that perturb a given image, our method can deliberately produce adversarial faces that impersonate the target identity while exhibiting high-level attributes chosen by the adversary.

Non-Adaptive Adversarial Face Generation
Sunpill Kim
Seunghun Paik
Chanwoo Hwang Minsu Kim Jae Hong Seo"
Department of Mathematics Research Institute for Natural Sciences, Hanyang University (ksp0352, whitesoonguh, aa5568, iayaho3248, jaehongseo) Saaaaaaaeeeaaaaggngnn.. ac. kr
Abstract
Adversarial attacks on face recognition systems (FRSs) pose serious security and privacy threats, especially when these systems are used for identity verification. this paper, we propose a novel method for generating adversarial siices-synthetie facial images that are visually distinct yet recognized as a target identity by the FRS. Unlike iterative optimization-basee approaches (e.g., gradient descent other iterative solvers), our method leverages the structural characteristics of the FRS feature space. We figure out that individuals sharing the same attribute (e.g., gender or race) form an attributed subsphere. By utilizing such subspheres, our method achieves both non-adaptiveness and remarkably small number of queries. This eliminates the need for relying on transferability and open-source surrogate models, which have been a typical strategy when repeated adaptive queries commercial FRSs are impossible. Despite requiring only a single non-adaptive query consisting of 100 face images, method achieves high success rate of over 93% against AWS's Comparel'aces API at its default threshold. Furthermore, unlike many existing attacks that perturb given image, our method can deliberately produce adversarial faces that impersonate the target identity while exhibiting high-level attributes chosen by the adversary.
1 Introduction
Computer vision has advanced significantly with the development Deep Learning (DI.) technolo- gies, which enable the extraction of discriminative features from images and have proven useful in various tasks such as classification and recognition. For example, with sufficient data, DL models
demonstrate remarkable accuracy in image classification 883 26 85
and face recognition [16] 3 40).
However, the high accuracy DL models has typically been evaluated using naturally generated (i.e., unaltered) images, and studies have shown that adversarially generated images, called adversarial examples, can fool DL models with high probability 23 53 76). Adversarial examples are artificially generated images that are perceived differently by humans and DL models. Both the generation of adversarial examples and the development of defense and detection methods are active areas of research, as adversarial examples present significant security and privacy risks in the practical For example, Di.-based Face Recognition Systems (FRSs) are widely
deployment DL. [53] 23 31
used for mobiles and website logins, as well for access control of buildings and airports (38 14]
In such systems, adversarial examples pose significant security and privacy risks 82
79 In this
paper, we focus FRS, but we believe that the techniques we developed can be spread to other
biometric fields such as voice [46] [2 47 (25). This is natural extension, as FRS a representative biometric authentication method, and the core objective of biometric systems is to achieve strong intra-class compactness and inter-class separability- princippltyppincipllee that apply across various modalities.
Several methods have been proposed to generate adversarial examples that can fool FRSs, and these are typically categorized based on the adversary's capabilities. If the adversary has full access the
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Target images from FairFace [35]
dataset
Sd
Adversarial faces using 100 queries (by column)
Scores from AWS API (default threshold: 0.8)
Projection to White
Projection Fomale
0.955
0.973
0.949
0.995
0.938
Figure 1: The core idea is to project feature vector 5d-1 onto an attributed subsphere Sy.
Figure 2: Examples of adversarial faces attack successfully against the AWS API 665
target DI. model's parameters, the attack is categorized as a "white-box attack" [53 23 18 If the adversary cannot access the model's parameters but can query, it is considered "black-box attack" [31 7 4. Black-box attacks can further be categorized based on the type of query result. For example, if the result is final prediction, such as an identity label or true/false verification
76
19 8. If the result includes logits or similarity outcome, it a ""hantlabel/decision -bssed attack" 4
scores, is Ssoft-label/score-bae
attack" 77]
31 24
71 50]. Regardless of these categories,
"GAN-based" approaches have been proposed to improve the imperceptibility [82] 288 68 [5) of
adversarial examples. Ordinary adversarial examples generated by directly modifying pixel values, which can occasionally introduce unnatural artifacts that are noticeable to humans. To address this limitation, GAN-based methods instead search the latent space of GAN synthesize natural- looking images. Other classifications include "optimization-buse -based" approaches, which solve discrete and non-continuous problems in the hard-label black-box setting [10], and "transferable attacks", which exploit the transferability of adversarial examples across DL models to bypass limited access in the black-box setting [52 62
Although the above adversarial attacks fall into different categorics, they all iteratively solve optimiza- tion problems under constraints that ensure the generated examples remain imperceptible to humans. For example, the PGD attack [53] is an iterative process that consists of two steps: (1) finding a perturbed image whose feature vector is far from that of the original, and (2) projecting it onto
set of small perturbations to ensure imperceptibility to humans. GAN-based attacks [2] 28 68 45
also involve iterative optimization process subject to objectives: maximizing the distance from the original image and minimizing perceptibility by humans. This is because these attacks rely on iterative solvers such as gradient descent (53) 23 18 49 76) and randomized gradient-free methods 71 [Q] However, these iterative solvers are cumbersome, especially in black-box settings, because they require large number of adaptive queries to the target DL model.
1.1 Our Contribution
The general idea of iterative solvers to approximate a (local) solution step-by-sted when the global landscape of the objective function is unknown. The objective function in DL contexts is often highly complex, as involves numerous factors, including the parameters of the neural network. Although there are attempts to analyze partial landscapes [1] 84 (57). understanding its entire landscape is nearly infeasible, and thus such iterative solvers may be the best approach until now. To overcome this fundamental limitation, Rather than embedding all aspects into the objective function and solving it iteratively, our approach interprets the feature space as much as possible and exploits its structural characteristics to refine the optimization problem. By leveraging this idea, we also show that our attack can be applied to black-box setting where the adversary can obtain confidence scores from queries. Note that this scenario corresponds to attacking several real-world commercial face matching APIs, e.g., provided by AWS [65] or Tencent [13] With additional techniques tailored for this setting, we successfully generate adversarial faces for these APIs using scores from a single non-adaptive query composed of 100 faces. The adversarial faces generated from our attack on AWS CompareFacce along with the corresponding confidence scores from the API, are presented in Fig- 2 All these pairs surpass the API's s default threshold of 0.8. More importantly, we emphasize that up to 13.7% of face pairs, consist of target face and adversarial face generated by our attack, surpasses the 0.99 confidence score-
propose novel method for non-adaptive adversarial face generation.

---

above the threshold suggested for law-enforeement according to Amazon's 's use-case guideline| These results demonstrate that our score-based non-adaptive approach can reliably generate adversarial faces even under realistic black-box constraints, highlighting the structural weaknesses of existing face recognition systems. For discussions on defensive strategies and responsible disclosure practices, please refer to Section 5.3 and the Ethics Statement
2 Related Works
We briefly survey adversarial attack methods against FRSs. Similar to attacks on image classifi-
cation 169 53
6 it is known that FRSs are vulnerable to adversarial attacks based on perturba- 27 In particular, recent studies have proposed attacks for black-box settings,
tions 19 80
successfully attacking real-world commercial FRSs, e.g., Tencent API [19] or Face++ [48] 27]. Along with these attacks, there is another branch of exploiting generative models for faces, e-g.. StyleGAN [37], to craft adversarial examples 778 82 28
68 45 63]. Most of these studies con-
ducted transfer attacks via ensembles of the adversary's 's FRSs while employing the naturalness loss to ensure that the resulting adversarial faces are perceived as natural in humans' eyes. A series of works 88 28, 688 45) attempted to craft adversarial faces via makeups that were guided by the GAN-based image editing techniques or the aid of a vision-language model. Recently, [63] utilized diffusion model and presented method to weaken the diffusion purification effect. Nevertheless, we point out that all these attacks either require a huge number of adaptive queries [19] or heavily rely on the transferability. The former can be defended by detection methods for adaptive queries (9 75 whereas the latter tends to exhibit lower attack success rate.
3
Attributed Subsphere S/ and Non-Adaptive Adversarial Face Generation
3.1 Our Approach to Avoid Iterative Solvers using Attributed Subsphere Projection
The most DL-based FR mode are trained by so-called "metric learning" to make the feature space particular, the above recent FR models utilize (d - 1)-sphere 5d 1 with angular distance metric d as feature space. Assume that the target FR model is well trained by metric learning. Then, for any attribute f (c.g., gender), we could naturally expect that the feature vector set Sy of all images having f lie close together in the feature space. Define the metric projection to S as ps, (i+ := argminges, d( ÿ7. For any iī E Sd-1 if we efficiently compute ps, (iz), then may use it for adversarial face generation; for example, if it is a feature vector without f and d(ps, ) i7) is sufficiently small, then ps, (it) is a feature vector of an adversarial example since it has attribute but f is not present in the original image. Using well-known inversion methods that reconstruct faces from the corresponding templates extracted from the given FRS [54] 66,
be like a metric space [51 74 30 55 3 39
77 3
67 34 [9]. can recover the adversarial face image of ps, (i7). However,
without assumptions on the structure of Sy, a naive computation of ps, becomes equivalent to exhaustive search, or it may require the use of generic iterative algorithms, e.g., gradient descent. To avoid such iterative methods, we establish a useful conjecture: the feature metric spaces (Sd-1, d) of all the metric-learning-bass based FR models share the following property.
We call the attribute that most humans possess dominant attributes. (c.g., number of eyes, nose, and mouth.) There exist nom-dominant attributes such that the feature vector set Sj of all images having f includes a k-sphere St with high probability Przesę ( E Sy]. In addition, there exists an efficient algorithm to find set of orthogonal unit vectors defining s7. called basis.
Conjecture 1.
If the above conjecture is valid, we can use the projection to the k-sphere Ps+ instead of ps, to efficiently compute without iterations. This is because naïve projection to á k-sphere is rather straightforward by basic linear algebra; if we have a basis of S/, we can first project to R including the $ and then normalize to be unit vector. However, the remaining issue is whether d(ps, (7), d) is sufficiently small. To address this, we present proposition concerning the expected distance between uniformly selected unit vector and its projection onto a subsphere.
Proposition 1. Consider the metric space (Sd-1,a d) and the metric projection to an arbitrary k- subsphere Sk C 1d-1, Pst (z) = argmingess ((E ÿ). Let be a uniformly chosen random variable
II this paper, FRS refers to the overall FR system whose final output is score. We use the term "FR model" for "feature vector extractor", which does not contain the score computation process to avoid confusion.
3

---

(a) Bilinear Interpolation the Principal Components (PCs)
(b) PCs
Figure 3: The visualization of attributed subspheres Sÿ (left) from principal components (right).
over Sd-1 and V := Psa (U). beta distribution Beta(
Then, we have that the random variable cos? (a(U, V)) follows the d_م That is, E|cos? (d(U, V)) +.
2 2
Note that Prop. provides the expectation of the squared cosine similarity, which may differ from the actual cosine similarity depending on the variances of and V. Although we experimentally verify that the expectation provides a sufficiently accurate approximation for our purposes, we defer both the proof and experimental validation to Appendix B due to space constraints. Importantly, the subsphere $* is independent of the distribution of U. In our setting, if an attributed subsphere S is fixed and the adversary arbitrarily selects a target face image reeerrraardlsss of S -ten there cxists a feature vector in St whose average cosine similarity with the target is v k/d. To show its concrete implication, we note that the decision threshold of many FRS 16 55 3 39 is typically set at most to 70", which corresponds
cosine similarity of approximately 0.3420. On the other hand, for k 128 and d 512, Prop. [ gives vk/d 0.5, i.e., 60 in angular terms. This means that for any 129-dimensional hyperplane, if we randomly select a face feature vector Sd-1 and compute its projection ग E S* then the reconstructed facial image corresponding to 0 would be recognized as the same identity as ū by the aforementioned FRS, thus achieving our main objective.
3.2 Validation of the Existence of the Attributed Subsphere sty and Conjecture L
Although the Proposition T and experimental results in Appendix show the feasibility of our strategy to eraft adversarial faces without iterative algorithms, it remains unclear whether Conj. iis indeed true, i.e., the existence of attribute-specific subspheres. Therefore, we now turn our attention to the S corresponding to attributes, e.g., race, or skin color, thus validating Conj- [ To this end, we applied Principal Component Analysis (PCA), a classical algorithm for extracting representative bases (i.e., principal components) from given distribution, the set of feature vectors from faces sharing specific attributes. Note that PCA is applied in the deep feature space (not in the pixel domain). We use PCA solely as an approximation to the basis of the attributed subsphere, not as a classical image- space preprocessor. We used the FairFace dataset (35), which provides nearly 1 10k annotated facial images, collect samples labeled with selected attributes. Specifically, we selected four attributes: male, female. White, and Black. For each attribute, we ran PCA to obtain principal components and reconstructed facial images from the components using pre-trained inverse model, Arc2Face [59]. These reconstructions are visualized and used in the subsequent analysis to evaluate whether the components span valid subspheres. In particular, by checking whether linear combination (followed by normalization) of the principal components results in facial image that still exhibits the same attribute as the original dataset used in PCA, we can empirically verify the existence of attributed subspheres. Fortunately, there is supporting evidence for this property: the semantic interpolation between facial images is known to be possible using inverse models by interpolating in the feature space (34 66 [9]. Since linear combination can be viewed as sequence of linear interpolations, the subsphere spanned by principal components can be interpreted as a valid attributed subsphere. As shown in Fig. although the pose may slightly vary, the interpolated facial images maintain identity coherence and preserve the intended shared attribute. To sum up, we conclude that Conj. is indeed true, therefore the adversary can conduct an adversarial attack by exploiting attribute-specife spheres.
We remark that our argument is not about specific choice of Arc2Face [59 we used, but about the inverse model itself of the metric learning-based FR model. Our argument still holds for other inverse models, such as NbNet [54]. Due to space constraints, we provide experimental results related to this aspect in Appendix D well as PCA results on other attributes or datasets and their interpolations.

---

Pre-processing
Black-box Pre-pracesing
F
PCA
FE
g
Images of attribbute
images of attribute
Representative
Matrix
Score matrix
Matrix R
White-Bo Attack
AAA IATAC - Cll
Black-Box Attack
Linear combination correction
g
Project to attribute space
ATR(-)
Target image img
Template x
Img with attribute
Target image img
Score Sy between 1 and img
[ATR( >0l
Img with attribute f
Figure 4: An overview of our adversarial face generation. In Figure, an attribute ff indicates male.
FL : I + 5d-1 and FT are adversary's 's own FR model and corresponding inverse model. Fr Ix I + [0, is target FRS whose output is confidence score and g sigmoid function.
Additional Note on Inverse Model Bias. While our main results Fig. and Appendix show that both Arc2Face (59] and NbNet [54] reconstruct faces consistent with the intended attributes, we observed that certain inverse models (c.g.. [66]) exhibit systematic demographic bias, often reconstructing young white male faces regardless of the input feature vector. We emphasize that such bias originates from the inverse model architecture and training data rather than our projection mechanism. Consequently, although the projection step faithfully preserves the target attribute within the feature space, the perceptual quality of the reconstructed adversarial face may vary depending on the generative capability and bias of the chosen inverse model. We have explicitly noted this limitation in our final analysis and Appendix
3.3 Non-Adaptive Adversarial Face Generation
An intriguing property of adversarial examples is the transferability, where adversarial examples generated for one local FRS can deceive another target FRS. This property convert "white-box attacks" to "black-bot attacks". Therefore, we first present our adversarial face generation algorithm in white-box setting, leveraging the insights discussed above. Let Z be the ideal collection of all facial images, and let I C I be the subset consisting of images with a specific attribute f. Given a facial image img E \ Iy, the goal of the adversary is to find another image img f such that is recognized as the same identity img by target FRS T. To achieve this, the adversary first runs PCA on ((D,))wheek
D, is an f-tttributed dataset and F is the adversary's ' own FR model to obtain PCA matrix M, whose i-th row is i-th principal components denoted by f. Using the inverse model F-1, the adversary then reconstructs the corresponding facial images O Fm) Next, A'AE [A AA where A is the matrix
the adversary defines a metric projection Psy (i) s + S as
whose i-th row is F(O,) and At is pseudo-inverse of A. Then, the adversarial face img is generated as F (ps (F(im)))) Regardless of whether the adversary generates img, such sample always exists in the attributed subsphere and is close enough img to be recognized as the same identity.
While the above (white-box) approach generates adversarial faces using F and F-1 alone, directly utilizing it for the transfer attack is insufficient for achieving high attack success rate. Notably, we observe that some facial images consistently fail in transfer attacks, and this phenomenon of lower success rates
limited our attack but can also be observed with the classical adversarial attack method based on iterative solver. Due to space constraints, detailed analysis of such cases is provided in Appendix E To overcome this, we extend our attack strategy by permitting the adversary to query the target FRS and exploit the obtained cosine similarity scores 5. To this end, we establish the following conjecture: there exists a universal basis over facial images whose interpolations via a FR model and its inverse always produce similar images under the same coefficients, regardless of the choice them. Our motivation
A A2 view the metric projection function + |[ Ad as linear combination of rows of A; note that, by the definition of pseudo-inverse, At A AT (AAT)-11 - Aě, and 5' = A.. Hence, if we appropriately treat the (AATT)1 term and the conjecture holds, then the adversary can produce the adversarial face by interpolating images in O through FR model and its inverse with scores 5, which are obtained from querying img and images O E O.
5

---

Conjecture 2. For E {1, 2), let F; be well-trained FR model and F be its inverse. Then for any well-trained FRS T with threshold TT, there exists set O of facial images s.t. for all $ [-1, 11%h
T(
A75
T
PI
||A
F2
|A]
TT,
(I)
where A, E Rlxd is a feature vector matrix whose j-th row is Fi(O,) O2 E O and j E [k].
Interestingly, found that the O constructed from an f-attributed subsphere-as realized by
400
350
Random Face Images
PCA and the inverse model-does satisfy the re- quired property in Conj. 2 To demonstrate this,
Number of Samples
Random
70"
for FR models Fi. F2. and corresponding in-
250
verse models FT- =iA'= F2 respectively, we measured the distance of the feature vector of two images in Eq. 1 extracted from another FRS T = F3. We sampled 10,000 score vectors from the uniform distribution over [-1, 1)k. For the choice of the image set O, we consid- ered three settings: (1) faces from our attribute- specific subsphere, (2) randomly sampled faces from the FairFace dataset, and (3) images con- sisting of uniformly sampled random pixels. The results are given in Fig. [5] We can observe that the measured distances from the random pixel images are hardly within the threshold (red line), whereas a non-trivial number of those from faces lie within the threshold. This necessitates the condition in Conj. 2 that O should consist of faces. More importantly, we can figure out that almost all measured distances from the attribute-specii than half of the randomly sampled faces lie outside the threshold. This indicates that our face image set O behaves well as the role of universal basis, therefore showing the validity of Conj 2
200
150
100
50
40
50
60
70
8C
90
100
Ang between Fr!
A/s
and F5
A/s
||4/=|12
\||A[=|I
Figure 5: Angle histogram for Conj. 2
subsphere are within the threshold, whereas less
sattiosship-paaticuleee in the black-box set- ting between the original image (without attribute f) and the crafted adversarial face image (with attribute f), by leveraging the projection Pst onto the f-attributed subsphere. In particular, to handle the )- term connect the linear combination of rows of A to the metrie projection mapping.
Building on Conj. - we can derive a formal
we introduce the correction matrix R. The equation is given as follows:
Subsphare projoction: Pot. (1))=
Al and T= F1 (img) TATA8112
Conj P and 12 (A1
A
AT(A44]) -3
AJR - || R 1
img
ST
PT
R FT
|| A
[[AT(AAA] -
face without attrbutee f
A[=AT(AAA]) and A==A (Query)
adversarial Face with f
From the above equations, F1 is the target FR model that the adversary can query in a black-box, and F2 is the FR model owned by the adversary. If we denote ş == Ar then each component of s corresponds to the cosine similarity between the target facial image img and the images in O. The left-hand side of the equation approximates facial image that lies on the attributed subsphere and is close enough to img to be recognized as the same identity, regardless of whether it is explicitly generated by the adversary. Note that if the adversary had access the full F, model, she could directly generate img using the white-box approach described earlier. However, since we can not access to the Fi, we proceed to reformulate the expression into a fully black-box compatible form. namely, A = Af [A]
The second equality comes from the definition of the pseudo-inverss Here, if we denote R- A₁ and 5 == R-13, 3, then we can observe that the numerator inside the FT- can be viewed as the linear combination of rows of A₁ with weights š. Hence, we can obtain the third equality by utilizing the Conj. (2 We can observe that all the involved values in the rightmost term in the equation are available to the adversary. Ag can be locally calculated by the adversary and 5 can be obtained through queries. In addition, can also be obtained from 122 cosine similarity scores by querying all the image pairs in O. Note that R is independent of the target image; the adversary can construct advance before conducting the attack. Therefore, the adversary can craft the adversarial image by the formula in the rightmost term.

---

NhNet 54
Are3Face
59
Target Image
ES Male
(35 Female
5 White
Bllkkck
VAsian
Male
B8 /Female
White
SVBlack
S MAsino
0.6747
0.5278
0,4518
0.5533
Scores
0.6154
0.4395
0.6791
0.5829
0.5248
0.5918
0.4522
0.0064
0.5020
0.4636
Table 1: Adversarial face examples using images from
29
(white-box setting; of F1: 0.2432).
One caveat that commercial FRSs typically do not return cosine similarity values, but rather confidence scores. Thus, we need an additional technique to convert the confidence scores into the cosine similarities. Fortunately, several methods have been proposed adopt them for conducting our attack against commercial FRSs. Due to space constraints, we defer the detailed analysis of the correction matrix and the score transformation technique to Appendix Finally, we provide the full description of our black-box attack in Alg. [
and we can directly
Algorithm Projection (line 1-6) and Adversarial Face Generation (line 7-8)
Require: f-attributed dataset D, target face image img E I J local FR model F I - S
its inverse model F-1 S - I, target FRS T Ix I -> [0, 1], and hyperparameter k E [d]
1: Run PCA on F(D;) to obtain My E RRxdd whose row vectors are top-k principal components
2: Set O₂ + F- (m),s) for vi E [k]. where my is i-th row vector of M
3: Set feature vector matrix A E Rkxdd, whose i-th row vector is F(O,) for E [k]
4: Query and set s + g (T'O;, O2)) for Vi,j [k:], where g()) is logistic sigmoid function
5: Set cosine similarity matrix R E [-1, 1] 1 whose ij-th component is K/.5 for Vi,j
6: Define the projection to the f-attributed k-sphere by PSy (3) ==
A
7: Query and set $ e [-1, 1] whose i-th element is g (T(O,, img)) for vi [K]
8: return img +- F=1 (ps) (1)
Experimental Results
4.1 Experimental Setting
We conducted evaluations on four face datasets: LFW 24
CFP-FP(64) and Age-DB (56), and the FairFace 35), which offers demographically balanced data to assess attack generalizability. We tested our attack using three sourcs FRSs (resp. two commercial FRSs) with three inverse models. For obtaining an appropriate set O for Conj. [ we extract stributed-speciffe PCA matrices (k 100) using VGGFace2 [5] with annotations from 70) and annotated FairFace data. Thresholds were selected per dataset: accuracy-optime values for Verification 3-sets and fixed thresholds for FairFace. Additional details and results for open-source FRSs and Tencent API are in Appendix A For commercial FRSs, we used two thresholds provided by the corresponding service provider. Additional details for each model and results for open-source FRSs [66 40 and Tencent API [33 given in appendix, due to space constraints. For evaluating our adversarial face generation, we use attack success rate (ASR). The ASR
ratio of generated images that are both classified as target identity and possess the target attribute and can be formulated as follows:
ASR ENY, ((T(img,) img,) 2 TT) 1 (img; E EI))I//I Zyl.
where Z represents the set of total images, img, E I Iy refers to each individual target image, and 1(.) function mapping I if the input statement is true and otherwise. To determine whether img, E I, in open-source and commercial target FRSs, we use an attribute classification model provided by FairFace [35] and corresponding APIs 65 [3] respectively.

---

4.2 Black-Box Attack
We first compare the ASR of the transfer attack and the black-box attack with score queries in Tab. uuiing underline. To save space, Black, and Asian, respectively, If the black- attack performs better, is underlined; otherwise, it is not. In most cases, the black-box attack with score queries outperforms the transfer attack without score queries in terms of ASR. We also present the effect of R using colored text. The blue-colored text indicates the ASR with correction matrix R is smaller than ASR without R. Since most of the ASRs are black-colored text, R is effective. We now turn to a real-world black-box setting where an adversary can only obtain unknown metric scores. In Tab. B we present our ASR against the AWS CompareFace API 65 using gender-attributed D,- Our attack achieves significantly high ASR with default threshold of 0.8. Even if we set the strict threshold of 0.99 recommended by Amazon for use cases involving law-enforcement, our attack achieves ASRs up to 13.70%. It is noteworthy that without matrix R. the ASR is only less than 1.5%. We also note that in the FairFace dataset, transfer attacks were not performed at all except for one case. Due to space constraints, we provide all ASR against Tencent CompareFace API [3] FT using acce-atrribtedd D in Appendix G
M. F, W, B, and A to denote Male, Female, White,
Target Dataset
Target
withour
F-1
f
LFW
CFP
AGE
FairFace
= U.8
U.50
=
U.S
= 0.90
29
64
[66]
FW
TCFP
TAGE
M
Fransfer Allack without Queties
N/A
47.96
7.06
M
97.29
99.41
99.18
98,28
99.28
99.42
61
25,80
54
W
F
99.94
93.33
99.95
96.02
97.44
100
98. 90
94.84
96.63
99.68
96.75
99.75
35
NIA
B
87.98
90.61
88.66
92.12
92.63
92.68
Direel Allack with Score Queries
A 73.82
73.09
72.27
78.91
79.53
79.63
51
89
91,45
5.95
60.19
71,25
1.49
M
70.82
86.66
91.16
81.31
91.94
93.63
95.87
13.70
60.33
59
F
38.48
80.25
86,20
75.21
88.31
91.02
91.00
12.35
37.23
0.78
W 84.42
90.43
94.84
73.32
87.96
90.49
69
29
79.35
0
24.35
37.92
B 80
93.68
94.97
86.95
95.23
96.38
84.16
3.27
44.58
0.20
A 73.97
$4.90
89.81
83.09
90.95
91.88
71.23
2.15
30.72
0
Table 2: Black-box ASR on ViT-KPRPE [40]
(F2)
Table 3: Black-box ASR on AWS (65] (FA)
5
Ablation Studies and Discussion
5.1
Comparison with Prior Work
63 aim to protect a given face image- the adversary's 's own-by manipulating so that the FRS classifies it as different identity. In contrast, our work pursues the opposite objective: to generate synthetic facial images that are visually different from the adversary but are still recognized as the adversary by the FRS. This represents a fundamental difference: prior methods aim for risual similarity with semantic difference, while our method seeks semantic similarity with visual difference. Nevertheless, it possible to adapt previous approaches to simulate our setting by reversing their direction: that is, by simply swapping the source image and the target image. It is worth noting that our method operates without source image. For comparison, we selected the most recent method (63. which follows diffusion-based iterative attack paradigm, and re-purposed in our setting..Snce Since they use random face images from [61] as inputs without attribute constraints, we also generated adversarial faces targeting all attributes categories in our setting to ensure a fair comparison. In Tab. g we first provide the transfer ASR of both methods, the adversary has white-box surrogate models and attacks against AWS CompareFace. Then, we further evaluated our method by issuing queries to the actual API. Diffusion-based iterative methods typically operate in white-box setting, requiring hundreds thousands of adaptive queries and explicit gradient computations on the target FRS. In contrast, our non-adaptive approach relies solely on the reported similarity score and succeeds with a single query per target, demonstrating comparable effectiveness with far greater efficiency. We also present a visual comparison in Tab. [ showing that our method generates facial images that are much more diverse and visually unrelated to the adversary, while still being classified as the same identity. On the other hand, the previous method tends to depend on the attribute of source image. Due to space constraints, additional details for Tab, 4 are provided in Appendix G
Most prior works 78 82
28
68
45
'Transfer attacks are performed using white-box surrogate models and therefore incur no queries to the target FRS (query count 0); the blx results report attacks that query the target for similarity scores.
8

---

Method
163
Ours (Transfer attack without queries)
Ours (Attack with 100 queries against AWS)
[6T
Male
Female
White
Black
Asian
Male
Female
White
Black
Asian
0.8
29.86
23.80
33.20
62.80
22.20
35.60
94.20
92.00
98.20
99.20
98.80
0.99
3.41
1.20
1.60
6.60
1.40
1.80
14.60
8.60
41.00
40.80
24.60
Table 4: ASR of [63 and ours evaluated on CelebA-HQ dataset [36 with different thresholds
66
Ours
Target
Source
Result
Source
Result
Transfer
Direct
Transfer
Direct
Scores
0.0101
0.3557
0.0170
0.9964
0.6876
0.9792
0.9921
0.9936
Table Visual comparison with 663 The target image shown on the left; ours used female and white attributed subspheres, respectively. To ensure a fair comparison, [63] used female and white source images, Additional results for male, black, and asian attributes are provided in Appendix
5.2 Black-box Attack on Non-facial Target
In Conj. 2 we did not impose any specific assumptions on the score vector g, which indicates that the extraction of scores does not necessitate the input being facial images. In Tab. [23] illustrated intriguing examples whose targets are non-facial images that provide some evidence that the proposed attack can be successfully performed only facial images but also on non-facial images, which are unrelated to the target model's s task. For more details, please refer to Appendix G
5.3 Possible Mitigation of Our Black-box Attack
We discuss possible mitigations against the proposed attack, focusing on the black box setting, since in white-box or transfer scenarios, the adversary is assumed to have control over the FRS model. straightforward defense would be return only decisions (c.g.. "accept"""reejctt) instead of confidence scores. However, such an approach may violate regulations such as the EU AI Act [1] and realized via confidence scores. Therefore,
GDPR 73 which mandate a right to explanation-ususee we investigate defenses under the current threat model where confidence scores remain accessible. From a theoretical perspective, our attack is grounded in Prop. [ and Conj- [ The former enables exploiting feature subsphere to approximate target vectors; the latter enables improved ASRs using queried confidence scores. To mitigate Conj. 2 one option is to add noise to the returned score. While this may reduce FRS accuracy, it also lowers the ASR, thereby neutralizing the advantage over transfer-based attacks. To address Prop I
we recall that the average cosine similarity between the original feature vector and its projection onto k-dimensional subspace is /k /k/d. d. r is the threshold for a successful match, impersonation requires at least 2 dr Thus, increasing either the dimension d would raise the required number of queries. However, both trade-off's are not explored. Increasing imposes heavier computational and storage costs, especially for training. In fact, enlarging the dimension d has not been actively studied and, as shown in the MFR benchmark (32), current models use only 128-1024 dimensions. Raising т, on the other hand, significantly lowers the TAR by increasing false rejections. Simply increasing on pre-trained models leads to severe performance degradation, making unsuitable for practical deployment. To explore this direction more effectively, we implemented prototype FRS trained from scratch with higher 7 as proof-of- concept. As shown in Table 6 this configuration led to a significant reduction in ASR by more than
Dr
Target
Fair/Male
Fair/Female
VGG/White
VGG/Black
VGG/Asian
F2
98.53
96.69
99.94
99.49
98.3
F3
99.62
98.55
100
99.68
99.29
Fp
6.02
3.07
15.52
4.55
2.50
Table 6: To isolate the effect of varying acceptance thresholds independent of attributes, we report identity matching rates (IMR; see Supplementary Appendix for definition). Reconstructions are obtained using F-11 the inverse model. Fp denotes the prototype FRS, which shares the Inception ResNet- 101 architecture with F3 and is trained on the MS1MV3 17 dataset.

---

80% compared to the ASRs against F2 and Fh-wh details are provided in Appendix G
while keeping the number of queries fixed. Further
Ethics Statement
While our study introduces an effective attack method, its primary purpose to provide a rigorous analysis that reveals structural weaknesses in FRSs and to promote the development of more robust and trustworthy recognition standards. Our work addresses a critical gap FRS security by showing that non-matching attributes (e.g., gender or race) can still yield high cosine-similarity scores due to suboptimal threshold tuning (typically 0.2-0.3). This vulnerability can affect real-world identity- verification platforms, potentially enabling unauthorized access or impersonation. By exposing these flaws, our study informs service providers, regulators, and researchers of the need for stronger, attribute-awe aware defense mechanisms. For instance, findings verification thresholds or additional semantic consistency checks, thereby enhancing user safety and trust. Section 5.3 outlines practical defense strategies toward more secure systems.
help platforms adopt stricter
To prevent misuse, we will not release the adversarial generation pipeline or related APIs. Bench- marking code for FRS evaluation will be shared under controlled access (e.g., to verified academic researchers via a private repository) to ensure responsible dissemination. Details specific to platform- level experiments are omitted in this paper to avoid potential misuse. These implementation details will disclosed only to the affected service providers upon request, balancing transparency with risk mitigation. The core algorithm and non- non-sensitive experimental setup are fully described in the paper to ensure reproducibility for academic research.
Although our method can be conditioned on demographic attributes such as gender or race, this was not intended for discriminatory targeting. Instead, we demonstrate that even with distinct attribute values, adversarial faces can achieve high matching scores high-ighthhhhhgggggg structural vulnerabilities of existing FRSs rather than exploiting demographic bias. This motivates future work toward attribute- aware robustness and more secure, trustworthy face-recognition standards.
7 Conclusion
In this paper, we have investigated how close feature vectors with different attributes are in the feature metric space of FRS. To this end, we develop a process of exploring the feature space using universul basis based on Conj [ which can serve a compass to navigate in the dark, so that we could successfully extend this idea to non-adaptive adversarial attack in the black-box setting. This shows that although the metric learning, the dominant training method for FRS, provides huge benefit for high accuracy, it is also useful for the adversary to design attacks with a high attack success rate. To the best of our knowledge, our attack is the first adversarial attack that is non-iterative, non-adaptive, and specialized to the metric learning-bate based DL technology. Rather than relying gradients perturbation constraints, our method leverages the intrinsic structure of conventional assumptions
the representation space to construct adversarial examples-challenye about what is necessary for attack success. We leave some open questions, such attacks specific to other DL-based systems trained in different ways than FRS. Another promising direction is to discover atrributeinformeed subspaces in data-driven manner, for instance by using unsupervised or weakly supervised techniques such as clustering or ditent-direction discovery. Such approaches could reveal more intrinsic structures of the feature space beyond manually defined attributes and further enhance both the theoretical and practical understanding of adversarial generation. In the opposite direction, it would also be interesting to investigate training paradigms that inherently reduce or restrict the exploitable structure for adversarial uses. Finally, we hope this study raises awareness of the vulnerability of commercial face APIs and encourages the development of secure and trustworthy face-recognition standards (e.g., ISO/IEC 24745).
Acknowledgments
This work was supported by Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2024(RS-202) 2024. 00332210)
10

---

References
[1] E. I. Act. The eu artificial intelligence act, 2024.
[2] X. An, X. Zhu, Y. Gao. Y. Xiao, Y. Zhao. Z. Feng, L. Wu, B. Qin. M. Zhang. D. Zhang, et al. Partial fc: Training 10 million identities single machine. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1445-1449, 2021.
[3] F. Boutros, N. Damer, F. Kirchbuchner, and A. Kuijper. Elasticface: Elastic margin loss for deep face recognition. In Proceedings of the IEEE/CVE conference on computer vision and paltern recognition, pages 1578-1587, 2022.
[4] W. Brendel, J. Rauber, and M. Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. arXiv preprint arXiv:1712.04245 2017.
[5] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman. Vggface2: dataset for recognising faces across pose and age. In 2018 13th JEEE international conference on automatic face & gesture recognition (FG 2018), pages 67-74. IEEE, 2018.
[6] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages 39-57. leee, 2017.
[7] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 15-26, 2017.
[8] W. Chen, Z. Zhang, X. Hu, and B. Wu. Boosting decision-basse based black-box adversarial attacks with random sign flip. In European Conference on Computer Vision, pages 276-293. Springer, 2020.
[9] Z. Chen. On the detection of adaptive adversarial attacks in speaker verification systems. IEEE Internet of Things Journal, 10(18):12271-1222 2023.
[10] M. Cheng, T. Le, P.-Y. Chen, J. Yi, H. Zhang, and C.-J. Hsich. Query-efficient hard-label black-box attack: An optimization- approach. arXiv preprint arXiv:1807.04453 04457, 2018.
[11] A. Choromanska, M. Henaff, M. Mathieu, G. Arous, and Y. LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pages 192-204. PMLR, 2015.
[12] J. S. Chung, J. Huh, S. Mun, M. Lee, H. S. Heo, S. Choe, C. Ham, S. Jung. B.-J Lee, and I. Han. In defence of metric learning for speaker recognition. arXiv preprint urXiv: /1982, 2020.
[13] T. Cloud. CompareFace API.
[14] S. Del Rio, D. Moctezuma, C. Conde, I. M. de Diego, and E. Cabello. Automated border control e-gates and facial recognition systems. computers & security, 62:49-72, 2016.
[15] J. Deng. J. Guo, E. Ververas, I. Kotsia, and S. Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5203-5212, 2020.
[16] J. Deng, J. Guo, N. Xue, and S. Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690-4699, 2019.
[17] Deng, Guo, D. Zhang, Y. Deng. X. Lu, and S. Shi. Lightweight face recognition challenge. In ICCV Workshops, pages 2638-2646. IEEE, 2019.
[18] Y. Dong, F. Liao, Т. Pang, H. Su. J. Zhu, X. Hu, and J. Li. Boosting adversarial attacks with the IEEE conference on computer vision and pattern recognition,
momentum. In Proceedings the pages 0185--193, 2018. 8.
[19] Y. Dong, H. Su, B. Wu, 7. Li, W. Liu, T. Zhang, and J. Zhu. Efficient decision-basee based black-box adversarial attacks on face recognition. In proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 77 7714-7722 2019.

---

[20] C. N. Duong, T.-D. Truong, K. Luu, K. G. Quach, H. Bui, and K. Roy. Vec2face: Unveil human faces from their blackbox features in face recogmition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1132-6141, 2020.
[21] J. Engelsma, A. Jain, and V. N. Boddeti. Hers: Homomorphically enerypted representation search. IEEE Transactions on Biometrics, Behavior, and Identity Science, 4(3):349-360, 2022.
[22] S. Gong, V. N. Boddeti, and A. Jain. On the intrinsic dimensionality of image representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3987-3996, 2019.
[23] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprini arXiv: 1412.6574 2014.
[24] C. Guo, Gardner, Y. You, A. G. Wilson, and K. Weinberger. Simple black-box adversarial attacks. In International conference on machine learning, pages 2484-2493 PMLR, 2019.
[25] B. Han, Z. Chen, and Y. Qian. Exploring binary classification loss for speaker verification. In ICASSP, pages 1-5. IEEE, 2023.
[26] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[27] C. Hu, Y. Li, Z. Feng, and X. Wu. Towards transferable attack via adversarial diffusion in face recognition. IEEE Transactions on Information Forensics and Security, 2024.
(28) S. Hu, X. Liu, Y. Zhang, M. Li, L. Y. Zhang, H. Jin, and L. Wu. Protecting facial privacy: Generating adversarial identity masks via style-robust makeup transfer. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 15014-15023, 2022.
[29] G. B. Huang. M. Mattar, T. Berg, and E. Learned-Mille Labeled faces in the wild: database forstudying face recognition in unconstrained environments. In Workshop on faces in RReal- Life' Images: detection, alignment, and recognition, 2008
[30] Y. Huang, Y. Wang, Y. Tai, X. Liu, P. Shen, S. Li, J. Li, and F. Huang. Curricularface: adaptive curriculum learning loss for deep face recognition. In proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 5901-5910, 2020.
[31] A. Ilyas, L. Engstrom, Athalye, and Lin. Black-box adversarial attacks with limited queries and information. In International conference on machine learning, pages 2137-2146. PMLR, 2018.
[32]
InsightFace. Mfr ongoing. Accessed: 2025-01-02.
[33] X. Jia, J. Zhou, I.. Shen, J. Duan, et al. Unitsface: Unified threshold integrated sample-to-sample loss for face recognition. Advances in Neural Information Processing Systems, 36:32732-32747, 2023.
[34] M. Kansy, A. Raël, G. Mignone, J. Naruniec, C. Schroers, M. Gross, and R. M. Weber. Con- trollable inversion of black-box face recognition models via diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3167-3177, 2023.
[35] K. Karkkainen and J. Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1548-1558, 2021.
[36] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 10/96, 2017.
[37] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4396-4405. IEEE, 2019.
12

---

[38] N. Khan and M. Efthymiou. The use of biometric technology at airports: The case of customs and border protection (cbp). International Journal of Information Management Data Insights, 1 1()): 2021.
[39] M. Kim, A. K. Jain, and X. Liu. Adaface: Quality adaptive margin for face recognition. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 18750-18759) 2022.
[40] M. Kim, Y. Su, Liu, A. Jain, and X. Liu. Keypoint relative position encoding for face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 244-255, 2024.
[41] S. Kim, K. Tan, Jeong. S. Mondal, M. M. Khin, and H. Seo. Scores tell everything about bob: Non-adaptia face reconstruction on face recognition systems. In 2024 IEEE Symposium on Security and Privacy (SP), pages 161-161. IEEE Computer Society, 2024.
[42] D. P. Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6911 2014.
[43] M. Knoche, T. Teepe, S. Hörmann, and G. Rigoll. Explainable model-agnostic similarity and confidence in face verification. In WACV, pages 1-8. IEFE, 2023.
[44] R. D. Labati, A. Genovese, E. Muñoz. V. Piuri, F. Scotti, and G. Sforza. Biometric recognition in automated border control: a survey. ACM Computing Surveys (CSUR). 49(2):1-39, 2016.
[45] M. J. Wang. H. Zhang. Z. Zhou, S. Hu, and X. Pei. Transferable adversarial facial images for privacy protection. In Pruceedings of the 32nd ACM International Conference on Multimedia, pages 10649-10658, 2024.
[46] Y. Li, F. Gao, Z. Ou, and J. Sun. Angular softmax loss for end-to-end speaker verification. In 2018 11th International Symposium Chinese Spoken Language Processing (ISCSLP), pages 190-194. IEEE, 2018.
[47] Z. Li, M. Mak, and H. M. Meng. Discriminative speaker representation via contrastive learning with class-aware attention in angular space. In ICASSP, pages 1-5. IEEE, 2023.
[48] 7. Li, B. Yin, T. Yao, Guo, S. Ding, Chen, and Liu. Sibling-attack: Rethinking transferable adversarial attacks against face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24626 22637, 2023.
[49] J. Lin, C. Song, K. Hc, L. Wang, and J. E. Hoperoft. Nesterov accelerated gradient and scale invariance for adversarial attacks. arXiv preprint arXiv:1908.062. 2019.
[50] J. Liu, J. Zhou, Zeng, and J. Tian. Difattack: Query-efficient black-box adversarial attack via disentangled feature space. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 3666-3674, 2024.
[51] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 212-220, 2017.
[52] Y. Liu, X. Chen, C. Liu, and D. Song. Delving into transferable adversarial examples and black-box attacks. arXiv preprint arXiv:1611.02777 1.207700, 2016.
[53] A. Madry. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: (706::/700600088 2017.
[54] G. Mai, K. Cao, P. C. Yuen, and A. K. Jain. On the reconstruction of face images from deep face templates. IEEE transactions on pattern analysis and machine intelligence. 41(5):1188-11)) 2018.
[55] Q. Meng, S. Zhao, Z. Huang, and F. Zhou. Magface: A universal representation for face recognition and quality assessment. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14225-14234, 2021.
13

---

[56] S. Moschoglou, A. Papaicannou, C. Sagonas. J. Deng, I. Kotsia, and S. Zafeiriou. Agedb: the first manually collected, in tn-the-wild age database. In proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 51-59, 2017.
[57] Q. Nguyen and M. Hein. Optimization landscape and expressivity of deep cnns. In International conference on machine learning, pages
PMLR, 2018.
[58] S. Paik, D. Kim, C. Hwang. S. Kim, and J. H. Seo. Towards certifiably robust face recognition. In European Conference on Computer Vision, pages 143-161. Springer. 2024.
[59] Papantoniou, A. Lattas, S. Moschoglou. Deng, B. Kainz, and S. Zafeiriou. Are2face: A foundation model for id-consistent human faces. In Proceedings of the European Conference on Computer Vision, volume I, page 6, 2024.
[60] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradbury, G. Chanan, T. Killeen, 7. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. DeVito, M. Raison, A. Te- jani, S. Chilamkurthy, B. Steiner, L. Fang, Bai, and S, Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[61] G. Photos. 100k Faces Generated by AL.
[62] Y. Qin, Y. Xiong, J. Yi. and C.-J. Hsich. Training meta- model for transferable adversarial attack. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9516-9524, 2023.
[63] A. Salar, Q. Liu, Y. Tian, and G. Zhao. Enhancing facial privacy protection via weakening diffusion purification. arXiv preprint arXiv:2503. 10350, 2025.
[64] S. Sengupta, J.-C. Chen, C. Castillo, V. M. Patel, R. Chellappa, and D. W. Jacobs. Frontal to profile face verification in the wild. In 2016 IEEE winter conference on applications of computer vision (WACV), pages 1 1-9. IEEE, 2016.
[65] A. W. Service. CompareFaces API.
[66] H. O. Shahreza and S. Marcel. Face reconstruction from facial templates by learning latent space of generator network. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[67] H. O. Shahreza and S. Marcel. Template inversion attack against face recognition systems using 3d face reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19662-19672. 2023.
[68] Shamshad, M. Naseer, and K. Nandakumar. Clip2protect: Protecting facial privacy using text-guided makeup via adversarial latent search. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 20595-20605 2023.
[69] C. Szegedy. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
[70]
Terhörst, Fährmann, N. Kolf, Damer, Kirchbuchner, and A. Kuijper. Maad-face: A massively annotated attribute dataset for face images. IEEE Transactions on Information Forensics and Security, 16:3942-3957, 2021.
[71] C.-C. Tu, P. Ting, P.-Y. Chen, S. Liu, H. Zhang, J. Yi, C.-J. Hsieh, and S.-M. Cheng. Autozoom: Autoencoder-basse zeroth order optimization method for attacking black-box neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 742-749, 2019.
[72] C. Voglis and I. Lagaris. A rectangular trust region dogleg approach for unconstrained and bound constrained nonlinear optimization. In WSEAS International Conference on Applied Mathematics, volume 7, pages 9780429081.38 130008881111111111 138, 2004.
[73] Voigt and A. Von dem Bussche. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10(3152676):0--22 2017.
14

---

[74] H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou. Z. Li, and W. Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5265-5274, 2018.
[75] X. Wang, K. Chen, X. Ma, Z. Chen, J. Chen, and Y.-G. Jiang. Advqdet: Detecting query-based adversarial attacks with adversarial contrastive prompt tuning. In Proceedings of the 32nd ACM International Conference Multimedia, pages 6212-6221 2024.
[76] X. Wang and K. He. Enhancing the transferability of adversarial attacks through variance tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1933-1933, 2021.
[77] Y. Wen, W. Liu, A. Weller, B. Raj, and R. Singh. Sphereface2: Binary classification all you need for deep face recognition. In International Conference on Learning Representations, 2022.
[78] X. Yang. Y. Dong, T. Pang, H. Su, J. Zhu, Y. Chen, and H. Xue. Towards face encryption by generating adversarial identity masks. Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3897-3907, 2021.
[79] X. Yang, C. Liu, L. Xu, Y. Wang. Y. Dong, N. Chen, H. Su, and J. Zhu. Towards effective adversarial textured 3d meshes on physical face recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4119 4128, 2023.
[80] X. Yang, D. Yang, Y. Dong, H. Su, W. Yu, and Zhu. Robfr: Benchmarking adversarial robustness on face recognition. arXiv preprint arXiv:2007.041 18, 2020.
[81] D. Yi, Z. Lei. S. Liao, and S. Z. Li. Learning face representation from scratch. arXiv preprint arXiv:1411.7922, 2014.
[82] B. Yin, W. Wang. T. Yao, J. Guo, Z. Kong, S. Ding, J. Li, and C. Liu. Adv-makeup: A new imperceptible and transferable attack on face recognition. arXiv preprint arXiv:2105.03162. 2021.
[83] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners image-text foundation models. arXiv preprint arXiv:2205.0199 2022.
[84] C. Yun, S. Sra, and A. Jadbahaie. Global optimality conditions for deep neural networks. arXiv preprint arXiv: 2017.
[85] X. Zhai, A. Kolesnikov. N. Houlsby, and L. Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12101-12113, 2022.
[86] H. Zhang, Y. Yu, J. Jiao, E. Xing. L. El Ghaoui, and M. Jordan. Theoretically principled raae-oof between robustness and accuracy. In International conference on machine learning. pages 7472-7482. PMLR, 2019.
[87] T. Zheng, W. Deng, and J. Hu. Cross-age Ifw: A database for studying cross-age face recognition in unconstrained environments. arXiv preprint arXiv:1708.08119 2017.
[88] Z. Zhu, G. Huang, J. Deng, Y. Ye, J. Huang, X. Chen, J. Zhu, T. Yang, J. Lu, D. Du, et al. Webface260m: A benchmark unveiling the power of million-scale deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10492 10502, 2021.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The Abstract and Introduction state our contributions: score-based non- adaptive adversarial face generation framework and its evaluation on commercial FRS APIs. Discussions on possible mitigations and responsible disclosure practices can be found in Section 5.3 and the Ethics Statement
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: As discussed in the main text, our attack relies on access to final similarity scores; if an API returns only binary accept/reject decisions (or completely hides scores), the score-based attack becomes infeasible. Results were evaluated on a subset of commercial APIs and coarse attributes (gender, race); generalization to other platforms and finer attributes remains future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex- system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledger in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
16

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We provided full proof of Proposition in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Ys]]
While we do not release the full attack pipeline to mitigate dual-use risks, we provide all datasets, hyperparameters, and evaluation settings. Evaluation scripts will be shared under controlled access with verified researchers to enable reproduction.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
17

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
To reduce misuse, we do not publicly release the attack pipeline platform- specific scripts. Public datasets and libraries are cited, and benchmarkinn/evaauutii scripts may be shared under controlled access with verified academic researchers.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ btliss.cc/puubic/guudeeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: For training the recognition models and the corresponding inverse models, followed the parameter setting provided in the original papers. Details are in Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We report aggregate ASR/MMR without formal significance tests because attacks are largely deterministic under fixed seeds and API responses; variance across runs was negligible our setting.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
We provided our hardware settings in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http::/...c....l.ccc.c.ccauaaaaa
Answer: [Yes]
Justification: Our method has the potential to be abused but we provided a mitigation method in Section 5.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
19

---

Justification: Our method could generate adversarial faces with different attributes, which could be abused to generate fake profile.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We implement safeguards by withholding platform-specific details and limiting release evaluation scripts to verified researchers; the core algorithm is documented for academic reproducibility (Ethics Statement
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cited all datasets, face recognition models, and APIs.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
20

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We did not provide any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No third-party human subjects were involved; images used for real-world validation were self-captured by the authors only (Ethics Statement 6).
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No third-party human subjects were involved; images used for real-world validation were self-captured by the authors only (Ethics Statement
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Our method not related to LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve L.MMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
A Additional Implementation Details
All experiments were conducted on single NVIDIA A100 GPU using PyTorch [60]. In addition, when open-source face recognition models and their inverse models were utilized, the official inference codes provided by each model were used.
A.1 Details of the Models Employed
In this section, we provide detailed descriptions of all the models used in our work. Specifically, we discuss the face recognition models, their inverse models, and the attribution classification models employed in our experiments.
FRSs and their inverse models The specifications of the face recognition models for our attacks were briefly introduced in the main text. However, due space limitations, provide detailed information, including the model architectures, loss functions, and training datasets. Therefore, we present Tab. which includes these details. For our experiments, F; was sourced from InsightFace (32), while the parameters for F2 and F, were provided by CVLFace We trained FiA and F using the loss functions and training dataset detailed in Tab. For Fig- we utilized the parameters provided by Are2Face [5] Additionally, for the face recognition models, we show the thresholds at which the highest accuracies achieved in evaluations not only LFW but also CFP-FP and AgeDB, all within Tab.
unable to
Face Attribute Classification model We mentioned that verify whether the results of our attack
reflect the intended attributes. i.e., check whether Img, E Iy. we utilized an attribute model. Specifically, we used publicly available model from FairFace, which is known to distinguish gender and four racial groups (White, Black, Asian, and Indian). To evaluate the performance this model, we compared the original labels from the FairFace Validation set with the outputs of the model in our experimental setup. The ACCs in Tab. represent the percentage of images, for which the original
'hurps//gitthubb "htnps//githuu.commmm-m https://gthhubcco///oovos//////t
22

---

Open-source FRS/ Inverse Model
Train Dataset
FARC?)
Notation
Architecture
Nare
LFW
CFP
AgeDB
ResNei- 100
AreFace
Glini360k 2
99. 99.7000000
98. 690.06
97.4700.87
NbNet-B
Perceptual
MSIMV3
N/A
N/A
N/A
Are2Face Vir-KPRPE
ID-oonditionig
WebFaco42m [8 FFHQ [37]
NIA
N/A
N/A
AdaFace
WebFace
99.67000000
98. 7T..0
97.07
Fg
NbNet B
Perceptual
MSIMV3 [
N/A
N/A
N/A
Pa
Inception KesNet-101
AreFace
WebFace-Im 88
99.73
98. 74.04.00.20
97.33 LI7
Cumpare API
Fr
Teneent CompareFace API
Table 7: Description of Open-Soure Face Recognition Systems (FRSs) and Their Inverse Models.
Note on Inverse Model Families. Among inverse models used our experiments, Arc2Face is a diffusion-based model that reconstructs high-fidelity faces from feature vectors, while NbNet [54] represents GAN-based deconvolutional inverse network. Although our main results focus on Arc2Face and NbNet for consistency, the proposed framework is fully compatible with other inverse architectures such as GAN-based models (c.g., Vec2Face (20].. This underscores the generality of approach across different inverse model families, as long as the model can reliably map feature vectors back to the pixel domain.
Dataset
LFW
CFP-FP
AgeDB
Fi
0.2432
0.2092
0.1833
F2
0.2272
0. 0.1892
0.1772
F33
0.2212
0818
0.1652
FA
0.8 (Default Threshold)
FT
0.6 (Default Threshold)
Table 8: Thresholds for FRSs Across Face Verification Datasets (LFW, CFP-FP, AgeDB).
label matches the attribute, that were correctly classified by the model. More specifically, in the case of FairFace, East Asians and South Asians were both considered Asians.
Male
Female
White
Black
Asian
Tyl
5792
5162
2085
1556
2965
ACC
95.7
96.07
93.72
94.6
96.93
Table 9: Performance of attribute classification models: The first row represents the attributes f. the second row shows the number of images in the FairFace validation set that the original label equals with f, and the last row shows the accuracy.
A.2 Statistics of the Datasets
This subsection presents the statistics of the image datasets targeted in the experimental attacks conducted in this study.
Target Image Datasets. We utilized both facial images and non-facial images as targets, with their
respective statistics summarized in Tab. 10
Facial
Non-Facial
Dataset
LFW
CFP-FP
AgeDB
FairFace 10,954/ N/A
CIFAR-10
Flower- 102
Random
IIgs/(Ds)
73.233/7749
7,000/500
16,488/568
10,000
6,149
10,000
Table 10: Statistics of the attack target datasets (both of facial and non- non-facial).
Also, when we perform our proposed attack, target images have to be chosen as images that don't 'tt possess the target attribute. So we provide Tab. TI which presents the number of images that possess attributes f. The attribute judgment for constructing this table was also carried out using the attribute classification model mentioned earlier. Of course, when original labels are available, such as in the case of FairFace, the original labels were used for classification. Specifically, original labels with East Asians and South Asians were both considered Asians in the FairFace case.
23

---

Dataset for D. To perform our attack, it is necessary to extract the PCA matrix from the dataset D with f-attribute. Therefore, we created D, using the attribution labels provided by FairFace and MAADFace about the VGGFace2 dataset. We selected five attributions: two for gender from FairFace and three for race from VGGFace2. The number of images per attribute provided in Tab. T2
Dataset
Male
Female
White
Black
Asian
LFW
9,341
2,659
4,286
3,322
1,755
CFP-FPP
10,433
3,567
7,604
3,442
1,546
AgeDB
7,259
4,741
5,568
3,559
733
FairFace
5,792
5,162
2,085
1,556
2,965
CIFAR-I0
1,264
Flower-100 102
445
Random
3,324
Table 11: Statistics of target datasets for Ty-
Imgs
Male
45,986
FairFace
Female
40,758
White
2,136,057
VGGFace2
Black 157,109
Asian 115,021
Table 12: Statistics of datasets for Dj-
B Validation of Proposition 3.1.
In this section, we provide an omitted proof for Proposition I We also experimentally verify Proposi- tion by measuring distances between feature vectors and random k-hyperplane k-hyperplanes derived from faces whose feature vectors are almost orthogonal to each other.
Proof of Proposition 3.1.. Let denote Pk as the k-hyperplane containing & and define a random variable V as a projection of U onto Px. Then we have that cos? (d(U, V)) = |V Hence, we focus on analyzing V [/ instead of cos? (d(U, V)).
Because of the radial symmetry of the hypersphere, the distribution of V is identical to the following random variable W =WW,,.. Wa) defined over Rd:
UUi i W. = 0
otherwise.
where U, is the i'th component of U for i E [d].
To analyze W. we first that for a random variable X = (X1...., X2) NN0, Ia), the random
x_ follows the uniform distribution over Sd-1 That is, analyzing the distribution of variable = |
[|V][3 equivalent
X?
(2)
X2
2 X7
where & means that two random variables are equivalent terms of distribution.
Here, we can observe that each X,, X, for i * j are pairwise disjoint. In addition, Σ X and Ej X² follow chi-squared distribution with degree of freedom & and d - k, respectively. That is, the RHS of Eq. (2) follows Beta( & d-k) by definition. By using the fact that the mean of the 2: Beta(a, 3) is a+3. finally obtain E|] V [ $. This completes the proof.

---

90'
Prop. 3.1.
From Faces
Thres, on 70
60
d(U, W.
30
0' 100
200
300
400
500
Dimension of Subspace
Figure 6: Measured distance d(U, V) of the projected face feature vector onto the subsphere. Red dots indicate the theoretically predicted value according to Prop.
With an inverse model, we can find a set facial images corresponding to the basis of the given subsphere. Using them, we attempted to simulate the settings and result of Prop. | First, for randomly selected k-subsphere St, we measured dxr, Ps* (x)) for the given feature vector All feature vectors extracted from the merge of the LFW, CFP-FP, and AgeDB datasets, excluding overlapping images. In addition, we also measure the same quantity under the same setting as above, except we sample k-subsphere from faces whose feature vectors are almost orthogonal to each other. We can view this as sampling a feature subsphere with considering the distribution of facial images, rather than independently from it. Such a set of faces is called orthogonal face set (OFS), which was first proposed by [41]. To generate them, we devised and exploited an efficient algorithm by utilizing the input space of the inverse model, whose description is given in Appendix A. We used the pre-trained ArcFace [16 as the FR model to obtain feature vectors, For the inverse model, we used the pre-trained NbNet (54) of the aforementioned FR model. The detailed description of each model is given in Tab. 7 as F1 and FTA respectively.
Sd-1
The results are illustrated in Fig. 6 From this figure, we can observe that the simulated result (blue line) well coincides with the theoretically predicted value via approximation (red dots). More importantly, we can observe that both the distance calculated from the OFS (orange line) and the blue line lie below the red line corresponding to 70 when 2 100, i.e., such choice is sufficient for generating a face that is identified as the same person with the target identity. One can figure out that the simulation result from faces is strictly less than that from Prop. [ for all dimensions of the subsphere. This result indicates that there exist good feature subspheres obtain more accurate feature vector than uniformly selected one, and more importantly, one way to obtain them is to select a feature subsphere derived from actual facial images.
C Efficient OFS Generator
For generating OFSs, Kim et al. [4]] utilized rather naïve approach by collecting lots of facial images and finding a subset being an OFS. However, their method is not scalable because the possible number of subsets is exponentially many with respect the size of the desirable OFS set, and more importantly, there is no guarantee whether such an OFS exists in a pre-selected set of facial images.
To mitigate these issues, propose an alternative approach by optimizing on the latent space of the inverse model. More precisely, instead of searching an OFS over the facial images, we focus on finding the set of latent vectors {21...., ] C gd-1 of the inverse model F-1 gd-1 + I of need to ensure
FRS F I - 5d-1, Note that the inverse model does not give an exact inverse, that the feature vectors corresponding to facial images (F-1 F- (2+)) are orthogonal to each other. Thus, if we denote Z [Rexd a matrix whose row vectors consist of {zi,..., Zu. and W = F F-1, Sd-1 + gd-1 sequential composition of F-1 and F, then formulate the optimization problem for finding an OFS as follows.
z" = arg min
| (W(2) -IAlF
25

---

Algorithm 2 Efficient OFS Generator
Require: A FRS model F I Sd-1 and its inverse F-1, Su-1 I, the size of OFS [d] and a learning rate a E R-00
Ensure: A set of facial images O being an OFS and |O| k. 1: Initialize {21....,
<- u( !) and set matrix Z Rkxd whose i'th row is for i [A]]
2: while Not Converged do
3: Compute O + (2) and ğ t F(O).
4: Compute L 1"2" Ix
5: Update Z +- Normalize(Z
02)
01
6: return F (Z).
where Is denotes the k x k identity matrix and || denotes the Frobenius norm for matrices. We can solve this problem via projected gradient descent with a constraint that each row vector belongs to Sd-1, We select the initial {=1,-..,2) #k random sample from the uniform distribution ((S--2) over the $d-1, which can be implemented by normalizing vectors sampled from the Gaussian distribution (0, J.). For simplicity, we denote Normalize as an operator that normalizes the row vectors of the given matrix to be unit vectors. We summarize the above idea as Algorithm 2
In our experiment for producing Fig. 6 we used the Adam optimizer 42 solve the optimization problem, selecting
0.1. In addition, we terminate the algorithm when it has not converged after 100 updates. We also remark that for large ki e.g., 256, the algorithm may not converge within 100 iterations. We suspect that this is because the face feature vectors would not occupy the whole hypersphere; only a subsphere would correspond to actual faces. There has been some evidence for this phenomenon, such studies on the dimensionality reduction techniques for face templates [22] 21). Nevertheless, our algorithm is sufficient for our purpose, and we leave more analysis on this aspect future work.
D Additional Examples of Attribue-Speccffcc Subspheres
We provide more examples of serribute-specffe feature subspaces for analyzing the validity of Conj.
D.1 More Finer Interpolation Results
Although the example in Fig. 3a is sufficient for our purpose, because of the space limit, the interpolations were done rather coarsely. To complement this, we also provide the 10 10-sized -eribute-specffic subspheres generated from the same algorithm as Sec. 3.3 The visualization result given in Fig. 7 Similar to Fig. 3a we can observe that each image contained in the subsphere still shares the common attribute, while the deviation between adjacent images is reduced because of the finer interpolation.
D.2 Interpolation from Other Attributes
We note that the FairFace dataset VGGFace datasets provide more attributes than experimented with, egg., more races such as Asian or Middle Eastern, attributes about age, or accessories such as glasses or baldness. In this section, provide more interpolation results about them investigate the non-dominaní features satisfying the Conj.
We selected the following attributes from each dataset: In FairFace, selected Asian, Indian, and Latino Hisponc for races, and ages range from 0-9, 20-39, and 50 or older. On the other hand, in VGGFace, we selected accessories having hat, glasses, baldness. We note that the size of all collected images across attributes is more than 9,000. We used the same FR model and its inverse model as the previous experiment.
We provide the facial images corresponding to each subsphere in Fig. 8 Fig and Fig. respec- tively. From these figures, we can observe that each subsphere largely catches the desired attributes. and interpolations between adjacent images seem to be done smoothly. However, for subspheres regarding ages in Fig. 9 we can observe that some images in the range 0-9 would not fit with their
26

---

(a) Male
(b) Female
(c) White
(d) Black
Figure 7: Examples of attribute-ppecifie subspheres with finer interpolation (10 x 10-sized).
(a) Asian
(b) Indian
(c) Latino
Figure 8: Attribute-specife subspheres from more types of races.
27

---

0-9
(b) 20-39
(c) 50+
Figure 9: Altribute-speciffe subspheres from various range of ages.
(a) Glasses Figure 10: Atrrbbue-specific subspheres from various accessories.
(b) Hat
(c) Bald
attribute, though we can observe that the faces seem to get older being placed in right. We guess the reason for this phenomenon is two-fold: first, the FR model and its inverse would not learn much about faces in the 0-9 age range. In fact, provided in the original paper of WebFace42M [8)] which is the training dataset of the Arc2Face model we used, the age distribution of the training dataset concentrated on ages more than 20. Hence, we suspect that the FR model and its inverse model would not be familiar with handling faces within this 0-9 regime.
On the other hand, also provide another interpretation by considering how much the attribute age contributes to extract a discriminative feature in terms of identities. As we can see in the benchmark results of datasets testing the ability of the FR model handle the variation in age, including AgeDB (56) or CALFW [8). recent FR models, including the model in our experiment, achieve these benchmark datasets. That is, we can expect that varying the age would not lead to a huge derivation on the feature vector, and thus failing to form a subsphere because of the collapsing effect of the FR model on faces with the same identity but different ages. From this argument, we further infer that such a phenomenon would occur for other attributes that would not play an important role extracting ddeentttyspeccinn features. As evidence for this, note that similar phenomenon occurs at the subsphere corresponding to hats, while this does not occur for other accessories, e.g., glasses or bald. We think that these factors complexly affected the production of these non-trivial results, and we leave further analyses about them and the effect of these attributes on our attack as interesting future work.
good accuracy
D.3 Subspaces from Other Inverse Models
To show that results in Sec. 3.3 are regardless of the choice of the inverse models, also provide the results of subspaces from other inverse models, including NbNet (54) and the StyleGAN based inverse model proposed by Shahreza and Marcel [66]. For NbNet, we used the same model as F in our experiments. On the other hand, for the inverse model by [66). we utilized their official implementation with a public pre-trained model. The details about the latter model will be found in their original paper. We conducted the same experiments in Sec. 3.3
28

---

The results are given in Fig. and Fig. 12 for each model, respectively. From this figure, we can observe that the subspheres made from NbNet showed the desired result, while almost all faces from [66] were alike to white males. We guess that this is because their inverse model uses unnormalized features as an input, so their inverse model not compatible with metric learning-based FR models using cosine similarity. In fact, we observed that the output image varies as we change the norm of the feature vector, so we multiplied the mean norm of the feature vector for each principal component. In addition, the authors of this inverse model reported that their inverse model struggled to invert facial images from some attributes, e.g., Asian, Black, or oldness. This result also indicates that the capability of the inverse model with respect to the diversity of the generated faces is also crucial for conducting our attack, especially for realizing the selected attribute of the adversarial face.
(a) Male
(b) Female
(c) White
(d) Black
Figure 11: Attribut--ppecifcc subspaces from NbNet.
(a) Male
(b) Female
(c) White
(d) Black
Figure 12: Attribute-spedd specific subspaces from 666
E Transfer Attack (Naive Approach)
An intriguing property of adversarial examples is the transferability, where adversarial examples generated for one local FRS can deceive another target FRS, often with a different architecture. This intriguing property can convert moxite-box&a attacks" to box attacks" that can circumvent the need for detailed knowledge of the target FRS. Similarly, we can expect that the adversarial faces generated by our attack in the white-box setting may deceive another target FRS. The corresponding experimental result on the LFW dataset is illustrated in Fig. 13 Although these transfer attacks show some success rates, they have fundamental limitations unless they do not use information from the target FRS. For example, the FR model trained from a strongly biased dataset suffers from inconsistent accuracy, and cannot expect the transfer attack work well if the target image of the adversarial example we generated is from a long-tailed distribution, as the similarity in the image pair is low. To support this argument, we illustrated histogram of angular distances between original target images and corresponding adversarial images from Fi-! in another target FRS F2 using the FairFace [35] dataset in Fig. [3] which is significantly lower than that using the LFW dataset. Note that while more than 75% of MSIMV3, which is representative public training dataset, LFW datasets, are comprised of Caucasian face images, FairFace is dataset comprised of more than 75% non-Caucasian face images.
Notably, this phenomenon of lower transfer attack rates is not limited to our attack but can also be observed in ad adversarial examples generated by the classical white-box adversarial attack method
29

---

White-box setting (F1)
Black-box setting IF2, Transfer Attack)
UPW
350
LFW
300
FairFace Threshald
Fairface Threshold
Number Samples
250
300
250
200
200
150-
150
100
100
50
oU
70
40
50
60
/0
80
90
Angular Dista nce between Target Image and Adversarial Face Image
Figure 13: Angular distance histograms. An adversarial face is accepted if its angular distance is below the threshold.
Noise Bound Dataset
E= 0.25
0.75
LFW
FairFace
LFW
FairFace
White-box
26.4%
89.9%
99.8%
99.8%
Transfer
6.3%
1.0% (-88.9%)
87.6%
5.3% (-94.5%)
Table 13: Comparison of attack success rates in different settings. Note that we only performed the untargeted attack to ensure fair comparison, since the attack success rate is affected by the starting image in targeted attack. Note that we use as the &2 norm bound for PGD attack with 20 iterations.
based on iterative solver. In Tab. 13 we record the white- box and black-box (transfer) attack success rates from the PGD algorithm using the same two datasets. As with our attack, there are still large gaps regardless of noise bound. That is, occurs because spaces around these difficult samples, and we intend to propose an attack that covers even these diffieult samples. Note that there are other ways to improve transferabilityy such using surrogate models that mimic the behavior of the target system. However, we do not consider such methods because training surrogate model requires a huge number of queries and is impractical when the target FRS is real-world application face API.
FR models do not share similar metric
F Additional Techniques
In this section, we introduce some techniques to proceed with our attack.
Confidence to Cosine Similarity. Our first technique is method to transform confidence scores into cosine similarity scores, which can then be applied to our attack when the target FRS API. A recent study by 43 proposed a method for caleulating confidence scores in FRS by assessing the cosine similarity between two input images. Their method deploys the DOGBOX algorithm [72] to determine the coefficients of logistic sigmoid function g(s) I+e to + b. where L, do. k and b are coefficients that need to be fitted. In [4]), which proposed a reconstruction attack on commercial FRS, they followed the method of 43 We note that they used only true-false image pairs that are different from the images used in actual queries when generating adversarial face images. Instead, we determined the coefficients using the images used in queries. Precisely, we fitted each coefficient by the following objective function:
arg min Σ Σ (s((F(())) -(O,, ((TT E,dg.k O,E0 0,E0
where F I Sd-1 is the local FR model and O is set of facial images defined in Conj. 2
Additional Technique: Correction Matrix. We now present our second technique, called correc- tion matrix, which converts scores to an appropriate form based on Conj. 2 We provide our analysis about why such a transformation is indeed effective. By the definition of pseudo-invers matrix, easily obtain the equality Af A[AAAT
A4æ. If our inverse model F-1 is the exact
30

---

inverse function of F, we obtain (A1AT) I where I& is the k-dimensional identity matrix. This is because y. are top-k principal components of the PCA matrix and then orthogonal to each other for all Vi E [k]. However, since our inverse model F~1 is an approximated version, we multiply the correction matrix R (A1AT)=1 by to obtain more accurate approximation for Conj. Note that Conj. 2 applies to the same $ and transpose matrix A7 of A, its pseudo-inverse A. Thus, here the correction matrix R. transforms to the same 8 in terms of Conj. 2 providing bridge between the FT-1
A.3
ET F
and the final generating term of the white-box attack defined
using the pseudo-inverse, img- PT-
A,
Ayalla
G Addditional Experiments
G.1 White-box Attack Setting
To evaluate the effectiveness of the white-box setting, we first report the attack success rates against the FR model F1. These results are summarized in [ab.[1] As expected, since the adversary has full access to the model architecture and parameters, the attack achieves consistently high success rates, mostly above 90%, with minimum above 80%, across all attributes and inverse models.
Target Dataset
F-1
Dj
FairFace (FF)
LFW
CFP
AGE
TLFW
TCPP
TAGE
FF/Male
95.83
94.31
96.79
93.14
93.37
93.47
FF/Feeale
91.29
89.02
92.41
89.92
90.14
90.14
LA
VGG/White hite
99.74
99.34
99.94
94.07
94.08
94.08
VGG/Black
96.35
95.15
97.03
90.76
90.81
90.83
VGG/Asian
83.45
83.41
80.24
90.22
90.3
90.3
FF/Male
92.25
93.92
94.07
87.83
88.28
88.45
FF/Feemalc
90.44
91.73
93.37
80.85
81.32
81.36
Fin
VGG/White
92.68
91.15
90.39
80.28
80.72
80.8
VGG/Black 97.9
97.74
98.74
87.74
88.94
89,25
VGG/Asian 94.83
95.42
97.1
93.49
94.43
94.72
Table 14: White-box ASR(%) on F1, "FF"
indicates FairFace.
Target Dataset
Prest
Dj
FairFace
L.FW
CFP
AGE
TiFW
TCPP
TAGE
Fair/Male
82.44
87.89
92.68
66.93
76
78.73
Fair/Female
77.14
83.96
90.95
64.66
73.43
75.47
F1
VGG/White
98.59
98.67
99.92
47.14
63.25
67.83
VGG/Black
77.48
90.38
93.59
38.33
52.11
57.02
VGG/Asian
68.03
78.67
78.43
46.98
39.94
64.2
F2
Pair/Male
60.74
74.49
71.93
59.94
69.26
72.22
Fair/Female
53.76
70.83
77.86
52.75
61.96
64.73
FT2
VGG/White
87.68
88.29
90.14
32.73
48
52.62
VGG/Black
60.57
79.14
84.44
28.07
40.67
45.52
VGG/Asian
56.35
76.55
86.19
39.69
54.51
59.52
Fair/Malc
94.25
93.78
96.41
84.31
88.
89.6
Fair/Female
89.21
88.45
92.31
78.28
83.67
85.43
FTA
VGG/White 99.68
99.34
99.94
80.69
87.9
90.01
VGG/Black 95.16
95.04
96.8
67.28
78.84
82.54
VGG/Asian
81.78
82.76
80.13
67.83
78.12
81.76
Fg
Fair/Malc
82.17
88.7
89.26
72.26
79.62
82.39
Fair/Female
75.92
85.06
90.51
62.15
70.17
73.12
Fig
VGG/White
91.68
90.57
90.36
57.2
69.16
73.15
VGG/Black
88.36
93.63
96.85
44.85
62.32
68.75
VGG/Asian
82.01
90.57
95.49
32.68
69.26
76.13
Table 15: White-Bop Transfer Attack Success Rate(%) ()), where F F.F.
To further evaluate transterability of the white-box attack, we performed transfer ättack; however, due to space limitations, the results are included in this section. The test models Fisst for the transfer
31

---

attack are F2 and F3. and the detailed settings for the attack are identical to those described in the main text. The results of the attack success rates are presented in Tab. [5] The results generally demonstrate high success rates. While the success rates are relatively lower compared to the black-box direct attack, which has more information about the test model, they still confirm that the attack retains sufficient transferability to the test models even in scenarios where no prior information is available.
G.2 Effect of Correction Matrix R in Black-box
As mentioned in the main text, we propose an additional technique, the correction matrix, to better refine the scores obtained in the black-box setting the local level. The correction matrix essentially means adjusting the scores obtained from the target model to make them more suitable for local use, which can be thought of as helping the adversarial face to align with the same identity as the target image. However, since the ASR used in the main text excludes images with the intended attribute from the selection of target images and also considers whether the attack results reflect the intended attribute when determining the success of the attack, it may not be the most suitable metric for analyzing the effect of R. Therefore, to analyze the effect of R, we define the Identity Matching Rate (IMR) as follows:
E.S 1(Steat (img,; img,) 2 Trest
IMR
[
where I represents the set of total images, img, I refers to each individual target image, and 1()) a function mapping if the input statement is 1, and 0 otherwise. Img, is an adversarial face generated by an attack algorithm. Searger (Img1, lmg2) gives a cosine similarity score (resp. confidence score) between Img, and Img2 from the open-source (resp. commercial) FRS. If the score exceeds the predefined threshold Ttarget of the test FRS, Img, and Img, are considered as the same identity in the target FRS.
We performed the black-box attack with the same setup mentioned the main text. We then compared the IMR before and after applying R and marked the results in Tab. [6] if the IMR increased after applying R, it is highlighted in blue, and if decreased, it is marked in red. As shown the table, R had positive impact on identity matching in most cases. The results in this table also reflected in the black-box direct attack success rate presented in Tab, of the main text, with only the areas where the effect of R has negative impact highlighted in blue.
Target Dataset
T
F
Dj
FairFace
LFW
CFP
AGE
7.F
TCPP
TAGE
Pair/Male
+1.20
+0.17
+0.11
+1.25
+0.32
+0.12
Fair/Female
+1.96
+0.36
+0.04
+0.75
+0.36
+0.23
FTA
VGG/White
+0.53
+0.07
+0.02
+1.57
+0.41
+0.22
VGG/Black
+0.09
0
+0.01
-0.06
-0.07
-0.06
VGG/Asian
+0.75
+0.20
+0.04
+0.54
+0.14
+0.17
F2
Fair/Malc
+0.88
+0.67
+0.67
+2.40
+1.39
+1.02
emalee
+2.12
+1.94
+1.76
+3.47
+1.64
+1.40
Fin
VGG/White
+0.97
+0.55
+0.13
+1.94
+0.61
+0.37
VGG/Black
+0.77
+0.46
+0.26
+1.64
+1.00
+0.74
VGG/Asian
+3.60
+2.07
+1.73
+2.81
+1.78
+1.22
Fair/Male
+0.03
0
+0.03
-0.02
-0.02
Fair/Temale
+0.19
-0.04
+0.10
-0.30
+0.02
+0.01
FT1
VGG/White
+0.02
0
0
+0.01
+0.02
+0.01
VGG/Black
-0.06
0
0
-0.05
-0.02
-0.01
VGG/Asian
+0.01
-0.14
0
-0.15
+0.01
F3
Fair/Male
+1.46
+0.73
+0.49
-0.41
-0.44
-0.34
Fair/Female
+0.30
+1.42
+0.82
+0.77
+0.49
+0.59
F
VGG/White
+0.41
+0.16
+1.10
+0.37
+0.28
VGG/Black
+1.80
+0.28
+0.50
+0.73
+0.73
+0.57
VGG/Asian
+2.79
+2.06
+1.31
43.55
+1.66
+0.81
Table 16: Change in IMR after applying correction matrix It: blue for increase, red for decrease.
32

---

In addition to the analysis of R, we provide a table to illustrate how well our proposed attack performs in terms of identity matching alone. The shared attack settings include the identity matching rates for the white-hox direct attack and the black-box direct attack, with the detailed settings identical to those mentioned in the main text. The results are presented Tab. 17 and Tab. T8) respectively, in sequential order. As shown in the numbers within the tables, our method achieves excellent IMRs.
Target Dataset
F-1
Dj
FairFace
LFW
CFP
AGE
F.
TCFP
TAGE
Fair/male
99.95
100
100
99.71
99.93
99.98
Fair/female
99.87
100
100
99.61
99.94
99.99
FIA
VGG/White
99.99
99.98
100
99.62
99.87
99.97
VGG/Black 99.97
100
100
99.55
99.86
99.95
VGG/Asian
100
100
100
99.87
99.98
99.98
Fair/male
98.58
99.52
99.84
99.09
99.65
99.88
Fair/Temale
96.08
99.03
99.88
98.96
99.63
99.86
Fin
VGG/White 99.87
99.85
99.98
98.79
99.58
99.78
VGG/Black 99.1
99.66
99.99
96.94
98.79
99.37
VGG/Asian 98.91
99.6
99.99
98.6
99.57
99.86
Table 17: White-Box Direct Attack IMR( %)).
Target Dataset
T
F-1
D
FairFace
LFW
CFP
AGE
TLFW
TCFP
TAGE
Fair/Male 98.53
99.71
99,96
98.52
99.7
99.7
Fair/Female
96.69
99.59
99.82
97.64
99.42
99.64
FT1
VGG/White 99.94
99.96
100
98.94
99.68
99.75
VGG/Black 99.49
99.94
100
99.19
99.84
99.89
VGG/Asian
98.3
99.94
99.99
98.5
99.54
99.78
F2
Fair/Male
79.02
92.89
95.37
82.79
94.31
96.1
Fair/Female
63.67
84.8
90.69
77.01
90.6
93.6
F
1s
VGG/White 87.39
94.74
98.54
76.1
91.6
94.23
VGG/Black 87.88
96.23
97.78
88.38 33
96.92
98.17
VGG/Asian 80.13
94.11
97.33
88.67
96. 78
97.96
Fair/Male
99.62
99.99
99.98
99.85
99.97
99.97
Fair/Female
98.55
99.75
100
99.4
99.97
100
F1A
VGG/White 100
100
99.92
VGG/Black 99.68
99.99
100
99.79
99.97
99.99
VGG/Asian 99.29
99.84
100
99.7
99.99
100
F3
Fair/Male 84.91
94.84
96.24
89.64
96.67
98.26
Fair/Female
65.93
83.33
93.4
81.22
93.16
96.34
Fin
VGG/White 94.24
97.7
99.51
91.77
97.75
99.01
VGG/Black
88.02
96
98
89.05
97.38
98.79 79
VGG/Asian 78.64
92.78
97.75
87.61
96.58
98.35
Table 18: Black-Boo Box Direct Attack IMR(%).
G.3 Black-box Attack against target FRS FT
In Tab. 19 we provide the ASR on commercial FRS FT using gender-attributed D,, Similar to ASR on FA in Tab. 3 we note that the ASR related to the FairFace dataset is significantly larger than the transfer attack. However, since the decrease in ASR obvious for the LFF dataset, we leave analyzing and improving this phenomenon
future topic.
G.4 Experimental Setup for Table and Table
We base our comparison on 633, which follows the experimental protocol CLIP2Protect [68 Specifically, 500 subjects are selected from the CelebA-HQ dataset (36), each with pair of facial images. In their setup, one image from each pair is used to generate adversarial examples (training set), and the other is used to evaluate the attack success rate (test set). In our comparison, used the same
33

---

with R
without
F
Target
Dy
0.99
0.8
(1.99
Transfer Attack without Queries (The images from Table
were used.)
VGG/White
N/A
91.56
61.78
LFW
VGG/Black
49.79
10.00
VGG/Asian
N/A
45,71
9,98
FT
F
VGG/White
N/A
0
0
FairFace
VGG/Black
N/A
0.02
0
VGG/Asian
N/A
Direct Attack with Score Queries
VGG/White 81.78
23.33
27.11
2.22
LFW
VGG/BBlack 75.26
10.74
20.84
2.84
VGG/Asian
56.26
4.41
13.23
0
FT
FTA
VGG/White
72.87
12.90
17.55
0.66
FairFace
VGG/Black
58.97
13.24
7.55
0.44
VGG/Astan
62.90
7:37
12.78
0.49
Table 19: Black-box ASR on FT using local FR model Fi
500 subjects and adopted the same data split. For the method of (63] we used our three open-source face recognition models (F1, F2, Fs) as surrogate models generate adversarial examples, consistent with the original protocol which involves multiple surrogate networks. However, our method differs fundamentally that does not require a source image. Instead, we project feature vectors into attribute-specific subspheres (e.g., gender, race) to generate adversarial faces. Therefore, for each training image, we created one adversarial face per attribute category and reported the attribute-wise transfer attack success rates against AWS ComparcFace, shown in Tab.
To ensure fair comparison. we also adapted the method of to our scenario by randomly sampling 500 images from the "100k Faces Generated by Al" dataset 61 as source images. Each source image was targeted toward a corresponding identity from the training set, and the adversarial faces were evaluated by querying AWS CompareFace. Importantly, unlike [63], our method supports attacks directly through actual API queries. We thus separately report our query-based black-box attack
63
Ours
Target Image
Source
Result
Transler
Direct
0.0028
0.0147
0.7835
0.9635
0.01TS
0.4349
0.5410
0.9755
scores
0.0052
0.0267
0.1366
0.9834
Table 20: Additional examples for Tab. using male, black, asian attributed subspheres, respectively.
34

---

success rates in Tab. 4 in addition to the transfer-based results. Furthermore, the evaluation against the set, following the full protocol of 633, is included in Tab. 21 for completeness.
Method
663
Ours (Transfer attack without queries)
Ours (Attack with 100 queries against AWS)
f
661
Male
Female
White
Black
Asian
Male
Female
White
Black
Asian
= 0.8
29.86
23.80
33.20
62.80
22.20
35.60
94.20
92.00
98.20
99.20
98.80
T= 0.99
3.41
1.20
1.60
6.60
1.40
1.80
14.60
8.60
41.00
40.80
24.60
Table 21: ASR of [63] and ours evaluated on CelebA-HQ dataset [36], evaluated against test set
images under different thresholds
G.5 Black-box Attack on Non-facial Target
We conducted our proposed attack in a black-box setting on non-facial images. Specifically, we used the CIFAR-10 10 and Flower- 102 test datasets and 10,000 randomly generated pixel images as the attack targets. CIFAR-10 is widely used benchmark dataset for image classification, consisting of 10 categories, such as airplanes, cars, and animals. On the other hand, Flower- 102 contains 102 categories of flowers with varying levels of visual complexity. In our black-box attack framework, Fi was employed as the local model,
F2 and F3 served the target models. The inverse models used were both of FiA and Fig! The ASR of the attack under the aforementioned settings was presented in Tab. [2 Since the target datasets selected are not verification datasets, the data is analyzed based the threshold values derived from the best accuracy of the F2 model on LFW, CFP-FP, and AgeDB. As evidenced by the attack success rates in the table, it was confirmed that the attack remains effective even when the target images are non-facial, but ASR is relatively lower compared to when the target image is facial images. Therefore, there is room for improvement in terms of extension broader target model, such as image classification models, of attack or non-facial adversarial examples on commercial systems.
G.6 Ablation Study about F-1 in Black-box
In our black-box algorithm, the inverse model for the local face recognition system is used in two distinct stages: once during preprocessing and the other once during the execution of the algorithm. Therefore, we conducted an ablation study using Fi, face recognition model with inverse models, F-11a and F-11g- B- The results of this study are shared in Tab. 26
G.7 Transfer Attack in Black-box
We confirmed that even in our black-box setting, the attack retains sufficient transferability to other test models that are not the target model. When the target model was F2, F3 was selected as the test model, and conversely, when the target model was Fs. F2 was used as the test model for the
Threshold :T
F-1
Target Dataset
TLFW
TCFP
TAGE
10
20.1
39.84
45.78
FT1
Flower- 102
15.8
31.87
37.9
Random
74.91
79.97
80.2
F2
CIFAR-10
16.4
39.55
49.76
FT1
Flower- 102
11.57
32.42
40.67
Random
61.22
91.16
94.82
CIFAR-TO
51.73
66.77
70.8
FiA
Flower- 102
58.35
63.83
64.62
Random
76.9
76.9
76.9
F3
10
32.92
63.85
76.33
FT2
Flower- I02
51.14
77.89
86.01
Random
92.47
99.19
99.7
Table 22: Attack Success Rate(%) of Black-box Direct Attack on Non-facial Target Images D, uses VGG/Black, and the ASR calculated based on the 7 values of Fz corresponding to each column.
35

---

Target Image
CIFAR-10
Scores
0,3289
0,2836
0,4439
0.3257
Flower- 102
Scores
0.3361
0.3140
0.3452
0.2742
Random
Scores
0.3649
0.3535
0.3698
0.3090
Table 23: Black-box attack on non-facial images
attack. All other settings for the black-box attack are identical to those described in the main text. Additionally, since an ablation study on the use of inverse models, as discussed in Sec. G.6) is also feasible in this setting, extracted the ASR results and included them in Tab. 27
G.8 Prototype of FRS for Strict Threshold
To construct prototype FRS with tight acceptance threshold, we designed loss function with two key modifications to the original ArcFace [16] loss LA. The loss function LA is defined as follows:
e cos(0 +m)
(x W)
log e (fga
+ Σ
Nia 25=1 e 0n
where gl is the index of the ground-truth, m is the angular margin term, and is the scaling factor.
First, instead of applying the angular margin only the target class in the original ArcFace, we apply an inverted margin to non-target classes as well. Specifically, for the target class, we use the standard cos + m). while for all other classes, use cos m). This contrastive strategy enhances intra-class compactness by pulling feature vectors of the same identity closer together, while still maintaining sufficient inter-class separation.
Second, rather than computing cosine similarity over the entire 51 512-dimensional feature vector, randomly split into two 256-dimensional subspaces in each batch and calculate the cosine similarity separately in each subspace. This regularization strategy prevents the dominance of specific dimensions and promotes a more uniform distribution of information across all feature dimensions. As result, the model learns to generate feature vectors that yield higher overall similarity between samples of the same identity. To formalize this idea, we define cosine similarity over each randomly selected subspace as follows. Let I, E Rd. Let L,, In C {1...., d} be two randomly sampled, disjoint subsets of indices such that 11l [22 d/2. Then, the partial cosine similarities is defined as follows:
(F1, Ur.)
cos2(1, 2) (1 V12)
cos (x, y)
|x,,
[1
|
|
where 21, and yr, denote the subvectors of and corresponding to the index set 4, respectively.
36

---

Then, for 1st (resp. 2nd) partial cosine similarities of j'th identity of (resp. 0f final loss function Lp for our prototype FRS is defined as follows:
cos(0 +m)
(x(x, W)
Σ log
e 00 ge +m)
Nia j=ll e c00( -m)
+
where ge is the index of the ground-truth, m is the angular margin term, and s is the scaling factor.
These two modifications lead to significantly improved intra-class compactness, as shown in Fig. [4] where the decision threshold of the prototype FRS is formed around 0.3, which is higher than that of standard open-source models (typically around 0.2). This higher threshold increases the overall system's strictness against false accept. In Table 6 we report the ASR against the prototype FRS on the LFW dataset, alongside the results for other target models, F2 and F3.
Number of Pairs
False Pair
125
False Peir Thrnshood
False Pair Threshoid Threshold
200
False Pair
100-
Thinshold
100
Threshold
Thrrshold
Threshooc
80
Threshold
75-
0.24
60-
0:22
100
2993
0.1035
40
25
0.00
0.25
0.50
0.75
1.00
-0.25
0.00
0.25
0.50
0.75
3.00
0.00
0.25
0.50
0.75
1.00
9625
0.00
0.25
0.50
0.75
1.50
Fp
Figure 14: Cosine similarity histograms. The first three plots correspond to the open-source FRSs used in our experiments, whose thresholds are typically formed around 0.2. In contrast, our prototype FRS exhibits a significantly higher threshold, forming around 0.3 to 0.4. The blue line indicates the threshold at FAR = 0.1%, while the red line marks the threshold corresponding to the best accuracy.
G.9 Adversarial Training and Certifiably Robust Face Recognition Models
In the literature of FR, several efforts have been made to achieve robustness against adversarial examples. Likewise to image classification, adversarial training (53) has been shown to be effective in defending against adversarial examples based on perturbations [8)) addition, recently, Paik et al. [58] proposed certifiably robust FR model against perturbation-basee adversarial attacks. More precisely, they derived an upper bound of the magnitude of the adversarial perturbation that does not change the decision of the FR model. Since the proposed adversarial face generation method can be considered as adversarial examples, we conduct additional experiments to assess the robustness of the aforementioned (certifiably) robust FR models against our attack.
For adversarially trained FR models, we utilize the RobFR library that provides various pre-trained FR models with adversarial training, c.g., PGD-AT [53] and TRADES 186). Among these models, use two FR models using IResNet50 as the backbone and trained with CASIA- CASIA-Webface [81] dataset, using ArcFace [16] and CosFace 77 loss functions, respectively. For the certifiably robust FR model, we used the pre-trained FR model provided by Paik et al. in their oofficial implementation, which uses custom 22-layer convolution neural network backbone, ElasticFace -Coee 3 loss function, and trained with the MSIMV3 [5] dataset. Detailed settings can be found in their original papers or their source codes.
We evaluate the ASRs of our attack against the above FR models under the same setting as in Sec. In particular, for the comparison with the zeroth-order optimization (7OO) based attack, we also provide the ASRs in the context of impersonation. More precisely, our implementation is based on Carlini & Wagner's 's white-box attack framework [6] with ADAM zeroth-order optimizer using batch size 128 with 20,000 iterations on one of the facial images. The goal is to make the image recognized the same identity a different person. We consider the attack successful if at least one perturbed image causes a successful impersonation.
The results are provided in Tab. 24 From this table, we can observe that, for the ZOO, all the adversarially trained FR models show low ASRs less than 4%, whereas the model from Paik et al. shows the ASR of at most 25%. On the other hand, for the proposed attack, we observe that the model by Paik al. is vulnerable to our attack, whereas other adversarially trained models successfully defend against it. We hypothesize that such dramatic difference is derived from two aspects: first, as
37

---

observed in the decision thresholds (T) of each target model, the threshold for adversarially trained models is substantially higher than those of Paik et al.'s FR model. In particular. the threshold for the latter is almost the same as that of non-robust FR models in the main text. From this, we can infer that the distribution of similarity scores from adversarially trained FR models significantly far from that from the adversary's 's local model. Hence, the scores from queries would interrupt the interpolation over the attribute subsphere. On the other hand, when attacking Paik et al.'s FR model, Conj. 2 would be valid because of its similar decision threshold to that of the adversary's 's local model, thus succeeding the attack. However, is important to note that, despite the success of the ZOO-based attack, the average of L2 norm of these examples ranges up 9, which is significantly exceeds the typical bounds adversarial examples.
To further analyze our hypothesis, we also evaluated the ASRs from the transfer attack based on our attack, whose results are provided in Table (5 particular, we considered two settings of the thresholds at FAR=0.1 % and the best accuracy. From this table, we can observe that the transfer attack shows higher ASR than the black box setting for adversarially trained FR models, whereas the opposite tendency appears for Paik et al.'s model. This result supports our hypothesis discussed above; such a difference in ASRs indicates whether the adversary can make use the queried scores for crafting adversarial face or not.
Note that direct comparison between ZOO and the our attack may be unfair because of the difference in the setting; for the former, the adversary adds perturbation to one of the given pairs of images, ie.. the source image, whereas there is no such source image in the latter. Nevertheless, our attack reveals the potential vulnerability of these FR models, even in certifiably robust ones, in practical setting. We leave the mitigation of our attack, along with an in-depth analysis on the relationship between our attack and prior perturbation-hanl based attacks, as interesting yet important future work.
ZOO
7
Ours
Target Model
TAR(%)
7
ASR (%)
Avg. L2
Male
Female
White
Black
Asian
PGD-Are
38.88
0,590
2.27
8.96
0
PGD-Cos
28.47
0.484
1.60
8.69
0
0
0
0
0
Trades-Are
12.84
0.918
1.67
5.79
0
0
0
Trades-Cos
51.90
0.768
3.73
5.77
0
Paik et al. [58]
83.97
0.231
25.53
6.71
79.62
76.92
91.18
75.65 78.82
Table 24: ASR(%) against adversarially trained models [80] and certifiably robust model [58] on the
LFW dataset. The FAR is set to 1e-3. Reconstructions are obtained using Fiz as the inverse model.
Ours
Target Model
TAR(%)
FAR(%)
Male
Female
White
Black
Asian
PGD-Are
38.88
0.590
0.46
0.88
3.95
0.62
1.19
PGD-Cos
28.47
0.484
0.15
0.47
1.19
0.55
0.57
Trades-Are
12.84
0.1
0.918
2.58
2.09
12.20
0.88
1.54
Trades-Cor Cos
51.90
0.768
0,76
0.43
2.18
1.13
0.62
Park et al. [58]
83.97
0.231
14.96
15.31
39.34
11.60
15.11
PGD-Are
84.23
7.23
0.357
9.26
13.94
39.65
17.06
19.99
PGD-Cos
84.03
9.67
0,260
6.91
7.20
22.36
10.54
10.48
Trades-Are
59.53
16.63
0.785
63.02
58.55
82.55
47.10
50.85
Trades-Con
88.46
8.67
0.678
6.61
5.32
19.66
11.23
4.14
Palk et [58]
94.80
3.2
0.165
35.31
39.50
67.31
31.71
37.20
Table 25: Transfer ASR(% (%) against ad adversarially trained models [80] and certifiably robust model [58)
on the LFW dataset, The FARs are set by both its value at FAR Ie-3, and by its value at the point of best accuracy, which is determined according to the LFW dataset evaluation protocol. Reconstructions are obtained using 1 as the inverse model.
38

---

F-1
Target Dataset
D5
FairFace
Pre
Alg
LFW
CFP
AGE
7LFW
7CPP
TAGE
Fair/Male
97.29
99.41
99.18
97.25
96.3
97.93
Fair/Female
93.33
96.02
97.44
93.26
94.16
93.75
PIA
VGG/White
99.94
99.95
100
99.92
99.78
99.98
VGG/Black 87.98
90.61
88.66 66
84.82
86.23
86.42
VGG/Astan
73.82
73.09
72.27
68.48
68.07
67.45
FIA
Fair/Malc
79.28
91.17
94.24
85.67
90.8
93.93
Fair/Female
73.38
88.31
91.36
75.91
86.93
92.88
PT-1
(Gt/
95.31
96.59
97.06
95.89
95.48
96.55
VGG/Black
91.16
96.16
96.94
91.52
96.07
97.03
VGG/Asian
82.77
90.4
91.93
81.23
89.08
90.17
F2
Fair/Male
90.45
97.87
99.07
93.19
96.61
97.91
Fair/Female
77.88
93.38
96.46
85.88
93.65
96.5
FTA
VGG/White 95.98
98.92
99.88
99.46
99.69
99.91
VGG/Black 85.42
89.9
87.96
85.25
86.38
85.17
VGG/Asian
61.76
66.27
71.31
58.62
63.92
64.6
Fia
Fair/Male
70.82
86.66
91.16
75.59
88. 14
90.53
Fair/Female
58.48
80.25
86.2
60.09
78.5
88.32
Pin 1
VGG/White
84.42
90.43
94.84
90.16
92.93
95.09
VGG/Black
85.1
93.68
94.97
83.45
92.53
94.38
VGG/Asian
73.97
84.9
89.81
70.66
81.57
86.4
Fair/Male
98.28
99.28
99.42
97.29
97.44
97.44
Fair/Female
94.84
96.63
96.75
95.65
96.24
96.27
PTA
VGG/Whit 98.9
99.68
99.75
99.61
99.7
99.7
VGG/Black 92.12
92.63
92.68
79.47
79.61
79.63
VGG/Asian 78.91
79.53
79.63
76.99
77.22
77.22
FIA
Fair/Male
91.79
97.97
98.84
91.53
94.17
94.54
Fair/Female
86.07
94.39
95.79
86.69
91.06
92.14
FT
VGG/White hite
89.58
97.25
98.16
89.27
90.71
90.97
VGG/Black
87.99
91.51
91.87
90.65
93.73
94.1
VGG/Asian 73.86
77.57
77.97
86.42
89.96
90.6
Fs
Fair/Male
87.58
94.46
95.31
96.98
98.37
98.51
Fair/Female 84.43
91.56
93.06
93.47
97.27
98.07
PTA
VGG/White 90.28
95.92
96.63
97.94
99.24
99.32
VGG/Black 91.75
96.34
96.78
82.38
83.98
83
VGG/Asian
89.19
93.44
94.02
74.83
77.22
77.64
Fi3
Fair/Male
81.31
91.94
93.63
85.76
93.1
94.65
Fair/Hemale
75.21
88.31
91.02
77.09
88
90.81
F
VGG/White
73.32
87.96
90.49
84.23
89.77
90.93
VGG/Black
86.95
95.23
96.38
86.77
93.74
95.16
VGG/Asian
83.69
90.95
91.98
82.61
90.01
91.56
Table 26: Ablation study (S) for the use of inverse models during preprocessing and algorithm execution in the black-box setting.
39

---

F-1
Target Dataset
T
Fiest
Dj
FairFace
Pre
Alg
LFW
CFP
AGE
TLFW
TCFP
TAGE
Fair/Male
64.57
78.92
81.48
28.03
42.02
49.5
Fair/Female
56.73
74.93
89.59
24.69
38.5
46.13
IA
VGG/White
95.54
96.58
99.7
17.88
32.24
40.67
VGG/Black
59.74
77.61
77.96
13.95
23.81
29.93
VGG/Asian
45.37
58.77
66.81
13.51
23.14
29
FiA
Fair/Male
37.08
50.21
53.07
13.89
26.66
33.18
Fair/Female
27.13
46.12
63.99
12.95
23.24
30.02
Fi
VGG/White
83.87
86.65
95.3
9.07
20.49
27.52
VGG/Black
42.01
63.19
66.83
6.5
13.6
18.89
VGG/Asian
30.1
51.15
68.85
7.7
15.56
21.6
F2
F'
Fair/Male
49.87
66.72
69.04
18.62
31
38.84
Fair/Female
34.71
57.85
72.93
14.87
26.92
34.41
FiA
VGG/White
83.07
87.76
98.59
9.61
21.67
29.83
VGG/Black 45.85
67.53
70.1
9.13
17.62
23.11
VGG/Asian
26.27
42.17
57.17
8.4
16.71
21.94
Fig
Fair/Male
25.57
37.06
41.4
10.02
18.38
24.33
Fair/Female
16.47
31.96
46.47
7.99
16.37
22.01
Fin
VGG//hite
57.34
66.49
86.26
4.14
10.61
15.9
VGG/Black 27.9
49.82
56.37
4.04
9.54
13.55
VGG/Asian
19.88
37.53
54.4
5,53
12.1
17.25
Fair/Male
66.64
81.08
80.19
26.97
39.98
44.61
Fair/Female
58.75
77.89
84.85
22.7
36.12
40.68
F
VGG/White
95,44
96.69
99.61
13.2
25.25
30.45
VGG/Black 57.04
73.79
74.4
11.32
18.98
22.09
VGG/Asian
43.13
57.56
62.14
11.1
20.33
23.92
FiA
Fair/Male
37.68
50.97
54.44
19.47
31.58
35.88
Fair/Female
33.75
53.53
64.54
18.34
29.92
34.63
F
VGG/White hite
84.05
84.83
93.92
8.64
18.74
22.9
VGG/Black
43.55
64.9
67.75
8.28
15.57
18.93
VGG/Asian 35.44
56.99
68.42
9.7
18.79
22.47
F3
Fs
Fair/Male
53.25
68.24
68.97
17.73
29.06
33.61
Fair/Female
39.36
62.68
73.98
13.42
23.29
27.43
PIA
VGG/White 85.44
88.88
98.88
7.63
16.87
21.41
VGG/Black 44.04
64.78
67.27
7.54
14.17
17.1
VGG/Asian
26.62
42.57
50.38
7.21
14.6
17.45
Fip1
Fair/Male
27.72
40.31
40.29
11.49
21.74
25.86
Fair/Female
19.79
36.45
47.89
10.33
19.54
23.2
F-1
VGG/White
63.13
70.14
86.12
4.72
11.67
15.14
VGG/Black 30.35
50.27
56.11
5.58
11.58
14.73
VGG/Asian
23.06
42.5
52.56
7.2
14.97
18.43
Table 27: Black Box Transfer ASR(%).
40


Reviews:
[{"review_id": "DYd1WSzykM", "reviewer": "Reviewer_osyU", "summary": "This paper proposes a novel non-adaptive, non-iterative black-box attack against face recognition systems, which constructs adversarial faces using a single batch query. The key insight is to exploit the geometric structure of deep face embeddings by identifying attribute-specific subspheres in the feature space. By projecting onto these subspheres and reconstructing faces through inverse models, the attack achieves high success rates against commercial APIs with minimal queries.", "questions": "-\tThe proposed subspace structure appears to be an empirically justified but heuristic assumption. Is there a more principled or data-driven way to discover alternative or more precise subspace structures that could potentially enhance attack effectiveness?\n-\tHow well does the proposed method generalize to other biometric modalities such as gait or iris recognition? The paper focuses on face recognition, but the core idea might extend further if the subspace structure exists similarly in other domains.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n-\tAdversarial face generation is an important topic worthy of further investigation.\n-\tThe non-iterative attack is efficient and practical for black-box FR systems.\n-\tThis paper well connects theoretical supports and empirical validations.\n-\tReal-world demonstration convincingly highlights the societal impact of the proposed attack.\n\n**Weaknesses**\n\n-\tMore ablation studies on the projection quality across different inverse models could testify to the attack’s generality on inverse model choices.", "was_revised": true, "final_justification": "I appreciate the authors' response. After considering their rebuttal and the feedback from my fellow reviewers, I have decided to maintain my original score.", "rebuttal": "### **Response to Weakness (Projection Quality across Different Inverse Models):**\n\nThank you for pointing this out. Among the inverse models we experimented with, the well-trained ones—NbNet [50] and Arc2Face [54]—consistently generate high-quality reconstructions that accurately preserve the intended semantic attributes (see Figures 3(a), 7, and 11). In contrast, the inflexible and biased inverse model from [61], which relies on a fixed pre-trained StyleGAN, often fails to reflect the desired attributes, producing reconstructions that are systematically biased toward young white male faces (Figure 12). Notably, this issue is also acknowledged by the authors of [61].\n\nPrecisely, all face feature vectors are extracted from the same recognition model, and each PCA-based projection matrix is computed using samples corresponding to a specific attribute (e.g., Male, Female, etc.). The resulting projected feature vectors—interpretable as adversarial features that remain close to the original identities but reflect different attributes—are obtained independently of the inverse model. Therefore, the *projection quality* primarily depends on how well the attributed subspheres are constructed, which in turn is influenced by the quality of attribute annotations in the attributed image dataset. This is evidenced by the successful reconstructions using Arc2Face and NbNet, where the reconstructed images consistently preserve the intended target attribute.\n\nHowever, when using the inverse model from \\[61], we observe that reconstruction fails to reflect the intended attribute and is biased toward young white male faces. This issue stems from limitations in the inverse model itself, not from the projection quality. As such, although the projection step operates correctly, the final *reconstruction quality* is adversely affected by the inverse model's biases—potentially impacting the overall attack success rate.\n\nWe will reflect this analysis more clearly in the final version of the paper and explicitly state that inverse model choice can affect the perceptual quality of generated adversarial faces, even when the projection is attribute-successful. Furthermore, we commit to including more qualitative examples of adversarial faces in the appendix—not only interpolations within principal components but also real-case adversarial samples per attribute, similar to Table 1.\n\n\n### **Response to Question 1 (Subspace Structure – Heuristic vs. Principled Approach):**\n\nThank you for this insightful question. While we report quantitative attack success rates only for gender and race attributes in the main paper, we also conducted the same principal component interpolation analysis for other attributes such as age, eyeglasses, baldness, and wearing a hat to qualitatively examine whether the semantic structure of the attribute-specific subspaces is preserved. As shown in Figures 9 and 10 (Appendix D), certain attributes like *baldness* exhibit consistent semantic variation along principal directions, suggesting the presence of a coherent subspace. In contrast, attributes like *wearing a hat* demonstrate only partial or inconsistent presence of the intended semantic attribute, indicating a weaker or noisier subspace.\n\n\nSeveral factors may contribute to this issue: noisy labels, insufficient sample size per attribute, or, in the worst case, the non-existence of a coherent attribute-specific subspace for certain attributes. Among these, we empirically demonstrate that increasing the number of attribute samples improves attack performance. In particular, we include results (see the following table and to be added in the final version) showing that using a larger set of attribute samples—e.g., constructing the gender-specific subspaces using CelebA [a] or VGGFace2 dataset instead of FairFace—leads to a consistent increase in attack success rates.\n\n| Attribute | Source Dataset | Number of Attributed Samples   | Attack Success Rate on $F_2$          | Attack Success Rate on $F_3$          |\n|:-----------:|:----------------:|:-------------:|:-------------------------:|-------------------------:|\n| Male      | FairFace       | 45,986      | 97.29% (Table 2)         | 98.28% (Table 23)        |\n| Female    | FairFace       | 40,758      | 93.33% (Table 2)         | 94.84% (Table 23)        |\n| Male      | CelebA        | 84,434       | 99.37% (+2.08)           | 98.48% (+0.20)           |\n| Female    | CelebA        | 118,165   | 99.53% (+6.20)           | 99.45% (+4.61)           |\n| Male      | VGGFace        | 1,958,913   | 99.48% (+2.19)           | 99.24% (+0.96)           |\n| Female    | VGGFace        | 1,349,127   | 99.77% (+6.44)           | 99.86% (+5.02)           |\n\n**Table A: Effect of Attribute Sample Size on Attack Success Rate (ASR)**\n\nTo further investigate which attributes are more likely to yield coherent subspaces, we also conducted an auxiliary experiment (see our response to Reviewer VhZn – Question & Weakness 1) that compares the similarity between features projected onto subspaces constructed from samples with and without a given attribute. This analysis provides additional insights into the intrinsic structure of each attribute in the feature space. Nevertheless, we acknowledge that our current approach is not a fundamentally data-driven solution and still relies on manually defined attributes. To overcome this heuristic assumption, we believe that future work could explore unsupervised or weakly-supervised techniques—such as clustering methods (e.g., K-means)—to identify more intrinsic and potentially effective subspace structures for adversarial generation.  \n\n\n### **Response to Question 2 (Generalization to Other Biometric Modalities):**\n\nThank you for the thoughtful question. As briefly mentioned in the introduction, we believe that extending our framework to other biometric modalities—such as voice—is a natural direction, especially for systems trained using metric learning approaches that enforce intra-class compactness and inter-class separability (e.g., feature vectors on a hypersphere). Such structural properties may allow the formation of attributed-specific subspaces in those domains as well.\n\nHowever, several challenges remain. As discussed in our response to weakness point, one key requirement for our method is the availability of a well-trained inverse model. While inverse models for face recognition are well-established—partly due to the standardized and cropped input format—learning reliable inverse models for modalities like voice is more difficult due to significant variations in input length and content. Similarly, gait recognition may face similar challenges due to the variable format of video sequences, which could make achieving high-quality reconstructions from projected feature vectors considerably harder, potentially hindering practical attack implementation.\n\nA second limitation concerns the availability and definition of meaningful attributes. As discussed in our response to Question 1, our framework relies on the ability to define attribute-specific subspaces based on semantically meaningful groupings such as gender or race. While such attributes are well-established and publicly annotated in the face domain, other modalities such as gait or iris may lack clearly defined, human-recognizable attributes that serve this purpose. Even when such attributes exist, collecting large-scale datasets with accurate annotations and sufficient intra-attribute variation remains a major obstacle. This issue further emphasizes the challenge of generalizing our method across modalities, where the subspace structure may be harder to define or less informative.\n\nNonetheless, we believe that extending our method to other modalities is a compelling direction for future work, especially with the advancement of multimodal systems and generative (inverse) models across biometric domains.\n\n[a] Liu, Ziwei, et al. \"Deep learning face attributes in the wild.\" Proceedings of the IEEE international conference on computer vision. 2015."}, {"review_id": "u6O6ZJLJob", "reviewer": "Reviewer_phZN", "summary": "This paper presents a method for generating adversarial facial images that can impersonate a target identity in face recognition systems (FRSs), even under black-box constraints. Unlike existing approaches that rely on iterative optimization or surrogate models, the proposed approach exploits the geometric structure of FRS feature spaces. Based on empirical observations that individuals with similar attributes (e.g., race or gender) cluster into specific subspaces, the paper uses projections onto these subspaces to generate adversarial images efficiently. The proposed method achieves over 93% attack success rate ono commercial APIs like AWS CompareFaces at default thresholds.", "questions": "Are there other subspace structures that would be more robust? Have the authors considered sparse representations such as those discussed in the 2009 PAMI paper by Wright et al. ? The authors point to the shortcomings of GAN-based methods. Have the authors considered the effectiveness of diffusion models?", "limitations": "Section 5.3 discusses the limitations of the proposed method.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe attack remains effective even without access to gradients or the internals of the FRS, marking it as non-adaptive and non-iterative. Impressive attack success rate on commercial APIs and dating websites even under black-box attack scenarios. \nWeaknesses: Using PCA dates the work back to the nineties. PCA representations are sentitive to illumination changes, rotation etc. What effcet will these have on attack performance? The paper is written in a style that makes it very difficult to comprehend. Too much notation for describing obvious concepts.", "was_revised": true, "final_justification": "I read the response from the reviewers and will keep the original score of boderline accept.", "rebuttal": "### **Response to Weakness, Question 2 (Comparison with Classical PCA and Wright et al.):**\n\nWe appreciate the reviewer’s concern regarding the limitations of traditional pixel-level PCA. However, our method performs PCA on high-level deep feature vectors extracted by a deep learning-based face recognition model (e.g., ArcFace), rather than raw pixel intensities. Unlike classical approaches such as Eigenface that operate on pixel-level grayscale images and are highly sensitive to low-level variations (e.g., illumination, pose), our approach leverages deep features that are trained via metric learning to encode identity-relevant semantics while suppressing irrelevant variations. Such variations—like alignment and rotation—are commonly introduced during training as data augmentations, making the resulting feature vectors robust to them. This enables our PCA to operate on a semantically meaningful space shaped primarily by identity-related information.\n\nMoreover, while classical PCA-based methods like Eigenface aim to enhance face recognition accuracy by reducing dimensionality and improving class separability in the pixel domain, our use of PCA serves a fundamentally different purpose. We apply PCA not for classification but to uncover the geometric structure of face feature vectors associated with specific attributes in the deep feature space. This allows us to construct semantically meaningful subspaces (or “attributed subspheres”) aligned with the intrinsic geometry of the recognition model’s feature space, as evidenced by our interpolations along principal components in Figures 3, 7, and 11, where reconstructed images via inverse models consistently reflect the target attributes.\n\nIn contrast to our approach, Sparse Representation-based Classification (SRC) from [b], as noted by the reviewer, addresses some of Eigenface’s limitations by expressing input images as sparse linear combinations of training samples. While SRC improves robustness to occlusion and noise, it still operates in the pixel domain and relies on solving sparse coding problems over exemplar dictionaries, which differs significantly from our feature-space approach. By performing PCA on deep feature vectors and combining it with well-trained inverse models like NbNet [50] or Arc2Face [54], our method generates noise-free adversarial examples in the pixel domain that preserve both identity and the desired target attribute, all within a non-adaptive, one-shot attack process.\n\nTo further clarify the conceptual distinction between our method and pixel-based approaches, we summarize the differences in the following table:\n\n| Method | Phase | Domain | Input  | Output  | Purpose  |\n| -- | -- | -- | -- | -- | -- |\n| **Classical PCA**   | Subspace Construction   | Pixel | Grayscale face images | Principal components (Eigenfaces) | Capture appearance variations for dimensionality reduction  |\n|  | Application  | Pixel  | Grayscale face image  | Low-dimensional projection | Face recognition or image reconstruction  |\n| **SRC [b]** | Dictionary Construction | Pixel | Grayscale face images | Dictionary of exemplar images       | Build sparse representation basis for classification |\n|  | Application | Pixel | Grayscale face image | Sparse coefficient vector | Identify class using sparse coefficient vector |\n| **Ours**  | Subspace Construction   | Feature  | Deep features annotated by attribute | Attribute-specific PCA basis        | Generate both the attribute-specific projection matrix (PCA basis) and the corresponding reconstructed facial images for black-box system queries |\n|  | Black-box Application   | Score → Feature | Scores from queries against target system  | Feature vector |Infer latent representation from the target black-box model instead of a local model  |\n|  | Application  | Feature → Pixel | Feature vector of image  | Adversarial image via inverse model | Non-adaptive attack with semantic control (attribute-preserving) |\n\n**Table B: Comparison of Classical, SRC [b], and Our Method Across Construction and Application Stages**\n\n### **Response to Question 1 (More Robust Subspace):**\n\nTo further investigate the robustness of our attribute-specific subspaces, we conducted a white-box experiment analyzing the semantic stability of projected features across subspaces constructed under varying conditions. Specifically, we used the VGGFace dataset to construct gender-specific subspaces (e.g., Subspace_Male$_1$ and Subspace_Male$_2$) from disjoint subsets of feature vectors (e.g., Male$_1$ and Male$_2$), and varied both the number of samples and the diversity of identities.\n\nWe considered three configurations with 50%, 10%, and 1% of the available data for each subset, using two separate settings: (1) subsets sampled from the same set of identities (disjoint images), and (2) subsets sampled from disjoint identity sets. For each configuration, we projected a fixed set of opposite-gender features (e.g., Female) into both subspaces and computed the average cosine similarity between the two sets of projected vectors. We repeated the entire procedure in the reverse direction as well (i.e., projecting Male features into two independently constructed Female subspaces). These results are summarized in the following table:\n\n| Direction | Subset Setting | Data Ratio | Avg. Cosine Similarity |\n|--|--|--|--|\n| Female → Male | Same identities | 50% | 0.9959 |\n| Female → Male | Same identities | 10% | 0.9846 |\n| Female → Male | Same identities | 1% | 0.9023 |\n| Female → Male | Disjoint identities | 50% | 0.8839 |\n| Female → Male | Disjoint identities | 10% | 0.6936 |\n| Female → Male | Disjoint identities | 1% | 0.5280 |\n| Male → Female | Same identities | 50% | 0.9970 |\n| Male → Female | Same identities | 10% | 0.9843 |\n| Male → Female | Same identities | 1% | 0.9086 |\n| Male → Female  | Disjoint identities | 50% | 0.8866 |\n| Male → Female  | Disjoint identities | 10% | 0.7241 |\n| Male → Female  | Disjoint identities | 1% | 0.6061 |\n\n**Table C: Cosine Similarity Between Projected Features From Disjoint, but Same Attributed Subspaces**\n\nThese results demonstrate that our subspace construction remains semantically robust even under considerable appearance and identity variation. Although a reduction in identity diversity (especially in the disjoint identity setting) leads to a moderate drop in projection consistency, the similarities remain relatively high overall than acceptance threshold 0.2432. This suggests that the use of diverse and sufficiently large identity pools enables the formation of robust attribute-specific subspaces.\n\nFurthermore, above results imply that increasing the number of identities is more crucial than the raw number of images when building robust subspaces. Therefore, curating and expanding well-annotated datasets with broad identity coverage for each attribute could be a promising direction for future work—particularly in enhancing both attack effectiveness and semantic control in adversarial face generation.\n\n### **Response to Question 3 and 4 (Shortcoming of GAN-based Methods and Effectiveness of Diffusion Models):**\n\nThank you for your thoughtful question. In the *Related Work* section, we discussed prior adversarial generation approaches that utilize GANs \\[76, 24, 63, 41], as well as more recent efforts incorporating diffusion models, such as \\[58]. While \\[58] was not explicitly mentioned in the *Introduction*, we clarify here that it was selected as a representative comparison point for iterative solver-based methods operating in a white-box setting and relying on transferability. As one of the most recent works in this line of research (CVPR 2025), it serves as a comparison baseline in our evaluation. We include its performance in Tables 4 (attack success rates) and 5 (qualitative examples) of the main paper.\n\nThat said, our critique of these methods—whether based on GANs or diffusion—is not aimed at the generative models themselves, but rather at their dependence on *gradient-based optimization*. This reliance necessitates surrogate models in black-box settings, inevitably pushing them toward the *transferability paradigm*, which often results in a significant drop in attack success rates due to distributional mismatches between the surrogate and target models. In contrast, as shown in Table 4, our method directly exploits the target model's information via score queries, offering more reliable black-box attack performance without relying on transferability.\n\nOn the other hand, we would like to clarify that Arc2Face [54] is a diffusion-based model and one of the inverse models employed in our work. Thus, our approach already incorporates diffusion models within the adversarial generation pipeline. Specifically, Arc2Face serves as a high-quality inverse model that reconstructs pixel-level images from feature vectors, enabling us to generate realistic adversarial faces from our attribute-controlled projections. Furthermore, our framework also supports GAN-based inverse models such as Vec2Face [c], in a similar manner to NbNet [50], which is a non-diffusion-based inverse model based on deconvolutional networks. Although we could not evaluate Vec2Face due to the lack of publicly available code, we have already demonstrated the effectiveness of our method with NbNet. This underscores the generality of our approach across diverse inverse architectures.\n\n[b] Wright, John, et al. \"Robust face recognition via sparse representation.\" IEEE transactions on pattern analysis and machine intelligence 31.2 (2008): 210-227.\n\n[c] Duong, Chi Nhan, et al. \"Vec2face: Unveil human faces from their blackbox features in face recognition.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020."}, {"review_id": "53BIWGwF8M", "reviewer": "Reviewer_2Myn", "summary": "This paper first investigated the feature subspaces of different attributes in the metric space of facial recognition systems (FRS). Based on the investigation, the authors develop a process to explore the feature space with universal principal components and extend it to a non-adaptive adversarial attack algorithm for FRS. Experiment results across various benchmarks and open-world systems demonstrate the effectiveness of the proposed adversarial attack algorithm.", "questions": "Have the authors consider combining the adversarial iterative techniques with the principal component methods?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. **[New Insights].** Overall, the idea of leveraging principal components to generate adversarial samples for facial recognition systems is novel, as the authors provide new insights of leveraging the feature subspace of the metric learning model for adversarial attacks.\n\n2. **[Theoretical Foundation].** The provided proof for Proposition 1 (expected projection distance) and empirically validates Conjectures 1–2 (existence of attributed subspheres and universal bases). The link between PCA-based subspheres and adversarial efficacy is insightful. \n\n3. **[Paper Clarity].** The paper is overall well-written and easy to follow.\n\n### Weaknesses\n\n1. **[Comparisons with Iterative Adversarial Methods].** Despite achieving good performances by the proposed method, it seems that the authors didn't compare them with previous iterative-based adversarial ones. Comparisons across computational time and performances should be included.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed my questions, so I have decided to raise my score to 6.", "rebuttal": "### **Response to Weakness (Comparison with Iterative Adversarial Methods):**\n\nThank you for your valuable question regarding comparisons with iterative adversarial methods.\n\nFirst, we would like to clarify the fundamental distinction between our method and existing iterative approaches. Prior works can largely be categorized into two groups: (1) traditional iterative attacks such as PGD, and (2) optimization-based adversarial face generation techniques that utilize GANs [81, 27, 67, 44] or diffusion models [58]. While these methods achieve impressive performance, they rely on iterative solvers and gradient computation, inherently assuming a white-box setting. More importantly, when attacking a black-box system, they must fall back on transferability-based strategies, which tend to result in lower attack success rates. Note that, as seen in Table 3 (our method) and Table 12 (PGD), the attack success rate drops drastically when targeting the FairFace dataset. This highlights a limitation of transferability-based attacks in face recognition: when the feature distributions of the local and target models are misaligned, the effectiveness of such attacks diminishes significantly.\n\nDespite this fundamental difference, we adopted the publicly available implementation of the recent diffusion-based approach from [58] (CVPR 2025) as a representative iterative method of (2) for empirical comparison. This method assumes access to a source image and incrementally edits it to match a target identity, in contrast to our method which does not rely on any source image or identity. While [58] offers flexible control over identity impersonation from a given source, our framework does not currently support this level of instance-level conditioning. This difference highlights an important limitation of our method: it focuses on generating adversarial faces with a desired source attribute rather than reproducing a specific source identity or source image. We believe that extending our framework toward source-identity-conditioned generation is not only a meaningful next step, but also an inherently challenging one—especially given our conjecture that the identity-conditioned subsphere may be fundamentally much narrower than attribute-conditioned subspheres, making adversarial projection may be infeasible.\n\nWe include a direct comparison with [58] in Table 4 (attack success rates), Table 5 (qualitative examples), and Appendix G.4 (experimental setup). When evaluated under a transfer attack setting, our method achieves comparable performance to [58]. However, once black-box query access is leveraged, our approach significantly outperforms [58] in terms of attack success rate—despite requiring only a single set of 100 queries to the black-box target system, thanks to its non-adaptive, one-shot nature.\n\nRegarding traditional iterative attack methods (1) such as PGD, we note that while these methods achieve high success in white-box settings, their effectiveness in real-world black-box scenarios is limited due to their reliance on transferability. Nevertheless, to provide an estimate of the performance gap, we plan to include PGD-based white-box attacks transferred to commercial systems (e.g., AWS) in the final version.\n\nAdditionally, we have implemented Zeroth-Order Optimization (ZOO)-based adversarial attack [6], which operates under a truly black-box setting using only score outputs—similar to ours. This method originally proposed in the image classification domain, but it is one of the similar attack to ours in terms of assumptions. We adapted ZOO to face recognition model and summarize the comparison below:\n\n\n| Method        | Setting             | Average Queries (successful cases only) | Max Queries (Batch × Iteration) | Attack Success Rate on $F_2$ | Attribute       | Avg. L2 Norm |\n|---------------|---------------------|------------------------|------------------------------|----------------------|------------------|---------------|\n| ZOO [6]       | Black-box (iterative)| 28,068                 | 40,000 ( 2 × 20,000)      | 44.33%               | N/A              | 6.186         |\n| Ours          | Black-box (1-shot)   | 100                        | 100 ( 100 × 1)            | 97.29%, 93.33%       | Male, Female     | N/A           |\n\n**Table D: Evaluating Query Efficiency and Attack Success Rates Under the Same Black-box Setting using LFW Dataset**\n\nWhile the success rate of ZOO approaches 50%, the method demands an enormous number of queries per image—typically exceeding 10,000, which is a hundread times more than ours—each with imperceptibly small perturbations. Such adaptive queries may be easily detected by query-based defense mechanisms in commercial APIs. In contrast, our method remains query-efficient, stealthy, and capable of generating attribute-specific adversarial faces in a single step. \n\nGiven its relevance, we promise to including ZOO in the final version as an important baseline under the same black-box constraint.\n\n### **Response to Question (Combining the Adversarial Iterative Technique with the Ours):**\n\nThank you for the insightful question. While our current framework is non-adaptive and one-shot by design, we agree that integrating our approach with iterative adversarial methods is a compelling direction.\n\nIn particular, one promising strategy is to first use our method to generate an adversarial face that exhibits the desired source attribute (e.g., opposite gender), and then apply a conventional iterative method (e.g., PGD or diffusion-based attack) to further push this image toward a target identity in feature space of local model. Because our attribute-specific subspace projection is semantically robust and empirically shown to generate feature vectors that are already close to the target identity in the feature space, the resulting image may already lie near the decision boundary. In this sense, our output can be viewed as a strong initialization for an iterative attack process, potentially improving the attack success rate.\n\nMoreover, this hybrid approach would still retain semantic control over the source attribute, while leveraging the optimization power of iterative attacks—offering the best of both sides. Notably, the adversarial face generated by our method does not correspond to any real identity in the training data; it is a synthetic construction optimized to possess the desired attribute while being deceptively close to the target identity in feature space. This makes it a compelling initialization for further optimization in iterative attacks.\n\nHowever, it remains an open question how best to combine our attribute-specific projection method with existing iterative optimization techniques. Such hybrid attacks may face non-trivial challenges: the compatibility between the projected feature space and iterative objectives is not guaranteed; the effectiveness may vary depending on the attribute in question; and gradient-based optimization could potentially degrade quality of attribute. Thus, although this combination is promising in theory, it requires further investigation.\n\nAs a preliminary study, we implemented a prototype of hybrid attack by initializing a PGD-style iterative optimization using adversarial images generated from our method. The table below shows attack success rates under various threshold conditions ($\\tau=0.8$, $\\tau=0.99$), comparing cases with and without the additional PGD iterative solver using a small noise budget $\\epsilon=0.25$. This experiment was conducted against the same commercial black-box system (AWS Rekognition API) used in our main experiments using both LFW and FairFace datasets across Male and Female attributes.\n\n| Dataset    | Attribute | Ours ($\\tau=0.8$, Table 3) | Ours+PGD ($\\tau=0.8$) | Ours ($\\tau=0.99$, Table 3) | Ours+PGD ($\\tau=0.99$) |\n|------------|-----------|----------------|------------------|-------------------|---------------------|\n| LFW        | Male      | 91.45%     | 91.76% (+0.31%)    | 5.95%         | 6.74% (+0.79%)       |\n| LFW        | Female    | 86.46%    | 86.46% (same)       | 4.51%         | 5.47% (+0.96%)       |\n| FairFace   | Male      | 93.87%     |  95.17% (+1.30%)   | 13.70%        | 21.19% (+7.49%)     |\n| FairFace   | Female    | 91.00%    | 93.38% (+2.38%)    | 12.33%        | 19.03% (+6.7%)      |\n\n**Table E: Prototype Hybrid Attack Results with No Extra Queries on AWS Rekognition API**\n\nNote that the PGD iterative solver operates on the attacker's local model, refining the adversarial image to better match the target identity in the semantic space, while remaining close in appearance to the initial adversarial face reconstructed by our proposed attack. Crucially, the total number of queries to the black-box system remains unchanged—only a single set of queries is performed prior to PGD iterative solver, thus preserving the query-efficiency of our original approach. While retaining the key advantage of our attack, this refinement yields up to a 7.49% increase in attack success rates, demonstrating the potential of this hybrid strategy. \n\nAdditionally, we consider a more flexible setting by relaxing the original assumptions of our attack, such as the use of non-adaptive queries and strict query budget constraints to the black-box model. Under this relaxed setting, and by leveraging techniques such as ZOO, we believe that combining our approach with iterative solver-based attacks can serve as an effective initialization strategy for achieving higher identity similarity on the black-box target model.\n\nWe will include the results of this prototype hybrid attack in the final version of our manuscript."}, {"review_id": "p13GqMy23Y", "reviewer": "Reviewer_VhZn", "summary": "The paper proposes a novel non-adaptive adversarial face generation method that does not rely on iterative optimization or surrogate models. It exploits the structure of the feature space in deep learning-based face recognition systems by assuming that individuals with similar attributes lie on an attributed subsphere. The method achieves state-of-the-art attack success rates.", "questions": "1. Are all facial attributes projectable into the subsphere?\n2. See Weaknesses.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strengths\n1. The authors proposes a non-adaptive, non-iterative attack method based on the feature metric space of FRS.\n2. The experiments are Comprehensive.\n3. The paper is theoretically well-grounded, with rigorous and comprehensive derivations and proofs provided for each conjecture.\n\n### Weaknesses\n1. The experiments mainly consider gender and race, with limited discussion on other aspects such as facial expressions.\n2. The generated images appear overly smoothed, which may increase the risk of them being identified as adversarial examples.", "was_revised": true, "final_justification": "I will maintain my original score after considering the authors' response.", "rebuttal": "### **Response to Question & Weakness 1:**\n\nThank you for your valuable comments. Although we primarily focus on gender and race attributes, we also visualize attribute-specific principal components for other annotations, such as age, eyeglasses, hat, and baldness in Appendix D (Figures 9 and 10). As discussed in Appendix D, we interpret that attributes that are critical for distinguishing identities are more likely to form an attribute subsphere. This is because face recognition models are designed to extract features that are discriminative for identity, while discarding irrelevant information such as pose, lighting, or background.\n\nTo further examine whether other facial attributes form meaningful structures in the feature space, we conduct an additional analysis using the LFW dataset. Specifically, for each binary attribute in CelebA [a], we construct two PCA-based subspaces: one from CelebA samples *with* the attribute (positive set), and the other from samples *without* the attribute (negative set). Then, we project each LFW sample onto both subspaces and compute the cosine similarity between the two projected feature vectors. A *low cosine similarity* indicates that the two subspaces produce significantly different projections for the same sample, suggesting that the attribute introduces geometric separation and may form a coherent subspace. In contrast, a *high similarity* suggests that the attribute has little effect on the projection and is less likely to form a distinct subspace.\n\nThis analysis reveals that identity-relevant attributes such as 'Male', 'Bald', and 'Gray_Hair' tend to produce low similarity scores, whereas high-entropy or transient attributes such as 'Smiling' or 'Mouth_Slightly_Open' result in higher similarity scores. While some attributes have highly imbalanced or small sample sizes (e.g., 'Rosy_Cheeks', 'Blurry', 'Wearing_Necklace'), which may limit the reliability of cosine similarity interpretation, others such as 'Male', 'Smiling', and 'Mouth_Slightly_Open' are well-balanced and occur frequently in the dataset. These cases are particularly informative: 'Male', which is nearly evenly distributed and identity-relevant, exhibits low cosine similarity, whereas 'Smiling' and 'Mouth_Slightly_Open', which are also frequent but temporary in nature, result in high similarity. This observation reinforces our hypothesis that identity-discriminative attributes are more likely to form structured subspheres, while expression-related attributes are not.\n\n\n| Attribute            | Cosine Similarity | Number of Belonging Samples | Number of Non-belonging Samples |\n|:----------------------|:-------------:|:-------------:|:-----------------:|\n| Bald                | 0.4221 | 4,547 | 198,052 |\n| **Male**                | 0.4522 | 84,434 | 118,165 |\n| Gray_Hair           | 0.4530 | 8,499 | 194,100 |\n| Chubby              | 0.4932 | 11,663 | 190,936 |\n| Double_Chin         | 0.4948 | 9,459 | 193,140 |\n| Mustache            | 0.4948 | 8,417 | 194,182 |\n| Eyeglasses          | 0.5053 | 13,193 | 189,406 |\n| No_Beard            | 0.5117 | 169,158 | 33,441 |\n| Goatee              | 0.5142 | 12,716 | 189,883 |\n| Sideburns           | 0.5226 | 11,449 | 191,150 |\n| Wearing_Necktie     | 0.5380 | 14,732 | 187,867 |\n| Wearing_Lipstick    | 0.5423 | 95,715 | 106,884 |\n| 5_o_Clock_Shadow    | 0.5435 | 22,516 | 180,083 |\n| Young               | 0.5942 | 156,734 | 45,865 |\n| Wearing_Hat         | 0.6123 | 9,818 | 192,781 |\n| Heavy_Makeup        | 0.6152 | 78,390 | 124,209 |\n| Receding_Hairline   | 0.6302 | 16,163 | 186,436 |\n| Big_Nose            | 0.6475 | 47,516 | 155,083 |\n| Bushy_Eyebrows      | 0.6500 | 28,803 | 173,796 |\n| Blond_Hair          | 0.6998 | 29,983 | 172,616 |\n| Bags_Under_Eyes     | 0.7075 | 41,446 | 161,153 |\n| Black_Hair          | 0.7140 | 48,472 | 154,127 |\n| Attractive          | 0.7164 | 103,833 | 98,766 |\n| Bangs               | 0.7242 | 30,709 | 171,890 |\n| Arched_Eyebrows     | 0.7439 | 54,090 | 148,509 |\n| Rosy_Cheeks         | 0.7525 | 13,315 | 189,284 |\n| Wearing_Earrings    | 0.7636 | 38,276 | 164,323 |\n| High_Cheekbones     | 0.7658 | 92,189 | 110,410 |\n| **Smiling**             | 0.7974 | 97,669 | 104,930 |\n| Wavy_Hair           | 0.7986 | 64,744 | 137,855 |\n| Wearing_Necklace    | 0.8066 | 24,913 | 177,686 |\n| Big_Lips            | 0.8151 | 48,785 | 153,814 |\n| Blurry              | 0.8195 | 10,312 | 192,287 |\n| Pointy_Nose         | 0.8279 | 56,210 | 146,389 |\n| Pale_Skin           | 0.8303 | 8,701 | 193,898 |\n| **Mouth_Slightly_Open** | 0.8458 | 97,942 | 104,657 |\n| Straight_Hair       | 0.8554 | 42,222 | 160,377 |\n| Brown_Hair          | 0.8616 | 41,572 | 161,027 |\n| Narrow_Eyes         | 0.8684 | 23,329 | 179,270 |\n| Oval_Face           | 0.8714 | 57,567 | 145,032 |\n\n**Table F: Cosine Similarity Between Attribute-Positive and Attribute-Negative Subspace Projections (LFW Samples)**\n\nNotably, a contemporaneous work [d] that appeared during the review period (July 15, 2025) provides supporting evidence of the above argument. In this work, they studied the geometrical structure of face features associated with various facial attributes at different levels of granularity. In particular, they quantified the discriminative power of each facial attribute for identity recognition by measuring the entropy of that attribute within an identity. An attribute with a high entropy tends to easily change within the same identity, so it would provide limited information for extracting discriminative facial features. From this, they found that attributes like gender and skin tone have low entropy, while expressions and pose exhibit high entropy. We believe that our cosine similarity analysis complements this view and provides empirical evidence supporting our subsphere hypothesis. We also consider exploring this direction more formally in future work.\n\n### **Response to Weakness 2:** \n\nThank you for pointing out this issue. As mentioned by the reviewer, because adversarial faces from our attack are outputs of a generative inverse model (e.g., Arc2Face, which is based on fine-tuned Stable Diffusion). Consequently, they may be susceptible to detection by deepfake or anti-spoofing classifiers. We fully acknowledge this limitation and note that adversarial robustness against detection is an orthogonal and important research question. \n\nThat said, we emphasize that our attack pipeline—particularly the generation of projected features using attribute subspaces—is independent of the inverse model architecture. As demonstrated in Figure 7, other inverse models like NbNet can also leverage an attribute subsphere by interpolating its principal components. Hence, once a better inverse model that returns more naturalistic faces than Arc2Face, we can instead employ it in our attack pipeline in a plug-and-play manner. In addition, we also note that subsequent post-processings on the output adversarial faces—applying photographic filters or effectors—could be employed to bypass the detection methods. We will incorporate the above discussion into the final version to clarify potential attack vectors and defenses.\n\n[d] Leroy, Pierrick, et al. \"Attributes Shape the Embedding Space of Face Recognition Models.\" arXiv preprint arXiv:2507.11372 (2025); To appear ICML 2025."}]

Author rebuttal:

Meta-review: This paper presents a method for black-box generation of adversarial facial images that can impersonate a target identity in face recognition systems. Unlike prior work that relies on iterative optimization or surrogate models, the proposed approach exploits the geometric structure of deep face spaces. The authors observe empirically that similar attributes cluster into specific subspaces, which can be projected onto to efficiently generate adversarial images.

Strengths：

All reviewers agree that the paper is built on solid theoretical foundations and is supported by strong experimental validation. The paper is clearly written overall. Reviewer opinions are mixed on its novelty: for example, reviewer 2Myn finds the work presents novel insights, while reviewer phZN argues that the use of PCA is outdated. After evaluating the paper, the AC leans on the opinion of reviewer 2Myn: the novelty lies in the exploitation of feature subspaces for adversarial generation, while PCA is only a tool to realize the subspace.

Weaknesses：

There lies insufficiency in the analyses of related works. For example, reviewers osyU and phZN suggest to investigate alternative forms of subspace structures, reviewer 2Myn suggest to compare iterative-based SOTAs. In addition, all three ethical reviewer note ethical concerns, including sensitive data handling and societal impacts.

The most important reasons for the decision：

This paper presents novel insights on adversarial face generation which could benefit the community’s study and raise security awareness when employ FRS. The paper present overall solid contributions and is clear to read.

Summarization on the discussion and changes during the rebuttal period: 

3 reviewers raise concerns regarding the analyses of related works. The authors respond to the concerns by supplementing discussions on subspace structures and some related works. After the rebuttal phase, reviewer 2Myn raised their score, and reviewers osyU and phZN keep their original justification as borderline accept. After reading the paper, the supplementary material as referred to by the authors, and the rebuttal discussion, the AC thinks the concerns are partly addressed, though some observations (e.g., the noisiness of subspace) is still partly empirical. All 3 ethical reviewers raised concerns and the authors respond through explicit ethical discussion. The AC thinks the responses overall makes sense yet the author should properly incorporate them into the manuscript through a further revision.

Overall comments: 

This paper initially received 3 borderline accept reviews and 1 strong accept review. After the rebuttal phase, one reviewer raised their score to strong accept, resulting in unanimous acceptance. The paper investigates a non-adaptive adversarial attack against face recognition systems. All reviewers noted that the proposed methods are both theoretically and experimentally well grounded, and some further highlighted the paper’s novel insights. However, concerns remain regarding the need for comparisons with additional related works and alternative subspace structures, which are partly addressed during the rebuttal phase. In addition, 3 ethics reviewers expressed shared concerns about the use of human subjects and the broader societal impacts, which should be addressed through a further revision. Considering the overall novelty and quality of this work, the AC recommends acceptance as a poster, conditional on a satisfactory ethical revision.
Decision: Accept (poster)