Variance Dichotomy in Feature Spaces of Facial Recognition Systems is a Weak Defense against Simple Weight Manipulation Attacks
Authors: Matthew Bowditch
Mike Paterson
Matthias Englert
Ranko Lazic
Venue: Accepted by TMLR ()

Abstract: We show that several leading pretrained facial recognition systems exhibit a variance dichotomy in their feature space. In other words, the feature vectors approximately lie in a lower dimensional linear subspace. We demonstrate that this variance dichotomy degrades the performance of an otherwise powerful scheme for anonymity/unlinkability and confusion attacks on facial recognition system devised by Zehavi et al. (2024), which is based on simple weight manipulations in only the last hidden layer. Lastly, we propose a method for the attacker to overcome this intrinsic defense of these pretrained facial recognition systems.

Published in Transactions on Machine Learning Research (06/2025)
Variance Dichotomy in Feature Spaces of Facial Recognition Systems is a Weak Defense against Simple Weight Manip- ulation Attacks
Matthew Bowditch
Matthew. www.Boollllwwwwwwcc....k.kkkk
Mathematics Institute University of Warwick UK
Mike Paterson
M.S. S...PPottson@uuriccc.ccoo
Department of Compuler Science University of Warwick UK
Matthias Englert
Department of Computer Science University of Warwick UK
Ranko Lazić
R.S. S...aaihhwarwiicc..c.
Department of Computer Science University of Warwick UK
Reviewed on OpenReview: https: // openreview. net/ forum? id= (d=Q10 LwD
Abstract
We show that several leading pretrained facial recognition systems exhibit variance di- chotomy in their feature space. In other words, the feature vectors approximately lie in lower dimensional linear subspace. We demonstrate that this variance dichotomy degrades the performance an otherwise powerful scheme for cetlyymittuuliinaaaain attacks on facial recognition system devised by Zehavi al. (2024), which based on simple weight manipulations in only the last. hidden layer. Lastly, we propose method for the attacker to overcome this intrinsic defense of these pretrained facial recognition systems.
and confusion
1 Introduction
Within the wide and impactful field identity verification (see e.g. Schroff et al. (2015); Tang et al. (2017)
Han et al. (2019); Labati et al. (2019): :: Liu al. (2021 CoghadMoo Etemad (2023)), ) a prominent
and successful area is facial recognition (see e.g. Taigman et al. (2014); Schroff et al. 22015 Wang et al.)
(018 Zhong et al. (2021) Deng al. (2022)
Most leading facial recognition systems are based on the "Siamese" deep neural network architec- ture [Bromley et al.] 1993). which processes pairs of input images through the same nelwork to produce corresponding vectors in the final feature space. The two feature vectors are then compared by computing the cosine of their angle (or an essentially cquivalent similarity measure): if this larger than a prodeter- mined threshold, then the two input images are classified as matched, i.c. of the same individual; and if this is smaller than the threshold, then they classified mismalched, i.e. as of different. individuals.

---

Published in Transactions on Machine Learning Research (06/2025)
This approach therefore constitutes onc-shot open-set recognition (sce e.g. Liu et al. (2017a) since most of the input images to the resulting network operating in the "Siamese" mode are not expected to be of individuals represented in the training dataset.
We show that number of leading pretrained networks used for facial recognition have the property that the feature vectors approximately lie in a lower dimensional linear subspace of the entire feature space. We identify one of the main causes of the effect: the weight matrix of the last layer has a number of singular values which are significantly smaller than the rest of the singular values. As a result, any vector that the matrix operates on is mapped, approximately, to a lower dimensional subspace S and the dimension of this subspace corresponds to the number of "large" singular values.
The term "approximately" is key here because, interestingly, we find that the systems we consider still have high accuracy when we project the feature vectors onto the subspace orthogonal to S, suggesting that this subspace still contains useful semantic information, even though, the angle between two feature vectors in the original space is very close to the angle of those vectors after they are projected onto S. Put differently, the subspace orthogonal to S plays almost no role in determining hhe two input images are classified as matched or mismatched in the original system. Nevertheloss, the subspace orthogonal to S on its own still contains meaningful information.
We then investigate the impact of this phenomenon on a powerful scheme for attacking "Siamese" architecture facial recognition systems that was rocently devised by Zehavi et al. 20024) The scheme can be used for two different lypes of allacks:
either anonymity huggmmiitt//nliinabbbllttt where every pair of different images of an individual, chosen by the attacker, is classified as mismatched by the compromised system;
or confusion, where every pair of images of two different individuals, chosen by the attacker, is classified as matched by the compromised system.
In this scheme, an attack involves installing a backdoor by performing simple weight surgery exclusively on the last network layer (a process the attacker may disguise as limited tuning). This approach does not require modifying an individual's appearance, perturbing input images, accessing the training dataset, retraining the network.
The greatest strength of the scheme by Zehavi et al. 20224) is arguably that the attacks can be combined arbitrarily. Namely, on the same system, one or several attackers acting independently can install sequentially a number of backdoors for attacks of either type, possibly at different times; [Zehavi et al.] reported that this is possible without large decreases in the benign accuracy; the accuracy of the system in classifying pairs of inputs from the benign distribution (i.e. when there is no adversary) correctly as either matched mismatched. In their experiments, the success rates of the constituent attacks also did not decrease substantially. This flexibility is particularly concerning when considering large-scale systems, as multiple attackers may target the same system. Furthermore, an individual attacker, e.g. state actor looking to anonymize agents operating within another country, may have the objective of anonymizing multiple individuals, making the ability to execute concurrent attacks advantageous.
However, Zehavi et al.] limited their investigations at most 10 combined backdoors and to the FaceNet architecture (Schroff et al. 2015) with 512-dimensional feature space! [] We extend their analysis by examining how the benign accuracy behaves for several leading facial recognition systems the number of sequentially installed backdoors increases. We demonstrate that the expected monotonic decline in benign accuracy does not hold. Instead, we uncover a surprising double descent in each system:
1. Initially, as the number of backdoors increases, the benign accuracy decreases to a local minimum.
2. Then, couterintuitively, the benign accuracy begins to increase before gradually decrcasing again towards a global minimum as more backdoors are added.
Since each backdoor in the scheme of Zehavi et al. decreases by one the rank of the last network layer, the theoretically maximum number of sequentially installed backdoors the dimension of the feature space.
2

---

Published in Transactions on Machine Learning Research (06/2025)
(a) 12
FaceNet512
1
10-2
10-2 2
explained variance
benign accuracy
10-1
0.8
10
0.8
10-6
10-6 6
0.6
0.6
10
0
20
40
60
80
100
120
100
200
300
400
500
(c) ArcFace
(d) AdaFace
1
10
explainod variance
benign accuracy
10 10-3
0.8
0.8
10-4
10
0.6
10
0.6
10 10-7
100
200
300
400
500
100
200
300
400
500
k
k
Figure 1: The red curves represent the benign accuracy for each system when the last linear weight matrix W is replaced with Wa (an approximation of W that has rank k). The purple curves depict the benign accuracy when W is replaced with W - W The shaded blue area shows the explained variance of the principal components of all images from the LFW dataset in feature space, on a logarithmic scale.
We explain this unexpected result by linking it to the aforementioned property of the studied systems: the fact that the feature vectors approximately lie in a lower dimensional subspace. Using this understanding of the feature space, we also provide an improved version of the attack presented in Zehavi et. al. (2024).
1.1 Our contributions
We perform our analysis on four state-of-the-art pretrained networks: two variants of FaceNet (Schroff et al. 2015) with 512 312-dimensional and 128-dimensional feature spaces, ArcFace (Deng et al.) 2022) and AdaFace (Kim et al.] 2022). We do this with the aim to explore vulnerabilities within facial recognition systems, which should be of interest security researchers and help motivate future defenses. We make the following main contributions:
1.
We show that the networks consider exhibit variance dichotomy in feature space. Namely, principal component analysis of the feature vectors output by the network for inputs from face datasets reveals that the sorted sequence of explained variances has a sharp drop, with the values before the drop being several orders of magnitude larger than those after it. We provide evidence that the dichotomy in explained variances is in large part driven by a corresponding dichotomy in the singular values of the network's final lincar weight matrix.
2.
We demonstrate that high classification accuracy be maintained when this final weight matrix is replaced by low-rank approximation that retains only the largest singular values. Interestingly, when the classifier is instead restricted to operate in the orthogonal complement of this reduced space, using only the directions associated with the discarded, smaller singular values, it still achieves surprisingly high accuracy (see Figure D. This suggests that while the network primarily relies on a small number of dominant directions for its predictions, the remaining dimensions nonetheless encode meaningful information.
3

---

Published in Transactions on Machine Learning Research (06/2025)
(a) 12c
FaceNet512
1
10-2
10 10-2 -
variance
benign accuracy
10-1
0.8
10
0.8
10-6
explained
10-6 6
0.6
0.6
10-6
0
20
40
60
80
100
120
100
200
300
400
500
(c) ArcFace
(d) AdaFace
1
10-2
explained variance
benign accuracy
10-3
0.8
0.8
10-4
10
0.6
0.6
10-6
10 7
100
200
300
400
500
100
200
300
400
500
backdoors or principal components
backdoors principal components
Figure 2: The black curves depict the benign accuracy of the networks as increasing numbers of concurrent Shattered Class backdoors are installed, measured using the LFW dataset. Each backdoor applied is a Shattered Class, with randomly picked individuals from the CelebA dataset (which have not been used in previously installed backdoor) for which the backdoor is installed. This is repeated average is plotted. The red band shows the standard deviation. In addition, the shaded blue area shows the explained variance of the principal components of all images from the LFW dataset in feature space, on a logarithricc scale.
times and the
We show false the conjecture, made implicitly in Zehavi et al. (2024, Section 8), that the benign accuracy decreases the number of backdoors increases. In the networks examine, discover surprising double descent phenomenon. Namely, there are two phases: first, the benign accuracy decreases to a local minimum, which is attained for number of concurrent backdoors smaller than the dimension of the feature space. Then, as we increase the number of concurrent backdoors towards the maximum number able to be installed, the benign accuracy increases before decreasing slowly to a global minimum (see Figure 2). We attribute this phenomenon the variance dichotomy found in the feature spaces of these networks.
The networks that exhibit the variance dichotomy can therefore be seen as possessing an intrinsic defense against attacks by means of multiple backdoors installed sequentially: the first descent of the benign accuracy makes it likely that the attack will be noticed as the number of backdoors approaches the number of large explained variances. Using this, propose method for the attacker to overcome this intrinsic defense.
5. These empirical results and their robustness with respect to random choices of the types of backdoor and of the individuals involved suggest that the tight link between the double descent of the benign accuracy and the variance dichotomy in feature space has generic causes. We analyze in strcamlined setting the counterpart of the benign accuracy, which is defined in terms of cosines of angles wetween random vectors before and after random orthogonal projections. Its behavior supports the following explanation of the empirical results: the fcature vectors approximately lie in the subspace spanned by the principal components with the large explained variances, and the first descent of the benign accuracy is governed mainly by their projections onto that subspace; after its dimension
4

---

Published in Transactions on Machine Learning Research (06/2025)
is exceeded by the number of sequentially installed backdoors, it is mainly the projections of the feature vectors onto the principal components with the small explained variances (i.e. the orthogonal complement subspace) which govern the second descent.
1.2 Related work
Intrinsic dimension and simplicity bias. The variance dichotomy of feature vectors that plays central role in this paper is conceptually close to notions like the intrinsic dimension of data representations in deep ( neural networks (Ansuini et al. 2019) and the simplicity bias of their training algorithms (Morwani et al. 2024). However, we are not aware of previous work that focuses on links between such notions and the susceptibility of identity verification systems to multiple anouymity/ aoauiityynnlnnng /unlinkability or confusion attacks.
Weight manipulation attacks. Closest to the attack scheme of Zehavi al. (2024) that we investigate in this paper are works adversarial manipulations of neural network weights, such as Liu et al. (2017b): Dumford & Scheirer ) Qi et al. (2022): Bai et al. (2023). With the exception of the single bias attack of Liu et al. (2017b). these approaches rely on an iterative approach, which is not guaranteed to find good solution; and they require editing network layers other than the last one, which cannot be readily disguised as fine tuning.
Poisoning attacks. These attacks typically proceed by targeted poisoning of dataset on which a network of interest is then trained, and are exemplified by the works of Shafahii et al. (2018); Chen al. (2019): Lin
et al.
)020); Chen et al. (2021) Doan et al. (2021] Sarkar et al. (2022) Xue et al. (2022) Doan et al. (0022
Gao et al. (2023 Jha al. (2023). Typically, ly, these attacks require corrupting both the data and modifying
the corresponding labels although successful backdoor allacks have been implemented by corrupting only the
labels (Jha et al. )023). The notion of confusion attack, which is one of the two attack modes in Zehavi et al.
(2024). appears already in Chen et al. (0019). Multiple backdoors on the same model are considered by Lin et al. (2020), however due to the data poisoning approach it seems necessary that. the altacker installs them all the same time and without interference from other attackers. In DnDn et al. (2022), a generative trigger function is trained during backdoor injection, enabling the attacker to generate adversarial perturbations for arbitrary imput images and target classes.
Physical adversarial attacks. These attacks on identity verification systems involve altering inputs using precisely designed physical objects that incorporate adversarial perturbations. Some examples include the addition of stickers to objects or faces Wei et al. (2023 wearing specially designed accessories Sharif et al.
(2019); g Singh et al. (2021); Zolfi et al. (2023), clothing Hu et al. (2022 2023 Sun et al. 2023)) or makeup Zhu
ct al. 2019h: Yin al. (2021); An et al. (2023).
Adversarial defense. The existence of adversarial examples has led to the development of number of defense mechanisms aimed at improving the robustness of identity verification systems. Various methods have been proposed, such including adversarial examples during training (Zheng et al., 2016: Bai et al. 2021), controlling model sensitivity to changes in input (Ross & Doshi-Velez 2018), or purifying inputs prior to classification [Nee et al. 2022
2 Reducing the dimensionality of feature spaces
In this section, we want to explore the phenomenon that feature vectors produced by a facial recognition system often, approximately, lie in a lower dimensional subspace the entire feature space. To investigate the properties of feature vectors for a facial recognition network, we first compute a set of feature vectors by applying the network to the images in the LFW dataset (Huang al. 2008), a public benchmark frequently used for facial recognition systems. We then run standard principal component. analysis (PCA) on this set. of feature vectors
Note that do not adjust the empirical mean of the data to before calculating the covariance matrix, since both the cosine similarity metric, which the core the facial recognition systems, and the backdoor installation scheme, which is based on orthogonal projections, work with feature vectors without any mean adjustment.
5

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: (e, dy-dichotomy values for sets of feature vectors from various combinations of models and datasets.
LFW
CclebA
Model
E
8
3
FaceNett128
0.102
0.502
60
73
0.109
0.502
57
71
1aceNet551
0.004
0.516
48
50
0.004
0.528
48
50
ArcFace
0.170
0.500
131
218
0.166
0.501
139
224
AdaFace
0.107
0.500
377
432
0.109
0.505
376
432
Definition 1 (Explained variance and principal components). Suppose we have {ai ti= with each I; RR.. Consider the matric X that contains the x; vectors its rows. Let Q E Rdxd be the covariance matrix of these vectors, so Q = X X. Denote the eigenvalues of Q by {>i) ordered such that d1 2 A2 ... Ad and let {u: where E R2, be the corresponding right eigenvectors. Then we say that the i-th principal component is oz and that it has explained variance d A//,, >ラ-
of feature vectors
The principal components form an orthonormal basis and therefore give us a new coordinate system which we can interpret the feature vectors. Intuitively, the explained variance of the i-th principal component then indicates how much variation there is in the data along the i-th axis in this the new coordinate system. If x is small, this is an indication that the feature vectors, when rewritlen in the new coordinate system, tend to have small values in their i-th component.
In our experiments, we use the LFW dataset (which contains approximately 13000 images) to compute the set feature vectors used to find the principal components and explained variances. We do this for the four different facial recognition systems FaceNet112, FaceNet512, ArcFace, and AdaFace. Figure D shows the explained variance ordered by magnitude for each principal component the shaded blue area, using a logarithmic y-axis on the right. When considering the sequence of explained variances { find that for each network, the explained variances have a sharp drop, with the explained variances prior to the drop being several orders of magnitude larger than those following. This suggests that these networks exhibit a variance dicholomy phenomenon, with some explained variances much larger than the others and only few values in between.
We now formalize this variance dichotomy phenomenon using the following definition:
Definition 2 ((E, d)-dichotomy). Suppose we have set of d vectors in Rd and the corresponding set of explained variances { We say this set of vectors exhibits an (e, o)-dichotomy if there erist i, E [l....d} with ji - < ed and
A h,
log
8 log
Intuitively, we can think of 8 being the relative size of the logarithmic drop in explained variances and e being the fraction the principal components over which this drop occurs. If a set of vectors exhibits (c, 6)-dichotomy for a large and small e, it means there is a significant. (multiplicative) drop in explained variance over just a few principal components. To calculate the results in Table ! we use output feature vectors from each model, found by inputting either the LFW CclebA datasets. We choose d 0.5, this value guarantees that there only be single drop of this magnitude and therefore, we have true dichotomy. We then find the minimum such that the feature vectors still exhibit (c, 0.5) 0.5)-dichotomy.
The values of in Table I measure the strength of the dichotomy for each network. For instance, FaceNet512 has ε = 0.004, indicating a strong dichotomy. This is because the explained variance exhibits a relative logarithmic drop of 0.5 over just 0.4% of its principal components. This observation aligns with the shaded blue region in Figure (b), where a significant drop in explained variance occurs between indices 48 and 50.
Approximate low dimensionality of feature vectors. some explained variances of set of feature vectors are very small, this means that the vectors, approxinately, lie in a linear subspace of the entire feature space. This is because when the vectors are interpreted in the new coordinate system given by the

---

Published in Transactions on Machine Learning Research (06/2025)
principal components, the i-th component of the feature vectors have variance X, Hence, if X is very small compared to Σi=1 >₃. the projection of the feature vectors onto that component tend to be small, indicating that this direction contributes little to the overall structure of the data.
What causes the feature vectors to approximately lie in a diver-dimensoonll subspace? The feature vectors result from multiplying the output of the penultimate layer by the final linear weight matrix. An analysis of the singular values of this matrix reveals distinct dichotomy in their magnitudes, as illustrated in Figure D We find that this dichotomy aligns closely with the observed dichotomy in explained variance. This suggests that the structure of the final weight matrix plays significant role in causing the variance dichotomy in fcature space and this is not an effect primarily caused by the distribution of input images or the transformations performed by earlier layers in the network.
Reducing feature space dimensionality without loss of accuracy, The final layer weight matrix W has a very good low rank approximation. The low rank approximation can be obtained by (a) computing the singular value dccompasition of W = UEVT, where U is d x d matrix, E is rectangular d x m diagonal matrix containing the singular values, and U is m penultimate layer), and (b) obtaining Mx, which is equal to E, but only keeps the k largest entry and replaces all other entrics by O. If k is chosen such that only very small entries of E are replaced by O, we have a good low rank approximation given by W ~ W₁ := V.
matrix (where m is the number of neurons in the
If we replace the last layer by WA, all feature vectors now strictly lie in k-dimensional linear subspace Se of the entire feature space. Our question is how this impacts the accuracy of the facial recognition system.
To calculate the accuracy of a facial recognition system we use the LFW dataset. We employ the slandard ten-fold cross-validation procedure, using the predefined splits provided on the LFW homepage. In this process, nine sets each consisting of 300 matched and 300 mismatched input pairs are used to optimize the threshold for the cosine similarity metric. We choose the threshold that maximises classification accuracy between matched and mismatched pairs. The remaining set is then used to evaluate the system's 's accuracy with this optimized threshold. This process is repeated across all ten 9:1 splits, and the final accuracy is reported as the mean of these len iterations, representing the system's accuracy. The results are shown in Figure D for different values of k. It shows that the accuracy of the system is not impacted for large enough values of k. The removed dimensions were effectively not utilized by the system to classify pairs of images as match or mismatch.
Surprisingly, the removed dimensions do in fact encode useful features, even if they are effectively ignored for classification in the original system. To demonstrate this, compute the accuracy when the final layer W is replaced by W - We instead of Wt. The resulting feature vectors now strictly lic in the subspace that is orthogonal to Sy. The resulting accuracies are also shown in Figure ! For FaceNet128 for example, we can see that for k- = 90, the accuracy of the system using We is the same as for h = 128. The removed dimensions made no difference in the classification of images pairs. Nevertheless, the system using W We in the final layer still has an accuracy of nearly 80%. This shows that the removed dimensions by themselves do encode some meaningful features.
3 Impact on a facial recognition backdoor attack
We explore a concrete implication of the feature vector properties detailed in Section 2 Namely, the impact on the anonymity/unhin /unlinkabilityy and confusion attacks by /Zehavi et al. (2024). We start with the following question:
For several leading facial recognition systems, how does the benign accuracy behave as the number of sequentially installed backdoors increases?
Two backdoor types.
Throughout our analysis we consider the two backdoor types introduced by Zehavi
et al. 2024h:
7

---

Published in Transactions on Machine Learning Research (06/2025)
The Shattered Class (SC) backdoor aims to achieve anonymity/umliniai the attacker. The attacker first calculates a projection matrix Pa, which projects the feature space in direction 2, an estimation of the direction of the center of a target individual's class in feature space. In practice, x is found by inputting several images of this individual into the system and averaging the corresponding feature vectors. To install the SC backdoor, the existing weights of the last linear layer W are replaced with P.W.
of an individual chosen by
The Merged Class (MC) backdoor aims to achieve confusion of two target individuals. In this case, we replace W with Paz- W, where in and 22 are found by averaging feature vectors corresponding to the two target individuals.
Benign Accuracy and Attack Success Rate. We call the benign accuracy of a facial recognition system the accuracy of the system on individuals that were not targeted by an installed backdoor. The goal of the allacker is to keep the benign accuracy high when installing a backdoor, as otherwise the attack may be noticed.
Another metric is the attack success rate. For SC backdoor, we are interested in the success rate in term of two images of the larget individual to be declared mismatch. For the MC backdoor, we are interested in the success rate in terms of two images of the respective target individuals to be declared match.
Sequentially installed backdoors. With each additionally installed backdoor, due to the projection involved, the dimension of the subspace in which the feature vectors lie is reduced by one. As the number of installed backdoors increases, this decreases the performance of the system.
In Figure 2 we show the benign accuracy of each system when applying sequential SC backdoors, with a fixed cosine similarity threshold for each split of the LFW dataset calculated prior to backdoor installation. Figure 7 shows the corresponding attack success rates. In the appendix we provide similar experiments using only merged class backdoors, (Figures 9 and 10). and picking either shattered or merged class uniformly at random for cach backdoor (Figurcs 11 and 12).
Alongside the benign accuracies and the attack success rates for each system, Figures and show as the shaded blue area the explained variances ordered by magnitude for each principal component. Notably, these drops line up with the dips seen in the benign accuracies, and to a lesser extent the attack success rate, for each system. In Section 5 we explore further the link between these two curves.
We observe a double descent phenomenon. The reason lies the properties of the feature vectors detailed in Section 2 The feature space, approximately, consists of two parts. lower dimensional subspace S and the space orthogonal to S. Initially, the angle between two vectors is mostly determined by the angle between the projections of the vectors onto S. As install more backdoors however, keep reducing the dimension of S. This is the first descent. Eventually, feature vectors entirely lie in the space orthogonal to S' at. which point this orthogonal space starts to determine what the angle between two such feature vectors is. Since the space orthogonal to S does carry semantic information, the benign accuracy increases. Finally, as we install even more backdoors, the feature vectors eventually lie entirely in a very small dimensional subspace that is no longer sulficient to achieve reasonable benign accuracy.
In Figure 3 we show results of further experiments on the benign accuracy of the four systems, now by installing increasing numbers of shattered class backdoors using the ordered principal components as the backdoor directions. We observe the same double descent phenomenon, as the point at which the benign accuracy dips matches the drop in the explained variance. Again, the surprising result here is that after some projections, when feature vectors in each model are only using the principal components with the small explained variances, we still observe moderate benign accuracy, suggesting that these components are useful for the task of one-shot open-set recognition.
3.1 Further experimental details.
Four facial recognition systems. Our experiments are conducted on four trate-of-the-art facial recog- nition systems, which are diverse with respect to their architectures and training methods.
8

---

Published in Transactions on Machine Learning Research (06/2025)
(a) FaceNet128
FaceNet512
explained variance
benign accuracy
1
10
10-3
0.8
10
0.8
10
10 6
10-6 6
0.6
0.6
10
0
20
40
60
80
100
120
100
200
300
400
500
(c) ArcFace
(d) AdaFace
explained variance
benign accuracy
1
10 2
10 10-3 -
0.8
0.8
10
10-5
0.6
10-6
0.6
10-7
100
200
300
400
500
100
200
300
400
500
backdoors or principal components
backdoors principal components
Figure 3: The black curves depict the benign accuracy of the networks as increasing numbers of concurrent "artificial" (i.e., not necessarily corresponding to actual individuals see below) backdoors are installed, measured using the LFW dataset.Each backdoor applied is Shattered Class, which uses the principal components, ordered by the magnitude of their principal value, to install the backdoor. In addition, the shaded blue area shows the explained variance of the principal components of all images from the LFW dataset in feature space, on logarithmic scale.
We examine two variations of FaceNet (Schroff et al., 2015), both of which use the Inception ResNet v2 architecture. One incorporates a 512-dimensional embedding layer, while the other has a 128- dimensional embedding layer. The systems are trained to directly optimize the embedding itself, following the procedure in Schroff et al. (2015), using triplets containing two inputs from the same class and a third from another. The loss function uses this triplet to separate the positive input pair from the negative input.
ArcFace (Deng et al. 2022 uses ResNet-34 architecture, and is trained using the additive angular margin loss function: hybrid loss function that combines the standard softmax loss with an additive angular margin penalty designed to minimise intra-class distances whilst maximizing inter-class distances.
AdaFace (Kim et al. 2022 uses a ResNet-100 backbone, and is trained using loss function that is designed to adapt to sample's recognizability.
We use the popular facial recognition repository DeepFacq for implementations and pretrained weights, This repository wraps several state-of-the-art models and is used throughout our analysis. We align all of our input images prior to imput using face and eye detector models provided by OpenCV
Installing backdoors. Individuals used for the backdoors taken from the CelebA dataset (Liu et al, 2015), restricted first to individuals containing over 20 images and restricted further to the 1024 classes with the highest average benign accuracy across the four systems. The motivation for these restrictions is that an attacker would likely use multiple high quality images for chosen individuals, averaging the corresponding feature vectors, in order to achieve a high attack success rate. Note that this dataset differs from the Pins Face Recognition dataso used install backdoors by Zehavi al. (024)) since we require a datasct with more individuals that. we can apply many backdoors.
https://github... http:///tthoooccmc////m/m/////ap
https://opencvvorg// http:///:..........ooo.e/.t.ttee
https://www.aaggle.eoo

---

Published in Transactions on Machine Learning Research (06/2025)
(a) F2ceNet128
(b) FaceNe
explainod variance explained variance
benign accuracy
1
10
10 10-2
0.8
0.8
10
10-4
0.6
10
0.6
0
20
40
60
80
100
120
100
200
300
400
500
(c) ArcFace
(d) AdaFace
benign accuracy
1
10 2
0.8
0.8
10-3
10 3
0.6
0.6
10-4
10
100
200
300
400
500
100
200
300
400
500
backdoors or principal components
backdoors principal components
Figure 4: The plots show the results of the same experiments as in Figure 2 but after first performing SVD on the last network layer and replacing the singular values with their mean. As before, the black curve and the red band show the average benign accuracies, and their standard deviation, respectively. The shaded blue area shows the explained variance of the principal components of all images from the LFW dataset, computed after the weight surgery and logarithmic scale.
For each system, we run each experiment ten times, each time using a different random order of classes within the restricted CelebA dataset to calculate the backdoor directions. We plot the average result of this ten runs and indicate the standard deviation using a shaded red region around the mean.
Computing benign accuracy and attack success rate. To compute the benign accuracy, we employ the standard ten-fold cross-validation procedure detailed in Section 2
To compute the attack success rate, after each installed backdoor, we average the proportion of pairs of inputs misclassified by the system for each individual backdoor currently installed. We take the mean over each cosine similarity threshold computed when performing the benign accuracy calculation.
A caveat of our methodology is the use of an entire CelebA class both install the backdoor and calculate the attack success rate. This choice was made due to limitations in the mumber of images per individual within the CelebA dataset. We found that the results were not significantly different if we split the classes that have moderate to high numbers of images randomly into two halves when installing a backdoor, and then use one half for implementation and the other for subsequent computations of the attack success rate.
4 The Revised Method
We identified the dichotomy of the magnitudes the singular values of the weight matrix of the last layer in the network as the main culprit behind the fast decrease benign accuracy as more backdoors are installed.
An observant attacker who wishes to implement multiple backdoors may notice a variance dichotomy in the feature space of facial recognition system, and seek to mitigate this effect to achieve higher benign accuracy and attack success rate. We show that this is indeed possible, by using revised method of implementing these backdoors that first applics transformation to the last lincar weight matrix.
To do so, we use a similar method to the backdoor hiding methodology found in Zehavi et al. 2024 Section 7.6), by first performing Singular Value Decomposition (SVD) on the last linear weight matrix and modifying the distribution of the singular values.
Concretely, we:
10

---

Published in Transactions on Machine Learning Research (06/2025)
1. Perform SVD the last linear weight matrix W to get W EE= owun0, where the o,'s are the singular values.
2. Find the mean of the singular values, a = Ei=1%
3. Replace the singular values with to construct new weight matrix, W = E ou,y and replace W with W in our facial recognition system.
We only perform this operation once, before installing the first backdoor. This reduces the dichotomy of explained variance in feature space, i.c the feature vectors now belong to a subspace with dimension approximately equal to the full dimension of feature space.
Figures 4 and [ show experiments using the revised methodology; other details of the experiment remain the same as in Figures 2 and 7 For each LFW split, the threshold is calculated as before using the nine sets. Again, we plot the explained variances for each system with the shaded blue area, calculated after replacing the last linear weight matrix with the modified version. Notably, after performing this modification, the benign accuracy of the unattacked system does not change more than 0.2% in any of the systems we consider. Also, the dips in benign accuracy and attack success rates, present in all four networks in Figures 2 and are now much less steep; in facl, AdaFace and ArcFace see this dip entirely eliminated. Importantly ly, we observe that this modified attack scheme achieves higher benign accuracies and attack success rates compared with the original for ranges of backdoor numbers around the climinated dips (sce Figure 15 for clearer comparison of the two hods)) This allows the allacker to stealthily and effectively install significantly greater number of backdoors compared to the previous method.
This modified method docs not cause this backdoor to take significantly longer amount of time to install. The SVD of the last linear weight matrix is quick (up to several seconds in our experiments), since the dimension of the feature space is low.
5 Synthetic data
Even with simple synthetic data, we can see how the benign accuracy decreases as the number of backdoors increases and we can obtain plots similar to those in Figures 2 and $ Each of the four architectures we study has optimal threshold, 1, which is compared to the angle between two vectors in feature space to determine whether the two images are classified as belonging to the same individual or not. This threshold is precomputed using the LFW dataset. We use this same threshold to study our synthelic data.
To generate the synthetic data, we compute the cosine of the angles of the vectors in feature space for every pair in the LFW dataset. Using these angles we record the mean, m and m', and standard deviation, s and 8', of matching pairs and mismatching pairs, respectively. With this information alone, we generate synthetic data. To obtain a matched pair, we sample cos(pp of an angle &p from a Gaussian distribution (restricted to []) with mcan and standard deviation We then sample two random unit vectors that have an angle of exactly y to one another. To obtain mismatched pair we do the same but with m' and s'.
To simulate the sharp drop in explained variance, we use the above procedure to find pair of vectors in k dimensions and a pair of vectors in d - k dimensions, where depends on where the drop in explained variance is. We scale down the second pair by a factor of 1000 and then concatenate both pairs to obtain a pair of vectors d dimensions. This final pair no longer has an angle of exactly y, but the angle still close to sp
We generate 1000 matched and an equal number of mismatched pairs. The benign accuracy on matched (mismatched) pairs is the fraction of pairs of vectors that have an angle of less than (greater than) t to one another. The benign accuracy overall is the average of the benign accuracy on matched and mismatched pairs.
To investigate the effect of installing backdoors, we repeatedly project all vectors our matched and mis- matched pairs in a random direction and recompute the benign accuracy after each projection. The resulting plots for all four architectures can be found in Figure We note that many characteristies from the plots in
11

---

Published in Transactions on Machine Learning Research (06/2025)
benign accuracy
(a) Mimicking FaceNet128
(b) Mimicking FaceNet512
1
0.8
0.8
0.6
0.6
20
40
60
80
100
120
0
100
200
300
400
500
benign accuracy
(c) Mimicking ArcFace
(d) Mimicking AdaFace
1
0.8
0.8
0.6
0.6
100
200
300
400
500
0
100
200
300
400
500
number of projections
number of projections
Figure 5: To mimic each of the four networks, we plot the benign accuracy over the synthetic data after repeated orthogonal projections. The direction of projection is chosen the same way as the vectors the synthetic data. When simulating the drop in explained variance, we pick a random k-dimensional unit vector and concatenate with random (d - -dddinnnio vector of length 1/1000. If not simulating the drop, the direction is just random d-dimensional vector. The blue curves show the values for synthetic data without the drop, and they resemble the real benign accuracios when using the modified method, plotted in Figure | The red curves for the first three networks show the values for synthetic data that tries to model the drop in explained variance, and they resemble the real benign accuracies as plotted in Figure
Figures 2 and 4 can already be observed in this simple model involving vectors that chosen completely at random, except that the distribution of angles between vectors is fixed in specific way.
6 Conclusion
We considered four state-of-the-art deep neural network facial recognition systems and found that all of them exhibit variance dichotomy in feature space. Specifically, their feature vectors approximately lie in a subspace of significantly lower dimension. We demonstrated that when these classifiers are restricted to the orthogonal complement of this subspace, they still achieve surpisingly high accuracy. This is strongly linked with a double descent phenomenon in the benign accuracy of the networks when installing multiple backdoors recently devised by Zehavi et al. (2024) for anonymity/ caayymmtty/nlinkkeet /unlinkability and confusion attacks. Furthermore, showed how an attacker can be installed stealthily and effectively. Supported by an analysis of the impact of random orthogonal projections on angles between random vectors, we also offered an explanation of the link between the double descent of the benign accuracy and the variance dichotomy in feature space.
exploit this behavior to significantly increase the number of backdoors that
We envisage that our results will be of interest lo researchers in archilectures and training of deep neural networks for identity verification, as well researchers in neural backdoor/Trope Trojan attacks and defenses.
Limitations. Our experiments were performed only the four pretrained facial recognition systems detailed in Section g However, we expect that the dichotomy property features in other systems and is common phenomenon. Our study of the link between the dichotomy property and the behavior of the backdoor attacks by Zehavi et al. (2024) is specific to this kind of weight manipulation scheme.
Future work. As we did not rely on any aspects of facial recognition other than to work with the standard datasets in this domain, our methodology should be applicable to a broad range of identity verification systems based on the "Siamese" deep neural network architecture. It would be interesting to examine state-
12

---

Published in Transactions on Machine Learning Research (06/2025)
of-the-art systems that work with other biometrics for the variance dichotomy in their feature spaces, the double descent of their benign accuracies under multiple backdoors, and the attack scheme enhancement discussed in Section 4 We envisage that many networks in this space would see comparable results to those shown here, based on the current understanding that the intrinsic dimension of data representations tends to decrease progressively the final layers of these networks (Ansuini al., 2019 It would also be interesting see if other attack schemes are affected by variance dichotomy feature spaces.
Our explanation of the link between the double descent and the variance dichotomy could be refined and further supported by deriving general theoretical bounds that in particular shed light on the behavior of the benign accuracy around its local minimum.
Broader impact statement. This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must specifically highlighted here.
Our research on backdoor attacks and defenses of identity verification systems based on deep neural networks is intended to highlight weaknesses, that those using them better understand the risks involved.
We have shown that relying on noticing reduced accuracy in the system may not be enough to identify a compromised system, even in the event of multiple backdoors. Instead, potential safeguards that avoid this reliance include:
Automated checking of the rank of the final layer matrix. Each backdoor reduces this by one due to the projection applied.
Automated checking of variance dichotomy in feature space.
Using an ensemble of norksrks to verify identity to mitigate the effect if one is compromised.
Acknowledgments
This research was supported by the Centre for Discrete Mathematics and Its Applications (DIMAP) at the University of Warwick and the Engineering and Physical Sciences Research Council through the Mathematics of Systems 11 Centre for Doctoral Training at the University of Warwick (grant number EP /SS02244//1) We acknowledge the Scientific Computing Rescarch Technology Platform at the University of Warwick for providing the compute cluster on which the experiments presented in this paper were run.
References
Shengwei An, Yuan Yao, Qiuling Xu, Shiqing Ma, Guanhong Tao, Siyuan Cheng, Kaiyuan Zhang, Yingqi Liu, Guangyu Shen, lan Kelk, and Xiangyu Zhang. ImU: Physical Impersonating Attack for Face Recognition System with Natural Style Changes In 2023 IEEE Symposium on Security and Privacy (SP), pp. 899-916, 2023.
Alessio Ansuini, Alessandro Laio, Jakob H. Macke, and Davide Zoccolan. Intrinsic dimension of data repre-
sentations in deep neural networks) In NewrIPS, pp. 6109 6119, 2019. 5 13
Jiawang Bai, Baoyuan Wu, Zhifeng Li, and Shu-Tao Xia. Versatile Weight Attack via. Flipping Limited Bits. IEEE Trans. Pallern Anal. Mach. Intell., 45(11):13653 13665, 2023.
Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. Recent advances adversarial training for adversarial robustness) Xio preprind arXiv:2102.00 01136660, 2021. 5
Jane Bromley, James W. Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. Signature Verification using a "Siamese" Time Delay Neural Network Int. J. Pattern Recognit. Artīf. Intell., 7(4):669 688, 1993.
Jinyin Chen, Haibin Zheng, Mengmeng Su, Tianyu Du, Chang- Thang-Ting Lin, and Shouling Ji. Invisible Poisoning:
Highly Stealthy Targeted Poisoning Attack Tn Inserypt, pp. 173 198, 2019. 5
13

---

Published in Transactions on Machine Learning Research (06/2025)
Jinyin Chen, Longyuan Zhang, Haibin Zheng. Xucke Wang, and Zhaoyan Ming. Deeppoison: Feature
transfer based stealthy poisoning attack for drns) TEEE Transactions on Circuils and Systems II: Pxpress Briefs, 68(7):2618 2622, 2021.
Jiankang Dong, Jia Guo, Jing Yang, Niannan Xue, Irene Kotsia, and Stefanos Zafeiriou. ArcFace: Additive Angular Margin Loss for Deep Face Recognition IEEE Trans. Pattern Anal. Mach. Intell., 44(10):5962 5979, 2022.
9
Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li. Lira: Learnable, imperceptible and robust backdoor
attacks In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11966-11976, 2021.
Khoa D Doan, Yingjie Lao, and Ping Li. Marksman backdoor: Backdoor attacks with arbitrary target class.
Advances in Newral Information Processing Systems, 35:38260-38273, 2022. 5
Jacob Dumford and Walter J. Scheirer. Backdooring Convolutional Neural Networks via Targeted Weight
Perturbations In IJCB, pp. 1-9, 2020. 5
Yinghua Gao, Yiming Li, Linghui Zhu, Dongxian Wu, Yong Jiang, and Shu-Tao Xia. Not All Samples Are
Born Equal: Towards Effective Clean-Label Backdoor Attacks Pattern Recognition, 139:109512, 2023.
ISSN 0031-3203. 5
Hu Han, Jie Li, Anil K. Jain, Shiguang Shan, and Xilin Chen. Tattoo Image Search at Scale: Joint Detection and Compact Representation Learning IEEE Trans. Pallern Anal. Mach. Intell., (1(0))2333 2348, 2019.
Zhanhao Hu, Siyuan Huang, Xiaopei Zhu, Fuchun Sun, Bo Zhang, and Xiaolin Hu. Adversarial Texture for Fooling Person Detectors in the Physical World, In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognétion (CVPR), pp. 13307-13316, June 2022.
Zhanhao Hu, Wenda Chu, Xiaopei Zhu, Hui Zhang, Bo Zhang, and Xiaolin Hu. Physically Realizable Natural- Looking Clothing Textures Evade Person Detectors via 3D Modeling, In Proceedings of the IEEE/O /CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16975-16984, June 2023. 5
Gary B. Huang, Marwan Mattar, Tamara Berg, and Eric Bearned-Millen Labeled faces in the wild: A database for studying face recognition in unconstrained environments. In Workshop faces in 'Real-Life" Images: detection, alignment, and recognition, 2008.
Rishi Jha, Jonathan Hayase, and Sewoong Oh. Label poisoning is all you need Advances in Neural Infor-
mation Processing Systems, 36:71029- 77052, 2023. 5
Minchul Kim, Anil K Jain, and Xiaoming Liu. AdaFace: Quality Adaptive Margin for Face Recognition In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recográtion, 2022. 3 9
Ruggero Donida Labati, Enrique Muñoz Ballester, Vincenzo Piuri, Roberto Sassi, and Fabio Scotti. Deep- ECG: Convolutional Neural Networks for ECG biometric recognition Pattern Recogrél. Lett., 126:78 85, 2019.
Junyu Lin, Lei Xu, Yingqi Liu, and Xiangyu Zhang. Composite Backdoor Attack for Deep Neural Network by Mixing Existing Benign Features) In CCS, pp. 113 131, 2020. 5
Li Liu, Linlin Huang, Fei Yin, and Youbin Chen. Offline signature verification using a region based deep metric learning network, Pattern Recognét.. 118:108009, 2021.
[
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. SphereFace: Deep Hypersphere Embedding for Face Recognition) In CVPR, pp. 6738 6746, 2017a. 2
Yannan Liu, Lingxiao Wei, Bo Luo, and Qiang Xu. Fault injection attack on deep neural network In ICCAD, pp- 131-138, 2017b. 5
14

---

Published in Transactions on Machine Learning Research (06/2025)
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Attributes in the Wild In ICCV, pp. 3730-3738, 2015. 9
Depen Morwani, Jatin Batra, Prateek Jain, and Praneeth Netrapalli. (Simplicity bias in 1-hidden layer neural networks In NewrIPS, volume 36, pp. 8048 8075, 2024. 5
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diflusion models for adversarial purification In International Conference on Machine Learning (ICML), 2022. 5
Xiangyu Qi. Tinghao Xie, Ruizhe Pan, Jifeng Zhu, Yong Yang, and Kai Bu. Towards Practical Deployment-
Stage Backdoor Attack on Deep Neural Networks In CVPR, pp. 13337-13347, 2022. 5
Andrew Ross and Finale Doshi Velez. Improving the adversarial robustness and interpretability of deep
neural networks by regularizing their input gradients In Proceedings of the AAAT conference on artificial intelligence, volume 32, 2018.
Esha Sarkar, Hadjer Benkraouda, Gopika Krishnan, Homer Gamil, and Michail Maniatakos. FaceHack:
[Attacking Facial Recognition Systems Using Malicious Facial Characteristics IEEE Trans. Biom. Behav. Identity Sci., 4(3):361-33 -3221 2022.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A unilied embedding for face recognition and clustering In CVPR, pp. 815-823, 2015. 3
Alireza Sepas-Moghaddam and Ali Etemad. Deep Gait Rocognition: A Survey
JEEE Trans. Pattern Anal.
Mach. Intell., 45(1)) 284, 2023. 1
Ali Shafahi, W. Ronny Huang. Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom In NewrIPS, pp.
Goldstein. Poison Frogs! Targeted Clean- Poisoning Attacks on Neural Networks
2018. 5
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. A General Framework for Adver- sarial Examples with Objectives ACM Trans. Priv. Secur., 22(3):16:1-16:30 2019.
Inderjeet Singh, Satoru Momiyama, Kazuya Kakizaki, and Toshinori Araki. On brightness agnostic ad- versarial examples against face recognition systems In 2021 International Conference of the Biometrics
Special Interest Group (BIOSIG), pp. 1-5. IEEE, 2021. 5
Jialiang Sun, Wen Yao, Tingsong Jiang, Donghua Wang, and Xiaoqian Chen. Differential evolution basod
dual adversarial camoullage: Fooling human eyes and object detectors Newral Networks, 163:256 271,
2023. ISSN 0893-6080. 5
Yaniv Taigman, Ming Yang, Aurcio Ranzato, and Lior Wolf. DeepFace: Closing the Gap to Human- Level Performance in Face Verification. In CVPR, pp. 1701-1708, 2014.
Yao Tang, Fei Gao, Jufu Feng, and Yuhang Liu. FingerNet: An unified deep network for fingerprint minutiac extraction In IJCB, pp.
IEEE, 2017.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. CosFace: Large Margin Cosine Loss for Deep Face Recognition, In CVPR, pp. 5265 5274, 2018.
Xingxing Wei, Ying Guo, and Jie Yu. Adversarial Sticker: A Stealthy Attack Method in the Physical World IEF. Transactions on Pattern Analysis and Machine Intelligence, 45(3):2711-277 1725 2023.
Mingfu Xue, Can He, Yinghao Wu, Shichang Sun, Yushu Zhang, Jian Wang, and Weiqiang Liu. PTB: Robust physical backdoor attacks against deep neural networks in real world Comput. Secur., 118:102726, 2022.
Bangjie Yin, Wenxuan Wang, Taiping Yao, Junfeng Guo, Zelun Kong, Shouhong Ding, Jilin Li, and Cong Liu. Adv--aakup:: A New Imperceptible and Transfcrable Attack on Face Recognition In Zhi-Hua Zhou (od.), Proceedings of the Thirticth International Joint Conference on Artificial Intelligence, IJCAI-21, pp.
1252-1258. International Joint Conferences Artificial Intelligence Organization, 8 2021. 5
15

---

Published in Transactions on Machine Learning Research (06/2025)
Irad Zehavi, Roce Nitzan, and Adi Shamir. Facial Misrecognition Systems: Simple Weight Manipulations Force DNNs to Err Only Specific Persons 2024.
2
4
10
12
Stephan Zheng, Yang Song, Thomas Leung, and lan Goodfellow. Improving the robustness of deep neural networks via stability training In Proceedings of the ieee conference on computer vision and pattern
recognition, pp. 1480-4488, 2016.
Yaoyao Zhong, Weihong Deng, Jiani Hu, Dongyue Zhao, Xian Li, and Dongchao Wen. SFace: Sigmoid- Constrained Hypersphere Loss for Robust Face Recognition IEEE Trans. Image Process., 30:2587-2595, 2021.
Zheng-Ar An Zhu, Yun-Zhong Lu, and Chen-Kuo Chiang, Generating Adversarial Examples By Makeup Atackk on Face Recognition, In ICIP, pp. 2516 2520, 2019.
Alon Zolfi, Shai Avidan, Yuval Elovici, and Asaf Shabtai. Adversarial Mask: Real-World Universal Adver- sarial Attack on Face Recognition Models) In Machine Learning and Knowledge Discovery in Databoses: European Conference, ECML PKDD 2022, Grenoble, France, September 19 23, 2022, Proceedings, Part
III, pp. 320. Springer, 2023. 5
16

---

Published in Transactions on Machine Learning Research (06/2025)
A
Appendix
(a) FaceNet122 128
FaceNe1512
10-2
10 10-2
explained variance
singular value
10-4
10-4
10-6
0.5
10-6
10
20
40
60
100
120
100
200
300
400
500
(c) ArcFace
(d) AdaFace
10-2
explained variance
singular value
10 10-3
3
10
2
10
0.5
10-6
10-7
100
200
300
400
500
100
200
300
400
500
component
component
Figure 6: For each network, the singular values for the last linear weight matrix and the explained variance of the principal components of all images from the LFW dataset in feature space are overlaid for comparison.
Table 2: (s, dicchhoyy values for sets of feature vectors from various combinations of models and datasets after replacing each singular value of the last lincar weight matrix with its mcan.
LFW
CelebA
Model
8
8
j
1aceNet128
0.195
0.501
53
78
0.188
0.504
53
77
FaceNet512
0.041
0.500
39
60
0.035
0.502
44
62
ArcFace
0.287
0.501
147
0.291
0.502
0
149
AdaFace
0.219
0.503
399
511
0.209
0.502
402
509
17

---

Published in Transactions on Machine Learning Research (06/2025)
(a) FaceNet128
(b) FaceNet512
1
attack success rate
10-2
10
10
10
0.8
10 10-4
0.8
10 6
10 6
0.6
0.6
10
20
40
60
80
100
120
100
200
300
400
500
(c) ArcPace
(d) Ada Aace
attack sucoess rate
10-2
explained variance
10-3
0.8
0.8
10-1
10
0.6
0.6
10-6
10
100
200
300
400
500
100
200
300
400
500
backdoors or principal components
backdoors or principal components
Figure 7: The black curves show the attack success rates after a number of Shattered Class backdoors are installed. The success rate of backdoor is the proportion of pairs of inputs from CelebA class or classes that are misclassified by the system. These rates are averaged over all the backdoors currently installed, over the len standard benchmark 90%-10% splits with the LFW dataset. This is repeated ten times, with different random choices of backdoored CelebA classes, and the average is plotted. The red band shows the standard deviation. In addition, the shaded blue area shows the explained variance of the principal components of all images from the LFW dataset in feature space, on logarithmic scale.
18

---

Published in Transactions on Machine Learning Research (06/2025)
(a) 2aceNet128
(b) FaceNet512
attack success rate
variance
10-2
10-2
0.8
0.8
10
explained
10
0.6
10-4
0.6
20
40
60
80
100
120
100
200
300
400
500
(c) ArcFace
(d) AdaFace
1
attack success rate
explained variance
10
0.8
0.8
10
10-3
0.6
0.6
10-1
10
100
200
300
400
500
100
200
300
400
500
backdoors or principal components
backdoors or principal components
Figure 8: The plots show the results of the same experiments as in Figure 7 but after first performing SVD on the last network layer and replacing the singular values with their mean. As before, the black curve and the red band show the average benign accuracies, and their standard deviation, respectively. The shaded blue area shows the explained variance the principal components all images from the LFW dataset, computed after the weight surgery and on a logarithmic scale
19

---

Published in Transactions on Machine Learning Research (06/2025)
(a) FaceNet128
(b) FaceNet512
1
10
10-2
explained variance
benign accuracy
0.8
0.8
10
10-4
10-6
0.6
0.6
10-6
10 8
0.4
0.4
0
20
60
80
100
120
100
200
300
400
500
(c) ArcFace
(d) AdaFace
10-2
explained variance
benign accuracy
10-3
0.8
0.8
10-4
10-5
0.6
0.6
10-0 6
0.4
0.4
10
0
100
200
300
400
500
100
200
300
400
500
backdoors or principal components
backdoors or principal components
Figure 9: The black curves depict the benign accuracy of the networks as increasing numbers of concurrent Merged Class backdoors are installed, measured using the LFW dataset. Each backdoor applied is Merged Class, with randomly picked individuals from the GelebA dataset (which have not. been used in a previously installed backdoor) for which the backdoor is installed. This repeated ten times and the average is plotted. The red band shows the standard deviation. addition, the shaded blue area shows the explained variance of the principal components of all images from the LFW dataset in feature space, on a logarithmic scale.
20

---

Published in Transactions on Machine Learning Research (06/2025)
(a) 2aceNet128
(b) FaceNet512
attack success rate
10-2
10
varidna sariance
0.8
0.8
10
10 10-4
10
e esplained
0.6
0.6
10-6
10
0.4
0.4
20
40
60
80
100
120
100
200
300
400
500
(c) ArcFace
(d) AdaFace
attack succoss rate
10 2
explained variance
10
0.8
0.8
10
10-5
0.6
0.6
10
0.4
0.4
10-7
100
200
300
400
500
100
200
300
400
500
backdoors or principal components
backdoors or principal components
Figure 10: The black curves show the attack success rates after number of Merged Class backdoors are installed. The success rate of backdoor is the proportion of pairs of inputs from its ColebA class classes that are misclassified by the system. These rates are averaged over all the backdoors currently installed, over the ten standard benchmark 90%-10% splits with the LFW dataset. This is repeated ten times, with different random choices backdoored CelebA classes, and the average is plotted. The red band shows the standard deviation. In addition, the shaded blue area shows the explained variance of the principal components of all images from the LFW dataset feature space, on logarithmic scale.
21

---

Published in Transactions on Machine Learning Research (06/2025)
(a) FaceNet122 128
(b) FaceNet512
benign aecuracy
10-2
10-2
e Replained variance
10
0.8
10
0.8
10-0
10
0.6
0.6
10
20
40
60
100
120
100
200
300
400
500
(c) ArcFace
(d) AdaFace
benign accuracy
10
explained variance
10-3
0.8
0.8
10 4
10-5
0.6
0.6
10
10-7
0
100
200
300
400
500
100
200
300
400
500
backdoors or principal components
backdoors or principal components
Figure 11: The black curves depict the benign accuracy of the networks increasing numbers of concurrent backdoors are installed, measured using the LFW dataset. For cach backdoor, uniformly random attack mode (Shatter Class or Merge Class) is selected, and individuals from the CelebA dataset (not used in previous backdoors) are randomly chosen to install the backdoor. This process is repeated ten times, with the mean plotted and the red band indicating the standard deviation. The shaded blue area represents the explained variance of the principal components of all LFW dataset images in feature space, plotted on a logarithmic scale.
22

---

Published in Transactions on Machine Learning Research (06/2025)
(a) FaceNet128
(b) FaceNet512
attack success rate
10-2
10
explainn asplained variaad aariaaee
10
0.8
10
0.8
10
10-5
0.6
0.6
10
20
40
60
80
100
120
100
200
300
400
500
(c) ArcFace
(d) AdaFace
1
attack sucooss rate
10-2 2
explained variance
10
0.8
0.8
10
10-5
0.6
0.6
10
10-7
100
200
300
400
500
100
200
300
400
500
backdoors or principal components
backdoors or principal components
The black curves show the attack success rates after mumber of backdoors installed. The success rate of a backdoor is the proportion of pairs of inputs from its CelebA class or classos that are misclassified by the system. These rates are averaged over all the backdoors currently installed, over the ten standard benchmark 90%-10% splits with the LFW dataset. This is repeated ten times over random choices of the two backdoor types and the backdoored CelebA classes, and the average is plotted. The red band shows the standard deviation. addition, the shaded blue area shows the explained variance of the principal components of all images from the LFW dataset in feature space, on logarithmic scale.
Figure 12:
23

---

Published in Transactions on Machine Learning Research (06/2025)
(a) FaceNet128
(b) FaceNet512
explained varianco
benign accuracy
10
10-2
0.8
0.8
10-3
10-3
10
0.6
10-4
0.6
10-5
20
60
80
100
120
100
200
300
400
500
(c) ArcFace
(d) AdaFace
1
explained variance
benign accuracy
10-2
0.8
0.8
10 10-3
10-3
0.6
0.6
10 4
10-1
0
100
200
300
400
500
100
200
300
400
500
backdoors or principal components
backdoors or principal components
Figure 13: The plots show the results of the same experiments in Figure 11 but after first performing SVD on the last network layer and replacing the singular values with their mean. As before, the black curve and the red band show the average benign accuracies, and their standard deviation, respectively. The shaded blue area shows the explained variance of the principal components of all images from the LFW dataset, computed after the weight surgery and on a logarithmic scale.
24

---

Published in Transactions on Machine Learning Research (06/2025)
(a) FaceNet128
(b) FaceNet512
attack success rate
variance
10-2
10
0.8
0.8
10
10-3
explained
10
0.6
10-4
0.6
10
20
40
60
80
100
120
100
200
300
400
500
(c) ArcFace
(d) AdaFace
1
attack success rate
explained variance
10
0.8
0.8
10
10-3
0.6
0.6
10-1
10
100
200
300
400
500
100
200
300
400
500
backdoors or principal components
backdoors or principal components
The plots show the results of the same experiments in Figure 12 but after first performing SVD on the last network layer and replacing the singular values with their mean. As before, the black curve and the red band show the average benign accuracies, and their standard deviation, respectively. The shaded blue area shows the explained variance of the principal components of all images from the LFW dataset, computed after the weight surgery and on a logarithmic scale
Figure 14:
25

---

Published in Transactions on Machine Learning Research (06/2025)
(a) 11c
(c) FaceNet512
benign accuracy
0.8
0.8
0.6
0.6
20
40
60
80
100
120
100
200
300
400
500
(b) ArcFace
(d) AdaFace
1
benign accuracy
0.8
0.8
0.6
0.6
100
200
300
400
500
0
100
200
300
400
500
number of backdoors
number of backdoors
The benign accuracy from the original method, shown in Figure and from the revised method, as shown in Figure 4 are overlaid for easier comparison. For example, for FaceNet512 we see that after approximately 70 backdoors, the revised method performs better until around 300 backdoors at which point they are very similar again.
Figure 15:
(a) FaceNet128
(b) FaceNet512
attack success rale
0.8
0.8
0.6
0.6
20
40
60
80
100
120
100
200
300
400
500
(c) ArcFace
(d) AdaFace
1
uttack success rale
0.8
0.8
0.6
0.6
100
200
300
400
500
100
200
300
400
500
number of backdoors
number of backdoors
Figure 16: The attack success rate from the original method, as shown in Figure and from the revised method, as shown in Figure are overlaid for easier comparison.
26

---

Published in Transactions on Machine Learning Research (06/2025)
(a) FaceNet 128
(c) FaceNet512
BA/ASR
0.8
0.8
0.6
0.6
20
40
60
80
100
120
0
100
200
300
400
500
(b) ArcFace
(d) AdaFace
1
BA/ASR
0.8
0.8
0.6
0.6
100
200
300
400
500
100
200
300
400
500
number of backdoors
mumber of backdoors
Figure 17: The benign accuracy (red) from the original method, as shown in Figure and the attack success rate (purple), as shown in Figure are overlaid for casier comparison.
27

---

Published in Transactions on Machine Learning Research (06/2025)
Original method
Revised method
200
Iacenet
Facenet512
20
Arclace
600
36
2
adaface
Figure 18: These surface plots show the explained variances of LFW dataset feature vectors as increasing numbers of concurrent Shattered Class backdoors are installed, both for the original method and our revised method. Other experimental details remain the same as in Figure)
28


Reviews:
[{"review_id": "ucNZRF13Ie", "reviewer": "Reviewer_9v8r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Overall, it's easy to follow.\n2. Evaluated four models.\n\nWeaknesses:\n1. The goal of this paper is unclear. Is it a defense or an attack? From the title, it seems a defense. However, from the main text, it seems an attack that improved the existing attack. Or is it just analyzing one existing attack? What is the takeaway to the audience?\n2. Because the paper only talks about one existing attack, the analysis and results may not be generalized to other attacks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper analyzes an existing attack that manipulates the weight of the last layer in a victim facial verification network and observes a phenomenon called variance dichotomy. When the number of injected backdoors increases, the benign utility first decreases and then increases before finally decreasing. Based on this observation, the authors propose a way to transform the weight of the last layer before injecting the backdoors to mitigate this dichotomy phenomenon which can be used as a potential defense against that existing attack. Experiments are conducted on four models with LFW and CelebA.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It's suggested that the goal of this paper be clearly stated and that generalizable observations and results be provided.\n\n2. From my understanding, the explained variance is dependent on the last layer's weight and the inputs according to the description in Section 3. That is, it changes after injecting each backdoor. Then, how can one draw the shade for the variance in Figure 1-2? Because when the backdoors increase along the x-axis, the variance matrix should also change.\n\n3. If the x and y axes are the same between Figures 1 and 2, the two figures may be merged for easier comparison between the trends of benign utility and backdoor ASR.\n\n4. It's unclear why the authors randomly chose two different backdoors. If one backdoor is used, will the same phenomenon be observed?\n\n5. According to Definition 1, $\\epsilon$ and $\\delta$ are not unique for each network and dataset. How are they decided, and why does the paper choose $\\delta=0.5$? Why are the values of $\\epsilon$ for Facenet512 in Table 1 0?\n\n6. In Section 4, is SVD needed for each backdoor? For example, injecting the first backdoor requires SVD on the benign weight. Does injecting the second backdoor require SVD on the already modified weight?\n\n7. Could you further explain why a synthetic dataset is needed in Section 5? It may not correspond to any real-face dataset. What extra evidence does it provide to support the paper? \n\n4. It would be better to mention some defenses in the related work. Also, since the authors mentioned physical adversarial attacks in the related work, it's suggested to add https://ieeexplore.ieee.org/document/10179360", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PgN6YYGxvw", "reviewer": "Reviewer_Urxe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts**\n\nI believe this paper to present a very interesting phenomenology. This double descent phenomenon as a function of number of backdoors is, in my opinion, very interesting, and quite unexpected. The authors present their intuition, which connects this non-regular descent to a sort of spectral gap in the covariance of the features, and their proposed method to remove this \"bump\" seems to be working, supporting their hypothesis.\n\n**Weaknesses**\n\nI believe this paper to have quite a few weaknesses, listed below:\n\n1 - At the moment, the narrative has a relative restricted scope. In other words, the open question this paper aim to address is the one at the end of page 2, which is very specific and very focused on an open istance left behind by Zehavi et al. (2024). From the perspective of a reader without strong familiarity with the area, this research goal is narrow and fairly not motivated. I recommend the authors to modify the introduction to make the narrative more engaging, for example describing the setting and the experiment from the very beginning, discussing the surprising phenomenology of Figure 1, and leaving the study and understanding of this problem as the purpose of the work. This is not a major request, I leave to the authors the ultimate choice regarding this.\n\n2 - The setting of the experiments is often not very clear. First, the _explained variances_ firstly introduced in Section 1.1 are not preoperly defined or linked to any definition. Even when better introduced in Section 3, the expanation is not very smooth: why do the authors mention PCA (that should generally more refer to a dimensionality reduction technique)? Are the authors computing the empirical covariance of the features of the penultimate layer on the target dataset? How large is this dataset and how large is the space of these features? I presume the authors report the renormalized eigenvalue of such empirical covariance.\n\nRegarding the same point, at point 4 in Section 1.1 (later remarked in the last paragraph of Section 2) the Authors mention that they take random choices of the attack. What does this mean? This, to me, seems like a very confusing variable in trying to understand what are the causes of the double descent phenomenon. Why did the Authors follow this choice, rather than focusing on one single attack, or carrying the study separately? Furthermore, computing the accuracy of the two attacks should be qualitatively different (the first one is in-class, while the second one regards two target classes). Thus, it is also not obvious to me how the attack success rate is exactly computed in Figure 2, for example. Also, how is the cosine similarity threshold computed? To optimize precision or recall? On which attack type?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is an empirical paper providing a more in depth analysis of the adversarial attack introduced by Zehavi et al. (2024), which involves manipulating the weights of the last layer of a deep neural network, to affect the capabilities of a facial recognition system on specific targets. The paper mainly explores how many backdoors can be inserted in the model (number of targets) without eccessively damaging the performance of the model on independent samples, framed as benign accuracy, and computed on a separate dataset. The authors identify a maybe surprising double descent behaviour of the benign accuracy with respect to both the benign accuracy and the success of the attack rate (the proportion in which a pair of the target dataset is misclassified).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My major requested changes regard Weakness number 2. I would appreciate if the authors introduce more definitions, maybe formalizing the exact quantities in the plots via equations and proper notation. The phenomenon investigated in this paper is in my opinion, very interesting, but requires a solid setting definition, also to enable later theoretical investigation by future work.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tGmH95ATHu", "reviewer": "Reviewer_SRe9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The discovery of ​variance dichotomy and its link to benign accuracy double descent is a significant contribution. This phenomenon explains why prior attacks (e.g., Zehavi et al.) face diminishing returns as backdoors accumulate, advancing understanding of intrinsic defense mechanisms in facial recognition systems.\n2. The work highlights critical vulnerabilities in widely used facial recognition systems, urging the community to address weaknesses in feature space design. The discussion of \"intrinsic dimension\" and simplicity bias bridges gaps between security and representation learning.\n\nWeaknesses:\n1. The theoretical explanation of the double descent phenomenon still relies on simplified models (such as random vector projection) and lacks rigorous mathematical proof.\n\n2. The causes of variance dichotomy (such as the influence of training objectives and loss functions) have not been explored in depth, which may affect the universality of the conclusions.\n\n3. Although this paper discovered the phenomenon of variance dichotomy, its intuitive explanation was not given.\n\n4. The expression of this paper needs further improvement. For example, the background and significance of the work are not very clear at present, and the correction of only one attack limits the influence of the paper. The experiments and methods are always expressed together, which is not conducive to everyone's understanding. In addition, the contribution should be simply summarized instead of lengthy output.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper analyzes the shortcomings of backdoor attacks on simple weight manipulations in only the last hidden layer. Through empirical experiments, the authors found that they exhibit a variance dichotomy in their feature spaces, which causes the benign accuracy of the attacked system to decrease fast as the number of sequentially installed backdoors increases. Finally, the authors proposed an improved backdoor attack.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Strengths And Weaknesses.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The AE highly encourages the authors to address the following two remaining concerns:

1. Clarification of Definition 2, as requested by the reviewer: https://openreview.net/forum?id=Q1Cf07flwD&noteId=dEEFCtTHIk
2. Addition of a discussion section at the end of the conclusion, outlining potential directions for future work, specifically, how one might further investigate the phenomenon through extensive evaluations and explore possible theoretical connections.

Additionally, the AE has a minor suggestion regarding the title:

1. Weak Defense from -> Weak Defense against
Decision: Accept (poster)