Year: 2021

Abstract:
Since training a large-scale backdoored model from scratch requires a large training dataset, several recent attacks have considered to inject backdoors into a trained clean model without altering model behaviors on the clean data. Previous work finds that backdoors can be injected into a trained clean model with Adversarial Weight Perturbation (AWP), which means the variation of parameters are small in backdoor learning. In this work, we observe an interesting phenomenon that the variations of parameters are always AWPs when tuning the trained clean model to inject backdoors. We further provide theoretical analysis to explain this phenomenon. We are the first to formulate the behavior of maintaining accuracy on clean data as the consistency of backdoored models, which includes both global consistency and instance-wise consistency. We extensively analyze the effects of AWPs on the consistency of backdoored models. In order to achieve better consistency, we propose a novel anchoring loss to anchor or freeze the model behaviors on the clean data, with a theoretical guarantee. Both the analytical and empirical results validate the effectiveness of our anchoring loss in improving the consistency, especially the instance-wise consistency.

heading: 1 INTRODUCTION
text: Deep neural networks (DNNs) have gained promising performances in many computer vision (Krizhevsky et al., 2017; Simonyan & Zisserman, 2015), natural language processing (Bowman et al., 2016; Sehovac & Grolinger, 2020; Vaswani et al., 2017), and computer speech (van den Oord et al., 2016) tasks. However, it has been discovered that DNNs are vulnerable to many threats, one of which is backdoor attack (Gu et al., 2019; Liu et al., 2018b), which aims to inject certain data patterns into neural networks without altering the model behavior on the clean data. Ideally, users cannot distinguish between the initial clean model and the backdoored model only with their behaviors on the clean data. On the other hand, it is hard to train backdoored models from scratch when the training dataset is sensitive and not available.
Recently, Garg et al. (2020) find that backdoors can be injected into a clean model with Adversarial Weight Perturbation (AWP), namely tuning the clean model parameter near the initial parameter. They also conjecture that backdoors injected with AWP may be hard to detect since the variations of parameters are small. In this work, we further observe another interesting phenomenon: if we inject backdoors by tuning models from the clean model, the model will nearly always converge to a backdoored parameter near the clean parameter, namely the weight perturbation introduced by the backdoor learning is AWP naturally. To better understand this phenomenon, we first give a visualization explanation as below: In Figure 1a, there are different loss basins around local minima, a backdoored model trained from scratch tends to converge into other local minima different from the initial clean model; In Figure 1b, backdoored models tuned from the clean model, including BadNets (Gu et al., 2019) and our proposed anchoring methods, will converge into the same loss basin where the initial clean model was located in.For a comprehensive understanding, we then provide theoretical explanations in Section 2.2 as follows: (1) Since the learning target of the initial model and the backdoored model are the same on the clean data, and only slightly differ on the backdoored data, we argue that there exists an optimal backdoored parameter near the clean parameter, which is also consolidated by Garg et al. (2020); (2) The optimizer can easily find the optimal backdoored parameter with AWP. Once the optimizer converges to it, it is hard to escape from it.
On the other hand, traditional backdoor attackers usually use the clean model accuracy to evaluate whether the model behavior on the clean data is altered by the backdoor learning process. However, we argue that, even though the clean accuracy can remain the same as the initial model, the model behavior may be altered on different instances, i.e., instance-wise behavior. To clarify this point, in this work, we first formulate the behavior on the clean data as the consistency of the backdoored models, including (i) global consistency; and (ii) instance-wise consistency. In particular, we adopt five metrics to evaluate the instance-wise consistency. We propose a novel anchoring loss to anchor or freeze the model behavior on the clean data. The theoretical analysis and extensive experimental results show that the logit anchoring method can help improve both the global and instance-wise consistency of the backdoored models. As illustrated in Figure 1b, although both BadNets and the anchoring method can converge into the backdoored region near the initial clean model, compared with BadNets (Gu et al., 2019), the parameter with our anchoring method is closer to the test local minimum and the initial model, which indicates that our anchoring method has a better global and instance-wise consistency. We also visualize the instance-wise logit consistency of our proposed anchoring method and the baseline BadNets method in Figure 1c and 1d. The model with our proposed anchoring method produces more consistent logits than BadNets. Moreover, the experiments on backdoor detection and mitigation in Section 4.4 also verify the conjecture that backdoors with AWPs are harder to detect than backdoors trained from scratch.
To summarize, our contributions include:
• We are the first to offer a theoretical explanation of the adversarial weight perturbation and formulate the concept of global and instance-wise consistency in backdoor learning.
• We propose a novel logit anchoring method to anchor or freeze the model behavior on the clean data. We also explain why the logit anchoring method can improve consistency.
• Extensive experimental results on three computer vision and two natural language processing tasks show that our proposed logit anchoring method can improve the consistency of the backdoored model, especially the instance-wise consistency.

heading: 2 PRELIMINARY

heading: 2.1 PROBLEM DEFINITION
text: In this paper, we focus on the neural network for a C-class classification task. Suppose D denotes the clean training dataset, θ ∈ Rn denotes the optimal parameter on the clean training dataset, where n is the number of parameters, L(D;θ) denotes the loss function on dataset D and parameter θ, and L ( (x, y);θ ) denotes the loss on data instance (x, y), here θ can be omitted for brevity. Since modern deep neural networks (DNNs) often have multiple local minima (Ding et al., 2019), and the optimizer does not necessarily converge to the global optimum, we may assume θ is a local minimum of the clean loss, ∇θL(D;θ) = 0. Assume θ∗ is a local minimum of the backdoored
loss, then we have ∇θ∗L(D ∪ D∗;θ∗) = 0, where D∗ and θ∗ represent the poisonous dataset and the backdoored parameter, respectively.

heading: 2.2 ADVERSARIAL WEIGHT PERTURBATIONS AND BACKDOOR LEARNING
text: Adversarial Weight Perturbations (AWPs) (Garg et al., 2020), or parameter corruptions (Sun et al., 2021), refer to that the variations of parameters before and after training are small. Garg et al. (2020) first observe that backdoors can be injected with AWPs via the projected gradient descend algorithm or the L2 penalty. In this paper, we observe an interesting phenomenon that, if we train from a well-trained clean parameter θ, θ∗− θ tends to be adversarial weight perturbation, namely finding the closest local minimum in the backdoored loss curve θ∗ to the initial parameter.
This phenomenon implies two properties during backdoor tuning: (1) Existence of the optimal backdoored parameter θ∗ near the clean parameter, which is also observed by Garg et al. (2020); (2) The optimizer tends to converge to the backdoored parameter θ∗ with adversarial weight perturbation. Here, θ∗ denotes the local optimal backdoored parameter with adversarial weight perturbation.
Existence of Adversarial Weight Perturbation δ. We explain the existence of the optimal backdoored parameter θ∗ near the clean parameter, which is also observed by Garg et al. (2020), in Proposition 1 and Remark 1. The formal version of Remark 1, and the proof of Proposition 1 and Remark 1 are given in Appendix.A.1. Proposition 1 (Upper Bound of ‖δ‖2). Suppose H denotes the Hessian matrix ∇2θL(D,θ) on the clean dataset, g∗ = ∇θL(D∗,θ). Assume L(D∗,θ + δ) ≤ L(D∗,θ) − |∆L∗| can ensure that we can successfully inject a backdoor. Suppose we can choose and control the poisoning ratio η = |D∗|/|D|, the adversarial weight perturbation δ (which is determined by η) is,
δ = δ(η) = −ηH−1g∗ + o(‖δ(η)‖2), (1)
To ensure that we can successfully inject a backdoor, we only need to ensure that η ≥ η0. There exists an adversarial weight perturbation δ,
‖δ‖2 ≤ |∆L∗|+ o(1)
‖g∗‖ cos〈g∗,H−1g∗〉 . (2)
Remark 1 (Informal. Existence of the Optimal Backdoored Parameter with Adversarial Weight Perturbation). We take a logistic regression model for classification as an example. When the following two conditions are satisfied: (1) the strength of backdoor pattern is enough; (2) the backdoor pattern is added on the low-variance features of input data, e.g., on a blank corner of figures in computer vision (CV), or choosing low-frequency words in Natural language processing (NLP), then we can conclude that: ‖δ‖2 is small, which ensures the existence of the optimal backdoored parameter with adversarial weight perturbation.
Proposition 1 estimates the upper bound of ‖δ‖2. In Remark 1, we investigate the conditions to ensure that ‖δ‖2 is small. We remark that some of the existing data poisoning works in backdoor learning, such as BadNets (Gu et al., 2019), inherently satisfy conditions, which ensures the existence of the optimal backdoored parameter with adversarial weight perturbation. Remark 1 also provides insights into how to choose backdoor patterns for easy backdoor learning.
The optimizer tends to converge to θ∗. We explain (2) in Lemma 1. θ∗1 is denoted as any other local optimal backdoored parameter besides θ∗. A detailed version of Lemma 1 is in Appendix.A.1. Lemma 1 (Brief Version. The mean time to converge into and escape from optimal parameters). The time (step) t for SGD to converge into an optimal parameter θ∗1 is log t ∼ ‖θ∗1 − θ‖ (Hoffer et al., 2017). The mean escape time (step) τ from the optimal parameter θ∗ outside of the basin near the local minimum θ∗ is, log τ = log τ0 + Bηlr (C1κ
−1 +C2)∆L∗ (Xie et al., 2021), where B is the batch size, ηlr is the learning rate, κ measures the curvature near the clean local minimum θ∗, ∆L∗ > 0 measures the height of the basin near the local minimum θ∗, and τ0, C1, C2 ∈ R+ are parameters that are not determined by κ and ∆L∗.
Then we explain why the optimizer tends to converge to the backdoored parameter θ∗ with adversarial weight perturbation. As Figure 1a implies, the optimal backdoored parameter θ∗ with adversarial weight perturbation is close to the initial parameter θ compared to any other local optimal
backdoored parameter θ∗1 , ‖θ∗ − θ‖ ‖θ∗1 − θ‖. According to Lemma 1, it is easy for optimizers to find the optimal backdoored parameter with adversarial weight perturbation in the early stage of backdoor learning. Modern optimizers have a large Bηlr and tend to find flat minima (Keskar et al., 2017), thus κ is small. ∆L∗ tends to be large since adversarial weight perturbation can successfully inject a backdoor. Therefore, it is hard to escape from θ∗ according to Lemma 1. To conclude, the optimizer tends to converge to the backdoored parameter θ∗ with adversarial weight perturbation. The formal and detailed analysis is given in Appendix.A.2.

heading: 2.3 CONSISTENCY DURING BACKDOOR LEARNING
text: An ideal backdoor should have no side effects on clean data and have a high attack success rate (ASR) on the data that contain backdoor patterns. The measure of ASR is easy and straightforward. Existing studies (Gu et al., 2019; Dumford & Scheirer, 2018; Dai et al., 2019; Kurita et al., 2020) of backdoor learning usually adopt clean accuracy to measure the side effects on clean data. However, Zhang et al. (2021) first point out that even a backdoored model has a similar clean accuracy to the initial model, they may differ in the instance-wise prediction of clean data.
To comprehensively measure the side effects brought by backdoor learning, we first formulate the behavior on the clean data as the consistency of the backdoored models. Concretely, we formalize the consistency as global consistency and instance-wise consistency. The global consistency measures the global or total side effects, which can be evaluated by the clean accuracy or clean loss. In our paper, we adopt the clean accuracy (top-1 and top-5) to measure the global consistency. The instance-wise consistency can be defined as the consistency of p∗i predicted by the backdoored model and pi predicted by the clean model on the clean data, or the consistency of logits s∗i and si (introduced in Section 3.1). In our paper, several metrics are adopted to evaluate the instance-wise consistency, including: (1) average probability distance, E(x,y)∈D[ ∑C i=1(p ∗ i − pi)2]; (2) average
logit distance, E(x,y)∈D[ ∑C i=1(s ∗ i − si)2]; (3) Kullback-Leibler divergence (Kullback & Leibler,
1951), KL(p||p∗) = E(x,y)∈D[ ∑C i=1 pi(log pi − log p∗i )]; (4) mean Kullback-Leibler divergence, mKL(p||p∗) = (KL(p||p∗) + KL(p||p∗))/2; and (5) the Pearson correlation of the accuracies of the clean and backdoored models on different instances (Zhang et al., 2021). Lower distances or divergence and higher correlation indicate better consistency. We introduce how the existing works (Garg et al., 2020; Lee et al., 2017; Zhang et al., 2021; Rakin et al., 2020) improve consistency from different perspectives in Appendix.B.

heading: 3 PROPOSED APPROACH
text: As analyzed in Section 2.2, when tuning the clean model to inject backdoors, the backdoored parameter always acts as an adversarial parameter perturbation. Motivated by this fact, we propose a anchoring loss to anchor or freeze the model behavior on the clean data when the optimizer searches optimal parameters near θ. According to Section 2.2, δ = θ∗− θ is a small parameter perturbation.

heading: 3.1 LOGIT ANCHORING ON THE CLEAN DATA
text: In a classification neural network with parameter θ, suppose si = si(θ,x) denotes the logit or the score for each class i ∈ {1, 2, · · · , C} predicted by the neural network, and the activation function of the classification layer is the softmax function, then we have
pi = pθ(y = i|x) = softmax([s1, s2, · · · , sC ])i = exp(si(θ,x)) C∑ i=1 exp(si(θ,x)) . (3)
Similarly, suppose s∗i = si(θ + δ,x) denotes the logit or the score for each class 1 ≤ i ≤ C predicted by the backdoored neural network with parameter θ + δ, and i = i(θ, δ,x) = s∗i − si = si(θ + δ,x) − si(θ,x) denotes the predicted logit change after injecting backdoors, then the anchoring loss on the data instance (x, y) can be defined as:
Lanchor ( (x, y);θ + δ ) = C∑ i=1 2i (θ, δ,x) = C∑ i=1 |si(θ + δ,x)− si(θ,x)|2. (4)
During backdoor learning, we adopt logit anchoring on the clean dataD. The total backdoor learning loss Ltotal consists of training loss on D ∪D∗ and the anchoring loss on the clean data D,
Ltotal(D ∪D∗;θ + δ ) = 1
1 + λ L(D ∪D∗;θ + δ
) + λ
1 + λ Lanchor(D;θ + δ
) . (5)
where λ ∈ (0,+∞) is a hyperparameter controlling the strength of anchoring loss. Difference with Knowledge Distillation. The formulation of knowledge distillation (KD) (Bucila et al., 2006) loss is ∑C i=1(s ∗ i − s′i)2, where s′i is the logit predicted by another clean teacher model, which is usually larger than the student model. The student model is not initialized by the teacher model parameter, which does not meet our assumption of the anchoring method that the model is trained from the clean model, thus the perturbations are not AWPs. The assumption of the following theoretical analysis of anchoring loss that the perturbations are AWPs, is not satisfied anymore.

heading: 3.2 WHY LOGIT ANCHORING CAN IMPROVE CONSISTENCY
text: In this section, we explain why the anchoring loss can help improve both the global consistency and the instance-wise consistency during backdoor learning.
Logit Anchoring for Better Global Consistency. Consider a linear model for example. Suppose x ∈ Rn is the input, the parameter θ consists of C weight vectors, w1,w2, · · · ,wC , where wi ∈ Rn, and the calculation of the logit is si(wi,x) = wTi x. Suppose the perturbations of weight vectors are δ1, δ2, · · · , δC , then i = δTi x. Suppose the loss function is the cross entropy loss. Proposition 2 estimates the clean loss change with the second-order Taylor expansion, here the clean loss change is defined as ∆L(D) = L(D;θ + δ)− L(D;θ). The proof for Proposition 2 is in Appendix.A.3. Proposition 2. In the linear model, suppose L = E(x,y)∈D[ C∑ i=1 2i ] denotes the anchoring loss, then,
∆L(D) = E(x,y)∈D ∑ i,j pipj( i − j)2 4 + o(L) ≤ E(x,y)∈D [ C∑ i=1 pi(1− pi) 2i ] + o(L). (6)
Proposition 2 implies that we can treat the clean loss change as a re-weighted version of the anchoring loss. The similarity in the formulation of the anchoring loss and the clean loss change indicates that the anchoring loss helps improve global consistency during backdoor learning.
Logit Anchoring for Better Instance-wise Consistency. Proposition 3 indicates that optimizing the anchoring loss is to minimize the tight upper bound of the Kullback-Leibler divergence KL(p||p∗), where p∗ is the probability predicted by the backdoored model. The proof of Proposition 3 is in Appendix.A.4. Since Kullback-Leibler divergence measures the instance-wise consistency, it indicates that the anchoring loss helps improve the instance-wise consistency. Proposition 3. The Kullback-Leibler divergence KL(p||p∗) is bounded by the anchoring loss, namely, the following inequality holds under α = 12 ,
KL(p||p∗) = C∑ i=1 pi log pi p∗i ≤ (α+ o(1)) C∑ i=1 2i . (7)

heading: 4 EXPERIMENTS

heading: 4.1 SETUPS
text: We conduct the targeted backdoor learning experiments in both the computer vision and natural language processing domains. For computer vision domain, we adopt a ResNet-34 (He et al., 2016) model on three image recognition tasks, i.e., CIFAR-10 (Torralba et al., 2008), CIFAR-100 (Torralba et al., 2008), and Tiny-ImageNet (Russakovsky et al., 2015). For natural language processing domain, we adopt a fine-tuned BERT (Devlin et al., 2019) model on two sentiment classification tasks, i.e., IMDB (Maas et al., 2011) and SST-2 (Socher et al., 2013). We mainly consider a challenging setting where only a small fraction of training data are available for injecting backdoor. We compare
our proposed anchoring method with the BadNets (Gu et al., 2019) baseline, L2 penalty (Garg et al., 2020; Lee et al., 2017), Elastic Weight Consolidation (EWC) (Lee et al., 2017), neural network surgery (Surgery) (Zhang et al., 2021) methods. Detailed experimental setups and backdoor details are reported in Appendix.B.
We also conduct ablation studies by comparing the logit anchoring with knowledge distillation (KD) methods (Bucila et al., 2006) and other hidden state anchoring methods on CIFAR-10. Here hidden statesH include the output states of the input convolution layer, four residual block layers, the output pooling layer, and the classification layer (namely logits) in ResNets. The anchoring term can also be the average square error loss of all hidden states (denoted as All Ave) or the mean of the square error losses of different groups of hidden states (denoted as Group Ave),
LAll Ave =
∑ h∈H ‖h∗ − h‖22∑
h∈H dim(h) , LGroup Ave =
1 |H| ∑ h∈H ‖h∗ − h‖22 dim(h) . (8)
where h∗ and h denote hidden states of the target model and the initial model, |H| denotes the number of hidden vectors (|H| = 7 in ResNets), and dim(h) denotes the dimension number of h. We conduct these ablation studies in order to illustrate the superiority of our logit anchoring method over knowledge distillation (KD) methods or sophisticated hidden state anchoring methods.

heading: 4.2 MAIN RESULTS
text: As validated by the main results in Table 1, on both three computer vision and two natural language processing tasks, our proposed anchoring method can achieve competitive backdoor attack success rate and improve both the global and instance-wise consistencies during backdoor learning, especially the instance-wise consistency, compared to the BadNets method and other methods. L2 penalty, EWC, and Surgery methods can also achieve better consistency compared to BadNets under most circumstances. We also adopt the metric ASR+ACC, which denotes the sum of backdoor
attack success rate (ASR) and the Top-1 accuracy (ACC), in order to evaluate the comprehensive performance of the backdoor accuracy (ASR) and the global consistency (ACC) at the same time. For all the methods, we report the gain of ASR+ACC with respect to the ASR+ACC of BadNets. Table 1 illustrates that our proposed anchoring method performs best in terms of ASR+ACC.
Ablation study. The results of the ablation study are shown in Table 2. Our proposed anchoring method significantly outperforms the knowledge distillation method in instance-wise consistency. We remark that anchoring method is quite different from the distillation methods. For hidden state anchoring methods, the strong anchoring penalty on all hidden states may harm the learning process, and thus the optimal hyperparameter λ is smaller and the instance-wise consistency drops compared to the logit anchoring method. Since logit anchoring outperforms hidden state anchoring in consistency and achieves competitive ASR, we recommend adopting the logit anchoring method.

heading: 4.3 FURTHER ANALYSIS
text: We conduct further analysis mainly on the CIFAR-10 dataset. Supplementary experimental results and analysis are provided in Appendix.C.
Results under multiple data settings. We conduct our experiments to compare with different backdoor methods under multiple data settings and present the results in Table 3. Our proposed anchoring loss can improve both the global and instance-wise consistency during backdoor learning with a small fraction of the dataset or the full dataset. Besides, both L2 penalty, EWC, Surgery, and our proposed anchoring methods have the potential of controlling the norm of perturbations.
It means that the theoretical analysis of backdoor learning with adversarial weight perturbation is applicable to their training processes since the perturbations are still small. When the full training dataset is available and the backdoored model is randomly initialized, the anchoring method would become equivalent to the knowledge distillation (KD) (Bucila et al., 2006) method with the initial clean model as the teacher model. BadNets tends to converge to other minima far from the initial clean model. L2 penalty can gain a minimal backdoor perturbation due to its inherent learning target. The perturbations of the KD method are large because no penalty of perturbations is applied, and the assumption of AWP is not satisfied. For better instance-wise consistency, we recommend initialization with the clean model.
Influence of the training data size and the hyperparameter λ. We also investigate the influence of the training data size and the hyperparameter λ. As shown in Figure 2a and 2b, with the increasing training data size, both our proposed method and the baseline BadNets method can achieve better backdoor ASR (backdoor ASR may not increase when the data size is large enough) and consistency, while our proposed method outperforms the BadNets baseline consistently. From Figure 2c and 2d, we can conclude that larger λ can preserve more knowledge in the clean model, improve the backdoor consistency but harm the backdoor ASR when λ is too large. Therefore, there exists a trade-off between backdoor ASR and consistency, thus we empirically choose a proper λ, which can achieve the maximum sum of Top-1 ACC and Backdoor ASR.

heading: 4.4 BACKDOOR DETECTION AND MITIGATION
text: We implement several backdoor detection and mitigation methods to defend against backdoored models with different backdoor methods on CIFAR-10. Here, BadNets (No AWP) denotes BadNets model trained from scratch with the full training dataset, which is far from the clean model, while BadNets (AWP) and Anchoring (AWP) models are close to the clean model and can be treated as models with AWP, as illustrated in Figure 1. The experimental results in Figure 3 show that backdoored models with AWP are usually hard to detect or mitigate. Our proposed anchoring method can further improve the stealthiness of the backdoor, as its curve is the closeast to the clean model.
Backdoor detection methods. Erichson et al. (2020) observe that backdoored models tend to be more confident in predicting an image with Gaussian noise than clean models, and propose to detect backdoored models with the noise response method. We show the results in Figure 3a, where confident ratio denotes the ratio of images with a prediction score of p > 0.95. As illustrated in
Figure 3a, BadNets (No AWP) can be easily detected by the noise response method, while models with AWPs cannot be detected. We also implement the backdoor detection method via targeted Universal Adversarial Perturbation (UAP) (Moosavi-Dezfooli et al., 2017) with the backdoor target as the target label following Zhang et al. (2020). From the results in Figure 3b, we can rank the detection difficulty as follows: Anchoring (AWP) > BadNets (AWP) > BadNets (No AWP).
Backdoor mitigation methods. Yao et al. (2019) propose to use standard fine-tuning to mitigate backdoor. Li et al. (2021) propose Neural Attention Distillation (NAD) to mitigate backdoor. In our experiment, we split 2000 samples for fine-tuning or NAD, and adopt another clean model as the teacher in NAD. As shown in Figure 3c, the order of the mitigation difficulty is the same as Figure 3b, i.e., Anchoring (AWP) > BadNets (AWP) > BadNets (No AWP). Standard fine-tuning can mitigate backdoor in BadNets (No AWP) completely, mitigate backdoor in BadNets (AWP) partly, but fail to mitigate backdoor in Anchoring (AWP). Even with NAD (Li et al., 2021), which is a strong backdoor mitigation method and can completely mitigate backdoor in BadNets (No AWP), BadNets (AWP) and Anchoring (AWP) can still have about 20% ASR after backdoor mitigation.

heading: 5 RELATED WORK
text: Backdoor Attacks. Backdoor attacks (Gu et al., 2019) or Trojaning attacks (Liu et al., 2018b) pose serious threats to neural networks, where backdoors may be maliciously injected by data poisoning (Muñoz-González et al., 2017; Chen et al., 2017). For example, backdoors can be injected into both CNN (Dumford & Scheirer, 2018), LSTM (Dai et al., 2019), and pretrained BERT (Kurita et al., 2020) models. Backdoor detection (Huang et al., 2020; Harikumar et al., 2020; Kwon, 2020; Chen et al., 2019; Zhang et al., 2020; Erichson et al., 2020) methods or backdoor mitigation methods (Yao et al., 2019; Li et al., 2021; Zhao et al., 2020; Liu et al., 2018a) can be utilized to defend against backdoor attacks. Backdoor detection methods usually identify the existence of backdoors in the model, via the responses of the model to input noises (Erichson et al., 2020) or universal adversarial perturbations (Zhang et al., 2020). Typical backdoor mitigation methods mitigate backdoors via fine-tuning, including direct fine-tuning models (Yao et al., 2019), fine-tuning after pruning (Liu et al., 2018a), and fine-tuning guided by knowledge distillation (Li et al., 2021).
Reducing Side-effects in Backdoor Learning or Continual Learning. Backdoor learning can be modeled as a continual learning task, which may inject new data patterns into models while conserving the knowledge in the initial model and avoiding catastrophic forgetting (Lee et al., 2017). Garg et al. (2020) propose to inject backdoors by Adversarial Weight Perturbations (AWPs). Rakin et al. (2020) propose to inject backdoors via targeted bit trojans. Yang et al. (2021) propose to inject backdoors into word embeddings of NLP models for minimal side-effects. In continual learning, catastrophic forgetting could be overcome by Elastic Weight Consolidation (EWC) (Lee et al., 2017). In backdoor learning, instance-wise side effects can be reduced by neural network surgery (Zhang et al., 2021), which only modifies a small fraction of parameters, and Dumford & Scheirer (2018) also observe that reducing the number of modified parameters can significantly reduce the alteration to the behavior of neural networks.

heading: 6 CONCLUSION
text: In this work, we observe an interesting phenomenon that the variations of parameters are always adversarial weight perturbations (AWPs) when tuning the trained clean model to inject backdoors. We further provide theoretical analysis to explain this phenomenon. We firstly formulate the behavior of maintaining accuracy on clean data as the consistency of backdoored models, including both global consistency and instance-wise consistency. To improve the consistency of the backdoored and clean models on the clean data, we propose a logit anchoring method to anchor or freeze the model behavior on the clean data. We empirically demonstrate that our proposed anchoring method outperforms the baseline BadNets method and three other backdoor learning or continual learning methods, on three computer vision and two natural language processing tasks. Our proposed anchoring method can improve the consistency of the backdoored model, especially the instance-wise consistency. Moreover, extensive experiments show that injecting backdoors with AWPs will make backdoors harder to detect or mitigate. Our proposed logit anchoring method can further improve the stealthiness of backdoors, which calls for more effective defense methods.

heading: APPENDIX

heading: A TECHNICAL DETAILS OF THEORETICAL ANALYSIS

heading: A.1 TECHNICAL DETAILS OF EXISTENCE OF ADVERSARIAL WEIGHT PERTURBATION
text: We first prove Proposition 1 and then provide more technical details of Remark 1.
Proposition 1 (Upper Bound of ‖δ‖2). Suppose H denotes the Hessian matrix ∇2θL(D,θ) on the clean dataset, g∗ = ∇θL(D∗,θ). Assume L(D∗,θ + δ) ≤ L(D∗,θ) − |∆L∗| can ensure that we successfully inject a backdoor. Suppose we can choose the poisoning ratio, and can control η = |D∗|/|D|, the adversarial weight perturbation δ (δ = δ(η) is determined by η) is,
δ = δ(η) = −ηH−1g∗ + o(‖δ(η)‖2) (9)
and to ensure that we can successfully inject a backdoor, we only need to ensure that η ≥ η0. There exists an adversarial weight perturbation δ,
‖δ‖2 ≤ |∆L∗|+ o(1)
‖g∗‖ cos〈g∗,H−1g∗〉 (10)
Proof. The definition of the loss function on the dataset can be rewritten as,
L(D;θ) = 1 |D| ∑ (x,y)∈D [L ( (x, y);θ ) ], L(D ∪D∗;θ) = 1 1 + η ( L(D;θ) + ηL(D∗;θ) ) (11)
According to the definition of optimal parameters,
∇θL(D;θ) = 0, ∇θL(D;θ∗) + η∇θL(D∗;θ∗) = 0 (12)( ∇θL(D;θ∗)−∇θL(D;θ) ) + η∇θL(D∗;θ∗) = ( Hδ + o(‖δ‖2) ) + ηg∗ = 0 (13)
Therefore, the following equation holds,
δ = −ηH−1g∗ + o(‖δ‖2) (14)
Adopting Tayler expansion, the clean loss change and backdoor loss change after injecting backdoor δ can be written as follows,
L(D;θ + δ)− L(D;θ) = 1 2 δTHδ + o(‖δ‖22) =
η2
2 g∗TH−1g∗ + o(‖δ‖22) (15)
L(D∗;θ + δ)− L(D∗;θ) = δTg∗ + o(‖δ‖2) = −ηg∗TH−1g∗ + o(‖δ‖2) (16)
To ensure that L(D∗;θ + δ) < L(D∗;θ)− |∆L∗|, we only need to choose η0 = |∆L ∗|
g∗TH−1g∗ + o(1)
and ensure η ≥ η0. There exists an adversarial weight perturbation, δ = δ(η0) that,
δ(η0) = − |∆L∗|H−1g∗
g∗TH−1g∗ + o(‖δ(η0)‖2) (17)
‖δ(η0)‖2 = |∆L∗|‖H−1g∗‖2(1 + o(1))
‖g∗‖2‖H−1g∗‖2 cos〈g∗,H−1g∗〉 = |∆L∗|+ o(1) ‖g∗‖ cos〈g∗,H−1g∗〉
(18)
Therefore, there exists δ that can successfully inject a backdoor with ‖δ‖2 ≤ ‖δ(η0)‖2.
Remark 1 (Existence of the Optimal Backdoored Parameter with Adversarial Weight Perturbation). We take a logistic regression model for a two-class classification task as an example. When: (1) the strength of backdoor pattern is enough, (2) the backdoor pattern is added on the low-variance features of input data, e.g., on the blank corner of figures in CV, or choosing a low-frequency word in NLP, then: ‖δ‖2 is small, which ensures the existence of the optimal backdoored parameter with adversarial weight perturbation.
The formally stated conditions in Remark 1 is that, when: (1) ‖g∗‖2 is large enough, (2) the direction of g∗ is close to that of H−1g∗, namely, the direction of g∗ is close to the eigenvector of H with the minimum eigenvalue, then ‖δ‖2 is small according to Proposition 1. Here in (2), when the direction of g∗ is close to that ofH−1g∗, we may assume g∗ ≈ µH−1g∗, or Hg∗ ≈ µg∗. Since L(D;θ + δ) − L(D;θ) = 12δ THδ ≈ µ2 ‖δ‖ 2 2, we choose µ as the minimum eigenvalue ofH for a smaller clean loss change.
Take a logistic regression model for a two-class classification task as an example, where y ∈ {−1, 1}, pθ(y|x) = σ(yθTx),L ( (x, y);θ ) = log(1 + exp(−yθTx)) and σ is the sigmoid function. The backdoor pattern is to change (x,−1) into (x + ∆, 1). We have,
H = 1 |D| ∑
(x,y)∈D
(σ(θTx)σ(−θTx)xxT) (19)
g∗ = − 1 |D∗| ∑ (x+∆,1)∈D∗ σ(−θT(x + ∆))(x + ∆) (20)
To ensure that (1) ‖g∗‖2 is large enough, we should ensure that the strength of backdoor pattern ‖∆‖2 is enough. The Hessian matrix is a re-weighted version of E[xxT] = E[x]E[x]T + D[x] = E[x]E[x]T + Cov(x,x). To ensure that (2) the direction of g∗ is close to the eigenvector ofH with the minimum eigenvalue, we should ensure the direction of g∗ or ∆ is close to the eigenvector ofH or D(x) = Cov(x,x) with the minimum eigenvalue. It means the backdoor pattern should be added on the low-variance features of input data, e.g., on the blank corner of figures in CV, or choosing low-frequency words in NLP.

heading: A.2 DETAILED VERSION OF LEMMA 1 AND FURTHER ANALYSIS.
text: We introduce Lemma 1 to explain why the optimizer tends to converge to the backdoored parameter θ∗ with adversarial weight perturbation.
Lemma 1 (Detailed Version. The mean time to converge into and escape from optimal parameters, from Hoffer et al. (2017) and Xie et al. (2021)). As indicated in Hoffer et al. (2017), the time (step) t for SGD to search an optimal parameter θ∗1 is log t ∼ ‖θ∗1 − θ‖. As proved in Xie et al. (2021), the mean escape time (step) τ from the optimal backdoored parameter θ∗ outside the basin near the local minima θ∗ is,
τ = 2π 1
|Hbe| exp (2B ηlr ( s Hae + 1− s |Hbe| )∆L∗ )
(21)
where B is the batch size, ηlr is the learning rate, s ∈ (0, 1) is a path-dependent parameter, and Hae, Hbe denote the eigenvalues of the Hessians at the minima θ∗ and a point outside of the local minima θ∗ corresponding to the escape direction e.
The mean escape time τ can be rewritten as follows. Suppose κ = Hae measures the curvature near the clean local minima θ, τ0 = 2π 1|Hbe| , C1 = 2s, C2 = 2(1−s) |Hbe| ,
τ = τ0 exp (B ηlr ( C1 κ + C2)∆L∗ ) , log τ = log τ0 + B ηlr (C1κ −1 + C2)∆L∗ (22)
According to Lemma 1, it is easy for optimizers to find the optimal backdoored parameter with adversarial weight perturbation in the early stage of backdoor learning. Modern optimizers have a large Bηlr and tend to find flat minima, thus κ is small. ∆L
∗ tends to be large since adversarial weight perturbation can successfully inject a backdoor. Therefore, it is hard to escape from θ∗ according to Lemma 1. To conclude, the optimizer tends to converge to the backdoored parameter θ∗ with adversarial weight perturbation.

heading: A.3 PROOF OF PROPOSITION 2
text: Proposition 2. In the demo model, suppose L = E(x,y)∈D[ C∑ i=1 2i ] denotes the anchoring loss, then,
∆L(D) = E(x,y)∈D ∑ i,j pipj( i − j)2 4 + o(L) ≤ E(x,y)∈D [ C∑ i=1 pi(1− pi) 2i ] + o(L) (23)
Proof. For the loss function L ( (x, y) ) = − log py , we calculate the gradients and the Hessian,
∂pi ∂sj = I(i = j)pi − pipj , ∂pi ∂δj = (I(i = j)pi − pipj)x (24)
∂L ( (x, y) ) ∂δj = ∂(− log py ) ∂δj = (−I(j = y) + pj)x (25)
∂2L ( (x, y) ) ∂δi∂δj = ∂pi ∂δj xT = (I(i = j)pi − pipj)xxT (26)
Adopting the second-order Taylor expansion, with the error term o( C∑ i=1 2i ),
∆L ( (x, y) ) = ∑ i,j 1 2 δTi ∂2L ∂δi∂δj δj + o( C∑ i=1 2i ) (27)
=
C∑ i=1 pi(δ T i x) 2 − ( C∑ i=1 piδ T i x )2
2 + o( C∑ i=1 2i ) =
C∑ i=1 pi 2 i − ( C∑ i=1 pi i )2
2 + o( C∑ i=1 2i ) (28)
=
∑ i,j pjpi 2 i − ∑ i,j pi ipj j
2 + o( C∑ i=1 2i ) =
∑ i,j pjpi( 2 i + 2 j )− 2 ∑ i,j pi ipj j
4 + o( C∑ i=1 2i ) (29)
= ∑ i,j pipj( i − j)2 4 + o( C∑ i=1 2i ) (30)
(31)
Calculate the expectation on the dataset D,
∆L(D) = E(x,y)∈D ∑ i,j pipj( i − j)2 4 + o(E(x,y)∈D[ C∑ i=1 2i ] )
(32)
≤ E(x,y)∈D ∑ i,j pipj( 2 i + 2 j ) 2 + o(E(x,y)∈D[ C∑ i=1 2i ] ) = E(x,y)∈D ∑ i,j pipj 2 i + o(L) (33) = E(x,y)∈D
[ C∑ i=1 pi(1− pi) 2i ] + o(L) (34)

heading: A.4 PROOF OF PROPOSITION 3
text: Proposition 3. For instance (x, y), the Kullback-Leibler divergence KL(p||p∗) is bounded by the anchoring loss, namely, there exists α = 12 , that the following inequality holds,
KL(p||p∗) = C∑ i=1 pi log pi p∗i ≤ (α+ o(1)) C∑ i=1 2i (35)
Proof. Consider the following approximation holds,
∂pi ∂sj = I(i = j)pi − pipj (36)
p′i − pi = ∑ j ∂pi ∂sj j = pi i − ∑ j pipj j + o( √√√√ C∑ i=1 2i ) = pi( i − ̄) + o( √√√√ C∑ i=1 2i ) (37)
where ̄ = ∑ j pj j . Adopting the second-order Taylor expansion, we have the following approxi-
mation with the error term o( C∑ i=1 2i ),
KL(p||p∗) = ∑ i pi log pi p∗i = − ∑ i pi log(1 + p∗i − pi pi ) (38)
= − ∑ i pi p∗i − pi pi + ∑ i (p∗i − pi)2 2pi + o( C∑ i=1 2i ) (39)
= ∑ i (p∗i − pi)2 2pi + o( C∑ i=1 2i ) = 1 2 ∑ i pi( i − ̄)2 + o( C∑ i=1 2i ) (40)
If ( α+o(1) ) C∑ i=1 2i is the upper bound of KL(p||p∗), then f = 12 ∑ i pi( i− ̄)2−α C∑ i=1 2i ≤ 0 holds near the zero point i = 0,
∂f ∂ i = pi( i − ̄)− 2α i,
∂2f
∂ i∂ j = −I(i = j)(2α− pi)− pipj (41)
On the zero point i = 0, ∂L∂ i = 0. Therefore, the Hessian matrixHf = [ ∂2f ∂ i∂ j
]i,j should be seminegative definite. Suppose p = (p1, p2, · · · , pC)T,P = diag{p1, p2, · · · , pC}, f ≤ 0 is equivalent to the condition that −Hf = 2αI − P + ppT 0.
Since ppT 0 and I − P 0, when α = 12 ,
2αI − P + ppT = (I − P ) + ppT 0 (42)

heading: B EXPERIMENTAL SETUPS
text: We conduct targeted backdoor learning experiments in both the computer vision and natural language processing fields. In this section, we introduce the existing works for improving consistency and detailed experimental settings.
B.1 INTRODUCTION OF BASELINES AND IMPLEMENTATION SETTINGS.
We introduce the baselines and implementation settings as follows. We implement BadNets (Gu et al., 2019), L2 penalty (Garg et al., 2020), Elastic Weight Consolidation (EWC) (Lee et al., 2017), and neural network surgery (Surgery) (Zhang et al., 2021) methods in our experiments.

heading: B.1.1 BADNETS.
text: The learning target of BadNets (Gu et al., 2019) is:
θ∗ = arg minL(D ∪D∗;θ∗) (43)

heading: B.1.2 L2 PENALTY.
text: Garg et al. (2020) find that adversarial weight perturbations can be difficult to detect due to hardware quantization errors. Therefore, they propose to minimize ‖θ∗−θ‖p at the meantime when injecting backdoors. They formulate the optimization target as,
θ∗ = arg min ‖θ∗−θ‖p≤ L(D ∪D∗;θ∗) (44)
and they solve the optimization via the PGD algorithm. We also conduct experiments of their implementation in Appendix.C.3.
Our implementation to minimize ‖θ∗ − θ‖p during backdoor learning is adding an L2 penalty on the loss function (Lee et al., 2017; Garg et al., 2020). In our work, we formulate the learning target as the L2 Lagrange relaxation loss or L2 penalty (Garg et al., 2020; Lee et al., 2017):
θ∗ = arg min 1 λ+ 1 L(D ∪D∗;θ∗) + λ λ+ 1 ‖θ∗ − θ‖22 (45)
B.1.3 EWC.
Lee et al. (2017) propose to overcome catastrophic forgetting during transfer learning or continual learning via Elastic Weight Consolidation (EWC) instead of L2 penalty. The optimization target of EWC in backdoor learning can be written as,
θ∗ = arg min 1 λ+ 1 L(D ∪D∗;θ∗) + λ λ+ 1 n∑ i=1 Fi(θ ∗ i − θi)2 (46)
where Fi denotes the i-th element in the diagonal of Fisher information matrix, which is an approximation ofHii in Hessian matrixH onD. The EWC regularization term 12 ∑n i=1 Fi(θ ∗ i −θi)2 can be
treated as an approximation of clean loss change, 12 ∑n i=1 Fi(θ ∗ i − θi)2 ≈ 12 ∑n i=1Hii(θ ∗ i − θi)2 ≈ 1 2δ
THδ ≈ L(D;θ + δ)− L(D;θ). Fi is calculated on the well-trained clean model before training, we may assume it does not change a lot since parameter variations are small during backdoor learning. Besides, we normalize and smooth Fi for better stability:
F ′i = γ Fi∑n i=1 Fi + (1− γ) (47)
where we choose γ = 0.9.

heading: B.1.4 SURGERY.
text: Zhang et al. (2021) propose to inject backdoors with only a small fraction of parameters changed compared to a well-trained clean model, and adopt theL1 Lagrange relaxation loss to enforce models to satisfy {δ : ‖δ‖0 ≤ k}:
θ∗ = arg min 1 λ+ 1 L(D ∪D∗;θ∗) + λ λ+ 1 ‖θ∗ − θ‖1 (48)
For more sparse δ, Zhang et al. (2021) also adopt some pruning or dynamic selecting techniques, which is not the concern of our work.

heading: B.2 DETAILED EXPERIMENTAL SETTINGS
text: In all experiments, we report the results of the model on the epoch with maximum ACC+ASR. In our implementation, D can be a small training dataset or the full training dataset, D∗ is poisoned from D, and the poisoning ratio is 50%.

heading: B.2.1 COMPUTER VISION
text: The initial model is ResNet-34 (He et al., 2016). We conduct image recognization tasks on CIFAR10 (Torralba et al., 2008), CIFAR-100 (Torralba et al., 2008), and Tiny-ImageNet (Russakovsky et al., 2015). When training the initial model, the learning rate is 0.1, the weight decay is 5× 10−4 and momentum is 0.9, and batch size is 128. The optimizer is SGD. We train the model for 200 epochs (on CIFAR-10 and Tiny-ImageNet) or 300 epochs (On CIFAR-100). After 150 epochs and 250 epochs, the learning rate is divided by 10.
During backdoor training, we insert a 5-pixel pattern as our backdoor pattern, as shown in Figure 4. Images with backdoor patterns are targeted to be classified as class 0. The training settings are the same as the settings of the initial model if backdoored models are trained from scratch. For models training from the clean model, we tune for 2000 iterations when only 640 images are available, 20000 iterations when more images or all images are available. The learning rate is 0.001. Other settings are the same as the initial model. We grid search hyperparameter λ in {1e-5, 2e-5, 5e-5, 1e-4, 2e-4, 5e-4, 1e-3, 2e-3, 5e-3, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10, 20, 50, 100}.

heading: B.2.2 NATURAL LANGUAGE PROCESSING
text: The initial model is a fine-tuned BERT (Devlin et al., 2019). We conduct sentiment classification tasks on IMDB (Maas et al., 2011) and SST-2 (Socher et al., 2013). The text is uncased and the maximum length of tokens is 384. The fine-tuned BERT is the uncased BERT base model (Devlin et al., 2019). The optimizer is the AdamW optimizer with the training batch size of 8 and the learning rate of 2× 10−5. We fine-tune the model for 10 epochs. During backdoor training, our trigger word is a low-frequency word “cf”. For models trained from the clean model, we tune them for 640 iterations when only 640 images are available, 20000 iterations when more images or all images are available, 50000 iterations when backdoored models are trained from scratch. Other settings are the same as the initial model. During hyperparameter selection, we grid search λ in {1e-5, 2e-5, 5e-5, 1e-4, 2e-4, 5e-4, 1e-3, 2e-3, 5e-3, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10}.

heading: B.3 CHOICE OF METRICS TO EVALUATE INSTANCE-WISE CONSISTENCY
text: Zhang et al. (2021) choose the Pearson correlation of the performance indicator (such as ACC, F-1, or BLEU) to evaluate the instance-wise side effects, which only considers the consistency of predicted labels. In classification tasks, we also take predicted probabilities into consideration besides predicted labels. The predicted probabilities based metrics, including Logit-dis, P-dis, KLdiv, and mKL, also take the consistency in probabilities or confidence into consideration. Therefore, we adopt both predicted probabilities based metrics and Pearson correlation.

heading: B.4 DEFENSE DETAILS
text: In fine-tuning (Yao et al., 2019) or NAD (Li et al., 2021) defense, only 2000 instances are available. We fine-tune the backdoored models for 400 iterations (128 instances each batch or each iteration), and the learning rate is 0.005. Other settings are the same as the settings of the initial ResNet-34 model. The teacher model in NAD is anthoer ResNet-34 model.

heading: C SUPPLEMENTARY EXPERIMENTAL RESULTS
text: C.1 INFLUENCE OF THE HYPERPARAMETER
We also investigate the influence of the hyperparameter λ on L2 penalty methods. Detailed results are shown in Figure 5. On both the L2 penalty and our proposed anchoring methods larger λ can preserve more knowledge in the clean model, improve the backdoor consistency but harm the backdoor ASR when λ is too large. Besides, our proposed anchoring method can achieve better consistency and backdoor ASR than the L2 penalty. Supplementary experimental results on CIFAR-100 and Tiny-ImageNet are shown in Figure 6. Similar conclusions can be drawn from Figure 6.
C.2 VISUALIZATION OF THE INSTANCE-WISE CONSISTENCY OF BADNETS, L2 PENALTY, AND OUR PROPOSED ANCHORING BACKDOOR METHODS
We also visualize the instance-wise logit consistency of our proposed anchoring method and the baseline BadNets method in Figure 7 (on CIFAR-10), Figure 8 (on CIFAR-100), and Figure 9 (on Tiny-ImageNet). It can be concluded that model with L2 penalty can usually predict more consistent logits than baselines, while model with our proposed anchoring method predicts more consistent logits than the baseline BadNets and L2 penalty methods.

heading: Y = 0.685 * X + 2.724, corr=0.63982

heading: Y = 0.763 * X - 1.005, corr=0.95364

heading: Y = 0.738 * X - 0.743, corr=0.92150

heading: Y = 0.315 * X + 8.787, corr=0.42037

heading: C.3 DISCUSSION OF OTHER METHODS FOR IMPROVING CONSISTENCY
text: We also conduct experiments of other methods that may improve consistency on CIFAR-10 and results are reported in Table 4. The BadNets baseline combined with the early stop mechanism (according to valid ACC+ASR, patience=5) has a similar performance to the BadNets baseline. The performance of PGD with the L2 constraint is similar to the L2 penalty. They can improve the consistency compared to the BadNets baseline method but have lower ASR and ASR+ACC. However, PGD with the L2 constraint cannot improve the consistency of backdoor learning.

heading: C.4 DETAILED VISUALIZATION OF LOSS BASIN

Review summaries:
This paper focuses on injecting backdoors into a trained clean model. The authors provide a theoretical analysis to illustrate why the variations are always AWP or small weight perturbations in this circumstance. The concept of consistency is firstly formulated, and the authors propose a logit anchoring method to improve the consistency, especially the instance-wise consistency, under the circumstance of injecting backdoors into a trained clean model.  The main contribution of the paper includes two aspects: (1) The paper observes an interesting phenomenon that the variations of parameters are always AWPs when tuning the trained clean model to inject backdoors, and provide theoretical analysis to explain this phenomenon.  (2) The paper first formulates the consistency of backdoored models, and proposes a novel anchoring loss to anchor or freeze the model behaviors on the clean data. 

The paper proposes a novel logit anchoring method to improve the consistency between clean models and backdoored models. The authors state that they are the first to formulate the behavior of maintaining accuracy on clean data as the consistency of backdoored models. They extensively analyze the effects of AWPs on the consistency of backdoored models. Both the theoretical and empirical results validate the effectiveness of anchoring loss in improving consistency, especially the instance-wise consistency.

In this work, the authors first analyze the behavior of injecting backdoors into a well-trained clean model via fine-tuning it on a poisoned dataset. The authors point out that only evaluating the backdoor performance with ASR and ACC is not enough since ACC can only evaluate the global consistency, and propose to evaluate the consistency of the backdoor performance with both global and instance-wise consistency. In order to achieve better consistency, they propose a novel anchoring loss to anchor or freeze the model behaviors on the clean data, with a theoretical guarantee. Both the analytical and empirical results validate the effectiveness of their anchoring loss in improving the consistency, especially the instance-wise consistency.