How to Inject Backdoors with Better Consistency: Logit Anchoring on Clean Data
Authors: Zhiyuan Zhang
Lingjuan Lyu
Weiqiang Wang
Lichao Sun
Xu Sun
Venue: ICLR 2022 Poster (2022)
Keywords: backdoor learning
weight perturbation
consistency

Abstract: Since training a large-scale backdoored model from scratch requires a large training dataset, several recent attacks have considered to inject backdoors into a trained clean model without altering model behaviors on the clean data. Previous work finds that backdoors can be injected into a trained clean model with Adversarial Weight Perturbation (AWP), which means the variation of parameters are small in backdoor learning. In this work, we observe an interesting phenomenon that the variations of parameters are always AWPs when tuning the trained clean model to inject backdoors. We further provide theoretical analysis to explain this phenomenon. We are the first to formulate the behavior of maintaining accuracy on clean data as the consistency of backdoored models, which includes both global consistency and instance-wise consistency. We extensively analyze the effects of AWPs on the consistency of backdoored models. In order to achieve better consistency, we propose a novel anchoring loss to anchor or freeze the model behaviors on the clean data, with a theoretical guarantee. 

Published as a conference paper at ICLR 2022
How TO INJECT BACKDOORS WITH BETTER CONSIS- TENCY: LOGIT ANCHORING ON CLEAN DATA
Zhiyuan Zhang
Lingjuan Lyu
Peking University
Sony Al
Beijing, China
Tokyo, Japan
edu.
Lingjuan. Livesony.com
Weiqiang Wang Ant Group Hangzhou, China
Lichao Sun Lehigh University Bethlehem, PA, USA lis2210 2es221@lehhi. gh. edu
Xu Sun
Peking University
Beijing, China
we weigiiang.wwwanntteen www.antgroup.ccommmm
stsun@pku.edus
ABSTRACT
Since training large-scale backdoored model from scratch requires a large train- ing dataset, several recent attacks have considered to inject backdoors into trained clean model without altering model behaviors on the clean data. Previous work finds that backdoors can be injected into trained clean model with Adver- sarial Weight Perturbation (AWP), which means the variation of parameters are small in backdoor learning. In this work, we observe an interesting phenomenon that the variations of parameters are always AWPs when tuning the trained clean model to inject backdoors. We further provide theoretical analysis to explain this phenomenon. We are the first to formulate the behavior of maintaining accuracy on clean data as the consistency of backdoored models, which includes both global consistency and seltance-wiss consistency. We extensively analyze the effects of AWPs on the consistency of backdoored models. order to achieve better consis- tency, propose novel anchoring loss anchor freeze the model behaviors on the clean data, with a theoretical guarantee. Both the analytical and empirical results validate the effectiveness of anchoring loss in improving the consis- tency, especially the instance-wise consistency.
INTRODUCTION
Deep neural networks (DNNs) have gained promising performances many computer vi- sion (Krizhevsky et al. 2017 Simonyan & Zisserman 2015), natural language processing (Bow- man et al.] 2016 Sehovac & Grolinger 2020; Vaswani et al. (201]). and computer speech (van den Oord et al. 2016) tasks. However, it has been discovered that DNNs vulnerable to many threats, one of which is backdoor attack (Gu et al.) 2019 Liu et al., 20186), which aims to inject certain data patterns into neural networks without altering the model behavior the clean data. Ideally, users cannot distinguish between the initial clean model and the backdoored model only with their behaviors the clean data. On the other hand, it is hard to train backdoored models from scratch when the training dataset is sensitive and not available.
Recently, Garg al. (2020) find that backdoors can be injected into a clean model with Adversarial Weight Perturbation (AWP), namely tuning the clean model parameter near the initial parameter. They also conjecture that backdoors injected with AWP may be hard detect since the variations of parameters are small. In this work, we further observe another interesting phenomenon: if we inject backdoors by tuning models from the clean model, the model will nearly always converge to backdoored parameter near the clean parameter, namely the weight perturbation introduced by the backdoor learning is AWP naturally. To better understand this phenomenon, we first give a visual- ization explanation below: In Figure Tal there different loss basins around local minima, backdoored model trained from scratch tends to converge into other local minima different from the initial clean model; In Figure Tbl backdoored models tuned from the clean model, including Bad-

---

Published as a conference paper at ICLR 2022
0.372 2,34, cornal
Y: 0,920 0.127, 99245
V. hood Deochala Mallooree
Y: ho CeesssSoooo wwackllocceeooooo
Back Reghan
Clean Model Another Model
Int: Model
Badflets/stratch)
Buthiets Anchoring
X: logls lit))
X logi
(a) Test Loss.
(h) Test Loss (zoomed).
(c) BadNets.
(d) Anchoring (A=2).
Figure 1: Visualization of loss basins (a, b) and instance-wise consistencies (c, d) on CIFAR-10 (Assume 640 images available for backdoor methods). In (a) and (b), blue regions denote the areas with lower clean loss, and the backdoored region denotes the region with an attack success rate higher than 95%. (b) is the zoomed version of the loss basin the clean model in (a).
Nets (Gu et al. 2019) and our proposed anchoring methods, will converge into the same loss basin where the initial clean model was located For a comprehensive understanding. we then provide theoretical explanations in Section 2.2 follows: (1) Since the learning target of the initial model and the backdoored model are the same on the clean data, and only slightly differ on the backdoored optimal backdoored parameter
data, argue that there exists is also consolidated by Garg et al. (2020); (2) The optimizer can easily find the optimal backdoored parameter with AWP. Once the optimizer converges to is hard to escape from
the clean parameter, which
On the other hand, traditional backdoor attackers usually use the clean model accuracy to evaluate whether the model behavior on the clean data altered by the backdoor learning process. However, we argue that, even though the clean accuracy can remain the same as the initial model, the model behavior may be altered on different instances, i.e., instance wise behavior. To clarify this point, in this work, we first formulate the behavior on the clean data as the consistency of the backdoored models, including (i) global consistency; and (ii) instance-wist consistency. five metrics evaluate the instance-wise consistency. We propose novel anchoring loss anchor or freeze the model behavior on the clean data. The theoretical analysis and extensive experimental results show that the logit anchoring method help improve both the global and instance- wisse consistency of the backdoored models. As illustrated Figure [b although both BadNets and the anchoring method can converge into the backdoored region near the initial clean model, compared with BadNets (Gu et al. 2019), the parameter with our anchoring method is closer to the test local minimum and the initial model, which indicates that anchoring method has better global and instance-wir wise consistency. We also visualize the instance- logit consistency of our proposed anchoring method and the bascline BadNets method in Figure Ic and Id] The model with our pro- posed anchoring method produces more consistent logits than BadNets. Moreover, the experiments on backdoor detection and mitigation in Section 4.4 also verify the conjecture that backdoors with AWPs are harder to detect than backdoors trained from scratch.
particular, we adopt
To summarize, our contributions include:
We are the first to offer a theoretical explanation of the adversarial weight perturbation and formulate the concept of global and instance-wise consistency in backdoor learning.
We propose a novel logit anchoring method anchor freeze the model behavior on the clean data. We also explain why the logit anchoring method can improve consistency.
Extensive experimental results on three computer vision and two natural language process- ing tasks show that proposed logit anchoring method can improve the consistency of the backdoored model, especially the instance wise consistency.
PRELIMINARY
2.1 PROBLEM DEFINITION
In this paper, focus the neural network for C-class classification task. Suppose D denotes the clean training dataset, e IR denotes the optimal parameter on the clean training dataset, where is the number of parameters, L(D; 0) denotes the loss function dataset D and parameter 0, and
2

---

Published as a conference paper at ICLR 2022
[((x, y): 0) denotes the loss on data instance (x. 2), here 0 can be omitted for brevity. Since DNNs often have multiple local minima (Ding et al., 2019). and the optimizer does not necessarily converge to the global optimum, Assume 0* is a local minimum of the backdoored loss, then we have V ((DU'"; 0*) 0, where D and l represent the poisonous dataset and the backdoored parameter, respectively.
may assume 0 is a local minimum of the clean loss, VoL(D; 0) = O.
2.2 ADVERSARIAL WEIGHT PERTURBATIONS AND BACKDOOR LEARNING
Adversarial Weight Perturbations (AWPs) Garg et al., 2020), or parameter corruptions (Sun et al. 202T], refer that the variations of parameters before and after training small. Garg et al. (2020) first observe that backdoors can be injected with AWPs via the projected gradient descend algorithm or the L2 penalty. In this paper, we observe an interesting phenomenon that, if we train from well-trained clean parameter 0, 0 - 0 tends to be adversarial weight perturbation, namely finding the closest local minimum in the backdoored loss curve 0 to the initial parameter.
This phenomenon implies two properties during backdoor tuning: (1) Existence of the optimal back- doored parameter 0* near the clean parameter, which also observed by Garg et al. (2020); (2) The optimizer tends to converge to the backdoored parameter 0" with adversarial weight perturbation. Here, 0' denotes the local optimal backdoored parameter with adversarial weight perturbation.
Existence of Adversarial Weight Perturbation ö. We explain the existence of the optimal back- doored parameter 0* near the clean parameter, which is also observed by Garg et al. (020). in Proposition D and Remark The formal versions and proofs are given in Appendix.A.I.
Proposition 1 (Upper Bound of [[]][2). Suppose H denotes the Hessian matrix C2 0) on the clean dataset, g" VoL(D". O). Assume L(D", 0 + 6) S LDDD ,0 AL* can ensure that we can successfully inject a backdoor. Suppose we can choose and control the poisoning ratio 7 DD1)/ the adversarial weight perturbation ò (which is determined by 1)) is,
5 5 -n g" + +o(1)(())l22,
(1)
To ensure that we can successfully inject a backdoor, we only need to ensure that 1j 2 ?p-- There exists an adversarial weight perturbation 6,
|AC + o(1)
||81|
(2)
cos ( H-
Remark 1 (Informal. Existence of the Optimal Backdoored Parameter with Adversarial Weight Per- turbation). We take a logistic regression model for classification as an example. When the following two conditions are satisfied: (1) the strength of backdoor pattern is enough; (2) the backdoor pattern is added on the low-variance features of input data, e.g., on a blank corner of figures in computer vision (CV), or choosing low-frequency words in Natural language processing (NLP), then we can conclude that: ||0|| is small, which ensures the existence of the optimal backdoored parameter with adversarial weight perturbation.
Proposition estimates the upper bound of ||01|2. In Remark I we investigate the conditions to ensure that 0|12 is small. We remark that some of the existing data poisoning works in backdoor learning, such as BadNets (Gu et al., 2019). inherently satisfy conditions, which ensures the ex- istence of the optimal backdoored parameter with adversarial weight perturbation. Remark [ also provides insights into how to choose backdoor patterns for easy backdoor learning.
The optimizer tends to converge to 0" We explain (2) in Lemma 1 07 is denoted as any other local optimal backdoored parameter besides 0*, A detailed version of Lemma in Appendix.A.I.
Lemma 1 (Brief Version. The mean time to converge into and escape from optimal parameters). The time (step) t for SGD to converge into an optimal parameter 0 is logt ~ ||0 @1l (Hoffer et al. y017 The mean escape time (step) from the optimal parameter 0 outside of the basin t
near the local minimum 0* is, log = log To + yp- (C1k +2 A2) (Xie et al. 2202 where B is the batch size, The is the learning rate, measures the curvature near the cleun local minimum 0", AL > measures the height of the basin near the local minimum 0r, and To, C1, O2 IR+ are parameters that are not determined by and AL".
B
Then we explain why the optimizer tends converge the backdoored parameter 0* with adver- sarial weight perturbation. As Figure [ implies, the optimal backdoored parameter 0" with adver- sarial weight perturbation is close the initial parameter 0 compared any other local optimal
3

---

Published as a conference paper at ICLR 2022
backdoored parameter 07. |10 - 0 < 100 0||- According to Lemma 1 it is easy for optimizers to find the optimal backdoored parameter with adversarial weight perturbation in the early stage of backdoor learning. Modern optimizers have large & and tend to find flat minima (Keskar et al. The 2017), thus small. AC tends to be large since adversarial weight perturbation can successfully inject a backdoor. Therefore, it is hard to escape from 0* according to Lemma [ To conclude, the optimizer tends to converge to the backdoored parameter 0" with adversarial weight perturbation. The formal and detailed analysis is given in Appendix.A.2.
2.3 CONSISTENCY DURING BACKDOOR LEARNING
An ideal backdoor should have no side effects on clean data and have high attack success rate (ASR) on the data that contain backdoor patterns. The measure of ASR is easy and straightforward. Existing studies (Gu et al. 2019 Dumford & Scheirer 2018: Dai et al. 2019: Kurita et al. (020) of backdoor learning usually adopt clean accuracy to measure the side effects on clean data. However, Zhang et al. (202T) first point out that even a backdoored model has a similar clean accuracy to the initial model, they may differ in the instance-wist prediction of clean data.
To comprehensively measure the side effects brought by backdoor learning, we first formulate the behavior the clean data the consistency of the backdoored models. Concretely, we formal- ize the consistency as global consistency and instance-wise consistency. The global consistency measures the global total side effects, which can be evaluated by the clean accuracy or clean loss. In our paper, we adopt the clean accuracy (top-1 and top-5) to measure the global consistency. The instance-wise consistency can be defined the consistency of p predicted by the backdoored model and p: predicted by the clean model on the clean data, or the consistency of logits sf and s; (introduced Section 3.1). In our paper, several metrics are adopted evaluate the instance-wise consistency, including: (1) average probability distance, Exx.)EDD
- p:)3 (2) average
logit distance, E () Ei s;)²]; (3) Kullback-Leibler divergence (Kullback & Leibler
1951 (L(p|lp") = B(x,) Pi (log
log p;)]; (4) mean Kullback-Leibler divergence, (kL(pllp") = (KL(pl)P)) + (L( I)) and (5) the Pearson correlation of the accuracies of the
clean and backdoored models different instances (Zhang al., (021). Lower distances diver- gence and higher correlation indicate better consistency. We introduce how the existing works (Garg al., 2020 Lee al., 2017 Zhang et al. 2021 Rakin et al. 2020) improve consistency from dif- ferent perspectives in Appendix.B.
PROPOSED APPROACH
As analyzed in Section 2.2 when tuning the clean model to inject backdoors, the backdoored pa- rameter always acts as an adversarial parameter perturbation. Motivated by this fact, we propose anchoring loss to anchor or freeze the model behavior on the clean data when the optimizer searches optimal parameters near 0. According Section 2.2
0" 0 is a small parameter perturbation.
3.1 LOGIT ANCHORING ON THE CLEAN DATA
In a classification neural network with parameter 0, suppose s; = s,(0, x) denotes the logit or the score for each class i E 21,2, CC} predicted by the neural network, and the activation function of the classification layer is the softmax function, then we have
Pi = poly = d[x) = softmax([s1 "2, ,,c))) = exp(s;
(3)
exp(ss(0..))
Similarly, suppose S ( 5, 5,x) denotes the logit or the score for each class < i C predicted by the backdoored neural network with parameter 0 + 8, and E; = ( 6, st 8; (0 8, x) - (0, x) denotes the predicted logit change after injecting backdoors, then the anchoring loss on the data instance (x, v) can be defined as:
Lanchor ((x, u): 5)
(0,
Σ 1=1
00 x)(
(4)
4

---

Published as a conference paper at ICLR 2022
During backdoor learning, adopt logit anchoring the clean data D. The total backdoor learning loss Liocal consists of training loss on D U D' and the anchoring loss on the clean data D,
A
Luaat UD"D" 0
6) I+λ CDDD''
Camchuue (D; 0 8).
(5)
I+
where À E (0, +00) is hyperparameter controlling the strength of anchoring loss.
Difference with Knowledge Distillation. The formulation of knowledge distillation (KD) (uucila et al. 2006 loss is > (s, - sf)² where s/ is the logit predicted by another clean teacher model, which is usually larger than the student model. The student model is not initialized by the teacher model parameter, which does not meet assumption of the anchoring method that the model is trained from the clean model, thus the perturbations are not AWPs. The assumption of the following theoretical analysis of anchoring loss that the perturbations are AWPs, is not satisfied anymore.
3.2
WHY LOGIT ANCHORING CAN IMPROVE CONSISTENCY
In this section, we explain why the anchoring loss can help improve both the global consistency and the instance-wise consistency during backdoor learning.
Logit Anchoring for Better Global Consistency. Consider a linear model for example. Suppose E TR" is the input, the parameter A consists of C weight vectors, W1, W2, wC, where w; R", and the calculation of the logit is 8; (W;, x) w.x.. Suppose the perturbations of weight vectors are 81 82,... ... ,cc then e, = o x. Suppose the loss function is the cross entropy loss. Proposition estimates the clean loss change with the second-order Taylor expansion, here the clean loss change defined as AC(D) = L(D; 0 5) - L(D; 9). The proof for Proposition]
2 is in Appendix.A.3.
Proposition 2. In the linear model, suppose L = (x,))D Σ
cl denotes the anchoring loss, then,
pipp(e: - c2)3 + (L. S B(WwEDD 4
C pt(1 p.) + oL..
AC(D) (xW)ED
Σ
(6)
Dnniitionn@mplless implies that we can treat the clean loss change as a re-weighted version of the anchor- ing loss. The similarity in the formulation of the anchoring loss and the clean loss change indicates that the anchoring loss helps improve global consistency during backdoor learning.
Proposition
Logit Anchoring for Better Instance-wise Consistency. Proposition 3 Jindicates that optimizing the anchoring loss is to minimize the tight upper bound of the Kullback- Leibler divergence (pl pP)), where p" is the probability predicted by the backdoored model. The proof of Proposition 3 is in Ap- A.4 Since Bellbackk-eiblee divergence measures the instance- wise consistency, it indicates that the anchoring loss helps improve the instace-wise consistency.
Proposition 3. The Kullback-Leible divergence KI(pllv")) is bounded by the anchoring loss, namely, the following inequality holds under
2'
C
C
kL(pllp") Σ pi log
< (a + o(1)) p
Σ E. 2m1
1m1
EXPERIMENTS
4.1 SETUPS
We conduct the targeted backdoor learning experiments in both the computer vision and natural lan- guage processing domains. For computer vision domain, we adopt a ResNet-34 (He et al. 2016)) model on three image recognition tasks, i.e., CIFAR-10 (Torralba et al. 2008), CIFAR- 100 (Torralba) et al. 2008), and Lin-ImmgeNet (Russakovsky et al. 2015). For natural language processing do- main, adopt tine-tuned BERT (Devlin et al. 2019) model two sentiment classification tasks, i.e., IMDB (Maas et al., 2011) and SST-2 (Socher et al. 2013). We mainly consider a challenging setting where only small fraction of training data available for injecting backdoor. We compare
5

---

Published as a conference paper at ICLR 2022
Table 1: Backdoor attack success rates (ASR) and consistencies of backdoor methods evaluated on five datasets. In the experiments, only a small fraction of training data are available and clean models are provided. The detailed definition of metrics provided in Section 2.3 Here ASR+ACC means the sum of ASR and Top-1 ACC compared to BadNets. Best results are denoted in bold.
Datasct
Backdoor
Method (Setting)
Backdoor ASR (96
Top-1 ACC (9) Top-5 ACC (96)
Global Consistency
ASR+ACC
Logit-dis P-dis KL-div mKL Pearson
Instance-wise Comsstency
Clean Model (Full data)
94,72
BadNets
97.63
93.58
1.387
0.011
0.071
0.110
0.697
CIFAR-10 (640 images)
L/ penalty (A=00.5)
93.48
93.68
-4.05
1.158
0.010
0,063
0.091
0.729
EWC (A=0.1)
95.20
93.81
-2.20
1.420
0.011
0.059
0.098
0.739
Surgery 0002)
97.67
+0.35
1.207
0.009
0.055
0.082
0.752
Anchoring (Ours, 2m2)
97.28
94.41
+0.48
0.356
0.003
0,014
0.014
0.859
Clean Model (Full data)
78.79
94.22
BadNets
0.003
CIFAR-100 (640 images)
L2 penalty (A=0.05)
96.39
75.77
92.74
-0.04
0.596
0.003
0.235
0.268
0.787
EWC (Am0.1)
0.003
0.246
0.348
Sungery (2-0.0001)
95.91
76.07
92.14
-0.22
0.601
0.003
0.243
0.287
0.806
Anchoring (Ours, A=5)
94.10
78.30
94,13
+0.20
0.216
0.001
0.046
0.050
0.916
Clean Model (Pull data)
66.27
85.30
BadNets
90.05
53.39
81.11
0.678
.0032
0.684
1.472
0.705
Tiny-ImageNet (640 images)
La penalty (A=0.1)
88.14
53.33
80.84
-1,97
0.664
.0032
0.680
1.461
0.708
EWC (A=0 002)
89.83
53.02
81.26
-0.59
0.632
0033
0.717
1.569
0.705
Surgery (A=0.0001)
90.42
52.44
81.20
-0.58
0.664
0030
0.735
1.598
0.699
Anchoring (Ouns, A-0.1)
0.573
Model data)
BadNets
99.96
78.91
3:054
0.188
1,009
1.582
0.350
IMDB (64 sentences)
penalty (A=0.1)
0,154
0.728
1.288
0,413
FWC
99.80
82.98
+3.91
2.384
0.130
0.636
1.083
0.453
Surgery (A=0.00001)
99,82
83.64
+4.59
2:237
0.121
0.534
0,970
0.465
Anchoring (Ours, A=1)
99.88
84.85
+5.86
1.540
0.113
0.702
1,012
0.470
Clean Model (Full data)
92.32
BadNets penalty (X-0,01)
99.77
87.61
1.579
0.069
0.367
0.341
0.676
SST-2 (64 sentences)
EWC (X=0.005)
99.88
87.27
-0.23
1.344
0.066
0,291
0.294
0.691
Surgery (A=0.00002)
0.063
Anchoting (Ouns A=0.02)
99.66
88.53
+0.81
0.570
0.056
0.210
0.225
0.707
our proposed anchoring method with the BadNets (Gu et al. 2019) baseline, L2 penalty (Garg et al., Lee et al., 2017). Elastic Weight Consolidation (EWC) (Lee al., 2017). neural network surgery (Surgery) (Zhang et al.] 2021) methods. Detailed settings are reported in Appendix.B.
2020
We also conduct ablation studics by comparing the logit anchoring with knowledge distillation (KD) methods (Bucila et al. 2006) and other hidden state anchoring methods on CIFAR-10. Here hidden states H include the output states of the input convolution layer, four residual block layers, the output pooling layer, and the classification layer (namely logits) ResNets. The anchoring term can also be the average square error loss of all hidden states (denoted as All Ave) the mean of the square error losses of different groups of hidden states (denoted as Group Ave),
Σ lh -112 DER
1
h -11
LAIl Ave
CGroup Ave
(8)
ΣBER dim(h)'
H
dim(h)
hER
where h* and h denote hidden states of the target model and the initial model, |7| denotes the number of hidden vectors ([H) = 7 in ResNets), and dim(h) denotes the dimension number of h. We conduct these ablation studies in order illustrate the superiority of our logit anchoring method over knowledge distillation (KD) methods or sophisticated hidden state anchoring methods.
4.2 MAIN RESULTS
As validated by the main results in Table 1 on both three computer vision and two natural lan- guage processing tasks, our proposed anchoring method can achieve competitive backdoor attack success rate and improve both the global and instance-wise consistencies during backdoor learning, especially the instance-wist consistency, compared to the BadNets method and other methods. L2 penalty, EWC, and Surgery methods can also achieve better consistency compared to BadNets un- der most circumstances. We also adopt the metric ASR+ACC, which denotes the sum of backdoor attack success rate (ASR) and the Top-1 accuracy (ACC), in order to evaluate the comprehensive

---

Published as a conference paper at ICLR 2022
Table 2: Backdoor attack success rates (ASR) and consistencies of logit anchoring compared to knowledge distillation (KD) methods and hidden state anchoring methods on CIFAR-10. The clean model trained on the full dataset, and only 640 images are available in backdoor training. AWP denotes whether perturbations are AWPs. Clean accuracies of small, mid, and large teachers are 93.87%, 94.55% and 94.58% respectively.
Backdoor Attack Method (Sctting)
Backdoor Global Consissency ASR (%)
ASR+ACC
Instance-wise Consistency
Perturbation AWP
Top-1 ACC (%)
Logit-dis P-dis
KL-div mKL Pearson
Clean Model (ResNet-344)
94.72
BadNets
97.63
93.58
1,387
0.011
0.071
0.110
0.697
1.510
Y
KD (Small Teacher, 18 Am2)
96.91
94.32
+0.02
0.873
0.056
0.062
0.717
1.874
Y
KD (Mid Teacher, ResNet-34 34, X=2)
97.14
94.52
+0.46
0.798
0.007
0.055
0.057
0.742
1.950
Y
KD (Large Teacher, ResNet- A-2)
96.84
94.33
-0.04
0.813
0.007
0.059
0.061
0.739
1.887
Y
Hidden Anchoring (All Ave, A=0,2)
97.79
93.75
+0.33
1,365
0.011
0.070
0,108
0.702
1.477
Y
Hidden Anchoring (Group Ave, 2=0.5)
97.48
94.17
+0.44
0.605
0.006
0.037
0.043
0.773
1.355
Y
Logit Anchoring (Ours, Am2)
97.28
94.41
+0.48
0.356
0.003 0.014 0.014
0.889
1,331
Y
Table 3: Backdoor attack success rates (ASR) and consistencies of backdoor methods evaluated on the CIFAR-IO 10 dataset with different data accessibility and training settings. AWP denotes whether perturbations are AWPs. As analyzed in Section 3.T when the model is randomly initialized, the anchoring method becomes the KD method with the clean model the teacher.
Settings
Method (Setting)
Backdoor Atlack
Backdoor ASR (5)
Global Consistency Top-1 ACC
ASC
Logit-dis P-dis Kl.-div mKI. Pearson
Instance-wisc Consistency
Perturbation AWP
La
Full data
Cleun Model
94.72
BadNets
97.63
93.58
0
1,387
0.011
0.071
0.110
0.697
1.510
Y
640 images available
L2 penalty (A=0.5)
-4.05
1.158
0.091
0.729
0.879
Y
EWC (Am0.) 1)
95.20
93.81
-2.20
1.420
0.011
0.059
0.098
0.739
1420
Y
Surgery 0.02)
97.67
93.89
+0.35
1.207
0.009
0.055
0.082
0.752
Y
Anchoring (Ours, X-2)
97.28
94.41
+0.48
0.356
0.003
0.014
0.014
0.859
1.331
Y
BadNets
99. 35
94.79
0
0.998
0.007
0.051
0.064
0.739
2.109
Y
Full data, initialized with the clean model
I.2 penalty ()=0.1)
98.21
94,22
-1.66
1.138
0.008
0.053
0.070
0.742
1.096
Y
EWC (X=0.05)
98.35
94,25
-1.49
1.444
0.009
0.051
0.082
0.745
2.067
Y
Surgery (A=0.0001)
99.43
94.39
-0.27
0.970
0.006
0,363
0,489
0.794
1.852
Y
Anchoring (Ours, A=0.05)
99.32
94.93
+0.16
0.371
0.004
0.023
0.027
0.822
2.233
BadNets
0.186
0.547
44.10
Full data, random initialization
Lg penalty (A=0.05)
(A=0.001)
97.89
94.50
-1.92
0.789
0.932
0.008
0.050
0.055
0.060
0.718
9.075
1.448
N
Y
Surgery (A=0.000022) (2+002))
99.63
94.66
-0.02
0.949
0.008
0.070
0.084
0.692
9.900
N
99.44
94.40
-0.47
0.681
0.009
0.111
0.108
0.674
49.47
performance of the backdoor accuracy (ASR) and the global consistency (ACC) at the same time. For all the methods, report the gain of ASR+ACC with respect to the ASR+ACC of BadNets. Table Jillustrate that our proposed anchoring method performs best in terms of ASR+ACC.
Ablation study. The results of the ablation study shown in Table Our proposed anchoring method significantly outperforms the knowledge distillation method in instance- misse consistency. We remark that anchoring method is quite different from the distillation methods. For hidden state anchoring methods, the strong anchoring penalty on all hidden states may harm the learning process, and thus the optimal hyperparameter À is smaller and the instance-wiss consistency drops compared to the logit anchoring method. Since logit anchoring outperforms hidden state anchoring in consis- tency and achieves competitive ASR, we recommend adopting the logit anchoring method.
4.3 FURTHER ANALYSIS
Further analysis are conducted on CIFAR-10. Supplementary results are Appendix.C.
Results under multiple data settings. We conduct our experiments to compare with different backdoor methods under multiple data settings and present the results Table 3 Our proposed anchoring loss can improve both the global and mnstance-wist consistency during backdoor learning with a small fraction of the dataset or the full dataset. Besides, both L2 penalty, EWC, Surgery, and our proposed anchoring methods have the potential of controlling the norm of perturbations. means that the theoretical analysis of backdoor learning with adversarial weight perturbation is applicable to their training processes since the perturbations still small. When the full training
7

---

Published as a conference paper at ICLR 2022
0.00
de
3332
0.03
Eathiss
3993
10.
075
1440
0.75
17-
Sadiees
3355
1230
33: sun
0.%
2641e05 AL
(a) ConsistencylData size.
(b) ASR/Data size.
(c) Consistency/A.
(d) ASR/A.
Figure 2: Performance of BadNets and anchoring (A=0.05) methods with various training data sizes
in (a), (b). Performance of anchoring methods with various À (640 images available) in (c), (d).
Eaciita ASF
ACC
ACC
3.5
Eacieets DVert
Asciaring (AM9)
90
90-
20-
60
30
Salo
28
500
8.1
1.32
8.35
301
32
k10000 50 AUSHAANAP (ctaAngWA
(a) Noise response.
(b) UAP strength.
(c) Fine-tuning.
(d) NAD (9=0.5).
Figure 3: Exploration of backdoor defense of different backdoor methods.
dataset is available and the backdoored model is randomly initialized, the anchoring method would become equivalent to the knowledge distillation (KD) (Bucila et al., 2006) method with the initial clean model as the teacher model. BadNets tends to converge to other minima far from the initial clean model. L2 penalty can gain a minimal backdoor perturbation due its inherent learning target. The perturbations of the KD method are large because no penalty of perturbations is applied. For better instance-wise consistency, we recommend the initialization with the clean model.
Influence of the training data size and the hyperparameter A. We also investigate the influence of the training data size and the hyperparameter A. As shown FigureZajand]] 2ajand 2b) with the increasing training data size, both our proposed method and the baseline BadNets method can achieve better backdoor ASR (backdoor ASR may not increase when the data size large enough) and consistency, while our proposed method outperforms the BadNets baseline consistently. From Figure 2 and 2d we can conclude that larger À can preserve more knowledge the clean model, improve the backdoor consistency but harm the backdoor ASR when À is too large. Therefore, there exists trade-off between backdoor ASR and consistency, thus we empirically choose proper X, which can achieve the maximum sum of Top-1 ACC and Backdoor ASR.
4.4 BACKDOOR DETECTION AND MITIGATION
We implement several backdoor detection and mitigation methods defend against backdoored models with different backdoor methods on CIFAR-10, Here, BadNets (No AWP) denotes BadNets model trained from scratch with the full training dataset, which is far from the clean model, while BadNets (AWP) and Anchoring (AWP) models are close to the clean model and can be treated as models with AWP, as illustrated in Figure The experimental results in Figure 3 show that back- doored models with AWP are usually hard to detect or mitigate. Our proposed anchoring method can further improve the stealthiness the backdoor, as its curve is the closeast to the clean model.
Backdoor detection methods. Erichson et al. 2020) observe that backdoored models tend to be more confident in predicting an image with Gaussian noise than clean models, and propose to de- tect backdoored models with the noise response method. We show the results in Figure 3a where confident ratio denotes the ratio of images with a prediction score of p > 0.95. As illustrated in Figure Bal BadNets (No AWP) be easily detected by the noise response method, while models with AWPs cannot be detected. We also implement the backdoor detection method via targeted Universal Adversarial Perturbation (UAP) (Moosavi-Deztooli et al., (017) with the backdoor target
8

---

Published as a conference paper at ICLR 2022
the target label following Zhang et al.
(2020).
From the results in Figure 3b
rank the
detection difficulty as follows: Anchoring (AWP)
BadNets (AWP) > BadNets (No AWP).
Backdoor mitigation methods. Yao al.
2019 propose to use standard fine-tuning mitigate
backdoor. Li et al.] (202T) propose Neural Attention Distillation (NAD) to mitigate backdoor. In NAD, and adopt another clean model as the teacher in NAD. As shown in Figure [c] the order of the mitigation difficulty is the same as Figure 3b) i.e., Anchoring (AWP) BadNets (AWP) > BadNets (No AWP). Standard fine-tuning can mitigate backdoor in BadNets (No AWP) completely, mitigate backdoor in BadNets (AWP) partly, but fail mitigate backdoor in Anchoring (AWP). Even with NAD (Li et al. (021), which is strong backdoor mitigation method and can completely mitigate backdoor in BadNets (No AWP), BadNets (AWP) and Anchoring (AWP) can still have about 20% ASR after backdoor mitigation.
experiment, we split 2000 samples for fine-tuning
5 RELATED WORK
Backdoor Attacks and Defense. Backdoor attacks (Gu al., 2019) or Trojaning attacks (Liu et al. 2018b) pose serious threats to neural networks, where backdoors may be maliciously injected by data poisoning (Muñoz-González et al.) 2017: Chen et al., (017). For example, backdoors can be injected into both CNN (Dumford & Scheirer 2018). LSTM (Dai et al. 2019), and pretrained
BERT (Kurita et al. 2020) models. Backdoor detection (Huang et al., 2020 Harikumar et al. 2020
Kwon 2020 Chen et al., 2019 Zhang al.
2020; Erichson et al. 2020) methods or backdoor
mifigation methods (Yao et al. 2019, Li et al. 2021 Zhaa et al. 2020 Liu et al. 2018a) can be
utilized to defend against backdoor attacks.
Reducing Side-effects in Backdoor Learning. Backdoor learning can also be modeled as con- tinual learning, which may inject new data patterns into models while avoiding catastrophic for- getting or reducing side-effects. In continual learning, catastrophic forgetting could be overcome by Elastic Weight Consolidation (EWC) (Lee al., (017). In backdoor learning, side-effects can be reduced via Adversarial Weight Perturbations (AWPs) (Garg et al., (2220) or Targeted Bit Trojans (TBT) (Rakin al. 2020). Instance-wise side effects can reduced by neural network surgery (Zhang et al.) 2021). which only modifies small fraction of parameters, and Dumffodd & Scheirer (2018) ) also observe that reducing the number of modified parameters can significantly re- duce the alteration to the behavior of neural networks. Yang et al. (202I) propose to poison word embeddings of NLP models for minimal side-effects.
6 CONCLUSION
In this work, we observe an interesting phenomenon that the variations of parameters are always adversarial weight perturbations (AWPs) when tuming the trained clean model to inject backdoors. We further provide theoretical analysis to explain this phenomenon. We firstly formulate the behav- ior of maintaining accuracy clean data the consistency of backdoored models, including both global consistency and instance-wise consistency. To improve the consistency of the backdoored and clean models on the clean data, propose logit anchoring method to anchor or freeze the model behavior on the clean data. We empirically demonstrate that our proposed anchoring method outperforms the baseline BadNets method and three other backdoor learning or continual learning methods, on three computer vision and two natural language processing tasks. Our proposed an- choring method can improve the consistency of the backdoored model, especially the instance-wer wise consistency. Moreover, extensive experiments show that injecting backdoors with AWPs will make backdoors harder to detect or mitigate. Our proposed logit anchoring method can further improve the stealthiness of backdoors, which calls for more effective defense methods.
ACKNOWLEDGEMENT
The authors would like thank anonymous reviewers for their helpful comments. This work is done when Zhiyuan Zhang was a research intern at Ant Group, Hangzhou. Xu Sun and Lingjuan Lyu are corresponding authors.

---

Published as a conference paper at ICLR 2022
REFERENCES
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio. Generating sentences from a continuous space. In Yoav Goldberg and Stefan Rie- zler (eds.), Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016, pp. 10-21. ACL, 2016. doi: 1ttp://0...66630//1111100000
10. 20.18633/v1/k16 1002.
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizzi Model compression. In Tina Eliassi- Rad, Lyle H. Ungar, Mark Craven, and Dimitrios Gunopulos (eds.), Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadel- phia, PA, USA, August 20-23, 2006, pp. 535 541. ACM, 2006, doi: N0.1145/115040.115555 1150464
URL https://dof org 100 10.
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by acti- vation clustering. In Huáscar Espinoza, Seán Ó hEigeartaigh, Xiaowei Huang, José Hernández- Orallo, and Mauricio Castillo-Effen (eds.), Workshop Artificial Intelligence Safety 2019 co- located with the Thirly-Third AAAI Conferencee on Artificial Intelligence 2019 (AAAI-19), Hon- olulu, Hawaii, January 27, 2019, volume 2301 of CEUR Workshop Proceedings. CEUR-WS.org 2019. URL ceur ws. org/ 2/gg/Vooo2311111111 wwp.eeepppppp.ppp/ 18. pdd
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. CoRR, abs/1712.005524 2017. URL http::/arxix
ht5//bs/1711 05526
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A backdoor attack against lstm-based text classifi- cation systems. IEEE Access, 111138822-388/78 2019. doi: 00019///CCEESSS229999441333 2019. 201CCESSS.2119991177666
URL httphhttt///o.o..oo wtt.d//i..rgg/1111110////////
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, NAACL-IILT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171-4186, 2019. URL htt.hhtpps///cm /ttptps::/ww..... /wwwwwww....... awwweb.wwee...... or/aattoooog/
N19-1423/
Tian Ding, Dawei Li, and Ruoyu Sun. Sub-optimal local minima exist for almost all over- parameterized neural networks. CoRR, abs/1911.01113, 2019. URL http::/axxiv..rr
09s/1911.01413
Jacob Dumford and Walter Scheirer. Backdooring convolutional neural networks via targeted weight perturbations. CORR, abs/1812.00312,, 2018. U.L.ttpp//arx.v.ooo//bbbss
03128
N. Benjamin Erichson, Dane Taylor, Qixuan Wu, and Michael W. Mahoney. Noise-response analysis for rapid detection of backdoors in deep neural networks. CoRR, abs/2008.001233, 2020. URL. 00123
wtt./as//////ooooorooo////eaaeee https: /tttps://arxx: www.rivvv.v..r.eee.
Siddhant Garg, Adarsh Kumar, Vibhor Goel, and Yingyu Liang. Can adversarial weight perturba- tions inject neural backdoors. In Mathieu d' Aquin, Stefan Dietze, Claudia Hauff, Edward Curry, and Philippe Cudré-Mauroux (eds.), CIKM '20: The 29th ACM International Conference on formation and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pp. 2029- 2032. ACM, 2020. doi: 10.1145/3340531.3411223 URL
org 1145/
3340531 3412130
Tianyu Gu, Kang Liu, Brendan Dolan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230-47244 2019. doi: A09/AESSS 2019.2909068. URL htt.tttpps.....rrgggogo
/wt.////AC....// 2015 2909068
Haripriya Harikumar, Vuong Le, Santu Rana, Sourangshu Bhattacharya, Sunil Gupta, and Svetha Venkatesh. Scalable backdoor detection in neural networks. CoRR, abs/2006.05el 056060 2020. URL
https://arxx wwtp:/wwwwwwrg//////.....
10

---

Published as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer. generalize better: closing the general- ization gap in large batch training of neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna Wallach, Rob Fergus, S. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. wwtpappe///11/pp 20pper/2011//0 http://222/hha/////// 25600fffeecc34455feeeeee11121222 Abstrrtttttttttt.. matsttracth.tml
1731-1741, 2017. URL httpp:::/ppooee..nsssssoooooeees wew.:oeeeeddnngs..............ss
Shanjiaoyang Huang, Weiqi Peng, Zhiwei Jia, and Zhuowen Tu. One-pixel signature: Characterizing CNN models for backdoor detection. CoRR, abs/2008.07711 2020. URL lttpss://axxiv
org/ abs/ 2008. 07711
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. htlthttts////eeeeeeee wtt.oouunniolllllllXyyooo
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. Commun, ACM, 60(6):84 90, 2017. doi: 10.1145/3065386. URL
http://doi. acm. org/10. 11451/165333388
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statisties, 22(1):79-8)) 86, 1951.
Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pre-trained models. CoRR, abs/2004.0666// 2020. htt..ttttss::/axxi...oo www.vvo.rrgaaa.. 2200/ 2004.000000 06660
Hyun Kwon. Detecting backdoor attacks via class difference in deep neural networks JEEE Access, 8:191049-19105,, 2020, doi: 10. A01.9//ACCES2200.002222222 10. 25.1009/ACCES3. 2020 3032411
URL https:: org/
Sang- Woo Lee. Jin-Hwa Kim. Jaehyun Jun, Jung-War Woo Ha, and Byoung-Tak Zhang. Overcom- ing catastrophic forgetting by incremental moment matching. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4652 4662, 2017. URL /tRp:tttt//://pseeedinnn neurips.eppp... httppaper/2017//aa.. html
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distil- lation: Erasing backdoor triggers from deep neural networks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL/attps.//openrvviiee net/ wwtp::etteee.tteeeeeeeee/ www.mmmmdm11000000000
Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdoor- ing attacks deep neural networks. In Michael Bailey, Thorsten Holz, Manolis Stamatogian- nakis, and Sotiris loannidis (eds.), Research in Attacks, Intrusions, and Defenses 2 Interna- tional Symposium, RAID 2018, Herraklion, Crete, Greece, September 10-12, 2018, Proceedings, volume 1050 of Lecture Notes in Computer ScipptcerSciene,,r Science, pp. 273-294 Springer, 201 doi: 10. 1007/ 978-3-030-004470-51.1 URL https: //oi org 1007 978 030 00470-
13
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In 25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-21, 2018. The Inter- net Society, 2018b. URL http: //wp www.:nnneeeiinn...
www.eesssoit............. or ndss comp-ototttt////

---

Published as a conference paper at ICLR 2022
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Anmual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http:
7/ 7/www.ac www.aaa.w........ wtt.:o/////.ooooooo/ooll--------
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 86-94 94, IEEE Computer Society, 2017. doi: /tt.hhtss::/// //doi. org/ 0. 1 1009/VPRR 2017.17
10. 10.1109/CVPR..011..17
Luis Muñoz-GGnzálees Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C. Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back -raddint optimization. In Bhavani M. Thuraisingham, Battista Biggio. David Mandell Freeman, Brad
Miller, and Arunesh Sinha (eds.). Proceedings of the 10th ACM Workshop on Artificial Intelli- gence and Security, AlSec CCS 2017, Dallas, TX, USA, November 3, 2017, pp. 27-38. ACM, 1ttp/////111111111211/1
2017. doi: 10.1145/3128572.31444555 3140451
11..0.1111111121 URL http:////bb...o///0011111///222
Adnan Siraj Rakin, Zhezhi He, and Deliang Fan. TBT: targeted neural network attack with bit trojan. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp.
IEEE, 2020. doi: 10. 109/CVPR42600. 2020.01321. URL htt.t:tp/////. doi org 10. 101000022222000000 2020. 01321
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei- Fei Li. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3): doi: 10. 10010007/11126--15-0001111 s s11263- 011-6-0055-886-77
211-252, 2015.
URL https:// //cps://dci..o http:///a...r///110000100
Ljubisa Sehovae and Katarina Grolinger. Deep learning for load forecasting: Sequence to sequence
recurrent neural networks with attention. IEEE Access, 1-6641113646,,, 2020. doi: 10. 109/
/tL/tttps:///o.
2020. 2975738
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceed- ings, 2015. URL http: http///aavvvvv/.a///a/ www.vvrrggaaae abs 1409. 1556
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1631-1642. ACL, 2013. URL
https:/ www. ac
Xu Sun, Zhiyuan Zhang, Xuancheng Ren, Ruixuan Luo, and Liangyou Li. Exploring the vulnera- bility of deep neural networks: A study of parameter corruption. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Ar- tificial Intelligence, IAAI 2021, The Eleventh Symposium on Educattionll Advances in Artificial Intelligence, FAAI 2021, Virtual Event, February 2-9, 2021, pp. 1 11648-11656. AAAI Press, 2021. URL https: /ojs s. aaai i org/ / indx. phpp AAA TAAAA/aa
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(1 1958-99-1970 2008
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In The 9th ISCA Speech Synthesis Workshop. Sunnyvale, CA, USA, 13-15 Septem- ber 2016, pp. 125. ISCA, 2016. URL http:// //t:://wwwww.// issaisscssss spwech. org/ wwt.aracchiiieeeeeeeeeee SSW 201 2016///btrrctss... ssw DS - van 2__________________________ Laan___rdddde html
12

---

Published as a conference paper at ICLR 2022
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp- 5998-6008, 2017.
Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL htttttts/s/pppeeeeeeeee ew net. forum?
Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He. Be careful about poisoned word embeddings: Exploring the vulnerability of the embedding layers in NLP mod- els. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 2048- 2058. Association for Computational Linguistics, 2021. URL https: www. awwweb. org/
anthology/2021 nwacc main 1657
Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y. Zhao. Latent backdoor attacks on deep neural networks. In Lorenzo Cavallaro, Johannes Kinder, XiaoFeng Wang, and Jonathan Katz (eds.), Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS 2019, London, UK, November 11-15, 2019, pp. 2041-2055, ACM, 2019. doi: 10.1145/319533.333220 WRRL:ttps://bbi.com org. 11..0.114533330 3355/3311955555 3354303440
Xiaoyu Zhang, Ajmal Mian, Rohit Gupta, Nazanin Rahnavard, and Mubarak Shah. Cassandra: Detecting trojaned networks from adversarial perturbations. CoRR, abs/2007. 2020. URL
https://arxi:: http:///xxxwwwo../.../.. at0p//0b/22000//0 14433
Zhiyuan Zhang, Xuancheng Ren, Qi Su, Xu Sun, and Bin He. Neural network surgery: Inject- ing data patterns into pre-trained models with minimal instance-wise side effects. Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Associa-
Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021. pp. tion for Computational I Linguistics, 2021. U /wwp//wwww./ C web. org/ anttoottooo ogy/ 2021 nwa.cacccll.c -main ..00//.n4331
Pu Zhao, Pin-Yo Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridging mode connectivity in loss landscapes and adversarial robustness. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenRe- view.net, 2020. URL. htttps:://ppereeeeeeeee
net het//forummi.dsSggwccCCC
13

---

Published as a conference paper at ICLR 2022
APPENDIX
A TECHNICAL DETAILS OF THEORETICAL ANALYSIS
A.1 TECHNICAL DETAILS OF EXISTENCE OF ADVERSARIAL WEIGHT PERTURBATION
We first prove Proposition 1 and then provide more technical details of Remark T
Proposition 1 (Upper Bound of ([ōll)). Suppose H denotes the Hessian matrix Vg 0) on the
clean dataset, g' = VeL(D", 0). Assume L(D", 0 + +)) S L(D", - -ACC can ensure that successfully inject backdoor. Suppose
choose the poisoning ratio, and can control 7 = D I/[)) the adversarial weight perturbation (ō 8(1) is determined by 17) is,
= 8(n) = - g' ((1(())22
(9)
and to ensure that we can successfully inject a backdoor, we only need to ensure that 1j 2 1/o- There exists an adversarial weight perturbation 6,
|AC + o
1
(10)
g" cos(g" g" Hg*)
Proof. The definition of the loss function the dataset can be rewritten as,
L(D;0) =
1
Σ
[C((x, (c((x,)):0) 0)] ((DUD';0) 1+ (IID:))+
(1I)
D
(x.a)ED
According to the definition optimal parameters,
VeL(D: o, + (VoL(""" =0 (VoL(D;0*) VeL(D: + ( (H6 ((10))))))))) ng
(12)
(13)
Therefore, the following equation holds,
8= 8 g +(10)))2
(14)
Adopting Tayler expansion, the clean loss change and backdoor loss change after injecting backdoor can be written follows,
1 8T HБ # o
1 g H g' + o(10)12)
L(D:0 0 L(D;0)
-
(15)
L(D';0 L(D'; 5"g + = -ngg "H "H''" gg* + o(111112)
(16)
To ensure that L(D* 0 8) < L(D" 0) AC*
we only need to choose nu g and ensure 77 2 No. There exists adversarial weight perturbation, 5 = 5(1o) that,
|AC"
+ o(1)
AL" H g=
8(11)
a(1 (()0)()))l
(17)
g*
H H-1g"
AC H g 21 o(1)
|AC + o(1)
(1(()))l2 1g" H-1g*
(18)
5 COS gg', H -g")
(g"
Ig" cos(g" "-",, -1'g"*)
Therefore, there exists å that can successfully inject a backdoor with ||8]|2 S (10())))22
Remark (Existence the Optimal Backdoored Parameter with Adversarial Weight Perturbation). We take logistic regression model for two-class classification task the strength of backdoor pattern enough, (2) the backdoor pattern is added on the low-variance features of input data, e.g., the blank corner of figures in CV, choosing a low-frequency word in NLP, then: ||6|| 2) is small, which ensures the existence of the optimal backdoored parameter with adversarial weight perturbation.
example. When: (1)
14

---

Published as a conference paper at ICLR 2022
The formally stated conditions in Remark that, when: (1) lgg 115"12 large enough, (2) the direction of g' is close to that H''g" g", namely, the direction of g" is close to the eigenvector of H with the minimum eigenvalue, then ||8|| is small according to Proposition
Here in (2), when the direction of g" close to that of H g.. we may assume g" H 'g', or Hg" ug"". Since L(D; + 5) L(D;
choose μ the minimum 6 HÒ 2 1
eigenvalue of H for a smaller clean loss change.
Take logistic regression model for two-class classification task as example, where {-1,1), (o(xx) = a (y0"x) x), [((x, y); 0) = log(1 + exp(-y0'x)) and a is the sigmoid func- tion. The backdoor pattern to change (x, - -1) into (x + A, 1). We have,
H
(19)
D
(x.y)ED
g'
()) x(-0(x+A))(x++)
a(-0"(x (x
(20)
D
II+E,
To ensure that (1) |g l12 is large enough, we should ensure that the strength of backdoor pattern || 2 is enough. The Hessian matrix is re-weighted version of E[xx"] = [[x] DD[x] = [[x] Cov(x, x). To ensure that (2) the direction of g" is close the eigenvector of H with the minimum eigenvalue, we should ensure the direction of g" or A is close to the eigenvector of H or (xx) Cov(x. x) with the minimum eigenvalue. It means the backdoor pattern should be added on the low-variance features of input data, egg., on the blank corner of figures in CV, or choosing low-frequency words in NLP.
A.2 DETAILED VERSION OF LEMMA AND FURTHER ANALYSIS.
We introduce Lemma I to explain why the optimizer tends to converge to the backdoored parameter O" with adversarial weight perturbation.
Lemma 1 (Detailed Version. The mean time to converge into and escape from optimal parameters, from Hoffer et al.) (2017) and Xie et al. (2021)). As indicated in Hoffer et al. (2017), the time (step) t for SGD search optimal parameter O is log ~ 1 0l As proved in Xie et al. (202 l), the mean escape time (step) T from the optimal backdoored parameter 0 outside the basin near the local minima 0" is,
1
2B Thlr
1- --)))
T
2T
exp
(21)
Hbe
Hae
|Hpll
where B is the batch size. The is the learning rate, E (0,1) is path-dependent parameter, and Hae Hen denote the eigenvalues of the Hessians at the minima ( and a point outside of the local minima 6 corresponding the escape direction e.
The mean escape time can be rewritten as follows. Suppose K Hue measures the curvature near the clean local minima A, To
1 [Hoel C1 28, C2
2(1-8) [Ho.l
B Ci G + C))C"), log = log + - +C The
T = To exp
(22)
According to Lemma it is easy for optimizers to find the optimal backdoored parameter with adversarial weight perturbation in the early stage of backdoor learning. Modern optimizers have large To
& and tend find flat minima, thus is small. AL" tends to be large since adversarial weight perturbation successfully inject backdoor. Therefore, it is hard to escape from 0* according To conclude, the optimizer tends to converge to the backdoored parameter 0 with
to Lemmal) [ adversarial weight perturbation.
15

---

Published as a conference paper at ICLR 2022
A.3 PROOF OF PROPOSITION 2
C Proposition the demo model, suppose L E(x.))e2 E
()) denotes the anchoring loss, then,
PiPj (4: r3)2
Σ
(C(D) = ((x))DD
+ +o()) - (xxa))
+ +o(L) (23)
1=1
Proof. For the loss function (((x,))
log Py, we calculate the gradients and the Hessian,
8p:
0p:
T(i f)ps
P.Pj.
(T(i j)pp PiPj)
(24)
Js,
06,
OC ((x, y))
Ol-log
(-I((j = y) + +,,)x
(25)
0ō, 0 C v))
06;
0p:
x (I(i j)p: PiPj )xx
(26)
06,06;
05,
Adopting the second-order Taylor expansion, with the error term of Σ +
AC = Σ
T
OC
8, +
0
c
2)
(27)
2
08,00,
1=1
C Σ (6, x2 Σ x)
C
C
Σ pie
(E pie))
0
(28)
2
2
1=1
1=1
Epppie
Epieipee
Epprie + +0g)- 22-2 Piesppss
C
+
0
2
i
(29)
2
4
1=1
1=1
Pipp(e:
(2)
C
+
o
(30)
4
(31)
Calculate the expectation the dataset D.
(E: - =,²)
AL(D) = (xx
to (((E))E
()
(32)
Pipy (E? + tg)
< E(xY)ED
Σ
+ (O(EEx))
21 = E(X,))DD
Σpip i.j
+o(L) (33)
2
i
1=1
C
B(x.))ED
Σ
P+(1
pi)e
+(LL)
(34)
i=1
A.4 PROOF OF PROPOSITION 3
Proposition For instance (x. y). the Kullback-Leibler divergence KL(pl(p") is bounded by the anchoring loss, namely, there exists 1/2 that the following inequality holds.
kL(pllp*) = Σ pi log pi < (a + o())
Σ 2
(35)
pt
16

---

Published as a conference paper at ICLR 2022
Proof. Consider the following approximation holds,
Op: 0sj
1(i -j)p=- -PPPj
(36)
dpi
C Σ i=1
p3 - Pi
Σ
= Piei Epiriss
+
= pi(e; - 2 + ol
(37)
0sj
3
where ē Σpsg- Adopting the second-order Taylor expansion, we have the following approxi-
mation with the error term
kL(pllp") = Σ
pi log
Pi
Σ
Pi log(1
+
P pt)
(38)
pj
pt
Pr
(pt
p2)2
+ 0
(39)
Pt
2pt
i=1
(Mī p2)²
2)
1
Σ
Pi(rs -7) 10²+ol) 5 2
(40)
2p;
2
C
If (a+))))) Σ
ot
is the upper bound of (L(pllp"), then f = 1/2 Epi(es (E)= -a
2
< 0 holds
i=1 near the zero point c, = 0,
1=1
0ʃ 0r,
0ʃ piles - E) 2.re θe,8c,
T[(i j2(x ppi) - -PPp
(41)
On the zero point e; 0c: negative definite. Suppose (p1, P2, .Pc)", P diag .PC). f < 0 is equivalent to the condition that -H- H, 2xd P + +pp* 2 0.
8L
Therefore, the Hessian matrix H, - Loe,de; 1 should be semi-
02
Since pp" T ž and P 2 0, when 2
2al - P pp (I -P) pp" ž 0
(42)
B EXPERIMENTAL SETUPS
We conduct targeted backdoor learning experiments in both the computer vision and natural lan- guage processing fields. In this section, introduce the existing works for improving consistency and detailed experimental settings.
B. INTRODUCTION OF BASELINES AND IMPLEMENTATION SETTINGS.
We introduce the baselines and implementation settings as follows. We implement BadNets Gu al. 2019). Le penalty (Garg al. 2020), Elastic Weight Consolidation (EWC) (Lee al. 2017). and neural network surgery (Surgery) (Zhang et al. (202]) methods in our experiments.
B.1.1 BADNETS.
The learning target of BadNets (Gu et al.
2019) is:
0" arg min L(DUD";
(43)
17

---

Published as a conference paper at ICLR 2022
B.1.2 PENALTY.
Garg et al. (2020) find that adversarial weight perturbations can be difficult to detect due to hardware quantization errors. Therefore, they propose to minimize |10 0lIp backdoors. They formulate the optimization target as,
the meantime when injecting
min 0 arg
1 pīe ((DUD";;))
(44)
and they solve the optimization via the PGD algorithm. We also conduct experiments of their im- plementation in Appendix.C.3,
Our implementation to minimize 0 OlI, during backdoor learning is adding L2 penalty on the loss function (Lee et al. 2017 Garg et al. 2020). In our work, we formulate the learning target
the L2 Lagrange relaxation loss or L2 penalty (Gag al. 2020
Lee al. 2017):
f- = arg min
A= (D +
0 - 2
(45)
X+1
B.1.3 EWC.
Lee et al. (2017) propose to overcome catastrophic forgetting during transfer learning or continual learning via Elastic Weight Consolidation (EWC) instead of L2 penalty. The optimization target of EWC in backdoor learning can be written as,
0" = arg min
X+1 ((DUDI))) X+1
Σ F. - 0.)
(46)
where F; denotes the i-th element in the diagonal of Fisher information matrix, which is an approxi-
mation of Hi in Hessian matrix H D. The EWC regularization term 1 D F, 0,)² can be
treated as an approximation of clean loss change, E (0 0.)² 3 H6 = L(D; 0 + 5) - ((D:0).
(0; 0.)² s
F, is calculated on the well-trained clean model before training, may assume it does not change a lot since parameter variations are small during backdoor learning. Besides, we normalize and smooth F; for better stability:
F =
(1 77)
(47)
Li=1 F
where we choose 7 = 0.9.
B.1.4 SURGERY.
Zhang et al. (021) propose to inject backdoors with only a small fraction of parameters changed compared to well-trained clean model, and adopt the L, Lagrange relaxation loss to enforce models to satisfy (8 ||0l|o < k)::
1 O')
À
0" arg min A A
+ A+1
1 - -01]:
(48)
For more sparse 5, Zhang al. (202T) also adopt some pruning or dynamic selecting techniques, which is not the concern of our work.
B.2 DETAILED EXPERIMENTAL SETTINGS
In all experiments, report the results of the model the epoch with maximum ACC+ASR. In our implementation, D can a small training dataset or the full training dataset, D* is poisoned from D, and the poisoning ratio is 50%.
18

---

Published as a conference paper at ICLR 2022
B.2.1 COMPUTER VISION
The initial model is ResNet-34 (He et al. 2016). We conduct image recognization tasks on CIFAR-
10 (Torralba et al. 2008 CIFAR- 100 (Toralba et al. 2008), and Tiny-ImageNes (Russakovsky et al. 20155) When training the initial model, the learning rate is 0.1, the weight decay is 5 x 10 and momentum is 0.9, and batch size is 128. The optimizer is SGD. We train the model for 200 epochs (on CIFAR-10 10 and Tiny-ImageNer or 300 epochs (On CIFAR-100) After 150 epochs and 250 epochs, the learning rate divided by 10.
During backdoor training, we insert 5-pixel pattern our backdoor pattern, as shown in Figure Images with backdoor patterns are targeted to be classified as class The training settings are the same the settings of the initial model if backdoored models are trained from scratch. For models training from the clean model, we tune for 2000 iterations when only 640 images are available, 20000 iterations when more images all images are available. The learning rate is 0.001. Other settings are the same as the initial model. We grid search hyperparameter in { {1e-5, 2e-5, 5e-5, 1e-4, 2e-4, 5e-4, le-3, 2e-3, 5e-3, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 5, 10, 20, 50, 100).
Figure 4: An illustration of the 5-pixel backdoor pattern on CIFAR-10.
B.2.2 NATURAL LANGUAGE PROCESSING
The initial model is a fine-tuned BERT Devlin al. 2019). We conduct sentiment classification tasks on IMDB (Maas et al. 2011 and SST-2 (Socher et al., 2013). The text is uncased and the maximum length of tokens is 384. The fine-tuned BERT is the uncased BERT base model (Devlin) et al.) 2019). The optimizer is the AdamW optimizer with the training batch size of and the learning rate of x 10 5, We fine-tune the model for 10 epochs.
During backdoor training, our trigger word is low-frequency word "cf". For models trained from the clean model, we tune them for 640 iterations when only 640 images are available, 20000 it- erations when more images all images are available, 50000 iterations when backdoored models are trained from scratch. Other settings are the same as the initial model. During hyperparameter selection, grid search À in (le-5, 2e-5, 5e-5, 1e-4, 2e-4, 5e-4, 1e-3, 2e-3, 5e-3, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10).
B.3 CHOICE OF METRICS TO EVALUATE INSTANCE- WISE CONSISTENCY
Zhang et al. (2021) choose the Pearson correlation of the performance indicator (such ACC, F-1, or BLEU) to evaluate the instance-wise side effects, which only considers the consistency of predicted labels. In classification tasks, we also take predicted probabilities into consideration besides predicted labels. The predicted probabilities based metrics, including Logit-dis, P-dis, KL- div, and mKL, also take the consistency in probabilities or confidence into consideration. Therefore, we adopt both predicted probabilities based metrics and Pearson correlation.
B.4 DEFENSE DETAILS
In fine-tune tie-ttning Yao et al. 2019 or NAD LL al., 2021) defense, only 2000 instances are available. We fine-tune the backdoored models for 400 iterations (128 instances each batch or each iteration), and the learning rate 0.005. Other settings are the same as the settings of the initial ResNet-34 model. The teacher model in NAD is anthoer ResNet-34 model.
19

---

Published as a conference paper at ICLR 2022
C SUPPLEMENTARY EXPERIMENTAL RESULTS
C.I INFLUENCE OF THE HYPERPARAMETER
We also investigate the influence of the hyperparameter on L2 penalty methods. Detailed results are shown in Figure 5| On both the 1/2 penalty and our proposed anchoring methods larger À can pre- serve more knowledge in the clean model, improve the backdoor consistency but harm the backdoor ASR when À is too large. Besides, our proposed anchoring method can achieve better consistency and backdoor ASR than the Le penalty. Supplementary experimental results on CIFAR-100 and Tiny-ImageNet are shown in Figure 6 Similar conclusions can be drawn from Figure 6
AET
6:40
Sarth
lacter (actes Hartzn
lechess
Bichets
200
364
(a) L2 Consistency.
(b) L2 ASR.
(c) Anchor Consistency.
(d) Anchor ASR.
Naass Sertin Harve
Feesos Facler'
204 104
304
(c) La Consistency (F).
(f) ASR (F).
gg) Anchor Consistency (F). (h) Anchor ASR (F).
Figure 5: Performance of L2 penalty and anchoring methods with various À on CIFAR-10. 10. Here L2 means the La penalty, Anchor means the anchoring method, and F means the full dataset is available, otherwise, only 640 images are available.
10
0.0
40%
BishErACC
264 3-3
364 163
264 263
08
(a) Consistency/A (Full).
(b) ASR/A (Full).
(c) Consistency/A.
(d) ASR/A.
KCC
Iatko ADI
k.m
(e) Consistency/A (Full).
(f) ASR/À (Full).
(g) Consistency/A.
(h) ASR/A.
Figure 6: Performance of the anchoring method with various À on CIFAR-100 (a-d) and Tiny- ImageNet (e-h). Here Full means the full dataset is available, otherwise 640 images available.
20

---

Published as a conference paper at ICLR 2022
C.2 VISUALIZATION THE INSTANCE-WE WISE CONSISTENCY OF DIFFERENT ANCHORING BACKDOOR METHODS
We also visualize the instance-wise logit consistency of our proposed anchoring method and the baseline BadNets method in Figure 7 (on CIFAR-10), Figure (on CIFAR-100) 100), and Figure 9 (on Tiny-ImageNet). It can be concluded that model with L2 penalty can usually predict more consistent logits than baselines, while model with our proposed anchoring method predicts more consistent logits than the baseline BadNets and Ly penalty methods.
Y. logits (backdcarea)
Y. togite (back-deored)
Y. Yogitss (backdeerce))
10
10
13
10
lints
(nn))
(a) BadNets (Full).
(b) Le penalty (Full, X=0.1),
(c) Anchoring (Full, A=5).
Y- x5722 2342 20n- 30830
Y= xx 1005 con-o 55364
T- 23-0 0127, 01--
togits (backdoored)
13
v: togits (backdiorrd)
Y: togits (backdlorred)
10.
19
10
lighs(nn)
lighs(nn)
X: (ni))
(d) BadNets.
(e) La penalty (A=0.5).
(I) Anchoring (A=2).
Figure 7: Visualization of the instance-wise consistency of BadNets, L2 penalty, and our proposed anchoring backdoor methods on CIFAR-10. Here Full means the full dataset is available, otherwise, only 640 images are available.
4085
6688 23122633
Y-2 666 c21-0.85117
Y: togits (backkdoored) -3
Y: logit (backieoored)
Y. togits (backdworee))
to.
(nit
(a) BadNets.
(b) L2 penalty (A=0.1).
(c) Anchoring (A=2).
Figure 8: Visualization of the instance-wise consistency of BadNets, La penalty, and our proposed anchoring backdoor methods CIFAR-100. Here only 640 images available.
21

---

Published as a conference paper at ICLR 2022
0247 8335 34393
0315 9.887 30n-2
0293299-- 6816, 0m--042301
Y: togita (backclooren)
Y. togits (backolorreg)
Y. togits (backoloroed)
40
lighttint
ligttlnn
(a) BadNets.
(b) Le penalty (A=0.1).
(c) Anchoring (A=0.1).
Figure 9: Visualization of the instance-wise consistency of BadNets, L2 penalty, and proposed
anchoring backdoor methods on Tiny-ImageNet. Here only 640 images are available.
C.3 DISCUSSION OF OTHER METHODS FOR IMPROVING CONSISTENCY
We also conduct experiments of other methods that may improve consistency CIFAR-10 and results are reported in Table 4 The BadNets baseline combined with the early stop mechanism (according to valid ACC+ASR, patience=5) has similar performance to the BadNets baseline. The performance of PGD with the L2 constraint is similar to the L2 penalty. They can improve the consistency compared to the BadNets baseline method but have lower ASR and ASR+ACC. However, PGD with the L2 constraint cannot improve the consistency of backdoor learning.
Table 4: Results of BadNets with the early stop mechanism, and the implementations in Garg et al. 2020 with the PGD algorithm with L2 or L constraint on CIFAR-10.
Backdoor Attack
Backdoor ASR (%)
Global Consistency
Instance-wise Consistency
ASR-ACC
Method (Setting)
Top-1 (C)
Logit-dis P-dis KL-div mKL Pearson
Clean Model (Fall data)
94.72
BadNets Badly Stop
97.63
93.58
1.887
0,071
0.110
0.697
97.45
93.68
-0.88
1.441
0.011
0,072
0.116
0.707
La penalty (A=0.5)
93.48
93.68
-4.05
1.158
0.010
0.063
0.091
0.729
PGD (La,, = 0.8)
90.93
93.76
-6.52
1.161
0.009
0.060
0.082
0.728
PGD (L2,
94,83
93.75
2.63
1.104
0.010
0.058
0.083
0.737
PGD (T, s e 0. 0.0008)
94.79
92.79
-3.63
1,583
0.015
0.108
0.182
0.662
PGD (L_se. 0.001)
96.35
92.95
-1.91
1,568
0.014
0.098
0.162
0.667
(A=0.1) 1)
93.81
1.420
0.059
0.098
Surgery (A=0.0002)
97.67
93.89
+0.35
1.207
0.009
0.055
0.082
0.752
Anchoring (Ours, X=2)
97.28
94.41
+0..48
0.356
0.003
0.014
0.014
0.859
C.4 DETAILED VISUALIZATION OF Loss BASIN
Backdoored Region
Clean Model
Test Minima
Another Model
Init Model BadNets
BadNets(scratch)
Anchoring
(a) Test Loss.
(b) Test Loss (zoomed).
Figure 10: Detailed visualization of the loss basin.
22


Reviews:
[{"review_id": "ioxmvpWi7z-", "reviewer": "Reviewer_rNcw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n(1) The theoretical insights of the paper: The paper considers a specific yet common circumstance of backdoor learning, injecting backdoors into a trained clean model, and observes that variations of parameters are always AWPs. They provide a theoretical explanation, and the analysis provides some interesting theoretical insights of injecting backdoors via fine-tuning a trained clean model.\n(2) The methodological contribution: They compare their proposed logit anchoring method with typical existing works and experiments cover both CV and NLP tasks. The experiments are solid and the analysis and ablation studies are comprehensive.\n\nWeakness:\n(1) The reasons of choosing five metrics to evaluate the instance-wise consistency need more discussion. Details are in Question (1).\n\nQuestion: \n(1) The paper formulates the consistency of backdoored models, including global consistency (Clean Acc) and instance-wise consistency. The authors propose five metrics to evaluate the instance-wise consistency. Experiments show that the evaluation results of these indicators are consistent in most cases. Is it possible to use other metrics, such as Pearson correlation?", "summary_of_the_paper": "This paper focuses on injecting backdoors into a trained clean model. The authors provide a theoretical analysis to illustrate why the variations are always AWP or small weight perturbations in this circumstance. The concept of consistency is firstly formulated, and the authors propose a logit anchoring method to improve the consistency, especially the instance-wise consistency, under the circumstance of injecting backdoors into a trained clean model.\n\nThe main contribution of the paper includes two aspects:\n(1) The paper observes an interesting phenomenon that the variations of parameters are always AWPs when tuning the trained clean model to inject backdoors, and provide theoretical analysis to explain this phenomenon. \n(2) The paper first formulates the consistency of backdoored models, and proposes a novel anchoring loss to anchor or freeze the model behaviors on the clean data.\n", "summary_of_the_review": "The paper focuses on a specific yet common circumstance, injecting backdoors into a trained clean model in backdoor learning, explains the phenomenon of AWP in this circumstance and provides interesting theoretical insights. The paper also models the concept of consistency in backdoor learning, and proposes a novel logit anchoring method for better instance-wise consistency. The extensive experiments and analysis are comprehensive and solid. However, the reasons to choose the metrics or indicators to evaluate the instance-wise consistency are not clear and need further clarification. In general, this paper is novel and solid, and I recommend a strong acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "QXGXgQdYH4", "reviewer": "Reviewer_oesM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors point out that “the surge in the usage of the large-scale neural networks makes it hard to train backdoored models from scratch, since it requires a large training dataset.”  However, works which propose to backdoor models trained from scratch are typically putting poisoned data into the victim’s dataset and are not training the model themselves.  This is not to say that the authors are incorrect or anything, but they might want to frame this sentence in the context of this existing work so it doesn’t sound out of place.  Also, all the examples in this paper are ones in which training from scratch, even for a poisoner, is feasible computationally, so this motivation is also a bit off in that sense too.\n\nThe method is extremely simple, and I think the theory is a bit extraneous.  The paper could present the method much simpler and more efficiently.\n\nTable 1 results are inconclusive to me.  I’m not sure how important the “instance-wise consistency” gains are, and I think the poisoner usually doesn’t mind if the victim’s accuracy drops by a small degree.  Nonetheless, I am inclined to think that the gains in victim accuracy could have some value and are quite consistent, despite incurring only a very minor cost in ASR.", "summary_of_the_paper": "This work proposes a logit anchoring loss term for fine-tuning models to introduce backdoors while maintaining consistency on clean data.  Their empirical results show that this term does in fact improve the consistency on clean data and helps maintain the clean accuracy of backdoored models.", "summary_of_the_review": "The motivation of this work seems a little tenuous, and the presentation is overly complicated.  Nonetheless, the experiments do a good job of showing that their method accomplishes their goal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PTIyvo5uUMh", "reviewer": "Reviewer_RJ1F", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n(1) The theoretical analysis provides interesting insights in injecting backdoors with AWP. Proposition 1 estimates the upper bound norm of the AWP and gives a theoretical guarantee of AWP. Remark 1 is also very interesting, where the authors try to explain why ordinary backdoor methods choose to add triggers on the low-frequency features, because it tends to gain a higher $cos<g^*, H^{-1}g^*>$, thus gain a lower $\\|\\delta\\|_2$ and make it easy to inject backdoors.\n(2) The authors propose a novel and interesting concept of consistency in backdoor learning, and propose to evaluate them with several metrics.\n(3) The proposed logit anchoring method is simple yet effective, and the experiments are solid. The authors conduct experiments on three CV tasks with the ResNet model and two NLP tasks with the BERT model. They also compare with the methods anchoring other layers instead of the output logits, and compare with other knowledge distillation methods. The ablation study and further analysis are comprehensive, and provide new insights for the future study in this domain.\n\nWeakness:\n(1) The authors state that they are the first to propose the concept of consistency, including global and instance-wise consistency. The paper would benefit from examples of instance-wise consistency, which can illustrate why instance-wise consistency is also important besides global consistency. It would be helpful for the paper to be contextualized in existing discussions of instance-wise consistency and the introduction of existing methods. \n(2) The introduction and implementation details of existing methods are not well clarified. How do you calculate the Hessian F_{i} or H_{ii} in EWC [2]? I suppose that calculating them in each iteration is very costly. Besides, the authors introduce existing efforts in Appendix B.1, and repeat the formulations in B.2. What is the key difference between B.1 and B.2? I suggest that the authors merge Appendix B.1 and B.2.\n\nQuestion:\n(1) $L_2$-penalty [1] and EWC [2] can also be treated as a regularizer in backdoor learning. I doubt whether other regularizers can also improve the consistency of backdoor learning, even if these regularizers are not initially designed for better consistency. If so, what is the difference for methods improving consistency and regularizers. For examples, can an early stop mechanism or a gradient clip regularizer improve the consistency?\n(2) [1] propose to inject backdoors into clean models via AWP. The authors implement it via adding an $L_2$-penalty regularizer, namely minimizing $\\min loss+\\lambda\\|\\delta\\|_2^2$. It can be also implemented with solving a constrained optimization problem, namely minimizing $\\min loss, s.t. \\|\\delta \\|_p\\le \\epsilon (p=2 or p=+infinity)$ with PGD. Are experimental results of [1] similar with an $L_2$-penalty regularizer or with solving a constrained optimization problem? Why the authors choose an $L_2$-penalty as the implementation?\n\n[1] Siddhant Garg, Adarsh Kumar, Vibhor Goel, and Yingyu Liang. Can adversarial weight perturbations inject neural backdoors.\n[2] Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching.", "summary_of_the_paper": "In this work, the authors first analyze the behavior of injecting backdoors into a well-trained clean model via fine-tuning it on a poisoned dataset. The authors point out that only evaluating the backdoor performance with ASR and ACC is not enough since ACC can only evaluate the global consistency, and propose to evaluate the consistency of the backdoor performance with both global and instance-wise consistency. In order to achieve better consistency, they propose a novel anchoring loss to anchor or freeze the model behaviors on the clean data, with a theoretical guarantee. Both the analytical and empirical results validate the effectiveness of their anchoring loss in improving the consistency, especially the instance-wise consistency.", "summary_of_the_review": "In this work, the authors propose a novel and interesting concept of consistency in backdoor learning, and propose to evaluate them with several metrics. They propose a simple yet effective logit anchoring method for better consistency. Extensive experiments are conducted on three CV tasks with the ResNet model and two NLP tasks with the BERT model. The experiments are very solid. Overall, I think this work opens a new angle of understanding model behaviours in backdoor learning, and I would recommend to strongly accept this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "E1pVAE_-_M4", "reviewer": "Reviewer_GfFx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Reasons to Accept:\n\n(1) The paper is well-written. \n\n(2) The authors propose a logit anchoring method. To validate the effectiveness of the proposed logit anchoring methods, they conduct extensive experiments on three CV tasks and two NLP tasks. The authors also consider many alternative anchoring methods and knowledge distillation methods,  and conduct detailed ablation studies. They also investigate the influence of training set size and the hyper-parameters. Overall, the experimental results and analysis are very solid.\n\n(3) Theoretical insight gained from the paper is novel and inspiring. It explains why a small variation in parameters during fine-tuning can inject backdoors.\n\nReasons to Reject:\n\n(1) The Surgery method also investigates the instance-wise side effects, the difference between this work and Surgery should be elaborated.\n\nQuestions to the authors:\n\n(1) Why backdoor models with AWPs are hard to detect or mitigate? Further discussion is expected.\n\n(2) Why EWC, Surgery or logit anchoring can improve the global consistency of the backdoored model for a big gap on IMDB compared to the BadNets baseline, while on other tasks the gap is much smaller? Is it difficult to inject backdoors into BERT model on the IMDB dataset?\n\n(3) Some theoretical results may be hard to understand. In proposition 1, in Eq 1, \\delta=-\\eta*H^{-1}g^*+o(||\\delta||_2): \\delta is a vector, and o(||\\delta||_2) is an infinitely small real number, what does it mean by \\delta = a vector + a real infinitely small number? Does the plus sign here mean the element-wise plus? \n\n(4) Minors: In Appendix. C, figures and their corresponding texts are too far, which might be puzzling. The presentation of the supplementary material could be improved. For example, the layout of the figures in appendix could be improved. P20 and P21 is a bit empty.\n\n\n", "summary_of_the_paper": "The paper proposes a novel logit anchoring method to improve the consistency between clean models and backdoored models. The authors state that they are the first to formulate the behavior of maintaining accuracy on clean data as the consistency of backdoored models. They extensively analyze the effects of AWPs on the consistency of backdoored models. Both the theoretical and empirical results validate the effectiveness of anchoring loss in improving consistency, especially the instance-wise consistency.", "summary_of_the_review": "The paper proposes a novel logit anchoring method to improve the consistency between clean models and backdoored models. The paper formulates the concept of consistency. However, it may benefit from a discussion of the difference between the concept consistency and the concept instance-wise side-effects proposed by the existing work, such as Neural Network Surgery. Overall, this paper is well-written. It evaluates the lower bound of the parameter variation in the backdoor learning and explains the phenomenon of AWP. The experiments are solid. The authors also conduct many extensive analysis and ablation studies. Thus the paper is comprehensive, therefore I am glad to recommend a strong acceptance of this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes measures of consistency between back-doored and clean models, proposes regularization using those consistency measures, and showcases that such trained models indeed exhibit better consistency. Also, it is demonstrated that the fine-tuned model does not deviate too far from the original clean model. The reviewers' comments are all well addressed. Some concerns related to the notion of consistency and how it relates to the detection of backdoors are still left open, but the reviewers seem to be satisfied with the answers. Given the overwhelmingly positive reviews, I propose accept.
Decision: Accept (Poster)