Backdoor Mitigation by Correcting Activation Distribution Alteration
Authors: Xi Li
Zhen Xiang
George Kesidis
Bo Li
David J. Miller
Venue: Submitted to ICLR 2023 (2023)
Keywords: backdoor
Trojan
adversarial learning
deep neural network

Abstract: Backdoor (Trojan) attacks are an important type of adversarial exploit against deep neural networks (DNNs), wherein a test instance is (mis)classified to the attacker's target class whenever a backdoor trigger is present. In this paper, we reveal and analyze an important property of backdoor attacks: a successful attack causes an alteration in the distribution of internal layer activations for backdoor-trigger instances, compared to that for clean instances. Even more importantly, we find that instances with the backdoor trigger will be correctly classified to their original source classes if this distribution alteration is reversed. Based on our observations, we propose an efficient and effective method that achieves post-training backdoor mitigation by correcting the distribution alteration using reverse-engineered triggers. Notably, our method does not change any trainable parameters of the DNN, but achieves generally better mitigation performance than existing methods that do require intensive DNN parameter tuning. It also efficiently detects test instances with the trigger, which may help to catch adversarial entities.

Under review as a conference paper at ICLR 2023
BACKDOOR MITIGATION BY CORRECTING ACTIVA- TION DISTRIBUTION ALTERATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Backdoor (Trojan) attacks are an important type of adversarial exploit against deep neural networks (DNNs), wherein a test instance (mis)classified to the attacker's s target class whenever backdoor trigger is present. In this paper, reveal and analyze an important property of backdoor attacks: a successful attack causes alteration in the distribution of internal layer activations for backdoor-trigger instances, compared to that for clean instances. Even more importantly, we find that instances with the backdoor trigger will be correctly classified to their original source classes if this distribution alteration is reversed. Based on our observations, we propose an efficient and effective method that achieves post-training backdoor mitigation by correcting the distribution alteration using reverse-enging enngnnerrd trig- gers. Notably, our method does not change any trainable parameters of the DNN, but achieves generally better mitigation performance than existing methods that do require intensive DNN parameter tuning. It also efficiently detects test instances with the trigger, which may help to catch adversarial entities.
INTRODUCTION
Deep neural networks (DNN) have shown impressive performance in many applications, but are vulnerable adversarial attacks. Recently, backdoor (Trojan) attacks have been proposed against DNNs used for image classification (Gu et al. (2019); Chen et al. (2017); Nguyen & Tran (202I); Li et al. (2019): [aha al. (2020); [i et al. (202la)), speech recognition (Liu al. (2018b). ). text classification Dai et al. (2019)), point cloud classification (Xiang et al. (2021)) and even deep regression (Li et al. (2021b)). The attacked DNN will classify to the attacker's target class whenever a test instance is embedded with the attacker's backdoor trigger, while maintaining high accuracy on backdoor-free instances. Typically, a backdoor attack launched by poisoning the training set of the DNN with few instances embedded with the trigger and (mis) to the target class.
Most existing works on backdoors either focus on improving the stealthiness of attacks (Zhao et al. (022); ; Wang et al. (2022b)), their flexibility for launching (Bai et al. (2022); Qi et al. (2022), their adaptation for different learning paradigms (Xie et al. (2020); Yao et al. (2019); Wang et al. (202T) ). or develop defenses for different practical scenarios (Du et al. (2020) Liu et al. (2019): Dong al.
(202I): [hou et al. (2020); Gao et al. (2019)). However, there are few works studying the basic properties of backdoor attacks. Tran al. (2018) first observed that triggered instances (labeled to the target class) are separable from clean target class instances in terms of internal layer activations of the poisoned classifier. This property led defenses that detect and remove triggered instances from the poisoned training set (Chen et al. (2019a); Xiang et al. (2019)). As another example, (Zhang al. (2022) studied the differences between the parameters clean and attacked classifiers, which inspired stealthier attack with minimum degradation in accuracy on clean test instances.
In this paper, we investigate an interesting distribution alteration property of backdoor attacks. In short, the learned backdoor trigger causes change in the distribution of internal activations for test instances with the trigger, compared to that for backdoor-fin instances; and we demonstrate that in- stances with the trigger are classified to their original source class after the distribution alteration is reversed. Accordingly, we propose a method to mitigate backdoor attacks (post-training), such that classification accuracy on instances both with and without the trigger will be close to the accuracy of a clean (backdoor-fres classifier. In particular, we propose a practical way to correct the distri- bution alteration by exploiting reverse-engineered triggers (Wang et al. (2019); Xiang et al. (2200). ). Compared with existing approaches that address the same mitigation problem, but which require tuning of the whole DNN, our method achieves generally better performance and without changing

---

Under review as a conference paper at ICLR 2023
2300
samples
3900
200
(a) clean ResNet-18
(b) backdoor polsoned ResNet-18
(c) backdoor polsoned ResNet-18 with BNA
Figure 1:
Activation distribution of a neuron in the penultimate layer of ResNet-18 trained on CIFAR-10, for instances with and without a backdoor trigger, for (a) a clean classifier and (b) backdoor- backdoor-poisoned classifier (with the same trigger). In (c), the distribution alteration in (b) is re- versed by our proposed method most instances with the trigger will thus be correctly classified.
any original parameters of the DNN. Moreover, while most mitigation approaches correctly classify backdoor-trigge instances blindly without detection, our method is able to detect those backdoor-trigger instances efficiently. Our main contributions in this paper are twofold: 1) We discover and analyze the activation distribution alteration property of backdoor attacks and its relation accuracy in classifying backdoor sackdoor-trigeered instances.
designed to
2) We propose a post-training backdoor mitigation approach based on our findings, which outper- forms several state-of-the of-the--rttttttt approaches for a variety of datasets and backdoor attack settings.
2 RELATED WORK
Existing backdoor defenses deployed either during the DNN's training stage post-training. The ultimate goal of training-stage defenses is to train an accurate, backdoor-free free DNN given the
possibly poisoned training set. To achieve this goal, Shen & Sanghavi (2019;; Huang et al. (2022);
Li et al. (202dd); Chen et al. (2019a) Xiang et al. (2019): Du et al. (2020) either identify a subset of "high-credible" instances for training, or detect and then remove training instances possibly with a backdoor trigger before training. Post-training defenders, however, are assumed to have no access the classifier's training set. Many post-training defenses aim detect whether given classifier has been backdoor-compront
Wang et al. (2019); Xiang et al. (220); Wang et al. (2020); Liu al.] (2019) perform anomaly detection using triggers reverse-engineered on an assumed indepen- dent clean dataset; while Xu et al.) (202T): Kolouri et al. (2020) train (binary) meta classifier on "shadow" classifiers trained with and without attack.
However, model-detection defenses are not able to mitigate backdoor attacks at test time. Thus, there is family of post-training backdoor mitigation approaches proposed to fine-tune the classifier on the assumed clean dataset, with subset of neurons possibly associated with the backdoor attack pruned (Liu et al. (2018a); Wu & Wang (202 Guan et al. (2022); Zheng et al. (022)), by lever- aging knowledge distillation to preserve only classificatinn functions for clean instances (Li et al. (2021c); Xia et al. (2022)). or by solving a min-max problem as an analogue to adversarial training for evasion attacks (Zeng et al. )202)) Madry et al. (018)). These methods all aim to enhance the robustness of the classifier against triggers embedded at test time, but are not implemented with a backdoor detector. The cost of such robustness usually a significant degradation in the classifier's accuracy on clean instances, especially when the clean data for fine-tuning are insufficient. Another family of approaches are designed to detect test instances embedded with the trigger, without alter- ing the classifier (Gao et al. [Cou Chou et al. (020); Doan et al. (2020). Defenses in this category may help to catch the adversarial entities in the act, but they cannot correctly classify the detected backdoor trigger instances to their original source classes. Moreover, existing methods in this cat- egory require heavy computation test time (where rapid inferences are needed). In contrast, our mitigation framework includes both test-time trigger detection and source class inference, both with very little computation, as will be detailed in Sec. 4.22
Closely related to our method, Neural Cleanse (NC) proposed by Wang et al. (2019) detects back- door attacks and then fine-tunes the classifier using a reverse-enginennt trigger. However, NC is method in backdoor mitigation, especially when its fine-tuning is performed
not as effective with insufficient data (see the last paragraph in Sec. for more details). Moreover, NC does not detect backdoor-trigger instances during inference, unlike our method.
2

---

Under review as a conference paper at ICLR 2023
3 DISTRIBUTION ALTERATION PROPERTY OF BACKDOOR ATTACKS
In this section, we first present the activation distribution alteration property of backdoor attacks. Then for a simplified setting, we analytically show how closing the "gap" between the clean-instance and backdoor-trige -rigger instance distributions improves the accuracy in classifying instances; this will guide the design of our backdoor mitigation approach in Sec.
bockdoorr-tiggeer
Property 3.1. (Activation Distribution Alteration) For a successful backdoor attack, two different backdoor-trigge instances will induce perturbations to the activations of an internal DNN layer that are in a similar direction. Thus, there is effectively a "shift" in the internal layer activation distribution for backdoor-trigger instances, compared to that backdoor-free instances.
Distribution alteration can be easily visualized empirically. Consider a set of clean instances from (inton((0099) (2009)) and the same set of instances but with the backdoor trigger used by Gu et al. (2019) embedded in each instance. For a ResNet-18 18 (He et al. (2016)) classifier that was successfully attacked using this trigger, there is divergence between the distributions of the internal layer activations induced by these two sets of instances. This is shown in Fig. Ib for neuron in the penultimate layer as an example. In comparison, for a clean classifier (not backdoor- attacked), the divergence between the Based on these visualizations, we ask the following question: Suppose the distribution alteration reversed for each neuron, e.g. by applying transformation to the internal activations of the triggered instances, so that the transformed distribution now closely agrees with the distribution for clean (without the backdoor-trigger) instances (see Fig. lc). Then, following this compensation, will the classifier accurately predict the true class of origin for these backdoor- -ackdoor-trigger instances?
CIFAR- 10 Krizhevsky & Hinton
distributions is almost negligible as shown in Fig. la.
Here, we investigate this problem in a simplified binary classification setting similar to the one considered by [lyas et al. (019). For a clean training random vector (X. Y) with a uniform class prior, i.e. Y ~ 27 11, } and with X NYY. μ. E), where μa E Rdd and E = g²I, I, consider a backdoor attack with target class "+1', triggered instance X, ~ NNu,,, Στ) with μ - + c, and Σ, = I. Here, class '-1' is automatically the source class of X,, since there are only two classes. With backdoor poisoning, multi-layer perceptron (MLP) classifier is trained with one hidden layer of J nodes, a batch normalization (BN) layer (loffe& & Szegedy (2015)) followed by linear activation, and two output nodes with functions f- Rd + R and f+ R R corresponding to classes '-1' and '+1' respectively. An instance x will be classified to class --1' if f (x) > f+ (x); else it will be classified to '+1'.
Definition 3.1. (7-erroneous classifier) A classifier is said be ny-rroneous if the error rate for each class is upper bounded by n-
Definition 3.2. (v--sucessfull attack) A backdoor attack is said to be sl-successful if its attack success rate (ASR), Le. the probability for triggered instances being (mis Jclassiid to the attacker's target class Li et al. (2022b)), ), is at least vú; in our case, this means that (X.) > f [Xx)] 2 tb
Given the settings above, for an arbitrary input x, the activation of the j-th node (j E (1. .)) (after BN with trained parameters Ti and (y), with weight vector wy in the hidden layer, is:
a, (x) w mj 7 + BJ,
(1)
where my and u, respectively are the mean and variance stored by the BN layer during training on
the poisoned training set. Then the activation distribution for clean source class instances (X|Y
-1) ~ (M,,,, Σ) is Gaussian specified by mean Ela,(X))
(1,
-1] and variance Varfa, (X)lY=
-1]; while for triggered instances X,, N (M,,, Ex), the activation follows a Gaussian specified by
mean ((a,(X.)) and variance Varlu,(X.)), An easy way to eliminate the divergence between these
two distributions is to create classifier for triggered instances x. X. replacing aj in Eq, I with
(z)) (w. x m //// Vuz + 33 for each node j. where (see Apdx. A.1 for derivation):
σi my + - 1)w3 la + uo and m
Ob
05-
(2)
'These can be constructed in practice, given estimated backdoor trigger, by embedding the trigger in clean instances available to the defender.
3

---

Under review as a conference paper at ICLR 2023
input
outpu: g(x ((r))
classifier j
detected target classes y
classifiers ( with distribution alteration reversed
(b0) (
delector
classifier f
classifier g( (()))
(re-jestimated trigger A, f
clean data
100
y(s))))0))
output f(x)
post-training mitigation (off-line procedure)
test-time inference
Figure 2: Illustration of our backdoor mitigation framework with a test-time inference rule.
such that E/a; (Xx)] = Ela, (X)lY -1] and Varla, (X+)] = Varla, (X) = -1] are achieved. But here, we aim study the quantitative relationship between the distribution divergence and the SIA metric of Def. 333] below. Thus, we consider an "intermediate state" with a classifier specified by output node functions g- (-la) Rd - R and g+(+)a) TRd - I. where for each output node i E {-,+), gi (a/la)
u, a(xaa) depends on a "transition variable" E [0, 1]. (z a), az(=))) is the activation vector for input æ where (z)(a) = (w) x ( + Sn. with mn((a) om,j + a and ēy(0)= (a + a) v vuj)² being the "intermediate" mean and variance respectively, Given these settings, our main theoretical results are presented below.
with u; the weight vector for the original output function fi. â(x)a)
Definition 3.3. (Souree inference accuracy (SIA)) SIA is the probability that triggered instance is classified to its original source class (Li et al. (2022a)), i.e. Plg-(X,)a) > 9+ (X, a:] here.
Theorem 3.1. (Monotonicity of SLA with Divergence) If the binary classifier with f- and f+ is q-erroneous with < 1/2, the attack is vusuccessful with ab > 1/2, and σb I a, then SIA of the modified classifier, i.e. Plg- (Xg a.)= 9+ (X)]a)), monotonically decreases
[0, increases.
The proof of the theorem is given in Apdx. A.2 Note that the assumptions for Thm. 3.1 are very mild and reasonable. For example, < 1/2 is a minimum requirement for the classifer and d > 1/2 is a minimum requirement for successful backdoor attack. Moreover, di S generally holds empirically since trigger embedding (e.g.. consider patch attack) typically reduces the vari- ance of source class instances (while additive attacks do not change the variance). Also note that merely gives way of quantifying distribution divergence for purpose of analysis. According to these results, the core part of our proposed backdoor mitigation approach should be to find a mod- ified classifier g(- ()) by minimizing (e.g.. using aub-gradient methods) measure of distribution divergence over a well-chosen set of parameters, O This approach is next explicated.
REVERSING DISTRIBUTION ALTERATION FOR BACKDOOR MITIGATION
4.1 PROBLEM DESCRIPTION
Threat model. For input space X' and label space C, classifier that has been successfully backdoor- attacked will predict to the attacker's target class l" E C when test instance x E X' is embedded with the backdoor trigger using an incorporation function A X' - X. In addition to this "all-to- one" setting, we also consider the "all-to-all" setting where a test instance from any class E C will be (mis)classified to class (c + 1)mod[(C when it is embedded with the trigger Gu et al. 2019
Defender's goal. Given trained classifier f X + C. that may possibly be attacked, the defender aims to mitigate possible attacks by producing mapping f X C which (a) has high accuracy in classifying clean instances, and (b) when there a backdoor attack, classifies triggered instances to their original source class, though there is trigger embedded, i.e., achieves a high SIA.
Defender's assumptions. We consider post-training scenario where the defender has no access the training set of the classifier. The defender does possess an independent clean dataset, but this dataset is too small to train an accurate classifier from scratch, and even too small to effectively fine- tune the full set of classifier parameters (Liu et al. (20T8a): Zeng et al. (202)): ; Wang et al. (2019). The defender has white box access to the classifier, but does not know whether it has been attacked and, if so, does not know the trigger pattern that was used, i.e., the defense is unsupervised.
4.2 METHOD
Based on Thm. 3.1 it would seem that good mitigation approach involves modifying the classifier f, i.e. creating new classifier g(- +) X C. from f by applying transformation function
4

---

Under review as a conference paper at ICLR 2023
hen(110,1) + to the activation of each neuron {1.... ,.} in each layer 1 E {1.... L}. The parameters θ = {0,u} should be jointly chosen so as to minimize the aggregation (c.g. sum) of the divergences between the distributions qz. g5.(0=z <t U (0,)) obtained using م (0 and the target distributions P2A for 2x,(()) for Vj,1, where X follows the clean data distribution, i.e.:
minimize (=
Σ 3,1
D& (p( U0,,))
(3)
where: 2,,1 : X' R and 2/1 X R are activation functions for neuron j in layer / for classifiers
f and ( respectively; 021 = (0,rrl'' < I) represents all transformation parameters prior to
layer 1; Px(plll) == F (ppgg) for convex function k: : [0, oo) - IR satisfying k(1) 0 and belonging to the family of f-divergences for any distributions and (Ali & Silvey ( (1666).
The defender does not know a priori whether there is an attack. When there is no attack, no distribution correction should be needed. Moreover, when there is an attack, while the classifier g(-1@) with optimal transformation functions for neuron activation will achieve a high SIA on triggered instances, its accuracy on clean instances (especially those not from the backdoor target class) may be degraded)) Challenge 2: If there is an attack, the attack setting, i.c. all-to-one or all-to-all, and the ground- truth backdoor trigger A are both unknown to the defender. Challenge 3: The density form for 25,1( ((x)) may get altered by the trigger A and will likely be different from the density form for (X) - moreover both will likely be non-Gaussian. Thus, (3) cannot be easily minimized, e.g. simply by matching the mean and variance.
However, in practice, we will face the following challenges. Challenge 1:
For Challenges 1&2, we leverage existing poxt-training backdoor detection approaches to infer: whether the classifier is backdoor attacked and the associated target classes when is attacked (Wang et al.] (2019); Chen et al. (2019b); Liu et al. (2019)). These detectors, following the same assumptions in Sec. 4.1 reverse-engineer a trigger for cach putative target class the clean dataset possessed by the defender. Then anomaly detection is performed on statistics derived from these reverse-engineered triggers, e.g. the estimated size for patch triggers used by Wang al. (2019).
Here, the detector in our framework is different from most existing ones in order to cover a broad range of attack settings including all-to-one and all-to-all attacks. We first reverse-engineer trigger by solving an optimization problem defined on the clean set to get a detection statistic for each ordered putative class pair (s.t) C x C. For the Xiang et al. (2020) method, this statistic is (the reciprocal of) the estimated perturbation size inducing high (mis) clasiifissis from s to t. For Wang et al.] (2019), is the estimated patch size inducing high (misfclassiiications from s to Then we apply the anomaly detection approach in Waan et al. (2019). based on the MAD criterion (Hampell (1974)) to all the obtained statistics to find all the outlier statistics. We denote the set of detected class pairs associated with these outlier statistics as P, and denote 7 = (1 E C | Es e C s.t. (s, E PP} as the set of detected target classes. For each t E T, we (re-)estimate a trigger å (as surrogate for the true backdoor trigger. which is unknown) using clean instances from all detected source classe (t))= = {s Cl(s, t) E P). Then.for each detected target class t E t, we construct a classifier g( (O)) by solving the distribution divergence minimization problem using its (re- irtimmett Á,
For any test input x E X. if classifier f is deemed attack-free, i.e. P. = 0. the classification output under our mitigation framework will be f(x) = f(x). Otherwise, if f(x) E C F, we trust the class decision and set f(x)= = f(x) both because æ is unlikely to possess a trigger and because a successful attack should not degrade the classifier's accuracy on clean instances. However,
(x) = E 7, there are two main possibilities: 1) xe is clean instance truly from class I; 2) x is classified to class i due to the presence of the trigger. To distinguish these two cases, we feed x to the optimized g(. If O,) # f(x), x likely contains a trigger, and thus we should set f(x) = g( O)), which is likely the original source class of x based on our theoretical results. The outline of our mitigation framework is summarized in Fig.
Note that in the test-time inference procedure above. the major (additional) computation for both backdoor trigger instance detection and source class inference a forward propagation for feeding x to g(- O.)) which is comparable
FFor example, the histogram for clean activations in Fig. 1b will be shifted away (to the left).
Mooe reliable trigger estimation be achieved in this way for detected target class.
5

---

Under review as a conference paper at ICLR 2023
to the computation required for classification using f. Moreover, such additional computation occurs only if an attack is detected and fxx = t; thus, our test-time inference is very efficient.
Now the remaining problem is to address Challenge 3, which is critical to the estimation of θ using the everss-engineered trigger Â for each detected target class E T. For simplicity, we will drop the subscript below without loss of generality. Our main goals are: (a) specifying the structure of the transformation function h1g with its associated parameters f, (b) empirical estimation of the distribution divergence in Eq. using a clean dataset (i.e. the subset of clean instances from classes in S() for each detected class t), and (c) choosing the convex function & for the divergence form. For (a), we consider the following transformation function with parameters 83 = {1ya, JJd U22,413h
h2,(2) = max max 2 Mjd
w 1.a
(4)
where UyJ and J. specify the location and scale of the activation distribution, respectively, while W5,, control the shape of the tail of the distribution. For goal (b), we quantize the real line into M _IM = [bx--1, 0o), for M sufficiently large. Then the distribution divergence in Eq. (3) for each node j and layer 1 is computed on discrete distributions Py. and âz over these intervals. Specifically, the discrete distributions are estimated using a subset D, of instances from classes S(t), with the probabilities for interval Iz computed by:
intervals Zn = (-o0, b1); I2 = [b,,b2),
1
Σ (x))(()) E I.] and 9 Dr ZED:
1
11,,( E Z.]. (5) EED;
Σ
[Del
To ensure that the distribution divergence is differentiable with reference to the parameters, such that it can be minimized using (e.g.) gradient descent, we approximate the non-differentiable indicator function [-] in Eq. (5) using differentiable functions such as the sigmoid, i.c. we redefine:
[[ E I = sigmoid(T(z - b2-1)) sigmoid(T(2 bb))
(6)
where 7 is scale factor controlling the error of approximation. For I₁ and Im. which have semi- infinite support, we use a single sigmoid in Eq. (6). The choice of the intervals and T is not critical to the performance, long the length of the finite intervals sufficiently small, will be shown in Tab. in Sec. [5] Finally, for goal (c), we consider several different divergence forms including the total variation (TV) divergence with k(r) = ir - 11/2, the -hasn--Shnnoon (JS) divergence with 2r k(r)- = r rlog- 74I + log 74T, and the Kullback-Leibb Leibler (KI.) divergence with k(r) rlogr. The choice of the divergence form is also not critical to the mitigation performance (see Apdx. E
2
5 EXPERIMENTS
5.1 EXPERIMENT SETUP
Datasets: Our main experiments are conducted on the benchmark CIFAR-10 dataset, which contains 60,000 32 x 32 color images from 10 classes, with 5,000 images per class for training and 1,000 images per class for testing (Krizhevsky & Hinton (2009).. We also show the effectiveness of our proposed mitigation framework other benchmark datasets including GTSRB (Houben et al. (2013). CIFAR-100 (Krizhevsky & Hinton (2009)), ImageNette (Howard (0200). and TmngemtgeNet. Details of these datasets be found in Apdx. [B.]] Data allocation in experiments strictly follows the assumptions in Sec. 4.1 For each dataset, we randomly sample 10% of the test set to form the small, clean dataset DDefense assumed for the defender. The remaining test instances, denoted by Drest are reserved for performance evaluation.
Attack settings: In this paper. we consider standard backdoor attacks launched by poisoning the training set of the classifier (Gu et al. (2019): [Chen et al. (2017). In particular, we consider both the all-to-one (A20) attacks and the all-to-all (A2A) attacks in our main experiments on CIFAR-10. For A20 attacks on CIFAR-10, we arbitrarily choose class 9 as the target class; while for A2A attacks, as described in Sec. [4.]] triggered instances from any class e E C are supposed to be (mis) to class (c + 1 D)modd()) For each attack setting, we consider the following triggers: 1) a 3 x random patch (BadNet) with a randomly selected location (fixed for all triggered images for each attack) used in Gu et al. (2019); 2) additive perturbation (with size 2/255) resembling a chessboard (CB) used in Xiang et al. (2020); 3) a single pixel (SP) perturbed by 75/255 with randomly selected location (fxxed for all triggered images for each attack) used by Tran et al.

---

Under review as a conference paper at ICLR 2023
Trigger type
BadNet
CB
In inv
15 Inv
SP
WaNer
A20
A2A
A20
A2A
A20
A2A
A20
A20
A2A
A20
A2A
ACC 0.9122
0.9121
0.9135
0.9098
0:9135
0.9131
0,9130
0.9126
0.9138
0,9060
0.9032
0.8994
Vanilla
ASR
0.9573
0.8658
0,9685
0,8692
0.9989
0.8973
0,989
0,8620
0.8912
0,8550
0.9153
0.8216
SIA
0.0397
0.0432
0.0293
0.0257
0.0010
0.0151
0,1107
0,0194
0,1016
0.0593
0,0772
0,0714
ACC
0.8797
0.8762
0.8735
038766
0.8835
0.8767
0.8500
0.690
0.8854
0.8614
0.8748
0.8756
NC
ASR
0.0130
0.0154
0,0064
0.0155
0.0120
0.0150
0.0080
0,0179
0.0355
0.0188
0.0144
0.1381
SIA
0.8532
0.8144
08312
0,8597
0,8654
0,8650
0,7932
0,8234
0.8362
0,8477
0,2211
0.7183
ACC
0.8500
0,8758
0.8812
0,8719
0.8452
0.8800
0,8225
0,8726
0.8666
0.8745
0,8777
0,8700
I-BAU
ASR
0.0094
0.0164
0.1973
0,0811
0.0091
0.0133
0,2600
0.3353
0,0172
0.0154
0,1339
0.1253
SIA 0.8301
0,8583
0.3999
0.7756
0.8277
0.8673
0.5549
0,4922
0.8479
0.8609
0.7059
0.7269
ACC 0.8644
0,8492
04240
038577
0.8455
0.8648
0.8145
0.8421
0,8195
0,8411
0,8298
0.8607
ANP
ASR
0.0474
0.1199
0.3351
0.0927
0.0836
0.1326
0,4703
0.2648
0,1229
0.0495
0,0263
0.0835
SIA
0.8184
0.7205
0,587
0,7168
0.7697
0.7324
0,3351
0,4976
0.7060
0.7942
0,7368
0.7451
ACC
0.8814
0.8819
0.8800
0.8908
0.8958
0,9047
0.991
0,8781
0,8813
0.8761
0,8592
0.8963
NAD
ASR
0.0193
0.7132
0.0871
0.0681
0.0356
0.0457
0,0254
0,0191
0.0667
0.0647
0.0571
0.1056
SIA
0.8498
0.1520
0.7711
0,8084
0.8504
0.8534
0,8221
0,8337
0,8123
0,8082
0,7710
0,7773
ACC
0.8689
0.8482
0.8800
0.8774
0,8880
0,8885
0.8669
0.8583
0.8899
0.8728
0.8739
0.8755
ARGD
ASR
0.0368
0.0839
0.0099
0.0117
0.0079
0.0122
0,0125
0,0179
0.0955
0,0452
0,0111
0.0362
SIA
0.8217
0.7544
0.8657
0.8690
0.8725
0.8786
0.8168
0.8297
0.7934
0.8295
0,8283
0.8241
BNA
0.9032
0.8951
0,9072
0.9068
0.8944
0,9055
0.9058
0.8921
0.8945
0.8792
ASR
0.0139 0.0189
0.0127
0.0202
0.0033
0.0111
0.0042
0.0168
0.0104
0.0225
0.0M44
0.0191
(ours)
0.8835
0.8841
0.8787
0.8820
0.8924
0.8942
0.8383
0.8522
0.8863
0.8811
0.8530
0.8607
Table 1: Average ACC, ASR, and SIA for our BNA, compared with NC, NAD, I-BAU, ANP, and ARGD, against all the created attacks applied to ResNet-18 trained on the CIFAR- 10 dataset.
20188; 4) invisible triggers generated with lo and l2 norm constraints (4 inv and Lا inv respectively) proposed by Li et al. (2021a); 5) a warping-basee
trigger (WaNet) proposed by Nguyen & Tran 2022 Details for generating these triggers are deferred to Apdx. B33 For experiments on CIFAR-10, we randomly create 5 attacks for each attack setting and each trigger (e.g. with random location). For experiments on the other four datasets, we only consider A20 attacks for subset of triggers where sufficiently high success rate can be achieved. For each dataset, we create one attack for each trigger being considered. A2A attacks are not considered for these datasets since they are not successful due to the insufficiency of data. More details about the attacks, including the number of back trigger images used for poisoning and the target class selected to create A20 attacks for the four datasets other than CIFAR- 10, are shown in Apdx. B.3
Performance evaluation metries: 1) The attack success rate (ASR) is the fraction of clean instances in Drest (mis)classified to the designated target class when the backdoor trigger is embedded. 2) The clean test accuracy (ACC) the DNN's accuracy on Drest without trigger embedding. 3) The SIA (.3) is the fraction of clean instances in Drea classified to the original source class when the trigger is embedded. For successful backdoor attack, ASR and ACC should be large, while SIA should be small. For successful mitigation approach, the resulting ASR should be small, while ACC and SIA should large.
Training settings: We train one classifier for each attack to evaluate our mitigation approach against existing ones. Training configurations, including the DNN architecture, batch size, number of epochs, etc., are detailed in Tab. 6 in Apdx. B.2] Data augmentation choices, including random cropping and horizontal flipping. are applied to each training instance. As shown in Tab. μ the defenseless "vanilla" classifiers being attacked achieve high ACC but suffer high ASR and low SIA (averaged over the five attacks we created) for all trigger types and for both A20 and A2A settings, i.e. the attacks are all successful and hence adequate for performance evaluation.
5.2 BACKDOOR MITIGATION RESULTS
We compare our mitigation approach (named BNA in the sequel) with five well-known and/or state- of-the-art methods, including NC (aagg et al. 2019)) NAD al. (202lc). I-BAU et al. 2222)), ), ANP(Wu & Wang (2021), and ARGD(Xia et al. (022)). ). For all these other methods, we used their officially posted code for implementation. For our BNA, following Sec. 4.2 we first perform detection by reverse-enginerrin a backdoor trigger for each class pair using objective functions from, e.g., Wang et al. (2019); Xiang et al. (2020) and then feeding the statistics obtained from the estimated trigger to an anomaly detector. Our anomaly detector is based on MAD, which is a classical approach also used by Wang et al. (2019; Chen et al. (201bb); Wang et al. (020). Here, set the detection threshold at "7-MAD" which easily catches all the backdoor class pairs More details, including pattern estimation and detection statistics are shown in Apdx. due to space limitation. Then, for each detected target class, we solve the divergence minimization problem
7

---

Under review as a conference paper at ICLR 2023
the numher of poisoned instances class
perturhation size (*255)
50
100
150
200
250
2
4
5
ACC
0.9112
0.9094
0,9098
0.9102
0.9015
0.5094
0.5041
0.9079
0.8992
0.8912
ASR
0.0195
0.0141
0.0121
0,0170
0.0090
0.0141
0,0395
0.0222
0.0109
0.0388
SIA
0.8851
0.8837
0.8728
0.8840
0.8662
0.8851
0.8783
0.8711
0.8814
0.8435
Table 2: ACC, ASR, and SIA for our BNA as a function of (1) the number of poisoned instances injected into the training set; (2) the perturbation size under all-to-one CB attack.
to optimize the transformation functions using learning rate 0.01 for 30 epochs. If a neuron is followed by a BN (which is very common), instead of applying an additional transformation function h22, we treat the mean and standard deviation of BN as the parameters μn and In associated with hyg respectively. Here, we only show results for BNA with the total variation divergence. Results for KL-divergence and JS JS-divergence are deferred to Apdx. E To compute the divergence, we use the "interval trick" (Eq. (5) to obtain the discrete empirical distribution. For simplicity, we let all finite intervals, I = b
1, ... M, have the same length A = 0.1. For each neuron, we set Umin and bmax the minimum and maximum activations, respectively, when feeding in clean instances from Decense to the poisoned classifier f. Then, the number of intervals is M = [ TTbaa-bmiss bmim] I; and all intervals can be specified by by = buii and b; = b + Ab. Finally, 150, which is obtained by line search minimize the total
Ab. the scale factor in Eq. (6) is set variation between the "soft" distribution and the empirical one. In fact, the choices for Ab and (over reasonable ranges) has little impact on our mitigation performance, shown in Tab.
In Tab. [ we show the ASR, ACC, and SIA for our BNA compared with the other five methods (which are all tuning-based) for attacks on CIFAR-10. Each metric averaged over the five attacks created for each trigger type and attack setting, with the highest ACC and SIA, and the lowest ASR in bold. Although the five tuning-based methods we compare here can effectively deactivate the backdoor attacks (iee, significantly reduce ASRs), there is a clear drop (3%-15%) in both ACC and SIA. This is possibly due to tuning many DNN parameters using very limited data. Moreover, we found these tuning-based methods are sensitive to the choices of tlper-parameters such as the learning rate. For ANP with neuron pruming, the performance is acceptable only for A2O with the BadNet trigger. One possible reason is that invisible, perturbation neurons only moderately (Wang et al. (2022a)); thus, pruning a small number of neurons cannot mitigate the attack. In contrast, our method successfully mitigates all these backdoor attacks (with generally the best ACC and ASR compared with the others) regardless of the trigger type and attack setting. Notably, since the purpose of BNA's divergence minimization is to maximize the SIA, unsurprisingly achieves the best SIA with a clear margin over all other methods, in all cases (the corresponding distribution divergnces are shown in Tab. 9. We also tune the poisoning ratio and perturbation size used in A20 CB attacks. and the performance for our BNA slightly declines as the attack is strengthened, as shown in Tab. 2 However, it still outperforms the other methods (see Tab. 11 in Apdx.
triggers affect most
Why tuning-based methods like NC cannot achieve SIA as high as our BNA without parameter altering? Note that NC tunes the classifier using instances embedded with the estimated trigger but without label flipping. This is equivalent to minimizing the divergence between internal activation distributions for clean and triggered instances, but with the parameters changed. Even for an optimal (zero) divergence, the best achievable SIA of NC is still upper-bounded bounded by the ACC of the classifier after tuming, which usually drops due to the data insufficiency. By contrast, the reference distribution for our divergence minimization is obtained by feeding clean instances to the poisoned classifier without changing its parameters; thus, it is "better" reference with higher upper-bound ACC.
Results of our BNA on other datasets are shown in Table 3 The ACC for DNNs trained without attack for GTSRB, CIO, ImageNette, and TinyImageNet are 0.9567, 0.6926, 0.8726, and 0.5224, respectively; while ACC, ASR, and SIA for attacked DNNs are shown in the row "Vanilla". We apply our BNA on the poisoned DNNs, with the same settings as for CIFAR- 10, which signifi- cantly reduces ASR (to less than 1.3% in all cases), with uniformly high SIA and ACC.
5.3 TEST-TIME BACKDOOR TRIGGER INSTANCES DETECTION
Different from other luning-based backdoor mitigation approaches, our BNA can also detect backdoor-trigee instances at test-time, as described in Sec. 4.2 and shown in Fig. 2 Here, we evaluate the accuracy of our test-time detection compared with state-of-the-art detector named
8

---

Under review as a conference paper at ICLR 2023
Trigger
GTSRB
CIFAR-100
TinyImageNer
ImageNetic
type
BadNes
CB
10V
11V
WaNer
BadNer
CR
Iny
1. inv
BadNet
BadNer
ACC
0.9517
0,9556
0,9531
0,9521
0,9408
0.679
0.6917
0.6863
0.6804
0.5192
0.8626
Vanilla
ASR
1.0000
1,0000
1,0000
0.9794
0,9000
0.9037
0.9199
0.9935
0.9097
0.8058
0.9144
SIA
0.0000
0,0000
0,0000
0,0169
0,0908
0.0781
0.0646
0.0063
0.0707
0.1134
0.0771
ACC
0.9491
0,5488
0.9505
0.9500
0.404
0.6770
0.6863
0.6858
0.6787
0.5178
0.7941
BNA
ASR
0.0000
0,0000
0,0122
0.0001
0,0041
0.0002
0.0524
0.0062
0.0016
0.0043
0.0016
SIA
0.9312
0.9454
0.9330
0.8945
0.3388
0.6526
0.5880
0.6169
0.5224
0.4965
0.7940
Table 3: ACC, ASR, and SIA for our BNA against all-to-one attacks on CIFAR-100, GTSRB, ImageNette, and TinylmageNet datasets.
(Afm0..I
10
100
200
300
400
500
600
700
ACC
0.9024
0.9025
0.9022
0.9019
0,9024
0,9019
0,9022
0.9018
0.9017
0,9015
0.9021
ASR
0,0257
0.022
0.0206
0.0207
0,0214
0,0202
0.0212
0.0209
0.0207
0.0201
0.0204
SIA
0,8744
0,8758
0.8774
0.8768
0.8768
0,8775
0,8773
0.8768
0.5772
0,8778
0,8777
Ab (7=130)
0.1
0.11
0.12
0.13
0.14
0.13
0.16
0.17
0.18
0.19
0.2
ACC
0.9028
0,9018
0.9019
0.9023
0,9019
0,9023
0,9019
0,9019
0.9022
0,9022
0,9022
ASR
0.0197
0,0202
0.0199
0.0204
0.0199
0,0204
0.0198
0.0206
0,0206
0,0207
0,0212
SIA
0.8799
0.8775
0.8779
0.8773
0.8772
0,8779
0,8773
0.8767
0.8779
0.8769
0.8764
Table 4: ACC, ASR, and SIA for our BNA as a function of scale factor and bin size on ResNet- 18 trained on CIFAR-10 poisoned by all-to-one BadNet attack.
Trigger
BadNet
CB
La inv
SP
WaNet
type
A20
A2A
A20
A2A
A20
A2A
A20
A2A
A20
A2A
A20
A2A
BNA
FPR
0,1390
0.0606
0.1144
0.1092
0,1413
0,0600
0.1976
0.1027
0.1323
0.0656
0.1406
0,0865
TPR
0.9872
0.9508
0.9872
0.9682
0.9967
0.9873
0.9958
0.9793
0.9894
0.9294
0.9959
0,9248
FPR 0.15
0.15
0.15
0.15
0.15
0.15
0.15
0.15
0.15
0.15
0.15
0.15
STRIP
TPR
0.9638
0.5147
0.6802
0,2089
0.9995
0.1053
0.9924
0.5272
0.8522
0.3123
0.0202
0.0411
Table TPR and FPR for our BNA, compared with STRIP, against all attacks created on CIFAR-10.
STRIP Gao et al. (019)). For any input image during inference, STRIP blends with clean im- ages possessed by the defender. The blended image is fed into the poisoned DNN, with an entropy calculated on the output posteriors. If the entropy lower than prescribed detection threshold, the input is deemed to be embedded with the trigger. Here, we set the detection threshold at 15% FPR for STRIP which achieves a generally good trade-off between TPR and FPR. In contrast, our BNA does not need to set a detection threshold. In Tab. [ we show the True Positive Rate (TPR, ie., the fraction of backdoor-trigger images correctly detected) and the False Positive Rate (FPR, i..., the fraction of clean test images from the backdoor target class(es) that are falsely detected) for both methods. Although STRIP performs well on A20 attacks for some trigger types, e.g., BadNet, attacks using human-imperceptible triggers, espe-
lo inv, and l2 inv, its TPR drops drastically cially the WaNet attacks. Moreover, it does not perform well on all A2A attacks, with largest TPR of only 0.5272. By contrast, our BNA is effective for all these attacks detects almost all the backdoor- trigger images, with FPRs comparable to STRIP.
5.4 MITIGATION PERFORMANCE AGAINST ADAPTIVE ATTACKS
A recent backdoor attack proposed by Doan et al. (202 minimizes metric similar to our activa- tion distribution alteration, in order to achieve better stealthiness. This attack can be viewed as an adaptive attack against our mitigation defense since the trained classifier will be more sensitive to even a smaller distribution divergence than for ordinary backdoor attacks. Nevertheless, our method successfully mitigates this attack. In our experiment on CIFAR-I0, the average distribution total variation divergence over all neurons is reduced from 8067 to 2789. Accordingly, the ACC/ASR before and after mitigation are 0.9162/0.99 and 0.8906/0.0072 respectively, with SIA 0.8496.
CONCLUSION
In this paper, we revealed an activation distribution alteration property for backdoor attacks. We found that by correcting such alteration, backdoor trigger instances will be classified to their original proposed backdoor mitigation approach without changing any
source classes. Accordingly, parameters of the classifier, which outperformed methods that use DNN fine-tuning. Moreover, our method can detect instances with the trigger during inference.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
The main purpose of this research is to understand the behavior of deep learning systems facing malicious activities, and enhance their safety level. The backdoor attack considered in this paper is well-known, with open-sourced implementation code. Thus, publication of this paper will be beneficial to the community defending against backdoor attacks. The code of our defense will be released if the paper is accepted.
REFERENCES
M. Ali and SAMUEL Silvey. A general class of coefficients of divergence of one distribution from another. Journal of the royal statistical society series b-methodological. 28: 28:131-142, 1966.
Jiawang Bai, Kuofeng Gao, Dihong Gong, Shu-Tao Xia, Zhifeng Li, and Wei Liu. Hardly percepti- ble trojan attack against neural networks with bit flips. In ECCV, 2022.
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian M. Molloy, and Biplav Srivastava. Detecting backdoor attacks deep neural networks by activation clustering. In AAAI, 2019a.
H. Chen, C. Fu, Zhao, and F. Koushanfar. Deepinspect: A black-box trojan detection and mitiga- tion framework for deep neural networks. In IJCAI, pp.
7 2019b.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv: 2017.
Edward Chou, Florian Tramèr, and Giancarlo Pellegrino. Sentinet: Detecting localized universal attacks against deep learning systems. In 2020 IEEE Security and Privacy Workshops, 2020.
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A backdoor attack against lstm-based text classifica- tion systems. IEEE Access, 2019.
Bao Gia Doan, Ehsan Abbasnejad, and Damith C. Ranasinghe. Februus: Input purification defense against trojan attacks on deep neural network systems. In Annual Computer Security Applications Conference, pp. 897-912, 2020.
Khoa Doan, Yingjie Lao, and Ping Li. In NeurlPS, volume 34, 2021.
Y. Dong, X. Yang, Z. Deng. T. Pang, Z. Xiao, H. Su, and J. Zhu. Icev. 2021.
M. Du, R. Jia, and D. Song. Robust anomaly detection and backdoor attack detection via differential privacy, In ICLR, 2020.
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith Chinthana Ranasinghe, and Surya Nepal. STRIP: defence against trojan attacks deep neural networks. In AC 2019.
Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 2019.
Jiyang Guan, Zhuozhuo Tu, Ran He, and Dacheng Tao. Few-shot backdoor defense using shapley estimation. 2022.
F. R. Hampel. The influence curve and its role in robust estimation. Journal of the American Statistical Association, 69, 1974.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detec- tion of traffic signs in real-world images: The German Traffic Sign Detection Benchmark. In International Joint Conference Neural Networks, 2013.
10

---

Under review as a conference paper at ICLR 2023
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXin:1744.04661 20177
Jeremy Howard. Imagenette, 2020.
Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In ICLR, 2022.
Andrew lyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In NeurlPS, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei (eds.), ICML, 2015.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
S. Kolouri, A. Saha, H. Pirsiavash, and H. Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In CVPR, pp. 298-307, 2020.
Alex Krizhevsky and Hinton. Learning multiple layers of features from tiny images. http:// www. cs. toronto. edu/-kriiii waarriizlleaanngggg features -eatueeeeeannnnneeeuueeeeeeee 2009 TR. pdf 2009.
S. Li, M. Xue, B. Zhao, H. Zhu, and X. Zhang. Invisible backdoor attacks on deep neural networks via steganography and regularization. IEEE Transactions on Dependable and Secure Computing, 18(05))2088-2105, 2021a.
Shaofeng Li, Benjamin Zi Hao Zhao, Jiahao Yu, Minhui Xue, Dali Kaafar, and Haojin Zhu. Invisible backdoor attacks against deep neural networks. arXiv:190..02449 2019
Xi Li, George Kesidis, David J. Miller, and Vladimir Lucic. Backdoor attack and defense for deep regression. arXiv:2109.0233: 2021b.
Xi Li, Zhen Xiang, David J. Miller, and George Kesidis. Test-time detection of backdoor triggers for poisoned deep neural networks. In ICASSP, 2022a.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Frasing backdoor triggers from deep neural networks. In ICTR, 2021c
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learn- ing: Training clean models on poisoned data. In NeurlPS, 2021d.
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning A survey. IEEE Transac- fions on Neural Networks and Learning Systems, pp. 1-18, 2022b.
Kang Liu, Brendan Davin-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdoor- ing attacks on deep neural networks. In RAID, 2018a.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In NDSS, 2018b.
Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS '19, pp. 1265-1282, 2019.
Yingqi Liu, Guangyu Shen, Guanhong Tao, Zhenting Wang, Shiqing Ma, and Xiangyu Zhang. Com- plex backdoor detection by symmetric feature differencing. CVPR, 2022.
A. Madry, A. Makelov, L. Schmidt, D. Tsipras. and A. Vladu. Towards deep learning models resistant adversarial attacks. ICLR, 2018.
Tuan Anh Nguyen and Anh Tuan Tran. Wanet imperceptible warping- warping-based backdoor attack. In ICLR, 2021.
11

---

Under review as a conference paper at ICLR 2023
Xiangyu Qi. Tinghao Xie, Ruizhe Pan, Jifeng Zhu, Yong Yang, and Kai Bu. Towards practical deployment-stage backdoor attack on deep neural networks. In CVPR, 2022.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 1 11(3:22112252, 2015.
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor at- tacks. AAAI, 2020.
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss mini- mization. In ICML. pp. 5739-5748, 2019.
Guanhong Tao, Guangyu Shen, Yingqi Liu, Shengwei An, Qiuling Xu, Shiqing Ma, Pan Li, and Xiangyu Zhang. Better trigger inversion optimization in backdoor scanning. In CVPR, 2022.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In NeurlPS, 2018.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy, 2019.
Hang Wang, Zhen Xiang, David J. Miller, and George Kesidis. Universal post-training backdoor 2022a.
detection. arXiv,
Lun Wang, Zaynah Javed, Xian Wu, Wenbo Guo, Xinyu Xing, and Dawn Song. Backdoorl: Back- door attack against competitive reinforcement learning. In IJCAI, 2021.
R. Wang, G. Zhang, S. Liu, P.-Y. Chen, J. Xiong, and M. Wang. Practical detection of trojan neural networks:
and data-free cases. In ECCV, 2020.
Zhenting Wang, Juan Zhai, and Shiqing Ma. Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning. In CVPR, 2022b.
Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In NewrlPS, 2021.
Jun Xia, Ting Wang, Jiepin Ding, Xian Wei, and Mingsong Chen. Eliminating backdoor triggers for deep neural networks using attention relation graph distillation. In IJCAI, 2022.
Z. Xiang, D.J. Miller, and G. Kesidis. A benchmark study of backdoor data poisoning defenses for deep neural network classifiers and novel defense. In MLSP, Pittsburgh, 2019.
Zhen Xiang, David J. Miller, and George Kesidis. Detection of backdoors in trained classifiers without access the training set. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Zhen Xiang, David J. Miller, Siheng Chen, Xi Li, and George Kesidis. A backdoor attack against 3d point cloud classifiers. ICCV, 2021.
Zhen Xiang, David J. Miller, Siheng Chen, Xi Li, and George Kesidis. Detecting backdoor attacks against point cloud classifiers. In ICASSP, 2022.
Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. Dba: Distributed backdoor attacks against feder- ated learning. In ICLR, 2020.
X. Xu, Q. Wang. H. Li, N. Borisov, C. A. Gunter, and B. Li. Detecting AI Trojans Using Meta Neural Analysis. In Proc. IEEE Symposiun Security and Privacy, 2021.
Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y. Zhao. Latent backdoor attacks deep neural networks. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communica- fions Security, CCS 19, 2019.
12

---

Under review as a conference paper at ICLR 2023
Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In ICLR, 2022.
Zhiyuan Zhang, Lingjuan Lyu, Weiqiang Wang, Lichao Sun, and Xu Sun. How to inject backdoors with better consistency: Logit anchoring on clean data. In ICLR, 2022.
Zhendong Zhao, Xiaojun Chen, Yuexin Xuan, Ye Dong, Dakui Wang, and Kaitai Liang. Defeat: Deep hidden feature backdoor attacks by imperceptible perturbation and latent representation constraints. In CVPR, 2022.
Runkai Zheng, Rongjun Tang, Jianze Li, and Liu. Data-free backdoor removal based on channel lipschitzness. 2022.
13

---

Under review as a conference paper at ICLR 2023
A PROOF OF THEOREMS IN THE MAIN PAPER
A.1 DERIVATION OF EQ.
Here, provide the derivation showing that m. and uz in Eq. (2) are the solutions
Elaj (Xx)] = Eaa (X)lY =-1
(7)
Var [aj [x)] = Varla, (X){ = -1]
(8)
Based on Eq.
the above equations can be expanded as the following:
X
m;
X
m
E
+ By] =
+ = =1]
(9)
J
w
X
w
X
m.
Var
+ By] = Varl
7/3 + +3118 = =-1]
(10)
/
Voj
We first solve Eq.
10 for Y = -1) ~ N2 (-p,,
μi 0 2 and X, ~ N of which leads to:
vj Uj
(11)
By substituting Eq.
111) into Eq.
9 and since μb - + c, we get the following:
m3
(m
w 11) + w p4b
Vij
σb
1)2} + w Ob
m
A.2 PROOF OF THEOREM 3.1
Proof. First, let's specify the following vector/matrix representations that will be used throughout this proof:
W [wi,'.. ,,]] T E R-xd
i (a)
V
R JXJ
v(a)=
()(
mi
T (a)
m
[
E R m(aa
E ER' r =
:
JxJ B= : ER'
mjj
m)(a)
[a)())
ā (+la)
[ai(-)
al-) =
E ER a(-la)= =
E ER.
a²(-)-
E ER'
as(+)
(s(+)a))
a5(1)
Let X_ (X|Y 11) NN 22I) denote random instances from the source class '-1' for simplicity. Let u = u - u with u. and u4 being the weight vectors associated with the node for class '-1' and the node for class '+1' respectively. Then, it is easy see that:
a(Xola))._ a(x)) and a(Xzla) = a" (Xx),
and taking one step further by setting a 1, we have the following:
(X, > 9+ (Xola)aa = 1] = P(u á(Xola) > 0/a 1]
(12)
PluTa(Xx (Xx)) > 0
= PXf-(Xu) > fX,))]
< 1
(13)
14

---

Under review as a conference paper at ICLR 2023
This is say that when 1, the classifier is not modified at all, thus the SIA will be larger than 1- v since the attack is sussucessfful (see Definition 3.2). On the other hand, by setting a = 0, will have the following:
Plg- (4 (Xxla)aa 0] P(u (Xx 0/a 0]
a (Xx)) > 0]
P(f-(x-) (x-))
(14)
2 -1
(15)
and this is to say that when O, the distribution shift will be fully recovered, such that SIA is equally high as the accuracy the source class. Recall that Eq. (14) due to Eq. (7) and Eq.
(15) is because the classifier specified by f- and f+ is assumed 77-erroneous (see Definition 3.1). Here, we prove the theorem by showing that the partial derivative of (X, >
(8). The inequality
9+ (X,/aa ax) over is strictly negative when σb I (i.e. triggered instances have smaller standard deviation than clean instances, which is generally true). To achieve this, we notice that
u T ((xl))) = u ((a)-
m((a)) + +u²
follows Gaussian distribution with
Elu a(Xx = u T We- 1are-milyy mm(a)) + tuTA
(16)
Varfu a(X; = 1 TV((( - 2
(17)
We also notice that for source class instances X_
u aTaaX_)== u T V -- - + - u
follows Gaussian distribution with Elu T a (X-]] = u T V T WH m)) + J
(18)
Varlu a(x_)] @ rV buli
(19)
Then we have
Elu à (X, [a)]
Plu" T a(Xola)) > 0] 4b
(20)
VVrlu u a (x
Elu aaxa
Plu
a P(uTa(X_) 0]
4
(21)
/Varlu' a(X)))
where & is the cumulative distribution function of standard Gaussian. Now let's consider Eq. 21 first. Since 7 < 1 as we have reasonably assumed (otherwise the classifier may be worse than random guess), and also according to Eq. [5). have
Plu a axxa 0] =
f-
Thus, based on Eq. (21) and Eq. [88 we get
V I
W μ m) T 1T>0
(22)
Next, let's focus on Eq. 20
Again, we set = 1. Based
12 13 and the reasonable
assumption that s > 2 (otherwise the attack not deemed successful since the success rate will be even lower than the accuracy on clean instances), have
1
P(u ((Xala) a 1] = (X.((X)) f4(x)) <
Then, based on Eq.
20 and Eq. 16 we get
v r WH We m) + + <<00
(23)
Subtract Eq.
23 from Eq.
we get:
V TWe>
(24)
15

---

Under review as a conference paper at ICLR 2023
Based on Eq.
20 also have
E(u a(X)(o)
aP X a) >
9+ (Xa
a)]
Var ja a[Xala]]
E u
((xx(a))
(25)
da
da
Var(u fur a(Xx)
where @ the probability density function (PDF) for standard normal distribution. Based on Eq.
16 Eq. [7p. and Eq. 2 we have
Elu [X)
Var u a(X, a)
V T W
WE (am (1 a)m")) (a + (1 a) tb )u g
Tbll W TV TV-iull full2
a : r
m + ( 1)W We] a (2 1)u T
+ constant
WWTVV-R 2ul12
and thus, based on Eq. 23 and Eq.
24
i 3/u (Xxa ai)
Varlu a(xx
da
a
1) V
Wi We) uT + 22 u V -We
TV
<
when σb S a. Substitute it into Eq. (55 and given Gaussian PDF being strictly positive, we have
OP g (Xa la) > g+ (Xola))
<
Oa
16

---

Under review as a conference paper at ICLR 2023
B DATASETS, TRAINING SETTINGS, AND ATTACK SETTINGS
B.I DATASETS
In experiments, we show the effectiveness of our proposed backdoor mitigation method on several benchmark datasets including CIFAR-10 10 (Krizhevsky & Hinton (2009)), GTSRB (Houben et al.] (.13)) CIFAR-I00 (Krizhevsky & Hinton (2009)) ImageNette (Howard (2020), and Tinylma- geNet. CIFAR-10 dataset contains 60,000 32 x 32 color images from 10 classes, with 5,000 images per class for training and 1,000 images per class for testing. GTSRB dataset has more than 50,000 traffic sign images with different sizes from 43 classes. Here, we resize all images in GTSRB to 32 x 32. CIFAR-100 contains 60,000 32 x 32 color images evenly from 100 classes, where 500 images per class are used for training. while the others used for testing. ImageNette is subset of 10 easily classified classes from Imagene(" with image size of 256 x 256. For each class, there are around 900 images for training and 400 images for testing. The TinyImageNet dataset is subset of the ImageNet dataset (Russakovsky et al. (2015)) It contains 100,000 64 x 64 color images evenly distributed in 200 classes (500 training images and 50 test images for each class).
B.2 TRAINING SETTINGS
Training settings for the 5 datasets are shown in Table We train a ResNet-18 (He al. (016)) on CIFAR-I0 10 and CIFAR-100 for 30 epochs and 40 epochs, respectively. We train a ResNet-34 (He al. (016)) on both TinylmageNet and ImageNette for 90 epochs. For GTSRB, we train MobileNet Howard et al. (0077) for 60 epochs. For all models. we use Adam optimizer (Kingma & Bal 2015)) for parameter learning and a scheduler to decay the learning rate of each parameter group by 0.1 every "scheduler step size" epochs (shown in the table). We choose batch size 32 for both CIFAR-10 and CIFAR-100, 64 for GTSRB and ImageNette, and 128 for Tiny IntgeeeeNet.
B.3 ATTACK SETTINGS
On dataset CIFAR-IO 10, we consider the following triggers: 1) a 3 x random patch (BadNet) with a
randomly selected location (fixed for all triggered images for each attack) used in Gu et al. (2019).
as visualized in Fig. 3b 2) an additive perturbation (with size 2/255) resembling a chessboard (CB) used in Xiang et al. 2020), as visualized in Fig. [c| 3) single pixel (SP) perturbed by 75/255 with a randomly selected location (fixed for all triggered images for each attack) used by TTran et al. (2018),
as visualized in Fig. 3d 4) invisible triggers generated with to and l2 norm constraints (lo inv and l2
inv respectively) proposed by Li et al. (2.21a2)))) as visualized in Fig. 3e and 3i
5) a warping-basee
trigger (WaNet) proposed by Nguyen & Tran 2021 visualized in Fig. 3g
Attack settings for CIFAR-10 10 are summarized in Tab. 7 For all-to-one attacks, we arbitrarily choose class 9 as the target class, and embed the backdoor triggers in 100 randomly chosen training samples per class (excluding the target class). To achieve similar effective attacks as other triggers, we poison 900 images per source class in the all-to-one attack using WaNet. For all-to-all attacks, we embed the backdoor triggers into 300 images for each class. For effective attacks, we poison 800 training images and 1500 training images in the all-to-all attacks using SP and WaNet, respectively.
Attack settings for other datasets are summerized in Tab. g Due to the insufficiency of data, we only conduct all-to-one attacks on these datasets for effective attacks. We arbitrarily choose class O as the target class for C10,R-100, GTSRB, and TinyImageNet, and class 9 for ImageNette. The classes other than the target class are all source classes. For CIFAR-IO0, we use the same BadNet, lo inv, and la inv triggers as CIFAR-10. 10. We increase the perturbation size to 6/255 for CB pattern for a effective backdoor attack. For each of the attack, we poison 10 images per source class using the above triggers. Trigger SP and WaNet are not considered since we can not launch successful backdoor attack using the trigger on CIFAR-100. For GTSRB, in addition to the same triggers as CIFAR-100 100, 2e also use the warping- based trigger (WaNet). We poison 2% of the training images per source class using BadNet trigger and l2 inv trigger, and 5% of the training images per source class with CB trigger and to inv trigger. To achieve similar effective attacks, we embed WaNet trigger into 24% of the training images per source class. For TinyImageNet and ImageNette, we
"The 10 classes are tench, English springer, cassette player, chain saw, church, French hom,, garhage truck, gas pump, golf ball, and parachute.
17

---

Under review as a conference paper at ICLR 2023
only consider BadNet the trigger, the DNN can not learn the backdoor mapping using the other (relatively simple and small) triggers in datasets that are much more complicated than CIFAR-10. To successfully plant backdoors, we increase the size the the BadNet patch to 6 x for TinylmageNet and to 21 x 21 for ImageNette. We embed the trigger 10 training images per source class in TinylmageNet and in 5% of the training images per source class for ImageNette.
Dataset
CIFAR-10
CIFAR-I00 100
TinyImageNet ResNet-34 Adam
ImageNette
GTSRB
DNN architecture
ResNet-18
ResNct-18
ResNet-34
MobileNet
Optimizer Batch size
Adam
Adam
Adam
Adam
32
32
128
64
64
Epuchs
Initial learning rate
lc-3
lc-3
1c-3
le-3
1c-3
scheduler step size
10
10
30
30
20
Table 6: Training configurations of the 5 datasets used our experiments.
BadNet
CB
la inv
Trigger type
A20
A2A
A20
A2A
A20
A2A
poisoned
100
300
100
300
100
300
per class
lo norm
3 x 3
3 X 3
NA
NA
1 6
1 x 6
12 norm
NA
NA
0.3074
0.3074
NA
NA
12
SP
WaNet
Trigger type
A20
A2A
A20
A2A
A20
A2A
# poisoned
100
300
100
800
900
1500
per class
lo norm
NA
NA
NA
NA
l2 norm
1.6106
1.6106
0.5094
0.5094
NA
NA
Table 7: Attack configurations CIFAR-10
CIFAR 100
TinylmageNet BadNet
ImageNette
Trigger type
BadNet
CB
la inv
lg inv
BadNet
Target class poisoned
0
0
0
9
10
10
10
10
10
5%
per class
to norm
3
3
NA
1 1 6
NA
6 x 6
21 x 21
12 norm
NA
0,9222
NA
1.6106
NA
NA
GTSRB
Trigger type
BadNet
CB
lo inv
le inv
WaNet
Target class
0
0
O
0
poisoned
2%
5%
5%
2%
24%
per class
to norm
3
3 NA
NA
x
NA
NA
1/ norm
0.9222
NA
1.6106
NA
Table Attack configurations on GTSRB, CIFAR-100, ImageNette, and Tiny Tine
18

---

Under review as a conference paper at ICLR 2023
clean
BadNet
(c) CB
(d) SP
(f) l2
(g) WaNet
Figure 3: Example of CIFAR-10 images embedded with the backdoor triggers considered in our experiments.
19

---

Under review as a conference paper at ICLR 2023
C PATTERN ESTIMATION AND BACKDOOR DETECTION
For our BNA, following Sec. we first perform detection by reverse-engineering a backdoor trigger for each class pair. For patch triggers like BadNet, we use the objective function from Wang et al. (2019) for trigger reverss-engineering. For other more subtle, perturbation-based trigger types, we use the objective function from Xiang et al.] (2020) for reverse-engineering The detection statistic is the reciprocal of the to norm of estimated patch triggers and 22 norm of reverse-engee eeverse-nggineered perturbation-batt triggers. Then we feed the statistics obtained from the estimated trigger to an anomaly detector.
Our anomaly detector based on MAD, which is a classical approach also used by Wang et al. 2019); [Chnn et al.] (2019b); Wang et al.] (2020). It first calculates absolute deviation between all detection statistics (the reciprocal of to norm of patch triggers and l2 norm of perturbation- triggers) and the median, and the median of the absolute deviations is called Median Absolute Devi- ation (MAD). For a class pair and its corresponding estimated trigger, if the trigger's 's anomaly score, which is defined detected as a backdoor class pair. The detection threshold can be easily found, as shown in the Fig, 4 and Fig. 5 Fig. 4 and Fig. [S] show the histograms of the anomaly scores for all class pairs under all-to-one and all-to-all attacks, respectively. Here, we set the detection threshold at 7, which easily catches all the backdoor class pairs under all the attacks, except for the all-to-all BadNet attack and both attacks using WaNet trigger.
the absolute deviation divided by MAD, is larger than given threshold, it is
For the all-to-all BadNet attack, the outlier detector finds source classes and 8 for the target class 1, where 0-1 is the true source-target class pair and 8-1 is falsely detected, as shown in Fig. [a The lo norm of the trigger estimated on class 0 clean images 3.02, and that estimated on class 8 images is 7.95. If class 0 and 8 are both the source classes involved in the backdoor attack, then the trigger estimated on the clean images from class and should also have a small to norm. Otherwise, the trigger estimated using class 8 images is intrinsic backdoor pattern (Xiang et al.) (2022); ; Liu et al. (2022); Tao et al. 2022)), and 0-1 is the true source class pair, since the trigger 0-1 has smaller size than 8-1. By optimizing
clean images from class 0 and 8, the lo norm of the trigger that causes mis classsifiiiatn to class with high confidence is 27.18 - much larger than the triggers estimated either class O images class 8 images. Thus, detect 0-1 as the true backdoor class pair and discard the trigger for class pair 8-1 backdoor mitigation.
For the attacks using warping-basee triggers (WaNet), unlike the other attacks, trigger size for clean class pairs and backdoor class pairs are both small. However, there is still a "gap" between the anomaly scores of clean class pairs and backdoor class pairs, shown in Fig. 4f and 5 The outlier detector successfully detects all the backdoor class pairs by using a threshold at 3.
20

---

Under review as a conference paper at ICLR 2023
clann class
ciaa caza
bededezz clea pairs
backdoor clice
20
10
30
(a) BadNet
(b) CB
claan para
claze poita
teckdesr class patns
60-
backdoor clee pars
30
20
10
(c) SP
(d) la inv
coan class pars Lenkeduz
12
Ciaan class
backdtan cluc
10-
20-
20
(c) lg inv
(f) WaNet
Figure 4: Histograms of anomaly scores for each class pair under all all-to-one attacks.
21

---

Under review as a conference paper at ICLR 2023
clann class
70-
ciaan caza
bededezz clea pairts
backdoor clice
10
25
(a) BadNet
(b) CB
para
clean claze poita
tecedesr class patns
backdoor clee pars
30-
40
20-
20
(c) SP
(d) la inv
coan class pars Lenkeduz
Ciaan class
backdaan cluc
60
30
10
50
(c) lg inv
(f) WaNet
Figure Histograms of anomaly scores for each class pair under all all-to-all attacks.
22

---

Under review as a conference paper at ICLR 2023
D DISTRIBUTION DIVERGENCES
As stated Thm. 3.1 for our backdoor mitigation method, the SIA monotonically increases as the divergence between clean instances and backdoor-trigger instances decreases. We show the ACC, ASR, and SIA for our method against all all-to-one attacks on CIFAR-I0 dataset in Tab. Here, we show the corresponding distribution divergences under attacks in Tab. 9 Tab. shows the average TV distance, JS divergence, and KL divergence between distributions of penultimate layer activations of clean images and backdoor-trigger images clean ResNet-18, backdoor poisoned ResNet- 18, and backdoor poisoned ResNet-18 18 with BNA. For the backdoor poisoned ResNet-18 with BNA, we use TV divergence backdoor mitigation. For all attacks, the three divergences small for clean DNN, while relatively large for backdoor-poisone DNN. The distribution of backdoor-trigget instances severely deviates from that clean instances. However, with our mitigation method, the distribution alteration is significantly relieved. All the three divergences drastically reduced, which is consistent with the results in Tab. [
Trigger type
BadNet
CB
to inv
Lz inv
SP
WaNet
KL divergence
clean poisoned
0.0022
0.001
0.0013
0.0085
0.0017
0.0037
0.3211
0.752
0.4029
0.873
0.4385
0.2708
BNA
0.0291
0.0141
0.0337
0.0402
0.0146
0.0389
JS divergence
clean
0.0239
0.0166
0.0185
0.0461
0,0214
0.0311
poisoned
0.2867
0.3528
0.3275
0.3898
0.3041
0.2215
BNA
0,0952
0,0583
0,0847
0,0986
0.0582
0.0611
TV distance
clean
634.1953
376.4219
461.9531
1240.3359
554.3594
805.9648
poisoned
8877.2734
11177.3867
9203.9609
12482.4336
10119.3281
7237.7812
BNA
2463.582
1705.0156
2490.457
2794.7578
1761.2148
1749.2461
Table 9: Average TV distance, JS divergence, and KL divergence between distributions of clean instances and backdoor-trigger instances in clean DNN, poisoned DNN, and poisoned DNN with BNA using TV divergence in backdoor mitigation.
23

---

Under review as a conference paper at ICLR 2023
E CHOICE OF DIVERGENCE FORMS
In Tab. [ we only show the results for our method of using TV distance backdoor mitigation (Eq. 3). Here we show that our method is not sensitive to the choice of distribution divergence form. We respectively use TV distance, JS divergence, and KL divergence to mitigate the 5 all-to-one CB attack against CIFAR-10, and show the average distribution similarity measured by the three measurements after mitigation. The distribution similarity is calculated on the penultimate layer activations. As shown in Tab. TO the distribution alteration is significantly relieved after mitigation, regardless the measurement used in mitigation (Eq. 3)
divergence form used in mitigation distribution similarity after mitigation |
TV
JS
KI.
TV
1742
1730
1719
IS
0,0622
0.06172
0.0613
KL
0.0165
0.0163
0.0161
Table 10: Average TV, JS, and KL between clean instances and backdoor-trigger instances using TV, JS, and KL for measuring distribution similarity in backdoor mitigation.
24

---

Under review as a conference paper at ICLR 2023
F IMPACT OF PERTURBATION SIZE AND POISONING RATIO ON BACKDOOR MITIGATION
To observe the impact of attack settings the performance of backdoor mitigation methods, tune the poisoning ratio (i.e. the number of poisoned instances per source class) and perturbation size used in all-to-one CB attacks, and apply all the mitigation methods on these poisoned DNNs. The results are shown in Tab. TT Generally, the metrics for all methods decrease with increasing poisoning ratio and perturbation size. Although the performance for our BNA slightly declines as the attack is strengthened, our method still outperforms other methods in terms of the SIA. Besides, achieves the best or comparable ACC and ASR to other methods.
Mitigation method
the number of poisoned Instances class
perturbation size (*255)
100
150
200
250
2
3
ACC
0,8953
0.8826
0.8943
0.8799
0.8734
0.8918
0.8667
0.8825
0.8709
NC
ASR
0.0056
0,0238
0,0148
0.0057
0.0064
0.0238
0.0042
0.0157
0.0133
0.0065
0.8512
0.8412
0.8560
0,8283
0,8194
0,7881
ACC
0,8708
0,8473
0.8818
0.8941
0.8594
0.8473
0.8646
0.8822
0,9004
0.8919
LBAU
ASR
0.0789
0.0043
0.0712
0.5074
0.0450
0.0043
0.3052
0.0247
0.0011
0.2048
0.5823
ACC
0.8523
0.8271
0.8612
0.8204
0.7614
0.227
0.8486
0.8156
0,8418
0,8249
ANP
ASR
0.1940
0,8535
0.5401
0.0031
0.0043
0.8535
0.9836
U.6394
0.3670
0.2548
SIA
0.5158
0.1047
0,2440
0.6157
0.3701
0.1047
0.0142
0,1911
0,3238
0,4606
ACC
0.8942
0.8745
0.8949
0.8902
0.8674
0.8767
0.8823
U.7745
U
0.9990
NAD
ASR
0.0147
0.0086
0.0095
0.0106
0.0125
0.0070
0.0096
0.0086
0.0642
0.0586
SIA
0.8646
0.8504
0.8709
0.8695
0.8514
0.8631
0.8574
0.8504
0.8070
0.7896
ACC
0.8743
0.8832
0.8693
0.8659
0.8415
0,8832
03872
0.8619
0.8508
0.8394
ARGD
ASR
0.0117
0.0108
0,0073
0,0083
0.0085
0.0153
SLA
0.8590
0,8685
0.8528
0.8482
0.8267
0.8685
0.8574
U.8467
U33733
0.8196
ACC
0,9112
0,9094
0,9098
0,9102
0,9015
0.90944
0.9041
0,9079
0.8992
0.8912
BNA (ours)
ASR
0.0095
0.0141
0.0121
0.0170
0.0090
0.0141
0.0395
0.0222
0.0109
0.0388
SIA
0.8851
0.8837
0.8728
0,8840
0.8662
0.8851
0.8783
0.8711
0.8814
0.8435
Table 11: ACC, ASR, and SIA for our BNA, NC, I-BAU, ANP, NAD, and ARGD as function of (1) the number of poisoned instances injected into the training set; (2) the perturbation size under all-to-one CB attack.
25


Reviews:
[{"review_id": "tv8yFwuTQY5", "reviewer": "Reviewer_2vpF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a trojan attack mitigation method. The paper first observe that the the neural representation of clean and triggered samples are different in distribution. The proposed mitigation method first detects whether a model is attacked or not based on existing reverse engineering methods. For models that are considered attacked, the mitigation method minimizes the divergence between activation distributions of a clean input and a triggered input. \n\n", "summary_of_the_review": "Overall, the idea is not quite novel. The solution involves multiple steps and heavily depends on existing attack detection methods. The mitigation-after-detection scheme is not very well justified.", "strength_and_weaknesses": "The writing/presentation of the paper is not particularly well structured. A pseudocode can be helpful. \n\nWhile the experiment does seem to support the claim, I have reservations in both novelty and motivation. \n\nThe observation that neural activations of clean and triggered samples at the penultimate layer have different distribution is not particularly surprising, given it is well known that the softmax output of clean and triggered samples are different. The idea of mitigating by pulling the distribution together also seems incremental. \n\nA more serious issue is the fact that the method is highly dependent on the success of the existing detection method, and dependent on the reverse engineered triggers. This makes it hard to believe that the method is robust and generalizes well. First, as attacks continue evolving, it is hard to believe the used classic detection method is still as effective. Second, with so many engineering components involved, the method will be extremely hard to tune. \n\nMotivation-wise it is also unclear: if a model is already detected to be attacked, why is it worthy to mitigate it anyway?  A convincingly good mitigation method should be easy and simple to use. It should work despite whether the model is attacked.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YvokFXlLtE", "reviewer": "Reviewer_zrrB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper observes that the activation distributions are different for clean inputs and backdoor samples. It hence proposes to change activation values of inputs such that the predictions of backdoor samples can be corrected. Particularly, this paper utilizes existing trigger inversion methods to find a set of suspicious targets and generate corresponding backdoor triggers. For inputs with predicted labels in the set, it adjusts the activation values by using a transformation function. The transformation function changes the mean and standard deviation of activation values, which is obtained by comparing the activation differences between a set of clean samples with and without the inverted trigger. The experiment is conducted on five datasets and six types of backdoor attacks. Compared to several backdoor removal methods, this proposed approach can achieve good defense performance.", "summary_of_the_review": "The idea of transforming activation values to remove backdoors is interesting and the results show potential. However, the paper lacks structure and is hard to understand. In addition, more empirical studies are needed to further validate the defense assumption and the performance on more complex backdoor attacks.", "strength_and_weaknesses": "Strength\n\n+ Important topic of backdoor removal\n+ Interesting perspective of adjusting activation values\n\nWeaknesses\n\n- Many parts of the method are not clearly explained\n- Strong assumption of backdoor attacks and the theorem may not hold\n- Lack of background for trigger inversion\n- No comparison with state-of-the-art backdoor removal techniques\n- No evaluation on recent and complex backdoor attacks", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "J2RyJQ_BTQj", "reviewer": "Reviewer_y2Ez", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Paper Summary: Deep neural networks (DNNs) are vulnerable to backdoor attacks, and this paper proposes a mitigation framework to detect malicious instances and classify the triggered instance to its original class in the inference time.\n", "summary_of_the_review": "see strength and weakness", "strength_and_weaknesses": "Strengths:\n\n- Trendy topic\n\nWeaknesses:\n\n- Lack of novelty\n- Hard-to-follow\n- More explanations needed\n\nComments for the authors:\n\nDefense algorithm against DNNs backdoor attack has been a popular topic. In this paper, the authors propose a post-training backdoor mitigation framework to detect triggered inputs and output their original classes. The main intuition of the proposed framework is using a property of backdoor attack: the triggered instances could cause an alteration in the distribution of internal layer activations. Therefore, the authors leverage reverse-engineered triggers to correct  this distribution. The advantages of the proposed framework are twofold: Non-interference in the training process of the model and low computational complexity.\n\nHowever, I do have the following concerns.\n\n- First, the main contribution of this paper needs to be stated more clearly. The authors claim that they discover the triggered inputs can result in distribution alteration.  However, a similar conclusion has been proposed by Tran et al. (2018). Meanwhile, the authors state that the detector in the framework is different from most existing works, so what is the difference between this paper and Wang et al. (2019) and Xiang et al. (2020).\n\n- The writing of this paper needs to be improved. For example, this paper uses lots of notations which makes it hard to follow. The presentation about the choice of the parameters and the construction of the transformation function should be more clear. The authors could simplify them by using simple notations or list them in a table.\n\n- In the defender’s assumptions, the authors assume that the defender has no access to the training set of the classifier, but does possess an independent clean dataset. What’s the meaning of \"independent\" here? What are the distribution requirements for this clean dataset? In the experiments, the authors leverage 10% samples from the test set, which has the same distribution as the training set. However, we can acquire a classifier from the wild and we cannot know its training dataset. So, how can we choose this clean dataset in such a case?\n\n- More experiments are needed to demonstrate the generality of the proposed framework. For instance, the theoretical analysis focuses on the DNNs with Batch Normalization (BN) layers. However, the DNNs could leverage Group Normalization (GN) instead of BN. In the experiments, the architectures of DNNs are ResNet and MobileNet (both using BN). I would appreciate it if the authors could consider more different architectures.\n\n- Recently, Jia et al. [1] proposed a novel entangled backdoor attack. This attack leverages the soft nearest neighbor loss to entangle representations extracted from clean data and triggered data, which allows the model to use the same subset of parameters to recognize clean data and malicious data. As a result, the divergence of activation distribution between clean and backdoored instances is small, so I am curious whether the proposed method can detect such entangled attacks.\n\n[1] Jia H, Choquette-Choo C A, Chandrasekaran V, et al. Entangled watermarks as a defense against model extraction[C]//30th USENIX Security Symposium (USENIX Security 21). 2021: 1937-1954.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject