In Search of Smooth Minima for Purifying Backdoor in Deep Neural Networks
Authors: Nazmul Karim
Abdullah Al Arafat
Umar Khalid
Zhishan Guo
Nazanin Rahnavard
Venue: Submitted to ICLR 2023 (2023)
Keywords: AI Security
Backdoor or Trojan Attacks on Deep Networks
Safe and Robust AI

Abstract: The success of a deep neural network (DNN) heavily relies on the details of the
training scheme; e.g., training data, architectures, hyper-parameters, etc. Recent
backdoor attacks suggest that an adversary can take advantage of such training
details and compromise the integrity of DNN. Our studies show that a backdoor
model is usually optimized to a bad local minima, i.e., sharper minima as compared
to a benign model. Intuitively, backdoor can be purified by re-optimizing the
model to a smoother minima through fine-tuning with a few clean validation data.
However, fine-tuning all DNN parameters often requires huge computational costs as
well as sub-par clean test performance. To address this concern, we propose a novel
backdoor purification technique—N atural G radient Fine-tuning (NGF)—which
focuses on removing backdoor by fine-tuning only one layer. Specifically, NGF
utilizes a loss surface geometry-aware optimizer that can successfully overcome
the challenge of reaching a smooth minima under one-layer optimization scenario.
To enhance the generalization performance of our proposed method, we introduce
a clean data distribution-aware regularizer based on the knowledge of loss surface
curvature matrix, i.e., Fisher Information Matrix. To validate the effectiveness of
our method, we conduct extensive experimentation with four different datasets—
CIFAR10, GTSRB, Tiny-ImageNet, and ImageNet; as well as 11 recent backdoor
attacks, e.g., Blend, Dynamic, Clean Label, etc. NGF achieves state-of-the-art
performance in most of these benchmarks.

Under review as a conference paper at ICLR 2023
IN SEARCH OF SMOOTH MINIMA FOR PURIFYING BACKDOOR IN DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
The success of a deep neural network (DNN) heavily relies the details of the training scheme; e.g., training data, architectures, hyper-parameters, etc. Recent backdoor attacks suggest that adversary can take advantage of such training details and compromise the integrity door model is usually optimized to a bad local minima, i.e., sharper minima compared to a benign model. Intuitively, a backdoor model can be purified by re-optimizing the model smoother minima through fine-tuning with few clean validation data. However, fine-tuning all DNN parameters often requires huge computational cost and often results in sub-par clean test performance. To address this concern, we propose a novel backdoor purification technique-Naiinn -Natuaal Gradient Fine-tuning (NGF)-whie) focuses on removing backdoor by fine-tuming only one layer. Specifically, NGF utilizes a loss surface ssometry-award optimizer that can successfully overcome the challenge of reaching smooth minima under a one-layer optimization scenario. To enhance the generalization performance of our proposed method, we introduce a clean data distribution-aware regularizer based the knowledge of loss surface curvature matrix, i.e., Fisher Information Matrix. Extensive experiments show that the proposed method achieves state-of-thee -frtth-art performance wide range of backdoor defense benchmarks: four different datasets CIFARIO, GTSRB, Net, and ImageNet; 13 recent backdoor attacks, e.g., Blend, Dynamic, WaNet, ISSBA, etc.
DNN. Our studies show that a back-
INTRODUCTION
Training deep neural network (DNN) with fraction of poisoned or malicious data is often security- critical since the model can successfully learn both clean and adversarial tasks equally well. This is prominent in scenarios where one outsources the DNN training to a vendor. such scenarios, an adversary can mount backdoor attacks (Gu et al., 2019; Chen et al., 2017) through poisoning portion of training samples so that the model will misclassify any sample with a particular trigger or pattern to adversary- set label. Whenever DNN is trained in such manner, it becomes crucial to remove the effect of backdoor before deploying it for real-world application.
Different defense techniques (Liu et al., 2018; Wang et al., 2019; Wu & Wang, 2021; Li al., 2021a; Zheng et al., 2022) have been proposed for purifying backdoor. Techniques such as fine-pruning (Liu al., 2018) and adversarial neural pruning (Wu & Wang, 2021) require a long training time due to iterative searching criteria. Furthermore, the purification performance deteriorates significantly as the attacks get stronger. In this work, explore the backdoor insertion and removal phenomena from the DNN optimization point of view. Unlike a benign model, a backdoor model is forced to learn two different data distributions: clean data distribution and poisoned/trigger data distribution. Having to learn both distributions, backdoor model optimization usually leads to a bad local minima or sharper minima W.E.t. clean distribution. We claim that backdoor can removed by re-optimizing the model to smoother minima. One easy re-optimization scheme could be simple DNN weights fine-tuning with a few clean validation samples. However, fine-tuning all DNN parameters often requires huge computational cost and may result in sub-par clean test performance after purification. Therefore, intend to fine-tune only one layer to effectively remove the backdoor.
Fine-tuning only layer creates shallow network scenario where SGD-based optimization becomes a bit challenging. Choromanska et al. (2015) claims that the probability of finding bad local minima poor quality solution increases the network size decreases. Even though there

---

Under review as a conference paper at ICLR 2023
good-quality solutions, it usually requires exponentially long time to find those minima (Choromanska et al., 2015). As a remedy to this, we opt to use a curvature aware optimizer, Natural Gradient Decent (NGD), that has higher probability of escaping the bad local minima as well as faster convergence rate, specifically in the shallow network scenario (Amari, 1998; Martens & Grosse, 2015). To this end, we propose novel backdoor purification technique Natural Gradient Fine-tuning (NGF) which focuses on removing backdoor through fine-tuning only one layer. However, straightforward application of NGF with simple cross-entropy (CE) loss may result in poor clean test performance. To boost this performance, we use clean distribution-awve aware regularizer that prioritizes the update of parameters sensitive to clean data distribution. Our proposed method achieves SOTA performance in wide range of benchmarks, e.g., four different datasets including ImageNet, 13 recent backdoor attacks etc. Our contributions can be summarized as follows:
We analyze the loss surface characteristics of DNN during backdoor insertion and purifica- tion processes. Our analysis shows that the optimization of a backdoor model leads a bad local minima or sharper minima compared to a benign model. We argue that backdoor can be purified by re-optimizing the model to a smoother minima and simple fine-tuning can be a viable way for that. To the best of our knowledge, this is the first work that studies the correlation between loss-surface smoothness and backdoor purification.
We conduct additional studies on backdoor purification process while fine-tuning different parts of DNN. We observe that <GD-based one-layer fine-tuning fails to escape bad local minima and a loss surface geometry-aware optimizer can be an easy fix to this.
We propose novel backdoor purification technique based on Natural Gradient Fine-tuning (NGF). In addition, we employ a clean distribution awwre regularizer to boost the clean test performance of our proposed method. NGF outperforms recent SOTA methods in wide range of benchmarks.
2 RELATED WORK
Backdoor Attacks: Backdoor triggers can exist in the form of dynamic patterns (Li et al., 2020), a sin- gle pixel (Tran et al., 2018), sinusoidal strips (Barni et al., 2019), human imperceptible noise (Zhong et al., 2020), natural reflection (Liu et al., 2020), adversarial patterns (Zhang et al., 2021), blending backgrounds (Chen al., 2017), etc. Based on target labels, existing backdoor attacks can generally be classified as poison-label or clean-label backdoor attacks. In poison-label backdoor attack, the target label of the poisoned sample different from its ground-truth label, e.g., BadNets (Gu et al., 2019). Blended attack (Chen et al., 2017), SIG attack (Barni et al., 2019), WaNet (Nguyen & Tran, 2021), Trojan attack (Liu et al., 2017), and BPPA (Wang et al., 2022). Contrary to the poison-label attack, clean-label backdoor attack doesn't 't change the label of the poisoned sample (Turner et al., 2018; Huang et al., 2022; Zhao et al., 2020b). Recently, Saha et al. (2022) studied backdoor attacks on self-supervised learning.
Existing backdoor defense methods can be categorized into backdoor detection or purifying techniques. Detection based defenses include trigger synthesis approach (Wang et al., 2019; Qiao et al., 2019; Guo et al., 2020; Shen et al., 2021; Dong et al., 2021; Guo et al., 2021; Xiang et al., 2022; Tao et al., 2022), or malicious samples filtering based techniques (Tran et al., 2018; Gao et al, 2019; Chen et al., 2019). However, these methods only detect the existence of backdoor without removing it. Backdoor purification defenses can be further classified as training time defenses and inference time defenses. Training time defenses include model reconstruction approach (Zhao et al., 2020a; Li et al., 2021b), poison suppression approach (Hong et al., 2020; Du et al., 2019; Borgnia et al., 2021), 1),1,, and pre-processing approaches (Li et al., 2021a; Doan et al., 2020). Although training time defenses are often successful, they suffer from huge computational burden and less practical considering attacks during DNN outsourcing. Inference time defenses are mostly based on pruning approaches such as (Koh & Liang, 2017; Ma Liu, 2019; Tran et al., 2018; Diakonikolas et al., 2019; Steinhardt et al., 2017). Pruning-based based approaches are typically based on model vulnerabilities to backdoor attacks. For example, MCR (Zhao et al., 2020a) and CLP (Zheng et al., 2022) analyzed node connectivity and channel Lipschitz constant to detect backdoor vulnerable neurons. ANP (Wu & Wang, 2021) prune neurons through backdoor sensitivity analysis using adversarial search on the parameter space. Instead, we propose simple one-layer fine-tuning based defense that is both fast and highly effective. To remove backdoor, our proposed method revisits the DNN fine-tuning paradigm from novel point of view.
Backdoor Defenses:
2

---

Under review as a conference paper at ICLR 2023
3 BACKGROUND
Attack Model. We consider an adversary with the capabilities of carrying backdoor attack on DNN model, fo IR" + IR, by training it on a poisoned data set Drrrinn = {Xtrain: Yrrain. Here, is the parameters of the model, d is the input data dimension and is the total number of classes. The data poisoning happens through specific set of triggers that can only be accessed by the attacker. The adversary goal is to train the model in way such that any triggered samples 2 + E I' will be wrongly misclassified to a target label. y. Here, æ is a clean test sample and 6 E R2 represents the trigger pattern with the properties of [|o|| S c; where is the trigger magnitude determined by its shape, size and color. We define the poison rate as the ratio of poison and clean data in Drrain- An attack is considered successful if the model behaves as: fo(x)) = y and fo (+) = y. where y the true label for x. We use attack success rate (ASR) for quantifying such success.
Defense Goal. We consider defender with a task to purify the backdoor model fo using small clean validation set (usually 1 ~ 10% of the training data). The goal is to repair the model in way such that it becomes immune to attack, i.e., fe, (.) = y. Here, fo, the final purified model.
Natural Gradient Descent (NGD). Let us consider a model p(y| 8) with parameters 0 E R be fitted with input data {ax, y1) [Baaiin from an empirical data distribution Pa... where E Xtrain is an input sample and y/ E Yirain is its label. We try to optimize the model by solving:
0" E arg min [(0),
(1)
8
where C()) = Cly, fo(=)) = E(...)))>PP [-log [-log p(y|z, 0)] is the expected full-batch cross-entropy (CE) loss. SGD optimizes for 0" iteratively following the direction of the steepest descent (estimated
by column vector, VoL) and update the model parameters by: ((++1) <- g(t) (1)) v/") L. where a
is the learning rate. Since SGD uses the Identity matrix as the pre-conditioner, it is uninformed of the geometry of loss surface.
In NGD, however, the Fisher Information Matrix (FIM) is used as a pre-conditioner, which can be defined as (Martens & Grosse, 2015),
F(0) = P.,,,))PP.. Vo log p(ular, 0)+ (Vo log ())² E RRN
As FIM is a loss surface curvature matrix, a careful integration of in the update rule of 0 will make the optimizer loss surface geometry aware. Such integration leads us to the update equation of NGD, g((+1) <- (00) - (t) ((0())-1V L. Here, the natural gradient is defined F(0())-1 From the perspective of information geometry, natural gradient defines the direction in parameter space which gives largest change in objective per unit of change in model (p(y) xz 0)). Per unit of change in model is measured by KL-divergence (Amari, 1998; Park et al., 2000). Note that KL-divergence well connected with FIM as it can be used as a local quadrature approximation of KL-divergence of model change. Eqn. 2 suggests that one requires the knowledge of the original parameter (0) space to estimate it. Therefore, FIM can be thought of as a mechanism to translate between the geometry of the model (p(y)ær, 0)) and the current parameters (0) of the model. The way natural gradient defined the direction in parameter space is contrastive to the stochastic gradient. Stochastic gradient defines the direction in parameter space for largest change in objective per unit of change in parameter (0) measured by Eucludian distance. That is, the gradient direction is solely calculated based on the changes of parameters, without any knowledge of model geometry.
C.
SMOOTHNESS ANALYSIS OF BACKDOOR MODELS
In this section, we analyze the loss surface geometry of benign, backdoor, and purified models. To study the loss curvature properties of different models, we aim to analyze the Hessian of loss, H -V where we compute L using the clean training set. The Hessian matrix I/ is symmetric and one can take the spectral decomposition H QAQ", where A diag(A1, A2 ... AN) contains the eigenvalues and Q = [2192.... ..x] are the eigenvectors of H. As a measure for smoothness, we take the maximum eigenvalue, Amax(= A1), and the trace of the Hessian, Tr(ff) = i==N (ifg Low values for these two proxies indicate the presence of highly smooth loss surface (Jastrzebski et al., 2020). The Eigen Spectral density plots in Fig. la- 1b tell us about the optimization of benign and backdoor models. To create these models, we use the CIFARIO dataset and train a PreActResNet118 architecture for 200 epochs. To insert the backdoor, we TrojanNet (Liu et al., 2017) and poison
3

---

Under review as a conference paper at ICLR 2023
Denuty Rog Scak)
109
Prsi 125.7
ASH 10.3
Dome Ea Scave
9030 7097.7
ACC
ASA: 100.0
Donsity kog Saak)
Trth:: 8258
61336
ACC: 87-4
ASR:
Demdty 2og SQaM)
334
97391 17.3
ADR:
24.
10-1
2011
20-
60-
20-4
32-4
Eigenwalus
(a) Benign Model
(b) Backdoor Model
(c) Purified Model (SGD)
(d) Parified Model (NGF)
Figure 1:
Eigen Spectral Density plots of Loss Hessian for (a) benign, (b) backdoor (TrojanNet (Liu et al.,
2017)), and (c & d) purified models. In each plot, the maximum eigenvalue (Amax), trace of Hessian (Tr( clean test accuracy (ACC), and attack success rate (ASR) are also reported. Here, low Amax and Tr hints at the presence of smoother loss surface which often results in low ASR and high ACC. (a & b). Compared to a benign model, backdoor model tends to reach a sharper minima as shown by the larger range of eigenvalues (x-axis). During purification, SGD optimizer (c) rarely escapes sharp bad local minima (similar Amax and Tr(H) as the backdoor model) while our proposed method, NGF, (d) converges to smooth minima. We use CIFAR10 dataset with a NreActResNet18 (He al., 2016) architecture for all cvaluations.
Table I:
Backdoor removal performance while fine-tuning (FT) different parts of DNN. Fine-tuning only the last layer creates a shallow network scenario. In such scenario, there is high probability that SGD does not escape bad local minima. Whereas, NGF consistently optimizes to a smooth minima (indicated by low Amax for 6 different attacks) which results in backdoor removal, i.e., low ASR and high ACC. We consider CIFAR10 dataset and Ne architecture for all evaluations. A clean validation set is used for all purification.
FT
Badnets ASR
Blend ASR
Treai ASR
Dynamic ASR
CLB ASR
SIG ASR
Methods
Aves.
ACC
Ans
ACC
Ares
ACC
Arus
ACC
Ava:
ACC
Avac
ACC
Initial
573.8
100
92.96
715.5
100
94.11
516.3
100
89.57
564.3
100
92.52
717.6
100
92.78
514.1
100
$8.64
Fell-Net
4.42
4.87
85.92
4.65
4.77
83.61
3.41
3.78
62.18
2.34
4.73
55.61
4.68
1.83
87.41
8.98
1.04
$1.92
CNN Bbone.
4.1
5.00
85.64
5.14
4.92
87.25
4.19
3.95
81.86
2,46
5.11
$7.54
5.19
2.08
56.67
9.74
1.61
$1.55
Cls. (SGD)
556.1
98.27
90.17
541.7
97.29
93.48
513.0
96.25
87.36
446.5
93.38
91.36
361.9
89.21
91.73
563.2
95.70
$6.92
Cls. (NGP)
2.79
1.86
88.32
2.43
0.38
91.17
2.74
2.64
84.21
1.19
1.17
90.97
3,13
1.04
58.37
1.45
0.12
$4.16
rate of 10%. From the comparison of Amax and Tr(H), we can conjecture that optimization of a benign model produces smoother loss surface. We observe similar phenomena for different datasets and architectures; details are Appendix F. The main difference between benign and a backdoor model is that the latter needs to learn two different data distributions: clean and poison. Based on our observations, we state following conjectures:
Conjecture 1. Having to learn two different data distributions, a backdoor model reaches sharper minima, Lee., large Amax and Tr(H), as compared to the benign model.
We support this conjecture with empirical evidence presented in Table 1. Looking at the Amax in the 'Initial' row for all attacks (details are in Appendix D), it can be observed that all of these backdoor models optimizes to sharp minima. As these models are optimized on both distributions, they also have high attack success rates (ASR) as well as high clean test accuracy (ACC). Note that, the measure of smoothness is done w.r.f. clean data distribution. The use of clean distribution in our smoothness analysis is driven from the practical consideration as our particular interest lies with the performance w.E.l clean distribution; more details in Appendix C. Since high ASR and ACC indicate that the model had learned both distributions. it supports Conjecture 1.
Conjecture 2. Through proper fine-tuning with clean validation data, a backdoor model can be re-optimized to smoother minima w.r.t. clean data distribution. Optimization to smoother minima leads to backdoor purification, i.e., low ASR and high ACC.
we imply that the fine-tuning will lead to an optimal solution w.E.t the data distribution we fine-tune the model with. To support Conjecture 2, we show the removal performances of fine-tuning based purification methods in Table 1. To remove backdoor using a clean validation set (~11% 5% of train-set), we fine-tune different parts of the DNN for 100 epochs with learning rate of 0.01. As shown in Table 1, after proper fine-tuning (Full-Net, CNN-Bbone), the backdoor model re-optimizes to a smoother minima that leads to successful backdoor removal.
By proper
We observe that one can remove the backdoor by fine-tuning either the full network or only the CNN backbone (using SGD). However, these methods can be computationally costly and less practical. Furthermore, such fine-tuning often leads to high drop in ACC. As an alternative, one could fine-tune only the last or classification (Cls.) layer. However, even with a small validation set, layer network becomes shallow network optimize. According the
One-Layer
4

---

Under review as a conference paper at ICLR 2023
spin-glass analogy in Choromanska et al. (2015), as the network size decreases the probability for the SGD optimizer to find sharp local minima or poor quality minima increases accordingly. In case of shallow network, the quality of minima is decided by their distances from the global minima. Choromanska et al. (2015) also observes that the process of finding a path from bad local minima to good quality solution global minima takes exponentially long time. Therefore, it is not always feasible to use the SGD optimizer for shallow network. Table 1 (row-Cls. (SGD)) corroborates this hypothesis as SGD optimizer fails to escape the sharp minima resulting in similar ASRs as the initial backdoor model. Instead of using SGD, one can use natural gradient descent (NGD) that has higher probability of escaping the bad local minima as well as faster convergence rate, specifically in the shallow network scenario (Amari, 1998; Martens & Grosse, 2015). Therefore, to effectively purify backdoor model, we propose a novel Fisher Information matrix based backdoor purification objective function and optimize it using the NGD optimizer.
4.1 NATURAL GRADIENT FINE-TUNING (NGF)
Let us decompose the model parameters 0 as,
0 = (Wo.1. W1,1, W2.3. WD-1,1}
here, Wi,++ is the parameters between layer i and layer i+ 1, commonly termed as (i (1+1)* 17th layer's parameters. WL L-L.L is the 4 layer's (Cls. layer) parameters and we are particularly interested in fine-tuning only this layer. Now, consider validation set, Dval {Xyat, Yval) that contains only clean samples. We denote Oz. W 1..)..)) as the L layer's parameters. To purify the backdoor model, we formulate the following loss
Cply. fo(z))
+ " Σ diag(F((L)) (0z, -2,,)
(3)
2
which is combination of the CE loss on the validation set and regularizer. Here, Or. (fixed) is Lth layer parameters of the initial backdoor model, i.e., 0 0z.
In a backdoor model, some neurons/parameters are more vulnerable than others. The vulnerable parameters are believed to be the ones that are sensitive to poison/trigger data distribution (Wu & Wang, 2021). In general, CE loss does not discriminate whether a parameter is more sensitive to clean or poison distribution. Such lack of discrimination may allow drastic/unwanted changes to the parameters responsible for learned clean distribution. This usually leads to sub-par clean test accuracy after purification and it requires additional measures to fix this issue. Motivated by Kirkpatrick et al. (2017), we introduce a clean distribution aware regularization term product of two terms: an error term that accounts for the deviation of 02 from 0,; ii) a vector, diag (0))), consisting of the diagonal elements of FIM (F(0z)). As the first term controls the changes of parameters w.r.t 0z, helps the model to remember the already learned distribution. However, learned data distribution consists of clean and poison distribution both. To explicitly force the model to remember the clean distribution, we compute F(0,.) using a clean validation set; with similar distribution as the learned clean data. Note that, diag(F((0)) represents the square of the derivative of log- ligeeeidd of clean distribution w.r.t. ÖL... [V or,,, log p(y 0)]² (ref. eqn. (6)). In other words, diag [ (0z)) is the measure of importance of dr.,, towards remembering the learned clean distribution. If diag(F((O,) (0)))) has a higher importance, we allow minimal changes to o..: over the purification process. This careful design of such regularizer improves the clean test performance significantly. We use 17 as regularization constant.
The overall optimization problem using the loss-function defined in (3) for purifying the backdoor model So is as follows:
Objective function:
0p == arg min
x E Xeats y E Yeal
(4)
0,
p e((+1) + d - -o
Update Policy:
(5)
n
where, F(Or) = n j=1
(V,, log p(usl 0)- (Ve, log (,ylæ,,)))?
(6)
Here, F E [2|00l is the FIM, and n is the validation set size. Notice that, as we only consider fine-tuning of -ather the computation of F and F-1 (00r) x (0r.] matrices) becomes tractable.
5

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison of different defense methods for four henchmark datasets. Backdoor removal performance, Le, drop in ASR, against a wide range of attacking strategies show the effectiveness of NGF, For CIFARIO and GTSRB, the poison rate is 10%. For T'ny-- ImygeNet and ImageNet, employ ResNet34 and ResNet50 architectures, respectively. We use a poison rate of 5% for these datasets and report performance successful attacks (ASR close to 100%) only. Average drop (J) indicates the changes in ASR/ACC compared to the baseline, ie. ASR/ACC of No Defense. Higher ASR drop and lower ACC drop is desired for a good defense.
Dataset
Mechod
No Defense
Vanilla
ANP
I-BAU
AWM
NGF (Ours)
Amacks
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
Bvsiga
B
95.21
13
92.25
D
93.98
0
93.56
95.80
()
94,10
Bodnets
100
92.96
4.87
85.90 92
2.84
85.96
9.72
87.85
4.34
86.17
1.86
S8.32
Blend
100
94.11
57.61
3.81
89.10
90.84
2.13
53.93
0.35
91.17
Truj one
100
5.47
2.64
Tioj-all
100
88.33
3.91
5.53
9.82
4.42
2.79
CIFAR-10
Dyn-one
SIG
100
100
88.64
92.52
4.75
1.04
88.66
81.92
5/2
0.37
1.78
86. 26
83.50
10.48
4.12
80. 16
83.57
0.90
3.35
83.38
88.4
1.17
0.12
$4.16
90.97
Dyn all
100
92.61
4.28
68.32
2.19
B4.51
10.30
89.74
2.46
K7.72
1.11
90.19
CLB
100
92.78
1.83
K7.41
1.41
85.07
5.78
56 70
1.89
84.18
1,04
88.37
CBA
93.20
83.71
85.63
$5.05
85.
24.60
85.97
FBA
100
Wañet
3.18
2.96
2.38
ISSBA
99. 80
92.50
6.76
85
3.82
89.2 89:20
12.48
90.03
4.57
89.50 59
4.24
90. 18
RPPA
99.70
93.82
9.94
90.23
10:46
90.57
9.94
00.68
10.60
90:88
7.14
91.84
Avg. Deup
92.61
0.005
87.09
4.98
88.882
91.32 4
4.53
95,01
3.33
Benign
0
97.87
93.08
0
95.42
96. 18
95.32
0
95.76
Rodness
100
97.38
0.35
93.17
01:34
11
Dlend
100
4.41
93.02
2.91
Trij-one
99.50
96.27
2.07
90:45
1.81
92:74
3.04
97.17
2:27
1:21
94
GTSRB
Troj
99.71
96.08
2.48
89.73
2.16
92.51
2.79
93.28
1.94
92.84
1.58
93.87
SiG
97.13
96.93
1.93
91.41
6.17
91. 01.82
2.66
93.10
5.32
92.68
3.24
93.48
Dyn-cise
100
97.27
2.27
91.26
2,08
93. 15
5.82
95.54
1.89
93.55
1.51
94.27
Dyn-all
100
07.05
2.84
91:42
2:49
92.88
4.87
93.95 98
2.74
93.17
1.26
94.14
RPPA
Avg. Deup
96.34 4
6.30
960110
99..1 4
2.83
916,022
3.,00
97.39
2.79
Resign
0
62.56
59.30
59.34
59.67
Rodnets
100
$9.50
3.84
$3.58
61.23
55.41
13:20
54.56
31.44
$4.81
2.34
55.84
Trijam
100
59.16
6.77
52.62
79.56
54.76
11.94
55.10
38.23
$4.28
3,38
54.87
Inag-EnageNNt
Hlend
100
66..1
2,18
51.22
83.58
54.70
17.42
54.19
41.37
53.78
1.55
54.98
SIG
98.48
66101
52.18
28.67
54.71
9.31
55.72
27.68
$4.11
2.81
54.63
CLB
60.33
16.24
4.06
Avg. Deup
94.55 4
7.63
45,38]
4
66.19 1
5.48
96.40 1
Benign
0
77.06
73.52
0
68.85
74.21
71.63
0
74.51
Rodnets
99.24
74.53
5.91
69.37
43.31
66.
21.57
69:46
21.18
69.44
4.61
70.46
Tiojan
99.21
74.00
4.63
69:15
38.81
66.
25.74
09:35
28.85
08.62
4.02
69.97
ImageNet
Blend
100
74.42
4.43
70
57.79
65.51
27.45
68.61
34.15
68.91
3.83
70.52
SIG
94.66
74.69
3.25
16.28
66.08
70.02
16.47
69.74
2.94
71.36
CLB
95.06
74.14
18:37
Aug. Drop
93.25
4.811
62.72]
8.28 5 76.22
4.88
72.80 1 5.15
93.94 5 3,85
After solving the above optimization problem, we will get modified parameters, W WL-1LL Finally, we get the purified model, fa, with 0p as
O, = (Wo,, WI,,, W2.3. ".. W W...I...
Fig. Ic-ld show that NGF indeed does reach the smooth minima as opposed to SGD hased fine-tuning. We provide additional results in Table for both NGF and SGD. Notice that the purified model seems to have a smoother loss surface than the benign model (2.7 vs. 20.1 for Amax). This, however, does not translate to better ACC than the benign model. The ACC of the purified model is always bounded by the ACC of the backdoor model. To the best of our knowledge, our study on the correlation between loss-surface smoothness and backdoor purification is novel. NGF is also the first method to employ second-order optimizer for purifying backdoor. More details are in Appendix C
5 EXPERIMENTAL RESULTS
5.1 EVALUATION SETTINGS
Datasets: To begin with, we evaluate our proposed method through conducting wide range of experiments on two widely used datasets for backdoor attack study: CIFARIO (Krizhevsky et al., 2009) with 10 classes, GTSRB (Stalkkamp et al., 2011) with 43 classes. As test of scalability, also consider Eiag-emaeeNet (Le & Yang, 2015) with 100,000 images distributed among 200 classes and ImageNet (Deng et al., 2009) with 1.28M images distributed among 1000 classes.
Attacks Configurations: We consider 13 state-of-the-an backdoor attacks: 1) Badnets (Gu et al., 2019), 2) Blend attack (Chen et al., 2017), 3 & 4) TrojanNet (Troj-one & Troj-all) (Liu et al., 2017), 5) Sinusoidal signal attack (SIG) (Barni et al., 2019), 6 & 7) Input-Aware Attack (Dyn-one and Dyn-all) (Nguyen & Tran, 2020), 8) Clean-label attack (CLB) (Turner et al., 2018), 9) Composite

---

Under review as a conference paper at ICLR 2023
Backdoor Insertion
Backdoor Insertion
Backdoor Purificattin
Backdoor Purification
3000
Beelon Trgus
5000
4000
Tropetion
Badnets (ANM)
Nax. Eignsaals
600
SiG
ACCIASR
Paitet 1449)
516
CLD (ASR)
200
M
15 S
CLB (ee
2000
ara (anR)
Erupertet (4E3)
CLD JACO
Tradecher (8GC)
1000
20
100
120
60
90
100
1.0
23
30
40
20
Number of Epochs
Number of Epochs
Number Epocrs
Number of Epochs
(a) Amax vs. Epochs
(b) ACC/ASR vs. Epochs
(c) Amax Epochs
(d) ACC/ASR vs. Epochs
Loss Surface characteristics of a DNN during backdoor insertion and purification processes. a & b) As the joint optimization on clean and poison distribution progresses, i.e., high ACC & ASR, the loss surface becomes less and less smoother, ie., high Amw)). с & d) One can purify backdoor by gradually making the loss surface smoother. We use CIFARIO dataset with four different attacks.
Figure 2:
backdoor (CBA) (Lin et al., 2020), 10) Deep feature space attack (FBA) (Cheng et al., 2021), 11) Warping- based backdoor attack (WaNet) (Nguyen & Tran, 2021). 12) Invisible triggers based backdoor attack (ISSBA) (Li et al., 2021c), and 13) Quantization and contrastive learning based attack (BPPA) (Wang et al., 2022). To ensure fair comparison, we follow the similar trigger patterns and settings as in their original papers. In Troj-one and Dyn-one attacks, all of the triggered images have same target label. On the other hand, target labels are uniformly distributed over all classes for Troj-all and Dyn-all attacks. For creating these attacks on CIFARIO and GTSRB, we use a poison rate of 10% and train a PreActResNet18 (He et al., 2016) and WideResNet-16-1 (Zagoruyko & Komodakis, 2016) architectures, respectively, for 250 epochs with an initial learning rate of 0.01. More details on paper-parameters and overall training settings can be found in Appendix D.
Defenses Configurations: We compare our approach with 4 existing backdoor mitigation methods: 1) Vanilla Fine-Tuuning (FT); where we fine-tune all DNN parameters, 2) Adversarial Neural Pruning (ANP) (Wu & Wang, 2021) with 1 1% clean validation data, 3) Implicit Backdoor Adversarial Unlearn- ing (I-BAU) (Zeng et al., 2021) 4) Adversarial Weight Masking (AWM) (Chai & Chen, 2022). We also compare NGF with another recent defense technique described in (Zheng et al., 2022). However, we present this comparison in the Appendix E due to several performance issues.' To apply NGF on CIFAR1O, we fine-tune the last layer of the DNN for Ep epochs with 1% clean validation data. Here, E, is the number of purification epochs and we choose value of 100 for this. For optimization, choose a learning rate of 0.01 with a decay rate of 0. 1/40 epochs and consider regularization constant n to be 0.1. Additional experimental details for NGF and other defense methods are in Appendix D).3. For GTSRB, we increase the validation size to 3% as there are less samples available per class. Rest of the training settings are same as CIFARIO. For NGF on Tiny-ImageNet, we consider a validation size of 5% as size less than this seems to hurt clean test performance (after purification). We fine-tune the model for 15 epochs with an initial learning rate of 0.01 with a decay rate of 0.3/epoch. Finally, we validate the effectiveness of NGF on ImageNet. For removing the backdoor, we use 3% validation data and fine-tune for 2 epochs. learning rate of 0.001 has been employed with decay rate of 0.005 per epoch. We define the effectiveness of a defense method in terms of average drop in ASR and ACC over all attacks. A highly effective method should have a high drop in ASR with a low drop in ACC. We define ASR as the percentage of poison test samples that are classified to the adversary-sed set target label.
5.2 PERFORMANCE EVAL.ATTION OF NGF
In Table 2, we present the performance of different defenses for four different datasets.
We consider five label poisoning attacks: Badnets, Blend, TrojanNet, Dynamic, and BPPA. For TorjanNet, we consider two different variations based on label-mapping criteria: Troj-one and Troj-all. Regardless the complexity of the label-mapping type, our proposed method outperforms all other methods both in terms of ASR and ACC. We also create two variations for Dynamic attack: Dyn-one and Dyn-all. Dynamic attack optimizes for input-aware triggers that are capable of fooling the model: making it more challenging than the static trigger based attacks (Badnets, Blend and Trojan). However, NGF outperforms other methods by satisfactory margin. We also consider attacks that does not change the label during trigger insertion, i.e., clean label attack. Two such attacks are CLB and SIG. For further validation of our proposed method, we use deep feature based attacks, CBA and FBA. Both of these attacks manipulates deep features for backdoor insertion. Compared
CIFAR10:
Based on our re-run, we notice significantly larger drop in ACC as compared other defenses.
7

---

Under review as a conference paper at ICLR 2023
to other defenses, NGF shows better effectiveness against these diverse set of attacks achieving an average drop of 95.01% in ASR while sacrificing an ACC of 3.33% for that. Table 2 also shows the performance of baseline methods such I-BAU and AWM. AWM performs similarly as ANP and often struggles to remove the backdoor.
GTSRB: In case of GTSRB, almost all defenses perform similarly for Badnets and Trojan. This, however, does not hold for blend as we achieve an 2.17% ASR improvement over the next best method. The performance is consistent for other attacks as well. Overall, we record an average 97.39% ASR drop with only an 2.79% drop in ACC. In some cases, ACC for I-BAU are slightly better as uses a much larger validation size (5%) for purification than other defense techniques.
ImageNet: For scalability test of NGF, we consider two large and widely used datasets, Tiny- ImageNet and ImageNet. In consistence with other datasets, NGF obtains SOTA performance in these diverse datasets too. The effectiveness of ANP reduces significantly for this dataset. In case of large models and datasets, the task of identifying and pruning vulnerable neurons gets more complicated and may result in wrong neurons pruning.
5.3 ABLATION STUDIES
Smoothness Analysis of Different Attacks: We show the relationship hetween loss surface smooth- ness and backdoor insertion process in Fig. 2a-2b. During backdoor insertion, the model is optimized for 2 different data distributions: clean and poison. Compared to benign model, the loss surface of a backdoor becomes much sharper as the model becomes well optimized for both distributions, i.e., model has both high ASR and high ACC. At the beginning of training, both backdoor and benign models are far from being well optimized. The difference between these models are prominent once the model reaches closer to the final optimization point. As shown in Fig. 2b, the training becomes reasonably stable after 100 epochs with ASR and ACC near saturation level. Comparing Amax of benign and all backdoor models after 100 epochs, we notice a sharp contrast in Fig. 2a. This validates our previous claim on loss surface smoothness of benign and backdoor models. During purification pe- riod as shown in 2c-2d, the model is being op- timized to a smoother minima. As a result, ASR becomes close to O while retaining good clean test performance. Note that, we calculate loss Hessian and Amax using all DNN parame-
Table 3: Performance comparison of NGF to other SGD-based optimizers. A more suitable sharpness-awee aware SGD-baser based optimizer also considered here. However, NGF is far more effective in purifying backdoor (lower ASR) due to its consistent convergence to smooth minima. We use CIFAR 10 dataset for these evaluations.
Defense
No Defense
AdaGrad
RMSProp
Adam
SAM
NGF (Ours)
Attakks
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
Badnets
100
92.96
96.54
91.16
98.33
91.73
97.68
91.45
91.08
90.12
1.86
$8.32
Blend
97.43
91.17
Trojan
100
89.57
95.52
88.51
94.87
88.02
74
87.98
92. 15
88.33
2.64
84:21
Dynamic
100
92.52
97.37
91.45
93.50
91.12
96.90
91.40
92.22 74
90.79
1.17
90.97
SIG
100
88.64
$6.20
87.98
86.31
87.74
85.66
87.75
68
88.04
0,31
83. 14
CLB
100
92.78
96.81
90.86
95.53
90.96
95.87
91.02
91.04
90.97
1.04
83.37
ters. This indicates that changing the parameters of only one layer impacts the loss landscape of
whole network. Even though the CNN-backbone parameters are frozen, NGF changes the last layer in way such that whole backdoor network behaves differently, i.e., like a benign model.
Evaluation of Different Optimizers: We compare the performance of NGF with different variants of first-order optimizer: (i) AdaGrad (Duchi et al., 2011), (ii) RMSProp (Hinton et al.), (iii) Adam (Kingma & :Ba, 2014), and (iv) Sharpness-Aware Minimization (SAM) (Foret et al., 2020) is a recently proposed SGD-based optimizer that explicitly penalizes the abrupt changes of loss surface by bounding the search space within small region. This forces the changes of model parameters in a way such that the optimization achieve smoother loss surface. Ta- ble 3 shows that NGF outperforms all of these variants of first-order optimizer by huge margin. At the same time, proposed method achieves comparable clean test performance. Although SAM usually performs better than vanilla SGD in terms of smooth DNN optimization, SAM's performance in shallow network scenario (our case) is almost similar to vanilla SGD. Two potential
Table 4: Avg. runtime comparison for different datasets. Here, #Parameters is the total number of parameters in the last layer. An NVIDIA RTX 3090 GPU used for all experiments.
Datavet
Parsmoters
Method
Runtime (Sec)
CIFAR10
5120
NGF
GUSRB
22016
NGF
Ting-EmageNet
409.6K
NGF
657.6
374.2
ImageNett
2.048M
FT NGF
1681.4
2771
reasons behind this poor performance are (i) using a predefined local area to search for maximum
8

---

Under review as a conference paper at ICLR 2023
Table 5: Performance of SGD-Long and NGF while fine-tuning only the last layer of DNN. For SGD-Loug, we consider long purification period with E, 2500. NGF performance with and without the regularization term underlines the importance of the proposed regularizer. The results shown here are for CIFAR 10 dataset.
Methods
Hadnets ASR
ACC
ASR
Bland
ACC
ASR
Trojan
ACC
Dynamic ASR
ACC:
ASR
CLB
ACC
ASR
SIG
ACC
ASR
CBA
ACC
Rumime (Secs.)
Initial
100
92.06
100
94.11
100
89.57
100
92.52
100
92.78
100
88.64
93.20
90.17
SGDS Long
82.34
90668
7:13
92.46
86.18
87.29
57.13
96.51
13.84
88.11
0.36
X5.74
84,41
86.87
907.5
NGF wlo Reg.
1.91
87.65
0.31
90.54
3,04
R3.31
1.28
90.24
0.92
87.13
0.16
84.46
25,58
84.81
37,8
NGF
88.32
91.17
84.21
1,37
90.97
88.37
84.16
24.60
85.97
38.3
Table 6: Evaluation of NGF on backdoor attacks with high poison rates, upto 50%. We consider CIFARIO dataset and two closely performing defenses for this comparison.
Attack
RodNets
Blend
Trojen
Poison Rote
25R8
35%
50%
25%
35%
50%
25%
55%
50%
Method
ASR
ACC
ASR
ACC|
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC|
ASK
ACC
ASR
ACC
No Defanse
100 88.
100
87.43
100 85.11
100 86.21
100
35.32
100
83.28
100
87.88
100
86.81
100
85.97
ANP
7.81 82.22
16.35
80.72
80 78.27
20.00
82.84
47,02
78.34
56.29
69.15
11.96
76 26
63.99
72. 10
89.83
70:02
5:21 78.11
11.52 09
1.41 68.73
4.56
7.97
55.20
72.05
5.59
70:98
NGF (Ours)
2.12 85.50
2.47
84.88
4.53
N2.32
0.83
80.62
1.64
79.62
2:21
76.37
3,02
83.10
3.65
81.66
4.66
80.30
loss, and (ii) using 'Euclidean distance" metric instead of geometric distance metric. In contrast, NGD with curvature geometry aware Fisher Information Matrix can successfully avoid such bad minima and optimizes to a global minima.
In Table 4, we show the average runtime for different defenses. Similar to purification performance. purification time is also an important indicator to measure the success of defense technique. In Section 5.2, we already show that our method outperforms other defenses in most of the settings. As for the run time, our method completes the purification (for CIFAR 10) in just 38.3 seconds; which is almost half as compared to FT. The time-advantage of our method also holds for large datasets and models, e.g., ImageNet and ResNet50. Runtime comparison with other defenses is in the Appendix H.
Runtime Analysis:
Effect of Proposed Regularizer: In this section, we analyze the effect of regularizer and long training with SGD. The effect of our clean distribution-aware regularizer can be observed in Table 5. NGF with the proposed regularizer achieves an 1 % clean test performance improvement over vanilla NGF. For long training with SGD (SGD-Long), we fine-tune the last layer for 2500 epochs. Table 5 shows the evaluations of SGD-Long on 7 different attacks. Even though the ASR performance improves significantly for CLB and SIG attacks, SGD- FT still severely underperforms for other attacks. Moreover, the computational time increases significantly over NGF. Thus, our choice of NGDD-based FT as a fast and effective backdoor purification technique is well justified.
Strong Backdoor Attacks: By increasing the poison rates, we create stronger version of different attacks against which most defense techniques fail quite often. We use 3 different poison rates, (25%, 35%, 50%]. We show in Table 6 that NGF is capable of defending very well even with a poison rate of 50%, achieving a significant ASR improvement over FT. Furthermore. there is a sharp difference in classification accuracy between NGF and other defenses. For 25% Blend attack, however, ANP offers a slightly better performance than our method. However, ANP performs poorly in terms of removing backdoor as it obtains an ASR of 29.96% as compared to 0.83% for NGF.
6 CONCLUSION
We propose novel backdoor purification technique based on natural gradient descent fine-tuning. The proposed method is motivated by our analysis of loss surface smoothness and its strong correlation with the backdoor insertion and purification processes. As a backdoor model has to learn an additional data distribution. it tends to be optimized to bad local minima or sharper minima compared to a benign model. We argue that backdoor can be removed by re-optimizing the model to a smoother minima. We further argue that fine-tuning single layer is enough to remove the backdoor. Therefore, in order to achieve a smooth minima in a single-layer fine-tuning scenario, we propose using an FIM-based DNN objective function and minimize it using a curvature NGD optimizer. Our proposed method achieves SOTA performance in a wide range of benchmarks. Since we fine-tune only one layer the training time overhead reduces significantly, making method one of the fastest among SOTA defenses. In the future, we aim to extend our smoothness analysis to 3D point-cloud attacks well attacks on contrastive learning.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):21-276. 1998.
Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 101-105. IEEE, 2019.
Eitan Borgnia, Valeriia Cherepanova, Liam Fowl, Amin Ghiasi, Jonas Geiping, Micah Goldblum, Tom Goldstein, and Arjun Gupta. Strong data augmentation sanitizes poisoning and backdoor attacks without an accuracy tradcoff. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3855-3859. IEEE, 2021.
Shuwen Chai and Jinghui Chen. One-shot neural backdoor erasing via adversarial weight masking. arXiv preprint ar 7XXx:2207.04497,, 2022.
Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Deepinspect: A black-box trojan detection and mitigation framework for deep neural networks. In ISCAI, volume 2, pp. 8, 2019.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint (rXivv1712.055222 2017.
Siyuan Cheng, Yingqi Liu, Shiqing Ma, and Xiangyu Zhang. Deep feature space trojan attack of neural networks by controlled detoxification. In AAAI, volume 35, pp. 1148-11500 2021.
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pp. 192-204. PMLR, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255. IEEE, 2009.
Ili Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. Sever: A robust meta-algorithm for stochastic optimization. In International Conference on Machine Learning, pp. 1596-1606. PMLR, 2019.
Bao Gia Doan, Ehsan Abbasnejad, and Damith Ranasinghe. Februus: Input purification defense against trojan attacks on deep neural network systems. In Annual Computer Security Applications Conference, pp. 897-912, 2020.
Yinpeng Dong, Xiao Yang, Zhijie Deng, Tianyu Pang, Zihao Xiao, Hang Su, and Jun Zhu. Black-box detection backdoor attacks with limited information and data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16491, 2021.
Min Du, Ruoxi Jia, and Dawn Song. Robust anomaly detection and backdoor attack detection via differential privacy. arXiv preprint arXiv:1911.07116, 2019.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awar minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021. URL https: http///pnnweeeeeceeeee
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference, pp. 113-125, 2019.
2019.
Tianyu Gu, Kang Liu, Brendan Dolan Gavitt,, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230-4724, 2019.
10

---

Under review as a conference paper at ICLR 2023
Junfeng Guo, Ang Li, and Cong Liu. Aeva: Black-box backdoor detection using adversarial extreme value analysis. arXiv preprint arXiv:2110.14880, 2021.
Wenbo Guo, Lun Wang, Yan Xu, Xinyu Xing, Min Du, and Dawn Song. Towards inspecting and eliminating trojan backdoors deep neural networks. In 2020 IEEE International Conference on Data Mining (ICDM), pp. 162-171. IEEE, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer. 2016.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Rmsprop: Divide the gradient by a running average of its recent magnitude. coursera: Neural networks for machine learning.
Sanghyun Hong, Varun Chandrasekaran, Yigitcan Kaya, Tudor Dumitras, and Nicolas Papernot. On the effectiveness mitigating data poisoning attacks with gradient shaping. arXiv preprint arXiv:2002. 1497, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. arXiv preprint urXiv:2202.03425. 2022.
Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks. arXiv preprint arXiv:2002.09572, 2020.
Zhiwei Jia and Hao Su. Information-theoretic local minima characterization and regularization. In International Conference on Machine Learning, pp. 4773-4783, PMLR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412,690, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka aaabska-Barainska. et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-355 -3326,, 2017.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894. PMLR, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009,
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models poisoned data. Advances in Neural Information Processing Systems, 34, 2021a.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distil- lation: Erasing backdoor triggers from deep neural networks. arXiv preprint arXiv:2101 100.05930, 2021b.
Yiming Li, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: survey. arXiv preprint 20X0:2007.08745, 2020.
Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with samplc-specific triggers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16463-16472, 2021c.
Junyu Lin, Lei Xu, Yingqi Liu, and Xiangyu Zhang. Composite backdoor attack for deep neural network by mixing existing benign features. In CCS, pp. 113-131, 2020.
11

---

Under review as a conference paper at ICLR 2023
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over- per-paaameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 2022.
Kang Liu, Brendan attacks on deep neural networks. International Symposium on Research in Artacks, Intrusions, and Defenses, pp. 273-294. Springer, 2018.
and Siddharth Garg. Fine-pruning Defending against backdooring
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack neural networks. 2017.
Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack deep neural networks. In European Conference Computer Vision, pp. 182-199. Springer, 2020.
Shiqing Ma and Yingqi Liu. Nic: Detecting adversarial samples with neural network invariant checking. In Proceedings of the 26th network and distributed system security symposium (NDSS 2019), 2019.
James Martens and Roger Grosse. Optimizing neural networks with factoree approximate curvature. In International conference on machine learning, pp. 2408-2417. PMLR, 2015.
Anh Nguyen and Anh Tran. Wapet-imperceptibbee warping-based backdoor attack. arXiv preprint arXiv:2/02. 10369, 2021.
Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. Advances in Neural Information Processing Systems, 33:3454-3464, 2020.
Hyeyoung Park, S-I Amari, and Kenji Fukumizu. Adaptive natural gradient learning algorithms for various stochastic models. Neural Networks, 13(7):755-764, 2000.
Ximing Qiao, Yukun Yang, and Hai Li. Defending neural backdoors via generative distribution modeling. Advances in neural information processing systems, 32, 2019.
Aniruddha Saha, Ajinkya Tejankar, Soroush Abbasi Koohpaycgani, and Hamed Pirsiavash. Backdoor attacks on self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13337-13348 2022.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chier Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma. and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm optimization. In International Conference on Machine Learning, pp. 9525-9536. PMLR, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for scage image recognition. arXiv preprint arXiv: 1556, 2014.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian lgel. The german traffic sign recognition benchmark: a multi-class classification competition. The 2011 international joint conference on neural nefworks, pp. 1453-1460. IEEE, 2011.
Jacob Steinhardt, Pang Wei Koh, and Percy S Liang. Certified defenses for data poisoning attacks. Advances in neural information processing systems, 30, 2017.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. 2014. arXiv preprint arXiv:1409 4442, 10, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
12

---

Under review as a conference paper at ICLR 2023
Guanhong Tao, Guangyu Shen, Yingqi Liu, Shengwei An, Qiuling Xu, Shiqing Ma, Pan Li, and
Xiangyu Zhang. Better trigger inversion optimization backdoor scanning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13368-13378, 2022.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. Advances in neural information processing systems, 31, 2018.
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Clean-label backdoor attacks. 2018.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. 2019 IEEE Symposium Security and Privacy (SP), pp. 707-723. IEEE, 2019.
Zhenting Wang, Juan Zhai, and Shiqing Ma. Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15074-15084, 2022.
Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In NeurIPS, 2021.
Zhen Xiang, David Miller, and George Kesidis. Post-training detection of backdoor attacks for two-class and multi-attack scenarios. arXiv preprint arXiv:2201.08474, 2022. 2022.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint wrXiv:1605.07146 2016.
Yi Zeng, Si Chen, Won Park, Z Morley Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. arXiv preprint arXiv:2110.0375:. 2021.
Quan Zhang, Yifeng Ding, Yongqiang Tian, Jianmin Guo, Min Yuan, and Yu Jiang. Advdoor: adversarial backdoor attack of deep learning system. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 127-138, 2021.
Pu Zhao, Pin- Yu Chen, Payel Das. Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridging mode connectivity in loss landscapes and adversarial robustness. arXiv preprint vrXiv:2005.00060 2020a.
Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu-Gang Jiang. Clean- label backdoor attacks on video recognition models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14443-14452 2020b.
Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitzness. arXiv preprint arXiv:2208.03//1 2022.
Haoti Zhong, Cong Liao, Anna Cinzia Squicciarini, Sencun Zhu, and David Miller. Backdoor embedding in convolutional neural network models via invisible perturbation. In Proceedings of the Tenth ACM Conference on Data and Application Security and Privacy, pp. 97-108, 2020.
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
Section B describes our proposed algorithm. Section C discusses the intuitions behind the uses of clean data distribution for smoothness analysis, more explanation on why backdoor behavior and weight loss-landscape are related and smoothness helps to mitigate the effect of backdoor, how FIM helps to smooth the loss-landscape, and why SAM (Foret et al., 2021) does not work in proposed problem setup. Section D contains the experimental details of different attacks and defenses. Section contain comparison with another recent defense technique. We present smoothness analysis with different datasets and architectures in Section F. Section G shows the Label Correction Rates for different defense techniques. Section H and contain additional runtime analysis. Section K contains more ablation studies purification with last layer re-initialization. Section L. and M discuss more attacks, all2all and combined attacks. Our code is available at anonymous GitHub link 2.
B NATURAL GRADIENT FINE-TUNING
In our proposed method, we aim to remove backdoor by fine-tuning only the last layer. The manner in which we perform that fine-tuning is described in Algorithm After purification, the model should behave like benign/clean model producing same prediction irrespective of the presence of trigger. Note that Kirkpatrick et al. (2017) proposed a similar regularizer, known as elastic weight consolidation (EWC) used in continual learning. EWC helps model to learn new task while keeping the knowledge of previously learned tasks. Our clean data distribution aware regularization, instead, only helps preserve the knowledge of previously learned task corresponding clean data distribution (not task corresponding to poison samples) through computing F using clean validation set. Thus, the design of our regularizer backdoor specific and needs careful attention as it is crucial for clean test performance.
Algorithm 1: Natural Gradient Fine-tuning
Input: Backdoor Model (fo()), 1% Clean Validation Set Dvat, Number of Purification Epochs N Deat
X,
F
Duad EvexweEy Ve, ylog fo(x) (Vs, ylog fo(z)) parameter of initial backdoor model.
11 0g is the last layer's
for = Ao do
Calculate loss, L Ccx(V, (X)) + Σ, (diag(F)); (0%.. -b,ss)²
F+ Peuil EvEX,VEY layer's parameter at 7th iterations F (C)
Voj ylog (x) Vog ylog for fx)
11 02 is the last
/1 a is the Learning rate // Wwn''s are frozen parameters 0p (Wo.1. WI.,.'.. W2-21-1,081 11 0p is the purified model's parameter Output: Purified Model, fap
0%1 - -0,, a F..F.F-VV Vog 0 (W,,, WI...'. WL-21-1,0 0/t1
C MORE EXPLANATIONS ON SMOOTHNESS AND BACKDOOR
C.I WHYY SMOOTIINESS ANALYSIS W.R.T CLEAN DISTRIBUTION?
In general, a backdoor model is usually well optimized w.r. clean and poison data distribution. Therefore, it is designed to perform well on both distributions. If we look at the smoothness analysis of backdoor models w.r.f. original poisoned training data (both data distributions), the loss surface will be smoother. However, looking from only clean distribution point of view the loss surface is sharper as we have described the paper. Backdoor purification implics that the model will only be sensitive clean data distribution and completely ignore any type of backdoor manipulations. Since the model's 's behavior w.t.t. clean distribution is of our particular interest, we perform the smoothness
Phttps://gi thub. com tttbb..cmm/krrraannnooo//////g
14

---

Under review as a conference paper at ICLR 2023
analysis w.z.t clean distribution in all cases. If we put it another way, making the loss surface smooth w.t.. clean distribution ensures that the model will automatically forget the poison distribution. i.e. backdoor purification. Another minor reason is that, we able to distinguish the behavior of benign and backdoor models because we consider clean distribution. This is the only common distribution between these models, and one has to perform smoothness analysis w.r.f. common distribution to distinguish them.
C.2 WHY SMOOTHNESS THE KEY TO REMOVING THE BACKDOOR?
One key observation from the smoothness study is that: there exists key difference between weight- loss surface smoothness (estimated by loss hessian) of a backdoor and a benign model w.r.t. clean distribution-the weight-loss surface of a backdoor model is less smooth compared to benign model. To further elaborate, let us consider feeding a clean sample to a backdoor model. By definition, it will predict the correct ground truth label. Now, consider feeding sample with backdoor trigger on it. The model will predict the adversary-set target label implying significant changes in prediction distribution. This significant change can be explained by the surface smoothness. In order to accommodate this significant change in prediction, the model must adjust itself accordingly. Such adjustment leads non-smoothness in the weight-loss surface. A non-smooth surface causes significant changes in loss gradient for specific inputs. In our case, these specific inputs are backdoor- triggered samples. As the magnitude of a trigger is usually very small compared to the total input magnitude, the model has to experience quite significant change in its weight space to cause large loss changes. We characterize this change in terms of smoothness. As for backdoor removal, we claim that making the non-smooth weight loss surface smoother removes the backdoor behavior. Based on the above discussion, a smoother surface should not cause a large change in loss or model predictions corresponding to backdoor related perturbations or triggers. In summary, for model to show certain backdoor behavior, there are some specific changes that take place in the weight space. In this work, we try to explain these changes in terms of weight-loss surface smoothness. Our intuition is well supported by our comprehensive empirical evaluations.
C.3 WIYY USE FISHER INFORMATION MATRIX FOR ACIIEVING SMOOTIINESS?
In Fisher Information Matrix (FIM) based optimization, the natural gradient is defined as F (ref. Eq. 5). From the perspective of information geometry, natural gradient defines the direction in parameter space which gives largest change in objective per unit of change in model (plyl,, 0)). Per unit of change in model is measured by KL-divergence. Note that KL-divergence is well connected with FIM as it can be used as a local quadrature approximation of KL-divergence of model change. Eqn. 2 suggests that one requires the knowledge of the original parameter (0) space to estimate Therefore, FIM can be thought of as mechanism to translate between the geometry of the model (p(ylz, 0) and the current parameters (0) of the model. The way natural gradient defined the direction in parameter space is contrastive to the stochastic gradient. Stochastic gradient defines the direction in parameter space for largest change in objective per unit of change in parameter (0) measured by Eucludian distance. That is, the gradient direction is solely calculated based on the changes of parameters, without any knowledge of model geometry.
As FIM-based optimization minimizes the changes in model. the model itself cannot significantly change at each iteration. So, the overall optimization process goes through comparatively smoother transition and finally reaches smoother minima in comparison with SGD-based optimization
C.4 WHY DOES SAM UNDERPERFORMS?
One-layer optimization becomes a shallow network optimization problem for which there can exist many bad local minima. For such an optimization problem, typically first-order optimizers perform poorly mainly for the unawareness of loss surface curvature geometry. In the case of SAM. it uses SGD as the optimizer. Informally, the working principle of SAM is: at each iteration, SAM tries to minimize the maximum loss within a certain area in the loss weight space. Note that the formulation of finding the maximum loss in certain area is based on 'Fuclidean distance" metric which does not capture the curvature information of the plane. Although SAM performs better than vanilla SGD
We refer the readers to (Jia & Su, 2020) for more discussion FlM-hased smoothness analysis.
15

---

Under review as a conference paper at ICLR 2023
in deep network in terms of smoother optimization point, SAM's performance in shallow network (our case) is almost similar to vanilla SGD. Two potential reasons behind this poor performance are (i) using predefined local area to search for maximum loss, and (ii) using 'Euclidean distance' metric instead of geometric distance metric. In contrast, NGD with curvature geometry aware Fisher Information Matrix can successfully avoid such bad minima and optimizes global minima.
D EXPERIMENTAL DETAILS
For creating backdoor models with CIFARIO (Krizhevsky et al., 2009), train PreActResNet (He et al., 2016) model using an SGD optimizer with an initial learning rate of 0.01, learning rate decay of 1/100 epochs for 250 epochs. We also use weight decay of 5e -4 with momentum of 0.9. We use a longer backdoor training to ensure a satisfactory attack success rate. We use a batch size of 128. For GTSRB (Stallkamp et al., 2011). we train WideResNet-16-1 (Zagoruyko & Komodakis, 2016) model for 200 epochs with a learning rate of 0.01 and momentum of 0.9. We also regularize the weights with a weight-decay of 5e We rescale each training image 32 x 32 before feeding them to the model. The training batch size is 128 and an SGD optimizer is used for all training. We further created backdoor models trained on the Viny-ImageNer and ImageNet datasets. For Tiny-ImageNet, we train the model for 150 epochs with a learning rate of 0.005, decay rate of 0/60 epochs, and weight decay of le-4, For ImageNet, we train the model for 200 epochs with a learning rate of 0.02 with decay rate of 0.1/75 epochs. We also employ 0.9 and le-4 for momentum and weight decay, respectively. The details of these four datasets are presented in Table 7.
D.1 DETAILS OF ATTACKS
We use 11 different attacks for CIFARIO. Each of them differs from each other in terms of either label mapping type or trigger properties. For label poisoning attack, we use fixed poison rate of 10%. However, we need to increase this rate to 80% for CLB and SIG. For Blend and SIG attacks, we use a image-trigger mixup ratio of 0.2. WaNet adopts universal wrapping augmentation as the backdoor trigger. Note that WaNet can be considered as an non-additive attack since it works like augmentation technique with direct information insertion or addition like Badnets or TrojanNet. ISSBA adds specific trigger to each input that is of low magnitude and imperceptible. Both of these methods are capable of evading some existing defenses. For BPPA attack, follow the Py Torch implementation'. For Feature attack, we create backdoor model based on this implementation Apart from clean-label attacks, we use a poison rate of 10% for creating backdoor attacks. The details of these attacks are presented in Table 8. In addition to theses attacks, we also consider 'AllAIl' attacks (Troj-all, Dyn-all) where have more than one target label. To implement this attack, we change the given label to the target label i + 1. For class 9, the target label is O.
Table 7: Detailed information of the datasets and DNN architectures used in our experiments.
Datasct
Classes
Image Size
Training Samples
Test Samples
Architecture
CIFAR-10
10
32 32
50,000
10,000
PreActResNet118
GTSRB
43
32 32
39,252
12,630
WideResNet-166.
ImageNet
200
64 x 64
100,000
10,000
ResNer34
ImageNet
1000
224 x 224
1.28M
100,000
ResNet50
D.2 NGF AND OTHER OPTIMIZER IMPLEMENTATION DETAILS
To implement our proposed algorithm, we freeze the CNN backbone of the model and only fine-tune tune the linear classification layer parameters. We perform the fine-tuning for 100 epochs with learning rate of 0.01, weight decay of 1e-4, momentum of 0.9, and a batch size of 128. For studies
'Deep network consists of degenerate local minima and manifold of connect global minima (Liu et al., 2022) implying that, in deep network, there is such bad local minima, unlike to shallow network, that could affect the performance of SAM.
'https: / / thub. com Soom/RJ-ae System So Soofware-and- ttwt 'htpps: / / thub. com /Megmm/gmmmmm /DSST
16

---

Under review as a conference paper at ICLR 2023
Table 8: Details of different backdoor attacks we have defended against.
Attacks
Trigger Type
Label
Description
Poison Rate
Target Label
Mapping
Checker Board
Label
Triggers placed at bottom
Badnets (Gi al,, 2019)
3 x 3
Poison
left corner of images
Checker Board
Clean
use PGD-based
CL.B (Turner et al,, 2018)
80%
0
3 3
Label
adversarial perturbations
Sinusoidal Signal
Clean
Use Mixup for adding the sinusoidal trigeer to whole image
(Barni al., 2019)
Label
Dynamic (Nguyen & Trun, 2020)
Optimization
Label
Generate image dependent triggers
10%
0
Poison
Label
Watermarks static for all
Trojan (Liu ol,, 2017)
Watermarks
Poison
poisoned samples
Random Pixels
Label
Each pixel of the trigger is sampled from uniform distribution [0,255]
Blend (Chen et al., 2017)
10%
0
Poison
Mixer
Label
Mixing existing benign feutures of two/more classes
CBA (Lin et ul., 2020)
10%
Constructor
Poison
Style Generator
Label
Use controlled detoxification to manipulate deep features
FBA (Cheng al., 2021)
10%
Poison
Quantization
Label
Image quantization contrustive adversarial learning hased
BPPA (Wang al., 2022)
10%
Trigger
Poison
Table 9: Comparison of NGF with another state-of-the-ar defense CLP (Zheng al., 2022). Even though CLP achieves satisfactory removal performance for some attacks, the clean test accuracy drops significantly for some attacks (Blend, TrojanNet, CLB). We consider CIFARIO dataset for this comparison.
Methods
Badncts
Blond
Troj-onc
SIG
CLB
Dyn-One
Dyn-All
BPPA
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
92.96
92.78
100
CL.P
2.58
90.90
0.81
81.17
1.81
56.08
1.46
53.03
13.61
90.38
13.84
90.84
14.78
89.72
11.39
90.21
NGP
1.86
88.32
0.38
91.17
2.64
84.21
0.12
84.16
1.04
88.37
1.17
90.97
1.61
90.19
7.14
91.84
with different optimizers (Adam, RMSProp, etc.), we use similar training settings as NGF. For sharpness- minimization, we restrict the search region for the SGD optimizer. We follow Pytorch implementation described here". We use batch size of 128 and a learning rate of 0.01 for SAM.
D.3 DETAILS OF OTHER DEFENSES
For experimental results with ANP (Wu & Wang, 2021), we follow the source code implementation After creating each the above mentioned attacks, we apply adversarial neural pruning on the backdoor model for 500 epochs with a learning rate of 0.02. We use the default settings for all attacks. For vanilla FT, we perform simple DNN fine-tuning with a learning rate of 0.01 for 125 epochs. We higher number of epochs for FT due its poor clean test performance. The clean validation size is 1% for both of these methods. For I-BAU (Zeng et al., 2021), we follow their Py PoTcch Implementation and purify the model for 10 epochs. We use 5% validation data for I-BAU. For AWM (Chai & Chen, 2022), we train the model for 100 epochs and use Adam optimizer with a learning rate of 0.01 and wight decay of 0.001. We use the default typer-parameter setting as described in their work a = 0.9, 8 = 0.1. Imageem
108, 77 1000. Above settings is for CIFAR 10 and GTSRB only. For keep most of the training settings similar except reducing the number of epochs significantly. We also increase the validation size to 5% for vanilla FT, ANP, and AWM. For I-BAU, we use a higher valiadtion size of 10%. For purification, we apply ANP and AWM for epochs, I-BAU for 5 epochs and Vanilla FT for 25 epochs. For ImageNet, we use a 3% validation size for all defenses (except for I-BAU, we use 5% validation data) and use different number purification
Phttps: http://:s//gitth..coooo
http::////gghho.b.co///////oog// /Nw__bcccbbbboorrrraarorooo Phttps: //gi thub. http:://gggtttt.ooooooo/e/ee/g/
17

---

Under review as a conference paper at ICLR 2023
Table 10:
Correction rate (%) different defense techniques. We define correction rate (CR) the percentage of poisonous samples correctly classified to their original classes. The higher the CR, the better is that method. We use CIFARIO dataset for these evaluations.
Method
Badnets
Trojan
CLB
SIG
Defense
Vanilla FT
85.74
80.52
84.72
43.35
ANP
85.56
80.69
82.04
45.64
NGF (Ours)
86.42
80.85
85.63
45.18
Table 11: Average run time for different defense methods. We consider CIFARIO dataset all attacks to calculate the average runtime. We do not show the runtime of CLP as it severely underperforms compared to other defenses. An NVIDIA RTX 3090 GPU was used for all computations.
Method
Vanilla FT
ANP
I-BAU
AWM
NGP (Ours)
NGF wo Regularizer (Ours)
Runtime (sec)
78.1
201.5
52.7
90.2
38.3
37.8
epochs for different methods. We apply I-BAU for 2 epochs. On the other hand, we train the model for 3 epochs for ANP, AWM and vanilla FT.
E COMPARISON WITH ADDITIONAL DEFENSE
In Table 9, we show the comparison of NGF with recently proposed defense technique based on channel Lipschitzness Pruning (CLP) (Zheng al., 2022) that works without any data. We follow the Github link Based on the trigger-activated change channel activation, CLP prunes channel. One disadvantage of pruning based method that in case of challenging scenarios, e.g.,, strong attacks, large datasets and models etc., it prunes neurons abruptly. This creates high possibility of pruning neurons sensible to clean data distribution. In turn, the clean test accuracy may decrease significantly in some scenario. As shown in Table 9, clean accuracies (ACCs) for Blend, Trojan and CLB attacks are much lower compared to NGE. Even though CLP performs reasonably well at removing backdoor, NGF still outperforms in that area.
MORE ON SMOOTHNESS ANALYSIS
For smoothness analysis, we follow the PyHessian implementation" !I and modify it according to our needs. We single batch with size 200 calculate the loss Hessian for all attacks with CIFAR10 and GTSRB datasets. We conduct further smoothness analysis for ImageNet dataset and different architectures. In Fig. 5, we show the Eigen density plots for different 5 different attacks. We used A40 GPUs with 96GB system memory. However, it was not enough to calculate the loss hessian if we consider all 1000 classes of ImageNet. Due GPU memory constraint, we consider ImageNet subset with 12 classes. We train a ResNet34 architecture with 5 different attacks. To calculate the loss hessian, we use a batch size of 50. Density plots before and after purification further confirms our proposed hypothesis. To test our hypothesis for larger architectures, we consider 5 different architectures for CIFARIO, i.e., VGG19 (Simonyan & Zisserman, 2014), MobileNetV2 (Sandler et al., 2018), 121 (Huang et al., 2017), GoogleNet (Szegedy et al., 2014), Inception-V3 (Szegedy et al., 2016). Each of the architectures is deeper compared to the ResNet18 architecture we consider for CIFAR10. Due their large size, showing the effectiveness of NGF in case of these architecture will strengthen our claim one layer NGF based finettnnin is enough for backdoor purification. In Fig. 6, show the performance of NGF when backdoor models are created using these architectures. Our proposed one layer fine-tuning successfully removes the backdoor in all of these scenarios.
1 - channeeedddcccaaeeeeeeee "https:/ //gi thub. com ami rghol ami /PyHessian
Phttps://gf
pscht tzness- www----aasedddeeeeeee
18

---

Under review as a conference paper at ICLR 2023
Table 12: Performance of NGF while fine-tuning layers DNN. The results shown here are for CIFARIO dataset.
Methools
ASR
ACC
ASR
ACC
Trojan ASR
ACC
ASR
Dynamic
ACC
ASR
ACC
ASR
ACC
ASR
ACC
Runtime (Sees.)
Initial
100
92.96
100
91.11
100
89.57
100
92.52
100
92.78
100
88.64
93.20
90.17
Namili-FT Layers)
4.87
85.92
4.77
87
3.78
82.18
4.73
88.61
1.83
87.41
1.01
81,92
27.80
83.79
78.1
layer)
88.32
90.97
(All layens)
38.65
2,05
90:42
0.18
85.12
19.86
173.2
Table 13: Purification performance after randomly re-initializing the last layer. Even after re-initialization, the purification task is similar before, ie., proper fine-tuning. Without proper fine-tuning, the backdoor behavior will be present after purification. contrast to SGD, NGF highly successful even after such re- re-initializatin. CIFARIO dataset is considered here.
BadNets ASR
Trojan
Blend
CLB
SIG
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
No Defense
100
92.96
100
89.57
100
94,11
100
92.78
100
88.64
SGD
93.51
89.35
89.63
87.55
71.22
92.21
88.01
90.01
76.83
86.12
NGF
3.34
89.65
18.78
84.21
0.33
86.39
4.45
82.50
1.25
83.95
G LABEL CORRECTION RATE OF DIFFERENT DEFENSES
In standard removal measurement, it is sufficient for backdoored images be classified non- target class. As we calculate ASR after removal, our evaluation follows standard measurement. We define the correction rate (CR) the percentage of poisonous samples correctly classified their original classes. We define the method with the highest value of CR as the best performing or SOTA method. We use the CIFAR10 dataset and different attacks for demonstration. It can be observed from Table 10 that our method obtains SOTA correction performance for most of these attacks.
H RUNTIME ANALYSIS OF OTHER DEFENSES
In Table 11, we show the average runtime of other defenses. It can be observed that ANP is almost 6x slower than NGE. Other defenses, NAD and MCR are also much slower than NGF. NAD uses transfer lcarning based distillation using teacher-student framework. However, the complexity of this method results in computational overhead. Instead, NGF revisits much simpler fine-tuning of
approach from lae-l optimization point of view. Our simple and effective method leads the fastest purification. We take the average of all run times against 11 attacks on CIFAR1O. Note only update
that, for each epoch in NGF, we have feed-forward all validation data. However, the parameters of last layer through poopppropagttion The reason behind this is that we use different data augmentations while fine-tuning. This does not allow save the CNN features one time and re-use it for upcoming all epochs.
I FINE-TUNING ALL LAYERS
We have considered fine-tuning all layers fusing NGF and SGD. Note that vanilla FT does fine-tune all layers. We report the performance of NGF for all layers in Table 12. While fine-tuning all layers seems to improve the performance, it takes almost 6x more computational time than NGF on last layer. We perform NGF with the regularizer here.
Table 14: Purification performance for various validation data size. NGF performs well even with very few validation data, e.g., 50 data points. All results are for CIFARIO and Badnets attack.
Validation size
50
100
250
350
500
Method
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
Defense
100
92.96
100
92.96
100
92.96
100
92.96
100
92.96
ANP
13.66
83.99
8.35
84.47
5.72
84,70
3.78
85.26
2.84
85,96
AWM
8.51
83.63
7.38
83.71
5.16
84.52
5.14
85.80
4.34
86.
NGF (Ours)
6.91
86.82
4.74
86.90
4.61
87.08
2.45
87.74
1.86
88.32
19

---

Under review as a conference paper at ICLR 2023
Table 15: Illustration of purification performance for All2All attack using CIFARIO dataset, where uniformly distribute the target lahels to all availahle classes. NGF shows hetter robustness and achieves higher clean accuracies for 3 attacks: Badnets, Blend, BPPA with 10% poison-rate.
BadNets-All
Blend-All ASR
BPPA-AIl
Method
ASR
ACC
ACC
ASR
ACC
Defense
100
88.34
100
88.67
99.60
92.51
FT
2.78
83.19
2.83
80.13
10.97
89.76
NAD
4.58
81.34
6.76
81.13
20.19
87.77
ANP
3.13
82.19
4.56
82.88
9.87
89.91
NGF (Ours)
1.93
84.29
1.44
83.79
6.10
90.56
Table 16: Performance of NGF against combined backdoor attack. We poison some portion of the training data using different attacks; Badnets, Blend, and Trojan. Each of these attacks have equal share in the poison data, All results are CIFARIO datasets containing different number of poisonous samples.
Poison
25%
35%
50%
Method
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
No Defense
100
88.
100
87.51
100
86.77
100
85.82
MCR
27.83
78.
31.09
77.42
36.21
75.63
40.08
72.91
ANP
6.51
79.93
78.06
NGF (Ours)
1.17
83.61
2.15
81.62
3.31
80.01
4.15
79.35
J EFFECT OF CLEAN VALIDATION DATA SIZE
We also present how the total number of clean validation data can impact the purification performance. In Table 14, we see the change in performance while gradually reducing the validation size from 1% 0. 1. We consider Badnets attack on CIFARIO dataset for this evaluation. Even with only 50 (0.1%) data points, NGF can successfully remove the backdoor by bringing down the attack success rate (ASR) to 6.91%. We also consider adversarial weight masking for this comparison. For both ANP and AWM, reducing the validation size has severe impact on the test accuracy (ACC).
K PURIFICATION USING LAST LAYER RE-INITIALIZATION
We also conduct studies on the behavioral difference of SGD and NGD while we re-initialize the last layer. Even though re-initialize the last layer, one still has to properly fine-tune the backdoor model to remove the backdoor. However, we see throughout our evaluations that SGD-based one- layer fine-tuning is not proper fine-tuning method and unable remove backdoor. In case of re-initialization, the shallow network optimization problem still stands as wells as the issue of bad local minima. Therefore, SGD shows similar behavior in this scenario too. Table 13 shows the performance of SGD and NGF in case one decides to re-initialize the layer than fine-tunes. As usual, NGF is able to reach smooth minima due its ability properly fine-tune the model.
L MORE ALL.AAL ATTACKS
Most of the defenses evaluate their methods on only All2One attack where we consider only one target label. However, there can be multiple target classes in practical attack scenario. We consider one such case: All2All attack where target classes are uniformly distributed among all available classes. In Table 15, we show the performance under such settings for 3 different attacks with poison rate of 10%. shows that AII2AII attack is more challenging to defend against as compared to All2One attack. However, the performance of NGF seems to be consistently better than other defenses for both of these attack variations. For reference, we achieve an ASR improvement of 12% over ANP while maintaining a lead in classification accuracy too.
M COMBINING DIFFERENT BACKDOOR ATTACKS
We also perform experiments with combined backdoor attacks. To create such attacks, we poison some portion of the training data using 3 different attacks; Badnets, Blend, and Trojan. Each of these
20

---

Under review as a conference paper at ICLR 2023
prvoction
46-
66
48-
-40
Poson 2uta Chznler
barget clss
22.8
40
(a) Cluster Structures Without Discriminator (Be- fore Purification)
(b) Cluster Structures Without Discriminator (After Purification)
CEARLO Talk can T:st endentee
B3
Cien des qutaa
compo
(c) Cluster Structures With Discriminator (Before Purification)
(d) Cluster Structures With Discriminator (After Purification).
Figure 3: t-SNE visualization of class features CIFARIO dataset with Badnets attack. visualization purpose only, we assign label "0" to clean data cluster from target class label "11" to poison data cluster. However, both of these clusters have same training label "0" during training. With discriminator 3c, the clusters are relatively closer and harder discriminate compared to without discriminator 3a. However, after fine-tuning using clean data, NGF remove the backdoor effect. Even though the clusters very close, classifier can discriminate the clusters as shown by 3b&3d. After purification, poison data are distributed among their original ground truth classes instead of the target class. To estimate these clusters after purification (3b& we take the output of classifier before softmax (embedding dim=10) and apply tSNE with 2 components.
attacks have an equal share in the poison data, As shown in Table we use different poison rates: 10% ~ 50%. NGF outperforms other baseline methods (MCR and ANP) by a satisfactory margin.
N DECISION HEATMAPS: HOW NGF REMOVES BACKDOOR?
While inserting the backdoor behavior, the model, especially the linear classification layer, memorizes the poison data distribution. By memorization, we mean memorizes the simpler trigger pattern. Whenever the model sees that pattern in the input, it prioritizes the trigger-specific feature instead of image-specific (clean part) feature and predicts the adversary-set target label. When we re-train or fine-tune the classifier with clean validation data, the classifier forgets the poison distribution as fine- tuning reinforces the dominance of clean features in model prediction. After fine-tuning, the model looks for image-specific features for prediction it has almost no memory of the trigger-specific features. We illustrate the decision heat- maps for clean, backdoor and purified model in Figure 4. We show the decision heatmaps for clean poison data. As clean model is only trained on clean data, not sensitive to the trigger. Our defense objective says that, a purified model should behave like a benign model, iee, the decision making process (for clean and poison data) should resemble clean
21

---

Under review as a conference paper at ICLR 2023
model. As for the poison data, NGF successfully removes the effect of the trigger. The purified model ignores the trigger while making decisions.
Clean Data
Poison Data
Clean Data
Poison Data
Clean Data
Poison Data
Clean Model
Backdoor Model
Purified Model
Figure 4: Decision heat-maps clean, backdoor and purificd models. Regions with more reddish color
more responsible towards a decision making. For each category, we show the heatmaps for clean and poison data. Trigger is al bottom left corner of each poison duta. Unlike backdoor model, clean model is insensitive to triggers in the poison sample. Wheres backdoor model causes the model make wrong decision based the trigger pattern. The purified model behaves like clean model does not at trigger while making decision. All heat heat-maps are generated for CIFARIO dataset attacked with BadNets. We choose this attack for better understanding of the context.
22

---

Under review as a conference paper at ICLR 2023
10
Amax 2771.9
ACC: 85.8
10
Amax 963.2
ACC: 81.3 81.3
Tr(H) 17015.6
ASR 100.0
Tr(H) 7711.9
ASR 33.8
Density (Log Scale)
Density (Log Scale)
10
10
10
10
10
-10"
-10
-10
-10"0 10"
10
10
10
-10
-10"
-10"
-10
10"
10
10
10
Eigenvlaue
Bigenvlane
(a) Badnets Attack
(b) Badnets Purification
10
Amax 3531.7
ACC: 85.0
10
Amax 448.11 Tr(H) 1193.5
ACC: 81.1
Tr(H) 12155.2
ASR 99.5
ASR 2.3
Density (Log Scale)
Density (Log Scale)
10-
10
10
10a
10
-10"
-10 -10"
-10% 0 10
10
10
10
-10
-10'
Ó
10
10
Eigenvlaue
Eigenvlaue
(c) Clean-Labbe (CI.B) Attack
(d) Clean-Tabbe (CLB) Purification
100
Amax 2649.8
ACC: 85.5
10
Amax: 796.7
ACC 80.7
Tr(H): 19228.8
ASR 99.1
Tr(H): :2816.2
ASR 2.7
Density (Log Scale)
Density (Log Scale)
10
10
10
10
10a
10
10
10
10'
0
10
10
10
10
10
10a
10'
10"
10
10
10
10
Eigenvlaue
Eigenvlaue
(e) SIG Attack
(I) SIG Purification
10
Amax :3196.1
:CC: 85.1
10a
Amax: : 768.1
ACC 81.9
Tr(H) 23445.2
ASR 100.0
Tr(H) 1347.44
ASR 3.5
Density (Log Scale)
Density (Log Scale)
1
10
10
10
10
10
-10
-10º
-10"
-10
10
10
10
10
-10²
-10²
-10"
-100
O 10
10
10
10
Bigenvlaue
Bigenvlaue
(g) Blend Attack
(h) Blend Purification
Figure 5: Smoothness analysis for ImageNet Subset (first 12 classes), ResNet34 architecture is trained on the subset. For GPU memory constraint, we consider only first classes while calculating the loss Hessian. Eigen Density plots of backdoor models (before and after purification) are shown here.
23

---

Under review as a conference paper at ICLR 2023
10
Amay: 75993
Amer: :20.1 T7(H) :28.9
:
Tr(H):54244
ASR :100.0 D
ASR 1.2
Densiry (Log Scale)
Denaiy (Log Scale)
1012
10
10
10
10"
10
-10
-10"
10"
-10
10
Eigenvlaue
Eigenvlaue
Backdoor Attack (VGG19)
(b) Backdoor Purification (VGG19)
Density (Log Scale)
109
:6906)
Amax 707.6 6
:600
:56
Density (Log Scale)
10a
Amax:
Tr(H) 28.8
ACC: :66.0
ASR 1.6
10"2.
10-4
10-6
10-8
10
-00
-10
-10 10
10a
102
10a
10
-101
-00
10
10a
Eigenvlaue
Eigenvlaue
(c) Backdoor Attack (MobileNerV22)
(d) Backdoor Purification (MobileNctV2)
10
Amer:
Amar :1.5 Tr(H): 12.4
:CC:91.8
Tr(H): 1479.4
ASR: :55::000 100.0
ASR: 0.1
Density (Log Scale)
16f
10
Densiy (Log
10
10
10
10
10
-10
-10"
10
10
10
Eägenrllaue
Ligenslans
(e) Backdoor Attack (GoogleNet)
(1) Backdoor Purification (GoogleNet)
Density (Log Scale)
10
Amwr: 195.6
Tr(H) :1598.3
:CC:90.9
100.0
Demsiyy (Log Scale)
16f
Amar :4.5
Tr(H)
:CC:87.9
10"2
10
10
10
10a
10"
10
-10
-160 0
10
10
-10"
10
Eigeavlauue
Eigenvlanc
(g) Backdoor Attack (Inception V3)
(h) Backdoor Purification (InceptionV3)
10'
Amax: Tr(H) 12836.6
Amer: 35,8 Tr(H):37.
:CC:86.7 :SA
ASR: 100.0 0
Density (Log Scale)
10*
10
Density (Log
10
10
10
10-
-10
-10
-10
-10"
10
10
10
10
-10'
-10
10
10'
Eigenvlaue
Eigenvlaue
(i) Backdoor Attack (DenseNet121)
(i) Backdoor Purification (DenseNet121)
Figure 6: Smoothness Analysis for different architectures. architectures, we consider badnets attack on CIFAR10.
24


Reviews:
[{"review_id": "RrnNPYr9s6e", "reviewer": "Reviewer_uefD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed NGF, which uses clean data to remove the backdoor from a model.  The proposed method works by fine-tuning the last layer of the model such that the false local minima are smoothed out and the effect of removing the backdoor from the model is achieved. Finally, this paper gives very extensive and comprehensive experiments to demonstrate the performance of their proposed model\n", "summary_of_the_review": "1. There lacks sufficient intuition on why smoothness is the key to remove the backdoor.  Essentially smoothness measures the changes in gradients wrt to changes in inputs. Then the authors’ conjecture is that triggers in input could lead to large changes in gradients. But why it must be a backdoor? It seems to me that the lipshitz pruning method (CLP, Zheng et al. 2022) is telling a more convincing story that if triggers in input could lead to large changes in loss, it might be backdoored.\n \n\n2. For runtime comparison, the author should compare with other baselines as well as NGF without regularization term. Also for ablation study, I would like to see the results for fine-tuning on all layers. \n\n3. The proposed method requires to have clean training data in order to function properly. The authors did not test the situation when the number of clean training data is not sufficient. To be comprehensive, I would suggest the authors to include the number of training samples as an ablation study.\n\n4. The performance reported for data-free backdoor removal by Zheng et al. 2022 seems a bit strange as the original paper actually gives quite a good performance on Blend, TrojanNet. Did the author tune the hyperparameters?\n\n5. There is also some recent work on backdoor removal:\n\n    \"One-shot Neural Backdoor Erasing via Adversarial Weight Masking.\" NeurIPS 2022.\n\n    The authors may want to comment on/compare with the above work.\n\n", "strength_and_weaknesses": "Strength:\n1. The motivation of this paper is clear, and the experiments are effective in proving the proposition of removing Backdoor from the model by smoothing the loss surface of the model.\n2. This paper demonstrates the effectiveness of the proposed approach through extensive and comprehensive experiments. The experimental results under various Backdoor Attack methods and models are included, and the effectiveness and superiority of the proposed approach is demonstrated by comparing various mainstream defense methods.\n \nWeaknesses:\n1. Not much intuition and discussion on how the proposed method will help eliminate the non-smoothness and why it is better than other solutions. \n2. lack comparison with new defense baselines \n3. technical contribution is not much aside from a regularization term in loss and the use of NGD\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8QX7hq5VBd", "reviewer": "Reviewer_6Zjq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new method to recover the model after an attack by finetuning only one layer. Moreover, it has assumed that the backdoor model tends to be trapped in the local minimum. Therefore, a new purification technique named NGF is invented based on the loss surface curvature matrix, i.e., Fisher Information Matrix. The proposed method is evaluated on multiple benchmarks across 11 backdoor attacks with promising results in ASR.", "summary_of_the_review": "Based on the comments above, this submission is around the borderline. I will consider changing the score after discussion.", "strength_and_weaknesses": "Pros:\n+ It is interesting to relate the model attack with optimization. The basic assumption of trapping in local minimum is convincing. \n\n+ The proposed NGF is inspired by the Fisher Information Matrix, which has solid theoretical foundations to support it.\n\n+ The entire manuscript is well written, with clear logic to follow.\n\n+ There are solid and comprehensive experimental analyses throughout the paper. The proposed method has achieved competitive results in ASR over many benchmarks against multiple backdoor attacks.\n\n+ The proposed method only needs one-layer finetuning which is efficient.\n\nCons:\n- The motivation of NGF does not fully convince me. The authors are suggested to explain more of geometric awareness in Fisher Information Matrix and why it is helpful for smoother optimization. Is NGF applicable for general optimization where the smoother surface is supposed to be helpful for many tasks?\n\n- The experimental results on ACC seem not good. Is that true? What is the reason for this, and how to solve it?\n\n- The baseline methods are not very recent (ANP in 2021). Since this is for ICLR 2023, is it any baseline in 2022?\n\n\nQuestions/Other Comments:\n\n1. Please consider changing the color of the references. The black references are hard to distinguish, and they may interrupt reading.\n\n2. Could authors explain more about ASR and ACC? How to compute ASR and ACC?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0OW3QXnf0Y", "reviewer": "Reviewer_1UnP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the parametric landscape in terms of the loss on clean data. Inspired by the phenomenon in the landscape, the authors propose to use natural gradient fine-tuning only to the last classifier layer for defense and also introduce clean data distribution-aware regularizer to keep the natural accuracy. Empirical results demonstrate the effectiveness of the proposed method.\n", "summary_of_the_review": "This paper introduces an interesting side effect caused by poisoned data and proposes a simple but promising method to defend against backdoors attacks. However, there are still some concerns about adaptive attacks and the correctness of the motivation. If the authors could address my concern, I will reconsider my score.", "strength_and_weaknesses": "**Strength**\n1. The paper demonstrates a side effect of backdoor attacks. When calculated only on benign data, the loss landscape of the minimum point becomes sharper. This might inspire further studies.\n2. The proposed method focuses on fine-tuning of last linear layer, which influences fewer parameters and consumes less computation. The empirical results also show notable improvements compared to previous SOTA methods, which verifies the effectiveness of the proposed method.\n3. Overall, the paper is easy to follow.\n\n**Weakness**\n1. What does $p$ in $\\mathcal{L}_p$ mean? Considering the author only provides an approximation of $\\mathcal{L}_p$, \nwhat is the exact formulation?\n\n2. In my opinion,  fine-tuning only to the linear layer has very limited ability, which makes me worried the performance. Although the authors show that the triggered data (images with trigger pattern) are not classified as the target class after defense (i.e., low ASR),  I am curious whether these triggered test data are classified into their ground-truth label. \n\n3. I think the success of the proposed method relies on the separation between (1) the cluster of benign data from the target class and (2) the cluster of poisoned data. Since they are separated, we can use a linear classifier to distinguish them.\nHowever, it might be easy to propose an adaptive attack, i.e., If the adversary is aware of the presence of the proposed defense, he can conduct an adaptive attack during backdoor attacks to escape. Specifically, he can introduce an extra linear discriminator on the top of the backbone in parallel with the classifier. The discriminator tries to predict whether the sample from the target class is benign or poisoned, while the classifier still predicts which class the sample belongs to. During training, he maximizes the discrimination loss while minimizing the classification loss. \nAfter training, since the discriminator cannot predict whether the sample is benign or poisoned anymore, the cluster of benign data from the target class is heavily mixed with the cluster of poisoned data. At this time, does the proposed method still works?\n\n4. Why does SAM fails in defense? The proposed method is inspired by the sharpness of the landscape caused by the additive poisoned data during training, while SAM is the most natural choice to recover the flatness. If SAM fails, does the success of the proposed method come from something else instead of the flatness?\n\n5. The natural gradient descent (NGD) contributes the most improvements, and the proposed regularization only introduces little improvements in natural accuracy (see Table 5). Considering NGD has been widely applied, the technical novelty in this paper is limited.\n\nMinor:\n- The $\\theta$ should be $\\theta = [ W_{0, 1}, W_{1,2}, W_{L-1, L} ]$ above Equation (3), i.e., starting from $0$. Otherwise, there are only L-1 layers in total.\n\n- In Equation (1) and Equation (3), the comma should follow the formula in the same line, rather than appearing in the front of the following sentence.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject