Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness
Authors: Pu Zhao
Pin-Yu Chen
Payel Das
Karthikeyan Natesan Ramamurthy
Xue Lin
Venue:  (2020)
Keywords: mode connectivity
adversarial robustness
backdoor attack
error-injection attack
evasion attacks
loss landscapes

Abstract: Mode connectivity provides novel geometric insights on analyzing loss landscapes and enables building high-accuracy pathways between well-trained neural networks. In this work, we propose to employ mode connectivity in loss landscapes to study the adversarial robustness of deep neural networks, and provide novel methods for improving this robustness.  Our experiments cover various types of adversarial attacks applied to different network architectures and datasets. When network models are tampered with backdoor or error-injection attacks, our results demonstrate that the path connection learned using limited amount of bonafide data can effectively mitigate adversarial effects while maintaining the original accuracy on clean data. Therefore, mode connectivity provides users with the power to repair backdoored or error-injected models.  We also use mode connectivity to investigate the loss landscapes of regular and robust models against evasion attacks. Experiments show that there exists a barrier in adversarial robustness loss on the path connecting regular and adversarially-trained models.  A high correlation is observed between the adversarial robustness loss and the largest eigenvalue of the input Hessian matrix, for which theoretical justifications are provided.  Our results suggest that mode connectivity offers a holistic tool and practical means for evaluating and improving adversarial robustness.

Published as a conference paper at ICLR 2020
BRIDGING MODE CONNECTIVITY IN LOss LANDSCAPES AND ADVERSARIAL ROBUSTNESS
Pu Zhao', Pin-Yu Chen², Payel Das Karthikeyan Natesan Ramamurthy?, Xue Lin' Northeastern University, Boston, MA 02115 2IBM Research, Yorktown Heights, NY 10598 zhao.pu @ husky.neu.ede edu, pin-yu. @ ibm.com, daspa@ us.ibm.com, knatesa us.ibm.com, xue. lin @ northeastemmeedu
ABSTRACT
Mode connectivity provides novel geometric insights on analyzing loss landscapes and enables building high-accuracy pathways between well-trained neural net- works. In this work, we propose to employ mode connectivity in loss landscapes to study the adversarial robustness of deep neural networks, and provide novel methods for improving this robustness. Our experiments cover various types of adversarial attacks applied to different network architectures and datasets. When network models are tampered with backdoor or error-injection attacks, our results demonstrate that the path connection learned using limited amount of bonafide data can effectively mitigate adversarial effects while maintaining the original accuracy clean data. Therefore, mode connectivity provides users with the power to repair backdoored or error-injected models. We also use mode connectivity to investigate the loss landscapes of regular and robust models against evasion attacks. Experiments show that there exists a barrier in adversarial robustness loss on the path connecting regular and adversarially-traine observed between the adversarial robustness loss and the largest eigenvalue of the input Hessian matrix, for which theoretical justifications are provided. Our results suggest that mode connectivity offers a holistic tool and practical means for evaluating and improving adversarial robustness
models. A high correlation is
INTRODUCTION
Recent studies on mode connectivity show that two independently trained deep neural network (DNN) models with the same architecture and loss function can be connected on their loss landscape using
high-accuraccyllo-llor path characterized by simple curve (Garipov et al. 2018 Gotmare et al., 2018 Draxler et al., 20T8). 8). This insight on the loss landscape geometry provides us with easy access to large number of similar-performing models on the low-loss path between two given models, and this devise new model ensembling method. Another line of recent research reveals interesting geometric properties relating to adversarial robustness of DNNs (Fawzi ct al. 2017 2018 Wang et al. 2018b Yu et al.) 2018). An adversarial data or model is defined to be one that is close to a bonafide data or model in some space, but exhibits unwanted or malicious behavior. Motivated by these geometric perspectives, in this study, propose to employ mode connectivity to study and improve adversarial robustness of DNNs against different types of threats.
Garipov et al. (2018)
DNN can be possibly tampered by an adversary during different phases in its life cycle. For example, during the training phase, the training data can be corrupted with a designated trigger pattern associated with target label to implant a backdoor for trojan attack on DNNs (Gu al.) 2019 Liu et al. 2018). During the inference phase when a trained model is deployed for vosk-solving, prediction-evasive attacks are plausible (Biggio & Roli, 2018, Goodfellow et al., 2013 Zhao et al.
2018), even when the model internal details are unknown an attacker (Chen et al. 2017 Ilyas et al.
2018 Zhao et al., 2019a). this research, we will demonstrate that by using mode connectivity
in loss landscapes, we can repair backdoored or error-injected DNNs. We also show that mode
'The code is available at htttt:://tpthhhaae////
HM/mode1 ssnnisait tiatiioiioonn

---

Published as a conference paper at ICLR 2020
connectivity analysis reveals the existence of robustness loss barrier the path connecting regular and udversarially-tran trained models.
We motivate the novelty and benefit of using mode connectivity for mitigating training- ad- versarial threats through the following practical scenario: as training DNNs is both time- and essurce-consuming, has become common trend for users to leverage pre-trained models released in the public domain Users may then perform model fine-tuning or transfer learning with a small set of bonafide data that they have. However, publicly available pre-trained models may carry an unknown but significant risk of tampering by an adversary. It can also be challenging to detect this tampering. as in the case of a backdoor attack] since a backdoored model will behave like a regular model in the absence of the embedded trigger. Therefore, it is practically helpful to provide tools to users who wish to utilize pre-trained models while mitigating such adversarial threats. We show that our proposed method using mode connectivity with limited amount of bonafide data can repair backdoored or irror-injected DNNs, while greatly countering their adversarial effects.
Our main contributions are summarized as follows:
For backdoor and error-injection attacks, we show that the path trained using limited bonafide data connecting two tampered models can be used to repair and redeem the attacked models, thereby resulting in high-ace accuracy and low-risk models. The performance of mode connectivity is significantly better than several baselines including fine-tuning, training from scratch, pruning, and random weight perturbations. We also provide technical explanations for the effectiveness of our path connection method based model weight space exploration and similarity analysis of input gradients for clean and tampered data.
For evasion attacks, we use mode connectivity to study standard and adversarial- loss landscapes. We find that between regular and an adversarially-traincd model, training a path with standard loss reveals no barrier, whereas the robustness loss on the same path reveals a barrier. This insight provides a geometric interpretation of the "no free lunch" hypothesis in adversarial robustness (Tsipras et al.) 2019 Dohmatob] 2018 Bubeck et al., 2019). We also provide technical explanations for the high correlation observed between the robustness loss and the largest eigenvalue of the input Hessian matrix on the path.
Our experimental results on different DNN architectures (ResNet and VGG) and datasets (CIFAR- 10 and SVHN) corroborate the effectiveness of using mode connectivity in loss landscapes to understand and improve adversarial robustness. We also show that our path connection is resilient to the considered adaptive attacks that aware of our defense. To the best of our knowledge, this is the first work that proposes using mode connectivity approaches for adversarial robustness.
2 BACKGROUND AND RELATED WORK
2.1 MODE CONNECTIVITY IN LOSS LANDSCAPES
Let wi and W2 be trained by minimizing any user-specified loss 1(w), such as the cross-entropy loss. Moreover, let (f with E [0, 1] be a continuous piece-wise smooth parametric curve, with parameters 0, such that its two ends are de (0)
of model weights corresponding
neural networks independently
and dig (1) W2.
To find high-accuracy path between w and w2, it is proposed to find the parameters 0 that minimize the expectation over a uniform distribution on the curve (Garipov et al. 2018).
[())= (2) (()o(())
where (t) is the distribution for sampling the models on the path indexed by t.
Since qo (1) depends on in order to render the training of high-accuracy path connection more computationally tractable, (Garipov et al. 2018 Gotmare et al. 2018) proposed to instead use the following loss term,
(()) E (()v(t)))
2 For example, the Model Zoo project:
httpss//mode: 200.
See the recent call for proposals on Trojans in Al announced by TARPA: https: http://ww...arr.. .gvv
index. hhtpsepppppreeeeeh----eeeeee http:///rreaeaaaaoooroorrrrrr/rr
-baa
2

---

Published as a conference paper at ICLR 2020
where 1)) is the uniform distribution [0, 1].
The following functions are commonly used for characterizing the parametric curve function dh (t). Polygonal chain (Gomes et al., 2012). The two trained networks wi and w2 serve as the endpoints of the chain and the bends of the chain are parameterized by 0. For instance, the case of a chain with one bend is
(t = {
2 + (0.5 2 ((t- + (1-1)0)
< <.5 0.5 1
(3)
Bezier curve (Farouki 2012). A Bezier curve provides convenient parametrization of smoothness on the paths connecting endpoints. For instance, a quadratic Bezier curve with endpoints w1 and Wa is given by
(l) = (1 R) + 2t (1 t) Pwz. O
It is worth noting that, while current research on mode connectivity mainly focuses on generalization analysis (Garipov et al. [2088] Gotmare et al. 2018 Draxler et al., 2018 Wang et al. 201 8a) and
has found remarkable applications such as fast model ensembling (Garipov et al. 2018). our results show that its implication on adversarial robustness through the lens of loss landscape analysis is a promising, yet largely unexplored, research direction. Yuu et al. (2018) scratched the surface but focused on interpreting decision surface of input space and only considered evasion attacks.
2.2 BACKDOOR, EVASION, AND
ADVERSARIAL ATTACKS
Backdoor attack. Backdoor attack on DNNs is often accomplished by designing designated trigger pattern with a target label implanted to a subset of training data, which is a specific form of data poisoning (Biggio al., 2012. Shafahi et al. 2018 Jagielski al., 2018). A backdoored model trained on the corrupted data will output the target label for any data input with the trigger; and it will behave as a normal model when the trigger absent. For mitigating backdoor attacks, majority of research focuses on backdoor detection or filtering anomalous data samples from training data for re-training (Chen et al., 2018: Wang et al., 2019; Tran al., 2018), while our aim is repair backdoored models for models using mode connectivity and limited amount of bonafide data.
Evasion attack. Evasion attack a type of inference-phase adversarial threat that generates adversar- ial examples by mounting slight modification on benign data sample to manipulate model prediction (Biggio & Roli 2018). For image classification models, evasion attack can be accomplished by adding imperceptible noises to natural images and resulting in misclassification (Goodfellow et al. 2015. Carlini & Wagner 2017 Xu et al. 2018). Different from training-phe -aaiing attacks, evasion attack does not assume access to training data. Moreover, it can be executed when the model details are unknown to an adversary, via black-box or transfer attacks (Papernot et al. 2017: Chen et al.
2017 Zhao et al.] 2020).
Error-injection attack. Different from attacks modifying data inputs, error-injection attack injects enrors to model weights the inference phase and aims cause misclassification of certain input samples (Liu et al., 20a7:Zhao system, can be made plausible via laser beam (Barenghi et al. 2012) and row hammer (Van Der Veen et al.) (016) to change or flip the logic values of the corresponding bits and thus modifying the model parameters saved in memory.
Zhao et al. 2019b).. At the hardware level of a deployed machine learning
3 MAIN RESULTS
Here we report the experimental results, provide technical explanations, and elucidate the effectiveness of using mode connectivity for studying and enhancing adversarial robustness three representative themes: (i) backdoor attack; (ii) error-injection attack; and (iii) evasion attack. Our experiments were conducted on different network architectures (VGG and ResNet) and datasets (CIFAR-IO and SVHN). The details on experiment setups are given in Appendix A When connecting models, we use the cross entropy loss and the quadratic Bezier curve as described in (4). In what follows, we begin by illustrating the problem setups bridging mode connectivity and adversarial robustness, summarizing the results of aach-accuracy (low-loss) pathways between untampered models for reference, and then delving into detailed discussions. Depending on the context, will use the terms error rate
3

---

Published as a conference paper at ICLR 2020
Inference on training set
Inference on test set
Legend
15
80
15
80
cross entropy loss
cross entropy loss
25000 images training, loss
80
enror rate (%)
60
1000 training,
10
10
images training,
250 Images training. loss
40
40
50 images training, loss
5
5
2500 training,
20
images training,
500 images training, anor rate
250 images training, error rale
magas training,
0.5
0.5
Figure Loss and error rate on the path connecting two untampered VGG models trained on CIFAR-10. The path connection is trained using different settings as indicated by the curve colors. The results using SVHN and ResNet are given in Appendix [] The inference results on test set are evaluated using 5000 samples, which are separate from what are used for path connection.
Table 1: Error rate of backdoored models. The rate of clean/backdoored samples means handdard-est-errorfaae failure-rate, respectively. The results are evaluated on 5000 non-overlapping clean/triggered images selected from the test For reference, the test errors of clean images on untampered models are 12% for CIFAR-10 (VGG), and 4% for SVHN (ResNet), respectively.
Backdoor attacks
Single-target attack
All-targets attack
Dataset
CIFAR-IO (VGG)
SVHN (ResNet)
CIFAR-I0 (VGG) SVHN (ResNet)
Clean images
15%
5.4%
14.2%
6.1%
Model (t 0)
Triggered images
0.07%
0.22%
12.9%
8.3%
Clean images
13%
7.7%
19%
7.5%
Model 1)
Triggered images
2%
0.17%
13.6%
9.2%
and accuracy on clean/adversarial samples interchangeably. The error rate of adversarial samples is equivalent to their attack failure rate as well as 100% attack accuracy.
3.1 PROBLEM SETUP AND MODE CONNECTION BETWEEN UNTAMPERED MODELS
Problem setup for backdoor and error-injection attacks. We consider the practical scenario as motivated in Section where a user has two potentially tampered models and limited number of bonafide data at hand. The tampered models behave normally as untampered ones on non- bnagered/non-targeed
inputs so the user aims to fully exploit the model power while alleviating adversarial effects. The problem setup applies to the case of one tampered model, where bonafide data to train a fine-tuned model and then connect the given and the fine-tuned models.
the
Problem setup for evasion attack. For gaining deeper understanding evasion attacks, consider the scenario where a user has access to the entire training dataset and aims to study the behavior of the models on the path connecting two independently trained models in terms standard and robust loss landscapes, including model pairs selected from regular and adversarially-traine
models.
Regular path connection between untampered models. Figure shows the cross entropy loss and training/test error rate of models on the path connecting untampered models. The untampered models are independently trained using the entire training data. While prior results have demonstrated high-accuracy path connection using the entire training data (Garipov et al.) 2018 Gotmare et al. 2018: Draxler al. 2018), our path connection trained using different portion of the original test data corresponding to the scenario of limited amount of bonafide data. Notably, when connecting two DNNs, a small number of clean data capable of finding models with good performance. For example, path connection using merely 1000/2500 CIFAR- 10 samples only reduces the test accuracy (on other 5000 samples) of VGG16 models by at most 10%/5% when compared to the well-traines models (those at t and = 1). respectively. In addition, regardless of the data size used for path connection, the model having the worst performance is usually located around the point 0.5, as is geometrically the farthest model from the two end models the path.
4

---

Published as a conference paper at ICLR 2020
Single-target backdoor attack
All-targets backdoor attack
Legend
100
100
80
80
2500 images traning
1000 images tearingg clean enro
500 mages tranning clcan ener
erfor rate
250 maaes training, clean enter
59 inages valling, clean enror
-500 imaee trwning ettack error
11000 images traning,
entr mapes training, attack enor
mages trining,
-50 images taining, atack error
0.5
Figure 2: Error rate against backdoor attacks on the connection path for CIFAR-10 (VGG). The error respectively.
rate of clean/backdoored samples means the
3.2 MITIGATING AND REPAIRING BACKDOORED MODELS
Attack implementation. We follow the procedures in (Gu al. 2019) to implement backdoor attacks and obtain two backdoored models trained on the same poisoned training data. The trigger pattern is placed at the right-hottom of the poisoned images shown in Appendix C Specifically, 10% of the training data are poisoned by inserting the trigger and changing the original correct labels the target lsbel(s). Here we investigate two kinds of backdoor attacks: (a) single-target attack which sets the target label T to specific label (we choose T = class I); and (b) all-targets attack where the target label T is set to the original label plus 1 and then modulo 9, i.e., T = i + 1 [mod 9). Their performance on clean (untriggered) and triggered data samples are given in Table D and the prediction errors of triggered images relative to the true labels are given in Appendix D The backdoored models have similar performance on clean data as untampered models but will indeed misclassify majority of triggered samples. Comparing to single-target attack, all-targets attack is more difficult and has a higher attack failure rate, since the target labels vary with the original labels.
Evaluation and analysis. We train a path connecting the two backdoored models with limited amount of bonafide data. As shown in Figure 2 at both path endpoints (t {0. 1)) the two tampered models attain low error rates on clean data but are also extremely vulnerable to backdoor attacks (low error rate on backdoored samples means high attack success rate). Nonetheless, connection with limited bonafide data can effectively mitigate backdoor attacks and redeem model power. For instance, the models at 0.1 or 0.9 can simultaneously attain similar performance on clean data as the tampered models while greatly reducing the backdoor attack success rate from close 100% to nearly 0%. Moreover, most models on the path (e.g. when E 10.1, 0.9) exhibit high resilience to backdoor attacks, suggesting mode connection with limited amount of bonafide data can be an effective countermeasure. While having resilient models backdoor attacks on the path, also observe that the amount of bonafide data used for training path has larger impact on the performance of clean data. Path connection using fewer data samples will yield models with higher error rates clean data, which is similar to the results of path connection between untampered models discussed in Section 3.1 The advantages of redeeming model power using mode connectivity consistent when evaluated different network architectures and datasets (see Appendix F
find that path
Comparison with baselines. We compare the performance of mode connectivity against backdoor attacks with the following baseline methods: (i) fine-tuning backdoored models with bonafide data; (ii) training a new model of the same architecture from scratch with bonafide data; (iii) model weight pruning and then fine-tuning with bonafide data using (Li et al. 2017); and (iv) random Gaussian perturbation to the model weights leading to a noisy model. The results are summarized in Table Lableeland and their implementation details are given Appendix [ Evaluated on different network architectures and datasets, the path connection method consistently maintains superior accuracy on clean data while simultaneously attaining low attack accuracy over the baseline methods, which can be explained by the ability of finding high-accuracy paths between two models using mode connectivity. For CIFAR- 10 (VGG), even using as few as 50 bonafide samples for path connection, the subsequent model in Table still remains 63% clean accuracy while constraining backdoor accuracy to merely 2.5%. The best baseline method is fine-tuning, which has similar backdoor accuracy as path connection but attains lower clean accuracy (e.g. 17% worse than path connection when using 50 bonafide samples). For SVHN (ResNet), the clean accuracy of fine-tuning can be on par with path connection, but its backdoor accuracy is significantly higher than path connection. For example, when trained with 250
5

---

Published as a conference paper at ICLR 2020
Table 2: Performance against single-target backdoor attack. The clean/backdoor accuracy means
wewpaaadddtttt-cureeeyyeeeee suceess-ree, respectively. More results are given in Appendix
Method / Bonafide data size
2500
1000
500
250
50
Path connection 0.1)
88%
83%
80%
77%
63%
Fine-tune
84%
82%
78%
74%
46%
Clean Accuracy
Train from scratch
50%
39%
31%
30%
20%
Noisy model (t 0)
21%
21%
21%
21 %
%
Noisy model (t = 1)
24%
24%
24%
24%
24%
CIFAR-10 (VGG)
Prune
88%
85%
83%
82%
81%
Path connection 0.1) Finc-tunc
1.1%
0.8%
1.5%
3.3%
2.5%
1.5%
0.9%
0.5%
1.9%
2.8%
Backdoor Accuracy
Train from scratch
0.4%
0.7%
0.3%
3.2%
2.1%
Noisy model (t = 0)
97%
97%
97%
97%
97%
Noisy model (t 1)
91%
91%
91%
91%
91%
Prune
43%
49%
81%
79%
82%
Path connection 0.2)
96%
94%
93%
89%
82%
Fine-tune
96%
94%
91%
89%
76%
Clean Accuracy
Train from scratch
87%
75%
61%
34%
12%
Noisy model (t 0)
13%
13%
13%
13%
13%
Noisy model (t 1)
11%
11%
11%
11%
11%
SVHN (ResNet)
Prune
96%
95%
93%
91
89%
Path connection [t 0.2)
2.5%
3%
3.6%
4.3%
16%
Finc-tune
14%
7%
29%
6.3%
60%
Backdoor Accuracy
Train from scratch
3%
3.6%
5%
2.2%
3.9%
Noisy model (t = 0)
51%
51%
51%
51%
51%
Noisy model (f = 1)
42%
42%
42%
42%
42%
Prune
80%
90%
88%
92%
94%
samples, they have the same clean accuracy but the backdoor accuracy of fine-tuning is 58.7% higher than path connection. Training from scratch does not yield competitive results given limited amount of training data. Noisy models perturbed by adding pero-mean Gaussian noises to the two models are effective against backdoor attacks and may suffer from low clean accuracy. Pruning gives high clean accuracy but has very little effect on mitigating backdoor accuracy.
Extensions. Our proposal of using mode connectivity repair backdoor models can be extended to the case when only one tampered model is given. We propose to fine-tune the model using bonafide data and then connect the given model with the fine-tuned model. Similar to the aforementioned findings, path connection can remain good accuracy clean data while becoming resilient to backdoor attacks. We refer readers to Appendix G for more details. In addition, we obtain similar conclusions when the two backdoored models trained with different poisoned datasets.
Technical Explanations. To provide technical explanations for the effectiveness of our proposed path connection method in repairing backdoored models, space exploration and (ii) data similarity comparison, For (i), we generate 1000 noisy versions of a backdoored model via random Gaussian weight perturbations. We find that they suffer from low clean accuracy and high attack success rate, which suggests that good model with high-clean-accuracyy and www-atackkaacuuace
sets of analysis: (i) model weight
unlikely be found by chance. We also report the distinct difference between noisy models and models on the path in the weight space to validate the necessity of using our path connection for attack mitigation and model repairing. More details are given Appendix H For (ii), we run similarity analysis of the input gradients between the end (backdoored) models and models on the connection path for both clean data and triggered data. We find that the similarity of triggered data is much lower than that of clean data when the model is further away in the path from the end models, suggesting that our path connection method can neutralize the backdoor effect. More details are given in Appendix [ The advantage of our path connection method over fine-tuning demonstrates the importance of using the knowledge mode connectivity for model repairing.
Adaptive Attack. To justify the robustness of proposed path connection approach to adaptive attacks, we consider the advanced attack setting where the attacker knows path connection is used for defense but cannot compromise the bonafide data that private user. Furthermore, allow

---

Published as a conference paper at ICLR 2020
CIFAR- 10 (VGG)
SVHN (ResNet)
Legend
100
100
2500 images training, dean error
80
erfor rate (%)
80
1000 im images training, clean amrrr
500 images training, dean
250 images training, dlaan errer
50 images training, clean error
images vraining, ettack
1000 im images training. attack arror
500 images training, allack
250 images training, attack
50 images traiiin, stlack enror
0.5
Figure 3:
Error rate against error-injection attack on the connection path for CIFAR-10 (VGG). The respectively.
enror rate of clean/targeted samples means sataaandeeestterrrraataaackiilla
Table 3: Performance against error-injection attack. The clean/injection accuracy means standard- restaacuraacy/attac--ccce satc respectively. Path connection has the best clean accuracy and can completely remove injected errors (i.e. 0% attack accuracy). More results are given in Appendix
Clean Accuracy
Injection Accuracy
Dutaset
Method Bonatide data
2500
1000
5U0
250
50
2500
1000
500
250
50
Path connection (1 0.1)
92%
90%
90%
90%
88%
0%
0%
0%
0%
0%
Fine-tune
88%
87%
86%
84%
82%
0%
0%
0%
0%
0%
CIFAR-I0 10
Train from scratch
45%
37%
27%
25%
10%
0%
0%
0%
0%
0%
(VGG)
Noisy model (1 = 0)
14%
14%
14%
14%
14%
36%
36%
36%
36%
36%
Noisy model (t = 1)
12%
12%
12%
12%
12%
19%
19%
19%
19%
19%
Prune
91%
89%
88%
88%
88%
0%
0%
25%
25%
25%
Puth connection (t 0.1)
96%
94%
92%
91%
90%
0%
0%
0%
0%%
0%
Fine-tune
94%
9,%
91%
89%
88%
0%
25%
0%
25%
25%
SVIIN (ResNet)
Train from scratch
90%
83%
75%
61%
21% 5
0%
0%
0%
0%
0%
Noisy model (f- - 0)
11%
11%
11%
11%
11 %
28%
28%
28%
28%
28%
Noisy model (t = 1)
11566
5%
1%
11%
%
18 %
18%
18%
18%
18%
Prune
95%
93%
92%
90%
0%
25%
0%
25%
the attacker to use the same path training loss function as the defender. To attempt breaking path connection, the attacker trains compromised path such that every model on this path is backdoored model and then releases the path-eware tampered models. We show that our approach is still resilient to this adaptive attack. More details are given in Appendix [
3.3 SANITIZING
MODELS
Attack implementation. We adopt the fault sneaking attack (Zhao et al. 2019b) for injecting errors to model weights. Given two untampered and independently trained models, the errors are injected with selected samples targets such that the tampered models will cause misclassification on targeted inputs and otherwise will behave as untampered models. More details are given in Appendix C
Evaluation and analysis. Similar the setup in Section 3.2 Figure shows the clean accuracy and attack accuracy of the models on the path connecting two error-injected models using limited amount of bonatide data. For the error-injected models (t {0, 1})). the attack accuracy is nearly 100%, which corresponds to 0% attack failure rate on targeted samples. However, using path connection and limited amount of bonafide data, the injected errors can be removed almost completely. Varying the size of path training data consistently sanitizes the error-injected models and mainly affects the standard test error. Most of the models on the path can attain nearly 100% fault tolerance (i.e. 100% attack failure rate) to the injected errors. The models on the path near t= 0 or / = have comparable performance on clean data and exhibit strong fault tolerance to injected errors. Similar findings are observed across different network architectures and datasets (see Appendix
Comparison with baselines and extensions. Table 3 we adopt the same baselines as in Section 3.2 to compare with path connection. We find that only path connection and can successfully sanitize the error-injected models and attain 0% attack accuracy, and other baselines less effective. Table 3 also shows the clean accuracy of path connection is substantially better than the effective baselines, suggesting novel applications of mode connectivity for finding accurate and adversarially robust models. The extensions other settings discussed in Appendix G
7

---

Published as a conference paper at ICLR 2020
Connection of regular and udversarially-traine
Connection of adversarially-traing
Connection of regular models (PCC=0.71)
Legend
models (PCC=0.88)
models (PCC=0.86)
argest genvetuu (log)
argeet genvelue (lgg)
orgest gwnveluu (lgg)
loss eigenvalue
cross ontreyy oss
0.
20
training loss
88
000 0e
test loss
0
10
tobustness loss
largest eigenvalue
-10
0.5
attack success rate
16
100
albakk succoss rate (8)
100
alback sucosss rater (%)
103
albok BUIDOSSA rato (%)
cnr rate (%6
enor rate (%6
emor rela (66
error rate
training data
80
test datar
80
10
attack suctoess rake
adverseriel examples
Figure 4: Loss, error rate, attack success rate and largest eigenvalue of input Hessian on the path con- necting different model pairs on CIFAR-10 (VGG) using standard loss. The error rate of training/test data means standard training/test error, respectively. In all cases, there is no standard loss barrier but a robustness loss barrier. There is also a high correlation between the robustness loss and the largest eigenvalue of input Hessian, and their Pearson correlation coefficient (PCC) is reported in the title.
Technical explanations and adaptive attack. Consistent with the results in backdoor attacks, we explore the model weight space to demonstrate the significant difference between the models on our connection path and random noisy models. We also show that the similarity of error-injected images are much lower than that of clean images. In addition, our path connection is resilient to the advanced path-aware prror-inecction attack. More details are given in Appendices H and
3.4 ROBUSTNESS LOSS LANDSCAPE AND CORRELATION ANALYSIS FOR EVASION ATTACK
To gain insights on mode connectivity against evasion attacks, here we investigate the standard and adversariall-obustten loss landscapes on the same path connecting two untampered and indepen- dently trained models. The path is trained using the entire training dataset for minimizing equation [ with standard cross entropy loss. The robustness loss refers the cross entropy of class predictions on adversarial examples generated by evasion attacks and their original class labels. Higher robustness loss suggests the model is more vulnerable to evasion attacks. In addition, we will investigate the robustness loss landscape connecting regular where the path is also trained with standard cross entropy loss. We will also study the behavior of the largest eigenvalue of the Hessian matrix associated with the cross entropy loss and the data input, which we call the input Hessian. As adversarial examples are often generated by using the input gradients, we believe the largest eigenvalue of the input Hessian can offer new insights robustness loss landscape, similar to the role of model-weight Hessian on quantifying generalization performance (Wu et al., 2017 Wang al. 2018a).
and adversarially-trainee (robust) models,
Attack Implementation. We uniformly select 9 models (t = 00.., 0.9}) on the path and evasion attacks each of them using the -orm--lll based projected gradient descent (PGD) method proposed in (Madry et al.] (2018). The robustness loss is evaluated using the non-targeted adversarial examples crafted from the entire test set, and the attack perturbation strength set to e = 8/255 with 10 iterations. We also use the PGD method for adversarial training to obtain adversarially-trainee models that are robust to evasion attacks but pay the price of reduced accuracy on clean data (Madry et al. 2018).
Evaluation and Analysis. To study the standard and robustness loss landscapes, we scrutinize the models on the path connecting the following pairs of models: (i) independently trained regular (non-robust) models; (ii) regular to adversarially-traine trained models; and (iii) independently adversarially- trained models. These results are shown in Figure 4 We summarize the major findings as follows.
No standard loss barrier in all cases: Regardless of the model pairs, all models on the paths have similar standard loss metrics in terms of training and test losses, which are consistent with the previous results on the "flat" standard loss landscape for mode connectivity (Garipov et al.) 2018 Gotmare al. 2018 Draxler et al. 2018). The curve of standard loss in case (ii) is skewed toward
8

---

Published as a conference paper at ICLR 2020
one end due to the artifact that the adversarially-triin trained model (f 1) has a higher training/test error than the regular model (t = 0).
Robustness loss barrier: Unlike standard loss, we find that the robustness loss the connection path has a very distinct characteristic. In all cases, there is a robustness loss barrier (a hill) between pairs of regular and adversarially-trainedd models. The gap (height) of the robustness loss barrier is more apparent in cases (ii) and (iii). For (ii), the existence of a barrier suggests the modes of regular and ddversariallly-tainee models are not connected by the path in terms of robustness loss, which also provides a geometrical evidence of the "no free lunch" hypothesis that adversarially robust models cannot be obtained without additional costs TTipras et al. 2019: [Dhhmatob] 20188] Bubeck et al., 2019). For (iii), robustness loss barriers also exist. The models on the path are less robust than the two adversarially-traine models at the path end, despite they have similar standard losses. The results suggest that there are essentially no better adversarially robust models on the path connected by regular training using standard loss.
High correlation between the largest eigenvalue of input Hessian and robustness loss: Inspecting the largest eigenvalue of input Hessian (x)x) of a data input z on the path, denoted by Amax(()) (l), we observe strong accordance between Amax (4) and robustness loss on the path, verified by the high empirical Pearson correlation coefficient (PCC) averaged over the entire test set as reported in Figure 4 As evasion attacks often use input gradients craft adversarial perturbations to x. the eigenspectrum of input Hessian indicates its local loss curvature and relates to adversarial robustness Yu et al. 2018). The details of computing Xmax (t) are given in Appendix K Below we provide technical explanations for the empirically observed high correlation between Xmux (t)
and the oracle robustness loss on the path, defined as max |8lls I(w(t), x 1-++ 5).
Proposition 1. Let (-) be neural network classifier with its model weights denoted by and let l(w, x) denote the classification loss (e.g. cross entropy of fau (z) and the true label of a data sample x)). Consider the oracle robustness loss max {(w(t), x 5) of the model t on the path, where 8 denotes a perturbation to confined by an e-ball induced by vector norm || ||. Assume (a) the standard loss Nww(t), x) on the path is a constant for all t E [0.
(b) Uw((t) x + 8) se I(w(t), x)) + V vwl((t)),z)' 2) + 1,8"
Hz) for small 6, where V V=l(w(t), is the input gradient and Hz(x) is the input Hessian of ((w(()),a x) at x.
Let denote the normalized inner product in absolute value for the largest eigenvector v of Hx)x) and V x)) V.((w(().)) [V
c. Then we have max [w](t), x + 5) Xmax (t) as
c+11. 1.
Proof: The proof's given in Appendix L Assumption (a) follows by the existence of high-accuracy
path of standard loss landscape from mode connectivity analysis. Assumption (b) assumes the local landscape with respect to the input can be well captured by its second-order curvature based on Taylor expansion. The value of is usually quite large, which has been empirically verified in both regular and adversarially-trainn
models (Moosavi-Dezfooli et al. 2019).
Extensions. Although we find that there is robustness loss barrier on the path connected by regular training, we conduct additional experiments to show that it is possible to find an robust path connecting two adversarially-traine or regularly-trained model pairs using adversarial training (Madry et al. 2018), which we call the "robust connection" method. However, model ensembling using either the regular connection or robust connection has little gain against evasion attacks, as the adversarial examples are known to transfer between similar models (Papernot et al. 2016 Su et al. 2018). We refer readers to Appendix M for more details.
CONCLUSION
This paper provides novel insights on adversarial robustness of deep neural networks through the lens of mode connectivity in loss landscapes. Leveraging mode connectivity between model optima, we show that path connection trained by limited number of clean data can successfully repair backdoored or error-injected models and significantly outperforms several baseline methods. Moreover, we use mode connectivity to uncover the existence of robustness loss barrier on the path trained by standard loss against evasion attacks. We also provide technical explanations for the effectiveness of our proposed approach and theoretically justify the empirically observed high correlation between robustness loss and the largest eigenvalue of input Hessian. Our findings are consistent and validated on different network architectures and datasets.
9

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENTS
This work was primarily conducted during Pu Zhao's internship at IBM Research. This work is partly supported by the National Science Foundation CNS-1929300.
REFERENCES
A. Barenghi, L. Breveglieri, and et. al. Fault injection attacks on cryptographic devices: Theory, practice, and countermeasures. Proceedings of the IEEE, 2012. ISSN 0018-9219.
Battista Biggio and Fabio Roli, Wild patterns: Ten years after the rise of adversarial machine learning.
Pattern Recognition, 84:317-331, 2018
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In International Coference on International Conference on Machine Learning, pp. 1467-1474, 2012.
Sébastien Bubeck, Eric Price, and Ilya Razenshteyn. Adversarial examples from computational constraints. International Coference International Conference Machine Learming, 2019.
Nicholas Carlini and David Wagner. Towards evaluating the robustness neural networks. In IEEE
Symposium Security and Privacy, pp. 39-57, 2017.
Bryant Chen, Wilka Carvalho, Nathalic Baracaldo, Heiko Ludwig. Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint arXiv: 1.00772 2018.
Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. 700: Zeroth order
optimization based black-box attacks deep neural networks without training substitute models. In ACM Workshop on Artificial Intelligence and Security, pp. 15-26, 2017.
Elvis Dohmatob. Limitations of adversarial robustness: strong free lunch theorem. International Conference on International Conference on Machine Learning, 2018
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International Conference on Machine Learning, volume 80, pp. 1309-1318, Jul 2018.
Rida T. Farouki. The bernstein polynomial basis: A centennial retrospective. Computer Aided Geometric Design, 29(6):379 419, 2012. ISSN 0167-8396. doi: hups://doi.org/10.100 2.c2gd.2012.03... URL http::/ www.sceiieeecce..ee rect com wtt.:/ccccccccceeeec wwt.////cienne//aticcccc
Pii 011/806673361200119
Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. The robustness of deep networks: A geometrical perspective. IEEE Signal Processing Magazine, 34(6):50-62, 2017.
Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfoolii Pascal Frossard, and Stefano Soatto. Empirical study of the topology and geometry of deep networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3762-3770, 2018.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of DNNs. In Advances in Neural Information Processing Systems, pp. 8789-8798, 2018.
Jonas Gomes, Luiz Velho, and Mario Costa Sousa. Computer Graphics: Theory and Practice. A. K. Peters, Ltd., Natick, MA, USA, 1st edition, 2012. ISBN 1568815808, 9781 1668818111 15800.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015.
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. Using mode connec- tivity for loss landscape analysis. arXiv preprint arXiv: 1806.06976777 2018.
10

---

Published as a conference paper at ICLR 2020
T. Gu, K. Liu, B. Dolan Gavat,, and S. Garg. BadNets: Evaluating backdooring attacks deep neural networks. IEEE Access, 7:47230 47244, 2019.
Kaiminaripov He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. International Coference on International Conference on Machine Learning, 2018
Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. In IEEE Symposium on Security and Privacy (SP), pp. 19-35, 2018.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. International Conference on Learning Representations, 2017.
Y. Liu, L. Wei, B. Luo, and Q Xu. Fault injection attack on deep neural network. In 2017 IEEE/ACM ICCAD, pp. 131-138, Nov 2017. doi: 10. D0.100CADD.2221....000
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuar Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In Network and Distributed System Security Symposium (NDSS), 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representationss 2018.
James Martens and Ilya Sutskever. Training deep and recurrent networks with hessian-free optimiza- tion. In Neural networks: Tricks of the trade, pp. 479-535, Springer, 2012.
Seyed-Mohsee
Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Ro- bustness via curvature regularization, and vice versa. IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07271 2016.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In ACM Asia Conference on Computer and Communications Security, pp- 506-519, 2017.
Ali Shafahi, W Ronny Huang. Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Advances in Neural Information Processing Systems, pp. 6103-6113, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1499.155 1556, 2014.
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin-Yi Yu-Yuu Chen, and Yupeng Gao. Is robustness the cost of accuracy?-a comprehensive study on the robustness 18 deep image classification models. In European Conference on Computer Vision, pp. 631-648. 2018.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In Advances in Neural Information Processing Systems, pp. 8000-8010, 2018.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Tumer, and Aleksander Madry. Ro- bustness may be odds with accuracy. In International Conference on Learning Representations, 2019.
Victor Van Der Veen, Yanick Fratantonio, and et. al. Drammer: Deterministic rowhammer attacks on mobile platforms. In ACM SIGSAC conference on computer and communications security, pp. 1675-1689. ACM, 2016.
11

---

Published as a conference paper at ICLR 2020
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In IEEE Symposium Security and Privacy, 2019.
Huan Wang, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. Identifying generalization properties in neural networks. arXiv preprint arXiv: (e:9:0009000000 2018a.
Timothy E Wang, Jack Gu, Dhagash Mehta, Xiaojun Zhao, and Edgar A Bernal. Towards robust deep neural networks. arXiv preprint 181. 2018b.
Lei Wu, Zhanxing Zhu, et al. Towards understanding generalization of deep learning: Perspective of loss landscapes. In International Conference Machine Learning, 2017.
K. Xu, S. Liu, P. Zhao, P.-Y. Chen, H. Zhang, D. Erdogmus, Y. Wang, and X. Lin. Structured Adversarial Attack: Towards General Implementation and Better Interpretability. ArXiv e-prints, August 2018.
Fuxun Yu, Chenchen Liu, Yanzhi Wang, and Xiang Chen. Interpreting adversarial robustness: view from decision surface in input space. arXiv preprint arXiv: 1810000144 2018.
Pu Zhao, Sijia Liu, Yanzhi Wang. and Xue Lin. An admm-basee universal framework for adversarial attacks on deep neural networks. In ACM Multimedia 2018, 2018. ISBN 978- 1-55----666---5- doi: 0g.114553240508.2446653
URL (RLt:tp:: acm. 10. 11w.11145/2222 11t.//...111/220551... 3240639
Pu Zhao, Sijia Liu, Pin-Yu Chen, Nghia Hoang, Kaidi Xu, Bhavya Kailkhura, and Xue Lin. On the design of black-box adversarial examples by leveraging gradient-free optimization and operator splitting method. In ICCV 2019, October 2019a.
Pu Zhao, Siyue Wang, Cheng Gongye, Yanzhi Wang, Yunsi Fei, and Xue Lin. Fault sneaking attack: stealthy framework for misleadingdeep neural networks. DAC, 2019b.
Pu Zhao, Yu Chen, Siyue Wang, and Xue Lin. Towards query-efficient black-box adversary with zeroth-order natural gradient descent. AAAI 2020, 2020.
12

---

Published as a conference paper at ICLR 2020
APPENDIX
A NETWORK ARCHITECTURE AND TRAINING
this paper we mainly use two model architectures, VGG and ResNet. The VGG model (Simonyan) & Zisserman 2014) has 13 convolutional layers and 3 fully connected layers. The ResNet model is based on the Preactivation -eeett implementation (He et al. 2016) with 26 layers. The clean test accuracy of untampered models of different architectures
CIFAR- 10 and SVHN given in Table A All of the experiments are performed on GTX 1080Ti GPUs. The experiments are implemented with Python and Pytorch.
Table A1: Test accuracy of untampered models for different datasets and model architectures.
architecture
VGG
ResNet
model (t 0)
88%
87%
CIFAR-10
model (t = 1)
86%
91%
model [[
96%
97%
SVHN
model {L = 98%
99%
B REGULAR PATH CONNECTION OF UNTAMPERED MODELS ON SVHN (RESNET)
The performance of regular path connection of untampered models on SVHN with ResNet presented in Figure AT
Inference on training set
Inference on test set
Legend
50
50
cross entropy loss
40
effor rate (%))
cross entropy loss
40
2500 Images training, loss
images training, 500 images training loss
30
30
BEFOO rale
250 images training,
50 mages training, loss
20
20
2500 images training,
1000 images training, error rate
10
10
images training, error
250 images training, error
0.5
0.5
50 mages treinina, error rate
Figure A1: Loss and error rate on the path connecting two untampered ResNet models trained on SVHN. The path connection is trained using different settings as indicated by the curve colors. The inference results on test set are evaluated using 5000 samples, which are separate from what are used for path connection.
C ILLUSTRATION AND IMPLEMENTATION DETAILS OF BACKDOOR AND ERROR-INJECTION ATTACKS
Backdoor attack The backdoor attack is implemented by poisoning the training dataset and then training backdoored model with this training set. To poison the training set, randomly pick 10% images from the training dataset and add trigger to each image. The shape and location of the trigger is shown in Figure Meanwhile, set the labels of the triggered images to the target label(s) as described in Section 3.2
Error-injection attack We select 1000 images from the test set and pick 4 images as targeted samples with randomly selected target labels for inducing attack. The target labels of the 4 selected images are different from their original correct labels. The goal of the attacker is to change the classification of the images to the target labels while keeping the classification of the remaining 996 images unchanged through modifying the model parameters. To obtain the models with injected errors on CIFAR-10, we first train two models with clean accuracy of 88% and 86%, respectively. Keeping
13

---

Published as a conference paper at ICLR 2020
CIFAR-10 12-
5
6.
SVHN
Figure A2: Examples of backdoored images on CIFAR-I0 and SVHN. The triggers white blocks
located at the right-bous bottob area of each image.
the classification of number of images unchanged can help to mitigate the accuracy degradation incurred by the model weight perturbation. After perturbing the model weights, the 4 errors can be injected into the model successfully with 100% accuracy for their target labels. The accuracy for other clean images become 78% and 75%, respectively.
D PREDICTION ERROR ON THE TRIGGERED DATA
We show the prediction error of the triggered data relative to the true labels on all datasets and networks in Figure A3 The error rate means the fraction of triggered images having top-1 predictions different from the original true labels. The prediction error rate of triggered data is high path ends (t = 0, 1) since the two end models are tampered. has similar trend as standard test error for models close the path ends, suggesting path connection find models having good classification accuracy on triggered data.
CIFAR-10 (VGG)
CIFAR-10 10 (ResNet)
SVHN (VGG)
SVHN (ResNet)
100
50-
20
Figure A3: Prediction error rate against backdoor attacks on the connection path.
E MORE RESULTS ON PATH CONNECTION AGAINST BACKDOOR ATTACKS
Implementation Details for Table 2 For our proposed path connection method, we train the connection using different number images as given in Table for 100 epochs and then report the performance of the model associated with a selected index f on the path. For the fine-tuning and training-fromt from-scratch methods, we report the model performance after training for 100 epochs. For the random Gaussian perturbation to model weights, we evaluate the model performance under Gaussian noise perturbations on the model parameters. There are two given models which are the models at f 0 and 1. The Gaussian noise has zero mean with standard deviation of the absolute value of the difference between the two given models. Then we add the Gaussian noise to the two given models respectively and test their accuracy for clean and triggered images. For Gaussian noise, the experiment is performed multiple times (50 times) and we report the average accuracy. We that adding Gaussian noise perturbations to the model does not necessarily
14

---

Published as a conference paper at ICLR 2020
change the model status from robust to non-robust from non-robust to robust. The path connection or evolution from the model at t = to that = follows a specifie path achieving robustness against backdoor attack rather than random exploration. For pruning, et al. 2017) to prune filters from convolutional neural networks (CNNs) that are identified as having small effect the output accuracy. By removing the whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. We first prune about 60% of its parameters for VGG or 20% parameters for ResNet. Then we retrain the network with for 100 epochs. The clean accuracy and backdoor different number of images given in Table for accuracy are as reported.
the filter pruning method (Li)
Figure A4 shows the error rates of clean and backdoored samples using CIFAR-10 on the connection path against single-target attacks.
VGG
ResNet
Legend
100
100
error rale (%)
60
11
effor rate (%)
80
mages triinigg
imaes trairing, clean
60
60
500 images troining, cleen encr
250 images training, clean encr
50 Images training, dean emror
40
40
2500 tnsining, allack attack 250 Images tralting, artack
images
-00 images training attack error
0.5
0.5
Figure A4: Error rate of single-target backdoor attack the connection path for CIFAR-10. The error rate clean/backdoored samples means standard-resst-eroo/attaa ffaluurr--allerrteeee. respectively.
Figure A5 shows the error rates of clean and backdoored samples using SVHN the connection path against single-target attacks.
VGG
ResNet
Legend
80
80
2500 mages trining, ciaan enor
arror rate (%)
eiror rate (%)
1000 mages traming, clean enor
60
60
500 Usining. clean
250
clean
1g
Eeining,
40
40
mages training.
1000 meçes trainigg allack
20
20
500 images training, attack enror
250 images training attack anorr
-00 Images training, attack enror
0.5
0.5
Figure A5: Error rate single-target backdoor attack on the connection path for SVHN, The error rate of clean/backdoored samples means sonddarddtest-errrr/atttaekk failurrrrae,alllleaaatteet respectively.
Table A2 shows the performance comparison of path connection and other baseline methods against single-targe backdoor attack evaluated on CIFAR-10 (ResNet) and SVHN (VGG).
F MORE RESULTS ON PATH CONNECTION AGAINST ERROR-INECTION ATTACKS
For our proposed path connection method, we train the connection using different number of images given in Ta for 100 epochs and then report the performance the model associated with a selected index on the path. The start model and end model have been injected with the same 4 errors (misclassifying 4 given images), starting from two different unperturbed models obtained with different training soper-parameters. For the fine-tuning and training- from--cratch methods, we report the model performance after training for 100 epochs. For the random Gaussian perturbation to model weights, we evaluate the model performance under Gaussian noise perturbations the model parameters. There are two given models which the
Implementation Details for Table 3
15

---

Published as a conference paper at ICLR 2020
Table A2: Performance comparison path connection and baselines against single-target backdoor attack. The clean/backdoor accuracy means standard- haabla/--e---cccccccyytttteeeeee
respectively.
Methods bonafide data size
2500
1000
500
250
50
Path connection 0.2)
87%
83%
80%
77%
67%
Fine-tune
85%
84%
83%
83%
72%
Clean
Train from scratch
42%
36%
33%
29%
22%
Accuracy
Noisy model (f = 0)
12%
12%
12%
12%
12%
Noisy model (f 1)
10%
10%
10%
10%
10%
CIFAR-10 (ResNet)
Prune
85%
82%
80%
79%
77%
Path connection 0.2)
0.6%
1.2%
1.3%
2.3%
5.35
Fine-tune
0.7%
8.7%
19%
20%
18%
Backdoor Accuracy
Train from scratch
3.7%
4.3%
3.5%
8.2%
6.5%
Noisy model (f 0)
95%
95%
95%%
95%
95%
Noisy model (1 1)
97%
97%
97%
97%
97%
Prume
37%
52%
78%
86%
88%
Path connection 0.7)
95%
93%
91 %
87%
75%
Fine-tune
93%
92%
90%
89%
77%
Clean Accuracy
Train from scratch
88%
82%
76%
54%
25%
Noisy model (f = 0)
22%
22%
22%
22%
22%
Noisy model (t = 1)
16%
16%
16%
16%
16%
SVHN (VGG)
Prune
94%
92%
91 %
89%
86%
Path connection 0.7)
2%
6%
2%
9%
22%
Fine-tune
11%
10%
13%
30%
Backdoor Accuracy
Train from seratch
2.6%
2.3%
3.3%
4.7%
13%
Noisy model (f = 0)
79%
79%
79%
79%
79%
Noisy model (f 1)
62%
62%
62%
62%
62%
Prune
75%
69%
78%
86%
91%
models & 0 and The Gaussian noise has zero mean with standard deviation of the absolute value of the difference between the two given models. Then we add the Gaussian noise the two given models respectively and test their accuracy for clean and triggered images. The experiment is performed multiple times (50 times) and we report the average accuracy. We can see that adding Gaussian noise perturbations to the model does not necessarily change the model status from robust to non-robust or from non-robust to robust. The path connection or evolution from the to that t = 1 follows a specific path achieving robustness against backdoor attack
model at rather than random exploration. The training- scrmtshratch baseline method usually obtains the lowest clean test accuracy, especially in the case of training with 50 images, where training with so little images does not improve the accuracy.
Figure A6 A shows the performance of path connection against error-injection attacks evaluated on CIFAR-I0 (ResNet) and SVHN (VGG).
CIFAR-10 (ResNet)
SVHN (VGG)
Legend
100
100
60
80
mages triiing,
efror rale (%)
effor rate (%)
maç traiiigg, clean
60
60
300 images training, clean encor
260 images trcining, clcan enrrr
30 images training dean error
40
40
2600 mages training, atlack
1000
attack 255 images training.
trchiing
imagesi trining staikk error
0.5
0.5
Figure A6: Error rate against error-injection attack the connection path for CIFAR- (VGG). The emrrr rate clean/targeted samples means
respectively.
Table 3 and show the performance comparison of path connection and other baseline methods against error-injection attacks evaluated on all combinations of network architectures and datasets.
16

---

Published as a conference paper at ICLR 2020
Table A3:
Performance evaluation against error-injection attack. The clean/injection accuracy means respectively. Path connection attains the highest clean accuracy and lowest (0%) attack accuracy among all methods.
Methods / bonafide data size
2500
1000
500
250
50
Path connection (f 0.1)
87%
83%
81%
80%
77%
Fine-tune
80%
79%
78%
75%
72%
Clean Accuracy
Train from scratch
46%
39%
32%
28%
18%
Noisy model (f = 0)
11%
11%
11%
1 11%
11 1
Noisy model (f
10%
10%
10%
10%
10%
CIFAR-10 (ResNet)
Prunc
84%
82%
81%
80%
78%
Path connection (f = 0.1)
0%
0%
0%
0%
0%
Finc-tunc
0%
0%
0%
25%
25%
Injection Accuracy
Train from scratch
0%
0%
0%%
0%
0%
Noisy model
33%
33%
33%
33%
33%
Noisy model (f =
26%
26%
26%
26%
26%
Prunc
%
0%
0%
25%
25%
Path connection (f = 0.1)
96%
94%
93%
91%
90%
Fine-tune
94%
93%
92%
90%
81%
Clean Accuracy
Train from scratch
88%
82%
76%
68%
28%
Noisy model (f =
33%
33%
33%
33%
33%
Noisy model (f 1)
37%
37%
37%
37%
37%
SVHN (VGG)
Prune
94%
93%
92%
90%
89%
Path connection (f = 0.1)
0%
0%
(%%
0%
0%%
Fine-tune
0%
25%
0%
25%
Injection Accuracy
Train from scratch
0%
0%
0%
0%
0%
Noisy model (f 0)
24%
24%
24%
24%
24%
Noisy model
29%
29%
29%
29%
29%
Prunc
0%
25%
0%
25%
25%
G EXTENSIONS OF PATH CONNECTION TO DIFFERENT SETTINGS
In Sections 3.2 and 3.3 we consider the scenario where the two given models are tampered using in
the same way using the same poisoned dataset for backdoor attack and the same targeted images error-injection attack. Here we discuss how to apply path connection to the case when only one tampered model is given. In addition, show that the the resilient effeet of path connection against these attacks still hold when the two given models are tampered a diffcrent way.
Backdoor and error-injection attacks given one tampered model We propose to first fine-tune the model using the bonafide data and then connect the original model with the fine-tuned model. The fine-tuning process uses 2000 images with 100 epochs. The path connection results are shown in Figure A7 The start model is backdoored model with high accuracy for triggered images. The end model is fine-tuned model where the triggers do not have any effects to cause any misclassification. We can see that through path connection, we can eliminate the influence of the triggers quickly in some cases. For example, with 250 images, the error rate of triggered images reaches 100% at t = 0.25 while the clean accuracy at this point is lower than the fine-tur fine-tuned model at
Similarly, for the case of crror-injection attack, we first fine-tune the model using the bonafide data and then connect the original model with the fine-tuned model. We follow the same settings of the backdoor attack for the finetuning and path connection. The performance of the tampered model case is shown Figure A8 a). We can see that the effects of injected errors can be eliminated quickly through path connection while the clean accuracy is kept high.
Backdoor and error-injection attacks given two differently tampered models the given two backdoored models are trained with different poisoned datasets (e.g. different number of poisoned images), the path connection method works as well in this case. We train two backdoored models by poisoning 30% and 10% of the training dataset, respectively. The performance of the path connection between the two models are shown in Figure A9 We can see that the connection can quickly remove the adversarial effects of backdoor attacks.
17

---

Published as a conference paper at ICLR 2020
Single target attack
Legend
100
2500 images training, clean error
error rate (%)
80
1000 images training, clean error
images training, clean error
60
250 images training, clean error
50 images training, clean error
40
-2500 images training, attack error
-1000 images training, attack
20
500 images training, attack error
250 images training, attack error
55 images training, attack error
0.5
Figure A7: Error rate under backdoor attack on path connection for CIFAR-10 (VGG). The error rate of clean/triggered samples means the pendddddees--emoraattekfalaiiin
respectively.
Two tampered models with
One tampered model
Legend
different injected errors
100
100
images training, deaan
rate (%)
80
enror rate (%)
80
1000 images training, diean enoo
500 Images training,
60
80
250 images training, deann enror
50 images training dlea error
40
40
images training
1000 images training. attack error
500 images training, atlack
250 images training, attack error
550 images makiingg attack enor
0.5
0.5
Figure A8: Error rate against error-injection attack on the connection path for CIFAR-IO (VGG). The eтror rate of clean/targeted samples means sandaard-estt-errrrttttaee aerror/attacc-faluurrrrattt failure-rate, respectively.
If the two given models with injected errors are trained with different settings (e.g. different total number of training images to inject the errors), case. For the start and end models, the number of injected errors is set to 4. The number of images with the same classification requirement is set to 996 for the start model, and 1496 for the end model,
path connection method works as well this
respectively. The performance of path connection is shown in Figure A8 (b).. We can observe that it is able to obtain robust model with high clean accuracy.
Single-target attack
All-targets attack
Legend
100
arror rate (%)
80
error rate (%)
80
2600 mages training. ciaan error
1000 mages tmaring. cinan emer
60
60
500 images training, clean encr
250 training clean
40
40
training
1000 imças trwinng.
20
20
+00 images troining. sttack emorr
-250 images training sttckk emoor
550 images training atacc error
0.5
0.5
Figure A9: Error rate against backdoor attack on the connection path for CIFAR-10 (VGG). The error rate of clean/triggered samples means the standamkeet-erro/ttttkkkllllnrn respectively.
H MODEL WEIGHT SPACE EXPLORATION
To explore the model weight space, add Gaussian noise to the weights of backdoored model to generate 1000 noisy models the backdoored model at = The standard normal Gaussian noise has zero mean and standard deviation of the absolute difference between the two end models the
18

---

Published as a conference paper at ICLR 2020
path. The distribution of clean accuracy and backdoor accuracy of these noisy models are reported in Figure/A10((a (a). The results show that the noisy models are not ideal for attack mitigation and model repairing since they suffer from low clean accuracy and high attack success rate. In other words, is unlikely, if not impossible, to find good models by chance. We highlight that finding path robust to backdoor attack between two backdoored models is highly non-intuitive, considering the high failure rate of adding noise. We can observe similar phenomenon for the injection attack in Figure A10 (b).
0.14
0.39
0.12
03
Probabilit distribution
Probability distribution
0.1
0.25
0.00
02
0.06
0.15
0.04
0.1
0.02
0.00
Clean accuracy (%)
Backdoor accuracy (%)
(a) backdoor attack
0.17
06
Pmnbbblitty distributtion tion
0.1
Probabiltty distributiin
0.
0.00
30
Clean accuracy (%)
Injection accuracy (%)
(b) error-injection attack
Figure A10: Clean and attack accuracy distribution for 1000 noisy models.
I DATA SIMILARITY OF INPUT GRADIENTS
To provide technical explanation on why our path connection approach can mitigate backdoor and injection attacks, we compare the similarity of input gradicnts between the models on the connection path (t E (0, 1)) and the end models (t 0 or t 1) in terms of clean data and tampered data. The rationale of inspecting input gradient similarity can be explained using the first-order approximation of the training loss function. Let ((z/wy) denote the training loss function of data input given model We on the connection path, where E [0, 1]. Then the first-order Taylor series approximation xz),, where
on ((x))) with respect V Val (wy) is the gradient of (x)(wy) when x = Izz Based on the flat loss of mode connectivity, we can further assume 1(x; (wz)) is constant for any & [0, 1]. Therefore, for the same data sample z,, the model (t (0, 1)) will behave similarly as the end model Wo (or w1 if its input gradient __1(x,) (xx) similar to [ (wo) (or [_( (ww)) Figure All shows the average cosine similarity distance of the input gradients between the models on the path and the end models for the backdoor attack and the injection attack. The pairwise similarity metric for each data sample is defined
data sample x; is ((x) w.) 1(x; (w:) + V=l(=//w.)'
Is 11/2, where is the cosine similarity of the input gradients between the models on the path and the end models. Smaller m means higher similarity the input gradients. Comparing the minimum of the solid and dashed curves of different colors the path respectively. which corresponds to the similarity to either one of the two end models, we find that the similarity of clean data are consistently higher than that of tampered data. Therefore, the sanitized models on the
19

---

Published as a conference paper at ICLR 2020
path can maintain similar clean data accuracy and simultaneously mitigate adversarial effects as these models are dissimilar to the end models for the tampered data.
Path for the backdoor attack
Path for the injection attack
Average similarity distance
Average similarity distance
0-
0.
cleen data simileriy wrt
0.2
backdoorec data similaniy wil t-0
2.1
iclean data similarit wrt
9.
gel imgges kiijenal lobvboo sieeleity
(backdoorec data siilanty 1=1
siibtrrty Earge ty
Figure A11: Similarity distance between the models the path and the two end models for CIFAR-10
(VGG). Smaller distance value means higher similarity.
J ADAPTIVE ATTACKS
We consider the advanced attack setting where the attacker knows path connection is used for defense but cannot compromise the bonafide data. Furthermore, we allow the attacker to use the same path training loss function as the defender.
Backdoor attack. To attempt breaking path connection, the attacker first separately trains two backdoored models with one poisoned dataset. Then the attacker uses the same poisoned dataset to connect the two models and hence compromises the models on the path. Note that when training this tampered path, in addition to learning the path parameter 0, the start and end models are not fixed and they are fine-tuned by the poisoned dataset. Next the adversary releases the start and end models (t = 0, 1) from this tampered path. Finally, the defender trains path from these two models with bonafide data. We conduct the advanced (path-aware) single-target backdoor attack experiments on CIFAR- 10 (VGG) by poisoning 10% of images in the training set with trigger. Figure A1 (a) shows the entire path has been successfully compromised due the attacker's poisoned path training data, yielding less than 5% attack error rate on 10000 test samples. Figure A12 (b) shows the defense performance with different number of clean data to connect the two specific models released by the attacker. We find that path connection is still resilient to this advanced attack and most models on the path (e.g. & [0.25, 0.75)) can be repaired. Although this advanced attack indeed decreases the portion of robust models on the path, it still does not break our defense. In Table A4] we also compare the generalization and defense performance approaches. Moreover, in the scenario where two tampered models are close in the parameter space (in the extreme case they are identical), we can leverage proposed path connection method with one tampered model to alleviate the adversarial effects. The details are discussed in the "Extensions" paragraph and Appendix G
demonstrated that path connection outperforms other
Table A4: Performance against path-aware single-target backdoor attack on CIFAR-10 (VGG).
Clean accuracy
Backdoor accuracy
Method/ Bonatide data size
2500
1000
500
250
50
2500
1000
500
250
Path connection (t 0.27)
90%
87%
83%
81 31%
75%
3.8%
2.9%
3.6%
4.2%
5.6%
Fine-ture Fine-tune
88%
84%
82%
80%
69%
4.1%
4.6%
4.4%
3.9%
5.9%
Train from scratch
58%
48%
36%
28%
20%
0.6%
0.5%
0.9%
1.7%
1.9%
Noisy model (t 0)
38%
38%
38%
38%
38%
91 %
91% 01%
91 %
91%
91%
Noisy model (t
35%
35%
35%
35%
35%
86%
86%
86%
86%
86%
Prune
87%
85%
83%
81 56
79%
29%
48%
69%
77%
81 %
20

---

Published as a conference paper at ICLR 2020
80
2500
images training, dican
error rale
10
error rate
500 imeges training, deen enoor
50000 training, clean
60
250 images training, deen
50000 imeges training, etack enror
40
2500 images training, attakk emar
images training
1000 images training, sttack enor
20
+500 images training
-20 images training, attack erro
-50 images training, sttack eriar
0.5
0.5
(a) Path with poisoned data
(h) Path with clean data
(c) Legend
Figure A12: Error rate against path-aware single-target backdoor attacks for CIFAR-10 (VGG).
Error-injection attack. For the advanced attack in the error-injection setting, the attacker first trains two models with injected errors and then connects the two models with clean data. Then, the attacker tries to inject errors to the three models W1, W2 and 6, Note that based on Equation (3) or (4), all the models on the path can be expressed path is expected to be tampered. Next, the attacker releases the start and end models from the "bad" path. Finally, the defender trains a path from these two models with bonafide data. We conduct the advanced (path-aware) error-injection attack experiments on CIFAR-10 (VGG) by injecting 4 errors. Figure A13 (a) shows that the entire path has been successfully compromised by the advanced path-aware injection. While the clean error rate is stable across the path, at least 3 out of injected errors (corresponding to 25% attack error rate) yield successful attacks. After training path to connect the two released models with the bonafide data, the clean and attack error are shown in Figure A13 (b). We can observe that path connection can effectively eliminate the injected errors on the path (i.e., high attack rate).
combination of these three models. Thus, the whole
Erjctiin with I0CC Inogge Inpetto arod
error rate 196)
80
2500 training, dlean eror images training.
60
500 Images training dican eror
250 images training, dlean enror
images training
OF
40
2500 images training, ellack enor
1000 images training, attack enor
20
500 traming, attack
images training, attack
550 images trainig, stlack error
0.5
(a) Path with injection attack
Path without injection attack
(c) Legend
Figure A13: Error rate against path- error-injection attacks for CIFAR-10 (VGG).
K IMPLEMENTATION DETAILS FOR EVASION ATTACK AND ADVERSARIAL TRAINING
Evasion attack After training the connection with the training set for 100 epochs, adversarial examples of the whole test set generated by the form based PGD method (Madry et al.) 2018) with attack strength = 8/23 8/255 and 10 attack iterations are used to test the performance of the path connection.
Adversarial training For adversarial training, at each model weight update stage, we first generate adversarial examples with the PGD method (Madry al., 2018). The attack strength set to = 8/ 8/255 with 10 attack iterations. After the adversarial examples are obtained, we then update the model weights based on the training losses induced by these adversarial examples with their correct labels.
Input Hessian As the adversarial examples perform small perturbations around the clean images to increase the robustness loss function, thus incurring misclassification, it relates to the Hessian
21

---

Published as a conference paper at ICLR 2020
matrix of the loss function with reference to the input images. A Hessian matrix (or Hessian) is the second-order partial derivatives of scalar-valued function, describing the local curvature of function with many variables. In general, large Hessian spectrum means the function reaches sharp minima, thus leading to a more vulnerable model as the input can leave this minima with small distortions. By contrast, in the case of flat minima with small Hessian spectrum, it takes more efforts for the input to leave the minima.
As we find the robustness loss barrier on the path, we are interested in the evolution of the input Hessian for the models on the path. We uniformly pick the models on the connection and compute the largest eigenvalue of the Hessian w.r.t. to the inputs. To deal with the high dimension difficulties for the Hessian calculation, the power iteration method (Martens & Sutskever, (2012) is adopted to compute the largest eigenvalue of the input Hessian with the first-order derivatives obtained by back-propagatie relative error of 1E-4.
Unless otherwise specified, we continue the power iterations until reaching
For ease of visual comparison, in Figure 4 we plot the log value of the largest eigenvalue of the input Hessian together with the error rate and loss for clean images and adversarial examples. We note that the Pearson correlation coefficient (PCC) is indeed computed using the original largest eigenvalue of input Hessian and robustness loss. As demonstrated in Figure 4 the evolution of the input Hessian is very similar to the change of the loss of adversarial examples. As we can see, the largest eigenvalue on the path does not necessarily have a high correlation with the error rate of the clean images in the training set and test set or the adversarial examples. Instead, it seems be highly correlated with the loss of adversarial examples as they share very similar shapes. This inspires us to explore the relationship between the largest eigenvalue of the input Hessian and the robustness loss.
L PROOF OF PROPOSITION
For simplicity, here we drop the the notation dependency on the input data sample x. It also suffices to consider two models w == w(l) and w + Aw == w(t + Al) for some small At on the path for our analysis. We begin by proving the following lemma.
Lemma 1. Given assumption (a), for any vector norm
- for any data sample
| 21 (w + Aw, x||l - V (w,, x) = 0
(A1)
Proof.
With assumption (a), we have (w+ + Aw, x) (w, x) const. Differentiating at both sides with respect to z and then applying the vector norm gives the lemma.
As the adversarial perturbation is generated using the PGD method (Madry et al. 2018 with an coo ball constraint, the optimal first-order solution of the perturbation
V (c)
8 = [V)
(A2)
Assume the PGD method can well approximate the robustness loss, the difference of the robustness losses of the two models w and w + Aw on the path can be represented as
max (w Aw, + 5) max I (Mr, x: + [
lww + Aw, + )w+aw) - 1 (w, x 5%))
(A3)
(w Aw,x) + V.l (w + Aw, 2² V.l (w + Aw,
2 Vel (w Aw,z)"
Var (w + Aw,
Hw Aw
| (w + Aw,
2 Vall (w Aw,x)]l
V2ll (w xw
(10,0) + Vzl Vzl(w, (w,w) w)) T' V (w, .)
2² (al(wxr)"
V.l (w. x))
H
+ +O((2)
(A4)
V. (w,, x>)
2 VVll x]]
V x)]l
2 1ww + Aw, x) 1 xw,
V (w Aw, x1) V 2.l [|]
2
V-l (w + Aw,x)² H Hutaw
V-l(w + Aw,
Val 2w,
H V.l (w,
(AS)
2
VZll (w + A0,
[Vzl (w0 + Aw, x]
(w,,))
| (w, I|
2
Val (w + 2w,x))
V.l (w0 + Aw, x)
V.l (w. x)
H V (w,
Hm Am
(A6)
2
VVal (w + Aw,x))
V.l (w Aw,
V2z (w,x)
V (w,x)]
22

---

Published as a conference paper at ICLR 2020
The first equality uses the definition of optimal first-order solution of the robustness loss. Using equa-
tion A2 and Taylor series expansion on ((:,-) with respect to its second argument gives equation A4
Rearranging equation A4 and using Assumption (b) (i.c., ignoring the O(e*) term in equation A4
gives equation A5 Finally, Based equation A5 the term 7 (w Aw., x: (w, x) due to
Assumption (a). Then, applying Lemma to equation A5 gives equation A6
Without loss of generality, the following analysis assumes || || to be the Euclidean
and consider The results can generalize to other norms since the correlation is invariant to scaling factors. With assumption (c), for any input Hessian H and its largest eigenvector v, we have
the case when V_J(w,a)'v
Val (w, x)"
Vi (w,
H V (1, I]
(w, x)
V (w, x))
T
V.l (w,
v - 2
H
1 1++9 +
V el (w,r)l
| (w,
1 (w,x:) x²)
T
V.l (w,
[w,(w)))] - +2 (ol(w,)) T V (w, x)|
-
H
Hu Amax
V / (w. x)
(2c 1)Amax 112
(A7)
where h2 ( V.l(ww.))
H Val(w..) -2) and the last equality uses the fact that Hu [[V=l(w,,)]
Amax and assumption (c). Note that the lower bound is tight when and , (w, x) are collinear,
i... U [[, as in this case hg = and = 1. On the other hand, using the definition of
largest eigenvalue gives an upper bound
Val (w Aw, x)²
V (w Aw, x)
Amax-
(A8)
V (w Aw, x)
V (w + Aw, x)
Finally, applying equation A8 and equation 7
to equation A6 we have
[w + Aw, x 5) - max I (w, + 8) s jóllse
max
Dmax (t + At) - -2c - 1) 1) (t) - h2 (t)
Se
(A9)
and
max 1 (ww + Aw, 5) - max (w, + 6) 2 2 [|
[2c 1)Amax + At) -maxx (t) + he + At]]
[
(A10)
As v becomes more aligned with 2d (w, æ), c increases toward and by decreases toward 0, implying 2c 1. Therefore, in this case have max 1 (w Aw, x + 5) max (w,, 6)
|| linear function of Amax (t + At) Amax (t) and max I (w((t, x + 8) Amax (t). S
||
If the higher order term O in equation A4 is non-negligible, which means Assumption (b) does not hold, then the following analysis will have an extra offset term of the order O(e) as e increases toward 1, i.e., max ( (w((f, + 8) Amaxe (t) (ee).
Se
M ROBUST CONNECTION AND MODEL ENSEMBLING
Robust connection
To train a robust path, we minimize expected maximum loss on the path
like the following.
min (2-U((011
max T(ob (4), 8) dES
(A11)
23

---

Published as a conference paper at ICLR 2020
Connection of regular and adversarially -traindd modols (PCC=0.98)
Connection of
Connection regular models (PCC=0.98)
adversarially-trrinee
Legend
modcls (PCC=0.94)
largest eennvllue (log
largest el eigovaaloo (log
attnns SUCDESS ras (%) largesl eigenvalue (log
cross entopy loss
cross entrapy loss
cr0s8 entopy loes
eigenvalue
loss &
80
20
training loss
test loss
30
-10
robusiness loss
largest elgenvalue
-15
-10
0.5
0.5
attack success rate
allads sucsoss rate (8)
100
altack BUCO6D role (%)
error &
error rate (9S)
error rate (%)
8
erfor rate
trakiing
50
50
attack succosss rala
adversorial examples
0
0.5
0.5
0.5
Figure A14: Loss, enror rate, attack success rate and largest eigenvalue of input Hessian on the path connecting different model pairs on CIFAR-10 (VGG) using robust connection. The path obtained with robust training method. The error rate of training/test data means standard training/test error, respectively. There is still a robustness loss barrier between non-robust and robust models, but not there is robustness loss barrier between robust model pair and non-robust model pair (verified by small loss variance and flat attack success rate on the path). There is also a high correlation between the robustness loss and the largest eigenvalue of input Hessian, and their Pearson correlation coefficient (PCC) is given in the title of each plot.
where
S s= f0 x + 10, $c}
(A12)
To solve the problem, we first sample i from the uniform distribution U0, 1) and we can obtain the model (t). Based on the model da (?), we find the perturbation maximizing the loss within the range S,
max L (t), x 8)
(A13)
RES
We projected gradient descent method for the maximization.
8k+1 = II (5x + sgn (VoL(oe(f), x +)))
(A14)
where [[ denotes the projection to the feasible perturbation space S, and sgn(-) denotes element-wise sign function taking the value of either 1 or -1.
After finding the perturbation å maximizing the loss, we would minimize the expectation. At each iteration, we make a gradient step for 0 as follows,
e = 0 - VoL(oo(t),x
(A15)
We show the robust connection for pair of non-robust and non-robust models, a pair of non-robust and robust models, and a pair of robust models in Figure A14 We can observe that with the robust training method, there is still a robustness loss barrier between the non-robust and robust models. However, for the robust connection of robust model pair and non-robust (regular) model pairs, there is no robustness loss barrier, verified by small loss variance and flat attack success rate on the path. Our results also suggest that there is always loss barrier between non-robust and robust models, no matter using standard or robust loss functions for path connection, as intuitively the two models are indeed not connected their respective loss landscapes. Moreover, the attack success rate of adversarial examples are relatively small on the whole path compared with the robust connection of non-robust and robust models.
Model ensembling Here we test the performance of (naive) model ensembling against evasion attacks. Given two untampered and independently trained CIFAR- 10 (VGG) models, we first build a regular connection of them. Then randomly choose models the path (randomly choose the
24

---

Published as a conference paper at ICLR 2020
value of t) and take the average output of these models as the final output if given input image for classification. The adversarial examples are generated based on the start model (t = 0) or end model (t = 1) and we assume the attacker does not have any knowledge about the connection path nor the models on the path. We use these adversarial examples to test the performance the model ensembling with the models on the connection. The attack success rate of adversarial examples can decrease from 85% to 79%. The defense improvement of this naive model ensembling strategy is not very significant, possibly due the well-known transferrability adversarial examples (Papernol et al. 2016 Su et al. 2018). Similar findings can be concluded for the robust connection method.
N
EXPERIMENTAL RESULTS ON CIFAR-100
To evaluate the proposed path connection method on more complicated image recognition benchmarks, we demonstrate its performance against backdoor and error-injection attacks on CIFAR-I00 as shown in Figure A13 The experimental setting is similar to that of CIFAR-10, 1-, where the two end models are backdoored/error-ine inneeeeeeeed..eeedddd and the connection is trained with various bonafide data sizes. We can observe that method is still able to remove the adversarial effects of backdooring error-injection and repair the model.
For the evasion attack, investigate cases, 1) the connection of regular models and 2) the connection of regular and adversarially-traine models. The performance of adversarial training on CIFAR-100 100 is not significant that CIFAR-10. So do not investigate the connection of two adversarially-trainee models. Their loss and eigenvalue on the path are shown in Figure[A16] We can observe that there also a high correlation between the robustness loss (loss of adversarial examples) and the largest eigenvalue of input Hessian.
100
80
images training,
error rate (%)
error rata (%)
1000 images training cican erro
500 images training, dean error
60
images training.
50 images training, clean error
40
40
2200 images training, sttack enrr
1000 training, attack
20
20
images training, attack
250 images training, attack errrr
-50 images trainig, allack
0.5
0.5
(a) Model connection against backdoor attack
(b) Model connection against
(c) Legend
error- inoorinjecttoon attack
Figure A15: Error rate against poison and unror-injection attacks for CIFAR-100 (VGG).
O FINE-TUNING WITH VARIOUS EYPER-PARAMETTRR
We demonstrate the performance of fine-tuning with various section. For CIFAR-I0 (VGG), we perform fine-tuning with different learning rate and the number of total epochs with the bonafide data of 2500 images and 1000 images, respectively. The clean accuracy and attack accuracy are shown Figure A17 We also plot the clean and attack accuracy obtained through our path connection method from Table 2 in Figure A17 as reference.
configurations this
As observed from Figure A17 (a), larger learning rate (such as 0.05) can decrease the attack accuracy more rapidly, but the clean accuracy may suffer from relatively large degradation. Small learning rate (such as 0.01) can achieve high clean accuracy, but the attack accuracy may decrease with a much slower speed, leading to high attack accuracy when fine-tuning stops. This is more obvious if we use less bonafide data (reducing data size from 2500 to 1000) as shown in Figure(A17 (b). Fine-tuning performs worse with lower clean accuracy and higher attack accuracy. Since fine-tuning is quite sensitive to these cyppe-parraetert we conclude that it is not easy to choose an appropriate learning rate and the number of fine-tuning epochs, especially considering the user is not able to observe the attack accuracy in practice. On the other hand, in Figure (A77 our path connection method can achieve the highest clean accuracy. In A17 (b), although the clean accuracy of br 0.01 is
25

---

Published as a conference paper at ICLR 2020
Connection of regular and adversarially-traine models (PCC=0.98)
Connection regular models
Legend
(PCC=0.74)
25
largest eigenvalus (log)
25
eigenvalue
entropy loss
20
-155
cross entropy loss
20
largest eigenvalus
loss &
15
training loss
test loss
10
2.5
10
robustness
3
largest eigenvalue
t
40
50
100
attack success rate
sttack success rate (%6)
error rate &
30
90
success
training data lest data
80
error
20
attack success of
10
edversarial examples
70
Figure A16: Loss, error rate, attack success rate and largest eigenvalue input Hessian on the path connecting different model pairs on CIFAR-100 (VGG) using standard loss. The error rate of training/test data means standard training/test error, respectively. In all cases, there is no standard loss barrier but robustness loss barrier. There is also a high correlation between the robustness loss and the largest eigenvalue of input Hessian, and their Pearson correlation coefficient (PCC) reported in the title.
higher than that of path connection, its attack accuracy remains high (about 40%), which is much larger than that of path connection (close to 0%).
100
100
deen scc
Accuracy (96)
80
Accuracy (96)
80
(r-0 02,02, clean acc
00.015 cicon acc
60
60
0.005, clean uoc
path connection. clean acc
40
40
0p=0.015, atlack soc
20
1r= 02, attack anc
1r=0 05, attack eoc
path connection. attack aod
epoc
epoc
(a) fine-tuning with 2500 samples
(b) fine-tuning with 1000 samples
(c) Legend
Figure A17: Test accuracy and attack accuracy for fine-tum fine-tuning on CIFAR-10 10 (VGG).
P STABILITY ANALYSIS
In Appendix E, we perform multiple runs for the Gaussian noise experiment and only report the average accuracy. The variance is not reported since the average accuracy is able to demonstrate that the Gaussian noise method is not good choice for removing adversarial effects.
To investigate the stability of the path connection method with respect to every possible factor, it will cost considerable amount of time and resource to run all experiment setups considering the various attack methods, datasets, and model architectures. So here we mainly perform one representative experiment setup with multiple runs and show their mean and standard deviation.
Figure A18 shows the error bars of the error rate computed over 10 runs for path connection against backdoor attack. The dataset CIFAR-10 and the model architecture is ResNet. For each bonafide
26

---

Published as a conference paper at ICLR 2020
data size, we train 10 connections with different hyper-parameter settings, that is, starting from random initializations and using various learning rates (randomly set learning rate to 0.005, 0.01 or 0.02). Their average value and standard deviation are shown in Table A3 We observe that although the connection may start from different initializations and trained with different learning rates, their performance the path close with relatively small variance, demonstrating the stability of our proposed method.
inages tinning M00 vags Luriig ralraagee tarring
100
etroy rate (96)
error rate (%)
60
images tnsine
toklie
40
40
20
20-
(a) test error bar
(b) attack error bar
Figure A18: Error rate against backdoor attack for CIFAR-10 (ResNet).
Table A5: Performance comparison of path connection against single-target backdoor attack. The clean/backdoor accuracy means staddad-eesacccmmmaaattkttuuuess respectively.
data size
2500
1000
500
250
= 0.2
874-0.21 %
$3.2
80.4 =0.72%
78.4 +0.82%
+1.9%
Clean Accuracy
80.510.37%%
73.5:10.19%
67.711.0%
62.111.0%%
40.5:12.55
0.6
78.8+0.16%
71.6+0.7%
64.8+1.3
58.8+1.4%
36.9+11.4
CIFAR-10 (ResNet)
0.8
85.4 +0.25%
81.8=0.35%
78.8=0.84 %
1..8
64=2.9%
== 0.2
1.2.0..77 %
1.610.54%
1.3.0.64
1.911%
2.610.54 %
Backdoor Accuracy
1 0.4
0.8+0.18 %
1.3++0.32%
1.6+0.54%%
1.8++0.9%
3.2 +.2+1.4%
f = 0.6
1.0=0.16%
1.5:0.411%
1. 1.6.2..42%
1.610.57%
3,24
1 0.8
0.3.1.37%%
0.86
1.0.10.78 0..00.78 %
1.1.1..52%%
1.8.1..3%
Q STRATEGY TO CHOOSE THE PARAMETER t
In our proposed method, we need to choose model on the path as the repaired model, that is, choosing the value of t. For different datasets/modell, the best choice of may vary. So discuss some general principles to choose an appropriate in this section.
We note that in practice the user is not able to observe the attack error rate the user does not have the knowledge about certain attacks. If the user is able to observe the accuracy on the whole clean test set, we suggest the
choose the model (a value of l. [0, 1) with test accuracy where a is the accuracy of the end model and Aa represents threshold. Based on the performance evaluations (Figures 2 A4 AS and A6). setting Aa 6% should be an appropriate choice, which is able to eliminate the effects of all attacks without significantly sacrificing the clean accuracy.
Aa,
If the user is not able access the accuracy of the whole clean test set, the user has to choose only based on the bonafide data. In this case, we suggest the user to use the k-fold cross-validation method assess the test accuracy. This method first shuffles the bonafide data randomly and splits into k groups. Then one group is kept to test the accuracy the learned path and the remaining k groups are used to train the path connection. The process repeated for each group. We perform additional experiments with the 5-fold cross validation method for CIFAR- 10 (VGG) and SVHN (ResNet). The average validation error rate on the hold-out set and attack error rate against backdoor attack is shown in Figure A19 The error-injection attack is easier to counter and hence do not explore the error-ineection experiments.
We observe that since the validation data size reduces much smaller value (one fifth of the bonafide data size), the test error rate becomes less stable. But generally follows the trends of the test error rate the whole clean test set (see Figure A4). So by utilizing the k-fold cross-validatee validation
27

---

Published as a conference paper at ICLR 2020
method, obtain the test accuracy the limited validation set. Then the aforementioned threshold method can be used to choose the model on the path. Here we suggest setting Aa to 10%, more conservative threshold than the former case, the user now does not has access to the accuracy of the whole test set. We also note that the performance of bonafide data with 50 images has large larger bonafide data size with the k-fold
suggest deviation from other data size settings, cross validation method.
2500 images, validation error
error rate (%)
80
1
error rate (%)
80
1000 images, validation
500 images, validation error
60
60
images, validation
images, validation error 2500 images, allack error 1000 Images, ettack 500 images, attack error
40
40
20
20
250 images, attack error
0.5
0.5
images, allack error
(a) Model connection for CIFAR-10 (VGG)
(b) Model connection for SVHN (ResNet)
(c) Legend
Figure A19: Average 5-fold cross validation and attack error rate against backdoor attack.
28


Reviews:
[{"review_id": "rJxEPld6KB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies leveraging mode connectivity to defend against different types of attacks, including backdoor attacks, adversarial examples, and error-injection attacks. They perform a comprehensive evaluation to show the benign test accuracy and attack success rate over the models in the connected path between pairs of models with the same or different properties, e.g., both are attacked, both are benign, one attacked and one benign, etc., where the connected path is learned using existing algorithms to find the high-accuracy path over two different models. Their evaluation suggests that in certain attack scenarios, exploring the mode connectivity could help find a model that has a high benign accuracy, while with a significantly lower attack success rate than the models at the end points.\n\nIn general I like this paper. Although mode connectivity has been studied in the literature, to my knowledge, this is the first work to extensively study this topic in the context of attacks. An interesting part of the paper is their evaluation on backdoor attacks, where they show that with a very small number of training samples for fine-tuning, they are able to find a model with decent test accuracy, while the watermarks are removed. Meanwhile, the attacks studied in this work include various settings, i.e., poisoning attacks, error-injection attacks, evasion attacks, and also adaptive attacks where the adversary knows that the defender will use path connection to improve the robustness. These make this paper a good reference as a systematic study of their proposed topic.\n\nHowever, my main question is that while the algorithm is pretty effective to defend against backdoor attacks and error-injection attacks, the results of evasion attacks are somehow negative. While it is helpful to show negative results if this is indeed the case, do you have some possible explanation why the models on the path are less robust than the two end models?\n\nAnother concern is that while the benign test accuracy of the model found by their algorithms looks good, both CIFAR-10 and SVHN evaluated in this paper have a small label set and may not be challenging enough. It would be great if the authors can provide some results on a more complicated image recognition benchmark, e.g., some intermediate-level datasets studied in previous work on attacks such as CIFAR-100.\n\n-------------\nPost-rebuttal comments\n\nI thank the authors for clarifying my questions, and I keep my original score.\n------------", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "rJeeI-l2FH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an adversarial defense method based on mode connectivity. The goal of the method is to repair tampered networks using a limited number of clean data examples. The authors consider two types of adversarial attacks: backdoor attacks and error-injection attacks. The proposed method takes two potentially tampered networks, then constructs a low-loss path connecting the weight vectors of the given models in the space of network parameters (the path is constructed using the small set of clean data examples), finally an intermediate point on the path is used as a weight vector corresponding to the “repaired” model. The authors analyze the properties of the paths and show that intermediate points on the mode-connecting paths deliver both high clean-data accuracy and low attack success rate. In the experiments the proposed method shows better results compared to baseline defense techniques including fine-tuning, training from scratch, and pruning followed by fine-tuning. The paper also analyzes evasion adversarial attacks from the perspective of mode-connectivity and observes the existence of barriers in the landscape of robustness loss on the paths connecting regular and adversarially-trained models.\n\nI would like to note that the text of the paper is well-structured and clear. Another strength of paper is that the authors consider multiple supporting experimental settings and extensions of the proposed method. The considered cases include adaptive attacks (the settings in which the attacker is aware about the employed defense method).\n\nAlthough there are many strong sides of this paper, I have identified several serious flaws in the justification of the approach and the results. Thus, I consider the paper to be marginally below the acceptance threshold. I am willing to increase the score if the authors address my concerns expressed below.\n\n1) What is the conceptual difference between using mode-connectivity and fine-tuning?  Both methods try to find a point in the weight space which (1) is close to the initial point and (2) delivers low loss on the bonafide data. What feature of the proposed method can explain its success? The “technical explanations” for the effectiveness of the proposed path connection method only show that the random sampling procedure fails to find models with high-clean-accuracy and low-attack-accuracy. However, this doesn’t necessarily mean that fine-tuning (which performs directed search) cannot find models which perform similarly to the models found by the proposed method (potentially the hyperparameters of the fine-tuning procedure can be tuned better). Given the aforementioned similarity of the proposed approach and fine-tuning, the explanation (either theoretical or empirical) for the effectiveness of the proposed method has to be provided in order to motivate the methodological proposal of the paper.\n\n2) Please report the error bars of the accuracy computed over multiple runs of the experiments (according to Appendix E multiple runs were performed, but only the mean values of the accuracy are reported). Including the error bars would help to quantify the statistical significance of the results.\n\n3) What strategy was used to choose the parameter t? The parameter has different values depending on the model/dataset. How should this value be chosen in general?\n\n4) In the formulation of Proposition 1 it is first stated (assumption (c)) that the directions of the gradient w.r.t. x and the vector v are aligned (the normalized dot product of the vectors is >= c), however then the proposition requires that the vectors must be similar. Why assumption (c) has to be included in this formulation, if the proposition also requires a stronger condition? Either the assumptions or the claim of the proposition has to be formulated clearly. For example, it can be explicitly stated that the proportionality holds approximately (up to terms which go to 0 as c goes to 1). I also recommend to include a term which accounts for the tail of the Taylor expansion in equation A4 instead of using the “approximately equals” sign. Moreover, I recommend to include the discussion of the effect of this tail term on the proof and the statement of the preposition.  \n\nAdditional minor comments, which do not affect the assessment:\n\nIn Section 3.1, paragraph 3 the description of Figure 1 reads as follows: “our path connection is trained using different portion of test data”. Another sentence below in the paragraph: “For example, path connection using merely 1000/2500 CIFAR-10 samples from the test set only reduces the test accuracy of VGG16 models by at most 10%/5%”. Which part of the dataset was used for finding the path connections? How the “test data”, which is claimed to be used for finding paths, is different from the test set used to evaluate the accuracy of the models? \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xXGumycH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper repurposes results on mode connectivity (that is, minimal loss paths between local optima) to improve adversarial robustness. The idea is new (to me) and appears both elegant and effective. The approach provides a fast way to repair models that have been attacked. \n\nMore broadly, mode connectivity provides a powerful window into adversarial robustness and model choice. It constructs approximately optimal paths between models, where the path depends on a choice of loss. It thus allows to investigate how various models relate to each other from the perspective of various losses. The paper barely scratches the surface of this approach. \n\nComments:\nThe claim that all models on paths have similar test losses (bottom p8) seems a bit of a stretch given the bottom panels (middle and right) of Figure 4.\nThe paper is difficult to read; it somehow feels quite disjointed.\nProposition 1 is difficult to parse.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates improving robustness to adversarial examples by using mode connectivity in the loss function. The paper received three reviews by experts working in related areas. In a strongly positive review, R1 recommends Accept, but gives some specific technical questions. The authors submitted a response to these questions; in post-review comments, R1 was satisfied and maintained the highly positive review. R2 recommended Weak Reject and also asked specific technical questions, including some additional details on experiments, statistical significance, etc. The author response also convincingly responded to these concerns. R3 recommended Weak Accept but suggested improving the writing, which authors have done in their revision. Given that R1 and R3 are highly positive and R2's concerns were addressed in the response and revision, we now recommend (weak) Accept.
Decision: Accept (Poster)