Sharpness-Aware Data Poisoning Attack
Authors: Pengfei He
Han Xu
Jie Ren
Yingqian Cui
Shenglai Zeng
Hui Liu
Charu C. Aggarwal
Jiliang Tang
Venue: ICLR 2024 spotlight (2024)
Keywords: Data poisoning attack; generalization; deep learning
Primary area: societal considerations including fairness, safety, privacy

Abstract: Recent research has highlighted the vulnerability of Deep Neural Networks (DNNs) against data poisoning attacks. These attacks aim to inject poisoning samples into the models' training dataset such that the trained models have inference failures. While previous studies have executed different types of attacks, one major challenge that greatly limits their effectiveness is the 
uncertainty of the re-training process after the injection of poisoning samples. It includes the uncertainty of training initialization, algorithm and model architecture. To address this challenge, we propose a new strategy called **Sharpness-Aware Data Poisoning Attack (SAPA)**. In particular, it leverages the concept of DNNs' loss landscape sharpness to optimize the poisoning effect on the (approximately) worst re-trained model. Extensive experiments demonstrate that SAPA offers a general and principled strategy that significantly enhances various types of poisoning attacks against various types of re-training uncertainty.

Published as a conference paper at ICLR 2024
SHARPNESS-AWARY DATA POISONING ATTACK
Pengfei He', Han Xu', Jie Ren', Vingqian "un',,ill Zeng', Hui Liu', Charu C. Aggarwal", Jilliang Tang'
!Department of Computer Science and Engineering, Michigan State University
"IBM T. J. Watson Research Center, New York
1 '[hepengfl, xuhanl, renjie3, cuiyingg, zengshel, 1iuhui7, cangjili j jmsu. edu Pcharu@us ibm. com
ABSTRACT
Recent research has highlighted the vulnerability of Deep Neural Networks (DNNs) against data poisoning attacks. These attacks aim to inject poisoning samples into the models' training dataset such that the trained models have in- ference failures. While previous studies have executed different types of attacks, major challenge that greatly limits their effectiveness is the uncertainty of the re- re-training process after the injection of poisoning samples. It includes the uncer- tainty of training initialization, algorithm and model architecture. To address this challenge, we propose a new strategy called "Sharrpess-Awee Aware Data Poisoning Attack (SAPA)". In particular, it leverages the concept of DNNs' loss landscape sharpness to optimize the poisoning effect on the (approximately) worst re-trained model. Extensive experiments demonstrate that SAPA offers a general and princi- pled strategy that significantly enhances numerous poisoning attacks against vari- ous types of re-training uncertainty.
INTRODUCTION
The rise of machine learning (ML) models that collect training data from public sources, such as large language models (Brown
2020 Radford et al.) and large visual models (Ramesh al., 2021 Radford et al. 2021). highlights the need of robustness against data poisoning attacks (Stein- hardt et al., 2017; Shafahi al., 2018: Chen et al., (017). Data poisoning attack refers to the threats of an adversary injecting poisoming data samples into the collected training dataset, such that the trained ML models can have malicious behaviors. For example, by injecting poisoning samples, the adversary's objecttv is to cause poisoned model has poor overall accuracy (known as un- targeted
attacks (Steinhardt et al., 2017 Li et al.) [020b]] Ren et al. (2022) ) or misclassifies a specified subset of test samples (known largeted attacks (Shafahi et al. 2018 Zhu et al. (2019)) Additionally, in buckdoor attacks (Chen et al. 2017; Saha et al. 2020 Tran et al. 2018)), the adversary aims to cre- ate "backdoors" in the poisoned model such that the model gives a specific output as the adversary desires if the backdoor trigger is presented, regardless of the actual input.
Many poisoning attacks in Deep Neural Networks (DNNs) face common obstacle that limits their effectiveness, which is the uncertainty of the re-training process after the injection of poisoning samples. This challenge is also highlighted in previous studies such (Schwarzschild et al.) 2021
Zhu et al., 2019 Huang et al. [0200] Geiping et al. 2020). In particular, most existing methods (as reviewed in Section and Appendix BD generate poisoning samples based the effect of only one victim model (Shatahi et al., 2018 Souri et al. 2022). However, on the poisoned dataset, the re-trained model may converge to very different point due to the re-training uncertainty. such as training algorithm, model initialization and model architectures. As a consequence, the injected poi- soning samples could lose their efficacy and the poisoning effect is compromised. To overcome this difficulty. recent methods (Geiping et al. 2020 Huang et al.) 2020) have introduced the Ensemble and Re-initialization (E&R) Strategy: which proposes to optimize the average poisoning effect of multiple models with various model-archirectures and initializations. However, this method tends to be computationally inefficient (Huang et al.] 2020). In general. if we consider the poisoning attacks general bilevel optimization problems (Bard) 2013 Colson et al. 2007) (Eq. 2 in Section 3). poisoning problems in DNNs can be categorized as "multiple inner minima" bilevel optimization
problems Sow al. 2022
Liu et al. 2020 Li et al. 2020a). Prior works (Sow et al. 2022 Li

---

Published as a conference paper at ICLR 2024
et al. 2020a Liu al. 2020 2021 have demonstrated the theoretical difficulty and complexity of solving "multiple inner minima" problems.
In this work, we introduce new attack strategy, Sharpness-Aware Data Poisoning Attack (SAPA). In this method, we aim to inject poisoning samples, which optimizes the poisoning effect of the approximately "worst" re-trained model. In other words, even the worst re-trained model (which achieves the relatively worst poisoning effect) can have a strong poisoning effect. Furthermore, we find that this strategy can be successfully accomplished by tackling the victim model's loss landscape sharpness (Foret et al., 2020). Notably, the loss landscape sharpness is more frequently used to ex- plain the generalization characteristic of DNNs. In our work, we show the possibility to leverage the algorithms of sharpness (such as (Foret et al.) (0200) to advance poisoning attacks. Through exper- imental studies, we show that our proposed method SAPA is a general and principle strategy and it can improve the performance of various poisoning attacks, including targeted attacks, un-targeted attacks and backdoor attacks (Section 5. | 5.2% (.3) Moreover, we show SAPA can also effectively
improve the poisoning effect under various re-training uncertainties, including for re-training algo- rithm as well as model architectures (Section 5.4). Compared to the E&R strategy (Huang et al.
2020), we show SAPA is more computational efficient (Section 5.5).
RELATED WORK
2.1 END-TO-END DATA POISONING ATTACKS
Data poisoning attacks (Biggio et al.) 2012 Steinhardt al., 2017) refer to the adversarial threat during the data collection phase of training ML models. These attacks manipulate the training data so that the trained models have malicious behaviors. Common objectives of poisoning attacks include the purposes of causing a poisoned model to have poor overall accuracy (un-targeted at- tacks (Steinhardt et al. 2017 Li et al. 2020b Ren et al. (022)). misclassifying a specified subset of test samples (targeted attacks (Shafahi et al. [0188] Zhu et al. (019)). insert backdoors (backdoor attacks (Chen et al., 2017] Saha et al. 2020; Tran et al. 2018)) Notably, in this work, we focus on the poisoning attacks in "end-to-end" manner, which means that the vietim model is trained on the poisoned dataset from scratch. is a practical setting for poisoning attacks as the attackers cannot take control of the re-training process.
In general, the feasibility of poisoning attacks highly depends on the complexity of the victim model. For linear models, such logistic regression and SVM, there are methods (Biggio al., 2012 Steinhardt et al.] 2017] Koh et al.] (2022) that can find solutions close to optimal ones. For DNNs, the exact solution is generally considered intractable, due the complexity and uncertainty of the re-training process. In detail, in the end-to-end scenario, the victim model can be trained with a great variety of choices, including various choices of model architectures, initialization, and training algorithms, which are hard to thoroughly consider while generating the poisoning samples. Faced with this difficulty, many works (Shaban et al. 2019) focus on the transfer learning setting (Shafahi et al. 2018 Zhu et al. 2019), where they loosen the "end-to-end" assumption. They assume the attacker has knowledge of pre-trained model, and the victim model is fine-tuned on this pre- trained model. However, these attacks are usually ineffective in the end-to-end setting. Recent works such as Gradient Matching (Geiping et al., 2020) make a great progress under end-to-end setting. However, these attacks still face obvious degradation when there are re-training variations such as model architectures. To further improve the attacks, |Huang et al. (2020) devise the "ensemble and re-initialization" strategy to take several victim models into consideration. However, it is time and memory-consuming, and cannot sufficiently cover the re-training possibilities. In Appendix B we provide more comprehensive review of existing poisoning attacks. In Appendix [B.2] we introduce some poisoning attacks which consider the scenario different from end-to-end scenario.
2.2 Loss LANDSCAPE SHARPNESS
In this paper, our proposed method involves calculating and optimizing the loss landscape sharpness of DNNs. The notion of the loss landscape sharpness and its connection to generalization has been extensively studied, both empirically (Keskar et al.) 2016 Dinh et al., 2017) and theoretically (Dz-) jugaite & Roy 2017 Neyshabur et al. (017)) These studies have motivated the development of methods (lzmallvv et al. 2018: Chaudhari et al., 2019) that aim to improve model generalization by manipulating penalizing sharpness. Among these methods, Sharpness- Aware Minimization (SAM) (Foret et al. 2020) has shown to be highly effective and scalable for DNNs. In this paper,
explore the use of sharpness for data poisoning attacks.
2

---

Published as a conference paper at ICLR 2024
3 PRELIMINARY
In this section, we introduce the definitions of the (loss landscape) sharpness, as well as the for- mulations of several most frequently studied data poisoning attacks. We start by providing some necessary notations. In this paper, we focus on classification tasks, with input I E X and label y E y following the distribution D which is supported on X' y. Under this dataset, a classifica- tion model f:; 8) X - Y is trained on the training set Drr = {z y:), = l....n]. n., whose n data samples follow D. The model parameter is from the parameter space O We define the loss function as [[[(2; 0), v). and the (training) loss as L(0; Drr) _ E ((f(;;;; (1);;
3.1 Loss LANDSCAPE SHARPNESS
We follow the precedent work (Foret et al. 2020) to define the loss landscape sharpness (refer to as measures how quickly the model's training loss can be increased by moving its parameter to nearby region. Following Eq.[]] it calculates the training loss increase after the model parameter is perturbed by v. whose 1p norm constrained by ||e||p S p:
"sharpness" for simplicity) as Eq. Il
5 D.-)
max S1
[( + Dr.)) L(0; Dzr)-
In this work, we focus on the definition of sharpness as in [q Note that there are other sharpness definitions (Andriushchenko & Flammarion| (022), we will investigate them as one future work. Given this definition of sharpness, previous works (Foret et al. 2020 Wen et al., (023) build the theoretical and empirical connections between sharpness and model generalization. Generally, smaller sharpness indicates a better generalization performance.
3.2 DATA POISONING ATTACKS
In this subsection, we give the definition of poisoning attacks we studied. Given a training set Der with n samples, we assume that the attacker selects subset D, from Day which takes an E [0, 1] percentage of Dtr. and replaces it with a poisoning set Dp. Usually, the samples in D, are from modifying samples in Dg- During the re-training stage, the model trained from scratch on the perturbed training set (Der Dg) D, and we denote it as "Dur + Dp" for simplicity. Although different attacks may have different purposes and formulations, we give a general formulation as in Eq[: the attacker aims to find the poisoning samples Dp, such that the trained model has an optimized malicious behavior towards data samples from victim set Dy:
min Q(0", Dr), s.t. 0" = argmin min L(e; Der + Dp) Dy
0
where C denotes the constraint of the poisoning samples. Notably, the design of "adversarial ob- jective" Q(-) and "victim set" Dr is based on the purpose of the adversary. By giving different formulations of Q(-) and definitions of Dr, the attacker can achieve various adversarial goals. For example, in targeted attacks (Shafahi et al.] [2018) Geiping et al. (020)) they aim to cause the model to misclassify a certain test sample (or set of test samples). Thus, they define the victim set Dr = (xxt ( ) and the adversarial objective as:
('",", Dr)=
Σ [[f(;;))) (,) (Z.BI))DT
where y nbi a designated class different from the In Appendix B. we provide detailed intro- duction to the threat models of two other types of poisoning attacks, such as backdoor attacks (Wu lt al. 2022) and un-targeted attacks (Huang et al. 2021
4 METHOD
In this section, we introduce our proposed method Sharpness-aware Data Poisoning Attack (SAPA). In Section 4. will first introduce SAPA in general form. In Section 4.2 introduce how to achieve SAPA in targeted and backdoor attacks. In Section 4.3] we introduce SAPA in untargeted attacks. From our discussion, we show the idea of leveraging sharpness is general and lightweight to be incorporated into various types of poisoning attacks.
4.1 THE OBJECTIVE OF SAPA
As discussed in Section D & 2 one major challenge for poisoning attacks originates from the exis- tence of multiple solutions during re-training. Therefore, we first define the Poisoned Model Space,
3

---

Published as a conference paper at ICLR 2024
denotes 2p which is the set of all models that are trained on poisoned dataset Dor Dp and have a small training loss:
= [0: L(,: + Dp) -T
(4)
In practice the space Op composes the models that are trained from different training algorithms. initializations and architectures. In the space 2g. we desire find the Worst-case Poisoned Model, which is the model that has the worst poisoning effect = arg max Q(0; Dr). Ideally, we
aim to optimize the poisoning effect of 0' to overcome the re-training uncertainty:
min Q(0'; D
(5)
where 0' = arg max Q(0; Dr), s.t. L(0; Der + DD)
(6)
By solving this problem, we can find poisoning samples Dr such that the worst poisoned model 0' can have an optimized poisoning effect (with small Q(''; D).)). Therefore, for the models that are re-trained on the poisoned dataset, the poisoning effect is likely to persist, and our experiments in Section provide empirical evidence. Admittedly, is also hard to solve the above problem exactly. As solution, we propose to approximate the term Q(0'; Dr), by leveraging the existing notion of model sharpness (Foret et al. 2020). In detail, given a fixed Dp. we approximate Q("'; Dr) by:
Q(0'; Dr) max Sp
Q(0" + U; Dr), where 0- = arg min L(0; Dtr + Dp)
(7)
AEE
In details, we first trained the model 0 on (Drr + D2) (i.e., via ERM), so that 0" satisfies the constraint L(0"; Der + DD)) S T. Then, we locally maximize the term Q(0" + v) by perturbing the model parameter 0* with vector (which has limited norm ||e||p
p). In this way. the
perturbed model 0 + v has a worse poisoning effect compared to 8", but it is still likely to fall into the space Op (because 0 + is not far away from 0*). Remarkably, the form of the objective Eq. 7 resembles the definition of model sharpness in Eq. T. while we focus on adversarial loss which is distinct from the sharpness on the standard loss landscape. Therefore,
call the term in Eq. 7 as Sharpness-aware adversarial objective, and we desire to find Dp to optimize this objective:
min max Q(0" + u; Dr), s.t. 0" = arg min L (0; Dr, + Dp)
(8)
D2 10P..SP
AEE
In general, the formulation of SAPA can be seen modification of existing attacks: it replaces the adversarial objective (E of existing attacks with Sharpness- awwre aware adversarial objective. In- tuitively, compared with the traditional adversarial objective (Eq. 2 in SAPA, the perturbation on the inner minima 8" with v enables the algorithm to escape from the poisoned models where the poisoning effect is unstable. Thus, SAPA has the potential induce the re-trained models to have a stable poisoning effect under the uncertainty of the re- re-training process. Notably, in our proposed approximation Eq 7 we locally search for worst model with the same model architecture of a trained model 8* which does not explicitly consider the uncertainty of different architectures. However, empirical results in Section [5.4] demonstrate that this strategy can also help improve the poisoning effect under if there is model architecture shift.
4.2 SAPA IN TARGETED /BACKDOOR ATTACKS
In this subsection, we take one representative algorithm (Geiping et al., 2020) in targeted attacks as an example, to show SAPA can be incorporated into existing attacks without requiring additional attacker's 's knowledge or computational overhead. In the work Gradient Matching (Geiping et al.) 2020), they assume the attacker is targeting on a victim set Dr, and the attacker has other informa- tion including a pre-trained model 0", which is trained on (or part of) the clean training set Drr and a loss function L(). In their attack, they solve the following problem to find the set D, satisfying:
(VoQ(O"; Dr). VOL D,r + D,)) arg min (1- I[V Dr)ll I[Vo D:r + D,)1l Dp
99)
It finds poisoning set Dp. such that the gradient of the training loss, VoL(O"; Drr + Dp), has a maximized alignment with )rr) In this way, during the re-training process, the model is updated by (oL D,r + Dp)) and likely to minimize Q(8"; Dr) and achieve the attack goal successfully. In our method SAPA, we denote the sharpness- -wwre poisoning objective as Q3 Q5(00;; DT) = max vp Q(0" + v; Dr) for simplicity, the objective of SAPA is to solve:
Dr).
(V.Q² (0"; Dr), VoL Der + Dp))
arg min (1 D3
(10)
Vo0 Dr)ll [V Dar + Dp)
4

---

Published as a conference paper at ICLR 2024
which directly replaces the adversarial objective in Eq 9 to be sharpness aware adversarial objective in Eq [0] In our algorithm to calculate the term V (8:; Dr). we follow the approach in the pre-
vious work (Foret et al. 2020) to first approximate Qo (0); Dr) by leveraging first-order method for 0:
é = p sign VoQQ(0" (+) [+) (IV (T)
(11)
where 1/p + 1/9 = and we consider = 2 as illustrated in Foret et al. 2020) if not specified. Then, we can have the approximation to calculate (0Q"(0"; Dr) via replacing H with 0* + é:
(0"; Dr) Q0Q(O; Dr) Dr) 10-00+0
(12)
In this way, by fixing (oQ we can solve Eq 10 find the poisoning samples Dp via gra- dient descent. Notably, the computation in Eq [ does not require additional attacker's 's knowl- edge compared to original Gradient Matching. In Appendix C.I] we provide the detailed algorithm of SAPA in Gradient Matching. We also analyze its computational efficiency to show SAPA does not introduce great computational overhead. Besides, the similar strategy of SAPA and algorithm can also be applied in backdoor attacks, given the similarity between the objectives and viable solutions of targeted attacks and backdoor attacks (see more discussions in Appendix C.2).
4.3 SAPA IN UN-TARGETED ATTACKS
For untargeted attacks, similar strategy to replace the (investigated) model with a worse model during generating the poisoning samples also helps. Note that conducting un-targeted attacks for DNN models with small poisoning budget e is usually considered to be a hard problem (Muñoz-Gonzálezz
al. 2017). Existing feasible solutions (Fowl et al. 202T Ren et al. 2022] are motivated from a "data protection perspective" (which are often referred as "un-learnable examples"). In detail, they perturb the whole (or at least large portion of) training set, such that the trained classifiers cannot effectively learn the true data distribution and have a low accuracy on the true data. Take the method Error-Min (Huang et al. 202T) as an example, to induce the model to have a low accuracy, generates "shortcut" perturbations ó; for each training sample to solve the following bilevel problem, such that the model trained on this perturbed dataset has a minimized loss:
min min GEO (6:).-1.....m
Σ (E..:)ED,
1(5(s; + 0):0)))))
(13)
As consequence, the found 6, via solving Eq. 13 has patterns with a strong correlation to the labels yi- The models trained on the perturbed dataset can predict (a; + 6,) to be yi mostly based on the information of 8; and prohibit the model to learn useful knowledge from the clean samples in Dtr. In the original algorithm proposed by (Huang et al., (022T), the problem in [3 is solved by updating the model parameter 0 and training set perturbation {ō. )= alternatively. See Algorithm (without SAPA), they first update the model parameter 0 with M steps (step 1) and then update data perturbation 6, for T steps (step 3). For our method, similar to the high-level idea as in targeted attacks, SAPA introduces one more step (step 2) to replace the model 8 with f + 0 during generating the poisoning samples. Notably, similar strategy also be adapted other un targege attacks, i.e., Error-Max Fowl et al. (202T). We provide its detailed algorithm in Appendix C
EXPERIMENT
In this section, we conduct experiments to show that SAPA can improve existing methods when is applied to targeted, backdoor and un-targeted attacks in Section 5.1 5.2 and 5.3 respectively. For all methods in experiments, we obtain the poisoning samples via ResNet18 models. We report the results when the re-training is under ResNet18 and ResNet50, to have a thorough comparison with previous works (many baseline methods are also evaluated in the same setting). In Section [.4 evaluate the robustness of SAPA under various re- re-training settings including different training algorithms (including some defense strategies) and several more model architectures. In Section 3.5) we compare SAPA with baselines incorporating ensemble and re-inifialization technique to illustrate the superiority of SAPA in both efřectiveness and efficiency. Through this section, we focus on image classification tasks on benchmark datasets CIFAR10 and CIFAR100. Meanwhile, we provide additional empirical results of the dataset SVHN in Appendix D For our method SAPA, we set the radius p (Eq. X in Section 4 to be 0.05 in all experiments. We provide detailed implementation in
Appendix C for all SAPA- based al
Code available in http:///gthhbboooo///eettltlltte
5

---

Published as a conference paper at ICLR 2024
Algorithm/ 1: Error-min+SAPA
Input : Clean training set { /=) Optimization step and M; epochs E.
Output: Sample-wise perturbation Dp {6, 16.
Randomly initialize perturbation Dp = {6,) for epoch in 1, E do
1. for m in 1,.., M do
Update via minimizing Elsw,) (f(az + 8;; v1)) end
SAPA: Fix 0, Dp, find worst direction è to maximize Σ E(Eon)ED Uf(a, + 6;;0 0), v.)
for t = I..,T steps do
Update 5, via gradient descent on the fixed model f(:0+ 60)
end end
5.1 PERFORMANCE OF SAPA IN TARGETED ATTACKS
Experiment Setup. In this experiment, we focus on targeted attack goal to cause the poi- soned model to misclassify one victim sample. During attacks, under each dataset, we assume that the attacker randomly chooses small proportion of training dataset with "poisoning ratio" e 1%, 0.2%. and inserts unnoticeable perturbations (whose Loo norm is limited by "perturbation budger" 16/22 (6/2555, 8/255, 4/255) on each of them. After crafting the poisoned dataset, the model is randomly initialized and re- re-trained from scratch, via SGD for 160 epochs with an initial learning rate of 0.1 and decay by 0.1 at epochs 80 and 120. For each setting, repeat the experiment 50 times and report the average performance. Multiple-target attacks are also considered in Appendix [ More details of the implementation can be found in Appendix C.I
Baselines. We compare SAPA with representative baselines, including BolllseyeAAhaahine et al. 202T). Poison-Frog (Sisonn et al. 2018). Maett-Poison (Huang al., (220), and Grad-Match (Geip- ing et al. 2020). Notably, our method is incorporated to the method Grad-Match (Geiping et al., 2020). For other baselines, MetaPoison leverages meta learning (Vilalta & Drissi (002) unroll the training pipeline and ensembles multiple models. Poison-frog and Bullseye, also known as "fea- ture collision", generate poisoning samples with representations similar to those of the victim. In Appendix
B provide detailed discussions of these baselines.
Performance comparison. In Table We report the "Success Rate" which is the probability that the targeted sample is successfully classified be the designated wrong label yoh From the results, can see that SAPA consistently outperforms baselines. The advantage of SAPA is obvious, especially when the attacker's capacity is limited (i.e., the perturbation budget is 8/255 4/255). Additional results on ResNet50 can be found in Appendix D
Table 1: Success Rate in Targeted Attacks(with standard error reported).
16/255
8/255
4/256
== 1%
0.2%
<= 1%
E= 0.2%
1%
0.2%
Bullseye
3.741.6
1.1+0.4
1-003
0+00
0-00
0+0
Poison-Frog
1.340.8
0::0
0+:0
0=:0
0-:
0+:0
CIFARIO
Meta-Poison
42.542.8
30.7=2.3
28.1=2.5
19.412.11
5.211.7
3.911.2
Grad-Match
72.9+3.4
63.4+3.8
35,4=3,1
26.8433
10.342.6
6,4228
SAPA+Grad-Match
80.1+33.5
70.8=2.7
48.422.8
31.51331
16.742.4
11.2=1.5
Bullseye
2.130.3
0:50
264007
1,240.2
0.5:0.1
0.120T1
CIFARI00
Poison-Frog
1.0.2.88
0::0
1.110.4
0::0
0,220.1
0#:
Meta-Poison
50.312..
24.122.5
43.2=2.8
22.741177
4.521.2
3.1108
Grad-Match
90.2.1..1
53.6.2.55
62.1
33.144
11.31225
74119
SAPA+Grad-Mater
91.6+1155
74.942.2
86.8-39
52.6+3.6
31.6+224
121-112
5.2 PERFORMANCE OF SAPA IN BACKDOOR ATTACKS
Experiment setup. In this subsection, study the effectiveness of SAPA in backdoor attacks. In particular, we focus on the "hidden-trigger" setting (Saha et al. 2020 Souri et al.] (022) where the attackers can only add imperceptible perturbations to the clean training samples. For backdoor attacks, the adversarial goal is to cause the samples from a victim class y to be wrongly classified
Results for baseline Grad-Match is lower than reported in the original paper, because the original paper only trains 40 epochs. Detailed analysis on the impact of epochs can he found in Appendix

---

Published as a conference paper at ICLR 2024
as designated class yoh by inserting triggers. Besides, we follow the setting in (Souri et al. 2022) that the attacker adds imperceptible perturbations to samples in class yoh. Therefore, in our study, randomly choose the two different classes ytin y for poisoning sample generation. In this evaluation, we constrain that the attacker can only perturb 1% of the whole training set and the re- repeated for 50 times and we report the average success rate. More implementation details are in Appendix C.2
training process resembles our settings in Section [.1] All experiments
Baselines. In the experiment, our proposed method SAPA is incorporated to Sleeper Agent (Souri et al. 2022) which also leverages Gradient Match (Geiping et al., 2020) to achieve the adversarial goal in backdoor attacks. We also show the results of the method, Hidden-Trige' Backdoor (Saha et al. 2020), which optimizes the poison over images with triggers to preserve information of trig- gers, and the Clean-Label Backdoor method (Turner et al., 2019) leverages calculating adversarial examples (Goodfellow et al. 2014) to train backdoored model.
Performance comparison. Our results are shown in Table where we report the "Success Rate", which is the ratio of samples (with triggers) in y that are classified as yod by the poisoned model. From the result, our method outperforms all baselines under all settings. Specifically, Hidden- trigger (Saha et al. 2020) and Clean-label (Turner et al.) 2019) suffer from low ineffectiveness, as they are either designed for transfer learning or require control over the training process. Compared to these methods, SAPA shows effectiveness for different model architectures and perturbation bud- gets. In comparison with Sleeper Agent (Souri et al., 2022), which is also based on Gradient Match- ing (Geiping al. 2020), our method can also have a clear improvement. Especially, under the perturbation budget 8/255 and CIFARIO0 dataset, SAPA can obviously outperform Sleeper Agent.
Table 2: Success Rate in Backdoor Attacks on CIFARIO and CIFAR100
ResNe118
ResNet50
VGGIT
16/255
8/255
16/255
8/255
16/255
8/255
Hidden-trigger
3.5=1.2
13300.4
3.240.9
1.3=03
5+1.4
1.8007
CIFARIO
Clean-label
2.711.1
0.910.7
2.610.7
09:02
4.1115
1.110,9
Sleeper Agent
90.9+2.2
31.5#4.2
94.142.7
21.2:44.3
85.8=3.4
26.7=3.9
SAPA+Sleeper Agent
98.1+-2.6
49.3+3377
98.4+4.9
33,2+3.11
94.3-226
35.5-2.8
Hidden-trigger
2.1+09
1.400.4
1,740.3
0.8-0.1
1.4112
1.2F0.6
CIFARIO0
Clean-label
1.510.7
0.910.2
1,2104
04.10.1
26100.
0.810.2
Steeper Agent
58.343.9
26.7+:4.3
47,244.5
18.543.7
416433
12.922.6
SAPA+Sleeper Apent
7244+3.6
4188=3.2
63.9+3.5
31,4+32
67.7=2.7
30.3-2.3
5.3 PERFORMANCE OF SAPA IN UN-TARGETED ATTACKS
Experiment Setup. The goal of un-targeted attacks is to degrade the models' test accuracy. We fol- low the line of existing works (Huang et al. 2021 Fowl et al. [022 Ren al.) 2022) (which are also called "un-learnable examples") ") to perturb a large portion of training samples (50% 80%, 100%) in CIFARIO and CIFAR 100 datasets, in order to protect the data against being learned by DNNs. In our experiment, we limit the perturbation budget to 8/255. We first generate poisoning samples tar- geting on ResNet18. Then we re-train the victim models under ResNet18 and ResNet50 following existing works (Huang et al. 2020; Fowl et al. 2021). The training procedure for each model also resembles the settings in Section 5.1 and we repeat experiments for each setting 5 times and report the average performance. More details of the implementation are in Appendix C.3
Baselines. We compare SAPA with representative "un-learnable" methods, such as Error-Min (Huang et al. 2020). Error-Max (Fowl et al.) (02II) Separable Perturbation (Yu et al. (022). and Autoregressive Perturbation (Sandoval-Segun) which refers to the accuracy of models without poisoning attack. Notably,
et al. 20022). We also report the clean performance
proposed method SAPA can be incorporated into either Error-Min or Error-Max, so we denote our method as "Error- Min+SAPA" and "Error-Max SAPA" respectively. We provide more details of the algorithm of "Ennor-Mx+SSAPA in Appendix C
Performance comparison. In Table test dataset of CIFARIO and CIFAR 100, so lower value indicates better attack performance. From the result SAPA can improve the poisoning effect for both Error-Min and Erxor-Max under all set- tings. For example, when incorporating SAPA to Error-Min, Error-Min+SAPA has a clear advantage, by reducing the accuracy to around 10% when the poisoning ratio is 100% the CIFARIO 10 dataset.
we report the accuracy of the re-trained model the clean
'Hidden trigger and Clean-label also have lower success rates than in their original papers. This is because they are originally proposed for finc-tuning settings (by fixing the early layers), while we focus on end-to-end setting in this work. Our results are consistent with results in work (Souri et al. 2022.
7

---

Published as a conference paper at ICLR 2024
In other settings, Error-Min+SAPA can also manage to achieve 2-4% accuracy reduction compared to Error-Min. Similarly, Error-Max+SAPA is also demonstrated to have a consistent improvement over Maxo.-Max. In addition, incorporating our strategy can boost Error-Max and Error-Min to achieve comparable performance with Autoregressive Perturbation which specifically targets on the vulnerability of CNNS.
Table 3: Test Accuracy of Models Trained Poisoned Datasets via Un-targeted Attacks.
CIFARIO
CIFARIO0
ResNet18
ResNet50
ResNet18
ResNet50
100%
80%
50%
100%
80%
50%
100%
80%
50%
100%
80%
50%
Clean'
94.8
94.8
94.8
95.0
95.0
95.0
74.8
74.8
74,8
75.2
75.2
75.2
Separable.
13.5
86.3
92.9
14.9
88.1
93.2
9.1
57.1
66.2
8.4
60.8
66.7
Autoregressive.
11.8
82.3
89.8
10.1
83.6
90.3
4.2
51.6
64.7
4.3
53,5
66.1
Error-Max
11.9
88.2
92.2
12.8
90.1
93.9
4.8
57.3
66.9
5.6
58.3
68.1
9.6
84.6
90.1
10.9
85.7
91.3
4.1
55.1
64.8
4.9
56.8
66.9
Error-Min
21.2
87.1
93.4
18.9
89.5
94.5
11.2
56,9
67.7
10.8
60.3
70.3
Error-Min+SAPA
10.9
83.7
90.0
10.3
85.2
91.8
8.7
53.1
65.3
9.5
57.9
67.6
5.4 ROBUSTNESS OF RE-TRAINING VARIANTS
We provide ablation studies on the effectiveness of SAPA under various re-training settings including different re-training algorithms, re- re-training schedules and model architectures. We also study the robustness of SAPA against adversarial training (Madry et al. 2017). Notably, we only focus on the CIFARIO dataset and ResNet18 (except for the studies backdoor attacks, we specify the perturbation budget to 16/ 26/255 with poisoning ratio e = 1% For un-targeted attacks, we specify the perturbation budget to be 8/255 with poisoning ratio e 100%.
various architectures). For targeted and
Various Re-training Algorithms. There are studies (Schwarzschild et al. 2021 Ren et al. 2022)
demonstrating that many poisoning attacks can lose efficacy when faced with training algorithms beyond Empirical Risk Minimization (ERM). Therefore, we provide additional experiments to test the performance of SAPA under various re-training algorithms. We mainly consider the algorithms including Cut-Out Vrevs & Taylor 2017). Mix-Up (Zhang et al. 2017), and Sharpness- aware Minimization (SAM) (Foret et al. | 2020). which is proposed to minimize model sharpness to improve model generalization. Two optimization methods-SGD and ADAM are also included. In Table we compare SAPA with its backbone attack as baselines for each type of poisoning attack, and we use the same evaluation metric as previous subsections. From this table, remains outperforming the baseline attacks. Notably, among these re- re-training algorithms, Mix-Up shows outstanding ability to reduce the poisoning effect for all attack methods that we studied. It may be because Mix-Up is a data augmentation strategy which drastically manipulates the training data distribution, which can weaken the poisoning effect of the injected poisoning samples.
that our method
Table 4: SAPA vs Strongest Baselines under Re- Re-training Scheme Variation.
Un-turgeted[1)
Targeted
Backdoor (1)
Emar-min
SAPA
GradMatch
SAPA
SleeperAgent
SAPA
SGD
21.2
(-10.))
73.1
(+6.8)
91.8
91.1 (+5.3)
ADAM
19.7
10.4 (-9.3)
80.2
85,4 (+5.2)
92.3
97.7 (45.4)
Cut-Out
22.6
11.2 (-11.4)
82.4
90.3 (+7.9)
97.8
100.0 (+2.2)
Mix-Up
40.8
36.7 (-4.1)
58.4
65.5 (+7.1)
69.7
76.5 (46.8)
SAM
28.9
11.3 (-17.6)
74.3
80.7 (+6.4)
79.9
85.3 (+5.4)
Un-targeted[[)
Targeted(1)
Backdoor(1)
Enor-min
SAPA
GradMatch
SAPA
SleeperAgent
SAPA
ResNet18
21.2
(-103))
73.1
(47.0)
91.8
(45.3)
MobileNetV2
21.5
11.9 (-9.6)
68.5
75.2 (+6.7)
30.2
(+7.4)
VGGHI
35.3
20.2 (-15.1)
42.9
47.6 (+4.7)
31.9
37.4
VIT
40.6
36.5 (-4.1)
36.2
41.5 (+5.3)
24.7
26.3 (+1.6)
Various model architectures. We also test different model architectures after the generation of poisoned data for all three types of attacks. In detail, we test on MobileNetV2 (Sandler et al.) 2018), VGG11 (Sionnyan& & Zisserman 2014) and pretrained Vision Transformer VIT, (Dosovitskiy et al, 2020 Results in Table 4 conclude that SAPA consistently improves the baseline methods, and shows a better ability to adapt to unknown model architectures during re- re-training.
Various Re-Training Schedules. There are also studies (Schwarzschild al. 2021 Huang et al. 2020) suggesting that a different re- re-training epoch number or schedule can significantly break the poisoning effect. Therefore, we provide additional experiments when: (1) the model is trained for
8

---

Published as a conference paper at ICLR 2024
500 epochs and the learning rate is updated by "steps" similar to previous studies, and (2) the training learning rate is updated "cyclicly". In Figure [] we plot the curve of the poisoning effect of SAPA and baselines in the backdoor and un-targeted attacks. Note that exclude targeted attacks because the poisoning effect is discrete in one model. From these figures, we can find that SAPA can stably converge the point with a strong poisoning effect that consistently outperforms baselines.
6
(a) Backdoor(t) (Step) (b) Backdoor(t) (Cyclic) (c) Un-targeted(L) (Step) (d) Un-targeted(4) (Cyclic)
Figure I: The Curve of Poisoning Effect for Various Re-training Schedules
Adversarial Training. In reality, these attacks can be faced to defense strategies. this part, take untargeted attack as an example, we conduct a case study to evaluate the resistance of SAPA against Adversarial Training (Madry et al., 2017). which is strong defense for un-targeted attacks (Tao et al. 202T). We test on SAPA as well as all baselines in un un-targeted attack. From the results,
Table 5: Adv. Train.
FRM
Adv Train
Separable
11.3
Autoererressive
13.5
78.7
Estur-min
21.2
82.0
Frror-min SAPA
10.9
73.2
Emmo-max
11.9
80.3
Erro-max SAPA
9.6
note that adversarial training can significantly increase the test accuracy to defend against all attack methods. However, SAPA can can still outperform other attacks. Notably, although Adv. Train. is usually considered strong defense to untargeted attacks, it can naturally reduce the accuracy on CIFAR10 10 from 95% to 85% without any poisoning attacks, based on extensive previous studies (Tsipras al. 2018
5.5 EFFICIENCY VS EFFECTIVENESS TRADE-OFF
Previous works (Huang et al. 2020) also leverage the Ensemble and Re-initialization (E&R) tech- nique, to take various model architectures and initializations into consideration to handle re-training uncertainty. In this part, we compare the efficiency and effectiveness trade-off between SAPA and E&R, when they incorporated to existing attacks, such as Gradient Matching and Sleeper Agent. In Figure 2 we report the attack successful rate (ASR) and computing time of SAPA and E&R with different options (K, R denote the number of ensembled model architectures and initializations respectively). Specifically, SAPA has higher ASR when K and R are small, and can still achieve comparable success rates when K and R are increasing. However, the running time grows dra- matically for large K and R making them much less efficient than SAPA. This result shows SAPA demonstrate much better efficiency and effectiveness trade-off compared with E&R. Notably, we exclude the result for untargeted attacks, as it generates poisoning samples for the whole training set, which makes E&R extremely inefficient.
971
27.3
15000
80
$4.5
75
1003
V13
10000
70
600
1337
250
60
34P4
Day R4D
Deeosbrreen (41))
caldie
Despphor
(a) Targeted
(b) Backdoor
Figure 2: ASR and running time for targeted attack (GM) and backdoor attack (Sleep.).
6 CONCLUSION AND LIMITATION
In this paper, we introduce a novel and versatile framework for data poisoning attacks that takes into account the model landscape sharpness. We apply this strategy to various types of poisoning attacks, including un-targeted, targeted, and backdoor attacks. Our experimental results demonstrate the superiority of our method compared to existing approaches. However, in this paper, our method focuses image classification. Therefore, will leave the relative studies, such as the poisoning attacks in self-super learning settings, and poisoning attacks in other domains such as texts and graphs for future investigation.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENT
This research is supported by the National Science Foundation (NSF) under grant numbers CNS 2246050, IIS1845081, IIS2212032, IIS2212144, IOS2107215, DUE 2234015, DRL 2025244 and IOS2035472, the Army Research Office (ARO) under grant number INT-211---, the Home Depot, Cisco Systems Inc, Amazon Faculty Award, Johnson&Johnson, JP Morgan Faculty Award and SNAP.
REFERENCES
Hojjat Aghakhani, Dongyu Meng, Yu-Xiang Wang, Christopher Kruegel, and Giovanni Vigna. Bullseye polytope: A scalable clean-label poisoning attack with improved transferability. In 202/ IEEE European Symposium on Security and Privacy (EuroS&P), pp. 159-178. IEEE, 2021.
Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-awar mini- mization, 2022.
Jonathan Bard. Practical bilevel optimization: algorithms and applications, volume 30. Springer Science & Business Media, 2013.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector ma- chines. arXiv preprint arXiv:1206.653: 2012.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 124018, 2019.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:/712.05526, 2017.
Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of operations research, 153:235-256, 2007.
Terrance Vrics and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: 2017.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp. 1019-1028. PMLR, 2017.
Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li. Lira: Learnable, imperceptible and robust backdoor attacks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11966-11976, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.I1 1999, 2020.
Gintare Karolina Dziugaite and Daniel Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint 1rXiv:1703.11008 2017.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. arXiv preprint arXiv:2010.01412. 2020.
Liam Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojciech Czaja, and Tom Goldstein. Adversarial examples make strong poisons. Advances in Neural Information Processing Systems, 04.33399-03551 2021.
10

---

Published as a conference paper at ICLR 2024
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning, pp. 1568-1577. PMLR, 2018.
Jonas Geiping, Liam Fowl, W Ronny Huang, Wojeiech Czaja, Gavin Taylor, Michael Moelerr, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. arXiv preprint arXiv:2009.0222 2020.
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):1563-1580, 2022.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230-47244, 2019.
Hanxun Huang, Xingjun Ma. Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearnable examples: Making personal data unexploitable. arXiv preprint arXiv:2101.04898 2021.
W Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. Metapoison: Prac- tical general-purpon purpose clcan-label data poisoning. Advances in Neural Information Processing Systems, 33: 10000-1209,, 2020.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:/803.05547, 2018.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On latge-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 04836, 2016.
Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses. Machine Learning, pp. 1-47, 2022.
Junyi Li, Bin Gu, and Heng Huang. Improved bilevel model: Fast and optimal algorithm with theoretical guarantee. arXiv preprint arXiv:2009.0006 2020a.
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterrized neural networks. In International conference artificial intelligence and statistics, pp. 4313-4324. PMLR, 2020b.
Risheng Liu, Pan Mu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. A generic first-order gorithmic framework for bi-level programming beyond lower-level singleton. In International Conference on Machine Learning, pp. 6305-6315, PMLR, 2020.
Risheng Liu, Yaohua Liu, Shangzhi Zeng, and Jin Zhang. Towards gradient-based bilevel optimiza- tion with non-convex followers and beyond. Advances in Neural Information Processing Systems, 34:8662-8675. 2021.
Yiwei Lu, Gautam Kamath, and Yaoliang Indiscriminate data poisoning attacks on neural net- works. arXiv preprint arXiv:2204.0009) 2022.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06088 2017.
Shike Mei and Xiaojin Zhu. Using machine teaching identify optimal training-set attacks machine learners. In Proceedings of the aaai conference on artificial intelligence, volume 29, 2015.
11

---

Published as a conference paper at ICLR 2024
Luis GonizzzGozzález, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back- muck-gradient optimization. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pp. 27-38, 2017.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally-normalizes margin bounds for neural networks. arXiv preprini urXiv:1707.09560 2017.
Anh Nguyen and Anh Tran. Wanet-imperceptible warping-based backdoor attack. arXiv preprint arXiv:2102.10369 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, PP- 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021.
Jie Ren, Han Xu, Yuxuan Wan, Xingjun Ma. Lichao Sun, and Jiliang Tang. Transferable unlearnable examples. arXiv preprint arXiv:22100 10:::2200.011144 14, 2022.
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor at- tacks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 11957- 11965, 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Pedro Landoval-Segura Vasu Singla, Jonas Geiping, Micah Goldblum, Tom Goldstein, and David Jacobs. Autoregressive perturbations for data poisoning. Advances in Neural Information Pro- cessing Systems, 35:27374-27386, 2022.
Avi Schwarzschild, Micah Goldblum, Arjun Gupta, John P Dickerson, and Tom Goldstein. Just how toxic is data poisoning? unified benchmark for backdoor and data poisoning attacks. In International Conference on Machine Learning, pp. 9389-998999898 99988 PMLR, 2021.
Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back prok-propagation for bilevel optimization. The 22nd International Conference on Artificial Intelligence and Statistics, pp.
PMLR, 2019.
Ali Shafahi, Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. Advances in neural information processing systems, 31, 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXiv:1409.111 1556, 2014.
Hossein Souri, Liam Fowl, Rama Chellappa, Micah Goldblum, and Tom Goldstein. Sleeper agent: Scalable hidden trigger backdoors for neural networks trained from scratch. Advances in Neural Information Processing Systems, 15:19955-19178 2022.
Daouda Sow, Kaiyi Ji, Ziwei Guan, and Yingbin Liang. A primal-dual approach to bilevel optimiza- tion with multiple inner minima, 2022.
Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. Certified defenses for data poisoning attacks. Advances in neural information processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2024
Fnu Suya, Saeed Mahloujifar, Anshuman Suri, David Evans, and Yuan Tian. Model-targeted poi- soning attacks with provable convergence. In International Conference on Machine Learning, pp- 10000-10010. PMLR, 2021.
Lue Tao, Lei Feng, Jinfeng Yi, Sheng-Jun Huang, and Songcan Chen. Better safe than sorry: Pre- venting delusive adversaries with adversarial training. Advances in Neural Information Process- ing Systems, 34:16209-16225, 2021.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. Advances in neural information processing systems, 31, 2018.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint a /2/52, 2018.
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv:1912.027771 2019.
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-leaming. Artificial intelligence review, 18:77-95, 2002.
Kaiyue Wen, Tengyu Ma, and Zhiyuan Li. How does sharpness-awe aware minimization minimize sharpness?, 2023.
Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor learning. Advances in Neural Information Processing Systems, 35: 155466099,,,, 2022.
Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eekert, and Fabio Roli. Is feature selection secure against training data poisoning? In international conference on machine learning, pp. 1689-1698. PMLR, 2015.
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Yan Liu. Availability attacks create shortcuts. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.
pp.
Hongyi Zhang. Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
Bingyin Zhao and Yingjie Lao. Clpa: Clean-label poisoning availability attacks using generative adversarial nets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp- 9162-91700 917, 2022.
Chen Zhu, W Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clean-label poisoning attacks on deep neural nets. In International Conference on Machine Learning, pp. 7614-7623. PMLR, 2019.
13

---

Published as a conference paper at ICLR 2024
A BOARDER IMPACT
Our research unveils powerful attacking framework that has the potential to compromise security- critical systems in a stealthy manner. As machine learning models, particularly large models that require extensive training datasets, become increasingly prevalent and assume significant roles in various domains, the importance of clean training data cannot be overstated. prioritize the quality of data to ensure the success and reliability of these models. By shedding light on the potential dangers associated with this threat model, our study aims to raise awareness about the importance of data security. We hope that our findings will serve catalyst for the development of stronger defenses against data poisoning attacks. Safeguarding against these threats requires proactive approach and increased vigilance to protect the integrity and robustness of machine learning systems.
is imperative to
B DISCUSSION OF EXISTING DATA POISONING ATTACKS. B. THREAT MODEL AND OBJECTIVES
In this section, we introduce the threat model, adversarial objective, and victim set for each type of data poisoning attack.
Un-targeted Attacks. un-targeted attacks (Steinhardt et al. 2017). the attacker aims to cause the trained model with an overall low test accuracy. The attackers are assumed have access the training data and can perturb part (Steinhardt et al. (011)) or the whole training data (Fowl et al., 2021 Huang al. 202I). However, because the attacker usually does not have knowledge of test distribution and the training process of the victim model, most works (Steinhardt et al., 2017 Fowl al. 2021 Huang et al. 202T) define the adversarial objective as the following maximize the model error on the clean training set Dtr:
Qui(0r,ii, -L00;
(14)
Targeted attacks. In targeted attacks (Shafahi et al., 2018), the attacker aims to cause the trained model to misclassify specified test sample
subset of test samples. For example, they are targeting on a victim person and have knowledge of & (k 2 1) photographs of this per- sor (x )*)) )===1 They aim to cause the model misclassify the photos of this person while preserving the overall accuracy of the rest. The attackers are only allowed to perturb small part of
the training data and have no knowledge of the victim model including the initialization and train-
and the adversarial ing algorithm. Therefore, they define the victim set Dr {(x)", ) objective as:
(T))= = E ("(x;00),,
(15)
(x.B1)DDr
where doo a designated class different from y
Backdoor attacks. In backdoor attacks (Chen et al. 2017 Souri et al., 2022). the attacker aims to take control of the model prediction by injecting samples with "triggers". particular, if there is trigger, such patch p. present in an image, the poisoned model will predict this sample to specified class yو Otherwise, the poisoned model will make a correct prediction. Similar to the targeted attack, attackers
only allowed to perturb small part of training data and have no control of the training process of the victim model. In backdoor attacks, most works target on samples from specific victim class y = the and define the victim set Dr {(x,y) Derly y" During the attack, they aim to solve the adversarial objective:
Q6(0", (r) = Σ
I(f(r, 0p:0),
(16)
(R.B))DD
where p denotes the process that p is attached clean image x. In this way, the poisoned model is likely to predict the samples with triggers to be class ybb
We assume that the samples of the victim are from the same class ric following most existing works (Sharahi al. 2018)
14

---

Published as a conference paper at ICLR 2024
B.2 ALGRITHTS
In this section, we discuss the details of existing data poisoning attacks.
Targeted attacks. Targeted attacks insert poisons into the clean training aiming at misclassifying targetsssamples or classes) as adversarial labels. Fundamentally, targeted attacks can be formu- lated as a bi-level optimization problem in Eq [5] which can be challenging especially for DNN because the inner problem has multiple minima. Some existing methods avoid solving it directly and apply heuristic approaches such as Feature Collision Attacks(i.e. Bullseye
et al., al., (018)) which manipulate the representation of victims to mislead models classify them as adversarial labels. These methods are well-suited for the transfer learn- ing setting(Goldblin et al., (022). where a model is pre-trained on clean data and fine-tuned on a smaller poisoned dataset. However, this brings natural drawback in that its poisoning effect is only kept for one surrogate (thel(the pre-trained model), thus these methods can hardly work for the retrain-from-ssene from-scratch scenario as the retrained model can be very different from the surrogate model. Another line of work tries to handle the bi-level optimization problem directly. For linear or convex models, the inner problem typically exhibits single minimum. As result, several methods such as those proposed Byigioo et al.) 2012 Xiao et al.) 2015) have been developed to successfully achieve the malicious objective. Theoretical analyses, such the work by(Mei & Zhu 2015), have provided guarantees in these cases. However, these methods become ineffective for DNN. (Muñoz-) González al., 2017) applies method called mhack-gradeent descent" in which the inner problem approximately solved using several steps of gradient descent, and the -reseet for the outer conducted by back propparoagg through the inner minimization routine. This method is both time and memory connuurc.onsuinng. thus impractical for models with multiple layers. MetaPoison (Huang et al.) 2020) draws inspiration from (Franceschi et al. 2018 Shaban et al. (019) and unrolling the training pipeline of inner. They also apply "ensembling" and "network re- re-initialization" to avoid overfitting to one single surrogate model and try to preserve the poisoning effect during re-training. However, the success of this unrolling still requires single minima assumption(Liu al. (020), leading to less effectiveness as shown in the empirical results. Brad-Match((CC)) (eeiinn et al.] 2020) leverages a "gra- dient alignment" technique to solve the problem in Eq [5 and applies "poison re-initialization" to select better poisons, but there still exists space for improvement as the formulation does not capture the nature of inner multiple minima well(Sow al. 2022). This is the reason why we rethink the formulation of targeted attacks and design a new objective in Eq
202T). poison--roogSSaaaan
Backdoor attacks. Backdoor attacks aim at inserting triggers into training samples and testing sam- plcs, such that triggered inputs will be misclassified. Note that there exist many types of backdoor attacks Wu et al. 2022) with regard to factors of backdoor, such as attacker's capabilitytraining con- trollable or not). characteristics of triggers trigger clean-label backdoor attacks, meaning attackers insert poisoned samples rather than triggers into the training data, and testing inputs attached with trigger will be misclassified. Hidden- trigger(Saha et al., 2020) generates poisoned images carrying information about triggered source images and confuses the model during fine-tuning Clean-label(TI versarial perturbations to cause a successful backdoored effect. However, both methods can hardly succeed in the retrain- trrrii-from-scratee scenario, as Hidden-trigger is designed for transferring learning and Clean-label needs to control the retraining process. Sleeper Agent(Souri et al., 2022) incorpo- rates the gradient matching technique to solve the problem in [6 and considers "model restarts" to adaptively update the model during the generation of poisons, but it can only cover a few mod- els and limits its effectiveness. Our SAPA utilizes the sharpness objective to capture the nature of multiple- iimplel--nee--neee -niie--mmimaa, and is shown to better preserve the poisoning effect during retraining. It worth noting that there exists a line of work that consider different scenario from end-to-end scenario discussed in this paper. Some of these attacks need additional assumptions that the attacker can control the re-training process such as WaNet (Nguyen & Tran 202T) and LiRA (Doan et al.) 2021]; some conduct attacks without involving any models and require noticeable triggers, such as BadNet (Gu et al. 2019). Therefore, this line of work is out of the main scope of our paper.
etc). In Eq 16] we focus on so-called hidden-
et al. 2019) leverages ad-
Un-targeted attacks. Un-targeted attacks insert poisons into the clean training and the goal is to degrade the overall accuracy over the clean testing. Generally, it can be formulated as bi-level opti- mization problem in Eq 14 Early works Biggii al.) 2012 Steinhardt et al. [017 Koh et al. 2022) primarily focused on simple models with desirable properties like linearity and convexity. In these cases, the inner problem typically possesses unique solution, and these methods demonstrate effec-
15

---

Published as a conference paper at ICLR 2024
tive performance. Theoretical analysis(Steinhard et al., 2017, Suya al. out to establish the feasibility of untargeted attacks on such simple models. Nevertheless, the pres- ence of non-convexity and multiple minima within the inner problem poses significant challenges for untargeted attacks on DNN models. scent", which is expensive and infeasible for complex models. (Lu al. 2022) borrows ideas from Stackelberg games and applies so-called "total gradient descent ascent" techniques to generate poi- son samples one by one. However, they can hardly preserve poisoning effects after retraining and have subtle influence on clean testing. CLPA (Zhao & Lao 2022) takes advantage of 'naturally poisoned data' which refers to the wrongly predicted samples by the clean model and generates them using GAN. Nevertheless, this method can only be applied for transferring learning as it relies on the clean model and suffers from the multiple-inner-minng 2020) tries to handle the muliple-inner-minina problem using ensembling and re- re-initialization, but the performance is not very satisfactory. stror-max(Fow) et al. 2021) solves the problem in Eq [4 by leveraging adversarial examples, and Error- piror-min(Huanng et al., 2021 inserts perturbations to build a relationship between labels and perturbations. These two methods only involve one surrogate model and can be improved through our proposed method. There also exists another line of work that generate perturbations without access to model or datasets, such as Synthetic Perturbations/ Yu Seruurratios(SSndoval-Seeee
2021) have been carried
et al.
(017) applies "back-gradient de-
problem. MetaPoison(HHang
al.)
al.) 2022). Though they do not solve the optimization problem directly and do not have the multiple- ceatiille-nnee-nnee inner-minima problem, their application may be limited to specific settings(such as un-targeted attacks and CNN models), and not as general as ours.
al. 2022) and Autoregressive
C IMPLEMENTATION DETAILS OF ALL SAPA METHODS
In this subsection, we provide details of the implementation of SAPA methods.
C.I TARGETED ATTACKS
solve the optimzation problem in Eq using Gradient Matching. To be more specific, we randomly choose one or a few target samples from a victim class(in testing) as the victim set DT. At the same time, we randomly choose a percentage of samples D, from the training to be modified in the algorithm, and the percentage is referred to as poisoming ratio. The attacking goal to make the model trained on Drr + D, misclassify samples Dr as a pre- specified ad adversarial class(different from the true class). To ensure the imperceptibility of poisons, we rigorously control the poisoning ratio and perturbation budger(L norm of the perturbation) and in our experiments, we consider the ratio of 1%, 0.2% and budget of 16/255, 8/255, 4/255.
As introduced in Section 4.1
Given a pre-trained clean model, we first randomly initialize the perturbation for Dp. and during each optimization step, we compute the gradient of the sharpness-aware objective using Eq [2] and do one step gradient descent to minimize the similarity loss defined in Eq 10 The detailed algorithm is shown in Algorithm 2
After the generation of poisons, we retrain the model from scratch via SGD for 160 epochs with an initial learning of 0.1 and decay by 0.1 at epochs 80, After training, evaluate the prediction on targets Dr. For the single-target case, only predicting the target as the adversarial label is con- sidered success, and sample 10 random poison-target cases success rate over 10 cases is referred to as the success rate for this trial. For each pettinggpoisoning ratio and perturbation budget), we do 50 trials and report the average success rate. For the multi- target case, we randomly choose a few targets, 4 or 8 in our experiments, and the average accuracy of predicting targets as adversarial labels is referred as the success rate of one experiment. We also repeat experiments 50 times and report the average success rate.
trial, for which the average
C.2 BACKDOOR ATTACKS
Similar to the targeted attack, we leverage the Gradient Matching method. To be more specific, we randomly choose a victim class and an objective class. Then we randomly sample the victim set Dr from the victim class and select D, from the objective class to be modified in the algorithm. Note that both Dy and D, are sampled from the training data. The attacking goal is to make the model retrained on Dar + D, misclassify images from the victim class(in testing) which attached with
16

---

Published as a conference paper at ICLR 2024
Algorithm 2: SAPA in Targeted Attacks(Grad-Matee)
Input : Pre-trained model f(:; 0*) on the clean training set Day; a victim set Dr; optimization
step M.
Randomly initialize the poisoning samples Dp Compute the gradient VoQ%(0"; Dr) using Eq 12 12
for m ..., M do | Update D, Eq with one-step gradient descent end
Output: Return Dr
a pre-specified trigger as the objective class. We also restrict the poisoning ratio to be 1% and the perturbation budget is bounded by 16/255, 8/255.
Given a pre-trained clean model, we first randomly initialize the perturbations for Dp. and during each optimization step, we compute the gradient in Eq 12 but different from targeted attacks, in [2 Dr are attached with the trigger which will be used to backdoor images in the testing. Then do one-step gradient descent to minimize the similarity loss defined in Eq E after optimizing for R steps, we update the model on poisoned training data Dr. + Dp. and then continue optimizing perturbations on the updated model. The detailed algorithm is shown in Algorithm B
After the generation, we retrain the model from scratch via SGD for 160 epochs with an initial leamning of 0.1 and decay by 0.1 at epochs 80,120, After that, we evaluate the prediction of triggered images from the entire victim class in the testing. The average accuracy of predicting triggered images as the adversarial label is referred to as the success rate. We repeat experiments 50 times and report the average accuracy as the final results. We conduct experiments on multiple datasets including CIFARIO, CIFARI00 and SVHN, along with multiple victim models including ResNet18, ResNet50 and VGGII.
Algorithm 3: SAPA in Backdoor Attacks
Input Pre-trained model f(;; #*) on the clean training set D,y a victim set Dy; retraining factor R: optimization step M.
Randomly initialize the poisoning samples Dp
for O...., MM- 1 do
m mod [ MM /R + 1)J = and m a then Update l on poisoned training D + Dp
Find the worst-case direction using Eq 11
Approximate (00;; Dr) by VoQ(6"; Dr)l=
end
Update D, with one step gradient descent
end
Output: Poisoning set Dp
C.3 UN-TARGETED ATTACKS
We implement SAPA based on two existing methods: Crror-min(Huang al. 2021 and Error- max et al. 202T).
Error-nin+SAPA We have discussed this method in Section 4.3 and the detailed algorithm is
shown in Algorithm In our experiments, we set T = 20, M 10. = e/10 and E = 100.
Error-max+SAPA. Different from Error-min, Error-max is used solve the optimization as fol-
lows:
Σ
(f((x: +
max Lo
(D..
17

---

Published as a conference paper at ICLR 2024
where H" denote model trained clean data and is fixed during poison generation. Thus SAPA+Error-ma focuses on the sharpness-awe aware objective as follows:
max (-.)il
max |I
E (BL.)))DL.
[[f(m + ō,;0" + v))
Similar to Ertor-Min+SAPA, we also solve this optimization problem using gradient descent, As shown in Algorithm 4 given the clean pre-trained model f(,,0, we first find the worst direction maximize max, (E.,))ED,. [[f(r, + 6 lr + u), 34) and then update 6, fixing 0" + 0.
After generating perturbations via either SAPA+Error-mnn model on poisoned training via SGD for 160 epochs with an initial learning of 0.1 and decay by 0.1 epochs 80.120. Then evaluate the prediction of clean testing and report the average ac- curacy. Our experiment is conducted on multiple datasets including CIFARIO, CIFARIO0, SVHN, along with multiple victim models including ResNetl and ResNet50.
S
retrain the
Algorithm 4: Error-max+SAPA
Input : Network f(: 0); clean training set {(ri, W perturbation bound e; PGD step T; pre-train steps R
Output: Sample-wise perturbation D, = {6, 1
for I,... R do
| Update 0 via minimizing L(0; D..) end
Randomly initialize perturbation Dp
Fix 0, Dp. find the worst direction i to maximize E(uw))D, (f(z; + 6,;0" + v),
for t = 1..... T PGD steps do Update D, via gradient ascent fixing end
18

---

Published as a conference paper at ICLR 2024
D ADDITIONAL EXPERIMENTS
In this section, we provide additional experiments further illustrate the effectiveness of SAPA.
D.I ADDITIONAL DATASET
We test the additional dataset, SVHN, to further illustrate the advantage of our method. All the experiments are conducted following the same procedure as in Section while some details are different. For un-targeted attacks, repeat all experiments conduct on the "'SingleViictii Victim" case and omit baselines Poison-Frog and Bulleyes because they do perform well under the train-from-scratee setting, For backdoor attacks, we also omit baselines Clean-label and Hidden-trigger which do not perform well.
SVHN. For targeted attacks,
Performance comparison. Results of un-targeted attacks are shown in Table Similar results on CIFARIO 10 and CIFARIOK our method outperforms nearly all baselines under all settings. Results of targeted attacks are shown in T Our method outperforms all baselines significantly especially for smaller budget sizes and poison ratios. Results of backdoor attacks are shown in Table g Our method has better performance than baselines under all settings.
Table 6: Test Accuracy of Models Trained on Poisoned Datasets via Un-targeted Attacks.
SVHN
ResNet18
ResNet50
100%
80%
50%
100%
80%
50%
Clean
96.0
96.0
96.0
95.9
95.9
95.9
Separable
8.3
92.5
94.3
7.8
90.9
93.1
Autoregressive.
7.2
89.5
92.5
7.1
89.1
91.2
Error-Max
92.9
93.4
92.5
93.7
SrTOr-Max+SAPA
4.7
92.3
92.6
5.1
91.8
92.1
Error-Min
13.8
92.8
95.3
13.7
91.3
94.2
Error-Min+SAPA
10.2
91.7
93.1
11.4
90.4
91.8
Table 7: Success Rate under the "Single-Victim" Victim" Setting Targeted Attacks.
16/235
8/255
4/235
1%
0..2%
1%
0.2%
1%
0.2%
Meta-Poison
52.1
28.7
35.3
21.9
23.8
11.2
SVHN
Grad-Match
62.6
37.4
44.2
25.7
31.3
17.4
SAPA
71.2
48.5
55.7
38.2
42.1
23.3
Table 8: Success Rate in Backdoor Attacks on SVHN
ResNet18
ResNet50
VGG11
16/255
8/255
16/255
8/255
16/255
8/255
Sleeper Agent
92.3
42.6
91.6
34.8
86.2
38.1
SAPA-backdoor
97.5
58.3
95.4
44.1
91.4
43.7
D.2 MULTI-TARGET TARGETED ATTACKS
We also perform tests on poisoning multiple targets simultaneously. We conduct experiments on model ResNet18 8 and two datasets Cifarl0 and Cifar100, We only compare with the most powerful baseline Grad-Match under multiple settings, including different number of victim targets(4,8) per- turbation size size(16/255, 8/255 and 4/255) and poisoning rate(1% and 0.25%). All results are shown Table It is obvious that SAPA consistently improves the performance under all settings.
D.3 HYPERPARAMETER
The radius p in Eq. is the most important hyperparameter in our proposed method, and it repre- sents the radius within which the locally worst model is searched. the work (Foret et al. 2020). the default value is 0.05 and adopt it in main experiments. However, interested in
19

---

Published as a conference paper at ICLR 2024
Table 9: Avg. Success Rate under the "Multiple-Vietin Victim" Setting in Targeted Attacks.
16/255
8/255
4/201
1%
0.25%
1%
0.25%
1%
0.25%
Grad-Match
62.9
36.2
34.7
25.1
20.3
7.5
CIFAR10: Vietims
SAPA
75.1
53.4
47.9
30.8
24.3
10.8
CIFARIO: Vietims
Grad-Match
52.1
23.2
27.9
18.4
12.7
5.6
SAPA
64.6
31.2
34.6
26.1
17.5
Grad-Match
30.1
37.2
17.8
CIFARI00: 4 Victims
SAPA
74.2
43.8
44.3
19.7
25.1
Grad-Match
43.6
23.2
16.7
4.9
13.8
2.7
CIFARI00: Victims
SAPA
52.7
31.3
24.8
8.3
18.7
4.2
whether this hyperparameter has large impact on the performance. Therefore, we for dif- ferent values, i.e. = 0.01, 0.05, aargted(SAPA+Grrd--aaa
0.5. To avoid the computational cost, we only test on and backdoor attack(APA~SSleeer//e Tabel 10 We notice that smaller p causes the algorithm to regress to the backbone attacks without SAPA, so the poisoning effect drops; while a very large value (p 2 0.2) also leads to poor attack performance.
and results are shown in
Table 10: Hyperparameter p
Method/p
0.01
0.05
0.1
0.2
0.5
SAPA+GM
76.5
80.4
82.1
77.5
69.3
SAPA-SleepAg
92.7
97.1
97.6
96.8
83.2
D.4 ENSEMBLE AND RE-INITIALIZATION
In Section compare SAPA with ensemble and re-initialization techniques. also worth noting that SAPA can leverage these techniques to further improve the performance. We conduct experiments on (Seeted(SAPA+Graadd this. Same as in Section 5.5 let K, R denote the number of ensembles and re-initializations respec- tively.
and SSAPAreeer attacks show
Table 11:
Impact of ensemble(K) and re-inistaillzation(R
Turgeted
Buckdoor
ASR(1)
Timels.)
ASR(1)
Timefe
KEL
80.0
156.9
97.1
1031.3
K-2,R=1
83.5
319.1
98.2
$129.7
K-4,R-1
K=8R=1
87.1
1562.5
100
30888.9 9
K=1.2=2
81.7
296.8
97.9
8006.4
K=1,aa
15986.7
K=1.R-8
86.2
1379.7
99.3
32459.8
D.5 MORE RESULTS
We also provide the performance of un-targeted attacks with standard error reported, and of targeted attacks for ResNet50 on C0 10 in Table 12 and respectively
E ADDITIONAL VISUALIZATIONS
Loss Landscape. We provide the visualization of the loss landscape for targeted and backdoor attacks to further illustrate that our method can indeed minimize the sharpness. For both attacks, losses are computed the victim set Dr, and visualize CIFAR10 and ResNet18 with per- turbation size 16/ 255 and poison ration 1 %. Visualizations are shown in Figure 3 It is obvious that method can achieve smoother loss landscape.
Effect of epochs on targeted. As we mentioned in Section we adopt different re- re-training epochs when applying baseline Grad-Match. In specific, the original paper(Geiping al. 2020) adapts
20

---

Published as a conference paper at ICLR 2024
Table 12: Model Test Accuracy under targeted Attacks, (with standard enror reported)
CIFARIO
CIFARIOU
ReaNet18
ResNet50
ResNet18
ResNet50
100%S
80%
50%
100%
B0%
50%
100%
80%
50%
100%
80%
50%
Separable
13SE0.11
14
BETERTT
33.236111
57.155111
26.230119
667:022
Anteregresive
11
22.2:025
89.44026
10,0+00000
63,610 15
90.3-0.11
4.4-0.08
51611013
64740010
4.32=0.11
53.++031
66 661:10.1
Ervor-lax
23
88213034
93.920.1
1.7500111
57.310144
3.61
58310.18
68.15027
Pr
1133-..11
84.640223
90.1+019
15.7640.34
01.3-023
4.13-0.06
1444111
64840009
4.87-0.17
568+0.14
669-023
10910.15
91.8-033
873-0:21
57.940.17
67.610.18
Table 13: Success Rate in Targeted Attacks(for ResNet50 on Cifarl0).
16/255
8/255
4/255
1%
0.2%
15
Bullseye
1.700.
0.34:0.2
1.1
0.4=0.1
0::0
0::0
Poison-Frop
2.3+0.7
0.6+00.4
0.8+0.3
0+0
0++0
0++0
CIFARIO
Peeaa-oisoon
46.9
34.1422.
23,5+:17
10.7
15.6
6.9.t1.2
Grad-Match
75.613.7
53.911.3
30.1133.
14.812..6
21,5-299
9.6112.3
SAPA
81.43..1
60.2:33.5
34.711366
17.611.9
26,3=2.1
12.41.9
Bullseye
2.41111
1.240..5
0.6
0.740.3
0:00
Poison-Frog
2.9.1.9
1.310.4
1.5.1.77
0.810.2
0.10
0.10
CIFARIOO
Meta-Poison
50.2:22.4
29.8:1..8
27.942.1
12.5
19.4216
8,2+188
Grad-Match
80,823.6
50.7=4.11
36.743.8
24.3:3399
26.9=2.7
10.422.1
SAPA
84,5=3.4
56.4=229
42.8+3.4
29.6+:2.7
30.1=2.4
13
(b) Backdoor(SAPA)
(c) Targeted(Grad-Mathh
(d) Targeted(SAPA)
Figure 3: Visualization of loss landscape for Backdoor and Targeted attacks.
(a) 40 epochs, 16/255
(b) 160 epochs, 16/255
(c) 40 epochs, 8/255
(d) 160 epochs, 8/255
Figure 4: Attack successful rate (ASR) for different training epachs of targeted attacks
40 epochs while we re-trained for 150 epochs. We plot the success rates for different epochs(40 and 160) under different perturbation sizes( 16/555 and 8/255) and show them in Figure Results of SAPA and Grad-Match are in red and blue respectively. From the figures, we notice that the 160-epoch has a smaller success rate than the 40-epoch and the success rate is slightly decreasing when the epochs are growing. These observations imply that re-training epochs indeed influence the poisoning effect and SAPA can improve the performance to some extent.
21


Reviews:
[{"review_id": "5HaRfUsdzR", "reviewer": "Reviewer_Y6BU", "summary": "This paper propses revisit exisitng \"end-to-end\" data poisoning attacks, finds that existing data poisoning attacks suffers from uncertanty issues in poison effects during the re-training process.  Then the reviewer proposes to leverage existing study on loss lanscape for DNNs and propose sharpness-award data poisoning attacks. Specifically, the authors improve upon previous work (e.g., Grad-Match, etc) with replacing original gradients with sharpness-aware gradients. Such sharpness aware loss is calculated by previous work. Through extensive experiements, sharpness-aware poisoning attackd can lead a mild improvement compared with existing approach.", "questions": "Can you evaluate the transferability of poison effects, such as crafting poisoning samples in VGGNet but used for training ResNet.  Can you test your approach across different surrogated models and target models?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The method is intuitive and sound.\n\n2. The evaluation is comprehensive.\n\n3. The results are good.", "weaknesses": "1. The presentation needs improvement.\n\n2. Limited Novelty. The only contribution for this work is that the author combines sharpness-award loss function (proposed by previous work) with existing poisoning approach (e.g., Grad-Match) to make stablize the pisoning effects during the re-training peocess.\n\n3. Lack of theoretical analysis compared with previosu work on data poisoning attacks, such as witches brew (ICLR 2020)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fRSNCXiYrk", "reviewer": "Reviewer_Ae7N", "summary": "The paper induces sharpness-aware training method towards 3 poisoning tasks: targeted attack (perturb few data to misclassify a sample), backdoor attack (perturb few data to misclassify a class of samples), and unlearnable examples (perturb all data to misclassify all clean test samples). The key design is an additional step to calculate the worst poisoning model before using it to update the poison samples. Experiments show that when plugged in to existing methods, SAPA improves the poison performance steadily.", "questions": "Response to rebuttal: Thanks for the good rebuttal and revision of the paper. I have no future concerns and thus keep my score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper extensively studies three poison attacks to showcase the effectiveness of SAPA. It would be very helpful for the audience since most papers only focus on one task but term the task as poisoning.\n2. The introduction of the sharpness-aware idea to poison attacks is straightforward and easy to plug in by adding an adversarial loop. But please make the additional computation amount (X + SAPA v.s. X) more clear.\n3. Experiments are comprehensive to demonstrate the steady improvement by SAPA, even under various training strategies. The study on efficiency is informative.", "weaknesses": "1. It would be good if the authors clearly distinguish between different poison tasks before introducing the method. Currently, the threat model is not clear for 3 tasks, and it may be confusing to distinguish the contribution of SAPA in a specific task.\n2. To better demonstrate the plug-in property of SAPA, it is good to be described without a specific attack method, e.g., gradient matching or error-min. And the results should be shown as X v.s. X+SAPA also in targeted and backdoor attacks.\n3. \"Self-Ensemble Protection: Training Checkpoints Are Good Data Protectors\" also focuses on efficient and effective poisoning in retraining. How does SAPA compare to it in the unlearnable examples task in terms of attack performance and efficiency?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0S9DUbmRhb", "reviewer": "Reviewer_jHfE", "summary": "Previous studies have developed different types of data poisoning attacks, but they are limited by the uncertainty of the retraining process after the attack. To address this challenge, the paper proposes a novel method called Sharpness-Aware Data Poisoning Attack (SAPA), which leverages the concept of deep neural networks' (DNNs') loss landscape sharpness to optimize the poisoning effect on the worst possible retrained model. Extensive experiments are conducted to show the method's effectiveness.", "questions": "1.  Regarding my concern about evalutation against more defense approaches, I would recommend adding experiments with some of the aforementioned defenses\n2. It would be better if there are qualitative comparison between clean and poisoned images.\n3. I find this sentence in section 5.5 confusing: \"In this part, we compare the efficiency and effectiveness trade-off between SAPA and E&R, when they are incorporated to existing defenses, such as Gradient Matching and Sleeper Agent.\" Do the authors mean \"existing attacks\" instead?\n4. Table 3 is quite hard to comprehend since it does not have best performances highlighted or average accuracy drops. The authors could improve its presentation a bit.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of leveraging the concept of loss landscape sharpness to improve data poisoning's efficiency is intriguing. The proposed method is also applicable to different attack objectives.\n2. The paper includes vastly extensive experiments with different attack goals, model architectures, re-training variants, etc. \n3. Overall, the experimental results show this method could constantly outperform other baselines.\n4. The paper is quite well-writtenm. Especially, the proposed method is explained quite clearly and detailedly.", "weaknesses": "1. The attack success rates are quite unimpressive when the perturbation budgets are low (4/255 for targeted attack and 8/255 for backdoor attack). I recognize there are improvements compared to other baselines, but I find it quite hard to considered the attacks are successful.\n2. The number of defense strategies evaluated in the paper is quite lacking. All defense strategies considered are different re-training variants. While the adversary in this theat model acts as the data provider, I think there should be experiments with data filtering defenses, such as [1], [2], [3], [4]. There is also no poisoned model detection defense, such as Neural Cleanse or STRIP, mentioned in the paper.\n3. The performance against adversarial training is also quite underwhelming.\n\n[1] Chen, Bryant, et al. \"Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering.\" (AAAI 2019)  \n[2] Tran, Brandon, Jerry Li, and Aleksander Madry. \"Spectral signatures in backdoor attacks.\" (NeurIPS 2018)  \n[3] Hayase, Jonathan, et al. \"Spectre: Defending against backdoor attacks using robust statistics.\" (ICML 2021)  \n[4] Zeng, Yi, et al. \"Rethinking the backdoor attacks' triggers: A frequency perspective.\" (ICCV 2021)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Qfrgw8mds", "reviewer": "Reviewer_X4g5", "summary": "This work proposes a modification to the objective function of several poisoning attacks wherein poisons are crafted at a parameter vector that has been modified from a standard (ERM trained parameter vector) to increase the attacker's target loss.", "questions": "1. Does SAPA also improve poisoning success when paired with objectives other than gradient matching? \n\n2. Eq. 7 is unclear - you craft $D_p$ on at a parameter vector $\\theta^* + \\nu$, but $\\theta^*$ was the result of ERM on $D_{tr} + D_p$? How was the initial $D_p$ (used in ERM) calculated? Do you have to generate poisons/train a victim model twice?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The authors do a good job of comparing to a wide range of existing attacks in several poisoning settings. \n\n2. The authors also do thorough experimentation in each of these settings, and include some results under existing defenses.\n \n3. Taking the results of the work at face value, the proposed method improves over SOTA poisoning methods, sometimes by a significant margin. \n\n4. The paper is easy to follow and generally well presented.", "weaknesses": "1. I think the motivation of poisoning the \"worst\" retrained model isn't fleshed out enough. Why does crafting on the worst-case poisoned model intuitively lead to more generalizable/potent poisons on *average*? \n\n  2. The authors should be more careful when talking about sharpness of minima in this context, as sharpness depends on the *objective* in question. Many might read this work and assume the sharpness the authors are referring to is the sharpness of the minima of the \"standard\" loss landscape, when this is not what is being discussed. \n\n3. I would avoid statements like \"with a high possibility, the re-trained model can converge to a point where the poisoning effect presents\" , and \"Therefore, for the models which are re-trained on the poisoned dataset, the poisoning effect is also very likely to persist\" as these haven't been justified anywhere in the work.\n\n4. A concern raised about existing attacks  in the introduction was that they are not architecture agnostic. While I agree with this concern, this work does nothing to explicitly mitigate this. \n\n5. Figure 2 shows that when using ensembles, and restarts, existing methods can beat SAPA. Did you try SAPA with these additions? In general I'm not sure poison crafting time is something to include in the main body/claim as an advantage, especially when the worst times are ~20 minutes.\n\n6. Gradient matching also considers ensembles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zd3pByR5pV", "reviewer": "Reviewer_kdwN", "summary": "This paper proposes a data poisoning attack by leveraging sharpness-aware minimization. The method can be applied to backdoor, untargeted, and targeted attacks to improve performance.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a unique approach using loss landscape sharpness to enhance poisoning attacks.\n\n2. The method can be applied to various attack settings. \n\n3. The performance shows the proposed method is effective against existing defenses.", "weaknesses": "The authors have improved the paper from their NeurIPS submission. The remaining concern is that many of the reported results are still not consistent with the results presented in the original papers without proper justification, including both Table 1 and Table 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c8bzfNmdlQ", "reviewer": "Reviewer_Twq5", "summary": "In this paper, the authors propose a new approach, called Sharpness-aware Data Poisoning Attack (SAPA), that use loss landscape sharpness to improve the effect of data poisoning attack under the worst scenario (model re-training). The authors elaborate on achieving SAPA in targeted, untargeted and backdoor attacks. SAPA can be incorporated with existing data poisoning approaches and achieve better performance. Evaluations are performed on CIFAR-10 and CIFAR-100 over different tasks (targeted,  untargeted, backdoor) and the results show that SAPA yields better ASR compared to existing attacks.", "questions": "Please see weaknesses for details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes an effective approach to tackle the long-last issue that poisoning attack effect may drop significantly due to worst-case training settings (e.g., re-training).\n\n2. The proposed SAPA is applicable to multiple attacks (i.e., targeted, untargeted, backdoor).\n\n3. The authors perform comprehensive experiments and ablations to show the performance of SAPA and the results are sufficient to support their claim that SAPA can enhance ASR by a large margin.", "weaknesses": "1. SAPA aims to solve the effectiveness of certain attacks (e.g., clean-label integrity and backdoor attacks) that suffer from re-training. On the other side, there existing a line of attacks (e.g., dirty-label attacks such as LIRA, WaNet, BadNets, etc.) that do not have such performance drop. I would suggest the authors clarify the discrepancy between different attacks.\n\n2. The authors only use CIFAR-10 and CIFAR-100 in the evaluations. However, in prior works such as the Hidden trigger backdoor and Gradient matching, ImageNet is always used to evaluate the attack performance. I suggest the authors also include ImageNet results in the experiments.\n\n3. It would be desired to evaluate SAPA against SOTA defenses. The authors provide the result against adversarial training, which is not the best option to defend against poisoning attacks. There are many defenses specifically designed for poisoning attacks such as ABL [1], Fine-pruning [2], Deep KNN[3], etc, it would be nice to evaluate SAPA against these defenses.\n\n[1]. Anti-Backdoor Learning: Training Clean Models on Poisoned Data\n\n[2]. Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks\n\n[3]. Deep k-NN Defense against Clean-label Data Poisoning Attacks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)