Can Adversarial Training Be Manipulated By Non-Robust Features?
Authors: Lue Tao
Lei Feng
Hongxin Wei
Jinfeng Yi
Sheng-Jun Huang
Songcan Chen
Venue: NeurIPS 2022 Accept (2022)
Keywords: Adversarial Training
Availability Attacks
Hypocritical Perturbations
Stability Attacks

Abstract: Adversarial training, originally designed to resist test-time adversarial examples, has shown to be promising in mitigating training-time availability attacks. This defense ability, however, is challenged in this paper. We identify a novel threat model named stability attack, which aims to hinder robust availability by slightly manipulating the training data. Under this threat, we show that adversarial training using a conventional defense budget $\epsilon$ provably fails to provide test robustness in a simple statistical setting, where the non-robust features of the training data can be reinforced by $\epsilon$-bounded perturbation. Further, we analyze the necessity of enlarging the defense budget to counter stability attacks. Finally, comprehensive experiments demonstrate that stability attacks are harmful on benchmark datasets, and thus the adaptive defense is necessary to maintain robustness.

Can Adversarial Training Be Manipulated By Non-Robust Features?
Lue Tao Lei Feng Hongxin Wei
Jinfeng Yi
Sheng-Jun Huang Songcan Chen
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
Chongging University, Chongqing, China
RRIENN Center for Advanced Intelligence Project, Japan
*Nanyang Technological University, Singapore
$ JJD AI Research, Beijing, China
MIIT Key Laboratory Pattern Analysis and Machine Intelligence,
Nanjing University of Aeronautics and Astronautics, Nanjing, China
Abstract
Adversarial training, originally designed to resist test-time adversarial examples, has shown to promising in mitigating training-time availability attacks. This defense ability, however, is challenged in this paper. We identify novel threat model named stability attack, which aims to hinder robust availability by slightly manipulating the training data. Under this threat, show that adversarial training using a conventional defense budget e provably fails to provide test robustness in simple statistical setting, where the non-robust features of the training data can be reinforced by e-bounded perturbation. Further, we analyze the necessity enlarging the defense budget counter stability attacks. Finally, comprehensive experiments demonstrate that stability attacks are harmful on benchmark datasets, and thus the adaptive defense is necessary maintain robustness
Introduction
Robustness to input perturbations is crucial to machine learning deployment in various applications, such as spam filtering 133 and autonomous driving (6]. One of the most popular methods for
improving test robustness is adversarial training (39 1. By augmenting the training data with
e-bounded and on-the-fly crafted adversarial examples, adversarial training helps the learned model
resist test-time perturbations 39
On the other hand, machine learning systems are vulnerable to training-time availability attacks [ In particular, small perturbations applied into the training data (before training) suffice to degrade the overall test performance naturally trained models [16] 28). Fortunately, recent work has proven that adversarial training (39 is capable of mitigating this type of threat [64]. In other words, even if the training data is manipulated to maximize the test error, considerable accuracy on clean test data can still be achieved by ad adversarially trained models. However, previous work hardly inspects the test robustness of the models, which is what adversarial training was originally proposed for 23 This naturally raises the following question:
Are the models adversarially trained on the manipulated data robust to test-time perturbations?
'Work done when Lue was master's student at Nanjing University of Aeronautics and Astronautics. CCorresponding author: Sonecan Chen <s.chen nuw..edu.cn. 'Our code is available at he/https:/// /tttps://githutt htt.t//.........aaaaaaaaoeeee.aa Porturraaitoottuurtiiiiooo
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Good accuracy
Adv. train.
Good robustness
Clean data
Stability attacks
Test data
Good accuracy
Adv. train.
Bad robustness
Perturbed data
Figure 1: An illustration of stability attacks, where the training data is slightly perturbed to hinder adversarial training.
In this work, we show that conventional adversarial training may fail to provide test robustness when the training data is manipulated by adversary, and thus adaptive defense is necessary resolve this issue. Our contributions are summarized as follows:
1. We introduce novel threat model called stability attack, where an adversary aims to degrade the overall test robustness adversarially trained models by slightly perturbing the training data. Figure |
Tillustrates the threat of stability attacks.
2. We show that adversarial training using a conventional defense budget provably fails under stability attacks in simple statistical setting. Specifically, defense budget of will produce models that are not robust to e-bounded adversarial examples when the training data is hypocritically perturbed.
3. We unveil that the aforementioned vulnerability stems from the existence of the non-robust (predictive, yet brittle) features [29] in the original training data. When the non-robust features are reinforced by hypocritical perturbations, the conventional defense budget will insufficient to offset the negative impact.
4. We further show that defense budget of 2c is capable of resisting any stability attack for adversarial training, while the budget can be reduced to e + 77 in a simple statistical setting, where is the magnitude of the non-robust features.
5. We demonstrate that stability attacks are harmful to conventional adversarial training on benchmark datasets. In addition, our empirical study suggests that enlarging the defense budget is essential for mitigating hypocritical perturbations.
To the best of our knowledge, this is the first work that studies the robustness of adversarial training against stability attacks. Both theoretical and empirical evidences show that the conventional defense budget is insufficient under the threat of e-bounded training- time perturbations. Our findings suggest that practitioners should consider larger defense budget of no more than 2e (practically, about 1.5c 1.75e) to achieve better e-robustness.
2 Threat Models
In this section, we formally introduce the threat model stability attacks. We begin by revisiting the concepts of natural risk, adversarial risk, and delusive attacks. These concepts naturally give rise to our formulation of stability attacks.
2.1 Preliminaries
Setup. We consider a classification task with input-label pairs (æ, from an underlying distribution D over X [k]. The goal to learn (robust) classifier / X - [k] that predicts label for given input ac
Natural training (NT). Most learning algorithms aim to maximize the generalization performance on unperturbed examples, i.c., natural accuracy. The goal is to minimize the natural risk defined as:
Reat(f)
IE
(I)
(*.a)~D

---

Table 1:
Comparisons between evasion attacks, delusive attacks, and stability attacks.
Training-time perturbation
Test-time perturbation
Threat model
Learning scheme
Test performance
None
X
X
NT
Good
NT
Bad
Evasion attacks 5
60
X
AT
Good
NT
Bad
Delusive attacks 44
16
64
X
AT
Good
AT (conventional)
Bad
Stability attacks (this paper)
AT (our improved)
Good
Adversarial training (AT). Since the risk of adversarial examples (a.k.a. evasion attacks) was found to be unexpectedly high [0] has become increasingly important defend the learner
against the worst-case perturbations [23 39] In this context, the goal is to train model that has low
adversarial risk given a defense budget et
Radv (f) =
E
max C( ( 80.
(ar,y))
BEA
where choose A to be the set of e-bounded perturbations, i.e., A {5 ||ا|| S e). This choice is the most common one in the context of adversarial examples [67]. To simplify the notation, we refer to the robustness with respect to this set e-robustness. It is worth noting that Ras-(f) Rus (f) always holds for any f. and the equation holds when e = O.
Delusive attacks. Delusive attacks, which belong to training-time availability attacks, aim to prevent the learner from producing an accurate model by manipulating the training data "imperceptibly" [4]. Concretely, the features of the training data can be perturbed, while the labels should remain correct (2, 28 81 [5] 64 18 19). This malicious task can be formalized into the following bi-level optimization problem:
max E [C(Fp(x)),)) v)] PES
(0.
fo E arg min (TTT
Σ [C[f(x; +p).
Here, the adversary aims to maximize the natural risk of the model fo (that is trained the manipulated training set) by applying the generated perturbations p = {p, {p,),_ into the original training set T {(rr; yi) ) The commonly used feasible region is S = ((p.)_.
|| S 6).
Generally, solving Equation B computationally prohibitive for neural networks
64 18 Thus,
various heuristic methods are proposed to achieve the goal. Among them, representative method is the hypocritical perturbation
663
64 crafted as follows:
min L(Jera + p.). y1), 1pp
where farat called the crafting model, pre-trained before generating poisons. Tao et al. (64] simply
adopted naturally trained classifier as the crafting model, while Huang et al. (28) proposed
min-min bi-level optimization process to pre-train the crafting model. Fu et al. [19] further built their
crafting model via min-min-mas three-level optimization process, and generated their poisons by replacing Equation 4 with min-max bi-level objective.
Another representative method of delusive attacks is the adversarial perturbation, crafted by solving
max Li + p:). y:). |p
Tao et al. 664 and Fowl et al. 18 both found that applying the adversarial perturbation to the training data is very effective compromising naturally trained models. However, adversarial training has proven to be promising in defending against various delusive attacks [64).
3

---

2.2 Stability Attacks
In contrast to delusive attacks that aim increasing the natural risk, stability attacks attempt to maximize the adversarial risk of the learner by slightly perturbing the training data:
E
max C(fr(z + 6). BEA
(6)
PES (2.9)n
where fr denotes the victim model, which is naturally or adversarially trained on the perturbed data. In other words, stability attacks seek to hinder the robust availability of the training data. Table 1 shows the comparisons among different threat models.
The goal of stability attacks can be immediately achieved for naturally trained models, since they have already incurred high adversarial risk, even if the training data is clean [60]. To ease the problem of high adversarial risk, adversarial training has been widely used to improve model's adversarial robustness 24 12). Hence, the main goal of stability attacks becomes to compromise the test. robustness of ad adversarially trained models.
Note that Equation (6) is multi-level optimization problem that is not easy to solve, our next question how to conduct effective stability attacks against adversarial training. In the following sections, we introduce an effective stability attack method and analyze the cost of resisting it.
This work focuses on adding bounded pertubrations as small as possible. We mostly assume that the adversary has full control of training data (instead of changing few) by following previous works [16 (28] 64 [7H19]. This is a realistic assumption [16. For instance, in some applications an organization may agree to release some internal data for peer assessment, while preventing competitors from easily building a model with high test robustness; this can be achieved by perturbing the entire dataset via stability attacks before releasing. Moreover, this assumption enables a worst-case analysis of the robustness of adversarial training, which may facilitate important theoretical implications.
Remark 1.
3 How to Manipulate Adversarial Training
Previous work suggests that adversarial training could defend against both evasion attacks and delusive attacks 33 64). However, in this paper, we show that adversarial training using a conventional defense budget e may not be sufficient to provide e-robustness when confronted with stability attacks. In particular, present simple theoretical model where the conventional defense scheme provably fails when the training data is hypocritically perturbed.
The binary classification task. The data model is largely based on the setting proposed by Tsipras et al. (67,, which draws a distinction between robust features and non-robust features. Specifically, consists of input-label pairs (æ, v) sampled from Gaussian mixture distribution D as follows:
u.a [-1,+1}, 21 ~N(y,
f.dd N(my, o²),
where 1) is much smaller than (i.e., < < 1). Hence, samples from consist robust feature (x1) that is strongly correlated with the label, and d non-robust features (#2...., Ed+ 1) that are very weakly correlated with it. Typically, an adversary can manipulate a large number of non-robust features e.g. d ((1/?²) will suffice.
Before introducing the way to hinder robust availability, we briefly illustrate the success of adversarial training when the training data is unperturbed.
Natural and robust classifiers. For standard classification, we consider a natural classifier:
fam( := sign TT)), where Was [1 ..).
which minimizer of the natural risk (T) with 0-1 loss on the data (7). i.e., the Bayes optimal classifier. However, in the adversarial setting, this natural classifier is quite brittle. Thus, it is imperative to obtain a robust classifier:
frab(a:): == s)) where wioh == (1,0,..,,0)
(9)
which relies only on the robust feature 21.

---

Illustration of adversarial accuracy. In the adversarial setting. an adversary that is only allowed to perturb each feature by a moderate can effectively subvert the natural classifier Tsipras et al. [67]. In particular, if 21y. an adversary can essentially force each non-robust feature to be anti-correlated with the correct label. The following proposition, proved in Appendix C.T gives the adversarial accuracies of the natural classifier fatt (8) and the robust classifier fob
Proposition 1. Let = 2n and denote by (f) the adversarial accuracy, i.e, the probability of a classifier correctly predicting y on the data 17) under too perturbations. Then, we have
(sor() = Pr Pr (( < 1- 1-2n Aadv (Snat) S SP P PP((NO.1) 0V1 + dip
Pr
1 dn
}
a
Proposition [ [implics that the adversarial accuracy of the natural classifier is < 50% when d 2 1/1²" Even worse, when I 1/3 and d 2 3/1²² the adversarial accuracy of the natural classifier (8) is always lower than 1%. In contrast, the robust classifier (9) yields a much higher adversarial accuracy (always > 50%); when I (1 /3, its adversarial accuracy will be higher than 99%.
3.1 Hypocritical Features Are Harmful
The results above reveal the advantages of robust classifiers over natural classifiers. Note that such robust classifier can be obtained by adversarial training on the original data (7). However, this defense effect may not hold when the adversary is allowed to perturb the training data.
We show this by analyzing two representative perturbations: the adversarial perturbation 664 18] and the hypocritical perturbation [63] 64). When applied into the training data, both perturbations are effective as delusive attacks for naturally trained models. In the following, we show that the former is harmless: adversarial training using defense budget e on the adversarially perturbed data can still provide test robustness. In contrast, the latter is harmful: we find that the same defense budget can only produce non-robust classifiers when the training data is hypocritically perturbed.
harmless case. Consider an adversary who is capable of perturbing the training data by an attack budget e. The adversary may choose shift each feature towards -y Hence, the learner would see input-label pairs (æ, v) sampled i.i.d. from a training distribution Tadv as follows:
y u.g. (-1,+1), 21 ()),)) T2 Tg...., Td+1 Lid N((nn -y,
(10)
where each feature of the samples from Tady is adversarially perturbed by a moderate c. While these samples are deviate significantly from the original distribution D (7), adversarial training on them using a defense budget e still able to neutralize the non-robust features. Formally, Appendix C.2 we prove the following theorem.
Theorem (Adversarial perturbation harmless). Assume that the adversarial perturbation in the training data Tad (TO) is moderate such that n1/2 S < 1/2. Then, the optimal linear -robust classifier obtained by minimizing the adversarial risk on Tado with a defense budget is equivalent to the robust classifier (9)
This theorem indicates that the adversarial perturbation is harmless: e-robustness can still be obtained by adversarial training on such perturbed training data.
A harmful case. perturbation. That is, the adversary can instead shift each feature towards Hence, the learner would see input-label pairs (xx y) sampled i.i.d. from a training distribution Thyp as follows
However, this defense effect can be completely broken by the hypocritical
y W-a." (11,+1),
~N11 + o),, 32, Te+1
Lid N((n + +),,03)
(11)
where each feature of the samples from Thyp is reinforced by magnitude of c. While these samples become more separable, adversarial training on them using the same defense budget will fail to neutralize the hypocritically perturbed features. Consequently, the resulting classifiers will inevitably have low adversarial accuracy. We make this formal in the following theorem proved in Appendix CC.
"To see how this relates to the hypocritical perturbation (4) let us consider the logistic loss [(f(x) v) log(1 + exp(-yf(x)) and use the natural classifier (8) as the crafting model. Then, the problem 4 has closed-form solution p ye sign(ws,) = [yc.... yc.. Applying p" to each æ yields the distribution Topp
5

---

Theorem 2 (Hypoeritical perturbation is harmful). The optimal linear -robust classifier obtained by minimizing the adversarial risk on the perturbed data Tryp to the natural classifier (8).
(TT) with a defense budget equivalent
This theorem implies that the conventional defense scheme can only produce non non-robust classifiers, whose adversarial accuracy is as low as that of the natural classifier (8). That is saying, if 2m, a $ 1/3 and 2 3/m²,, the classifiers cannot get adversarial accuracy better than 1%.
Implications. As it turns out, the seemingly beneficial features in Tayp Therefore, the adversary is highly motivated to hide such hypocritical features in the training data, intending to cajole an innocent learner into relying the non-robust features. Intriguingly, notice that the natural classifier (8) (i.e., the crafting model used to derive the distribution Thyp) actually has 17-robustness. This essentially because the non-robust features in the data (7) can resist small-magnitude perturbations by design. This motivates us to use "slightly robust" classifiers as the crafting model in practice. Indeed, our experimental results show that training the crafting model with 0. 25e-robusss performs the best for conducting stability attacks, This is different from the that use naturally trained models as the crafting model for poisoning.
actually hypocritical.
previous works 664 18
4 The Necessity of Large Defense Budget
We have shown that the hypocritical perturbation is harmful to the conventional adversarial training scheme. Fortunately, it is possible to strengthen the defense by using larger defense budget, while the crux of the matter is how large the budget is needed.
We find that the minimum value of the defense budget for a successful defense depends on the specific data distribution. Let us first consider the hypocritical data in Thyp (TT. In this case, we show that a larger defense budget is necessary in the following theorem proved Appendix C.4
Theorem 3 (E + 1) is necessary). The optimal linear tso-robua classifier obtained by minimizing the adversarial risk on the perturbed data Thop (77) with a defense budget e + 17 is equivalent to the robust classifier (0). Moreover, any defense budget lower than + 1j will yield classifiers that still rely on all the non-robust features. featuress.
This theorem implies that, in the case of the mixture Gaussian distribution under the threat of c- bounded hypocritical perturbations, the learner needs slightly larger defense budget c + 17 to ensure c-robustness.
While it is challenging to analyze the minimum value of the defense budget in the general case, the following theorem provides upper bound of the budget.
Theorem 4 (General case). For any data distribution and any adversary with attack budget c, training models to minimize the adversarial risk with a defense budget 2e on the perturbed data is sufficient to ensure e-robustness.
C.5 It implies that a defense budget twice to the The proof of Theorem 4 is deferred in Appendix
attack budget should be safe enough under the threat stability attacks. Theorem B also suggests that the minimum budget might be much smaller than 2e, and it depends on the specific attack methods and data distributions. In the following section, we empirically search for an appropriate defense budget on real-world datasets.
5 Experiments
In this section, we conduct comprehensive experiments to demonstrate the effectiveness of the hypocritical perturbation stability attacks popular benchmark datasets and the necessity of an adaptive defense for better robustness.
We conduct stability attacks by applying hypocritical perturbations into the training set. We focus on an lxx adversary with an attack budget e, 8/255 by following [28
Our crafting model is adversarially trained with a crafting budget e, 2/255 for 10 epochs before generating perturbations. Unless otherwise specified, we use R8 (66 as the default architecture for both the crafting model and the learning model. For adversarial training, mainly follow the settings in

---

Table 2: Test robustness (%) PGD-AT using defense budget cd = 8/255 on CIFAR-10 -
Attack
Natural
FGSM
PGD-20
PGD- RGD-100
CW
AutoAttack
None (clean)
82.17
56.63
50.63
50.35
49.37
46.99
DeepConfuse 16
81,25
54.14
48.25
48.02
47.34
44.79
Unlearnable Examples
28
83.67
57.51
50.74
50.31
49.81
47.25
NTGA [8
82.99
55.71
49.17
48.82
47.96
45.36
Adversarial Poisoning
18
77.35
53.93
49.95
49.76
48.35
46.13
Hypocritical Perturbation (ours)
88.07
47.93
37.61
36.96
38.58
35.44
Table Test robustness (%) PGD-AT using a defense budget 6d = 8/255 across different datasets.
Dataset
Attack
Natural
FGSM
PGD-20
PGD-100
CW
AutoAttack
None
93.95
71.83
57.15
56.02
54.93
50.50
SVHN
Adv.
87.50
56.12
46.71
46.32
45.70
42.48
Hyp.
96.06
59.41
38.17
37.29
40.54
35,43
None
56.15
31.50
28.38
28.28
26.53
24.30
CIFAR-100
Adv.
52.14
28.59
26.19
26.09
24.36
22.71
Hyp.
62.22
26.38
21.51
21.13
21.13
18.74
None
49.34
25.67
22.99
22.86
20.67
18.54
Imageem
Adv.
49.52
22.93
20.01
19.91
18.75
16.83
Hyp.
55.92
20.21
15.61
15.26
14.99
12.53
Table Test robustness (%) of PGD-AT using defense budget e, = 8/255 on CIFAR-10 across different architectures. Test robustness is evaluated by PGD-20. Values in parenthesis denote the accuracy on natural test data.
Attack
VGG-16
GoogLeNet
DenseNet-
MobileNetV2
WideResNet-286
None
47.37 (77.15)
50.67 (83.03)
49.92 (80.08)
48.51 (80.83)
53.91 (85.81)
Adv.
44.70 (73.24)
47.72 (79.34)
48.00 (78.17)
45.90 (74.61)
51.01 (82.43)
Hyp.
34.34 (87.20)
37.03 (87.61)
37.58 (88.04)
35.58 (87.04)
41.07 (89.14)
By convention, the defense budget is equal to the attack budget, i.c., <d 8/255. More details on experimental settings are provided in Appendix
previous studies 83 70 50].
5.1 Benchmarking
Attack evaluation. We compare our crafted hypocritical perturbation to existing methods, which were originally proposed delusive attacks, including DeepConfuse (which builds adversarial auto-encoder to generate their perturbations) [16]. Unlearnable Examples (which use a min-min bi-level optimization process to pre-train their crafting model)
(28). NTGA (which adopts neural tangent kernels as its crafting model) [8I] and Adversarial Poisoning (whose crafting model is simply naturally trained classifier) [18]. It is noteworthy that none of these previous works evaluated the
test robustness of their poisoned models.
Results using ResNet-18 18 on CIFAR-10 are summarized in Table "Natural" denotes the accuracy on natural test data. Various test-time adversarial attacks are used to evaluate test robustness, including FGSM, PGD-20/100, CW oo (foo version of CW loss (9] optimized by PGD-100), and AutoAttack (a reliable evaluation metric via an ensemble of diverse attacks (T]). We observe that the hypocritical perturbation widely outperforms previous training-time time perturbations in degrading the test robustness of PGD-AT [39 This demonstrates that stability attacks are indeed harmful to the conventional defense scheme. We note that method increases the natural accuracy. This is reasonable, since our analysis in Section 3.1 has implied that the hypocritical perturbation can increase model reliance on the non-robust features, which predictive but brittle (29].
Moreover, we evaluate hypocritical perturbation on other benchmark datasets including SVHN, CIFAR-100 100, and ImageNe. Both the crafting model and the victim model use the ResNet-

---

robustresss (96)
perturbation perturbation
rabustness (86
perturtation perturboriss
Adx pettuiaaion perturtation
Tost roburouuttnn cobustresss (16)
30
65-
45
RB
45
40-
40
40-
PGDJAT TRADES MAAT
15
Epoch
PGD steps
(a) Crafting budget
(b) Crafting epoch
(c) Number of PGD steps
(d) Large defense budget
Figure The ablation study experiments on CIFAR-10 10. Test robustness (%) is evaluated by PGD-20.
Table 5: Test robustness (%) of various adaptive defenses on the hypocritically perturbed CIFAR-10.
Defense
Natural
FGSM
PGD-20
PGD-100
CW nc
AutoAttack
PGD-AT (Ea 8/255)
88.07
47.93
37.61
36.96
38.58
35.44
+ Random Noise
87.62
47.46
38.35
37.90
39.07
36.25
+ Gaussian Smoothing
83.95
50.96
42.80
42.34
42.41
40.07
+ Cutout
88.26
49.23
39.77
39.25
40.38
37.61
AutoAugment
86.24
48.87
40.19
39.65
37.66
35.07
PGD-AT (e.r 14/255)
80.00
56.86
52.92
52.83
50.36
48.63
TRADES (Ea 12/255)
79.63
55.73
51.77
51.63
48.68
47.83
MART (ea 14/255)
77.29
57.10
53.82
53.71
49.03
47.67
Table 6: Test robustness (%) of PGD-AT by adjusting the amount of clean data included in the manipulated CIFAR-10. Test robustness (%) is evaluated by PGD-20. Values in parenthesis denote the accuracy on natural test data.
Attack' Clean proportion
0.1
0.4
0.6
0.8
None (clean subset)
30.65 (63.90)
37.99 (70.99)
44.95 (77.11)
47.17 (80.33)
49.78 (81.60)
Adversarial Perturbation
48.33 (77.71)
48.23 (76.94)
49.68 28.8 (78.54)
50.15 (82.46)
51.21 (82.03)
Hypocritical Perturbation
41.51 (87.49)
43.66 (88.30)
46.98 (86.46)
49.20 (85.29)
50.56 (82.72)
architecture. Results summarized in Table "Hyp." denotes the hypocritical perturbation gener- ated by our crafting model. As a comparison, we also evaluate the adversarial perturbation generated using the same crafting model (denoted "Adv."). Again, the results show that the hypocritical perturbations are more threatening than the adversarial perturbations to standard adversarial training. This phenomenon is consistent with analytical results in Section 3.1
Besides, we find that the hypocritical perturbation can transfer well from ResNet- 18 to other architec- tures, successfully degrading the test robustness of wide variety of popular architectures including VGG-16 [57], GoogLeNet [61, DenseNet-121 (27], MobileNetV2 [52], and WideResNet-28: 10 (82). shown in Table Note that this is completely black-box setting where the attacker has knowledge of the victim model's 's initialization, architecture, learning rate scheduler, etc.
Adaptive defense. prevent the harm of stability attacks, our analysis in Section suggests that larger defense budget would be helpful. We find that this is indeed the case on CIFAR-10. As shown in Table 5 a large defense budget E = 14/255 for PGD-AT performs significantly better than the conventional defense budget ed 8/255. We also combine several data augmentations with PGD-AT as defenses by following Fowl et al. [8]. The results show that they are beneficial, while their improvements are inferior to PGD-AT with e
14/ /255. In addition, we adopt other adver- sarial training variants including TRADES 33 and MART 770 to defend against the hypocritical perturbation, and find that they achieve comparable defense effects with large defense budgets.
Finally, we note that the adaptive defense has several limitations: i) robust accuracy is improved at the cost of natural accuracy; ii) finding appropriate defense budget is time-consuming for adversarial training; úi) adversarial training with large budgets may lead to learning obstacles such as inherent large sample complexity (53]. We leave the detailed study of these questions as future work.

---

5.2 Ablation Studies
In this part, we conduct set of experiments to provide an empirical understanding of the proposed attack. We train ResNet-18 18 using PGD-AT on CIFAR-10 by following the same settings described in Appendix Dunneess otherwise specified.
Analysis on the crafting method. Different from previous work, we use "slightly robust" classifiers as our crafting model. Figure Figure((a)) shows that this technique greatly improves the potency of the hypo- critical perturbation, where the crafting budget & 2/255 performs best in degrading test robustness. We also observe that training the crafting model for 10~80 epochs works well in Figure 2(b) and that optimizing the crafted perturbations over 100 steps performs well in Figure 2(c) Finally, note that Fowl et al. [88 also tried to use adversarially trained models as the crafting model, but they failed to produce effective attack in this way. This is mainly because they adopted adversarial perturbations as poisons, which, as we observed, are inferior in degrading test performance.
Ablation on defense budget. As discussed in Section we are motivated to find the appropriate defense budget ea in the range je ~ 2e]. Figure 2(d) shows that the optimal defense budgets against the hypocritical perturbation are 14/255, 12/255, and 1 14/255 for PGD-AT, TRADES, and MART, respectively. We also observe that all these adversarial training variants are inferior when using the conventional defense budget 8/255.
Less data. We follow Fowl et al. [8] to test the effectiveness of attacks by varying the proportion of clean data and perturbed data. Attacks are then considered effective they cannot significantly increase performance over training on the clean subset alone. As shown in Table the proposed attack often degrades the test robustness below what one would achieve using full clean dataset. More importantly, the hypocritical perturbations consistently more harmful than the adversarial perturbations. This again verifies the superiority of hypocritical pertubrations as stability attacks.
Effect on natural training. As a sanity check, we include the test accuracy of naturally trained models on CIFAR-10 in Table shows that without adversarial training, the test robustness of the models becomes very poor-all models only have 0% accuracy under PGD-20 attack. Thus, the goal of stability attacks is immediately achieved. On the other hand, We find that our method degrades the test accuracy from 94.23% to 75.92% though this is not the main focus of this work. We also observe that Adversarial Poisoning [18 is the most effective method in degrading the test accuracy of naturally trained models. This observation is consistent with Fowl et al. [8]
Table 7: Test robustness (%) of natural training on CIFAR-10.
Attack
Natural
PGD-20
None (clean)
94.23
0.00
DeepConfuse 16
17.22
0.00
Unlearnable Examples
28
22.72
0.00
NTGA [1
11.15
0.00
Adversarial Poisoning 18
8.60
0.00
Hypocritical Perturbation (ours)
75.92
0.00
Related Work
Adversarial training. The presence of non-robust features has been demonstrated popular benchmark datasets [1] which naturally leads to model vulnerability to adversarial examples [67] 58). To improve test robustness against adversarial examples, adversarial training methods have been
developed [33 39
62 [69]. Usually, adversarial training using a defense budget is expected to improve model robustness against e- e-bounded adversarial examples. Thus, to break this defense, a direct way is to enlarge the typical e-ball used to constrain the attack; however, this may risk changing the true label 88 66). In this work, we aim to show that it is possible achieve this by slightly perturbing the training data without enlarging the e-ball.
B
83
65
46
76
86
Data poisoning. Data poisoning attacks, which manipulate training data to cause the resulting models to fail during inference 3. can be divided into availability attacks (to degrade over- all test performance) 4
[8] and integrity attacks (to cause specific misclassifica- While the stability attacks studied in this work may be reminiscent of we note that they share several key differences. First, stability attacks aim to hinder adversarial training with well-defined e-robustness, while backdoor attacks mainly focus
77
tions) 32 10
55
87
21
54
buckdoor attacks
10

---

embedding malicious behaviors (that can be invoked by pre-specified triggers) into naturally trained models [22 68]. Second, stability attacks only perturb the imputs slightly, while most works on
backdoor attacks require mislabeling 25 37 45
36
74). Thus, backdoor defenses 7 75 35
might
not be directly applied to resist stability attacks.
Additional related works are discussed in Appendix A
7 Conclusion
In this work, we establish a framework to study the robustness of adversarial training against stability attacks. We unveil the threat of stability attacks-smal hypocritical perturbations applied into the training data suffice to hinder conventional adversarial training. The conventional defense budget e is insufficient under the threat. To resist it, we suggest a larger defense budget of no more than 2e. Our theoretical analysis explains why hypocritical perturbations are effective as stability attacks- they can mislead the learner by reinforcing the non-robust features. Experiments demonstrate that hypocritical perturbations are harmful to conventional adversarial training on benchmark datasets, and enlarging the defense budget is essential for mitigating hypocritical pertubrations. Future work includes relaxing the assumption that the adversary perturbs the entire training set and designing more effective stability attacks against adversarial training.
Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China (Grant No. 61732006. 62076124, 62076128, 62106028), the National Key R&D Program of China (2000AA00107000), Lei Feng was also supported by Chongqing Overseas Chinese Entrepreneurship. Innovation Support Program and CAAl-Huawei MindSpore Open Fund. The authors wish thank the anonymous reviewers for their helpful comments and suggestions.
References
[1] Anish Athalye. Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses adversarial examples. In ICMI., 2018.
[2] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In ICML, 2018
[3] Battista Biggio and Fabio Roli. Wild pattern:: Ten years after the rise of adversarial machine learning. Puttern Recognition, 84:317-331, 2018
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In ICML, 2012.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šmdić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In ECML-PKDD, 2013.
[6] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. In NeurlPS Deep Learning Symposium, 2016.
[7] Eitan Borgnia, Valeriaa Cherepanova, Liam Fowl, Amin Ghiasi, Jonas Geiping, Micah Goldblum, Tom Goldstein, and Arjun Gupta. Strong data augmentation sanitizes poisoning and backdoor attacks without accuracy tradeoff. In /CASSP, 2021.
[8] Nicholas Carlini. A critique of the deepsec platform for security analysis of deep learning models. arXiv preprint
2019.
[9] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In S&P, 2017.
[10] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep leaming systems using data poisoning. arXiv preprint /rx.::/7 2017.
[11] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fiee attacks. In ICML, 2020.
10

---

[12] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion. Mung Chiang, Prateek Mittal, and Matthias Heii.. Robustbench: a standardized adversarial robustness benchmark. In NeurlPS Datasets and Benchmarks Track, 2021. URL https://openreviert net/
[13] Nilesh Dalvi, Pedro Domingos, Sumit Sanghai, and Deepak Verma. Adversarial classification. In KDD, 2004.
[14] Edgar Dobriban, Hamed Hassani, David Hong, and Alexander Robey. Provable tradeoffs in adversarially robust classification. arXiv preprint arXiv:2006.00 05161, 2020.
[15] Ivan Evtimov. Ian Covert, Aditya Kusupati, and Tadayoshi Kohno. Disrupting model training with adversarial shortcuts. In JCML 2021 Workshop, 2021.
[16] Ji Feng, Qi-Zhi Cai, and Hua Zhou. Learning to confuse: generating training time adversarial data with auto-encoder. In NewrIPS, 2019.
[17] Liam Fowl, Ping-yeh Chiang, Micah Goldhlum, Jonas Geiping, Arpit Bansal, Wojtek Czaja, and Tom Goldstein. Preventing unauthorized use proprictary data: Poisoning for secure dataset release. arXiv preprint arXiv: 103.:::10030028.33 2021.
[18] Liam Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojtek Czaja, and Tom Goldstein. Adversarial examples make strong poisons. In NeurIPS, 2021.
[19] Shaopeng Fu. Fengxiang He, Yang Liu, Li Shen, and Dacheng Tao. Robust unlearnable examples: Protect- ing data privacy against adversarial learning. In International Conference Learning Representations, 2022. URL httpp:://oppnrreiiess wwtp::rwwwwwweeeeeerooee/daaaaaa wwwp://oooourr.ddaauPpPuu.
[20] Yinghua Gao, Dongxian Wu, Jingfeng Zhang, Shu-Tao Xia, Gang Niu, and Masashi Sugiyama. Does adversarial robustness really imply backdoor vulnerability?, 2022. URL httppee://ppeeeeeieeeeee
Forum?id=nCADkene
[21] Jonas Geiping, Liam H Fowl, W Ronny Huang, Wojeiech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching, In ICLR, 2021.
[22] Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. arXiv preprint arXiv:2012. 10544, 2020.
[23] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
[24] Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering limits of adversarial training against norm- bounded adversarial examples. arXiv preprint arXiv:2010.03593 2020.
[25] Tianyu Gu, Brendan lcarning model supply chain. arXiv preprint /ti::/70006633 2017.
and Siddharth Garg, Badnets: Identifying vulnerabilities in the machine
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[27] Gao Huang, Zhuang Liu. Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[28] Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearnable examples: Making personal data unexploitable. In JCLR, 2021.
[29] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry, Adversarial examples are not bugs, they are features. In NewrlPS, 2019.
[30] Adel Javanmard and Mahdi Soltanolkotabi. Precise statistical analysis classification accuracies for adversarial training. arXiv preprint arXiv:2010.1 /1213, 2020.
[31] Junho Kim, Byung-Kwan Lee, and Yong Man Ro. Distilling robust and non-robust features in adversarial examples by information bottleneck. In NewrIPS, 2021.
[32] Pang Wei Koh and Percy Liang. Understanding black-bor box predictions via influence functions. In ICML, 2017.

---

[33] Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
[34] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[35] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. In NewrlPS, 2021.
[36] Yuezun Li, Yiming Li. Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In CVPR, 2021.
[37] Yingqi Liu, Shiqing Ma. Yousra Aafer, Wen Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. 2017.
[38] Zhuoran Liu, Zhengyu Zhao, Alex Kolmus, Tijn Berns, Twan van Laarhoven, Tom Heskes, and Martha Larson. Going grayscale: The road to understanding and improving unlcarnable examples. arXiv preprint arXiv:2111.13244 2021.
[39] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In /CLRR 2018.
[40] Mohammad Mehrabi, Adel Javanmard, Ryan A Rossi, Anup Rao, and Tung Mai. Fundamental tradeoffss distributionally adversarial training. In ICML, 2021.
[41] Luis Muioz-Gonzzlle,, Battista Biggio. Ambra Demontis, Andrea Paudice, Vasin Wongrassamee,, Emil Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. In ACM Workshop on Artificial Intelligence and Security, 2017.
[42] Preetum Nakkiran. A discussion of 'adversarial examples are not bugs, they are features': Adversarial examples are just bugs, too. Distill, 2019. doi: 10.2391 1//disiil..000119.5 bugs-discussioo/responss-
http://diiiill.pub//1999//
[43] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[44] James Newsome, Brad Karp, and Dawn Song. Paragraph: Thwarting signature learning by training maliciously. In International Workshop on Recent Advances in Intrusion Detection, 2006.
[45] Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In NeurIPS, 2020.
[46] Tianyu Pang. Xiao Yang. Yinpeng Dong, Hang Su, and Jun Zhu. Bag of tricks for adversarial training. In ICLR, 2021.
[47] Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Accumulative poisoning attacks on real-time data. In NewrlPS, 2021.
[48] Tianyu Pang, Min Lin, Xiao Yang. Jun Zhu, Shuicheng Yan. Robustness and accuracy could be reconcilable by (proper) definition. arXiv preprins orXiv:2202.10100 2022.
[49] Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John Duchi, and Percy Liang. Understanding and mitigating the tradcoff between robustness and accuracy. In ICML, 2020.
[50] Leslie Rice, Fric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In ICMI., 2020.
[51] Aniruddha Saba, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In AAAI, 2020.
[52] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018
[53] Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adversarially robust generalization requires more data. In NeurlPS, 2018.
[54] Avi Schwarzschild, Micah Goldblum, Arjun Gupta, John P Dickerson, and Tom Goldstein. Just how toxic is data poisoning? unified benchmark for backdoor and data poisoning attacks. In ICML, 2021.
[55] Ali Shafahi, W Ronny Huang, Mahyar Najibi. Octavian Suciu. Christoph Studer, Tudor Dumitras. and Tom Goldstein. Poison frogs! targeted clban-label poisoning attacks on neural networks. In NewrIPS, 2018.
[56] Juncheng Shen, Xiaolei Zhu, and De Ma. Tensorclog: An imperceptible poisoning attack on deep neural network applications. IEEE Access, 2019.
12

---

[57] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In ICLR, 2015.
[58] Jacob Springer. Melanie Mitchell, and Garrett Kenyon. little robustness goes long way: Leveraging robust features for targeted transfer attacks. NeurIPS, 2021.
[59] Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin-Yu Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy?-a comprehensive study on the robustness 18 deep image classification models, ECCV, 2018.
[60] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
[61] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet. Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
[62] Jihoon Tack, Sihyun Yu, Jongheon Jeong. Minseon Kim, Sung Hwang, and Jinwoo Shin. Consistency regularization for adversarial robustness. In ICML 2021 Workshop on Adversarial Machine Learning. 2021.
[63] Lue Tao, Lei Feng, Jinfeng Yi, and Songean Chen. With false friends like these, who can notice mistakes? arXiv preprint 20fi::2012.17733:0 2020.
[64] Lue Tao, Lei Feng, Jinfeng Yi, Sheng-Jun Huang, and Songcan Chen. Better safe than sorry: Preventing delusive adversaries with adversarial training. In NewrlPS, 2021.
[65] Florian Tramer and Dan Bonch. Adversarial training and robustness for multiple perturbations. In NeurlPS, 2019.
[66] Florian Tramèr, Jens Behrmann, Nicholas Carlini, Nicolas Papernot, and Jörn-Henrik Jacobsen. Funda- mental tradeoffs between invariance and sensitivity to adversarial perturbations. ICML, 2020.
[67] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robust- ness may be at odds with accuracy. In ICLR, 2019.
[68] Alexander Turner. Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv:1912.02771 2019.
[69] Qizhou Wang, Feng Liu, Bo Han, Tongliang Liu, Chen Gong, Gang Niu, Mingyuan Zhou, and Masashi Sugiyama. Probabilistic margins for instance reweighting in adversarial training. In NewrIPS, 2021.
[70] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020.
[71] Zhirui Wang, Yifei Wang, and Yisen Wang. Fooling adversarial training with induction noise, 2022, URL https://openreevier wwt.:eeweeeeooooor/sddddooo.....
[72] Cheng- Hsin Weng. Yan-Ting Lee, and Mtan-Hung Brandon Wu. On the trade-off between adversarial and backdoor robustness. In NeurlPS, 2020.
[73] Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In ICML, 2018.
[74] Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, Chao Shen, and Hongyuan 7ha. Backdoorhench: A comprehensive henchmark of backdoor learning, arXiv preprint arXiv:2206. 12654, 2022.
[75] Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In NeurlPS, 2021.
[76] Dongxian Wu, Shu- Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In NeurlPS, 2020.
[77] Huang Xiao, Battista Biggio. Gavin Brown, Giorgio Fumera, Claudia Eckert, and Fabio Roli. Is feature selection secure against training data poisoning? In ICML, 2015.
[78] Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. To be robust or to be fair: Towards fairness in adversarial training. In ICML, 2021.
13

---

[79] Yua-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R Salakhutdinov, and Kamalika Chaudhuri. A closer look accuracy vs. robustness. NeurlPS, 2020.
[80] Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie- Yan Liu. Indiscriminate poisoning attacks are shortcuts. arXiv preprint arXiv:2111.0099) 2021.
[81] Chia-Hung Yuan and Shan- Hung Wu. Neural tangent generalization attacks. In ICML, 2021.
[82] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 1605.07146. 2016.
[83]
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoreti- cally principled trade-off between robustness and accuracy. ICML, 2019.
[84] Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankanhalli. Attacks which not kill training make adversarial learning stronger. ICML, 2020.
[85] Jingfeng Zhang, Xilie Xu, Bo Han, Tongliang Liu, Gang Niu, Lizhen Cui, and Masashi Sugiyama. Noilin: Do noisy labels always hurt adversarial training? arXiv preprint arXiv:2105. 14676, 2021.
[86] Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry- aware instance-reweighteed adversarial training. In ICLR, 2021.
[87] Chen Zhu, Ronny Huang, Hengduo Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clean-label poisoning attacks on deep neural nets. In ICML, 2019.
[88] Jianing Zhu. Jingfeng Zhang, Bo Han, Tongliang Liu, Gang Niu. Hongxia Yang. Mohan Kankanhalli, and Masashi Sugiyama. Understanding the interaction of adversarial training with noisy labels. arXiv preprint 1rXiv:2102.03482. 2021.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 2.2 and Section 7.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix F.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes] See the supplemental material.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? Yes] See Section 5.1.
(c) Did you report error bars (e.g.. with respect to the random seed after running experiments multiple times)? [Yes] See Appendix B.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] All experiments run on single NVIDIA GcForce RTX 3090 GPU.
4. If you using existing assets (c.g., code, data, models) curating/roleasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention license of the assets? [N/A]
(c) Did you include any new either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you's 're us-
ing/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
14

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent
participant compensation? [N/A]
15


Reviews:
[{"review_id": "hXC1-SbXJKP", "reviewer": "Reviewer_J69n", "summary": "This paper introduces the problem of adversarial training when they face the new type of attacks called stability attacks. The stability attacks aim to compromise the robust availability by slightly manipulating the training data. Most of existing methods neglect that the test robustness of the adversarial trained models when they are under the training-time availability attacks. Under the threat of the stability attacks, they demonstrate that the adversarial trained network with epsilon perturbation budget is not enough to defend against the epsilon bounded adversarial perturbation. The authors argue that it is necessary to enlarge the epsilon perturbation budget when they conduct the adversarial training.", "questions": "According to the paper, stability attack consists of training-time perturbation and test-time perturbation in Table 1. In Table 2, the authors compare the stability attacks with other training-time availability attacks (DeepConfuse, Unlearnable Examples, NTGA …). However, the training-time availability attacks have training-time perturbation only. Therefore, the test-time perturbation of FGSM, PGD-20, PGD-100, CW, and AutoAttack in Table 2 should be different for the stability attacks and for other training-time availability attacks. What is the exact experiment setting for the test-time perturbation under the threat of stability attacks?", "limitations": "Yes, the authors have addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths :\n1. Clear writing. Easy to understand and well-organized paper.\n2. Important experimental result. The test-robustness of adversarial trained network against evasion attacks when they are under the delusive attacks is intriguing result in the adversarial research.\n\nWeaknesses : \n1. Low originality: Missing comparison with the recent key reference [Ref_1] which is somewhat similar method of the training-time availability attacks. The attack generation algorithm of training-time and test-time perturbation is similar to [Ref_1]. Furthermore, [Ref_1] achieved the state-of-the-art attack performance against adversarial trained network on clean data. Thus, it is unclear that this paper made the fair comparison between current SOTA poisoning attacks on adversarial trained network. In this regard, the novelty of the threat model on the adversarial trained network by perturbing the training data is marginal.\n2. Insufficient explanations on the relationship with non-robust features. This paper passes the buck to the non-robust feature for the experiment results of the increase of standard accuracy and the decrease of robust accuracy when they are under the stability attacks. The authors emphasize the responsibility of the non-robust feature in the title, abstract, and throughout the paper. However, I am not convinced that non-robust feature is the only reason for the experiment result in Table 2. Natural test robustness has increased and test robustness under evasion attacks has decreased when they are adversarially trained under the stability attacks. But, considering the fact that there always exists a trade-off between the standard accuracy and the robust accuracy [Ref_2], non-robust feature can’t be solely blamed. As the authors followed the process of theoretical analysis with the [Ref_3], they need to present another empirical evidence, feature-level analysis, or visualizations for the explanations on relationship with non-robust features as [Ref_3] did.\n3. Confounding usage of the term for the similar concept. It is very confused when the term ‘Hyp’, ‘stability attacks’, and similar concept appears throughout the paper. \n\n[Ref_1] Fu, S., He, F., Liu, Y., Shen, L., & Tao, D. (2021, September). Robust unlearnable examples: Protecting data privacy against adversarial learning. In International Conference on Learning Representations.\n\n[Ref_2] Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., & Jordan, M. (2019, May). Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning (pp. 7472-7482). PMLR.\n\n[Ref_3] Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., & Madry, A. (2018). Robustness may be at odds with accuracy. arXiv preprint arXiv:1805.12152.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fyGZh3-Tg8B", "reviewer": "Reviewer_UnFX", "summary": "### Summary:\n\nThis paper introduces a novel data poisoning attack against adversarial training called *stability attacks*.\nThe goal is to temper the training data such that the robust performance of adversarial training over this manipulated dataset is degraded.\nTo construct this attack, a *hypocritical perturbation* is built: unlike *adversarial perturbations*, the aim of *hypocritical perturbations* is to reinforce the non-robust features in the training data.\nThese perturbations can be generated by negating adversarial example generation objectives.\n\n**Motivation:**\nThe paper motivates stability attacks from the perspective of robust vs. non-robust features.\nSpecifically, a simple binary classification task over a mixture of Gaussians is considered.\nStatistical analysis on this task shows that adversarial training over *hypocritically perturbed data* is destructive to adversarial robustness.\nMoreover, it is shown that a larger perturbation magnitude is needed to guard adversarial training against stability attacks.\n\n**Implementation:**\nThe effectiveness of stability attacks against adversarial training is demonstrated through extensive experimental results.", "questions": "### Questions:\nAs mentioned above, there are a few questions:\n\n1. How the *hypocritical perturbation* objective is derived? Are there any other objectives that can be used?\n2. What is the relationship between this work and prior work on the trade-off between clean and adversarial accuracy?\n3. While increasing the perturbation bound would help combat stability attacks, how can one maintain the clean accuracy as the current solution jeopardizes this?", "limitations": "A potential discussion on the real-world negative impacts of the current work is missing. This reviewer would encourage the authors to discuss this matter explicitly.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n- The paper is clear, and it guides the reader skillfully.\n\n- The paper is well-motivated.\nThe statistical analysis of the binary classification task is thorough, and the implications of the theoretical results are discussed comprehensively.\n\n- This paper sheds light on the implications of robust vs. non-robust features from a novel perspective and utilizes these studies to introduce a new threat against adversarial training.\n\n- The experimental settings are discussed in detail, and the effects of different hyper-parameters and architectures on the performance are investigated.\n\n### Weaknesses:\n- While the theoretical justifications of *hypocritical perturbations* on the binary classification task are discussed, the relationship of these results with the attack generation process (Eq. (10)) is obscure.\nAlthough the given example for the logistic loss and the binary classification task is appreciated, the origins of the objective function in Eq. (10) need a better justification.\n\n-  Furthermore, a thorough discussion on the relationship of this work with existing works on the trade-off between the clean and robust accuracy of neural networks with adversarial training seems missing.\nAs the experimental results suggest, the implications align with the observations of Tsipras et al. [63] on the trade-off between the clean and robust accuracy (e.g., see Table 4).\nFrom this perspective, it seems like stability attacks are somehow just exploiting this trade-off to pose their threat on adversarial training.\nThus, a comprehensive discussion on the differences between this work and prior work in this area is required. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SbwxwXDdHi", "reviewer": "Reviewer_bFP2", "summary": "This paper propose a new threat model called stability attack. The goal of stability attack is to hinder model from being robust to adversarial attacks. The author  proposes hypocritical perturbation as a method for stability attack and shows that hypocritical perturbation is harmful in terms of adversarial robustness in a simple gaussian mixture setting. Finally, the author shows that 2 adversarial training is enough for protecting stability attack.", "questions": "I have no question.", "limitations": "- As mentioned in the manuscript, the standard accuracy of a robustly leanred prediction model based on training data poisoned by the stability attack is  better than the standard accuracy of a robustly trained model based on training data poisoned by other methods. In this sense, the stability attack is less serious than other poisoned methods which degrade the standard accuracy as well as the robust accuracy simultaeously.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nPros: \n- As far as I know, this is the first work that studies the robustness of adversarial training against train data poisoning called stability attack.\n- As [1] noted, hypocritical perturbation is known to be a weak attack in terms of the standard accuracy, but this paper illustrates that it can be a strong attack in terms of the robust accuracy.\n- The intuition of stability attack for vulnerability of adversarial attack is well demonstrated.\nCons:\n- It seems that hypocritical perturbation is main content, but is not well described. Especially, the comparison of adversarial perturbation and hypocritical perturbation lacks.\n- It seems that Stability Attack in Table 2 and Adversarial Poisoning and Hyp. and Adv. in Table (3, 4) are identical, respectively. The consistency of naming the attacks is necessary.\n- The description of the poisoning methods in Table 2 is not given.\n- In Table 3,  Adv. and Hyp are only considered for the methods of data poisoning. It would be more appropriate to do evaluation against the various attacks as in Table 2.\n- The final message of the paper is that by using a larger perturbation size, the hypocritical perturbation can be defended. However, this message would be obvious and so the novelty would not be high. \n\n[1] Lue Tao, Lei Feng, Jinfeng Yi, Sheng-Jun Huang, and Songcan Chen. Better safe than sorry: Preventing delusive adversaries with adversarial training. In NeurIPS, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "K89wUs1jNCE", "reviewer": "Reviewer_dZbx", "summary": "This paper presents sability attack against the conventional adversarial training process, aiming to reduce the eventual robust accuracy of the resulting model. Specifically, the corresponding hypocritical perturbations are applied into training data as a training-time attack. Theoretical analysis is provided to support the idea of hypocritical perturbations. Experimental results on commonly used classification datasets like CIFAR-10 demonstrate the effeciveness of the proposed method.", "questions": "My questions are summarized in the weaknesses.", "limitations": "Limitations are discussed as pointed out in the checklist.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1. This paper is technically sound and clear around the theoretical analysis. Experimental results are significant, which well support the theory.\n\n2. The writing quality of the paper is good overall. Specifically, the background of problem is smoothly introduced.\n\n3. The proposed method is sufficiently evaluated. To be specific, key attacks like FGSM, PGD, CW, and AutoAttack are present for robustness evaluation. Meanwhile, the experiments cover four datasets, namely CIFAR-10/100, SVHN, and Tiny-ImageNet, which are sufficient to demonstrate the effectiveness for the proposed method.\n\n4. The proposed method successfully compromised adversarial training methods. However, countermeasure (adaptive defense) is also proposed and evaluated.\n\n# Weaknesses\n\n1. Overall this is a good paper and I did not find many problems.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept