Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training
Authors: Lue Tao
Lei Feng
Jinfeng Yi
Sheng-Jun Huang
Songcan Chen
Venue: NeurIPS 2021 Poster (2021)
Keywords: adversarial examples
adversarial training
data poisoning
delusive attacks
Wasserstein distance

Abstract: Delusive attacks aim to substantially deteriorate the test accuracy of the learning model by slightly perturbing the features of correctly labeled training examples. By formalizing this malicious attack as finding the worst-case training data within a specific $\infty$-Wasserstein ball, we show that minimizing adversarial risk on the perturbed data is equivalent to optimizing an upper bound of natural risk on the original data. This implies that adversarial training can serve as a principled defense against delusive attacks. Thus, the test accuracy decreased by delusive attacks can be largely recovered by adversarial training. To further understand the internal mechanism of the defense, we disclose that adversarial training can resist the delusive perturbations by preventing the learner from overly relying on non-robust features in a natural setting. Finally, we complement our theoretical findings with a set of experiments on popular benchmark datasets, which show that the defense withstands six different practical attacks. Both theoretical and empirical results vote for adversarial training when confronted with delusive adversaries.

Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training
Lue Tao
Lei Feng"
Jinfeng Yi'
Sheng-Jun Huang
Songean Chen 12f
'College of Computer Science and Technology, Nanjing University of Acronauties and Astronautics MIIT Key Laboratory of Pattemn Analysis and Machine Intelligence CCollege of Computer Science, Chongqing University IJD AI Research
Abstract
Delusive attacks aim to substantially deteriorate the test accuracy of the learning model by slightly perturbing the features of correctly labeled training examples. By formalizing this malicious attack as finding the worst-case training data within specific -o-aasserttein ball, we show that minimizing adversarial risk the perturbed data equivalent to optimizing an upper bound of natural risk on the original data. This implies that adversarial training can serve as principled defense against delusive attacks. Thus, the test accuracy decreased by delusive attacks can be largely recovered by adversarial training. To further understand the internal mechanism of the defense, we disclose that adversarial training can resist the delusive perturbations by preventing the learner from overly relying on non-robust features in natural setting. Finally, we complement our theoretical findings with a set of experiments on popular benchmark datasets, which show that the defense withstands six different practical attacks. Both theoretical and empirical results vote for adversarial training when confronted with delusive adversaries.
Introduction
Although machine learning (ML) models have achieved superior performance on many challenging tasks, their performance can be largely deteriorated when the training and test distributions different. For instance, standard models are prone to make mistakes on the adversarial examples that considered as worst-case data at test time [10, 113]. Compared with that, easily overlooked threat is the malicious perturbations at training time, i.e., the delusive attacks [82] that aim to maximize test error by slightly perturbing the correctly labeled training examples [6. 7].
threatening and
In the era of big data, many practitioners collect training data from untrusted sources where delusive adversaries may exist. In particular, many companies are scraping large datasets from unknown users or public websites for commercial use. For example, Kaspersky Lab, a leading antivirus company, has been accused of poisoning competing products [7]. Although they denied any wrongdoings and clarified the false rumors, one can still imagine the disastrous consequences if that really happens in the security-critical applications. Furthermore, recent survey of 28 organizations found that these industry practitioners are obviously more afraid of data poisoning than other threats from adversarial MI. [64]. In nutshell, delusive attacks has become realistic and horrible threat to practitioners.
Recently, Feng et al. [32] showed for the first time that delusive attacks are feasible for deep networks, by proposing "training-time adversarial data" that can significantly deteriorate model performance on clean test data. However, how to design learning algorithms that are robust to delusive attacks still remains open question due to several crucial challenges [82, 32]. First of all, delusive attacks
Corresponding author; Songcan Chen <s.chen muaa.edu.en>.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

100
Standard training
Adverrattal traiing
bind
horse
ship
80
Text Arcureyy
cXr
brnd
CX
Dps
20
Training using 1>
Training using Dys
Figure 1: An illustration of delusive attacks and adversarial training. Left: Random samples from the CIFAR-IO [63] training set: the original training set D; the perturbed training Dps, generated using the P5 attack described in Section 4. Right: Natural accuracy evaluated on clean test set for models trained with: standard training on D; ii) adversarial training on D; iii) standard training on Dps; iv) adversarial training on Dp5. While standard training on Dp5 incurs poor generalization performance on D, adversarial training can help lot. Details are deferred to Section 5.1.
cannot be avoided by standard data cleaning [59], since they does not require mislabeling, and the perturbed examples will maintain their malice even when they are correctly labeled by experts. In addition, even if the perturbed examples could be distinguished by some detection techniques, it is wasteful to filter out these correctly labeled examples, considering that deep models are data-hungry. In extreme case where all examples in the training set are perturbed by delusive adversary, there will leave no training examples after the filtering stage, thus the learning process is still obstructed. Given these challenges, we aim examine the following question in this study: Is it possible to defend against delusive attacks without abandoning the perturbed examples?
In this work, we provide affirmative answer to this question. We first formulate the task of delusive attacks as finding the worst-case data at training time within specific oo- oo-Wasserttein ball that prevents label changes (Section 2). By doing so, we find that minimizing the adversarial risk on the perturbed data is equivalent to optimizing an upper bound of natural risk on the original data (Section 3.1). This implies that adversarial training [44, 74] on the perturbed training examples can maximize the natural accuracy on the clean examples. Further, we disclose that adversarial training resist the delusive perturbations by preventing the learner from overly relying the non robsro features (that are predictive, yet brittle or incomprehensible to humans) in a simple and natural setting. Specifically, two opposite perturbation directions studied, and adversarial training helps in both cases with different mechanisms (Section 3.2). All these evidences suggest that adversarial training a promising solution to defend against delusive attacks.
Importantly, our findings widen the scope of application of adversarial training, which was only considered as principled defense method against test-time adversarial examples [74, 21J. Note that adversarial training usually leads to a drop in natural accuracy [119]. This makes less practical in many real-world applications where test-time attacks are and high accuracy
clean test data is required [65]. However, this study shows that adversarial training can also defend against a more threatening and invisible threat called delusive adversaries (see Figure 1 for illustration). We believe that adversarial training will be more widely used in practical applications in the future.
Finally, we present five practical attacks to empirically evaluate the proposed defense (Section 4). Extensive experiments on various datasets (CIFAR-10, 10, SVHN, and a subset of ImageNet) and tasks (supervised learning, self-supervised learning, and overcoming simplicity bias) demonstrate the effectiveness and versatility of adversarial training, which significantly mitigates the destructiveness of various delusive attacks (Section 5). Our main contributions summarized as follows:
Formulation of delusive attacks. We provide the first attempt to formulate delusive attacks using the oo-aasserttein distance. This formulation is novel and general, and can cover the formulation the attack proposed by Feng et al. [32].
The principled defense. Equipped with the novel characterization of delusive attacks, we are able to show that, for the first time, adversarial training can serve principled defense against delusive attacks with theoretical guarantee (Theorem 1).
Internal Mechanisms. We further disclose the internal mechanisms of the defense in popular -Gaussin setting (Theorem 2 and Theorem 3).
Empirical evidences. We complement our theoretical findings with extensive experiments across wide range of datasets and tasks.

---

2 Preliminaries
In this section, we introduce some notations and the main ideas we build upon: natural risk, adversarial risk, Wasserstein distance, and delusive attacks.
Notation. Consider classification task with data (æz, y) X' x y from underlying distribution D. We seek to learn model f : + ) by minimizing a loss function L (z)) y). Let A X' x X - R be some distance metric. Let ())
fa' X A(x, x') S c} be the ball around with radius e. When A is free of context, we simply write ((x, A) = B(x,,), e). Throughout the paper, the adversary is allowed perturb only the inputs, not the labels. Thus, similar to Sinha et al. [109], we define the cost function e: 2 x 2 + R U (2o) by c(2, = A (ar, + 0c 1 * y'), where (2r y) and Z is the set of possible values for (ac, y)- Denote by P22) the set of all probability
measures Z. For any l E RRd and positive definite matrix E Rdxd denote by N (H,, Σ) the d-dimensional Gaussian distribution with mean vector μ and covariance matrix Σ.
Natural risk. Standard training (ST) aims to minimize the natural risk, which is defined as
Rna , D) = (())) [((f(z). y)]
The term "natural accuracy" refers to the accuracy of a model evaluated on the unperturbed data. Adversarial risk. The goal of adversarial training (AT) is to minimize the adversarial risk defined as
Radv (J, D) = E(a.))>D | (a('),
which is a robust optimization problem that considers the worst-case performance under pointwise perturbations within an e-ball [74]. The main assumption here is that the inputs satisfying A (a., (a) S preserve the label y of the original input æ.
Wasserstein distance. Wasserstein distance is a distance function defined between two probability distributions, which represents the cost of an optimal mass transportation plan. Given two data distributions D and D', the p-th Wasserstein distance, for any p 2 defined as:
W WD(,,
(inf (inf,empp, Saxz c(z, x' 2'),
(3)
where [ID,, D') is the collection of all probability measures 2 x 2 with D and D' being the marginals of the first and second factor, respectively. The xo- Wasserstein distance is defined as the limit of p-th Wasserstein distance, i.e., Wx (D, D') -immp=o (D, D'). The p-th Wasserstein ball with respect to DD and radius 2 0 is defined as: (D,e) (D' E P(Z): W, (D, S c).
Delusive adversary. The attacker is capable of manipulating the training data, long the training data is correctly labeled, to prevent the defender from learning an accurate classifier [82]. Following Feng et al. [32], the game between the attacker and the defender proceeds as follows:
n data points are drawn from to produce a clean training dataset Dn,.
The attacker perturbs some inputs in D., by adding small perturbations to produce x' such that A(x, æ²) S where e is small constant that represents the attacker's s budget. The (partially) perturbed inputs and their original labels constitute the perturbed dataset D,,-
The defender trains on the perturbed dataset D,, to produce a model, and incurs natural risk.
The attacker's goal is to maximize the natural risk while the defender's 's task is to minimize it. We then formulate the attacker's goal as the following bi-level optimization problem:
max
Reae Bn
s.t. f arg min R. D).
(4)
BeBw.. (D.x)
In other words, Eq. (4) is seeking the training data bounded by the oc- Wasserstein ball with radius e, so that the model trained on it has the worst performance on the original distribution.
Remark 1. It is worth noting that using the oo- Wasserstein distance constrain delusive attacks possesses several advantages. Firstly. the cost function used in Eq. (3) prevents label changes after perturbations since we only consider clean-label attacks. Secondly, our formulation does not restrict the choice of the distance metric A of the input space, thus our theoretical analysis works with any metric, including the loo threat model considered in Feng et al. [32]. Finally, the oc- Wasserstein ball is more aligned with adversarial risk than other uncertainty sets [110, 147].
3

---

Remark 2. Our formulation assumes an underlying distribution that represents the perturbed dataset. This assumption has been widely adopted by existing works [111, 118, 145]. The rationale behind the assumption is that generally, the defender treats the training dataset as an empirical distribution and trains the model on randomly shuffled examples (e.g., training deep networks via stochastic gradient descent). It is also easy to see that our formulation covers the formulation of Feng et al. [32]. On the other hand, this assumption has its limitations. For example, the defender treats the training examples sequential data [24], the attacker may utilize the dependence in the sequence to construct perturbations. This situation is beyond the scope of this work, and we leave it as future work.
3 Adversarial Training Beats Delusive Adversaries
In this section, we first justify the rationality of adversarial training as principled defense method against delusive attacks in the general case for any data distribution. Further, to understand the internal mechanism of the defense, we explicitly explore the space that delusive attacks can exploit in simple and natural setting. This indicates that adversarial training resists the delusive perturbations by preventing the learner from overly relying on the non-robust features.
3.1 Adversarial Risk Bounds Natural Risk
Intuitively, the original training data is close to the data perturbed by delusive attacks, so it should be found in the vicinity of the perturbed data. Thus, training models around the perturbed data can translate to a good generalization on the original data. We make the intuition formal in the following theorem, which indicates that adversarial training on the perturbed data is actually minimizing an upper bound of natural risk on the original data.
Theorem 1. Given a classifier f : X - for any data distribution 'D and any perturbed distribution D such that D E Bw (D,e), we have
R, S D'EBw, (D.r) Rnae(f,D') = Radv (..
max.
The proof is provided in Appendix C.I. Theorem suggests that adversarial training is principled defense method against delusive attacks. Therefore, when our training data is collected from untrusted sources where delusive adversaries may exist, adversarial training can be applied to minimize the desired natural risk. Besides, Theorem 1 also highlights the importance of the budget e. On the one hand, if the defender is overly pessimistic (i.c., the defender's budget is larger than the attacker's 's budget), the tightness of the upper bound cannot be guaranteed. On the other hand, if the defender is overly optimistic (i.e., the defender's budget is relatively small risk on the original data cannot be upper bounded anymore by the adversarial risk. Our experiments in Section 5.1 cover these cases when the attacker's budget is not specified.
equals to zero), the natural
3.2 Internal Mechanism of the Defense
To further understand the internal mechanism of the defense, in this subsection, we consider simple and natural setting that allows us explicitly manipulate the non-robust features. It turns out that, similar to the situation in adversarial examples [119, 56], the model's reliance on non-robust features also allows delusive adversaries to take advantage of it, and adversarial training can resist delusive perturbations by preventing the learner from overly relying on the non-robust features.
As Ilyas al. [56] has clarified that both robust and non-robust features in data constitute useful signals for standard classification, we are motivated to consider the following binary classification problem mixture of two Gaussian distributions D:
x x~~ [-1,+1},
14, o2
(5)
where (1. (1. E R is the mean vector which consists of 1 robust feature with center I and non-robust features with corresponding centers 7,, similar to the settings Tsipras al. [119]. Typically, there are far more non-robust features than robust features (i.e., d > 1). To restrict the capability delusive attacks, here we chose the metric function A(r, x²) = ||ar - x a''looo We assume that the attacker's budget satisfies 2 2n and 1 S. 1/3, that the attacker: i) can shift

---

each non-robust feature towards becoming anti-correlated with the correct label; it) cannot shift each non-robust feature to be strongly correlated with the correct label (as strong as the robust feature).
Delusive attack is easy. For the sake of illustration, here we choose e = 2n and consider two opposite perturbation directions. One direction is that all non-robust features shift towards -y, the other is to shift towards y. These settings are chosen for mathematical convenience. The following analysis can be easily adapted any E 2nj and any combinations of the two directions on non-robust features.
Note that the Bayes optimal classifier (i.e., minimization of the natural risk with 0-1 loss) for the distribution D is fa( sign(μ' (1 [x), which relies on both robust feature and non-robust features. As a result, an -ouunded delusive adversary that is only allowed to perturb each non-robust feature by moderate distribution D can be perturbed to the delusive distribution Dj:
take advantage of the space of non-robust features. Formally, the original
y U-a-T {-1,+1}, x ~ N fI,o²1);
where f4 (1, -1,-7.....,- is the shifted mean vector. After perturbation, every non-robust feature correlated with yy thus the Bayes optimal classifier for D, would yield extremely poor performance on 'D, for d large enough. Another interesting perturbation direction is to strengthen the magnitude of non-robust features. This leads to the delusive distribution D2:
y U-g-T {-1,+1},
N P2,0²²1.
where N2g (1, 31, ........................... 37) is the shifted mean vector. Then, the Bayes optimal classifier for D2 will overly rely on the non-robust features, thus likewise yielding poor performance on D.
The above two attacks are legal because the delusive distributions are close enough to the original distribution, that is, W (D, S and Wo (D, D2) Meanwhile, the attacks are also harmful. The following theorem directly compares the destructiveness of the attacks.
Theorem 2. Let fD. J., and JD. be the Bayes optimal classifiers for the Sixture-Gaussiar distri- butions D. Dr, and D2, defined in Eqs. (5), (6), and (7), respectively. For any 1) > O, we have
Rna D) Rn D) < Rna
The proof is provided in Appendix C.2. Theorem indicates that both attacks will increase the natural risk the Bayes optimal classifier. Moreover, D2 is more harmful because it always incurs higher natural risk than Dy. The destructiveness depends on the dimension of non-robust features. For intuitive understanding, we plot the natural accuracy of the classifiers as a function of in Figure We observe that, as the number of non-robust features increases, the natural accuracy of the standard model fs continues decline, while the natural accuracy of fö, first decreases and then increases.
Ad veresaaial training matters. Adversarial train- ing with proper e will mitigate the reliance on non- robust features. For D, the internal mechanism is
Test Accuracy on D (%)
similar the case in Tsipras et al. [119], while for
D2 the mechanism is different, and there was no
Standard training on
Standard training on
such analysis before. Specifically, the optimal lin-
Standard training
ear Loo robust classifier (i.e., minimization of the
Adversorial training on 9.
Adversarial training
adversarial risk with 0-1 loss) for D will rely solely on the robust feature. In contrast, the optimal ro- bust classifier for D2 will rely on both robust and non-robust features, but the excessive reliance on non-robust features is mitigated. Hence, adversarial training matters in both cases and achieves better nat- ural accuracy when compared with standard training. We make this formal in the following theorem.
2000
6000
5000
10200
# Non-Robust Features
Figure 2: els trained on the tions function of the number of non-robust features. As a concrete example, here we set a= 1, 1) = 0.01 and vary d.
The natural accuracy of five mod-
distribu-
Theorem 3. Let Sob rob and rob be the optimal linear too robust classifiers for the delusive distribations D, and Dyy, defined in Eqs. (6) and (7), respectively. For any 0 < 1) < 1/3, we have Rnalffo D) Rnat Cnat(f rob, and Roalf D) Reat ( (o D).
5

---

The proof's is provided in Appendix C.3. Theorem 3 indicates that robust models achieve lower natural risk than standard models under delusive attacks. This also reflected in Figure 2: After adversarial training on D1 natural accuracy is largely recovered and keeps unchanged as increases. While on D2, natural accuracy can be recovered better and keeps increasing as increases. Beyond the theoretical analyses for these simple cases, also observe that the phenomena in Theorem 2 and Theorem generalize well to our empirical experiments on real-world datasets in Section 5.1.
Practical Attacks for Testing Defense
Here briefly describe five heuristic attacks. A detailed description is deferred Appendix D. The five attacks along with the L2C attack proposed by Feng et al. [32] will be used in next section for validating the destructiveness of delusive attacks and thus the necessity of adversarial training.
In practice, we focus on the empirical distribution Dn, over n data points drawn from D. Inspired by "non-robust features suffice for classification" [56], propose to construct delusive perturbations by injecting non-robust features correlated consistently with a specific label. Given a standard model fo trained on D,, the attacks perturb each input (with label y) in D, follows:
P1: Adversarial perturbations. It adds small adversarial perturbation to ensure that it is misclassified as target by minimizing (ff(x'), such that x' E (xx c), where t is chosen deterministially based on y.
P2: Hypocritical perturbations. It adds small helpful perturbation to by minimizing & (fo(z'), )) y) such that x' E 18 c), so that the standard model could easily produce a correct prediction.
P3: Universal adversarial perturbations. This attack is variant of P1. It adds the class- specific universal adversarial perturbation & æ. All inputs with the same label y are perturbed with the same perturbation &,, where t is chosen deterministially based on
P4: Universal hypocritical perturbations. This attack is a variant of P2. It adds the class-specifhs universal helpful perturbation Ey æ. All inputs with the same label y perturbed with the same perturbation Ey-
P5: Universal random perturbations. This attack injects class-specific random perturba- tion ry to each æ. All inputs with the label is perturbed with the same perturbation Ty. Despite the simplicity of this attack, we find that it are surprisingly effective in some cases.
Figure visualizes the universal perturbations for different datasets and threat models. The perturbed inputs and their original labels constitute the perturbed datasets Dp1 Dps.
(a) CIFAR-10 10 (l2)
(b) SVHN (E2)
(c) CIFAR-10 (Ex0)
(d) ImageNet (E:0) (c) SSL (l2)
Figure Universal perturbations for the P3 and attacks across different datasets and threat models. Perturbations are rescaled to lie in the [0, 1] range for display. The resulting inputs are nearly indistinguishable from the originals a human observer (see Appendix Figures 10, 11, and 12).
Experiments
In order demonstrate the effectiveness and versatility of the proposed defense, conduct ex- periments on CIFAR-I0 [63], SVHN [81]. subset of ImageNet [95], and MNIST-CIFAR [104] datasets. More details experimental setup provided in Appendix A. Our code is available at https:// /ig thub. com/ TLMi chwel Sawaaae/Deelliieeeeeeeeaaaeeeeee Adversary.
Firstly, we perform a set of experiments on supervised learning to provide comprehensive under- standing of delusive attacks (Section 5).. Secondly, we demonstrate that the delusive attacks can also obstruct rotation-based self-supervised learning (SSL) [41] and adversarial training also helps lot in this case (Section 5.2). Finally, we show that adversarial training a promising method to overcome the simplicity bias the MNIST-CIFAR dataset [104] if the e-ball is chosen properly (Section 5.3).

---

5.1 Understanding Delusive Attacks
Here, we investigate delusive attacks from six different perspectives: i) baseline results on CIFAR-10 10. fi) transferability of delusive perturbations to various architectures, iii) performance changes of various defender's budgets, iv) a simple countermeasure, v) comparison with Feng et al. [32], and vi) performance of other adversarial training variants.
Baseline results. We consider the typical &g threat model with e = 0.5 for CIFAR-10 by follow- ing [56]. We use the attacks described in Section 4 to generate the delusive perturbations. To execute the attacks P1 ~ P4, we pre-train a VGG-16 [108] as the standard model fp using standard training on the original training set. We then perform standard training and adversarial training on the delusive datasets Dp1 Dps- Standard data augmentation (i.e., cropping, mirroring) is adopted. The natural accuracy of the models is evaluated on the clean test set of CIFAR-10.
Results are summarized in Figure 4. We observe that the natural accuracy of the standard models dramatically decreases when training the delu- sive datasets, especially on Dp3, Dp4 and Dps-
Standard training I Adversarial training
Natural Aczuracy (%)
33.5
85.0
60
t
The most striking observation to emerge from the results is the effectiveness of the P5 attack. It seems that the naturally trained model seems to rely exclusively on the small random patterns in this case, even though there are still abundant nat- ural features in Dps. Such behaviors resemble the conjunction learner' studied in the pioneering work [82], where they showed that such learner is highly vulnerable to delusive attacks. Also, we point out that such behaviors could be attributed to the gradient starvation [91] and simplicity bias [104] phenomena of neural networks. These recent studies both show that neural networks trained by SGD preferentially capture a subset of features relevant for the task, despite the presence of other predictive features that fail to be discovered [50].
36.7
19.4
Tpp Ppp
Do1
Do4
Dee
Figure 4: Natural accuracy CIFAR-IO using VGG- 16 under lg threat model. The horizontal line indicates the natural accuracy of a standard model trained on the clean training set.
Anyway, our results demonstrate that adversarial training can successfully eliminate the delusive features within the e-ball. As shown in Figure 4, natural accuracy can be significantly improved by adversarial training in all cases. Besides, we observe that P1 is more destructive than P2, which is consistent with our theoretical analysis of the hypothetical settings in Section 3.2.
Evaluation of transferability. A more realistic setting to attack different classifiers using the same delusive perturbations. We consider various architectures including VGG-19, ResNet- 18, ResNet-50, and 1enseNet-121 as victim classifiers. The delusive datasets are the same as in the baseline experiments. Results are deferred to Figure 8 in Appendix B. We observe that the attacks have good transferabilityy across the architectures, and again, adversarial training can substantially improve natural accuracy in all cases. One exception is that the P5 attack is invalid for DenseNet- 121. possible explanation for this might be that the simplicity bias of DenseNet-121 on random patterns is minor. This means that different architectures may have distinct simplicity biases. Due to space constraints, a detailed investigation is out of the scope of this work.
What if the threat model is not specified? Our theoretical analysis in Section 3. highlights the importance of choosing a proper budget for AT. Here, we try to explore this situation where the threat model is not specified by varying the defender's 's budget. Results on CIFAR- 10 using ResNet-18 under lg threat model are summarized in Figure slightly hurt performance, while budget that is too small is not enough to mitigate the attacks. Empirically, the optimal budget for P3 and is about 0.4 and for P1 and P2 it about 0.25. P5 the easiest to defend-An AT with small budget (about 0.1) can significantly mitigate its effect.
We observe that budget that too large may
simple countermeasure In addition to adversarial training, simple countermeasure adding clean data to the training set. This will neutralize the perturbed data and make it closer to the original distribution. We explore this countermeasure on SVHN since extensive extra training examples are available in that dataset. Results are summarized in Figure 6. We observe that the performance of standard training is improved with the increase of the number of additional clean examples, and the
'The conjunction learner first identifies a subset of features that appears in every examples of a class, then classifies an example as the class if and only if it contains such features [82].
7

---

3
Tranin
57
6.
Defender's Sudger
#AAditioral Exampkc
Addditooal Clean Exumpios
Figure 5: Natural accuracy as a function of the defender's bud- get on CIFAR-10.
Figure Natural accuracy (left) and adversarial accuracy (right) as a function of the number of additional clean examples on SVHN using ResNet- 18 under & threat model.
100
Standard training
Adversarial training
Standard training
Adversarial training
Natural Accuraty (80)
36.0
80
Natural Accuracy
40
Bix
D0
Dos
Bes
By
Dux
10
18z
By
8y
(a) CIFAR-10
(b) Two-class ImageNet
Figure 7:
Natural accuracy on CIFAR-10 using VGG-11 (left) and two-class ImageNet using
R8 (right) under fx.. threat model. The horizontal orange line indicates the natural accuracy of a standard model trained on the clean training set.
performance of adversarial training can also be improved with more data. Overall, it is recommend that combining this simple countermeasure with adversarial training to further improve natural accuracy. Besides the focus natural accuracy in this work, another interesting measure is the model accuracy on adversarial examples. It turns out that adversarial accuracy of the models can also be improved with more data. We also observe that different delusive attacks have different effects on the adversarial accuracy. A further study with more focus on adversarial accuracy is therefore suggested.
Comparison with L2C. We compare the heuristic attacks with the L2C attack proposed by Feng et al. [32] and, show that adversarial training can mitigate all these attacks. Following their settings on CIFAR- 10 and two-class ImageNet, the fxc -nom bounded threat models with e = 0.032 and = 0.1 are consid- ered. The victim classifier is VGG-11 and R8 for CIFAR-10 10 and the two-class ImageNet, respec- tively. Table reports the time cost for executing six attack methods on CIFAR-10. We find that the heuris- tic attacks are significantly faster than L2C, since the bi-level optimization process in L2C is extremely time- consuming. Figure shows the performance of stan- dard training and adversarial training on delusive datasets. The results indicate that most of the heuristic attacks are comparable with L2C, and AT can improve natural accuracy in all cases.
Table 1: Comparison of time cost. The L2C attack needs to train an autoencoder to gen- erate perturbations. The P1 ~ P4 attacks need to train standard classifier to generate perturbations, and P5 needs not.
Time Cost (min) Generating
Method Training
Total
L2C
7411.5
0.4
7411.9
P1 /P2
25.9
12.6
38.5
P3 PP4
25.9
4.6
30.5
P5
0.0
0.1
0.1
Performance of adversarial training variants. It is noteworthy that AT variants are also effective in our setting, since they aim to tackle the adversarial risk. To support this, consider instance- dependent-based variants (such as MART [125], GAIRAT [141], and MAIL. [123]) and curriculum- based variants (such as CAT [13]. DAT [124], FAT [140]) Specifically, we chose experiment with the currently most effective variants among them (i.c.. GAIRAT and FAT, according to the latest leaderboard at RobustBench [2]]). Additionally, we consider random noise training (denoted as RandNoise) using the uniform noise within the c-ball for comparison. We also report the results of standard training (denoted as ST) and the conventional PGD-based AT [74] (denoted as PGD-AT) for reference. The results are summarized in Table 2. We observe that the performance of random noise training is marginal. In contrast, all AT methods show significant improvements, thanks to the theoretical analysis provided by Theorem 1. Besides, we observe that FAT achieves overall better results than other AT variants. This may due to the tight upper bound of the adversarial risk pursued by FAT. In summary, these results successfully validate the effectiivnees of AT variants.
8

---

Table 2: Natural accuracy CIFAR-10 1- using ResNet-18 under &oo threat model with 8/255.
The column of "Clean" denotes the natural accuracy of the models trained on the clean training set.
Method
Clean
L2C
P2
P3
P4
P5
94.62
15.76
15.70
61.35
9.40
13.58
10.12
RandNoiie
94.26
17.10
17.32
63.36
10.52
14.37
27.56
PGD-AT
85.18
82.84
84.18
86.74
86.37
83.18
84.57
GAIRAT
81.90
79.96
79.61
82.68
82.05
82.81
82.28
FAT
87.43
85.51
86.05
88.98
84.39
84.22
87.78
Table 3: Adversarial training on MNIST-CIFAR:
The table reports test accuracy on the MNIST- CIFAR test set and the MNIST-randomized test set. Our customized AT successfully overcomes SB, while others not. The MNIST-randomized accuracy indicates that our adversarially trained models achieve nontrivial performance when there are only CIFAR features exist in the inputs.
Test Accuracy on MNIST-CIFAR
SUIST-Randomized Accuracy AT [104]
Model
ST
AT [104]
AT (ours)
ST
AT (ours)
VGG-16 ResNet-50
99.9
100.0
91.3
49.1
51.6
91.2
100.0
99.9
89.7
48.9
49.2
88.6
DenseNet-121
100.0
100.0
91.5
48.8
49.2
90.8
5.2 Evaluation on Rotation-based Self-supervised Learning
To further show the versatility of the attacks and defense, we conduct experiments rotation-based self-supervised learning (SSI.) [41], a process that learns representations by predicting rotation angles (0", 909, (09,90", 180", 270%). SSL methods seem to be inherently resist to the poisoning attacks that require mislabeling, since they do not use human-annotann -anotated labels to learn representations. Here, we examine whether SSL can resist the delusive attacks. We use delusive attacks to perturb the training data for the pretext task. To evaluate the quality of the learned representations, the downstream task is trained on the clean data using logistic regression. Results deferred to Figure 9 in Appendix B.
We observe that the learning of the pretext task can be largely hijacked by the attacks. Thus the learned representations are poor for the downstream task. Again, adversarial training can significantly improve natural accuracy in cases. interesting observation from Figure 9b) that the quality of the adversarially learned representations is slightly better than that of standard models trained on the original training set. This is consistent with recent hypotheses stating that robust models may transfer better [98, 121, 69, 22, 2]. These results show the possibility of delusive attacks and defenses for SSL, and suggest that studying the robustness of other SSL methods [58, 17] against data poisoning is promising direction for future research.
5.3 Overcoming Simplicity Bias
A recent work by Shah et al. [104] proposed the MNIST-CIFAR dataset to demonstrate the simplicity bias (SB) of using standard training to learn neural networks. Specifically, the MNIST-CIFAR images are vertically concatenations of the "simple" MNIST images 2m and the more complex CIFAR- 10 images x. (i.e., x = [mm (m.). They found that standard models trained on MNIST-CIFAR will exclusively rely on the MNIST features and remain invariant to the CIFAR features. Thus randomizing the MNIST features drops the model accuracy to random guessing.
From the perspective of delusive adversaries, we can regard the MNIST-CIFAR dataset as delusive version of the original CIFAR dataset. Thus, AT should mitigate the delusive perturbations, as Theorem pointed out. However, Shah et al. [104] tried AT on MNIST-CIFAR yet failed. Contrary their results, here we demonstrate that AT is actually workable. The key factor the choice of the threat model. They failed because they chose an improper ball B(x, c) = (x' E X' || x'I] < 0.3), while we set B(x, e) = (æ' E X ||.,., x'm lolo + xe | - o' < 1}. Our choice forces the space of MNIST features to be non-robust region during AT, and prohibits the CIFAR features from being perturbed. Results are summarized in Table 3. We observe that our choice leads to models that do not rely the simple MNIST features, thus AT can eliminate the simplicity bias.

---

6 Related Work
Data poisoning. The main focus of this paper is the threat of delusive attacks [82], which belongs to data poisoning attacks [6, 7, 43]. Generally speaking, data poisoning attacks manipulate the training data to cause a model to fail during inference. Both integrity and availability attacks were extensively studied for classical models [5, 42, 82, 80, 8, 9, 76, 132, 143]. For neural networks, most of the existing works focused on targeted misclassification [61. 103, 145, 1, 55, 39, 105, 20, 92] and backdoor attacks [47, 18, 71, 120, 72, 83, 86, 84]. while there was little work on availability attacks [102. 78, 32. 106]. Recently, Feng et al. [32] showed that availability attacks are feasible for deep networks. This paper follows their setting where the perturbed training data is correctly labeled. We further point out that their studied threat is exactly the delusive adversary (a.k.a. clean-label availability attacks), which was previously considered for classical models [82]. Besides, other novel directions of data poisoning are rapidly evolving such as semi-supervised learning [70, 35, 14], contrastive learning [15, 96], domain adaptation [75], and online learning [88]. etc.
Existing defenses. There were many defense strategies proposed for defending against targeted attacks and backdoor attacks, including detection-basee defenses [111, 118, 16, 23, 38, 90, 48, 28], randomized smoothing [94, 127], differential privacy [73, 51], robust training [12, 66, 67, 40], and model repairing [19, 68, 130J. while some of them may be overwhelmed by adaptive attacks [62, 122, 107]. Robust learnability under data poisoning attacks can be analyzed from theoretical aspects [11, 1 126, 36). Similarly, our proposed defense is principled and theoretically justified. More importantly, previous defenses mostly focus on defending against integrity attacks, while none of them are specially designed to resist delusive attacks. The work most similar to ours is that of Farokhi [31]. They only handle the linear regression model by relaxing distributionally robust optimization (DRO) as regularizer, while we can tackle delusive attacks for any classifier.
Adversarial training. Since the discovery of adversarial examples (a.k... evasion attacks at test time) in neural networks [10, 113], plenty of defense methods have been proposed to mitigate this vulnerability [44, 89, 3]. Among them, adversarial training is practically considered as a principled defense against test-time adversarial examples [74, 21] at the price of slightly worse natural accuracy [112, 119, 135] and moderate robust generalization [101, 93], and many variants were devoted to improving the performance [139, 140, 26, 131, 87, 37, 29, 114, 54, 45, 27]. Besides, it has been found that adversarial training may intensify backdoor attacks in experiments [128]. In contrast, both of our theoretical and empirical evidences suggest that adversarial training can mitigate delusive attacks. On the other hand, adversarial training also led to further benefits in robustness to noisy labels [146), distiiibuttoonnonnnnnnn generalization [133, 136, 60], transfer learning [98, 116, 121], domain adaption [4], novelty detection [46, 97], explainability [142, 85], and image synthesis [100].
Concurrent work. The threat of delusive attacks [82, 32] is attracting attention from the community. Several attack techniques are concurrently and independently proposed, including Unlearnable Exam- ples [53], Alignment [33], NTGA [137], Adversarial Shortcuts [30], and Adversarial Poisoning [34]. Contrary to them, we mainly focus on introducing a principled defense method (i.e., adversarial training), while by-products, five delusive attacks are presented and investigated in this paper. Meanwhile, Huang et al. [53] and Fowl et al. [34] also experiment with adversarial training, but only for their proposed delusive attacks. In contrast, this paper not only provides empirical evidence on the success of adversarial training against six different delusive attacks, but also offers theoretical justifications for the defense, which is of great significance to security. We believe that our findings will promote the use of adversarial training in practical applications in the future.
7 Conclusion and Future Work
In this paper, we suggest applying adversaria training in practical applications, rather than standard training whose performance risks being substantially deteriorated by delusive attacks. Both theoret- ical and empirical results vote for adversarial training when confronted with delusive adversaries. Nonetheless, some limitations remain and may lead to future directions: i) Our implementation of adversarial training adopts the popular PGD-AT framework [74, 871, which could be replaced with certified training methods [129, 138] for better robustness guarantee. ii) The success of our proposed defense relies on generalization, just like most MI. algorithms, so an analysis of robust generalization error bound for this case would be useful. iīi) Adversarial training may increase the disparity of accuracy between groups [134, 1 117], which could be mitigated by fair robust learning [134].
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China (Grant No. 62076124, 62076128) and the National Key R&D Program of China (2020AAA0107000)
References
[1] Hojjat Aghakhani, Dongyu Meng, Yu- Xu-Xiang Wang, Christopher Kruegel, and Giovanni Vigna. Bulls- eye polytope: A scalable clean-label poisoning attack with improved transferability. arXiv preprint arXiv:2005.00191 2020.
[2] Anonymous. Adversarially robust models may not transfer better: Sufficient conditions for domain transferabilityy from the view regularization. Submitted The Tenth ICLR, 2022. URL https:
/ //openreview net/foormmidd_iiiieeeee.ee Ltte__ixxH_RR-- FZ. under review.
[3] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018.
[4] Yang Bai, Xin Yan, Yong Jiang. Shu-Tao Xia, and Yisen Wang. Clustering effect adversarial robust models. In NeurlPS, 2021.
[5] Marco Barreno, Blaine Nelson, Russell Sears, Anthony D Joseph, and Doug Tygar. Can machine learning secure? Proceedings of the 2006 ACM Symposium on Information, computer and communications security, 2006.
[6] Marco Barreno, Blaine Nelson, Anthony D Joseph, and Doug Tygar. The security of machine learning. Machine Learning, 81(2)121-144, 2010.
[7] Batrista Biggio and Fabio Roli. Wild patterns: Ten years after the rise adversarial machine learning.
Pattern Recognition, 84:317-331, 2018.
[8] Battista Biggio, Blaine Nelson, and Pavel Laskov. Support vector machines under adversarial label noise. In ACMI, 2011.
[9] Battista Biggio, Blaine Nelson, and Pavel Laskov, Poisoning attacks against support vector machines. In ICML, 2012.
[10] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šindić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine lcarning at test time. In ECML-KDD, 2013.
[11] Avrim Blum, Steve Hanneke, Jian Qian, and Han Shao. Robust learning under clean-label attack. In COLT, 2021.
[12] Eitan Borgnia, Valeriia Cherepanova, Liam Fowl, Amin Ghiasi, Jonas Geiping, Micah Goldblum, Tom Goldstein, and Arjun Gupta. Strong data augmentation sanitizes poisoning and backdoor attacks without accuracy tradeoff. In ICASSP, 2021.
[13] Qi-Zhi Cai, Chang Liu, and Dawn Song. Curriculum adversarial training. In IJCAI, 2018.
[14] Nicholas Carlini. Poisoning the unlabeled dataset of semi-supervised learning. USENIX Security Symposium, 2021.
[15] Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. arXiv preprint arXiv:2106.0966 2021.
[16] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, lan Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint arXiv:1811.0372 2018
[17] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[18] Xinyun Chen, Chang Liu, Bo Li. Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv: 1712.0552, 2017.
[19] Xinyun Chen, Wenxiao Wang, Chris Bender, Yiming Ding. Ruoxi Jia, Bo Li, and Dawn Song. Refit: A unified watermark removal framework for deep learning systems with limited data. In ACM Asia Conference Computer and Communications Security, 2021.
11

---

[20] Valeriia Cherepanova, Micah Goldblum, Harrison Foley, Shiyuan Duan. John P Dickerson, Gavin Taylor, and Tom Goldstein. Lowkey: Leveraging adversarial attacks to protect social media users from facial recognition. ICLR, 2021.
[21] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Dehenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robusthench: standardized adversarial robustmess benchmark. In Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021. URL https: / / http///peneeeeeeeeeeeeeeeeeeeeee htr///ormmm/// wwwrooooounnnniiir. didd/dddddsssdsSo JCt7B.
[22] Zhun Deng, Linjun Zhang, Kailas Vodrahalli, Kenji Kawaguchi, and James Zou. Adversarial training helps transfer learning via hetter representations. In NeurlPS, 2021.
[23] Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. Sever: A robust meta- aega-aaggoi for stochastic optimization. In ICML, 2019.
[24] Thomas G Dietterich. Machine learning for sequential data: A review. In Joint IAPR international workshops on statistical techniques in pattern recognition (SPR) and structural and syntactic pattern recognition (SSPR), 2002.
[25] Edgar Dobriban, Hamed Hassani, David Hong. and Alexander Robey. Provable tradeoffss in adversarially robust classification. arXiv preprint arXiv:2006.05/66 2020.
[26] Yinpeng Dong, Zhijie Deng. Tianyu Pang, Jun Zhu, and Hang Su. Adversarial distributional training for robust deep learning. In NeurlPS, 2020.
[27] Yinpeng Dong, Ke Xu, Xiao Yang, Tianyu Pang, Zhijic Deng, Hang Su, and Jun Zhu. Exploring memorization in adversarial training. arXiv preprint urXiv:216600166 2021.
[28] Yinpeng Dong, Xiao Yang, Zhijie Deng, Tianyu Pang, Zihao Xiao, Hang Su, and Jun Zhu. Black-box detection of backdoor attacks with limited information and data. In ICCV, 2021.
[29] Xuefeng Du, Jingfeng Zhang. Bo Han, Tongliang Liu, Yu Rong, Gang Niu, Junzhou Huang, and Masashi Sugiyama. Learning diverse-structuree networks for adversarial robustness. In ICMI,, 2021.
[30] Ivan Evtimov, Ian Covert, Aditya Kusupati, and Tadayoshi Kohno. Disrupting model training with adversarial shortcuts. In JCML Workshop, 2021.
[31] Farhad Farokhi. Regularization helps with mitigating poisoning attacks: Distribattonaally-ohur machine learning using the wasserstein distance. arXiv preprint arXiv:2001.1063 2020.
[32] Ji Feng. Qi-Zhi Cai, and Zhi- Hua Zhou. Learning confuse: generating training time adversarial data with auto-en In NeurIPS, 2019.
[33] Liam Fowl, Ping-yeh Chiang, Micah Goldblum, Jonas Geiping, Arpit Bansal, Wojtek Czaja, and Tom Goldstein. Preventing unauthorized use of proprictary data: Poisoning for secure dataset release. arXiv preprint arXiv:2103.c 02683, 2021.
[34] Liam Fowl, Micah Goldblum, Ping-ych Chiang, Jonas Geiping, Wojtek Czaja, and Tom Goldstein. Adversarial examples make strong poisons. In NewrlPS, 2021.
[35] Adriano Franci, Maxime Cordy, Martin Gubri. Mike Papadakis, and Yves Le Traon. Effective and efficient data poisoning in semi-supervised learning. arXiv preprint 2000:201 2020.
[36] Ji Gao, Amin Karbasi, and Mohammad Mahmoody. Learning and certification under instance-targeted poisoning. In UAI, 2021.
[37] Ruize Gao, Feng Liu, Jingfeng Zhang, Bo Han, Tongliang Liu, Gang Niu, and Masashi Sugiyama. Maximum mean discrepancy test is aware of adversarial attacks. In ICML, 2021.
[38] Yansong Gao, Change Xu, Derui Wang. Shiping Chen. Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Annual Computer Security Applications Conference, 2019.
[39] Jonas Geiping, Liam Fowl, Ronny Huang, Wajeiech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. arXiv preprint mrXiv:2009.022770 2020.
[40] Jonas Geiping, Liam Fowl, Gowthami Somepalli, Micah Goldblum, Michael Moeller, and Tom Goldstein. What doesn't 't kill you makes you robust (er): Adversarial training against poisons and backdoors. arXiv preprint 102. 2021.
12

---

[41] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR. 2018.
[42] Amir Globerson and Sam Roweis. Nightmare at test time: robust learning by feature deletion. In ICML, 2006.
[43] Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. arXiv preprint arXiv:2012. 2020.
[44] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
[45] Sven Gowal, Sylvestre Sylvestre-Alvise Rebuffi, Olivia Wiles. Florian Stimberg, Dan Andrei Calian, and Timothy Mann. Improving robustness using generated data. In NeurlPS, 2021.
[46] Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Vardhan Simhadri, and Prateck Jain. Droce: Deep robust one-class classification. In ICML, 2020.
[47] Tianyu Gu, Kang Liu, Brendan Bolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230-4724 2019.
[48] Jonathan Hayase. Weihao Kong, Raghay Somani, and Sewoong Oh. Spectre: defending against backdoor attacks using robust statistics. In ICML. 2021.
[49] Kaiming Hc, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[50] Katherine L Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. In NeurlPS, 2020.
[51] Sanghyun Hong. Varun Chandrasekaran, Vigitcan Kaya, Tudor Dumitras, and Nicolas Papernot. On the effectiveness of mitigating data poisoning attacks with gradient shaping. arXiv preprint 2rXiv:2002.11497, 2020.
[52] Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[53] Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlcamable examples: Making personal data unexploitable. In ICLR, 2021.
[54] Hanxun Huang, Yisen Wang, Sarah Monazam Erfani, Quanquan Gu, James Bailey, and Xingjun Ma. Exploring architectural ingredients of adversarially robust deep neural networks. In NewrlPS, 2021.
[55] Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. Metapoison: Practical general- clcan-label data poisoning. In NewrlPS, 2020.
[56] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In NeurlPS, 2019.
[57] Saumya Jetley, Nicholas Lord, and Philip Torr. With friends like these, who needs adversaries? In NeurlPS, 2018.
[58] Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. TPAMI, 2020.
[59] Sean Kandel, Andreas Paepcke, Joseph Hellerstein, and Jeffrey Heer. Wrangler: Interactive visual specification of data transformation scripts. In CHI Conference Human Factors in Computing Systems (CHI), 2011.
[60] Klim Kireev, Maksym Andriushchenko, and Nicolas Flammarion. On the effectiveness of adversarial training against common corruptions. ICLR RobustMI. Workshop, 2021.
[61] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In JCML, 2017.
[62] Pang Wei Koh, Jacob Steinhardt, and Percy Liang, Stronger data poisoning attacks break data sanitization defenses. arXiv preprint arXiv:1811.0074 2018
[63] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
13

---

[64] Ram Shankar Siva Kumar. Magnus Nyström, John Lambert, Andrew Marshall, Mario Goertzel. Andi Comissoneru, Malt Swann, and Sharon Xia. Adversarial machine learning-industry perspectives. In IEEE Security and Privacy Workshops (SPW), 2020.
[65] Mathias Lechner, Ramin Hasami, Radu Grosu, Daniela Rus, and Thomas A Henzinger. Adversarial training is not ready for robot learning. In ICRA, 2021.
[66] Alexander Levine and Soheil Feizi. Deep partition aggregation: Provable defenses against general poisoning attacks. JCLR, 2021.
[67] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. NeurlPS, 2021.
[68] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li. and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In ICLR, 2021.
[69] Kaizhao Liang, Jacky Zhang, Boxin Wang. Zhuolin Yang, Sanmi Koyejo, and Bo Li. Uncovering the connections between adversarial transferability and knowledge transferability. In ICML, 2021.
[70] Xuanqing Liu, Si Si, Xiaojin Zhu, Yang Li, and Cho-Jui Hsich. unified framework for data poisoning attack to graph- based semi supervvie learning. In NewrIPS, 2019.
[71] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lec, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In Anmual Network and Distributed System Security Symposium (NDSS), 2018.
[72] Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In ECCV, 2020.
[73] Yuzhe Ma, Xiaojin Zhu Zhu, and Justin Hsu. Data poisoning against differentialll-priieeee private learners: Attacks and defenses. In IJCAI, 2019.
[74] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[75] Akshay Mehra, Bhavya Kailkhura, Pin- Yu Chen, and Jihun Hamm. Understanding the limits of unsuper- vised domain adaptation via data poisoning. In NeurlPS, 2021.
[76] Shike Mei and Xiaojin Zhu. Using machine teaching to identify optimal training-set attacks machine learners. In AAAI Conference on Artificial Intelligence (AAAI), 2015.
[77] Soyse-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In CVPR, 2017.
[78) Luis Gonzálzz, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back gradient optimization. ACM Workshop on Artificial Intelligence and Security, 2017.
[79] Preetum Nakkiran. A discussion of adversarial examples are not bugs, they are fcatures': Adversarial examples are just bugs, too. Distill, 2019, doi: 10.2391 5/disttll.00019.5. https://distill pub/20 1advexx- nugg-ddiscusioonrcssosss-
[80] Blaine Nelson, Marco Barreno, Fuching Jack Chi, Anthony D Joseph, Benjamin IP Rubinstein, Udam Saini, Charles Sutton, JD Tygar, and Kai Xia. Exploiting machine learning to subvert your spam filter. In Usenix Workshop on Large-Scale Exploits and Emergent Threats, 2008.
[81] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[82] James Newsome, Brad Karp, and Dawn Song. Paragraph: Thwarting signature learning by training maliciously. In International Workshop on Recent Advances in Intrusion Detection, 2006.
[83] Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In NeurlPS, 2020.
[84] Tuan Anh Nguyen and Anh Tuan Tran. Wanet - imperceptible warping-based backdoor attack. In ICLR, 2021.
[85] Adam Noack, Isaac Ahern Dejing Dou, and Boyang Li. An empirical study on the relation between network interpretability and adversarial robustness. SN Computer Science, 2021.
14

---

[86] Ren Pang, Hua Shen, Xinyang Zhang, Shouling Ji, Yevgeniy Vorobeychik, Xiapu Luo, Alex Liu, and Ting Wang. A tale of evil twins: Adversarial inputs versus poisoned models. In CCS, 2020.
[87] Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Bag of tricks for adversarial training. In ICLR, 2021.
[88] Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Accumulative poisoning attacks on real-time data. In NeurlPS, 2021.
[89] Nicolas Papernot. Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as defense to adversarial perturbations against deep neural networks. In IEEE symposium on security and privacy (SP), 2016.
[90] Neehar Peri, Neal Gupta, W. Ronny Huang, Liam Fowl, Chen Zhu, Soheil Feizi, Tom Goldstein, and John P. Dickerson. Deep k-nn defense against Meeenllbbll data poisoning attacks. In Adrien Bartoli and Andrea Fusiello, editors, ECCV Workshop, 2020.
[91] Mohammad Pezeshki, Sékou-Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guil- laume Lajoie. Gradient starvation: A learning proclivity in neural networks. In NeurlPS, 2020.
[92] Evani Radiya- Dixit and Florian Tramer. Data poisoning won't tt save you from facial recognition. In ICML 2021 Workshop on Adversarial Machine Learning, 2021.
[93] Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In ICML, 2020.
[94] Elan Rosenfeld, Ezra Winston, Pradcep Ravikumar, and Zico Kolter. Certified robustness to label-flipping attacks via randomized smoothing. In ICMI., 2020.
[95] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.
[96] Aniruddha Saha, Ajinkya Tejankar, Soroush Abbasi Koohpayegani, and Hamed Pirsiavash. Backdoor attacks on supf-sup learning. arXiv preprint (0t.::2000.10222.. 2021.
[97] Mohammadreza Salehi, Atrin Arya, Barbod Pajoum, Mohammad Otoofi, Amirreza Shaeiri, Moham- mad Hossein Rohban, and Hamid R Rabiee. Arae: Adversarially robust training autoencoders improves novelty detection. Neural Networks, 2021.
[98] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? In NewrlPS, 2020.
[99] Hadi Salman, Andrew Ilyas, Logan Engstrom, Sai Vemprala, Aleksander Madry, and Ashish Kapoor. Unadversarial examples: Designing objects for robust vision. In NeurlPS, 2021.
[100] Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Image synthesis with single (robust) classifier. In NeurlPS, 2019.
[101] Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adversarially robust generalization requires more data. In NeurlPS, 2018
[102] Avi Schwarzschild, Micah Goldblum, Arjun Gupta, John P Dickerson, and Tom Goldstein. Just how toxic is data poisoning? a unified henchmark for backdoor and data poisoning attacks. In ICMI,, 2021.
[103] Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In NeurIPS, 2018.
[104] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In NeurIPS, 2020.
[105] Shawn Shan, Emily Wenger, Jiayun Zhang, Huiying Li, Haitao Zheng, and Ben Y Zhao. Fawkes: Protecting privacy against unauthorized deep learning models. In USENIX Security Symposium, 2020.
[106] Juncheng Shen, Xiaolei Zhu, and De Ma. Tensorclog: An impereeptible poisoning attack on deep neural network applications. IEEE Access, 1411498-41506 2019.
[107] Reza Shokri et al. Bypassing backdoor detection algorithms in deep learning. In EuroS&P, 2020.
15

---

[108] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[109] Aman Sinha, Hongseok Namkoong, and John Duchi. Certifiable distributional robustness with principled adversarial training. In ICLR, 2018.
[110] Matthew Staib and Stefanic Jegelka. Distributionally robust deep learning as a generalization of adversarial training. In NeurlPS workshop on Machine Learning and Computer Security, 2017.
[111] Jacob Steinhardt, Pang Wei Koh, and Perey Liang. Certified defenses for data poisoning attacks. In NeurIPS, 2017.
[112] Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi. Pin-Ye Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy? comprehensive study on the robustness of 18 deep image classification models. In ECCV, 2018.
[113] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
[114] Jihoon Tack, Sihyun Yu, Jongheon Jeong, Minscon Kim, Sung Ju Hwang, and Jinwoo Shin. Consistency regularization for adversarial robustness. In ICML 2021 Workshop on Adversarial Muchine Learning. 2021.
[115] Lue Tao, Lei Feng, Jinfeng Yi, and Songean Chen. With false friends like these, who can notice mistakes? In AAAI, 2022.
[116] Mattco Terzi, Alessandro Achille, Marco Maggipinto, and Gian Antonio Susto. Adversarial training reduces information and improves transferability. In AAAI, 2021.
[117] Qi Tian, Kun Kuang, Kelu Jiang, Fei Wu, and Yisen Wang. Analysis and applications of class-wise robustness in adversarial training. In KDD, 2021.
[118] Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In Neur!PS, 2018.
[119] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry, Robust- ness may be at odds with accuracy. In ICLR, 2019.
[120] Alexander Turner, Dimitris Tsipras, and Aleksander Madry, Label-consistent backdoor attacks. arXiv preprint arXiv: 1912.02771, 2019.
[121] Francisco Utrera, Evan Kravitz, N. Benjamin Frichson, Rajiv Khanna, and Michael W. Mahoney. Adversarially deep nets transfer better: Illustration on image classification. In ICLR, 2021.
[122] Akshaj Veldanda and Siddharth Garg. On evaluating neural network backdoor defenses. arXiv preprint arXiv:2010.122 12186, 2020.
[123] Qizhou Wang, Feng Liu, Bo Han, Tongliang Liu, Chen Gong, Gang Niu, Mingyuan Zhou, and Masashi Sugiyama. Probabilistic margins for instance rewcighting in adversarial training. In NeurlPS, 2021.
[124] Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. ICMI., 2019.
[125] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. ICLR, 2020.
[126] Yunjuan Wang, Poorya Mianjy, and Raman Arora. Robust learning for data poisoning attacks. In JCML, 2021.
[127] Maurice Weber, Xiaojun Xu, Bojan Karlaš, Ce Zhang, and Bo Li. Rab: Provable robustness against. backdoor attacks. arXiv preprint arXiv:2003.08M0 2020.
[128] Cheng-Hsin Weng, Yan-Ting Lee, and Shan-Hung Brandon Wu. On the trade-off between adversarial and backdoor robustness. In NeurIPS, 2020.
[129] Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In ICML, 2018.
[130] Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In NeurIPS, 2021.
16

---

[131] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In NeurIPS, 2020.
[132] Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, and Fabio Roli, Is feature selection secure against training data poisoning? In ICML, 2015.
[133] Cihang Xie, Mingxing Tan, Boging Gong, Jiang Wang, Alan L Yuille, and Quoe V Le. Adversarial examples improve image recognition. In CVPR, 2020.
[134] Han Xu, Xiaorui Liu, Yaxin Li, and Jiliang Tang. To be robust or to be fair: Towards faireess in adversarial training. arXiv preprint arXiv:2010.061 20122100..01.1..... 2020.
[135] Yao-Yuan Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R Salakhutdinov, and Kamalika Chaudhuri. A closer look accuracy vs. robustness. In NeurlPS, 2020.
[136] Mingyang Yi, Lu Hou, Jiacheng Sun, Lifeng Shang. Xin Jiang, Qun Liu, and Zhiming Ma. Improved ood generalization via adversarial training and pretraing. In ICML, 2021.
[137] Chia-Hung Yuan and Shan-Hung Wu. Neural tangent generalization attacks. In ICMI., 2021.
[138] Bohang Zhang, Tianle Cai, Zhou Lu, Di He, and Liwei Wang. Towards certifying I-infinity robustness using neural networks with dinf-dist neurons. In ICML, 2021.
[139] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoret- ically principled trade-off hetween robustness and accuracy. In ICMI,, 2019.
[140] Jingfeng Zhang, Xilie Xu. Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In ICML, 2020.
[141] Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry- aware instancc-reweigghedd adversarial training. In ICLR, 2021.
[142] Tianyuan Zhang and Zhanxing Zhu. Interpreting adversarially trained convolutional neural networks. In ICML, 2019.
[143] Mengehen Zhao, Bo An. Wei Gao, and Teng Zhang. Efficient label contamination attacks against black-box learning models. In IJCAI, 2017.
[144] Shihao Zhao, Xingjun Ma. Yisen Wang. James Bailey. Bo Li, and Yu-Gang Jiang. What do deep nets learn? class-wise patterns revealed in the input space. arXiv preprint arXiv:2101.6680 2021.
[145] Chen Zhu, W Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transfer- able clean-label poisoning attacks deep neural nets. In JCML, 2019.
[146] Jianing Zhu, Jingfeng Zhang, Bo Han, Tongliang Liu, Gang Niu, Hongxia Yang, Mohan Kankanhalli, and Masashi Sugiyama. Understanding the interaction of adversarial training with noisy labels. arXiv preprint orXiv:2102.0340000 2021.
[147] Sicheng Zhu, Xiao Zhang, and David Evans. Learning adversarially robust representations via worst-case mutual information maximization. /CML, 2020.
17


Reviews:
[{"review_id": "Q5f0Ubs5YWd", "reviewer": "Reviewer_5bn7", "summary": "The paper tackles the problem of delusive attacks. It reveals that adversarial training, traditionally considered as one of the best defense methods against adversarial examples, can serve as a powerful and elegant defense against delusive attacks. This expands the defense ability of adversarial training from test-time evasion attacks to training-time availability attacks, making the technique significantly more applicable in practice.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\n\nThe paper pioneers the use of adversarial training to mitigate delusive attacks. It shows that adversarial training on perturbed examples minimizes an upper bound of natural risk on clean examples. Theoretical analyses also reveal the internal mechanisms of the defense method. As far as I know, these points have never been theoretically justified before.\n\n### Quality\n\nThe paper is technically sound and is a solid contribution to the community. Theoretical results are novel and experimental results are remarkable. Both results indicate that adversarial training is a promising solution to defend against delusive attacks.\n\n### Clarity\n\nThe writing of this paper is overall clear and straightforward. The experiments are thorough and well explained. \n\nA few minor observations: line 106: \"Eq. (3)\" -> \"Eq. 3\", line 128: \"good\" -> \"a good\", line 141: \"this case\" -> \"these cases\", line 199: \"varying d\" -> \"vary d\", line 140: \"changes of\" -> \"under\", line 300: \"and, show\" -> \", and show\", line 354: \"optimize\" -> \"optimizes\".\n\n\n\n### Significance\n\n1. The paper solves an important and challenging problem of defending against delusive attacks. It provides a novel way to utilize all training data in an elegant manner, rather than abandoning the correctly labeled examples. I believe that many practitioners will be glad to adopt the defense method when their training data is collected from untrusted sources. After all, it's better to be safe than sorry.\n2. It is well-known that adversarial training usually leads to a drop in natural accuracy. The paper shows that adversarial training instead can improve the test accuracy on clean examples in the settings of delusive threat models. I found this phenomenon very intriguing. At the same time, this provides a new and convincing argument for the application of adversarial training in many real-world applications.\n3. Extensive experiments validate the superiority of the proposed method. The authors also show the versatility of the proposed defense with several experiments; in particular, the authors show the possibility of delusive attacks and defense for rotation-based self-supervised learning. I thought this would inspire researchers to explore the vulnerabilities of SSL and find countermeasures. I also noticed some orthogonal works [1] [2] in this direction that you may find helpful.\n\n### Experiments\n\n1. The authors mention that their implementation of adversarial training minimizes a lower bound of adversarial risk (they adopt the popular PGD-AT method). Why not adopt SOTA certification methods such as CROWN-IBP [3] or Linf-dist Net [4] to directly optimize the adversarial risk?\n2. There is still a small gap between the natural accuracy of a standard model trained on the clean data and that of a robust model trained on the perturbed data. Is the gap inevitable?\n3. The authors may want to discuss the computation cost of adversarial training compared to standard training.\n4. From Figure 5, it turns out that the optimal budgets for P1/P2 and P3/P4 are different. Why is this case? The authors may want to give a reasonable explanation.\n\n\n[1] Saha et al. Backdoor Attacks on Self-Supervised Learning. 2021\n\n[2] Carlini et al. Poisoning and Backdooring Contrastive Learning. 2021\n\n[3] Zhang et al. Towards stable and efficient training of verifiably robust neural networks. ICLR 2020\n\n[4] Zhang et al. Towards Certifying $\\ell_{\\infty}$ Robustness using Neural Networks with $\\ell_{\\infty}$-dist Neurons. ICML 2021", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Many limitations and negative aspects are discussed in the paper. ", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8hrs"}, "extra_text": {}}, {"review_id": "8l-IwWU_oif", "reviewer": "Reviewer_LKRg", "summary": "This paper shows adversarial training (AT) can resist delusive attacks naturally. \n\nAT can naturally prevent the learner from overly relying on non-robust features. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1 May I know what the main difference between delusive attacks and poisoning attacks is? \n\n2 A piece of work [1] has shown that adversarial training may intensify the backdoor attack. \nWould the claim contradict the finding in [1]? \n\n3 Have authors tried different epsilon_train budgets for adversarial training in defending delusive attacks?\n\n4 AT has a very strong smoothing effect [2], which can smoothen out the non-robust adversarial features. \nIf AT encounters robust adversarial features, it may strengthen it.\n\n[1] On the trade-off between adversarial and backdoor robustness, in NeurIPS 2020\n\n[2] Understanding the Interaction of Adversarial Training with Noisy Labels.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above.", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "81wjFFQI9zT", "reviewer": "Reviewer_ZbSz", "summary": "This paper address the problem of adversarial training for delusive attacks with perturbed yet correctly labeled training data. By characterizing delusive adversaries as maximizing the natural risk with perturbed training data (worst-case data) within a specific Wasserstein ball around the original data, the adversarial training (e.g. in [7]) is shown to minimize an upper bound of such adversarial risk (Theorem 1) and hence is principle to defend against such delusive attacks. Furthermore, some settings of perturbing non-robust features are given and discussed to identify several examples of delusive attacks and show how adversarial training is able to achieve lower natural risk than standard models (Theorem 3). To my understanding, the main contribution is the formal analysis of the performance of adversarial training in defending the delusive attacks, with the new metrics employed such as the Wasserstein distance to generate the perturbed data.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is overall well written in presenting the theoretical analysis of adversarial training against delusive attacks. However, my main concern is in the significance of the contribution in the paper. It seems the main idea is to characterize the delusive attacks using Wasserstein distance as done by [30], and then show the existing adversarial training formulation (e.g. [7]) is also effective in resisting such delusive adversaries by actually minimizing the worst-case adversarial risk. While the results are interesting, I am not sure if such analysis is significantly novel given that no new adversarial training theory nor attacks are proposed to advance the current state of the art adversarial training algorithms, but to verify the performance in different scenarios. To help readers better appreciate the contribution, authors are  encouraged to provide an explicit statement of contribution and specify any actual contributions presented by the theorems.\n\nOn the other hand, I have some questions regarding the generality of the adversarial training in resisting against delusive attacks. While Section 3.2 provides some examples and settings to justify how adversarial training will naturally prevent the learner from relying too much on the non-robust features, I was wondering if such training process is able to deal with other classification problems beyond the example of a mixture of two Gaussian distributions? Besides the given binary examples, it would be helpful to give additional discussions on how generalizable the obtained theorem 3 is, e.g. to non-binary classification problems with non-Gaussian data?\n\nSome other questions:\n\n- It is mentioned in the paper that while adversarial training usually would negatively affect its performance on clean data without attacks, the adversarial training using Wasserstein ball is able to achieve good performance even without adversaries. However, I am not able to find explicit discussions to justify this claim. Is this concluded from Theorem 1 that the adversarial training is also minimizing the upper bound of “natural risk” besides the “adversarial risk”, and hence standard training objective is also implied?\n\n- While this paper deals with adversarial training, the reported results in Fig. 2, 5, 6 and 7 do not contain any performance variance information. In order to further justify the results and performance, authors are strongly suggested to provide more quantitative results such as the number of trials and performance variance besides the average results.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed some limitations of the work, but the reviewer has some questions regarding other limitations such as the generality of the adversarial training in classification problems other than the discussed binary classification task with Gaussian data. Authors are suggested to add more discussions to address the comments above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "0Z_rN8pVW6w", "reviewer": "Reviewer_5L8s", "summary": "The paper proposes a defense method based on adversarial training to defend against delusive adversaries at the training time. The paper shows that adversarial training can largely improve the accuracy under delusive attacks if the defender applies pessimistic budget. Both experiments and theory verify the claims. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n1. The delusive attacks are indeed a threat to the training of networks. And adversarial training can largely improve the accuracy under the attacks.\n2. The analysis and the theory of the methods are good and reasonable, which point out why adversarial training improves the accuracy.\n3. The paper is clearly written and easy to follow.\n\n**Cons**\n1. It has been pointed out in [1] (Section 4.3, Under Noise Filtering Techniques) that adversarial training can largely improve the accuracy of delusive attacks. Although that paper does not carefully analyze the phenomenon as this paper, it limits the contribution and novelty of this paper.\n2. One biggest drawback of the method is the choices of adversarial budget $\\varepsilon$. In Figure 5, the accuracy seems to be sensitive to choice of $\\varepsilon$ of the defender. Another question is the generalization across different attack norm. If the attacker chooses attack in $\\ell_2$ norm and the defend chooses to defend in $\\ell_\\infty$ norm, will the accuracy still be improved? The transfer across different attack norm is also a big problem in adversarial training.\n3. The analysis is based on the Wasserstein distance. However, the experiments seems to be performed in single $\\ell_p$ norm ball, which does not allow different perturbations budget for different training samples.\n\n\nOverall, I think the paper proposes a good way to prevent delusive attacks. I vote for rejection beacause of \n1. Lack of novelty mentioned in point 1 of Cons senction.\n2. It also cannot solve more practical problems for different norms in point 2 of Cons section.\n\nI would increase my score if both of the concerns can be resolved.\n\n\n[1] Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, Yisen Wang. Unlearnable Examples: Making Personal Data Unexploitable, ICLR 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyzes in a particular context how adversarial training can help with preventing delusive adversaries affect training. Delusive adversaries are defined to be those that may modify training data (feature, not labels) to make the resulting classifier have bad test error. The analysis is specifically in the case of Gaussian noise. There is also experimental evidence provided. This paper resulted in a long exchange between reviewers and authors and the reviewers turned increasingly positive about the paper. The authors must include all of the additional details that have been included in the discussion in the next iteration of the paper.
Decision: Accept (Poster)