Indiscriminate Poisoning Attacks Are Shortcuts
Authors: Da Yu
Huishuai Zhang
Wei Chen
Jian Yin
Tie-Yan Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: Data Security
Data Poisoning
Shortcuts

Abstract: Indiscriminate data poisoning attacks, which add imperceptible perturbations to training data to maximize the test error of trained models, have become a trendy topic because they are thought to be capable of preventing unauthorized use of data.  In this work, we investigate why these perturbations work in principle. We find that the perturbations of advanced poisoning attacks are almost linear separable when assigned with the target labels of the corresponding samples. This is an important population property for various perturbations that were not unveiled before. Moreover, we further confirm that linear separability is indeed the workhorse for poisoning attacks. We synthesize linear separable data as perturbations and show that such synthetic perturbations are as powerful as the deliberately crafted attacks.  Our finding also suggests that the shortcut learning problem is more serious than previously believed as deep learning heavily relies on shortcuts even if they are of an imperceptible scale and mixed together with the normal features. It also suggests that pre-trained feature extractors can be a powerful defense.

Under review as a conference paper at ICLR 2022
INDISCRIMINATE POISONING ATTACKS ARE SHORTCUTS
Anonymous authors Paper under double-blind review
ABSTRACT
Indiscriminate data poisoning attacks, which add imperceptible perturbations to training data to maximize the test errof" have become a trendy topic because they are thought to be capable of preventing unauthorized use of data. In this work, we investigate why these attacks work in principle. We find that the perturbations of advanced attacks are almost linear separable when assigned with the target labels of the corresponding samples. This an important population property for various perturbations that were not unveiled before. Moreover, further confirm that linear separability is indeed the workhorse for recent attacks. We synthesize linear separable data perturbations and show such synthetic perturbations are as powerful as the deliberately crafted attacks. Our finding also suggests that the shortcul learning problem is more serious than previously believed as deep models heavily relies on shortcuts even if they are of an imperceptible scale and mixed together with the normal features. It also suggests that pre-trained feature extractors can be a powerful defense.
INTRODUCTION
Big datasets crawled from the Internet keep
advancing the state-of- of-th-of-the-rt results (De vlin et al. (0088) He et al. 2020 Chen et al.] 2020). However, there increasing con- cerns about the unauthorized use of per- sonal data (Hill Krolik 2019 Prabhu & Birhane 2020: Carlini et al., (020). For
Imperceptible
Low Accuracy
Perturbations
Publish
Train
ML Models
instance, private company has collected more than three billion face images to build commercial face recognition models without acquiring any user consents (Hill 2020). To address those concerns, many data poisoning attacks have been proposed to prevent data from being learned by unauthorized deep models (Feng et al.) 2019; Shen et al. 2019 Huang et al.) 2021 Yuan Wu 2021 Fowl et al. They add imperceptible perturbations the training data that the model accuracy unseen data is arbitrarily bad. We refer to these attacks as indiscriminate poisoning attacks as the adversary targets at all examples. This type of attack also know availability attack (Biggio & [Rol] or delusive attack (Tao et al.) 20211. We note that the word is also used to denote the adversary does not have a specific target class (Muñoz- MonzzlzGonzález al. 2017 Figure we give an
Perturbed Samples
Clean Semples
Figure 1: An illustration of indiscriminate poisoning attacks.
2021a b).
illustration of the attacks studied in this paper.
Roughly speaking, there are three methods available construct the indiseriminate poisoning a The first method formulates the perturbations as the solution of a bi-level optimization
problem (Biggio et al. 2012 /Fng et al. 2019 Fowl al., [021a]] Yuan Wul 2021). The bi-level
optimization problem requires models trained on perturbed data to have the maximum loss on unseen
'There some attacks have the same objective but they inject malicious training samples instead of perturbing existing ones, e.g., (Biggio et al, (002)). In this work, we focus on the latter approach.
2This paper focuses on recent attacks that are designed for deep neural networks. Some earlier attacks against.
SVM covered in this work.

---

Under review as a conference paper at ICLR 2022
DeepConfisse
60
-40
-40
Cass
-10
-00
-00
-03
-70
60
Figure 2: T-SNEs of the first three classes of clean CIFAR-I0 data and the perturbations generated via DeepConfuse (Feng al. (019) and error-minimizing noises (Huang al. 202T). The perturbations are flattened and normalized into unit norms.
data. Then Huang et al. 2021) conceive a simpler poisoning attack called error-minimizing noises, where the perturbation on training data is crafted by minimizing the training loss so that nothing is left to backpropagate in the regular training procedure. The last method uses common adversarial
examples an indiscriminate poisoning attack (Nakkiran) 2019 Tao et al. [021 Fowl et al. 202 lb).
Despite of these quite different approaches, they all give strong poisoning attacks. Intrigued by this observation, we ask the following question:
What is the underlying workhorse of recent indiscriminate poisoning attacks?
To answer the above question, we first take a closer look at the perturbations of existing attacks. We visualize the perturbations of two advanced attacks via two-dimensional T-SNEs (Van der Maaten & Hinton] 2008) in 2 Surprisingly, the perturbations with the same class label are well clustered, suggesting that the perturbations would be linear separable in the original high-ddmensional space. We confirm this by fitting the perturbations with linear models. The perturbations are assigned with the labels of their target examples. It turns out that simple softmax regression models can fit the perturbations of four representative attacks with > 90% accuracy. This finding suggests that using linear separable perturbations may be the key for an indiscriminate poisoning attack succeed.
To further confirm that the linear separability is sufficient (not only necessary) condition, we reverse the above procedure: synthesizing some linear separable data to see if they can serve as data poisoning attacks. Extensive experiments on benchmark datasets and models demonstrate that the synthetic perturbations can be as powerful as advanced indiscriminate poisoning attacks. Notably, generating synthetic data perturbations is significantly easier and cheaper than existing attacks as it does not require solving any optimization problems. This finding reveals that linear separability is indeed the workhorse to the success of recent indiscriminate poisoning attacks.
The above finding coincidentally matches a recent concept called shorteut learning (Geirhos et al. 2020). Shortcut learning stands for the behavior that deep models tend to rely features that do not generalize on realistic test data. Such features are referred to as shorteuts. With this concept, the perturbations of indiscriminate poisoning attacks are also shortcuts. We note that the shortcuts in previous works are usually part of natural data, which are somehow heuristic, e.g., "grass" is shortcut for recognizing "cow" in natural images (Beery et al. 2018 Geirhos et al. 2020). In this work, we expose a more explicit form of shortcuts and discuss extensively how to construct such shortcuts. We unveil that deep learning models would overwhelmingly rely on spurious shortcuts even though the shortcuts are scaled down to an imperceptible magnitude. This finding exposes fundamental vulnerability of deep models and hence may of independent interest to the community.
Finally, our understanding of the working principle of indiscriminate poisoning attacks also motivates a powerful defense against them. As the attacks succeed by providing shortcuts, they will be less effective if those shortcuts are filtered out. To achieve this, we use pre-trained models as feature extractors. a pre- are-trained model is trained on datasets that have similar distributions as the target dataset, it can extract useful features and avoid shortcuts. Our experiments suggest that training only the linear classification layers with the features extracted from the pre-traine trained model can successfully defend existing attacks.
2

---

Under review as a conference paper at ICLR 2022
Our findings and contributions are summarized as follows:
We find that the perturbations used in several advanced indiscriminate poisoning attacks are (almost) linear separable. We further validate that this property is the workhorse for the perturbations to be effective. To the best of our knowledge, we are the first to unveil this important property, which is fundamental to understand existing poisoning attacks and may inspire future attacks.
We connect this property to the shortcut learning problem and demonstrate the omnipres- ence of shortcuts in learning. We show that one can construct invisible shortcuts that machine learning models would heavily rely on. We believe this finding greatly widens the understanding shorteuts in machine learning.
Motivated by breaking the shortcuts, we use pre-trained feature extractors to defend against indiscriminate poisoning attacks. Experiments show the proposed defense is very powerful.
1.1 RELATED WORK
Data poisoning. In general, data poisoning attacks perturb training data to intentionally cause some
malfunctions of the target model (Biggio & Roli 2018 Goldblum al. 2020 Schwarzschild et al.,
202T). A common class of poisoning attacks aims to cause test-time error on some given samples
(Koh & Liang| 2017 Munioz-Gnzzllez et al. [2017 7 Chen et al. 2017 Koh et al.
2018 Shafahi et al.
2018 Zhu et al. 2019 Shan et al. 2020 Geiping et al. [2200] Huang et al. [2020] Cherepanova et al.
2021 or on all unseen samples (Biggio et al. 2012 Feng al. 2019 Liu Shrofi [019 Shen et al.
2019
Huang et al. 1 Yuan & Wu 2021 Fowl et al. 202 b). The latter attacks are also known
as indiscriminate poisoning attacks as they do not have specific target examples (Barreno et al. 2010). Backdoor attack is another type of poisoning attack that perturbs training data so that the attacker
can manipulate the target model's ' output with a designed trigger (Chen et al. 2017]] Shafahi et al.
2018 Turner et al.) 2018 Xie et al. 2019 Bagdasaryan et al. |020 Nguyen & Tran 2020 Saha ct al. 2020 Nguyen & Tran 1021). In this work, we investigate indiscriminate poisoning attacks and
reveal the workhorse of them. We show that the perturbations of advanced attacks are (almost) linear separable. We further confirm that using linear separable perturbations is a sufficient condition to perform strong attacks.
There are defenses against indiscriminate poisoning attacks. Huang et al. (2021) show training with advanced data augmentation methods improves the test performance. Another defense is to train the target model with adversarial training (Madry et al. 2018). which currently achieves the best performance (Huang et al. 2021] Fowl et al.) 2021b Tao et al.) 202I). Recently, Radiya-Dixit & Traner (2021) challenge the security of poisoning attacks from another view. They argue that unless existing attacks can fool all future defenses, they cannot protect the data well because the adversary can simply save the perturbed data and wait for better ") In this work, we propose to use pre-trained models to defend indiscriminate poisoning attacks. Our experiments demonstrate that using pre-trained models is powerful defense against indiscriminate attacks. We note that Cinà et al. (2021) have explored running poisoning attacks against pre- trained feature extractors. Nonetheless, they focus on backdoor attacks and do not advocate using pre- trainedid models as defense.
Shorteut learning. Recently, the community has realized that deep models may rely on shortcuts to make decisions (Beery et al. 2018 Niven & Kao 2019 Ilyas et al. 2019 Geirhos al. 2020 Huh et al.] (021). Shorteuts are spurious features that are correlated with training labels but do not generalize test data. [Beery et al. (2018) show that deep model would fail to recognize cows when the grass background is removed, suggesting that the model takes "grass" as shortcut for "cow".. Niven& & (20g) show that large language models use the strong correlation between some simple words and labels to make decisions, instead of trying to understand the sentence, For instance, the word "not" is directly used to predict negative labels. In this work, we show the shortcut lcarning problem is more serious than previously believed. Our experiments in Section 3 demonstrate that deep models only pick shorteuts even if the shortcuts are scaled down to an imperceptible magnitude and mixed together with normal features. These experiments reveal another form of shortcut learning, which has already been unconsciously exploited by indiscriminate data poisoning attacks. We note that there also exist other synthesize datasets that offer stratification of features (Ross et al. 2017]
(RadiyaDDixhi & Tramer 2022] only verify this is achievable for targeted poisoning attacks (Shan et al. 2020 Cherepanova et al. 2021) though the general idea may also apply indiseriminate poisoning attacks.
3

---

Under review as a conference paper at ICLR 2022
Jacobsen et al. 2018 Hermann & Lampinen 2020 Shah et al. 2020). Those synthetic data contain shortcuts that can not be used as perturbations as they are visible and affect the normal data utility, e.g., vertically concatenations of images from MNIST and CIFAR-10.
1.2 NOTATIONS
We use bold lowercase letters, e.g., v. and bold capital letters, e.g., M, denote vectors and matrices, respectively. The Lp norm of a vector v is denoted by || A sample consists of feature æ and label y. We use denote a dataset which is sampled from distribution D. The loss of model f on given sample is denoted by (z),
2 INDISCRIMINATE POISONING ATTACKS USE LINEAR SEPARABLE PERTURBATIONS
In this section, we investigate the perturbations several advanced indiscriminate poisoning attacks. First, introduce three approaches of poisoning attacks. Then, we visualize them with two- dimensional T-SNEs. Finally, we verify that the perturbations of four advanced attacks are almost linear separable by fitting them with linear models and two-layer neural networks.
2.1
THREE TYPES OF INDISCRIMINATE POISONING ATTACKS
2.1.1
THE ALTERNATIVE OPTIMIZATION APPROACH
We first introduce the alternative optimization approach to generate perturbations for indiscriminate poisoning attacks. It solves the following bi-level objective,
arg max LE ( [(f (x), (ō)E)
s.t. f E arg min
Σ (a.y)ED
e(f( 5).
where 8 is sample-wise perturbation and constraint set of all perturbations. To put other words, the optimal solution perturbed data (specified by the second objective) should have maximum loss on clean data (specified by the first objective). The perturbations are restricted to not affect the normal data utility.
Directly solving Equation (T) is intractable for deep neural networks and recent works have designed multiple approximate solutions (Feng al. 2019 Fowl et al., 202 Yuan & Wu 2021). Feng et al. (2019) use multiple rounds of optimization to generate perturbations. At each round, they first approximately optimize the second objective by updating target model perturbed data for few steps. Then they approximate the first objective by updating generator for a few steps. The outputs of the generator are used as perturbations. Another example is the Neural Tangent Generalization Attacks (NTGAs) in Yuan & Wu (2021). They rewrite the entire Equation (I) into a single objective based on the recent development Neural Tangent Kernels (Jacot al. 2018 Then they solve the new objective with lightweight surrogate model.
2.1.2 THE ERROR-MINIMIZING NOISE
Huang et al. 2021) propose a simple way to generate data poisoning perturbations. Instead of solving Equation (1) they use the following objective,
min E (e.)) larg min {(f(x: 8),
(8)E
That is to say, the perturbations are intentionally optimized to reduce the training loss. A randomly initialized model is used as surrogate of the target model. They use multiple rounds of bi-level optimization to generate perturbations. At each round, they first train the surrogate model for a few steps to minimize the loss on perturbed data. Then they optimize the perturbations to also minimize
4

---

Under review as a conference paper at ICLR 2022
Table 1: Training accuracy (in %) of simple models the perturbations of different attacks.
Algorithm
Linear Model Two-layer NN
Clean Data
49.9
70.1
DeepConfuse (Feng al. 2019
100.0
100.0
NTGA (Yuan & Wu (021)
100.0
100.0
Error-minimizing (Huang ct al. 2021
100.0
100.0
Adversarial Examples (Untargeted) (Fowl et a
2021b)
91.5
99.9
Adversarial Examples (Targeted) (Fowl et al. 20216)
100.0
100.0
the loss of the surrogate model. They repeat the above process until the loss on perturbed data is smaller than pre-defined threshold. The main motivation is that if the training loss is zero, then the target model will have nothing to learn there is nothing backpropagate.
2.1.3 ADVERSARIAL EXAMPLES
Instead of using bi-level objectives, Fowl et al. 2021b) show that the common objectives of adversarial examples are sufficient to generate powertul data poisoning perturbations. They use both untargeted (the left objective) and targeted adversarial examples (the right objective),
arg max Ba [e (f (æ + ,)]] (B)
arg min (( [ (f +)) ,))),, (8)E
where y/ * incorrect label and f is trained model. Surprisingly, Fowl al. (2021b) demonstrate that these simple objectives can generate perturbations that achieve state-of-the-an attack performance.
2.2 VISUALIZZING THE PERTURBATIONS
Although the three approaches in Section 2.1 have different objectives, they all manage to generate perturbations with the same effect. Intrigued by this observation, we seek to find the underlying working principle of those approaches if there is common pattern of different types of perturbations.
To find such a common pattern, we first visualize different types of perturbations by computing their two-dimensional t-SNEs Van der Maaten & Hinton 2008). We generate perturbations using DeepConfuse (Feng et al.) 2019), NTGA Yuuan& & Wu [021). minimizii noises (Huang et al.) 2021 and adversarial examples (Fowl et al.) 202 These four attacks achieve advanced attack performance and cover all the three approaches in Section 2.1. We use the official implementations to generate perturbations. Detailed configurations are in Appendix
The wooddimensional I-SNEs of DeepConfuse and error-minimizing noises are shown in Figure 2 The plots of NTGA and adversarial examples are presented in Appendix B Surprisingly, for all the attacks considered, the perturbations for the same class are well clustered, suggesting that even linear also compute the t-SNEs of the clean data. As
models can classify them well. For comparison, shown in Figure[] in contrast with the t-SNEs of perturbations, the projections different classes of the clean data are mixed together, which indicates that they require complex neural network to be correctly classified.
2.3 PERTURBATIONS OF POISONING ATTACKS ARE ALMOST LINEAR SEPARABLE
In order quantify the 'linear separability' of the perturbations of different indiscriminate poisoning attacks, we fit the perturbations with simple models and examine the training accuracy. The perturba- tions are labeled with the labels of the corresponding target examples. The simple models include linear models and two-layer neural networks. Details can be found in Appendix
The results are presented in Table [ Compared to the results on clean data, simple models can easily fit the perturbations. On all attacks considered, linear models achieve more than 90% training accuracy and two-layer neural networks achieve nearly 100% training accuracy. These results confirm that the perturbations of advanced indiscriminate poisoning attacks all (almost) linear separable. We note that existing attacks against deep neural networks all use ReLU activation functions so their crafting models learn piecewise linear functions in input space. In Appendix C replace the ReL.U

---

Under review as a conference paper at ICLR 2022
layers with Tanh layers and show the perturbations are still linearly separable. This suggests that the linear separability is not stemming from the property of ReLU.
2.4 CONNECTING OUR FINDINGS To SHORTCUT LEARNING
The fact that the perturbations can be easily fitted by linear models naturally connects to recent concept named shortcut learning (Geirhos al., 2020). Shorteut learning summarizes a general phenomenon when any learning system makes decisions based on spurious features that do not generalize on realistic test data Shortcut features have been found different fields. For vision tasks, Beery et al. (2018) show deep models fail to recognize cows when the grass background is removed from images, suggesting the grass background is shortcut for predicting cows. In the field of natural language processing, Niven & Kao (2019) show language models the strong correlation between some simple words and labels to make decisions, instead of really understanding the data.
With the presence of shortcut learning, it seems sonable to postulate that the perturbations existing attacks succeed by creating shortcuts the target model. We give an illustration in Figure 3 A ma- jor difference between the perturbations of poisoning attacks and existing shortcut features is that the per- turbations are of an imperceptible scale and mixed together with useful features. Since there is no direct evidence show deep models will take this kind of shortcuts, the observation that the perturbations are linear separable may only be some superficial results of the underlying root cause. In the next sec- tion, we design experiments to confirm the postulated explanation. We synthesize imperceptible and lin- ear separable data and show deep models are very vulnerable to such synthetic shortcuts.
Normal Data From
Neural Networks
Training
0.03
Model only learns 'easy' from fails recognize features from y-
Parturbations x'
Figure 3: An illustration of the shorteut learn- ing phenomenon in this paper.
IS LINEAR SEPARABILITY A SUFFICIENT CONDITION FOR POISONING ATTACKS TO SUCCEED?
Although we have demonstrated that the perturbations of four advanced attacks are all almost linear separable, it is a bit early to claim that 'linear separability' is the underlying working principle of indiscriminate poisoning attacks. For example, perturbations are linear separable may only be necessary but not sufficient condition for poisoning attacks to succeed. In order to verify this postulated explanation, we use simple synthetic data serve as perturbations and compare their effectiveness with existing poisoning attacks. It turns out that the synthetic perturbations are as powerful as the perturbations of advanced attacks.
The rest of this section is organized as follows. In Section 3.1 we first give an algorithm for generating synthetic data as perturbations. Section 3.2 we verify the effectiveness of synthetic perturbations on different models and datasets.
3.1 GENERATING SIMPLE SYNTHETIC DATA
The synthetic data in this section are generated via two building blocks. In the first block, we use an algorithm in Guyon (2033) to generate samples from some normal distributions. In the second block, we transfer the samples into two-dimensional images in order to apply them to benchmark datasets.
The first building block proceeds as follows. We first generate some points that are normally distributed around the vertices of hypercube. The points around the same vertex are assigned with the same label. Then for each class, we introduce different covariance. Any two classes of the
Geirhos et al. (220)) use more specific definition of shortcuts. They denote shortcuts as those features that do not generalize out-of-distribution (OOD) data. We note that poisoning attacks would change the distribution of training data and hence make the clean test data 'OOD' with respect the trained model.

---

Under review as a conference paper at ICLR 2022
generated points can be easily classified by hyperplane as long the side length of the hypercube is reasonably large. We give the pseudocode of this block in Appendix A
Algorithm 1: Transform A 1-D Sample Into A 2-D Image
1: Input: input vector E R2, target image size
//For simplicity, we assume the image is square and is divisible by vd.
2: Compute patch size p= ==s/vd
3: Get æ TR by duplicating each dimension of a for p² times.
4: Rearrange x' to get d patches so that the pixel values in each patch are the same.
In the second building block, we pad each dimensional of the sampled points and reshape them into two-dimensional images. The padding operation introduces local correlation into the synthetic images. Local correlation is an inherent property of natural images. In Appendix F we show the padding operation necessary to make the synthetic perturbations remain effective when data augmentation methods are applied. The pseudocode of this block is given in Algorithm I
The synthetic images are scaled down before being used as perturbations. We visualize the synthetic perturbations and corresponding perturbed images in Figure 4 We also visualize the perturbations in Huang et al. (202T) I for a comparison. The details of perturbations can be found in Section 3.2 As
shown in Figure 4 the synthetic perturbations do not affect data utility.
Original
a)
c)
Figure 4: Visualization of perturbed images and normalized perturbations. Columns a) and b) use synthetic perturbations. Columns c) and d) use the attack in Huang al. (2011)
3.2 SYNTHETIC PERTURBATIONS ARE HIGHLY EFFECTIVE AS DATA POISONING ATTACK
Now we verify the effectiveness of synthetic perturbations and make comparisons with existing poisoning attacks. We perturb the entire training set following the setup in previous works (Feng et al. 2019; Huang et al. 2021 Yuan & Wu, 2021 Fowl et al. 2021b That is, we synthesize perturbation for every training example. We L2-norm for synthetic perturbations to keep the
sample-wise variation in the same class. We normalize the synthetic noises into L2-norm ball with
radius de', where d is the dimension of the input.
We evaluate synthetic perturbations on three benchmark datasets: SVHN (Netzer 2011 CIFAR- 10, and CIFAR-100 (Krizhevsky & Hinton] (2009). The target model architectures include VGG (Simonyan & Zisserman (2014). ResNet (He al. 2016), and DenseNet (Huang et al., 2017). We adopt standard random cropping and flipping as data augmentation. The hyperparameters for training are standard and can be found in Appendix We use 6/25 for synthetic perturbations. The
patch size in Algorithm T is set as 8.
We first compare synthetic perturbations with existing poisoning attacks. The comparisons are made on the CIFAR-10 10 dataset with ResNet- 18 as the target model. The perturbations in previous works
In Appendix D show synthetic perturbations are still effective when different percentages of training data are poisoned (range from 20% to 90%).
7

---

Under review as a conference paper at ICLR 2022
SVHN
CIFAN-30
CPAR-100
Trin Paaka wat tho (Chiue
Tain (hasonadi
Tar shaidr lus bes othiart
Tas (choet Tesi ithioe
43
22-
7033
Figure 5: Training curves of ResNet- 18 models on perturbed and clean data. The word 'poisoned' denotes the model trained on perturbed data. The test performance is evaluated on clean data. The test accuracy is low throughout training when synthetic perturbations are added.
Table 2: Accuracy on clean test data of CIFAR-10. 10. The target model is ResNet-18 18. The training data are poisoned with different attacks. The smaller the accuracy, the better the attack efficiency.
Algorithm
Test Accuracy (in %)
No Perturbation
94.69
TensorClog (Shen et al.
2019
48.07
Alignment (Fowl ct al.
2021a
56.65
DeepConfuse (Feng et al.
2019
28.77
NTGA (Yuan & Wu 2021
33.29
Error-minimizing (Huang
2021
19.93
Adversarial Examples (Fowl cl. Synthetic Perturbatrons
202
6.25
13.54
are normalized with Los bound with 8/255. We present the comparisons in Table Then we evaluate synthetic perturbations on different models and datasets. The final test accuracy of target models in Table 3 We also plot the training curves of target models on both clean and perturbed
data in Figure
As shown in Table synthetic perturbations are as powerful as advanced poisoning attacks and reduce the test accuracy close to that of random guessing. The results in Table 3 and Figure 3 further confirm the effectiveness of synthetic perturbations. Notably, generating synthetic perturbations data irrelevant and only takes several seconds using a single CPU core. We compare the computational complexities of synthetic perturbations and recent attacks Appendix
In summary, experiments demonstrate that using linear separable perturbations is indeed sufficient condition for indiscriminate poisoning attacks to succeed. Moreover, these results also expose that deep models are very vulnerable obscured shortcuts. This finding has two meanings to the community. First, it confirms that advanced indiscriminate poisoning attacks do succeed by providing shorteuts. Second, further exposes the shortcut learning problem, which fundamental vulnerability of deep models.
Table 3: Accuracy (in %) on clean test data. The target models are trained on clean data (D.) and data perturbed by synthetic perturbations (I) (Dsyn).
SVHN
CIFAR-10
CIFAR-100
Target Model
De
Bey
Deyn
De
Dayn
VGG-11
95.4
18.1
91.3
28.3
67.5
10.9
ResNet-18
96.2
94.7
13.5
74.8
ResNet-50
96.4
7.8
94.8
14.9
75.2
8.4
DenseNet-1e 121
96.7
9.7
95.0
10.6
76.5
7.6
8

---

Under review as a conference paper at ICLR 2022
4 DEFEND INDISCRIMINATE POISONING ATTACKS WITH PRE-TRAINED FEATURE EXTRACTORS
We now explore possible defenses against indiscriminate poisoning attacks with our understanding. The experiments in Section and suggest that deep models only use shortcuts to make predictions and ignore the useful features. A reasonable defense should be able to filter out such shortcuts and make the decision based on normal features. To achieve this goal, we explore using pre-trained models to extract useful features. Pre-training is a popular technique that trains the model on some auxiliary data before learning on target data. If the auxiliary data is similar to the target data, the pre-trained models can recognize some useful features and hence avoid relying on shortcuts.
Our implementation is follows. We with SimCLR (Chen et al.) 2020). To apply the pre-trained model on CIFAR-10, we remove the final layer and add ten-class readout layer. The output dimension of the penultimate layer is 4096. We try two different approaches to take advantage of the pre-trained model. The first approach uses the pre-trained weights initialization and fine-tunes the whole model. The second approach only uses the outputs of the penultimate layer and trains a linear classifier on them.
ResNet-152 model pre-trained unlabeled ImageNet
We run experiments on five types perturbed CIFAR-10 datasets that are crafted via DeepConfuse (Feng et al. 2019), NTGA (Yuan & Wu 202T), error- aror-minimizing noise (Huang et al.) 202Ip. error- maximizing noise (Fowl et al. (02Tb), and synthetic perturbations. The strength of perturbations the same as that in Section [] In Appendix G we increase up to 32 and show the proposed defense remains effective. The hyperparameter choices of fine-tuning can be found in Appendix
B
Our baseline method is adversarial training. It is powerful defense against existing attacks (Huang jet al., 2021; Tao et al. [202T] Fowl et al., 2021b). We use Fast Gradient Sign Method (FGSM) (Goodfellow et al. (2144) with the Loo norm bound to generate adversarial examples. We choose
from a list (2/255, 4/255, 8/255] and report the best test accuracy for adversarial training.
Table 4: Test accuracy (in %) when different countermeasures are applied. 'Linear eval.' means fix the pre-trained weights and only train the classification layer. Adv. Training
Algorithm
Pre-train
Pre-train (Linear eval.)
No Perturbation
93.8
95.9
95.4
DeepConfuse (Feng et al. 2019
86.5
17.6
94.1
NTGA (Yuan & Wu 2021
90.8
39.5
95.2
Error metoo-mmnimizing (Huang al. 2021
91.2
21.6
94.0
Error- miximiiaxiiiinn (Fowl et al. 20216
85.4
37.3
87.6
Synthetic Perturbations
87.8
15.7
93.8
The results are shown in Table When fine-tuning the full models, the models still pick the shortcuts and achieve poor test accuracy. In contrast, when we freeze the pre-trained weights and only train linear classifiers on the extracted features, pre-training substantially increases the test accuracy. Moreover, the pre-trained feature extractor achieves better defense than adversarial training.
Although the proposed defense achieves state-of-the-art results against advanced attacks, it could have potential risks when facing white-box attacks. The four advanced poisoning attacks evaluated in this section are all black-box, i.e., they do not have access the target model or try recover the target model first. The power of our defense may be compromised if the perturbations are generated invalid the chosen pre- pre-trained model.
5 CONCLUSION
This work gives an explanation of the working principle of indiscriminate poisoning attacks. We show advanced attacks coincidentally generate linear separable perturbations. We further synthesize linear separable perturbations demonstrate that using linear separable perturbations is a sufficient condition for attacks to succeed. Our findings also suggest deep models are more prone to shorteuts than previously believed as they will find and heavily rely shorteuts when the shortcuts are scaled down to an imperceptible magnitude. Finally, our explanation also motivates us to use pre-trained feature extractors as powerful defense.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We upload our source code in the supplementary materials. The README files in the uploaded code provide example commands to verify our results. Our implementation is based Pytorch (bttps://pytorcch.orz which is a popular open-source machine learning framework.
REFERENCES
Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units. arXiv preprint arXiv:1611.01491 2016.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics. PMLR, 2020.
Marco Barreno, Blaine Nelson, Anthony D Joseph, and J Doug Tygar. The security of machine learning. Machine Learning. 2010.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), 2018.
Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning.
Pattern Recognition, 2018.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines.
arXiv preprint arXiv: /20.//006666633 2012.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. arXiv preprint arXiv:2012.07800, 2020.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint 1721v:17 2017.
Valeriia Cherepanova, Micah Goldblum, Harrison Foley, Shiyuan Duan, John Dickerson, Gavin Taylor, and Tom Goldstein. Lowkey: leveraging adversarial attacks to protect social media users from facial recognition. arXiv preprint arXiv:2101.0799 2021.
Antonio Emanuele Cinà, Kathrin Grosse, Sebastiano Vascon, Ambra Demontis, Battista Biggio, Fabio Roli, and Marcello Pelillo. Backdoor learning curves: Explaining backdoor poisoning beyond influence functions. arXiv preprint arXiv:2106.07214 2021.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04805, 2018.
Ji Feng, Qi-Zhi Cai, and Zui-Hua Zhou. Learning to confuse: generating training time adversarial data with auto-encoder. arXiv preprint 1005v:1905.0900 2019.
Liam Fowl, Ping-yeh Chiang, Micah Goldblum, Jonas Geiping, Arpit Bansal, Wojtek Czaja, and Tom Goldstein. Preventing unauthorized use of proprietary data: Poisoning for secure dataset release. arXiv preprint arXiv: 103.022333.0,,3, 2
Liam Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojtek Czaja, and Tom Goldstein. Adversarial examples make strong poisons. arXiv preprint 10Xiv:2106. 10807, 2021b.
Jonas Geiping, Liam Fowl, W Ronny Huang, Wojeiech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. arXiv preprint ar arXiv:2009.022777 2020.
10

---

Under review as a conference paper at ICLR 2022
Robert Geirhos, Sorn-eeriik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020.
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. arXiv preprini (rXi::0012.1054:: 2020.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Isabelle Guyon. Design of experiments of the nips 2003 variable selection benchmark. In NIPS 2003 workshop on feature extraction and feature selection, 2003.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Conference Computer Vision and Pullern Recognition, 2020.
Katherine L Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training, arXiv preprint arXiv:2006. 12433, 2020.
Kashmir Hill. The secretive company that might end privacy know it. The New York Times, 2020.
Kashmir Hill and Aaron Krolik. How photos of your kids are powering surveillance technology. The New York Times, 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearnable examples: Making personal data unexploitable. International Conference on Learning Representa- tions, 2021.
W Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. Metapoison: Practical general-purpose clean-label data poisoning. arXiv preprint arXiv:2004.0022: 2020.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung. Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprint 2rXiv:2103.10427, 2021.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry, Adversarial examples are not bugs, they are features. arXiv preprint vrXivv1905.0211 2019.
Jörn-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance causes adversarial vulnerability, arXiv preprint arXiv:1811.00440 2018.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. arXiv preprint arXiv:1806.07577 2018.
Pang Wei Koh and Perey Liang. Understanding black-box predictions via influence functions. In International Conference Machine Learning, 2017.
Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses. arXiv preprint 18X 2018
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 1989.
11

---

Under review as a conference paper at ICLR 2022
Fang Liu and Ness Shroff. Data poisoning attacks stochastic bandits. In International Conference on Machine Learning. PMLR, 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018.
Luis Muñoz- González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee,
Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back trck-gradient optimization. In ACM Workshop on Artificial Intelligence and Security, 2017.
Preetum Nakkiran. A discussion of adversarial examples are not bugs, they are features': Adversarial examples are just bugs, too. Distill, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. arXiv preprint ar arXi::2010.08138, 2020.
Anh Nguyen and Anh Tran. inpee-impe warping- warping-based backdoor attack. arXiv preprint arXiv:2102. 10369, 2021.
Timothy Niven and Hung-Yu Kao. Probing neural network comprehension of natural language arguments. arXiv preprint 1rXiv:190..07355, 2019.
Vinay Uday Prabhu and Abeba Birhane. Large image datasets: A pyrrhic win for computer vision? arXiv preprint arXiv:2006. 16923, 2020.
Evani Radiya-Diss Dixit and Florian Tramèr. Data poisoning won't save you from facial recognition. arXiv preprint 2rXiv:2106.4440 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, 2015.
Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez. Right for the right reasons: Training differentiable models by constraining their explanations. arXiv preprint arXiv: 1703.03777 2017.
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Avi Schwarzschild. Micah Goldblum, Arjun Gupta, John P Dickerson, and Tom Goldstein. Just how toxic is data poisoning? unified benchmark for backdoor and data poisoning attacks. In International Conference on Machine Learning, 2021.
Ali Shafahi. W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clcan-label poisoning attacks on neural networks. arXiv preprint arXiv: 1804. 00792, 2018
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. arXiv preprint arXiv:2006.07710, 2020.
Shawn Shan, Emily Wenger, Jiayun Zhang, Huiying Li, Haitao Zheng, and Ben Y Zhao. Fawkes: Protecting privacy against unauthorized deep learning models. In USENIX Security Symposium, 2020.
Juncheng Shen, Xiaolei Zhu, and De Ma. Tensorelog: An imperceptible poisoning attack on deep neural network applications. IEEE Access, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
12

---

Under review as a conference paper at ICLR 2022
Lue Tao, Lei Feng, Jinfeng Yi, Sheng-Jun Huang, and Songean Chen. Provable defense against delusive poisoning. arXiv preprint arXiv:2102.04716. 2021.
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Clean-label backdoor attacks. 2018.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2008
Chulin Xie, Keli Huang, Pin-Yu Yu Chen, and Li. Dba: Distributed backdoor attacks against federated learning. In International Conference Learning Representations, 2019.
Chia-Hung Yuan and Shan-Hung Wu. Neural tangent generalization attacks. In International Conference on Machine Learning. PMLR, 2021.
Chen Zhu, W Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clean-label poisoning attacks on deep neural nets. In International Conference on Machine Learning, 2019.
13

---

The computational complexity of generating synthetic perturbations is O(nd/ p²), where n is the size of dataset, d is the dimension of clean data, and p the patch size in Algorithm This complexity is mainly from introducing covariance into synthetic data (Line 6 in Algorithm 2). We note that the complexity of generating synthetic perturbations significantly smaller than that of recent attacks. The complexity of running the algorithms in recent attacks is OTTLn(dw + w²)), where T is the number of iterations generating the poisons, the network depth, and the network width (the Under review as a conference paper at ICLR 2022
A GENERATE SYNTHETIC DATA FROM NORMAL DISTRIBUTIONS
We adopt algorithm in Guyon 2003) to generate synthetic data from normal distributions. Algo- rithm shows the pseudocode. The algorithm places data of different classes at different vertices of hypereube. It also introduce different random covariance each class.
Algorithm 2: Generating Synthetic Data From Normal Distributions
1: Input: number of classes k, number of examples in each class {n,}i= data dimension d.
2: Create d-dimensional hypercube.
3: for 1 to & do
4: Generate ( Ra.xd where each of D is sampled from \O, Iaxa)-
5: // Introduce random covariance among columns.
6: Uniformly sample the elements of A Raxd from [-1,
7: Compute ( D A.
8:
Randomly choose unused vertex and let el E IRd be its coordinates.
9: // Move the sampled points to the chosen vertex.
Compute ( (() c! i.e., c(9) is added each row of (
Assign the rows of D with label
12:
end for
cost of one forward and backward process). This complexity is strictly worse than that of generating synthetic perturbations.
B ADDITIONAL T-SNE PLOTS
Here we plot the t-SNEs of two other attacks Table 1 i.e., adversarial examples (Fowl et al.) 2021b) and NTGA Yuan & Wu 202I). We use their official implementations to generate the perturbations (see Appendix D for details). The t-SNEs are plotted in Figure 6 The perturbations for the same class
are well clustered. This observation is similar to that from Figure 2
Adversarial Examples
NTGA
30
40
20
20
-10-
-00
-20
Class 0
-30
Class
-60
Class 2
Class 2
-50
-40
-20
20
40
-6
-4
-2
Figure 6: T-SNEs of targeted adversarial examples (Fowl et al.) 2021b) and NTGA Yuan & Wu 2021I. Perturbations from the same class are well clustered. Notably, many embeddings of NTGA are overlapped, suggesting that it uses very similar perturbations for some examples.
14

---

Under review as a conference paper at ICLR 2022
Table 5: Training accuracy (in %) of simple models on the perturbations of different attacks. The perturbations are generated with Tanh-DNNs.
Algorithm
Linear Model
Two-layer NN
Error-minimizing (Huang et 2021
100.0
100.0
Adversarial Examples (Untargeted) (Fowl el a.
202
92.7
100.0
Adversarial Examples (Targeted) Fowl et al.
20216
100.0
100.0
Table 6: Test accuracy (in %) with different poisoning percentages p. Training with the poisoned subset does not improve the test accuracy much compared to training with clean data only.
Method
p -90%
-0%
=50%
p =20%
Clean Data Only (100%-p)
82.6
86.5
92.4
93.9
Aaror-minimizingg (Huang al. 2021
85.2
86.8
92.8
94.1
Adversarial Examples (Fowl et al. 2021b)
85.3
88.2
92.2
93.7
Synthetic Perturbations
85.7
86.3
92.9
94.0
C LINEAR SEPARABILITY IS NOT STEMMING FROM RELU
Existing indiscriminate poisoning attacks against deep neural networks all use ReLU activation functions in their crafting models. well known the community that ReLU-DNN learns piecewise linear function in input space (Arora et al. 2016). To verify that whether the linear separability of perturbations is stemming from the property of ReLU, we replace the ReLU layers with Tanh layers in the crafting models of error- error-minimizng noises (Huang et al. (202I) and adversarial examples (Fowl et al. 2021b). We fit the new perturbations with the same simple models as those in Section The results are presented in Table The new perturbations are still almost linearly separable: linear models achieve more than 90% training accuracy and two-layer neural networks achieve 100% training accuracy. This suggests the linear separability is not stemming from the property ReLU.
D POISONING DIFFERENT PERCENTAGES OF THE TRAINING DATA
In this section, we show synthetic perturbations are as effective as advanced attacks when only a given percentage of the training data is poisoned. We follow the method in Huang al.) (202T) Fowl et al. (20215). For each poisoning percentage, we train two models. One model uses both the clean subset and the poisoned subset its training data and the other one only uses the clean subset. The difference between the performances of those two models represents how much information the former model gains from the poisoned data.
We test four different poisoning percentages (from 20% to 90%) on the CIFAR- 10 dataset. The experiments are run on ResNet- 18 models. We compare the performance of synthetic perturbations with adversarial examples and arror-minimzzing noises (Huang et al. 2021 Fowl et al. (0216). The results are presented in T The performance gain of using the poisoned subset is small for all three attacks. This suggests that synthetic perturbations are still effective in this setting.
E IMPLEMENTATION DETAILS OF EXPERIMENTS
Implementation details of the experiments in Section 2 We generate perturbations for the CIFAR- 10 dataset using the official implementations of DeepConfuse NTGA7 error-minimizing noise") and adversarial examples") The configuration is set to be the one that achieves the best attack performance on CIFAR-I0. Specificallyy DeepConfuse uses an 8-layer U-Net (Ronneberger et al. 2015) as the
htte:://thhbblooo/waaaa////noonn Kxamp lem
15

---

Under review as a conference paper at ICLR 2022
SVHN
CIFAR-10
CIFAR-100
100
80
Accuracy
60
Test (Algo. 1)
(Aggo.
Test (Nge. 1)
(Algo.
Test (Agg. 1)
(Algo.
Train (initial)
Train (ihitial)
Train (initia))
40
Test (initial)
Test (Initial)
Test (initial)
20
40
60
100
60
80
100
20
40
60
Epoch
Epoch
Epoch
Figure Training curves ResNet18 models trained on SVHN, CIFAR-10, 10, and CIFAR- 100 datasets. The word "initial" denotes the synthetic perturbations are generated directly using Algorithm) 2
crafting model. NTGA uses a 3-layer convolutional network. Error-minimizing noises and adversarial examples use standard ResNet- models.
The experimental setup for training the simple models are as follows. We train the simple models with standard cross-entropy loss. Before training, all perturbations are flattened into I-dimensional vectors and normalized to unit norm. The two-layer neural networks have a width of 30. All models are trained with the L-BFGS optimizer (Liu & Nocedal]
1989) for 50 steps.
Implementation details of the experiments in Section 3 We use the Stochastic Gradient Descent (SGD) optimizer with a momentum coefficient 0.9 for experiments. For all datasets, we use batchsize of 128. The learning rates of all models are set to follow the choices in the original papers (Simonyan & Zisserman 2014 He et al., 2016 Huang et al. 2017). The learning rate for ResNet and DenseNet models is 1.I. The learning rate for VGG models is 0.01. All models are trained for 100 epochs. The learning rate is divided by 10 epoch 50 and 75.
Implementation details of the experiments in Section 4 To fine-tune the pre are-trained model on the CIFAR-10 dataset, we first use bicubic upsampling to resize the image into 224 224. For the first approach, use the same hyperparameters as SectionBJte fine-tune the full model. For the second approach, we first extract the features of all samples and train linear classifiers on the extracted features. The linear classifiers trained with the L-BFGS optimizer for 50steps.
F THE EFFECT OF THE PADDING OPERATION IN ALGORITHM
Here we explain why we duplicate each dimension of the initial data points into two-dimensional patches in Algorithm [ Intuitively, it is more convenient to directly generate synthetic perturbations that have the same dimension as the original images. We will show this straightforward approach has unstable performance when common data augmentation methods are applied.
We implement the above straightforward by directly using the output of Algorithm 2 i.c.. the dimension of synthetic data the same as the dimension of flattened images and we simply reshape the synthetic data into two-dimension. Other configurations The models are trained with standard augmentation methods including random crop and flipping. We compare this straightforward approach with the that further processes the outputs with Algorithm
the same those in Section 3
[ The training curves of the target models are plotted in Figure 7 When using the straightforward approach, the test accuracy sometimes increases a high point which violates the requirement of indiscriminate poisoning attacks.
G INCREASING THE STRENGTH OF PERTURBATIONS
In Section we only use a single level of perturbation. Here we increase the perturbation strength and examine whether the proposed defense remains effective. Specifically, try 16. 32 for existing attacks and é' 12, 24 for synthetic perturbations. Other settings are the same as those in Section 4 The results presented in Table Even when strong perturbations applied (e 32
16

---

Under review as a conference paper at ICLR 2022
Table 7: Test accuracy (in %) when the proposed defense is applied. We fix the pre-trained weights and only train the linear readout layer.
Algorithm
16
(c'
12)
c= 32 (c' 24)
DeepConfuse (Feng et al. 2019
92.4
82.4
NTGA Yuuan & Wu
2021
92.9
84.4
Error-minimizing (Huang et al.
2021
92.2
87.9
Error-maximizing (Fowl et al. Synthetic Perturbations
2021b
82.6
76.6
92.0
87.4
and ='= 24), the proposed defense still achieves decent test accuracy. We note that increasing the perturbations strength will make the perturbations visible. This will ultimately violate the designing goal of indiscriminate poisoning attacks in this paper that is to reduce the test accuracy without hurting the normal data utility.
17


Reviews:
[{"review_id": "vLMGokTokkg", "reviewer": "Reviewer_baCa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The finding that existing attacks produce linearly-separable datasets is interesting and somewhat counters the standard explanation offered as to why these attacks are effective.\n\nAt the same time, the rest of the paper's contributions are rather minor:\n- The finding that shortcuts are sufficient for poisoning is relatively well-known. There has already been work that shows that if the training set contains a class-indicative, easy-to-learn pattern the classifier will disproportionately rely on that (e.g., [DecoyMNIST](https://arxiv.org/abs/1703.03717) or [https://arxiv.org/abs/1811.00401](https://arxiv.org/abs/1811.00401) in the context of interpretability). In fact, Tao et al. 2021 (cited in the manuscript) propose a very similar yet simpler attack (P5 in their paper) with virtually the same justification.\n- As the authors acknowledge, the proposed defense is only demonstrated to be effective when pre-training is performed on clean data and the adversary has no knowledge of the defense in place. However, these are relatively strong assumptions. An adversary could devise methods to circumvent the defense were this not the case.", "summary_of_the_paper": "The authors study data poisoning attacks that aim to degrade the overall (test) accuracy of a model by adding small perturbations to all training samples. They find that existing attacks introduce features into the data that make the training dataset easy to fit without learning the underlying concept (and thus not in a way that would generalize to natural test examples). They then show that this principle is sufficient to create poisoning attacks by proposing a simple method relying on random, class-specific pattern. Finally, the authors propose defending against these such poisoning attacks by relying on a pre-trained model and only fine-tuning the last (linear) layer.", "summary_of_the_review": "While the work does contain some interesting findings, the overall contribution is rather minor when taking into account prior work. Thus, I do not find it suitable for the general NeurIPS audience.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "sd_cBKERyoH", "reviewer": "Reviewer_sUUe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Better understanding the root causes behind the success of indiscriminate poisoning attacks would be a valuable contribution. Unfortunately, this work in its current form presents different issues.\n\nAmbiguous usage of the nomenclature. The authors refer to the poisoning attacks that maximize the error on the test dataset as \"indiscriminate poisoning attacks.\" However, this may be confusing, and it may be worth clarifying the terms. Poisoning attacks that aim to maximize the test error are usually categorized as availability attacks. The term \"indiscriminate\" in Barreno et al. (2010) has a different meaning, i.e., that the attacker is not targeting a specific user/test sample (but rather any of them – think, e.g., to phishing vs spearphishing). In more recent papers, however, multiclass classification has been considered, and targeted/indiscriminate has been used to refer to the class which the attacker aims the test sample to be assigned to. I think here it would be good to try to disambiguate the terms – see, e.g., Munoz-Gonzalez et al., 2017.\n\nSome claims are not supported by experimental evidence. The authors have made different claims, but unfortunately, the following are not supported experimental evidence.\n\n1) The authors claim the poisoning samples generated adding imperceptible perturbations to the training data are well separable (accordingly to the class of the original sample) in input space with hyperplanes. However, this might be due to the type of classifier considered. For example, does this hold when the attacker targets a Support Vector Machine with an RBF kernel? The authors should consider different types of classifiers and not only deep neural networks with ReLU activation functions (ReLU-DNNs indeed learn a piecewise linear function in input space).\n\n2) The authors claim that linear separability is necessary for indiscriminate poisoning attacks to succeed. However, this claim is not supported by evidence. The authors have only shown that the existing poisoning attack algorithms produce linearly separable perturbations when run against a deep neural network. Nevertheless, this does not exclude that poisoning samples that are not linearly separable remain effective. It may be thus worth investigating the possibility of crafting “adaptive” poisoning attacks that remain undetected when considering separating hyperplanes. Another issue may be related to the high dimensionality of the input space. Is the fact that such attacks are linearly separable due to the high dimensionality of the data, other than the given model?\n\n3) The authors claim the proposed attack is more efficient than the state-of-the-art ones. They should add a comparison of their computational complexity. \n\n4) The authors claim that using a pre-trained model is sufficient to defend against clean-label poisoning attacks. However, their experiments have considered a single level of perturbation that the attacker can inject. To support their claim, they should instead assess the attack effectiveness for an increasing level of perturbation the attacker can add. The same observation has also been done in this work (https://arxiv.org/abs/2106.07214) for backdoor attacks, so it may be worth clarifying that this finding is not completely novel.\n\nOverall, the authors should present a more extensive experimental analysis to support their claims.\n\nThe experimental setting is not sufficiently described. For some experiments, the experimental details are not provided. For example, against which deep neural network have the authors crafted the poisoning samples to generate Table 1? \n\nConfusing manuscript organization. The description of the experimental setup is fragmented into different sections.\n\n", "summary_of_the_paper": "The authors empirically investigate the underlying reasons behind the effectiveness of indiscriminate poisoning attacks. According to the authors' claims, their analysis shows that the poisoning samples generated adding imperceptible perturbations to the training data are well separable (accordingly to the class of the original sample) in input space with hyperplanes. This discovery enables the authors to propose a more efficient attack, and to understand that using a pre-trained neural network as a feature extractor can enable defending such poisoning attacks.", "summary_of_the_review": "The paper tries to clarify an issue, but more insights need to be developed. For instance, whether such linear separability is induced by the kind of model, input dimensionality, and if it generalizes beyond poisoning attacks – similar evidence are probably existing for backdoor poisoning and adversarial examples too.\n\nI’m also quite concerned about the threat model, as poisoning large fractions of points (more than 20%) is not quite realistic in many practical scenarios (I don’t think it’s realistic to poison up to 50% of the dataset). This should be clarified in the paper, and more details on the practicality of the threat model should be discussed or acknowledged as specific limitations of this work (and other papers that use similar setups).\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CVIP7g397Pg", "reviewer": "Reviewer_nWuM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I lean towards recommending accepting this paper.\n\n---\nStrengths:\n\n- The understanding of this paper is interesting. The understanding applies to a wide range of existing works in this area. The indiscriminate data poisoning attacks are a form of linear separable features. The experimental results (t-SNE and shallow models) supports this claim. \n- Based on this understanding, the proposed synthesized linear separable data is technically sound and shows a effective performance. \n- This paper's presentation is easy to follow and most related works have been discussed.\n\n---\n\nQuestion/weakness:\n- Are other attacks evaluated in Table 1/2 also showing linear separable on t-SNE? \n- The experiments baseline is not consistent. For tables 2 and 4, why not include Alignment/TensorClog in table 4?\n- Legends and captions in Figure 4 are not informative enough. What does test (poisoned) mean? Is it training on poisoned training data evaluated against clean test data? Or is it the poisoned test data? \n- Need clarifications in the shortcut features. At the beginning of section 2.4, you describe shortcut learning as spurious features that are useful for training but do not generalize to test data. I believe in Geirhos et al., 2020, it is described as do not generalize to the OOD test set, but these features do generalize to the IID test set. I understand the connection of indiscriminate poisoning attacks, but some evidence/figures would be very helpful to establish this connection. \n\n", "summary_of_the_paper": "This paper demonstrated existing indiscriminate data poisoning attacks are a form of linear separable features. They exploit the shortcut learning of deep learning. Based on the understanding, this paper also shows a simple synthetic data can be used to achieve the same effect as existing indiscriminate data poisoning attacks. ", "summary_of_the_review": "This paper made an interesting observation on the linear separability of a wide range of indiscriminate data poisoning attacks. This reveals the reason why such attacks can succuss. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0En-HrKGmdM", "reviewer": "Reviewer_ngSk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. This paper shows that linear separability is indeed an underlying principle across many delusive attacks. I thought this finding is very interesting and would help the community to better understand the mechanisms behind the attacks.\n2. The proposed synthetic noise is very effective, which makes the finding about the linear separability more solid.\n3. The connection between linear separability and shortcut learning is thought-provoking. This shortcoming about shortcuts may also be related to the simplicity bias of neural networks, as observed in [3].\n\nWeaknesses:\n\n1. Some citations in the fourth sentence in the introduction section are not appropriate. Specifically, Fawkes (Shan et al., 2020) and LowKey (Cherepanova et al., 2021) are actually targeted attacks (i.e., causing test-time error on some given samples), rather than indiscriminate attacks (i.e., causing test-time error on all unseen samples) in the context.\n2. The criticism from Radiya-Dixit & Tramer (2021) may fail to challenge the effectiveness of the indiscriminate attacks considered in this paper, since their criticism was directed only at those targeted attacks such as Fawkes and LowKey.\n3. The first sentence in the abstract describing the indiscriminate attacks is somewhat imprecise. Specifically, not all indiscriminate data poisoning attacks work by adding imperceptible perturbations. Actually, another type of indiscriminate attacks can increase test error by injecting a small amount of arbitrarily crafted examples into the training set (e.g., [1]). Both two types of poisoning attacks belong to indiscriminate attacks. Therefore, it would be better to call the former \"clean-label indiscriminate attacks\" or, simply, \"delusive attacks\" as in [2-3].\n4. The last sentence on page 1 is imprecise. Specifically, it was [4] that pointed out for the first time that \"adversarial examples can serve as an indiscriminate poisoning attack\". Later, [3] and [5] both experimented with this idea. The difference is that [3] only verified targeted adversarial examples, while [5] further tested untargeted adversarial examples.\n5. In Table 1, it is suggested to report the results of the linear separability for both targeted and untargeted adversarial examples [3-5]. *The content in Section 2.1.3 may also need to be modified to precisely reflect this subtle difference.*\n6. In Table 4, it is suggested to further report the performance of adversarial training with small $\\epsilon$ as baselines, since it has been found that adversarial training with smaller $\\epsilon$ may have better defense performance [3].\n\n[1] Biggio et al., Poisoning Attacks against Support Vector Machines, ICML 2012  \n[2] Newsome et al., Paragraph: Thwarting Signature Learning by Training Maliciously, RAID 2006  \n[3] Tao et al., Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training, NeurIPS 2021  \n[4] Nakkiran, \"A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Adversarial Examples are Just Bugs, Too\", Distill, 2019.  \n[5] Fowl et al., Adversarial Examples Make Strong Poisons, NeurIPS 2021  \n[6] Huang et al., Unlearnable Examples: Making Personal Data Unexploitable, ICLR 2021", "summary_of_the_paper": "This paper unveils an intriguing \"linear separable\" property of various existing indiscriminate poisoning attacks, and then designs a new attack using linearly separable features. Experiments show that the proposed attack is fast and efficient. Additionally, pre-trained feature extractors are suggested as a powerful defense strategy.", "summary_of_the_review": "Overall, I feel that the finding of linear separability would be of great interest to the community, and I consider the proposed synthetic noise as a supporting evidence for the main claim about the underlying principle and as a direct application of the linear separability. Therefore, I am leaning to accept. I would like to increase my score if the concerns are well addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies poisoning attacks of small perturbations to training samples. Existing attacks introduce features that allow for easily fitting the data, but do not lead to good generalization. They use this principle to generate attacks based on random class-specific patterns. They finally propose defense using a pre-trained model whose last layer is fine tuned. Several criticisms of novelty of the work in comparison to prior work were raised during the reviews. At the same time the validity and effectiveness of defenses remains unsubstantiated to a large extent. Although the authors put an effort in addressing these concerns, for the most part some reviewers and myself remain critical of the contributions and novelty of this work.
Decision: Reject