Poisoning Attacks with Generative Adversarial Nets
Authors: Luis Muñoz-González
Bjarne Pfitzner
Matteo Russo
Javier Carnerero-Cano
Emil C. Lupu
Venue:  (2020)
Keywords: data poisoning
adversarial machine learning
generative adversarial nets

Abstract: Machine learning algorithms are vulnerable to poisoning attacks: An adversary can inject malicious points in the training dataset to influence the learning process and degrade the algorithm's performance. Optimal poisoning attacks have already been proposed to evaluate worst-case scenarios, modelling attacks as a bi-level optimization problem. Solving these problems is computationally demanding and has limited applicability for some models such as deep networks. In this paper we introduce a novel generative model to craft systematic poisoning attacks against machine learning classifiers generating adversarial training examples, i.e. samples that look like genuine data points but that degrade the classifier's accuracy when used for training. We propose a Generative Adversarial Net with three components: generator, discriminator, and the target classifier. This approach allows us to model naturally the detectability constrains that can be expected in realistic attacks and to identify the regions of the underlying data distribution that can be more vulnerable to data poisoning. Our experimental evaluation shows the effectiveness of our attack to compromise machine learning classifiers, including deep networks.

Under review as a conference paper at ICLR 2020
POISONING ATTACKS WITH GENERATIVE ADVERSARIAL NETS
Anonymous authors Paper under double-blind review
ABSTRACT
Machine learning algorithms are vulnerable to poisoning attacks: An adversary can inject malicious points in the training dataset to influence the learning process and degrade the algorithm's 's performance. Optimal poisoning attacks have already been proposed to evaluate worst-case scenarios, modelling attacks as bi-level optimization problem. Solving these problems computationally demanding and has limited applicability for some models such as deep networks. In this paper we introduce a novel generative model to craft systematic poisoning attacks against machine learning classifiers generating adversarial training examples, i.e. sam- ples that look like genuine data points but that degrade the classifier's accuracy when used for training. We propose Generative Adversarial Net with three com- ponents: generator, discriminator, and the target classifier. This approach allows to model naturally the detectability constrains that can be expected in realistic attacks and to identify the regions of the underlying data distribution that can be more vulnerable data poisoning, Our experimental evaluation shows the effee- tiveness of our attack to compromise deep networks, bypassing different type of existing defences against data poisoning.
INTRODUCTION
Despite the advancements and the benefits of machine learning, has been shown that learning algorithms are vulnerable and can be the target of attackers, who can gain significant advantage by exploiting these vulncrabilities (Huang et al. 2011). At training time, learning algorithms are vulnerable to poisoning attacks, where small fractions of malicious points injected the training set can subvert the learning process and degrade the performance of the system in an indiscriminate or targeted way. Data poisoning is one of the most relevant and emerging security threats in applica- tions that rely upon the collection of large amounts of data in the wild (Joseph et al., 2013). Some applications rely on the data from users' feedback or untrusted sources of information that often collude towards the same malicious goal. For example, in IoT environments sensors can be compro- mised and adversaries can craft coordinated attacks manipulating the measurements of neighbour sensors evading detection (Illiano et al. (016)). In many applications curation of the whole training dataset is not possible, exposing machine learning systems to poisoning attacks.
In the research literature optimal poisoning attack strategies have been proposed against different machine learning algorithms (Biggio et al., 2012 Mei & Zhu 2015; Muñoz-González et al.) 2017] Jagielski et al.) (018)), allowing assess their performance in worst-case scenarios. These attacks can be modelled as a bi-level optimization problem, where the outer objective represents the at- tacker's goal and the inner objective corresponds to the training of the learning algorithm with the poisoned dataset. Solving these bi-level optimization problems is challenging and can be computa- tionally demanding. especially for generating poisoning points at scale. This limits its applicability against some learning algorithms such as deep networks or where the training set is large. In many cases, if no detectability constraints are considered, the poisoning points generated are outliers that can be removed with data filtering (Paudice et al. 2018 80). Furthermore, such attacks are not realistic as real attackers would aim to remain undetected in order to be able to continue subverting the sys- tem in the future. As shown in (Koh et al., 2018), detectability constraints for these optimal attack strategies can be modelled, however they further increase the complexity of the attack, limiting even more the application of these techniques.

---

Under review as a conference paper at ICLR 2020
Taking entirely different and novel approach, in this paper we propose a poisoning attack strategy against machine learning classifiers with Generative Adversarial Nets (GANs) (Goodfellow et al. 2014). This allows us to craft poisoning points in more systematic way, looking for regions of the data distribution where the poisoning points are more influential and, at the same time, difficult to detect. Our proposed scheme, pGAN, consists three components: generator, discriminator and larget classifier. The generator aims to generate poisoning points that maximize the error of the tar- get classifier but minimize the discriminator's ability to distinguish them from genuine data points. The classifier aims to minimize some loss function evaluated on training dataset that contains a fraction of poisoning points. As in standard GAN, the problem can be formulated as a minimax game. pGAN allows to systematically generate adversarial training examples (Koh & Liang] (017), which are similar to genuine data points but that can degrade the performance of the system when used for training. The use of a generative model allows us to produce poisoning points at scale, enabling poisoning attacks against learning algorithms where the number of training points is large or in situations where optimal attack strategies with bi-level optimization are intractable or difficult to compute, as it can be the case for deep networks. Additionally, our proposed model also includes mechanism to control the detectability of the generated poisoning points. For this, the generator maximizes a convex combination of the losses for the discriminator and the classifier evaluated on the poisoning data points. Our model allows to control the aggressiveness of the attack through parameter that controls the weighted sum of the two losses. This induces a trade-off between ef- fectiveness and detectability of the attack. In this way. PGAN can be applied for systematic testing of machine learning classifiers at different risk levels. Our experimental evaluation in synthetic and real datasets shows that PGAN is capable of compromising different machine learning classifiers bypassing different defence mechanisms, including outlier detection Paudice et al.) (2018a), Sever Diakonikolas et al. 2019). PCA-based defences Rubinstein et al. (2009) and label sanitization Pau- dice et al. (2018b). We analyse the trade-off between detectability and effectiveness of the attack: Too conservative strategies will have a reduced impact on the target classifier but, if the attack is too aggressive, most poisoning points can be detected as outliers.
2 RELATED WORK
The first practical poisoning attacks were proposed in the context of spam filtering and anomaly detection (Nelson al. 2008 Kloft & Laskov (012). But these attacks do not easily generalize to different learning algorithms. [iggio et (2012) presented a more systematic approach, modelling optimal poisoning attacks against SVMs for binary classification as a bi-level optimization problem, conditions in the inner problem. A sim- ilar approach proposed by Xiao et al. (2015) for poisoning embedded feature selection methods, including LASSO, ridge regression, and elastic net. [Mei & Zhu (2015) proposed more general framework to model and solve optimal poisoning attacks for convex classifiers. They exploit the implicit function theorem to compute the gradients required to solve the corresponding bi-level op- timization problem. Munoz--Gnzálee et al. (2017) proposed back- greddgradeent optimization to estimate the gradients required to solve bi-level optimization problems for optimal poisoning attacks against multi-class classifiers. This approach allows to attack a broader range of learning algorithms and re- duces the computational complexity with respect to previous works. However, all these techniques are limited to compromise deep networks trained with a large number of training points, where many poisoning points are required even to compromise a small fraction of the training dataset. Previous attacks did not model explicitly appropriate detectability constraints. Thus, the resulting poisoning points can be far from the genuine data distribution and can be easily identified as outliers (Paudice et al. 20 2018a; Steinhardt et al.] 2017; Paudice et al., 2018b). Recently, Koh et al. (2018) showed that is still possible craft attacks capable of bypassing outlier-detection based defences with an iterative constrained bi-level optimization problem, where, at each iteration, the constraints change according to the current solution of the level problem. However, the high computational complexity of this attack limits its practical application in many scenarios.
which can be solved by exploiting the Karush-Kuhn-Tucee
Koh & Liang (017)) proposed a different approach to craft targeted attacks against deep networks by exploiting influence functions. This approach allows to create adversarial training examples by learning small perturbations that, when added to some specific genuine training points, change the predictions for a target set of test points. Shafahi et al. (2018) showed that it is possible to perform targeted attacks when the adversary is not in control of the labels for the poisoning points. (Ynng et al. (0077) introduced poisoning attack with generative models using autoencoders to generate the
2

---

Under review as a conference paper at ICLR 2020
malicious points. Although this method is more scalable than attacks based on bi-level optimization, the authors do not provide a mechanism to control the detectability of the poisoning points.
3 POISONING ATTACKS WITH GENERATIVE ADVERSARIAL NETS
Our model, pGAN, is a GAN-based model with three components (generator, discriminator and target classifier) to generate systematically adversarial training examples. First, shortly describe the considered model for the attacker. Then, we introduce the formulation of pGAN and, finally, we provide some practical considerations for the implementation of pGAN.
3.1 ATTACKER'S S MODEL.
The attacker's knowledge of the targeted system depends different aspects: the learning algo- rithm, the objective function optimized, the feature set or the training data. In our case we consider perfect knowledge attacks, where assume the attacker knows everything about the target sys- tem: the training data, the feature set, the loss function and the machine learning model used by the victim. Although unrealistic in most practical scenarios, this assumption allows us to perform worst-case analysis of the performance of the system under attack. However, our proposed attack strategy also supports limited knowledge, exploiting the transferability property of poisoning attacks (Muzoz-González et al., (017). For the attacker's 's capabilities, we consider here a causative attack (Barreno et al.J 2006; 2010), where the attacker can manipulate fraction of the training data to influence the learning algorithm. We assume that the attacker can manipulate all the features to craft the poisoning points as long as the resulting points are within the feasible domain for the distribution of genuine training points. Finally, we also assume that the attacker can also control the labels of the injected poisoning points.
3.2 PGAN
In multi-class classification task, let X' Rdd be the d-dimensional feature space, where data points are drawn from distribution (x) and y is the space of class labels. The learning algorithm, C, aims learn the mapping f X' + y by minimizing a loss function, Lc, evaluated on set of training points Str The objective of the attacker is to introduce fraction, À E (0, 1), of malicious points S,y to maximize Le when evaluated on the poisoned training set.
The Generator, G, aims to generate poisoning points by learning a data distribution that is effective at increasing the error of the target classifier, but that is also close to the distribution of genuine data points, i.e. the generated poisoning points are similar to honest data points evade detection. Thus, G receives some noise z ~ (=(z)))) as input and implicitly defines a distribution of poisoning points, (x)), which is the distribution of the samples ( Y p)) conditioned on Y, C Y the set of target class labels for the attacker. The Discriminator, D, aims to distinguish between honest training data and the generated poisoning points. It estimates D D(x Yp,, the probability that came from the genuine data distribution Ps rather than Pp- As in G, the samples used in the discriminator are conditioned on the set of labels Y, The Classifier, In perfect knowledge attacks C can have the same structure as the actual target classifier. For black- box attacks we can exploit attack transferability, and then, use C as surrogate model that can be somewhat similar to the actual (unknown) classifier. During the training of PGAN, C is fed honest and poisoning training points from pa and Pp respectively, where the fraction of poisoning points is controlled by a parameter X E [0, 1).
is representative for the attacked algorithm.
In contrast to traditional GAN schemes, 9 PGAN plays a game against both and This can also be formalized Similar to conditional GANs (Mirza & Osindero 2014), the objective function for D (which also depends G) can be written
minimax game where the maximization problem involves both D and C.
V(D,9) = Be [log(1 D(g(>)Y,)))) + (xx (oxp
(1)
The objective function for C' is given by:
W(C.9) -0 -AB
A) Exxps [c(())
(2)
3

---

Under review as a conference paper at ICLR 2020
where À is the fraction of poisoning points introduced in the training dataset and Le is the loss function used to train C. Note that the poisoning points in (2) belong to a subset of poisoning class labels Y p. whereas the genuine points used to train the classifier are from all the classes. The objective in (2) is just the negative loss used to train C evaluated on a mixture of honest and poisoning points (from the set of classes in y) controlled by X.
Given and pGAN can then be formulated as the following minimax problem:
min max a V(D, 9) + (1 a) a) W(C,g) D.C
(3)
with a E [0. 1]. In this case, the maximization problem can be seen as multi-objective optimization problem to learn the parameters of both the classifier and the discriminator. Whereas for C and the objectives are decoupled, the generator optimizes convex combination of the two objectives in (T) and (2). The parameter a controls the importance of each of the two objective functions towards the global goal. So, for high values of a, the attack points will prioritize evading detection, rendering attacks with (possibly) a reduced effectiveness. Note that for a = we have the same minimax game in a standard conditional GAN (Mirza & Osindero 2014). On the other hand, low values of a will result in attacks with higher impact in the classifier's s performance. However the generated poisoning points will be more detectable by outlier detection systems. For a O. pGAN does not consider any detectability constraint and the generated poisoning points are only constrained by the output activation functions in the G. In this case pGAN can serve as suboptimal approximation of the optimal attack strategies in (Biggio et al. [012 Mei & 7hu 2015 Muñoz-GGnnzález et al. 2017) where no detectability constraints are imposed.
Similar to (Goodfellow et al. 2014) we train PGAN following a coordinated gradient-based strategy to solve the minimax problem in 33. We sequentially update the parameters of the three components using mini-batch stochastic gradient descent/ascent. For the generator and the discriminator data points are sampled from the conditional distribution on the subset of poisoning labels Y,. For the classifier, honest data points are sampled from the data distribution including all the classes. A different number of iterations can be considered for updating the parameters of the three blocks. The details of the training algorithm are provided in Appendix A.
3.3 PRACTICAL CONSIDERATIONS
The formulation of PGAN in (3) allows to perform both error-generic and error-specific poisoning attacks (Muñoz-González et al. 2017). which aim increase the error of the classifier in an indis- criminate or specific way. However, the nature of these errors can be limited by Y, p. i.c. the classes for which the attacker can inject poisoning points. To generate targeted attacks or to produce more specific types of errors in the system we need to use surrogate model for the target classifier in PGAN, including only the classes or samples considered in the attacker's goal. For example, if the attacker wants to inject poisoning points labelled as to increase the classification error for class j. we can use binary classifier in PGAN considering only classes and j. where the generator aims to produce samples from class i. As in other GAN schemes, pGAN can also be difficult to train and can be prone to mode collapse. To mitigate these problems, we used in our experiments some of the standard techniques proposed improve GANs training, such as dropout or batch-normalizztiion (Salimans et al., 2016). We also applied one-side label smoothing, not only for the labels in the dis- criminator but also for the labels of the genuine points in the classifier. As suggested by Goodfellow
eet al. (2015), to avoid small gradients for G from the discriminator's s loss function (1). especially in early stages where the quality of the samples produced by G is poor, we train g to maximize log( (()(())))))) rather than minimizing log(1 - D(G(2) p))).
In contrast to standard GANs, PGAN the learned distribution of poisoning points pp is expected to be different from the distribution of genuine points P.. Thus, the accuracy of the discriminator in PGAN will always be greater that 0.5. Then, the stopping criteria for training pGAN cannot be based on the discriminator's 's accuracy. We need to find saddle point where the objectives in (I) and ( are maximized for D and G respectively (i.e. pGAN finds local maxima) and the the combined objective in 33) is minimized w.r.t. G (i.e. PGAN finds local minimum). Finally, the value of X plays an important role in the training of pGAN. If A is small, the gradients for g from the classifier's loss in (2) can be very small compared to the gradients from the discriminator's 's loss in ( Thus, the generator focuses more on evading detection by the discriminator rather than increasing the error of the target classifier, resulting in blunt attacks. Then, even if the expected fraction of poisoning
4

---

Under review as a conference paper at ICLR 2020
Figure 1: Synthetic experiment: Distribution of genuine (green and blue dots) and poisoning (red dots) data points for different values of a. The poisoning points are labelled as green.
points to be injected in the target system is small, larger values of À are preferred to generate more successful poisoning attacks. In our experiments in Sect. 4 we analyse the effectiveness of the attack as a function of X.
4 EXPERIMENTS
4.1 SYNTHETIC EXPERIMENT
To illustrate how PGAN works we first performed synthetic experiment with binary classification problem, generating two bivariate Gaussian distributions that slightly overlap. We trained pGAN for different values of with 500 training points from each Gaussian distribution. We targeted logistic regression classifier with À 0.8. In Fig. T we show the distribution of poisoning (red dots) and genuine (green and blue dots) data points. The poisoning points are labelled as the green data points. Thus, G aims to generate malicious points, similar to the green ones (i.e. D aims to discriminate between red and green data points). For a = we have the same result as in a standard GAN, so that the distribution of red points matches the distribution of the green ones. But, as decrease the value of a, the distribution of red points shifts towards the region where both green and blue distributions overlap. We can observe that for 0.2 the poisoning points are still close to genuine green points, i.e. we cannot consider the red points as outliers in most cases. For a 0 the generator does not have detectability constraints, focusing only on increasing the eтгоr of the classifier. It is interesting to observe that, in this case, pGAN does not produce points interpolating the distribution of the two genuine classes, but the distribution learned by the generator far from the region where the distributions of the blue and green points overlap This suggests that for a * 0 pGAN is not just producing a simple interpolation between the two classes, but G looks for regions close to the decision boundary where the classifier is weaker. The complete details of the experiment and the effect on the decision boundary after injecting the poisoning points can be found in Appendix B.
4.2 ATTACK EFFECTIVENESS
We performed our experimental evaluation on MNIST (LeCun al. 1998). Fashion-MNIST (FM- NIST) (Xiao et al. 2017), and CIFAR- Koizhieevskkkk (2009) datasets, using Deep Neural Networks (DNNs) for MNIST and FMNIST and Convolutional Neural Networks (CNNs) for CIFAR. All details about the datasets used and the experimental settings in our experiments are described in Ap- pendix C. To test the effectiveness of pGAN to generate steulthy poisoning attacks we applied the defence strategy proposed by Paudice et al. (20 Sap: We assumed that the defender has fraction of trusted data points that can be used to train one outlier detector for each class in the classification problem. Thus, we pre- filter the (genuine and malicious) training data points with these outlier de- tectors before training. As in (Paudice et al.] 2018a) we used the distance- based anomaly detector proposed by Wu & Jermaine (006)) which attacks (Biggio et al. 2012 Muñoz-Gonzálezz et al. (017). The outlierness score is computed based on the euclidean distance between the tested data point and its k-nearest neighbours from subset of s points, which are sampled without replacement from the set of points used to train the outlier detector. In our experiments we used the same values proposed in (Paudice al. 2018a): k for the number of neighbours and s = 20 for the number of training points to be sampled. We set the threshold of the outlier detector so that the a-percentile is 0.95. The a-percentile controls the fraction of genuine points that is expected to be retained after applying the outlier detector (i.e. 95%
proven to be effective against optimal poisoning
Note that the result would he significantly different the target classifier were non-linear.
5

---

Under review as a conference paper at ICLR 2020
MNIST
FMNIST
0-0
7.5-
7.0
Error
6.5
6.0
15
20
30
10
20
30
35
% Polsoning Points
%
Poisoning Points
Figure 2: Test classification error (%) function of the percentage of poisoning points using pGAN with different values of a for MNIST (left) and FMNIST (right).
in our case). To provide a better understanding of the behaviour of PGAN we first trained and tested our attack targeting binary classifiers. For this, in MNIST selected digits 3 and 5, for FMNIST we picked the classes sneaker and ankle bool and, for CIFAR, automobile and truck classes. The poisoning points were labelled as 5, ankle boot and truck respectively.
First, we analysed the effectiveness of the attack as a function of a. For each dataset we trained 5 different generators for each value of a explored, [0.0, 0.1, 0.3, 0.5, 0.7, We set 0.9-Pr(Y,)), where Pr(Y,) is the prior probability of the samples from the poisoning class, Y, (i.e. digit 5, ankle boot, and truck). For testing, we used 500 (genuine) samples per class to train the outlier detectors and 500 samples per class to train a separate classifier. We evaluated the effectiveness of the attack varying the fraction of poisoning points, exploring values in the range - 40%. To preserve the ratio between classes we substitute genuine samples from the poisoning class with the malicious points generated by pGAN (rather than adding the poisoning points to the given training dataset). For each pGAN generator and for each value of the fraction of poisoning points explored, we did 10 independent runs with independent splits for the outlier detectors and the classifier training sets.
MNIST and FMNIST: In Fig. 2 we show the test classification error for MNIST and FMNIST as function of the fraction of poisoning points averaged over the 5 generators and the 10 runs for each generator. In MNIST, the attack is more effective for a 0.1, increasing the error from 2.5% when there's no attack to more than 12% when 40% of the training dataset is compromised. For bigger values of a the effect of the attack is more limited. For a 0, i.e. when no detectability constraints are considered the effect of the attack is more limited than for a = 0.1, although in this case, the points capable of bypassing the outlier detector can still produce some degradation the target system. Similarly, for FMNIST the attack with a = 0.1 produces more effective poisoning data points, although the overall effect of the attack more limited compared to MNIST. For a = 0 the attack is mitigated by the outlier detector in most cases, and has only some effectiveness for larger fractions of poisoning points. It is interesting to observe that, despite the baseline error (i.e. when there is attack) is lower for MNIST (2.5% vs 4.75% in FMNIST), it is more difficult to poison FMNIST. This suggests that the impact of the attack not only depends on the separation between the two classes but also on the topology of the classification problem. In Fig. 3 we show some of the poisoning examples generated by PGAN (with a = 0.3). For MNIST the malicious data points (labelled as 5) exhibit features from both, digits 3 and 5. In some cases, although the poisoning digits are similar to a 3, it is difficult to automatically detect these points as outliers, as many of the pixels that represent these malicious digits follow a similar pattern compared to genuine 5s, i.e. they just differ in the upper trace of the generated digits. In other cases, the malicious digits look like a that have some characteristics that make them closer to 3s. the case of FMNIST, the samples generated by PGAN (labelled as ankle boots) can be seen as an interpolation of the two classes. The malicious images look like high-top sneakers or low-top ankle boots. Thus, it is difficult to detect them as malicious points, as they clearly resemble some of the genuine ankle boots in the genuine training set. Actually, for some of the genuine images, it is difficult to identify them as a sneaker or an ankle boot. More examples for different values of a are also shown in Appendix D.
The results in Fig. leftt show that for a = 0, Le. with no detectability constraints, the attack is significantly more effective than for the other values of a explored. In this case, observed that the outlier detector can be easily bypassed in this dataset, as the number of features and the complexity of the classification tasks are higher compared to MNIST and FMNIST. This
CIFAR:

---

Under review as a conference paper at ICLR 2020
Genuire 31
PGAN
Genuine
Genuine Sneakers
PGAN
Genuine Ankle Boots
3 3 333
3 5 3
555 5 5
3 3333
3
33
5 5 S
3333
3
5 3333
5555
3 3333
3 333
5555
3333
5 3 3 3
55
Figure 3: Examples from pGAN (with a = 0.3) for MNIST (left) and FMNIST (right),
CIFAR-10
Genuine Automoblles
PGAN
Geruine Trucks
24
Error
18-
14.
12
10
25
40
%
Poisoning Points
Figure 4: (Left) Average error on CIFAR function of a. (Right) Examples of images generated
by pGAN for CIFAR (a- = 0.3).
shows some of the limitations of some existing defences to mitigate data poisoning attacks. We can observe that in this case, the test classification error increases from 12% to 24% after injecting 40% of poisoning points. The increase is more moderate for larger values of a. In Fig. rightt we show some of the examples generated by PGAN with a 0.3, which exhibit characteristics from the two classes: cars and trucks.
Outlier detection: In Fig. (centre) we show the fraction of data points pre-filtered by the outlier detectors in MNIST dataset as a function of a. We explored two values for the a-percentile (the threshold of the detectors): 0.90 and 0.95. As expected, the fraction of rejected genuine data points is, on average, 10% and 5% respectively. However, the fraction of rejected malicious points for 2 0.1 smaller than for the genuine points for the two detectors. This is because the generator pays less attention to samples that are in low density regions for the data distribution of the genuine points, and then, the generated poisoning points are conservative. For a = 0 the fraction of rejected malicious points is also not very high. This can be due to the similarity between the two classes. Then, even if the generated poisoning points, labelled as 5, look like a they are still close to the distribution of genuine 5s when targeting a non-linear classifier.
Sensitivity w.r.t X: For analysing the sensitivity of pGAN w.r.t. X, the fraction of poisoning points used for C. we performed experiment on MNIST dataset (digits 3 and 5). We set 0.2 and explored different values for X' = X/Pr(Y,,) ranging from 0.1 to 1. With the same experimental settings as before we trained generators for each value of X. We also tested the effectiveness of the attack on a separate classifier, with 10 independent runs for each generator and value of X explored. For the attacks we injected 20% of poisoning points. In Fig. (left) we show the averaged classification eтror on the test dataset as function of X' We can observe that, for small X' the effect of the attack is more limited. The reason that, when training PGAN the effect of the poisoning points on C is very reduced, and then, the gradients of (2) w.r.t. the parameters of G can be very small compared to the gradients coming from the discriminator. Then, G focuses more on optimizing the discriminator's 's objective. In this case, even for X' = the attack is still effective, just slightly decreasing the error rate compared to X' = 0.9.
Comparison with label flipping: Comparison with existing poisoning attacks in the research lit- erature is challenging: Optimal poisoning attacks as in Muñoz-Gonzálee et al. (2017) are computa- tionally very expensive for the size of the networks and datasets used in our experiments in Fig. 2 This is even worse if we consider detectability constraints as in Koo et al. (2018). . On the other side, comparing with standard label flipping results in an unfair comparison for PGAN, as label flipping
7

---

Under review as a conference paper at ICLR 2020
6.5
20-
Geoudie
Pitson E-85
Classification Error (%)
10
Clean
6.0-
Outliers removed
23-
Poison
5.5
5.0
30-
4.0
3.-
2
0.1
0.3
0.5
0.7
0.9
1.0
0.0
0.2
0.4
0.5
0.8
1.0
1000
3000
4000
5200
Number of training points
Figure 5: (Left) Average test error on MNIST as a function of X A/Pr(Y,)) (Centre) Outlier detection on MNIST as function of for a-percentiles of 0.95 and 0.90. (Right) Average test emorr on MNIST as a function of of the number of training examples for a clean and a poisoned classifier (with 20% of poisoning points).
False Negative Rate (%)
11
PGAN
- 18
PGAN Flipping
7-
10
Flipping
Flipping
2
6
12 Pabs Peat
5
15 % of Palsaning Points
10
20
30
20 %6 Polsoning Points
15
30
10
15
25
of Polsoning Points
Figure 6: Average error (left), false positive (centre) and false negative rates (right) function of the percentage of poisoning points for pGAN and label flipping attacks on MNIST.
do not consider detectability constraints. In other words, effective than PGAN when no defence is applied, but this attack clearly more detectable (Paudice eet af., 20186). To provide fairer comparison, flipping attacks with detectability constraints. Thus, we flipped the labels of training samples from the target class that are closer the source class. For this, we computed the distance of the training points from the target class to the mean of the training points of the source class. Then, we flipped the labels of the closest points, so that the malicious points should be more difficult to detect. In Fig. 6 we show the comparison of this label flipping strategy with pGAN (a 0.1) for MNIST, using the same settings as in the experiment in Fig. 2 tive than the label flipping attack and that the effect of the two attacks is different. Label flipping increases both the false positive and false negative rates of the target classifier, whereas pGAN aims only to increase the false positive rate, i.e. pGAN is producing attacker more control on the kind errors to be produced in the classifier.
expect label flipping to be more
implemented an heuristic for generating label
We can observe that pGAN is more effec-
error-specific attack, giving the
function of the number of training points: In Fig. 5 (right) show how the number training data points impact the effect of the attack. For this, we trained 5 pGAN generators with
Attack effectiveness
0.1 and tested classifiers with different number of training points ranging from 1, 000 to 10, 000 and injecting 20% of poisoning points. For each generator and value of the number of training points explored we did independent runs. We also used 500 samples per class to train the outlier detectors. The results in Fig. 5 (right) show that the difference in performance between the poisoned and the clean classifier reduces as the number of training samples increases. This is expected, as the stability of the learning algorithm increases with the number of training data points, limiting the ability of the attacker to perform indiscriminate poisoning attacks. This does not mean that learning algorithms trained with large datasets are not vulnerable to data poisoning, as attackers can still very successful at performing targeted attacks, focusing on increasing the also
error particular instances creating backdoors (Gu al. (017). In these scenarios use PGAN to generate more targeted attacks using a surrogate model for the classifier including the subset of samples that the attacker aims to misclassify.
Multi-class classification: Finally we performed an error-specific attack on MNIST using the 10 classes. In this case the objective of the attacker is increase the error of digit 3 being misclassified as 5. For this, we trained PGAN using a surrogate classifier including only digits and 5, and then, tested against multi-class classifier trained on 10, 000 data points (see the details of the architecture
8

---

Under review as a conference paper at ICLR 2020
0
-0
-0.1
0
0
-0
-0
Overall Error
0
-0.1
0
0.1
0
0
Error as
N
4
0
-0.4
D
0.2
0.1
0.1
0
0.1
-11
0
12
0
0.1
Error (%)
4
-0
0.1
0
0.1
8-
02
0
0.1
0.3
0.4
-1.5
0.1
0.5
0.1
0.1
-0.1
0.1
02
0.1
00
0.1
0
0
-0.1
0
0.1
0
0
0.4
0
0.6
-.1
-02
4.1
0.2
0.7
0.1
-01
-0.1
1.5
Poisoning Points
Figure 7: (Lefft Overall classification error and error of digit being classified as 5 in MNIST (with all classes), as a function the attack strength with PGAN. (Right) Difference in the confusion matrix between the clean and the poisoned classifier (4% poisoning).
used in Appendix C). For PGAN we used a - 0.1 and À 0.9 Pr(Y,) p We varied the fraction of poisoning points exploring values in the range 0- The results in Fig. (left) show that, although the overall test classification eπror only increases slightly. the test error of digit 3 being classified as a is significantly affected by the attack, increasing from 1.1%. when there is no attack, to 13.1% with just 4% of poisoning points. In Fig. (right) we show the average difference in the confusion matrix evaluated on the clean dataset and the poisoned dataset (4% poisoning). We can observe that the detection rate of digit 3 decreases up to 1 11, and that this decrease is due to increase of 12% on the error of digit being incorrectly classified as a 5. This experiment support the usefulness of PGAN to generate targeted attacks, showing that even with small fraction of poisoning points can craft successful targeted (error-specific) attacks.
4.3 BYPASSING DEFENCES
Apart from the outlier detection defence described previously, we also tested pGAN against 3 dif- ferent types of defences: First, we considered Sever Diakonikoass et al. (2019). for robust optimization that aims remove outliers and points that can have a negative impact on the learning algorithm at training time. We followed the settings described in Diakonikolas et al. (019) applying the algorithm separately for each class. We set the value of 6, the parameter that controls the fraction of points to be removed, to 0.1 (results for different values of are also shown in Appendix 1. For the sake of computational tractability we applied Sever using the parameters in the last layer of the target classifier, as these parameters are more influenced by the poisoning attack P Secondly, we considered the defence introduced by (Rubinstein et al. 2009) which relies on Principal Component Analysis (PCA) to detect poisoning points. This model assumes that the clean data lies in low-rank subspace and that poisoning points will have a large component out of this subspace. As in Sever, this defence has a parameter e that controls the fraction of points to be rejected. In our experiment we set E = 0.05. Further results for different values of are shown in Appendix E.2. Finally we also tested pGAN attack against the label sanitization technique troduced by (Paudice et al.) (018b), which relabels training data points according to KNN-based algorithm so that a point is relabelled if, at least, n4 neighbours have the same label, different from the label of the evaluated point. Following similar settings as in Paudice et al. (0018b), we set A 3 for KNN and nış = 2.
algorithm
Using the same settings in our previous experiments we tested the 3 defences against PGAN attacks with a = 0.1 for MNIST and FMNIST. This value of a produced the most effective attacks against the outlier detection defences shown in Fig. 2 The results in Fig. g show that pGAN successfully bypasses all the defences in the two datascts. We can observe that Sever performs worse than the outlier detector defence for MNIST although the degradation with the increasing number poisoning points follows a similar trend. Tn the case of FMNIST, Sever is more effective compared to the other defences when the number of poisoning points is reduced, but in this case, the
Thhe experimental evaluation included in Diakonikolas testing the defence.
2019) only considers linear classifiers for
NNte that the outlier detection defence assumes stronger model for the defender, which is in control of fraction of trusted (clean) data points train the outlier detector.

---

Under review as a conference paper at ICLR 2020
MNIST
FMNIST
MNIST
33
Dutlee Det.
10
733
Label Senit.
Error (96)
Error
15
Outleer Det.
Sever
0.0
Sever
Outleer Det.
10
5 10 15
20
25
30
0
15
20
25
30
10
15
20
25
%6 of Paisoning Points
% of Poisoning Points
% of Poisoning Points
Figure 8: Test classification error (%) function of the percentage of poisoning points using PGAN with a = 0.1 for MNIST (left) and FMNIST (centre) using outlier Setection, (with 0.1), and the PCA-based defence (with e 0.05). Right-most plot compares the test error of outlier detection and label sanitization defences in MNIST.
algorithm degrades faster when we increase the fraction of malicious points. In Appendix E. we show that PGAN produces similar effect Sever for different values of e.
The PCA-based defence performs worse than the outlier detector in MNIST, whereas FMNIST the difference between the defences very small. In both cases, both algorithms degrade in the same way as we increase the number of poisoning points. In Appendix E.2 we show that for larger values of this defence performs worse. Finally, in Fig. rright we can observe that label sanitization completely fails to defend against our attack in MNIST. As pGAN produces poisoning points that are correlated, the KNN-based algorithm proposed to do the relabelling not capable of detecting the poisoning points. Furthermore, some of the genuine points from the target class are incorrectly relabelled, making the problem even worse. Similar results are obtained for FMNIST, as shown in Appendix E.3.
The results in Fig. S along with the previous results in Figs. show that pGAN not only can generate successful poisoning points scale even when using detectability constraints but also that PGAN is successful against mate-offthee-a defences published in the literature.
5 CONCLUSION
The PGAN approach we introduce in this paper allows to naturally model attackers with different levels of aggressiveness and the effect of different detectability constraints the robustness of the algorithms. This allows to a) study the characteristics of the attacks and identify regions of the data distributions where poisoning points are more influential, yet more difficult detect, b) systematically generate in an efficient and scalable way attacks that correspond to different types of threats and e) study the effect of mitigation measures such as improving detectability. addition to studying the tradeoffss involved in the adversarial model, pGAN also allows to naturally study the tradeoffs between performance and robustness the system as the fraction of poisoning points increases. Our experimental evaluation shows that PGAN effectively bypasses different strategies to mitigate poisoning attacks, including outlier detection, label sanitization, PCA-based defences and Sever algorithm.
REFERENCES
Marco Barreno, Blaine Nelson, Russell Sears, Anthony D Joseph, and J Doug Tygar. Can Machine Learning be Secure? In Symposium on Information, Computer and Communications Security, pp. 16-25, 2006.
Marco Barreno, Blaine Nelson, Anthony D Joseph, and Doug Tygar. The Security of Machine Learning. Machine Learning, 1122)121 2010.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning Attacks against Support Vector Ma- chines. In International Conference Machine Learning, pp. 1807-1814, 2012.
10

---

Under review as a conference paper at ICLR 2020
Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. Sever: A Robust Algo-rign for Stochastic Optimization. In International Conference on Machine Learning. pp. 1596-1606, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations, 2015.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv preprint (rXiv:1708.0633,, 2017.
Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and JD Tygar. Adversarial Machine Learning. Workshop on Security and Artificial Intelligence, pp. 43-58, 2011.
Vittorio Illiamo, Luis Muñoz González, and Emil C Lupu. Don't Fool Me!: Detection, Char- acterisation and Diagnosis of Spoofed and Masked Events in Wireless Sensor Networks. IEEE Transactions on Dependable and Secure Computing, 14(3):279-22) 2016.
Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Ma- nipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning. In IEEE Symposium on Security and Privacy, pp. 19-35, 2018.
Anthony D Joseph, Pavel Laskov, Fabio Roli, Doug Tygar, and Blaine Nelson. Machine Learning Methods for Computer Security (Dagstuhl Perspectives Workshop 12371). Dagstuhl Manifestos, 3(1), 2013.
Marius Kloft and Pavel Laskov. Security Analysis of Online Centroid Anomaly Detection. Journal of Machine Learning Research, 13:3681-3724, 2012.
Pang Wei Koh and Percy Liang. Understanding Black-box Predictions via Influence Functions. In International Conference on Machine Learning, pp. 1885-1894, 2017.
Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger Data Poisoning Attacks Break Data Sanitization Defenses. arXiv preprint arXiv: 2018.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Dataset, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et Gradient- based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(1 1):2278-2324 1998.
Shike Mei and Xiaojin Zhu. Using Machine Teaching to Identify Optimal Training-Se SSt Attacks on Machine Learners. In AAAI, pp. 2871-2877, 2015.
Mehdi Mirza and Simon Osindero. Conditional Generative Adversarial Nets. arXiv preprint arXiv:1411. 2014.
Luis MunózzGonzález, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards Poisoning of Deep Learning Algorithms with Back- Gradient Optimization. In Workshop on Artificial Intelligence and Security pp. 27-38, 2017.
Blaine Nelson, Marco Barreno, Fuching Jack Chi, Anthony D Joseph, Benjamin IP Rubinstein, Udam Saini, Charles A Sutton, J Doug Tygar, and Kai Xia. Exploiting Machine Learning to Subvert Your Spam Filter. LEET, 8:1-9, 2008.
Andrea Paudice, Luis Muñoz Gonzálzz, Andras Gyorgy, and Emil C Lupu. Detection of Ad- versarial Training Examples in Poisoning Attacks through Anomaly Detection. arXiv preprint arXiv: 0304001, 2018a.
Andrea Paudice, Luis Munoz--Gozzáezz and Emil Lupu. Label Sanitization against Label Flipping Poisoning Attacks. Nemesis' 18 Workshop on Recent Advancements in Adversarial Machine Learning, 2018b.
11

---

Under review as a conference paper at ICLR 2020
Benjamin IP Rubinstein, Blaine Nelson, Ling Huang. Anthony D Joseph, Shing- hon Lau, Satish Rao, Nina Taft, and Doug Tygar. Antidote: Understanding and Defending against Poisoning of Anomaly Detectors. In SIGCOMM Conference on Internet Measurement, pp. 1-14. 2009.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved Techniques for Training GANs. In Advances in Neural Information Processing Systems, pp. 2234-222222 2016.
Ali Shafahi, Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks. In Advances Neural Information Prucessing Systems, pp. 6103-6113, 2018.
Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. Certified Defenses for Data Poisoning Attacks. In Advances in Neural Information Processing Systems, pp. 3517-3529, 2017.
Mingxi Wu and Christopher Jermaine. Outlier Detection by Sampling with Accuracy Guarantces. In International Conference on Knowledge Discovery and Data Mining, pp. 767-772, 2006.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Bench- marking Machine Learning Algorithms. arXiv preprint /7Xiv:/78..07700 2017..
2017.
Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, and Fabio Roli. Is Fea- ture Selection Secure against Training Data Poisoning? In International Conference Machine Learning, pp. 1689-1698, 2015.
Chaofei Yang, Qing Wu, Hai Li, and Yiran Chen. Generative Poisoning Attack Method against Neural Networks, arXiv preprint 1rXiv:1703.011 2017.
12

---

Under review as a conference paper at ICLR 2020
A PGAN TRAINING ALGORITHM
We train PGAN following a coordinated gradient-based strategy by sequentially updating the param- eters of the three components using mini-batch stochastic gradient descent/ascent. The procedure is deseribed in Algorithm | For the generator and the discriminator data points are sampled from the conditional distribution
the subset of poisoning labels Y,. For the classifier, honest data points are sampled from the data distribution including all the classes. We alternate the training for the three components with the i, and & number of steps for the discriminator, classifier, and generator respectively. In practice, we choose i.j >> k, i.c. we update more often the discriminator and the classifier. For example, our experiments we j.j 4 and k
Algorithm pGAN Training
for number of training iterations do for steps do
sample mini-batch of m noise samples (Z,, MY, Do,YY)mme p>== from P. (2 pp)
get mini-batch of m training samples (xn um I==1 from Pa (x YY)))
update the discriminator by ascending its stochastic gradient
Log (xxnllyy + log D(g(zn) Y,
0p
n
end for for steps do sample mini-batch of m noise samples {z,, Y pm from (=(z Yp)
get mini-batch of training samples 1m from (x)
update the classifier by ascending its stochastic gradient
xC)( ()((x)))
Bcc
m
end for for k steps do sample mini-batch of noise samples {zn, Y p from p=(z) Y, update the generator by descending its stochastic gradient
m
VOG
log(1 D(g(z,, (Y))) (1-
771
end for end for
B SYNTHETIC EXAMPLE: EXPERIMENTAL. SETTINGS AND EFFECT ON THE DECISION BOUNDARY
For the synthetic experiment shown in the paper sample training and test data points from two bivariate Gaussian distributions, (uo, Mo) and A(μ1, E1), with parameters:
2.5
[..8
0.7
En
-1.0
0.7
2.0
0.55
1.0
0.3
1.0
0.3
1.4
We trained PGAN with 500 training data points for each class with À = 0.8 and ar E 0., 0.8, 1]. We set the number epochs to 3, 000, the batch-size to 500, and the parameters in Algorithm [ used one-hidden-layer neural networks with
j.j.k k 1. For the generator and the discriminator Leaky ReLU activation functions. For the classifier we used logistic regression with cross-entropy loss function. The details about the architecture of the three components detailed in Table [
13

---

Under review as a conference paper at ICLR 2020
Table 1: pGAN architecture for the Synthetic experiment (Notation: SGD stands for Stochastic
Gradient Descent)
Architecture: DNN (2 x 20 x 2)
Generator
Hidden layer act. functions: Leaky ReLU (negative slope 0.1)
Output layer act. functions: Linear
Optimizer: Adam (learning rate = 10-1)
Architecture: DNN 250 1)
Discriminator
Hidden layer act. functions: Leaky ReLU (negative slope = 0.1)
Output layer act. functions: Sigmoid
Optimizer: SGD (learning rate 10 momentum = 0.9)
Architecture: Logistic Regression
Classifier
Loss function Le: Cross- Cross-entropy
Optimizer: SGD (learning rate 10 10-3, momentum = 0.9)
0.0
a=0.2
a 1.1.0
Figure 9: Synthetic experiment: Distribution of genuine (green and blue dots) and poisoning (red dots) data points for different values of a. The poisoning points are labelled as green.
In Fig. we show the effect of the poisoning attack on the decision boundary. For testing pGAN we trained a separate logistic regression classifier with 40 genuine training examples (20 per class) and adding extra 20% poisoning points (8 samples). We trained the classifier using Stochastic Gradient Descent (SGD) with a learning rate of 0.01 for 1, 000 epochs. applied to pre-filter the training points. The results in Fig. show that for effective, although the poisoning points depicted in red (which are labelled as green) are far from the genuine distribution of green points. Then, In this synthetic example, the classifier is quite stable: the number features is very small (two), and the topology of the problem is simple (the classes are linearly separable and the overlapping between classes is small) and the classifier is simple. Thus, the effect of the poisoning attack when considered, i.e. f is very reduced. Note that the purpose of this synthetic example is just to illustrate the behaviour of pGAN as function of À rather than showing an scenario where the attack can very effective.
this case, no outlier detector 0 the attack is very
increase the value of À the attack is blunt.
detectability constraints
C EXPERIMENTAL SETTINGS
Here provide complete details about the settings for the experiments described in the paper. In Table 2 we show the characteristics of the datasets used in our experimental evaluation. The param- eters for training PGAN for MNIST and FMNIST are shown in Table 3 whereas the parameters for CIFAR are described in Table H In all cases, for PGAN generator we used (independent) Gaussian noise with zero mean and unit variance.
14

---

Under review as a conference paper at ICLR 2020
Table 2: Characteristics of the datasets used in the experiments
Name
# Training Examples
# TTst Examples
Features
MNIST (3 vs 5)
6, 131 /5, 421
1, 1,010/892
784
MNIST (all)
10, 000
10, 000
784
FMNIST (sneaker vs ankle bool)
6, 000 / 000
1, 0,00/ 000
784
CIFAR (automobile vs truck)
5, 000/5, 000
1, 1,000/11000
3,072
Table 3: PGAN architecture for MNIST and FMNIST
Architecture:
DNN (100 x 784 x 1, 024 x 784)
Generator
Hidden layer act. functions: Leaky ReLU (negative slope 0.1)
Output layer act. functions: Tanh
Optimizer: Adam (learning rate = 10-4)
Dropout: p 0.5
Architecture: DNN (784 x 1, 024 x 512 x 1) Hidden layer act. functions: Leaky ReLU (negative slope 0.1)
Discriminator
Output layer act. functions: Sigmoid
Optimizer: SGD (learning rate 10-3, momentum = 0.9)
Dropout: p 0.5
Architecture: DNN (784 x 1, 024 x 512 x 1)
Classifier
Loss function Le: Cross-entropy
Hidden layer act. functions: Leaky ReLU (negative slope = 0.1)
Output layer act. functions: Sigmoid
Optimizer: SGD (learning rate 10-3, momentum = 0.9)
Dropout: p == 0.5
For MNIST we trained pGAN for 2, 000 epochs using batch-size of 200, setting i.j 4 and 1 in Alg. [ For FMNIST we used similar settings but training for 3, 000 epochs. For CIFAR trained PGAN using batch-size of 25 for 300 epochs, with i.j and k = 1. Finally, the architecture the Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs) trained test the attacks is described in Tables and 6
D GENERATION OF POISONING SAMPLES WITH PGAN
In Figs. 10 12 we show samples generated with PGAN for different values of in MNIST, FMNIST
and CIFAR respectively. The class labels the poisoning points are and ankle boot and truck for each of the datasets. In all cases the generated examples exhibit characteristics from the two classes involved in the attack, although PGAN tries preserve features from the (original) poisoning class to evade detection. For values of close to 1, the samples generated by PGAN are similar to those we can generate with a standard GAN.
observe that for small values of (but with > O),
EXTENDED COMPARISON OF DEFENCES
E. SENSITIVITY ANALYSIS FOR SEVER
Here we show the sensitivity analysis of Sever performance to pGAN attack w.r.t. its parameter e, which controls the fraction of training points removed. For this, using the same experimental settings in the paper, tested the performance of Sever for [0.05, 0.1, 0.2]. The results in Fig. shoo that the performance of the attack is not very sensitive to the value for both MNIST and FMNIST.
15

---

Under review as a conference paper at ICLR 2020
Table 4: pGAN architecture for CIFAR
Architecture: DCNN:
Generator
Layer 1: 2D transposed convolutional: input channels: 100; output channels: 1024; kernel size: (2x2); stride: 1; padding: no bias terms; batch normalization
Layer 2D transposed convolutional; input channels: 1024; output channels: 256; kernel size: (4);); stride: 2; padding: 1; no bias terms; batch normalization
Layer 3: 2D transposed convolutional; input channels: 256; output channels: 128; kernel size: (4x4); stride: 2: padding: 1; bias terms; batch normalization
Layer 4: 2D transposed convolutional; input channels: 128; output channels: 64; kernel size: (4x4); stride: 2: padding: no bias terms; batch normalization
Layer 5: 2D transposed convolutional; input channels: 64; output channels: 3; kernel size: (4x4); stride: 2; padding:
Hidden layer act. functions: ReLU
Output layer act. functions: Tanh
Optimizer: Adam (learning rate = 10-3)
Architecture: DCNN:
Discriminator
Layer 1: 2D convolutional; input channels: 3; output channels: 64;
kernel size: (4x4); stride: padding: 1
Layer 2: 2D convolutional; input channels: 64; output channels: 128;
kernel size: (4x stride: 2; padding: I; no bias terms;
batch normalization
Layer 2D convolutional; input channels: 128; output channels: 256; kernel size: (4x stride: 2: padding: 1; no bias terms; batch normalization
Layer 4: 2D convolutional; input channels: 256; output channels: 512: kernel size: (4x4); stride: 2; padding: 1; no bias terms;
batch normalization
Layer 5: 2D convolutional; input channels: 512; output channels: 1;
kernel size: (2x2); stride: 1; padding: 0
Hidden layer act. functions: Leaky ReLU (negative slope = =02)
Output layer act. functions: Sigmoid
Optimizer: SGD (learning rate 1.5 10-4 momentum 0.5)
Architecture: DCNN:
Classifier
Layer 1: 2D convolutional; input channels: 3; output channels: 32;
kernel size: (4x4); stride: 2; padding: 1
Layer 2D convolutional; input channels: 32; output channels: 128;
kernel size: (4x4); stride: 2; padding: 1; no bias terms;
batch normalization
Layer 3: 2D convolutional; input channels: 128; output channels: 256:
kernel size: (4x stride: 2; padding: 1; no bias terms; batch normalization
Layer 4: 2D convolutional; input channels: 256; output channels: 512: kernel size: 44x stride: 2; padding: 1; no bias terms;
batch normalization
Layer 5: 2D convolutional; input channels: 512; output channels: 1;
kernel size: (2x2); stride: 1; padding:
Hidden layer act. functions: Leaky ReLU (negative slope 0.2)
Output layer act. functions: Sigmoid
Loss function Le: Cross-entropy
Optimizer: SGD (learning rate 10-1, momentum = 0.5)
16

---

Under review as a conference paper at ICLR 2020
Table 5: Architecture of the classifiers to test the attacks on MNIST, FMNIST and CIFAR.
Classifier binary MNIST and FMNIST
Architecture: DNN (784 x 1, 024 x 512 x 1)
Loss function Le: Cross-entropy
Hidden layer act. functions: Leaky ReLU (negative slope 0.1)
Output layer act. functions: Sigmoid
Optimizer: SGD (learning rate 10-3, momentum 0.9)
Batch size: 200
Epochs: 2, 000
Dropout: p = 0.5
Classifier binary CIFAR
Architecture: Deep CNN:
Layer 1: 2D convolutional; input channels: 3; output channels: 32;
kernel size: x3); stride: I; padding: 0; no bias terms;
batch normalization; 2D max pooling (2 x 2); dropout: = 0.3
Layer 2: 2D convolutional; input channels: 32: output channels: 64;
kernel size: (3x3); stride: 1; padding: O; no bias terms;
batch normalization; 2D max pooling (2 2); dropout: p 0.4
Layer 3: 2D convolutional: input channels: 64; output channels: 128;
kernel size: (3x3); stride: 1; padding: no bias terms;
batch normalization; 2D max pooling (2 x 2); dropout: 0.5
Layer 4: flattening + fully connected (512 x
Hidden layer act. functions: ReLU
Output layer act. functions: Sigmoid
Loss funetion Le: Cross-entropy
Optimizer: SGD (learning rate 10 10-3, momentum = 0.9)
Batch size: 100
Epochs: 1, 000
Table Architecture of the classifiers to test the attacks on multi-class MNIST (i.e. using all 10
class labels).
Classifier multi-class MNIST
Architecture: DNN (784 1, 024 512 x 10)
Loss function Le: Cross-entropy
Hidden layer act. functions: Leaky ReLU (negative slope 0.1)
Output layer act. functions: Softmax
Optimizer: SGD (learning rate 0.01, momentum 0.9)
Batch size: 500
Epochs: 1, 000
Dropout: p 0.5
17

---

Under review as a conference paper at ICLR 2020
a=
0.0
a 0.3
3
3
3
3
3
3
3
3
3
3
3
5
3
3
3
3
3
5
3
3
3
3
3
3
3
3
5
3
3
5
3
5
3
$
3
3
3
3
3
3
3
3
5
3
8
5
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
5
3
5
3
5
5
3
5
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
8
3
3
3
3
3
3
3
3
3
3
3
3
3
3
5
3
3
3
3
3
3
3
3
3
3
3
3
3
3
5
3
3
3
3
5
3
8 3 9
5
3
3
3
3
3
3
3
3
3
3
3
3
5
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
5
3
5
3
3 3
3
3
3
3
3
3
33
3
3
5
8
3
5
5
3
3
a 0.5
a 0.9
5
5 5
5
5
5
3
3
3
5 5 5 5 5
5
3
9
3
}
5
3
3
5
5
5
5.5
$
5
5
5
5
5
3
5 5555555555
5
5
4
5
5
5
5
5
3
3
5555555555
5
5
5
3
3
3
5
5555555555
3
5
5
5
5
3
33
3
5555555555
5
3
5
5
5
5
5
0
5
55555$5555
3
3351
3
5
3
5
5
5555555558
5
5
3
3
53
5555555551
3
3
3
5
5
5
5
5 5
5 5 5
Figure 10:
Examples from PGAN on MNIST dataset for different values of a.
18

---

Under review as a conference paper at ICLR 2020
0.0
a 0.3
a= 0.5
a= 0.9
Figure 11: Examples from PGAN on FMNIST dataset for different values of a.
19

---

Under review as a conference paper at ICLR 2020
a = 0.1
a == 0.3
a= 0.5
a = 0.9
Figure 12: Examples from PGAN with CIFAR dataset for different values a.
MNIST
FMNIST
8.0
10
0.1
7.5
0.1
Error (%)
g 0.2
(%6)
7.0
E 0.2
6.5
6.0
5
10
15
20
25
30
10
15
20
25
30
% of Polsoning Points
% of Polsoning Points
Figure 13: Test classification error(%) for Sever (with
[0.05. 0.1, 0.2] function of the
percentage poisoning points using PGAN with a = 0.1 for MNIST (left) and FMNIST (right).
20

---

Under review as a conference paper at ICLR 2020
MNIST
FMNIST
12
0.05
9-
Error (%)
10-
0.2
7-
0.05
0.2
10
20
25
Poisoning Points
Polsoning Points
Figure 14: Test classification error(%) for the PCA-based defence (with [0.05, 0.1, 0.2))
function of the percentage of poisoning points using pGAN with 0.1 for MNIST (left) and FMNIST (right).
30
Outlier Det.
20.0
22.5
Outlier Det.
25
Label Sanit.
Label Sariit.
(%)
20
15
12.5
10.0
7.5
10
25
% of Polsoning Points
% of Polsoning Points
Test classification error(%) for label sanitization and outlier detection defences as function of the percentage of poisoning points using pGAN with 0.1 for MNIST (left) and FMNIST (right).
Figure 15:
E.2 SENSITIVITY ANALYSIS FOR THE PCA-BASED DEFENCE
As for Sever, tested the performance of the PCA defence for different values of which in the previous case, controls the fraction of training points removed. We explored the values = [0.05, 0.1, 0.2). The results in Fig. 14 show that, in this case, the performance is further degraded as we increase the value of especially for FMNIST.
E.3 LABEL SANITIZATION
we compared performance of the label sanitization defence and outlier detection defence for both MNIST and FMNIST. We can observe that label sanitization performs very poorly against pGAN attacks compared to outlier detection. The average test error increases up to 30% with 30% of poisoning points for MNIST, whereas for FMNIST the error increases up 22% for the same fraction of poisoning points.
In Fig. 15
21


Reviews:
[{"review_id": "r1eDh8QAFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposed a method pGAN based on Generative Adversarial Networks to generate poisoning examples in order to degrade the performance of classifiers when trained on the poisoned training data. The authors evaluated pGAN on both synthetic datasets and commonly used MNIST and Fashion MNIST datasets in machine learning.\n\nThe paper is self-contained and easy to read. My main concern is on the experiment results. The detailed questions are as follows:\n\nQ1: Has the authors tried more complicated datasets such as CIFAR-10 to evaluate the pGAN method? It would make the paper more convincing to add results on more complex datasets.\n\nQ2: Can the authors structure the experimental results with different sections? Currently it is just a single section which is difficult to read. \n\nQ3: The authors noticed that “But, as we decrease the value of α, the distribution of red points shifts towards the region where both green and blue distributions overlap”. This observation is interesting as it finds that the poisoned input tends to lie on the overlap of two classes. But this can easily lead to a defense method: remove those training examples that are close to the other class. This defense mechanism can be used together with other sanitization approaches. So I would like to see how would pGAN perform in this case?\n\nQ4: The authors mentioned “Comparison with existing poisoning attacks in the research literature is challenging: Optimal poisoning attacks as in Munoz-Gonzalez et al. (2017) are computationally very expensive for the size of the networks and datasets used in our experiments in Fig. 2.”. \nHowever, I can not agree because you can simply generate poisoned data and train the neural networks on the poisoned data regardless of the underlying approach that is targeted in generating the poisoned data. This would be an effective baseline to compare. (Correct me if I am wrong here.)\n\nI will change my score if the authors can address my concerns here.\n\n================================================================\nThanks for the rebuttal. I am more convinced now.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Skgk1VhAYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new generative poisoning attack method against machine learning classifiers. The authors propose pGAN with three components to maximum the error of classification and guarantee undistinguished poisoning data for the discriminator. The experimental results show that the hyperparameter \\alpha significantly affects the poisoning data distribution and pGAN leads to specific error in a classification task.\n\nThis paper should be weekly accepted, considering the following aspects.\n\nPositive points: (1) The experiments seem solid. The overall performance with different parameters and the corresponding error type have been evaluated. (2) The error-specific and performance-control characteristics of pGAN seem to be interesting. (3) The paper is well organized.\n\nNegative points: (1) The authors should provide more justification on equation-3. Why do the authors directly average different loss for the discriminator and the classifer? (2) The function of the discriminator is not very clear, especially for the classification error test. Does the discriminator exclude the poisoning data according to certain rule? It would make more sense if the classification error measured from the data the discriminator selects. (3) pGAN can produce error-specific attack without sufficient justifications. Why can pGAN lead to the inclination? Is it possible for pGAN to control the specific error tendency? (4) For the error-specific attack task, it would be better to provide an ablation experiment. For example, authors could implement pGAN by ignoring the detectability of the discriminator (i.e. \\alpha=0) or typical pGAN when they compare with the label-flip operation. Please explain which component contribute to the error-specific inclination.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeZZKSxqB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles vulnerability to poisoning. An important subtopic of adversarial ML.\nThe authors propose using a GAN to generate poisoning data points, as an alternative to existing methods.\n\nWhile most (or all) of the paper is devoted to illustrate the effectiveness of the approach against *non-protected* ML. My only and biggest concern with this paper is that no defense mechanism has been tested against, and there are many in the literature. (see e.g. Diakonikolas et al ICML 2019).\n\nThus my question for the rebuttal period: How would pGAN perform when defense mechanisms are deployed during the learning phase? (ideally, a thorough experiment illustrating the strength of pGAN against a few defense mechanisms would help re-evaluating the score)", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a GAN-based approach to producing poisons for neural networks.  While the approach is interesting and appreciated by the reviewers, it is a legitimate and recurring criticism that the method is only demonstrated on very toy problems (MNIST and Fashion MNIST).  During the rebuttal stage, the authors added results on CIFAR, although the results on CIFAR were not convincing enough to change the reviewer scores; the SOTA in GANs is sufficient to generate realistic images of cars and trucks (even at the ImageNet scale), while the demonstrated images are sufficiently far from the natural image distribution on CIFAR-10 that it is not clear whether the method benefits from using a GAN.   It should be noted that a range of poisoning methods exist that can effectively target CIFAR, and SOTA methods (e.g., poison polytope attacks and backdoor attacks) can even target datasets like ImageNet and CelebA.
Decision: Reject