BUZz: BUffer Zones for defending  adversarial examples in image classification
Authors: Phuong Ha Nguyen*
Kaleel Mahmood*
Lam M. Nguyen
Thanh Nguyen
Marten van Dijk
Venue:  (2020)
Keywords: adversarial machine learning
machine learning security

Abstract: We propose a novel defense against all existing gradient based adversarial attacks on deep neural networks for image classification problems. Our defense is based on a combination of deep neural networks and simple image transformations. While straight forward in implementation, this defense yields a unique security property which we term buffer zones. In this paper, we formalize the concept of buffer zones. We argue that our defense based on buffer zones is secure  against state-of-the-art black box attacks. We are able to achieve this security even when the adversary has access to the entire original training data set and unlimited query access to the defense. We verify our security claims through experimentation using FashionMNIST, CIFAR-10 and CIFAR-100. We demonstrate <10% attack success rate -- significantly lower than what other well-known defenses offer -- at only a price of a 15-20% drop in clean accuracy. By using a new intuitive metric we explain why this trade-off offers a significant improvement over prior work.

Under review as a conference paper at ICLR 2020
BUZz: BUFFER ZONES FOR DEFENDING ADVERSAR- IAL EXAMPLES IN IMAGE CLASSIFICATION
Anonymous authors Paper under double-blind review
Abstract
We propose a novel defense against all existing gradient based adversarial attacks on deep neural networks for image classification problems. Our defense based on a combination of deep neural networks and simple image transformations. While straight forward in im- plementation, this defense yields a unique security property which we term buffer zones. In this paper, we formalize the concept of buffer zones. We argue that defense based on buffer zones is secure against state-of-the-art black box attacks. We are able to achieve this security even when the adversary has access to the entire original training data set and unlimited query access to the defense. We verify our security claims through experimen- tation using MNSSS CIFAR-10 and CIFAR-IOK We demonstrate < 10% attack success rate significantly lower than what other well-known defenses offer only price of 15-20% drop in clean accuracy. By using a new intuitive metric we explain why this trade-off offers a significant improvement over prior work.
INTRODUCTION
There are many applications based on Convolution Neural Networks (CNNs) such image classifi- cation (Krizhessky et al. (2012); Simonyan & Zisserman (2015), ). object detection (Girshick (2015); Ren et al. (2015)), semantic segmentation (Shelhamer al.) (2017) and visual concept discovery (Wang et al. ()177). However, it is well-known that CNNs are highly susceptible to small pertur- bations which are added to benign input images. As shown in (Szegeyy et al. 2013); Goodfellow et al. (2014)), by adding visually imperceptible perturbations to the original image, adversarial ex- amples can be created. These adversarial examples are misclassifed by the CNN network with high confidence. Hence, making CNNs secure against this type of attack is a significantly important task.
The strength of an attack relative to the considered adversarial model. In the white-box setting parameters of the target/defense model f are given to the adversary white-box attacks use this information to model access to an oracle that returns gradients f-)). From our analysis of previous literature, it is clear that secure white-box defense is extremely challenging to design. We can also question the realism of having an adversary that knows the model's weights and architecture. Many online machine learning services by default only allow black-box access their models and do not publish their model parameters (Papernot et al. (2017)). Therefore in this paper, we keep the classifier defense parameters secret. This disallows white-box attacks and so we focus exclusively on black-box adversaries in this paper.
In black-box setting, the adversary may know some general features about the classifier (i.e. that CNN is being used) and how the target model is trained. Most importantly, the adversary has query access to the target model itself and/or access to (part of the) training dataset. The adversary uses this information to train a synthetic model g. Using the synthetic model, adversarial examples can be created. The underlying assumption the black-box setting is that large percent of the adversarial examples created with the synthetic model will also fool the target model f.
Our proposed defense assumes black-box adversary with two important attributes. First, the ad- versary has unlimited access to oracle (target model) which returns the final class label F.)(.)) Here f(x) indicates the score vector enumerating confidence scores for each possible class label and F()) computes the class label with the maximum confidence score. The second important at- tribute our adversary has is access to the entire original training dataset. In this sense, we model the strongest known black-box adversary which has not yet been studied in the literature.
Defense based on Buffer Zones. We first explain the concept of buffer zones. Next, we argue how "wide" buffer zones force the black-box adversary produce "sufficient large" noise m,: As

---

Under review as a conference paper at ICLR 2020
discussed in Papernot et al. (2016a) count an attack successful only if the adversarial noise has small magmitude, say | I e, which cannot be recognized by human beings. Forcing noise ||||| accomplishes our security goal as either the human eye or our defense detects 17.
Figure a describes a 2D snapshot of the landscape with class labels A, B and C' are depicted. Clearly, for any image which is close the boundaries between the regions, the adversary can produce an adversarial example x' by adding a small noise " æ. The resulting adversarial image is what we would consider true adversarial image. Here we say true in the sense that the difference between & and x' is almost visually imperceptible to humans, but it makes the classifier produce wrong label.
normal classifier. Three different regions
B
B
c
(b)
Figure 1: (a) Landscape without buffer zone, and (b) Landscape with buffer zone 1.
To force the adversary to use large perturbation 17, create buffer zones 1 between all the regions as presented in Figure Now the classifier outputs one of the class labels (A, B, or C) or L. where I means "no class is assigned". We can think of as the class label of the buffer zones. In other words, the adversarial noise yj (i.e., perturbation v = x' - .)) must cross over the buffer zones 1 in order modify label to label l'. To cross from to B in Figure [ is initially not visible by the human eye as long as one remains in the original A-region of Figure I and starts to transition into the B-region of Figure I However, to cross over into the smaller B-region in Figure T we need an additional perturbation which we expect will become visible.
Since we only focus on small noise which is not recognized by human beings, the buffer zones 1 allow us to defend against adversarial images with small noise. Only large adversarial noise, which out of the interest of the attacker as can be detected by the human eye, can possibly fool our defense. If the buffer zones are wide, then we accomplish our security goal since small adversarial noise cannot cross over the buffer zones, however, the prediction/clean accuracy of the defense will decrease (as it becomes more noise intolerant to both clean and adversarial examples). Notice that it well-known in the security community that to provide security to any system, there is an associated cost; believe the buffer zone concept is reasonable as experiments for our proposed techniques for creating mostly wide buffer zones show that the clean accuracy suffers only about 15-20% while reducing the attacker's success rate to about 5-10% (a 90% drop in success). Being able to reach such small attacker's success rates is a main contribution of our paper.
Performance of defense. We introduce a new metric to properly understand the combined effect of (I) a drop y in clean accuracy from an original clean accuracy p to clean accuracy Pa p for the defense and (2) small attacker's success rate a against the defense. The attacker's success rate is defined as the fraction of adversarial examples that produce an adversarial label for images that are properly classified by the defense. So, in a non- non-malicious environment we have the original clean accuracy while in the malicious environment the probability of proper/accurate classification by the defense model (1-7)(1-a) (since the defense properly labels a fraction p- if no adversary is present and out of these images fraction ar is successfully attacked if an adversary is present).
We call (p- (1 a) the effective clean accuracy of the defense. Going from a non-malicious environment to malicious environment with defense gives drop in the effective clean accuracy of
8 -
- (p-
As an example of the usefulness of 5 suppose one wishes to classify new object by taking say images and submit these images to registration service which implements a classifier with defense. In malicious environment the camera which used for taking pictures or any man-in-hhe-middle can attempt to transform the images into adversarial examples (with a targeted new label) which cannot be detected by the human eye. Nevertheless, the service will see agreement among the produced labels of (in expectation) (p d'm images that are correct labeled with in the worst case the remaining images having the adversarial targeted label. This is drop of &n compared to registration of an object without adversary. The smaller ,, the better the defense. In order trust a majority vote among the image labels we need p > 0.5.
2

---

Under review as a conference paper at ICLR 2020
In order to minimize this drop 6, it turns out to be very important to have small enough, which is accomplished in this paper. We use subscripts t and u in óf and óy when differentiating for targeted attacks and untargeted attacks (since untargeted attacks are easier to pull off, d, 6,).
Contributions. (a) We focus on state-of-the-an (pure and oracle-based) black box attacks where strengthen the adversary in that allow the adversary access to the entire original training data with query access to the target classifier. Related work on defense strategies is detailed in Supplemental Material [ where we argue and/or experimentally demonstrate into what extent this strong adversary can successfully attack well-known defense strategies; summary is given in Table I These defenses typically (except [ramerr al. (2017)) have an attacker's success rate of 2 50% with a relative small drop in clean accuracy of < 0.05 with often almost no (close to zero percent) drop, for data sets such as MNIST and CIFAR-10. 10. Precise calculations show that this corresponds
to 8 values of at least 0.29 (Tramèr et al. (2017)), 0.43 (Srisakaokul et al. (2018), and 2 0.65 for the other defenses. It turns out that for our strong adversary a vanilla network as defense i.e., we implement no defense at all against the black-box attacker - has å 0.63 for CIFAR-10 10. The other considered defenses for CIFAR-10 do not do better than implementing no defense at all.
Defone
Date
Allack
Cb. Ascunecy
Det. Aucunacy
Bt
CIFAR 10
Mised BB JPGMM
6,0% (this paper)
88.35% (CHFAR 10)
68. 32% (CIEAR 10)
0.247
TTramer dt ml. 2017
CIPAK-TO
Mosed BB
-RISM
34M (dhi
85% (CIPAT-TO)
R0S6 (CIFARRI0
0.25
Snsskankul
2018)
CIEMR-10
Mised DD -IISMM
50%
SPX (CTAAI--
MPA (CIEAK-- 10)
0.43
Ni defemse
CIFAR-10
Mised BB -IFISM
72%
85% (CIFAR-10)
88% (CIFAR-IO)) 10)
0,63
2017
CO
BB
FOSM
(CTERR
SUSTCIFAK
Fanmen
CIFAR-TO
B8
['anlin Wapner 7117
(CIFATETO
8EB8S TCIFAR-TT
Experoo
201de
FOSM
Taperoor crai p017
Fue 2018
CIPARCIO
Mixed
TO
Mirwen 2017) 7
MNIST
Paro RR CSOW
51%
(arlini Wagner 2017h0
51 (MNIST))
91.8% (MNIST)
0.79
Meng Cntn 2017)
MNIST & CITRR TO
Pars DB
CRW
09%
Kanini & Wipter
20170
WOON ICIPAR TVI
S6.8 ICIPAR TO
0397
Table 1: Attacker's success rate of -lack-ox attacks for state-of-the-art defenses
(b) To the best of our knowledge, there are no papers on defenses against adversarial machine learn- ing based on our concept of buffer zones. This new concept offers a conceptually simple and efficient defense strategy with rigorous security argument.
(c) We realize wide buffer zones by combining multiple classifiers with majority vote based on a threshold together with image transformations that are unique for each of the classifiers. In Sec- tion we verify our security claims through experimentation using FashionMNIST, CIFAR-10 and CIFAR- 100 and show for untargeted attacks a drop in clean accuracy of 0.158, 0.200, and 0.170 in return for small attacker's success rates a 9%, 7%, and < 10%, respectively. This gives 8, values 0.226, 0.247, and 0.216 showing a significant improvement over prior work it makes sense sacrifice some clean accuracy in return for a much smaller attacker's 's success rate For CIFAR-10 we conclude that BUZz with å = 0.247 (and
7%) improves over Tramèr et al. (2017) with 0.29 (and a 34%); both are far better than any other defense, and since both use complimen- tary techniques we expect to be able to combine both improve d in future work. (All the above and this observation demonstrate the usefulness of our new d-metric.)
Finally, we will make all our code, including replicated defenses and attacks, available online.
Outline. We give an overview of known attacks Section 2 and a mathematical formulation of a black-box adversary is given in Supplemental Material Related work on white-box and black-box defenses is given in Supplemental Material B where we also analyze (by argument and experiment) white-box defenses against black-box adversaries this benchmarks our work. In Section we detail BUZz, our defense based on buffer zones. Section 4 has experiments and simulations showing the attacker's success rates versus clean rates. Supplemental Material C Cnumeraaes pseudo code for the implemented attacks and more experimental results are given in Supplemental Material D We
conclude in Section 5
ATTACKS
Adversarial Examples in an image classification task. See (Yuan et al. (2017)) the general scheme of a successful attack can be described as follows. The adversary is given a trained im-
2 BUZZ does better for targeted attacks with attacker's success rates a < 3.5%. < 5%, and < 7% leading to de values 0.144, 0.161, and 0.04. Here, in order to achieve 6, = 0.0423 for CIFAR- 100 it turns out to be best not to implement any defense (no BU7z)) the data set must have made it already hard for a successful targeted statc-of-the-art black-box attack!
3

---

Under review as a conference paper at ICLR 2020
age classifier (e.g, CNN network) f which outputs a class label / for given input data (i.e., image) r. The adversary will add a perturbation 77 to the original input x to get an adversarial example (or modified data input) x', i... x' = + n. Normally, 77 should be small make the adversar- ial example barely recognizable by humans. Yet, the adversary may be able to fool the classifier to produce any class label I' 1) as she wants. Assume that f())) (8,,88,....3 8k) is k- dimensional vector of confidence scores 8, of class labels j. We call f(x) the score vector with
0 8, S E5= 8j and ls the number of class labels. The class label is computed
I =(f(z)) argmax {51, (81,82..... $2,..,,,,888 8k]. (E(1....kk
Given x E [0, 1jd and l' =1= F(f(x)), the attacker wishes to ideally solve
min || - z|| such that F(J(=') = +1= F((x)), 2'E(0.1]4
(2)
where and l' are the output label of x and x', || denotes the distance between two data samples, x' x is the perturbation added on x. In this
and d is the number of dimensions of x. optimization problem, we minimize the perturbation r/ while the label I' is fixed (this represents a targeted attack). This problem becomes easier when the attacker has more information about [()): In some adversarial models, the adversary may know parigeters/weights that define the target model f. Some classification applications may directly output vector f() instead of F(f(x)) and this gives more information about the target model.
By adding a sufficient large noise 7) any given benign input image, can fool any existing image classifier. However, as discussed in (Papernot et al. (2166)) we count an attack successful only if the adversarial noise 17 has small magnitude, say |Ml] S c, which cannot be recognized by human beings (e indicates this transition from noise not being recognized to noise that is visually perceptible). In this sense the attacker is already successful soon sufficiently small perturbation v) (lzl S is found that realizes label l'. That is, finding the minimal possible perturbation 17 in 2 is not necessary.
Attack Methodologies. Attacks can be partitioned into two main categories based on their ap- proach. The first kind are white-box attacks where the adversary knows the parameters of de- fense/target model/classifier f and uses these parameters to compute gradients. One can think of this scenario as having oracle access gradients fxx for input images x. The attacker only uses this type of oracle access to compute adversarial examples.
The second kind are black-box attacks where the adversary does not know the parameters of f. but does have black-box access to the target model itself. One can think of this scenario as having oracle access to class labels Fxf()) or score vector values f(x) (the latter gives more information and models stronger attacker). In addition to having black-box access to the target model, the adver- sary may know and use (part of) the original training data (this can be used to train an adversarial synthetic model which resembles the target model). Since the oracles given to the white-box and black-box attackers are different/compliientaa white-box defenses and black-box defenses deal with different attack methodologies.
White-box Attacks. (Yuan et al. (207)) constructs perturbation with the help of gradient V V((): for example, 17 = e sign( V L(z () in the Fast Gradient Sign Method (FGSM) by (Goodfellow ct al. (2014)), where 0 represents the parameters of f. L is a loss function (c.g, cross entropy) of model 7. (E can be thought of as relating the maximum amount of noise which is not visually perceptible.)
Black-box Attacks. Black- box attacks use non non-gradient information of classifier such as (part of) the original training data set Xo (Papernot et al. (2016b)) and/or set X, of adaptively chosen queries to f (i.e., {xx f(x))
F()())) x E }) (Papernot et al. (201)) querries in Xi are not in the training data set Xn. These type of attacks exploit the transferability property of adversarial examples (Papernot et al. (2016b); Liu et al. (017)): Based on information Xo and X, the adversary trains its own copy of the proposed defense. This is called the adversarial synthetic network/model and is used to create adversarial examples for the target model. [Liu et al. (2017)) shows that the transferability property of adversarial examples between different models which have the same topology/architecture and are trained over the same dataset is very high, i.c., nearly 100% for ImageNet (Russakovsky al. (2015)). This explains why the adversarial examples
E } or {
4

---

Under review as a conference paper at ICLR 2020
generated for the synthetic network often be successful adversarial examples for the defense network.
Black-box attacks can be partitioned into three following categories:
Pure black-box attack (Szegedy et al. 2014); Papernot et al. (2016b); Athalye et al. (2018a): Liu et al. (2017)) The adversary is only given knowledge of training data set Ag- Based on this information, the adversary builds his classifier g which is used to produce adversarial examples using an existing white-box attack methodology. These adversarial examples of y may also be the adversarial examples of due to the transferability property between f and g.
Oracle based black-box attack (Papernot et al. (2017). The adversary is allowed to adaptively query target classifier f. which gives information 24 Based on this information, the adversary builds his own classifier 9 which is used to produce adversarial examples using an existing white-box attack methodology. Again, the generated adversarial examples for g may also be able fool classifier due to the transferability property between f and g. Compared to the native (pure) black box attack, this attack is supposed to be more efficient because g is intentionally trained to be similar f. Hence, the transferability between and may be significantly higher.
Zeroth Order Optimization based black-hox attack (Chen et al. (2017) )) The adversary does not build any assistant classifier as done in the previous black-box attacks. Instead, the adversary adaptively queries {ar, fx)) Fz)())) to approximate the gradient V based on a derivative free optimization approach. Using the approximated ff, the adversary can build adversarial examples by directly working with the classifier f.
In this paper, we analyze a mixed black-box attack where the synthetic network g is built based on the training data set Ag of the target model ʃ and is based on adaptively chosen queries Xi. Our mixed black-box attack more powerful than both the pure black-box attack and oracle based black-box attack. Supplemental material provides pscudo code.
As explained and motivated in the introduction, we restrict ourselves to the black-box setting where we keep secret the parameters of our defense classifier, called BUZz (see next section), and we do not reveal score vectors- this disallows white-box attacks and zeroth order optimization based black-box box attacks.
3 BUZZ: A DEFENSE BASED ON BUFFER ZONES
Design Philosophy. Since each single network gives a classifier with aligned boundaries (i.e. no buffer zones), we propose to combine multiple classifiers (each with its own aligned boundaries) to produce composed classifier which will provide a non-empty buffer zone. To create a buffer zone we use majority voting among the individual classifiers with a threshold mechanism. E.g., the threshold may be such that only if all individual classifiers agree the same label 1, then the composed classifier outputs 1, otherwise, it outputs 1. Because the transferability among all the individual classifiers not perfect, they will disagree if image has "too much' noise and this leads to an output L. The area where they disagree is the buffer zone
Although majority voting based on threshold leads the existence of buffer zone, the resulting buffer zone may not be wide enough to prevent a successful attack using small adversarial noise. If we are able to decrease/diminish the transferability among the different classifiers, then this leads to a wider buffer zone. To decrease the transferability, we must make the individual classifiers more unique. This can be done by, for each CNN network, first uniquely transforming the inputted image Jr. Since the transformations are different for each of the classifiers that participate in the majority
"Tee buffer concept offers first immediate insight: A defense with only one single classifier with buffer zones may be hard to develop because of nature a single network does not produce buffer zones between regions unless (1) another class label 1 for the buffer zone can be trained, but how does one construct proper training data? or (2) the score vector is used to mathematically define a subspace of score vectors that should map to I, but how one achieve acceptable clean accuracy at the same time? For this reason technique for creating non-empty buffer zone uses multiple classifiers.
5

---

Under review as a conference paper at ICLR 2020
I
c(r) = Ax +
Protected Layer
unjoority
#12
CNN
Prolected Layer
(a) Design of protected layer.
(b) BUZz: majority voting and threshold x.
Figure 2: General Design of BUZZ.
vote, transferability will be reduced. We will discuss what kind of transformations reduce the clean accuracy the least.
The created buffer zones may be mostly wide, but there will exist narrow parts. Since the narrow parts allow a successful attack based on small adversarial noise, the attacker will want to search for a narrow part around a given image x. order to make such a search less efficient and less often successful, the distinct transformations are kept secret. That is, only the distribution (or set) from which the transformations are drawn is known to the adversary (the concrete transformation selections/parameetee are kept secret). We will make the adversarial model explicit once have detailed our defense mechanism:
BUZz using multiple classifiers and secret image transformations. Our defense is composed of multiple CNNs as depicted in Figure 2b Each CNN has two unique image transformations as shown in Figure 2a The first is a fixed randomized linear transformation c(x) Ar b, where is matrix and d is a vector. After the linear transformation a resizing operation is applied to the image before is fed into the CNN. The CNN corresponding to and trained on clean data fi(c(z))) }. This results in a weight vector w. The m protected layers in Figure 2b are described by 'parameters' (ey, f w
When a user wants to query the defense, input x is submitted to each layer which computes its corresponding image transformation and executes CNN. The outputs of the layers are class labels (4)) The final class label of BUZz, i.e., the composition of the m protected layers, is a majority vote based on a threshold K. In our experiments we use unanimous voting, i.e., if the networks do not all output the same class label then the adversarial/undetermiin class label 1 is given as the output (i.e., K = m).
Adversaiaal Model. In our adversarial model assume that score vectors are revealed (which makes the attacker incapable of executing zeroth order optimization based black-box attack) and kept secret, i.e., the attacker has direct knowl-
assume that the parameters (c,, i Wy edge about the weights of the CNN networks, matrices and vectors b, and the amount of image resizing for each layer (this makes the attacker incapable of executing a white-box attack).
Our adversarial model allows a mixed black-box attacker having more capabilities (is stronger) than the strongest black-box adversary in literature: Just like in (Papernot et al. (2017)) the adversary is allowed to query the defense as many times as they desire and is practically possible, they may generate synthetic data Xi and they can train synthetic network. Based on this synthetic network they can carry out white-box attacks and test their efficiency (attack success rate) on the defense. We give the adversary one additional and extremely powerful ability that (Papernot al. (217)) does not. We allow the adversary access to the entire original training data set Xg as an initial set. This gives the adversary access to a huge amount of training data, an order magnitude higher than what (Papernot et al. (2017)) gives in the original attack.
In supplemental material A we mathematically formalize the adversary (as is done in crypto/security literature) an adversarial algorithm in order to make precise the adversary's capabilities.
Image Transformations. From (Guo et al. (017)) we know adversarial examples are sensitive to image transformations which either distort the value of the pixels in the image or change the original spatial location of the pixels. However, it is well established (Goodfellow et al. (2016)) that CNNs rely certain spatial patterns being present in the data in order to classify clean (non-adversarial) data. Hence, we want an image transformation that keeps such patterns 'invariant' while introducing distortions that make the attacker's task less likely to succeed.

---

Under review as a conference paper at ICLR 2020
In the literature, previous defenses with only a single network have suggested multiple different image transformations (Meng & Chen (2017); Xie et al. (2018)). Through experimentation we decided to employ two image transformations for each protected layer. The first transformation is simple resizing operation, where we resize the image before giving it as input to the CNN. Resizing to smaller dimension than the original image may result in lost pixel information and by extension hurt the network's 's performance on clean data. Therefore we only consider resizing operations which increase the size of the image. The other transformation we use is a linear transformation: c(z) Az + b where z is an n by n pixels input image, A is a fixed n by n matrix and b fixed by pixels 'noise" sample. Depending on the data set we can control the trade off between the attacker's success rate and clean accuracy using the linear transformation. For example if only b is random (with small magnitude) and is identity, it results in less image distortion (so higher clean accuracy) but also less security (more adversarial samples bypass the defense) |
Security A Argument. In the context of Papernot's oracle based black-box attack and pure black-box attack, the adversarial noise 17 is created based on white-box attack for synthetic network g of
BUZZ. It means that the noise 1) is specifically developed for g. Since the x' = x + 77 is inputted into every protected layer of BUZZ, the j-th layer will apply its CNN network on noisy image (xy)) = i + 17)). which due to the linearity of ē (((cg())) is equal to i (=)) + ty(C (1)). Therefore, layers receive different inputted noises (r(r)(1)) # 17- Hence, the protected layers have different behavior from one another and from synthetic network for any given adversarial example
x' x 17. This widens the buffer zones as is less likely that each protected layer reacts the same to in terms of clissslaasiiiiationn
4 EXPERIMENTAL RESULTS
In this section we diseuss our experiments for CIFAR- Supplemental Material [ has also exper- iments for FashionMNIST and CIFAR-100. Supplemental material C provides exact details in the form of pseudo codes and tables with attack parameters; below is a concise summary in words.
We implemented the mixed black-box attack, i.e., the sfate-of-the-art oracle based black-box attack of Papernot et al.] (2017) where in addition the adversary has access to the entire training data set. The attack first generates synthetic network by taking the initial training data set X6 and learning the parameters 8g of a single vanilla network g - for the training we use Adam (Kingma & Ba )2044) with learning rate 0.0001, batch size 64, 100 epochs, and no data augmentation. During the first iteration the Jacobian matrix of the score vector g is computed in each image x E Xg. The signs of the entries in the Jacobian matrix that correspond to x's class label (according to the target model) form a vector which is scaled with À = 0.1 and added to x. This leads to an augmented data set X' which consists of these new images together with Xg. We use black-box access to the target model to learn the labels for the images in N1. During next iterations we double the data set 5 times and learn a more accurate synthetic network g.
Second, the attacker uses Carlini's single synthetic network (Carlini & Wagner (2166) and ap- plies a targeted/non-targetee https: // thub. c tees/eenoorrroo wwwp////wwwwwaaaeeeeeeeeeee
iarative/non-tteraivee FGSM attack (we use the cleverhans library, see to produce an adversarial example x' for image x. (We do not use the C&W attack (Carlini & Wagner (2016)) because our experiments show a much lower attacker's success rate.) During each FGSM iteration the black-box ad adversary has the capability to use black-box access to the target model to verify whether the produced x' in that iteration has desired label L'. We use 10 iterations in iterative FGSM with 10/256 (giving a 1/256 scaling vector in each of the 10 iterations). In non-iterative FGSM we pick e = 1/20.
The CNN networks in the vanilla network and each of the protected layers of BUZz (the target model) each have the VGG16 architecture (see (https://neurohh) popular-networks /vg11 6 8) and are trained clean data (z)),, where 7 is the lo-be-leamed class label for I. BUZz uses anonymous voting with k equal to the number of net- works. The image transformations are selected from mappings c(x) x + b (we use the identity matrix for A) where the entries of b are non-zero with probability p 0.35 and non-zero entries
1olee.o//ee 1o/en/ /
'We note that the positive results reported in this paper open the door for the exploration of many more pos- sible image transformations. Other transformations such as the affine transformation. zero padding images and pixel shuffling could yield even better trade off hetween security and clean accuracy in multi model defenses. We leave these possibilities as future work.
7

---

Under review as a conference paper at ICLR 2020
Table 2: Mixed black-box attack on CIFAR-10
Vanilla
2-Nerworks
4-Newworks
NeNeworkss
Clean Accuracy
0,8355
0.1599
0.6832
0.6731
FGSM Targoted
0.831
0
0,985
0.995
IFGMM Targeted
0.766
FGSM Untargeted
0.297
0.797
0.931
0.966
1FGSM Untargeted
0.282
0.814
Table 3: 8 values for mixed black-box attacks on FashionMNIST, CIFAR- -O, and CIFAR-100
Detense targeted
6.
Detense untargeted
MashionMNIST CIFAR-10 10
4-Newworks
0.144
8-Netwerks
0.226
2 2-Neiworks
0.161
Netwworss
0.247
CIPAR
Vanilla
070223
Neiworks
0.216
are uniformly selected from the interval [-0.5, +0.5]. Images x are 32 x 32 x 3 where 3 stands for the red, blue, green values (each in [-0.5, +0.5) of a pixel. The same non-zero entry of b is added to each of the red, blue, green values of the corresponding pixel in æ. (We can think of an image transformation oj (z) as an extra randomly fixed layer added to the layers which form the j-th CNN.) We tested three BUZz designs: One with 8 networks each using different image resizing operation from 32 to 32, 40, 48, 64, 72, 80, 96, 104. The second with networks being the subset of the networks that use image resizing operations from 32 to 32, 48, 72, 96. The third with 2 networks being a subset of the networks that use image resizing operations from 32 to 32 and 104.
ddeiets the results of the four possible target/untargeted iterative/non -atetiiv/non--erativee FGSM attacks against a plain vanilla network without image transformation and BUZz with 2, 4, and 8 networks respectively. The defense rale is defined as the fraction 1000 test data (with the property that the target model produces the correct label) for which the adversary produces successful adversarial example z' that changes the label to desired label P' The clean rate and defense rate are an average over 5 runs where each run anew selects random image transformations in BUZz. The experiments show that the min and max values over runs are not far apart indicating that most random image transformation selections according to the recipe described above give similar results.
where is the attacker success rate computed as
For BUZz with 4 networks for CIFAR-I0 we see a 0.2003 drop in clean accuracy from 0.8835 to 0.6832. In return for this drop in clean accuracy we defense rate of 0.931, i.e., attacker's success rate < 7%. Also for FashionMNIST and CIFAR- 100 (for BUZz with 8 resp. 2 networks) see drops clean accuracy of 0.1577 and 7.3 in order to achieve an attacker's success rate of < 10% and < 9% (see supplemental material D This leads to the values reported in the introduction. Table shows the best BUZz configurations with targeted and untargeted 5 values.
CONCLUSION
We introduced a new concept called buffer zones which is at the core of new adversarial ML de- fense, coined BUZZ. BUZz defends against black-box adversaries with oracle access to the target model (BUZz) and knowledge of the entire training data set. BUZz uses threshold voting over mul- tiple networks that each are preceeded with a secret/hidden image transformation. Experiments for FashionMNISTT, CIFAR- 10, and CIFAR- 100 for carefully designed classes of image transformations in BUZz show that at the cost of drop in clean accuracy of 15-20% the attacker's 's success rate is only 5-10% much less than the best attacker's 's success rates 2 34% achieved by other well known de- fenses for these data sets. We have argued this to be acceptable trade-off for better security by using a new metric (called 8 value).
'One would expect lower defense rate for iFGSM attacks compared to FGSM attacks. This is not reflected in the table due to the choices of taken from literature which are different for iPGSM and FGSM. For future work we anticipate detailed sensitive study with respect
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated Gradients Give False Sense of Security: Circumventing Defenses to Adversarial Examples. ICML 2018, pp. 274- 24332 2018a.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing Robust Adversarial Examples. ICML 20/8, pp- 284-293, 2018b.
Xiaoyu Cao and Neil Zhenqiang Gong. Mitigating Evasion Attacks to Deep Neural Networks via Region-based Classification. CoRR, abs/1709.05583, 2017. URL httt:://paaxiiie org.abs//rg////////
1709.05583
Nicholas Carlini and David Wagner. Adversarial examples
easily detected: Bypassing detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, 2017a.
Nicholas Carlini and David A. Wagner. Towards Evaluating the Robustness of Neural Networks. CoRR, 2016.
Nicholas Carlini and David A. Wagner. MagNet and "Efficient Defenses Against Adversarial At- tacks" Not Robust to Adversarial Examples. CoRR, abs/1711.0847 2017b.
Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. 700: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks Without Training Substitute Models. In ATSec. ACM, 2017.
Reuben Feinman, Ryan R. Curtin, Saurabh Shintre, and Andrew B. Gardner. Detecting Adversarial Samples from Artifacts. CoRR, abs/1703.00440, 2017.
Ross B. Girshick. Fast R-CNN. In IEEE-ICCV, pp. 1440-1448,2011
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning The MIT Press, 2016. ISBN 0262035618, 9780262035613.
lan Goodfellow, Jonathon Shiens, and Christian Szegedy. Explaining and Hamessing Adversarial Examples. CoRR, abs/1442.6572,, 2014. URL
6572
Chuan Guo, Mayank Rana, Moustapha Cissé, and Laurens van der Maaten. Countering Adversarial Images using Input Transformations. CoRR, 2017.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint arXhv:1412.6980, 2014.
Alex Krizhevsky, Ila Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Con- volutional Neural Networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), NIPS, pp. 1097-1105. 2012.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial Machine Learning at Scale. CoRR, abs/1611.01236. 2016.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into Transferable Adversarial Examples and Black-box Attacks. ICLR (Poster), 2017.
Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. In CSS. pp. 135-147, ACM, 2017.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On Detecting Adver- sarial Perturbations. CoRR, abs/ Abs/1702..0426,, 2017. URL httph:::/axx...o///bbaa...
04267
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pp. 582-597. IEEF, 2016a.

---

Under review as a conference paper at ICLR 2020
Nicolas Papernot, Patrick D. McDaniel, and lan J. Goodfellow. Transferability in Machine Learning: from Phenomena to Black-Boo Attacks using Adversarial Samples. CoRR, 2016b.
Nicolas Papernot, Patrick D. McDaniel, Jan Goodfellow, Somesh Jha, Z. Berkay Celik, and Anan- thram Swami. Practical Black-Box Attacks against Machine Learning. In ACM AsiaCCS 2017, pp. 506-519, 2017.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In NIPS, pp. 91-99, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)21112255 2015.
Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully Convolutional Networks for Semantic Segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 39(4):640-651, April 2017. ISSN 0162- 8828.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. ICLR, 2015.
Siwakorn Srisakaokul, Zexuan Zhong, Yuhao Zhang, Wei Yang, and Tao Xie. Muldef: Multi-modellbased defense against adversarial examples for neural networks. arXiv preprint 00065, 2018.
Christian Szegedy, Wojeiech Zaremba, Ila Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. CoRR, abs//312.66 2013, URL htttp://arxiv.org//ee 1112. 6199
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Dan Boneh, and Patrick D. McDaniel. Ensemble Adversarial Training: Attacks and Defenses. CoRR, abs/1705.07204, 2017. URL http://
arxiv. org/ abs 705...00000 07204]
Jianyu Wang. Zhishuai Zhang, Cihang Xie, Yuyin Zhou, Vittal Premachandran, Jun Zhu, Lingxi
Xie, and Alan L. Yuille. Visual Concepts and Compositional Voting. CoRR, 2017.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. ICLR (Poster), 2018.
Xiaoyong Yuan, Pan He, Qile Zhu, Rajendra Rana Bhat, and Xiaolin Li. Adversarial Examples: Attacks and Defenses for Deep Learning. CoRR, ab//1/22.0000000 2017. URL httt/ttt:///axxv org/ exq/abs//e 0711100771070
10

---

Under review as a conference paper at ICLR 2020
A ADVERSARIAL MODEL
The strength of an attack is relative to the considered adversarial model. adversarial Machine Learning (ML) the assumed capabilities of an attacker are mix of:
Having knowledge of the parameters and architecture of the defense network/classifeer The architecture and methodology of how the defense network is trained is about how the defense operates and its underlying philosophy. In cryptography this is similar to the actual design of security primitive and this is never assumed to be hidden as this would lead to the undesirable practice of security by obscurity. The secret key of the security primitive kept private; the method of how the secrect key is generated using probabilistic algorithm is public. Similarly, the parameters of defense network can be considered to be private while the method of their training is public. If the adversarial model does not assume the parameters to be private, then all is public and we call this a white-box setting. the parameters are considered to be private and not given to the attacker (for 'free'), then call this a black-box setting.
Having access to the defense classifier a.k.a. target model. If the parameters are kept private, then the next best adversarial capability is having access to the target model black- box. The idea is that the adversary can adaptively query the target model with own input images for which class labels are being returndd. Here, we have two flavors: only class label is returned, or more information is given in the form of a score vector which entries represent confidences scores of each of the classes (the returned class label the one with maximal confidence score).
In the white-box setting where all parameters are known, the attacker can reproduce the target model and access the target model black-box. Confusing in adversarial ML is that white-box attacks are the ones that 'only' use the parameters in the white-box setting learn gradients of the target model which are used produce adversarial examples these attacks do not consider/use black-box access. This means that white-box defenses are not necessarily analysed against black-box attacks where the adversary only has black-box access to the target model with possibly the added capability described below.
Having access to (part of the) training data. The training data which is used to train the parame- ters of the defense network is often public knowledge. Knowing the methodology of how the defense network is trained, an adversary can apply the same methodology to train its own synthetic defense network and this can be used find adversarial examples. The synthetic network will not be exactly the same as the defense network since training is done by randomized (often SGD--ike) algorithms where training data is used in random order. This means that knowledge of the training data is less informative than knowledge of the parameters as in the white-box setting.
The white-box setting describes the capabilities of the strongest adversary, while the black-box setting describes a weaker adversary who cannot exactly reproduce the target model (and can only estimate the target model by training a synthetic network). White-box attacks on the other hand restrict the adversary in that only oracle access to the gradient of f allowed. Black-box attacks only allow oracle access to the target model itself and oracle access to training data. In this sense white-box attacks adversarial ML literature exclude access to the above black-box oracles. This means that even though white-box defense may be able to resist a white-box attack, it can still be vulnerable to a black-box attack. Vice versa, even though a white-box defense may be broken under white-bos box attack, it may still survive in the black-box setting.
Taking BUZz as an example, we may mathematically formalize the black-box adversary (as is done in crypto/security literature) as adversarial algorithm Ar which has access
a random oracle representing / <- BUZza (x) where parameters 0 = (cjy ij w; ) input or is an image, and / the outputted label of the target classifier BUZz (the collected (x,, 1) is the set 2, of Section 2). and
"Similarly white-box adversary with only oracle access gradient information can be modeled.
11

---

Under review as a conference paper at ICLR 2020
random oracle ED which outputs most D times 'training data' according to the distribu- tion from which the training data is taken from (the collected (æ, is the set Xo of Section by abuse of notation { denotes the distribution itself.
Subscript T denotes the allowed number of computation steps plus oracle accesses to BUZZ, and ED. In our experiments we test the most powerful existing black-box attacks and do not mention T; here, T just means the amount of steps allowed by existing practical attack methodologies. If an attacker with unlimited access (T = oo) to BUZza can scan region boundaries, then this achieves optimal success rates.
Subscript D in En indicates the number of training data an attacker is allowed to use for the attack. D represents an important metric in machine learning as the amount of training data cannot be assumed infinite (with respect to the application there is a concrete limit to how many training data available; collecting samples from È not straightforward. e.g., making true new picture/image of a plane takes effort). How strong we make e.g. Papernot's black-box attack is based on how much training data we give it: In their paper D 150 used for MNIST in order to train a synthetic network while in our experiments we use D = 50K which is the entire original training data set of CIFAR-IO (and leaves 10K test data). Since the attack uses the synthetic network in a (targeted or untargeted) white-box attack (with small enough e) to generate adversarial example, the probability of successfully changing the label depends on how similar the synthetic network classifies data compared to the target model with defense it depends on the tranferability between the synthetic and defense classifiers and transferability improves for larger D.
The aim of the adversary is produce perturbation 1) f AEp.BUZze (x) (just based on the oracle accesses described above indicated by superscripts and based on input x) which is visually imperceptible, i.e. ||||| S c, and for which x' x + 1) gives different label: The attacker's success rate for untargeted black-box attacks is defined as the probability
Prz A BUZZa (x), ||lll S e, BUZza + v) & (1,ZUZZ BUZzz 1).
For targeted black-box attacks we replace BUZzg(I+1) & BUZZ by BUZz, (x+ 1', replace BUZza(2) * 1 by (UZ))( & {1,7"), and take the probability over both x + & and I' - [1..A]. In the above notations we do not explicitly state that the adversary also has knowledge of the distribution from which 0 is taken, i.e., the adversary knows the philosophy behind our defense together with what type of image transformations are being used and knows the architecture in terms of number of nodes and connections at each layer of the CNN networks and how they trained.
The above formalism helps in making the adversarial model in terms of adversarial capabilities precise. We will not explicitly use the formalism as we cannot prove statements about general classes of adversarial algorithms AA (our defense does not allow "standard' reduction proofs to some hard computational problem as is done in crypto).
From a cryptographer's perspective we want the above probability (i.c., the attacker's success rate) to be very small, even negligible in some security parameter À where image is (o)s(A))sized. First, in ML we have concrete data sets with images of certain fixed sizes for which we want to design defense strategies against adversarial ML. So, such asymptotic goal makes no sense. Second, it turns out that it is very difficult to obtain a defense strategy that can make the attacker's success rate very small, say 0. 0.1% without sacrificing the clean accuracy all together. In this paper we have been able to reduce the attacker's s success rate down to about 5-10% while only reducing the clean accuracy by 15-20%
B RELATED WORK: COMPARISON TO KNOWN DEFENSES
White-Box Defenses. White-box defenses are any defense with an adversarial model that allows
the adversary oracle access to the gradient the target model. These defenses include (Papernot ct al. (201 Kurakin et al. (2016); Tramer al. (2017); Cao & Gong (2117) Metzen et al. (2017) 7)::
Feinman et al. (2017): Xie et al. (2018) ; Meng & Chen 2017); SSisskkkkk et al. (2018)) to name a few. A complete list is given in Athalyy ye et al. (2018a); Carlini & Wagner (201 (2017a)) except for the
unpublished work (Srisakaokul et al. (2188) which appeared later. So far, any defense with public
weights and architecture turns out to be vulnerable FGSM, IFGSM, or Carlini type attacks (Carlini
12

---

Under review as a conference paper at ICLR 2020
& Wagner 22017a b): Athalye et al. (2018a); Liu et al. (0117); we will argue the vulnerability of Srisakaokul et al. (2018)) below).
In order to implement white-box attack (Yaan et al. 2017) constructs perturbation 7 with the help of gradient f.(:): for example, 7) = e x sign( V L(x 1; 8) in the Fast Gradient Sign Method (FGSM) by (Goodfellow et al. (2014)), where I represents the parameters of f. L is loss function (e.g, cross entropy) of model f. (c can be thought of as relating to the maximum amount of noise which is not visually perceptible.) To defend against adversarial examples, many methodologies have been proposed and they all employ the same strategy, i.e., gradient masking (Papernot et al. (2017) respectively obfuscated gradient (Athalye et al. (2018a)). As pointed out in (Athalye et al. 20188)), ) there are three main methods for realizing this strategy: shattered gradients, stochastic gradients and exploding vanishing gradients. In (Athalye et al. (2018a)), the authors propose three different types of attacks:
1. Backward Pass Differentiable Approximation (BPDA). The attack is applied for pro- tected network f(((z)) where t(z) is not differentiable and t(x) = x. The adversary will replace f(x:) in the backward phase for computing the gradient by and thus, he can com- pute the approximated gradient (t= R (=)(()))
2. Expectation over Transformation (EOT). In this case, the adversary computes the gradi- ent of Ex-r f (t(z)) where t(z) is a random transformation and is sampled from a distri- bution T. The gradient can be computed as ((tz)) E ((t(z).
3. Reparameterization. The protected network f(((x)) has t(z) which performs some opti- mization loop to transform the input x to a new input 4. This step will make the gradient exploding or vanishing, i.e., the adversary cannot compute the gradient. To cope with this defense, Athalyy et al. (2018a) proposes to make a charialof-varrialle x = h(z) for some (t()) such that ((h(z)) h(z) for all but f(-) is differentiable.
In literature, many white-box defenses have shown a predictable cat and mouse type of security game. In this repeated chain of events, the defender creates a network defense and the attacker comes up with new type of attack that breaks the defense. The defender then creates a new defense which the attacker again breaks. While this occurs frequently in security, a simple example of this occurring in the field of adversarial machine learning is the FGSM attack breaking standard CNNs. the distillation defense mitigating FGSM, and the distillation defenses subsequent break by (Carlini (Papernot et al. (a): Carlini Wagner (2017))). Alternativeyy, in an even worse case, the defense can be immediately broken without the need for new attack strategies. In adversarial machine learning an example of this is the autoencoder defense of (Meng & Chen (2017)) which is vulnerable to the attack in (Carlini & Wagner (2017b)) From our analysis of the previous literature it is clear that a secure pure white-box defense is extremely challenging to design.
Black-Box Defenses based on Single Network. We discuss how the white-box defenses of (Pa- pernot et al. ; Kurakin et al. (2016); Tramer et al. (2017); Cao & Gong (2017); Metzen et al. (2017) Feinman et al. ( et al. 20188);Meng & Chen (2017)g Guo al. (2017 Srisakaokul et al. (2188) are vulnerable in a black-box setting. As shown in (Papernot et al. (201)). 7), the adver- sary can build a synthetic network y which simulates the target vanilla network. This can be used to produce high transfcrability adversarial examples (that transfer to the target model with significant success). Boundary alignment is the well-known explanation, see (Papernot et al. (2016b)).
PPapernot et al. (2016a)) proposes a single network defense with a better adversarial robustness property based on distillation: First, given a training data set, a network is built and trained. After this, the softmax output (i.e., score vector) of the network is used to train another network with the original training data set. This process called 'distillation' and the distilled network argued to have better robustness against white-box attacks. However, (Carlini & Wagner (2016)) showed a white-box attack against this defense. Moreover, (Papernot et al. ((017)) showed that for the MNIST dataset, the success rate of Papernot's black-box attack (untargeted) is at least 70%.
In (Kurkkin et al. (2166)), the authors discuss how to train the network for ImageNet with adversarial examples to make it robust against adversarial machine learning. During each epoch in the training process, adversarial examples are generated and again used in the training process. According to Table 4 in (Kurakin et al. (216)), the success rate of (untargeted) pure black box attack ImageNet using FGSM is high 2 50%. The authors in (Tranèr et al. (2017)) also claim that the adversarial training in (Kurakin et al. (2016)) may not be useful.
13

---

Under review as a conference paper at ICLR 2020
Tramer et al. (017)) proposes another type of adversarial training method. The adversarial ex- amples are generated by doing attacks on different networks with different attack methods. After this the designer trains the network with the generated adversarial examples. The authors gued that this adversarial training can make the adversarially trained network more robust against (pure) black-hox attacks because it is trained with adversarial examples from different sources (i.e., pre-trained networks). In other words, the network is supposed to have better robustness against black-box attack generalization across models. As shown in (Athalye et al. (2018b)), the adversari- ally trained network is vulnerable to white- box attack. Regarding pure black-box attack, as reported in Table in (Tramer al. 2017)), the success rate of (untargeted) pure black box attack on Ima- geNet using FGSM the best known black box attack that has been executed on this defense is 2 27%. We verify the efficiency of this approach for CIFAR-10 this paper: We do the adversarial retraining using data from 8 other networks to build 9th network. The 8 other networks are from the Mul-Def paper Srisakaokul et al. (018)) (we also rigorously discuss this paper next few para- graphs). Network 0, is a vanilla VGG. Network is VGG with 30% adv training from network O. Network 2 is a VGG with 30% adv training (15% from network O, 15% from network 1). Etc. until get to Network 8. After training we run the full Papernot attack on Network 8. The full result we can find in Table i.e., Adv Retrained I I-Net. The defense has clean prediction accuracy of 85% and the best attack on the defense untargeted FGSM or iFGSM with success rate of 34%.
Cao & Gong (017]) proposes a white-box defense based on the following trick: for a given sample 2, the defense collects many samples z'i.... 2y, in a small hypercube centered at z. Then, the outputted class label is the one which gains the majority vote among where F' is the output function of network f. We argue that this defense is vulnerable to black- box attacks because of the following reasons. The adversary can build a synthetic network g with very high transferability between f and g (Papernot et al. (2016b); Liu et al. (2017)). After this, the attacker looks for adversarial examples which can gain the majority vote in the same setting as proposed above in (Cao & Gong (2017). We believe that there exist many such adversarial examples, in particular. if the clean x is very close to the decision boundary, then the adversarial example x' = x + 17 can lie deeply in another region and its distance to the decision boundary is larger than that of x. Hence, x' now gains the majority vote and fools the defense. This example also shows the importance of our buffer zone concept as a defense mechanism.
In (Metzen al. 2017) ), the authors constructed a defense of a single network with an additional 'detector' network g. The 'detector network' is built based on the training data set of the main network f together with adversarial examples generated for training data samples. The detector network is used to distinguish clean samples from adversarial samples. The authors in (Carlini & Wagner (2017a)) showed white-box and black-box attacks on this defense. The success rate of untargeted pure black-box attack on MNIST using the C&&W attack by (Carlini & Wagner (2016)) is at least 84%.
In (Feinman et al. 2017)) the authors built a detector to distinguish adversarial examples from clean examples using Bayesian uncertainty estimate or Kernel Density Estimator. The key idea is that since the adversarial and clean examples do not belong to the same manifolds, the defender can build a detector. (Carlini & Wagner (2017a)) showed a white-box attack on this defense and clearly claim that the detense does not work if the dataset is CIFAR-10 for both white-box attack and black-box attack, i.c., the success rate of untargeted pure attack seems at least 80% based on their explanation.
Table 4: Mixed black box attacks on defenses Randomized I-Net (ie et al. 1-Net TTamer et al. (2017)). Mul-Def 2,4,8-Net (Srisakaokul et (2018). Mixed Arch 2-Net (Liu et al. (217)) and BUZz 2, 4, Net.
20 8 ), Adv Retrained
Clean Accuracy
FGSM largeled
IFOSM targeted
FcSM unlargeted
IFGSM unturgeted
Randomized 1-Not (Xio et al. 2018)
0.58
0.83
0.72
0.52
0.13
Adv Retrained I-Not [Trnnr 207))
0185
0193
0.93
D166
0.67
0.73
0.26
Mul Nel Srisakaokul et 2018)
0.86
0.85
0.82
0.44
0.39
Mixed Arch 2:Net (Luu
2017)
0.82
BUZs 2-0
0.75
095
0.97
0.2
0.83
Mul-Def Net aokil
20188)
0.54
BUZZ Not
0.68
0.98
0.99
0.3
0.96
Mol-Teef Net (Skikakakkul
2018)
0.85
0.92
0.87
0.59
0.50
BUZZ 8 Net
0.61
0.99
1.00
0.97
0.98
14

---

Under review as a conference paper at ICLR 2020
Clean Accuracy
FGSM Targeted
IFGSM Targeted
FGSM Untargeted
IFGSM Untargeted
0.9
0.8
Accuracy
0.5
0.2
Randomized Adv Retrain
Buzz 1- Network
Mul-Def 2-
Mixed Arch
BuZz 2-
Mul-Def 4- Networks
BuZ7 4- Networks
Mul-Def 8-
BuZb
Networks
Figure 3: Mixed black-box attack defenses Xie al. (2018) (randomized network), Guo et al. (2017) (mul-sec network), and Srisakaokul et al. aaaaal2010 (218)) (mul-def 2, 4, and network) for CIFAR- 10. The setup of the attacks for this experiment is the same the attacks in Section 4 and Supple- mental Materials[C and
(Xie al. 2018)) has single network and uniformly selects image transformation from a-priori fixed set of a small number of image transformations to defeat white-box attacks. In the white-box setting (Athalye et al. (2018a)) shows that this defense does not work. But is this de- fense secure against black-box attacks? To maintain sufficiently high clean accuracy, the random image transformations should not have high randomness. Hence, the boundaries of any single net- work/classifier and the network/classifier with one of the random image transformations may be highly aligned. This implies that the adversarial examples created for any classifier will likely trans- fer to the network with randomization operations. This is confirmed by experiments reported in the column "Randomized Network" Figure 3 We can see that the defense accuracy (i.e., clean ac- curacy for the defense model) drops to 0.59 and the attacker's ss success rate of the untargeted mixed black-box attack (using iFGSM) is 0.865 (the table entry depicts the defense rate 035 defined as minus the attacker's success rate).
Similarly, the defense proposed in (Meng & Chen 2017) defense with a single network and multiple different auto auto-encoders image transformations from which one is selected at random per query ii is not secure against pure black-box attacks, i.e., the success rate of targeted pure black-box attack the defense using C& W attack for CIFAR-10 and MNIST is at least 99%, (Carlini & Wagner 2017b)).
In (Guo al. (2017)), the designer selects a of possible image transformations for single net- work and keeps the selection of the chosen image transformation secret. The image transformation will distort the noise as explained in (Guo al. (2017). ). This is BUZz for a single protection layer (without multiple networks and threshold voting). However, there is no buffer zones for any sin- gle network and thus, there exist many adversarial examples with small noises. We have validated this claim for BUZz with single protection layer because it is very close to the one in (Guo et al. (2017). In column "BU7z 1-Ner" in Figure the best attack untargeted one and the success rate for CIFAR-10 is 77% (corresponding defense rate 0.226).
Black-Bac Box Defenses based on Multiple Networks. (Liu et al. (017)). the authors study the transferability between different networks which have different structures for the ImageNet dataset. The authors report that the transferability between the networks is small (claimed to 'close to zero'). For this reason, it may be possible to have the BUZz defense where protected layers represent different networks with different architectures. However, experimentally we have built a defense with VGG16 and Resent20 trained on CIFAR10. As reported in Table 4 and Figure 2 the 2-net
15

---

Under review as a conference paper at ICLR 2020
BUZz with 2 VGGI6 and 2-net BUZz with VGG and 1 Resnet (Mixed Arch 2-Net) have the same performance. It implies that having different architecture does not give us any advantage in BUZz defense.
In unpublished work (Srisakaokul et al. (2018) ) the authors have proposed a defense against white- box attacks based on multiple networks with the same architecture. The authors develop their de- fense based on a retraining technique. First, the authors apply adversarial attacks on each network to generate a set of adversarial examples. For example, for each network, j a white-box attack produces a set of adversarial examples Sj. Next network j will be retrained with the clean training data set together with some of the adversarial sets Sn. + j. The authors argue that all the networks cannot be fooled at the same time for a given adversarial example and this leads to a low(er) attacker's s success rate. The final outputted class label is the predicted label of one of the networks chosen at. random among all networks; this gives high clean accuracy.
With respect to white-box attacks, the defense in (Srisakaokul et al. (2018)) seems not secure: For verifying resistance against white-box attacks, (Srisakaokul et al. 2018)), only attacks each model separately instead of attacking all the models at the same time as is done Ensemble Pattern Attacks in Section 4 in (Xie et al. (2018). Hence, the authors should do the Ensemble Pattern Attack on their defense to have a completed claim on white-box resistance.
For testing resistance against Papernot's black-box attack, the authors only work with an initial set of 150 samples and 5 runs. This gives an augmented set of only 24 150 = 2400 samples used for building the synthetic model (compared to augmented set of 2- samples in our experiments). Hence, the performance of synthetic model g is very poor and as a result a lower attacker's success rate. Nevertheless, even with poor synthetic network (that is, very weak black-box adversary) the reported success rate of the Papernot's attack with FGSM is still high, i.e., around 18% /27% for MNIST and CIFAR- 10 (see Table 5 in (Srisakaokul et al. (2018)). We performed experiments for our strong mixed black-box adversary and found an attacker's 's success rate of 50% for the best attack on the best defense (Mul-Def' 8 Network in Figure
We summarize the attacker's s success rate of the black-box attacks on the aforementioned defenses in Table 5 In the table the original accuracy denotes the clean accuracy for the classifier without implemented defense and defender accuracy denotes the clean accuracy with implemented defense. We can see that all defenses do not want to sacrifice clean accuracy p - generally at most a drop of about 5%, see (Meng& & Chen (0177), with larger drop of 25% in (Xie et al. (2018). As result of aiming at keeping the clean accuracy the same, the attacker's success rate & remains very high, typically > 0.75 for experiments with MNIST, 110AR-10. and CIFAR-100 as in this paper with (Tramer et al. 2017); Srisakaokul et al. 2018)) being the exception with about 50%. These high attacker's s success rates give drop in the effective clean accuracy of 8 p (p 7) a) > p- p1 - 0.5) = 0.5p; the 5 values are listed in the table. We do not differentiate the 8 values for targeted resp. untargeted attacks. We see that ô- 0.29, 0.43 for (Tramèr et al. (2017): Srisakaokul et al. (018)) and 5 > 0.65 for other defenses.
For ImageNet, not studied in our paper, we have 8 equal to 0.211 in (Tamer al. (017 and 0.396 in (Kurakin et al. (0166)) albeit for the weaker pure black-box box adversary (without access to the entire training data set). First, for the stronger adversary analysed in this paper we expect these 5 values to become higher. Second, since Tramer et al. (2017) has (compared to 5 = 0.247 for BUZz) a higher = 0.29 for the stronger adversary for CIFAR-10, we expect to also see this reflected in smaller â for ImageNet for BUZz compared to Tramer et al. (2017) (after fine tuning image transformations). We leave exact experiments for future work. Third, even if BUZz would do similar when comparing 8 to Tramer et al. (2017) for ImageNet, BUZz would still be the better choice: This is because even though the 8 values are the same, the attack success rate for BUZz is expected to be 5-10, much less than 27% of Tramèr et al. (2017) for the weaker adversary. This means that there is much less adversarial control in BUZz compared to Tramer et al. (017). (This has already been shown to be true for CIFAR-10 in Table
C PSEUDO ALGORITHMS: BLACK-BOX ATTACK & BUZZ
Synthetic network. Algorithm I depicts the construction of a synthetic network g for the oracle based black-box attack from Papernot et al. (2017). The attacker uses as input oracle O which
16

---

Under review as a conference paper at ICLR 2020
Defause
Dasa
Attack
Ari. suscess rile
Or. Accusscy
Del. Accunccy
Lomar
InugeNar
Func BX
Timmss 2017
76% (mmageNas
75% (EangeNen)
Kirakor
ImageNer
Pure DBV- TGSM
15 15.50mmapeNe
77.855 (lmageNe))
0.396
Tormer ct
2017
CIFAR-TO
Mixel RR -FGSM
35% (thes
55% (CIFAR-TO) 1o)
86% (CIFAR-10)
0.39
2018
CIEAR
Mined DG TFGSM
pipen
SEX ICIDAR T0)
(CIVAR
CI
BB -IFGSM
SIS ICIPAR
(CIFAR--10)
Wagner
2017a)
$2.0% (CTRAR-10)
Mised
Shazen papei
SEW (CTPAR TO
197
Ming Likcn
2017
MNIST & CIPAR-TO
Punc -CC
5973
Carlin Wagnet 20170
WOOS (CIFAR-10)
80.555 (CIFAICTT)
Papemunt sl.
Milna
MINIST
Oracle Mi8 FOSM
7VX
Prrennt al
1017
BUSTT (MINST))
18.10% (MNIST)
0.00
MINST
BB CKW
84%
70
91.8% (MINST))
91.3% (MNIST)
Table 5: Attacker's success rate of black-box attacks for state-of-the-e defenses
represents black-box access to the target model f which only returns the final class label F(((z) for a query x (and not the score vector f (x)).. Initially, the attacker has (part of) the training data set X, i.e. he knows D ((x, F(f(=)) x E Ko} for some X, C X. Notice that for a single iteration N = Algorithm I therefore reduces to an algorithm which does not need any oracle access to O; this reduced algorithm is the one used in the pure black-box attack (Carlini & Wagner (2017b); Athalye et al. (2018a); Liu et al. (2077). In this paper we assume the strongest black-box adversary in Algorithm| for evaluating the attack success rate).
I with access to the entire training data set Xo X' (notice that this excludes test data
In order construct synthetic network the attacker chooses a-priori substitute architecture G for which the synthetic model parameters Og need to be trained. The attacker uses known image- label pairs in D train 0g using training method M (e.g., Adam (Kingma & Ba ()) In each iteration the known data is doubled using the following data augmentation technique: For each image in the current data set D, black-box access the target model gives label Ozx). The Jacobian of the synthetic network score vector g with respect to its parameters 6g is evaluated/computel for image x. The signs of the column in the Jacobian matrix that correspond to class label I are multiplied with a (small) constant À this constitutes vector which is added to x. This gives one new image for each x and this leads to a doubling of D. After N iterations the algorithm outputs the trained parameters lg for the final augmented data set D.
Algorithm 1 Construction of synthetic network in Papernot's oracle based black-box attack
1: Input:
2:
O represents black-box access to F(f()) for target model f with output function F: Ng X' where X' is the training data set of target model f; substitute architecture training method M; constant X;
3:
4:
5:
6:
7:
number N of synthetic training epochs
8: Output:
9:
synthetic model g defined by parameters 6, (g also has output function F which selects the max confidence score; g fits architecture G)
10:
12: Algorithm:
14: for N iterations do
D ((x, ((r)) x E X.)
16: Og M D)
17: X++1 + {o + sgn(Ja, (z)(O())) x e X,) U
18: end for
19: Output 6g
The precise set-up for our experiments is given in Tables 6 7 and T the used training method M in Algorithm For the evaluated data sets Fashion MNIST, CIFAR-10, and CIF without data augmentation, we enumerate in Table the amount [X) of training data together with parameters À and N in Algorithm (A 0.1 and N = 6 are taken from the oracle based black-box attack paper of Papernot et al. (2017): notice that a test data set of size 10.000 is standard practice; all remaining data serves training and this is entirely accessible by the attacker).
17

---

Under review as a conference paper at ICLR 2020
Table 8 depicts the architecture G of the CNN network of the synthetic network g for the different data sets; the structure has several layers (not to be confused with "protection layer' in BUZz which is an image transformation together with whole CNN in itself). The adversary attempts attack BUZz and will first learn a synthetic network with architecture G (used as input Algorithm T that corresponds to Table X Notice that the image transformations are kept secret and for this reason the attacker can at best train a synthetic vanilla network. Of course the attacker does know the set from which the image transformations in BUZz are taken and can potentially try learn synthetic CNN for each possible image transformation and do some majority vote (like BUZZ) on the outputted labels generated by these CNNs. However, there are exponentially many transformations making such an attack infeasible. For future research we will investigate whether small sized subset of "representative" image transformations can be used to generate a synthetic model which can be used to attack BUZz in more effective way. Nevertheless, we believe that BUZz will remain secure because the security argument given in Section B where is shown how a single perturbation 7) leads to very different perturbations at each protected layer in BUZz. This leads to 'wide' buffer zones and their mere existence is enough to achieve our security goal - security is not derived from keeping the image transformations private. Keeping these transformations private just makes it harder for the adversary to construct a more effective attack but the resulting attack is expected to still have small attacker's success rates. We leave this study for future work.
Table 6: Training parameters used in the experiments
Training Parameter
Value
Optimization Method
ADAM
Learning Rate
0.0001
Batch Size
64
Epochs
100
Data Augmentation
None
Table Papernot Black-Box Attack Parameters
Xo
N
X
Testing set
CIFAR-IO
50000
6
0.1
10000
CIFAR- 100
50000
0.1
10000
Fashion MNIST
60000
0.1
10000
Table 8: Architectures of synthetic neural networks y from (Carlini & Wagner 2017a
Layer Type
Fashion MNIST and CIFAR-10
CIFAR- 100
Convolution + ReLU
3 x 64
3 x 64
Convolution + ReLU
3 x 3 x 64
x 3 x 64
Max Pooling
2
Convolution + ReLU
3 x 3 x 128
3 3 x 128
Convolution + ReLU
3 x x 128
x 128
Max Pooling
2 2
2 x 2
Fully Connected + ReLU
256
256
Fully Connected ReLU Softmax
256
256
10
100
White-box attack on the synthetic network. The targeted iterative Fast Gradient Sign Method (FGSM) of (Goodfellow et al. (2014)) is given in Algorithm 2 (for our implementation we use the cleverhans library, see patpps:n
The non- iterative variant has outer loop size H = 1. For untargeted attack no adversarial label I' is given as input and the loss function L is only a function of x. I, and 6,, (the loss function L should be properly defined to make the attack targeted attack untargeted). The algorithm walks in H iterations along
com/ wwom/teesorffecc http://ennrrooo/c/eeeaaaaaat
18

---

Under review as a conference paper at ICLR 2020
the gradient towards the boundary of x's region (where region with label I' should start). The maximum perturbation will have entries in the interval [-c, +e] (since each of the H iterations at most add e/H).
When Algorithm 2 is used as a pure black-box box attack, then no oracle access is available and com- parison I' (x) is replaced by comparison 1' = F(g(x)), which uses the synthetic network. In this paper we assume the stronger black-box adversary who has oracle access in Algorithm 2
Algorithm 2 Targeted iterative Fast Gradient Sign Method
1: Input:
2:
O represents black-box access to F(((-) for target model f with output function F; parameters 0g of synthetic network g:
3:
4:
loss function L for the attack, 5: threshold >
6:
outer iteration size H, benign image x with true label 1, adversarial target label I'
7:
8:
9: Output:
10:
adversarial example x'
11:
12: Algorithm:
13: for H iterations do
14: 3
(e/ H) sgn V (x,,,,,,,
15: if' L' (z) then
output x
17: break;
end if
19:
end for
In (Carlini & Wagner (016)), the authors propose a powerful non-iterative attack (coined C&W attack), which can also be used to produce adversarial examples with small noise. The authors use some optimization method OPT (e.g Adam (Kingma & Ba (2014)) to find n which minimizes 8g) for a untargeted attack (or 1nlp L(x + +n. I, I', fg) for a targeted attack); the output of OPT is x. x +7. Here, -1p denotes the p-norm, L is the objective function defined in Carlini & Wagner (2016) for given synthetic network with parameters 0,. We do not use the C&W algorithm of (Carlini & Wagner (2016) because it achieves lower attacker's success rate compared to iterative FGSM in our setup: For example, the C& w untargeted attacker's success rate for 1000 samples only 0.006 on our Network BUZz CIFAR- 10 defense. This translates to an overwhelmingly large 99.4% defense rate. Contrast this to the attacker's s success rate based on the untargeted iFGSM for our 2-Network BUZz defense which is 0.186 0.006 (see Table ?
|| + c- L(x + n,, 7,),8,)
Table 9: Parameters for iFGSM and FGSM in the mixed black- box attack on BUZz for CIFAR-10, CIFAR- 100 and Fashion MNIST
Cuntargeted
H Huntargeted
Etargeted
Htargeted
CIFAR- 10
10/256
10
1/20
10
CIFAR-
10/256
10
1/20
10
Fashion MNIST
0.1
10
0.3
10
We enumerate in Table the parameters and H used in our experiments we have taken these from literature (Meng & Chen (2017)). Notice that literature often reports </ as the step size in FGSM while we list values for which corresponds to the size of the final perturbation that leads to the adversarial example (and as such we can interpret e as the threshold for noise being visually perceptible by the human eye).
We use the library httpp::/gghe http:///oo.iooooooooo/////////// ot implement the
C& W attack.
19

---

Under review as a conference paper at ICLR 2020
Success rate black-box attack. In order implement the black-box attack we first run Algorithm I which outputs the parameters of a synthetic network g. Next, out of the test data (each data set has 10,000 samples in our set-up) we randomly select 1000 (this is custom in literature) samples (z, () which the target model f (i.e., BUZz in this paper) correctly classifies. For each of the 1000 samples run Algorithm@ produce 1000 adversarial examples. The attacker's success rate is the fraction of adversarial examples which change / to the desired new randomly selected l' in a targeted attack or any other label (' for an untargeted attack.
Image transformations for BUZz. In the BUZZ, we use image transformations that are com- posed of a resizing operation i(x) and linear transformation c(x) Ax + b. An input im- age I at a protected layer in BUZz is linearly transformed into an image i(c(x)) before it en- ters the corresponding CNN network with VGG16 architecture for CIFAR-10 and CIFAR-100 www.:/o///nnooooe wwwpee/poplaaeeteteeoee net /vggl 6/8) or minivgg architec- ture for FashionMNIST (see TODO). In network implementation one can think of i(c(x)) as an extra layer in the CNN architecture of VGGI6 itself (here 'layer' should not be confused with the terminology protected layer').
(see htt.t::/t roh ve ve. io/
For the resize operations i(-) used in each of the protected layers in BUZz, we choose sizes that are larger than the original dimensions of the image data. We do this to prevent loss of information in the images that down sizing would create (and this would hurt the clean accuracy of BUZz). In our experiments we use BUZz with 2, 4, and 8 protected layers. Each protected layer gets its own resize operation i(.). When using 8 protected layers, 32 to 32, 40, 48, 64, 72, 80, 96, 104, Each protected layer will be differentiated from each other protected layer due to the difference in how much resizing each layer implements. This will lead to less transferability between the protected layers and as a result we expect to see a wider buffer zone which diminishes the attacker's 's success rate. When using 4 protected layers, we use a copy of the protected layers from BUZz with 8 networks that correspond to the image resizing operations from 32 to 32, 48, 72, 96. When using 2 protected layers, we use a copy of the 2 protected layers from BUZz with 8 networks that correspond to the image resizing operations from 32 to 32 and 104. In our implementation we use resizing operation from github battps://githi: Batp:___ddcchhaaale
image resizing operations from
com/ cihangxie/
Xic et al. (2018)).
For each protected layer, the linear transformation c(x) - Ar + is randomly chosen from some statistical distribution (the distribution is public knowledge and therefore known by the adversary). Design of the statistical distribution depends on the complexity of the considered data set (in our case we experiment with FashionMNIST, CIFAR-10, and 110) Transformation c(x) takes an image of size n1 X 02 X3 as input and considers this as a vector of length k = nn 12233 Here, n1 and n2 denote the horizontal and vertical width in pixels of image x; n3 3 means that each pixel has a red, blue, and green values; f13 = means that each pixel only has one black/white value. CIFAR-10 and CIFAR-I00 have 32 x 32 3 images and FaShionMNIST has 28 28 images. All the values in vector are converted from integers [0..255] to the range [-0.5, +0.5| of real numbers. Notice that the entries of c(x) may have their values outside of this range.
In our implementation we do not consider to be in vector representation; we think of x as ny times a 01 X ng matrix. For example, x = (X1, X2, Xz) for ng- = 1. We restrict e(x) = Ar + b to linear operations
c( c(X,, X2) AX + b, X²A2 + b2, X²Ag bz).
where A; are n2 X n2 matrices and b; are n x n2 matrices.
For CIFAR-10 10 and CIFAR-100 we take matrices A, be identity matrices (this also makes A the identity matrix in the vector representation of c(z)) and we use the same matrix for each of the matrices 6;, i.e.
1' = bj = by
This means that we use the same random offset in the red, blue, and green values of a pixel. The reason for making this design decision is because for CIFAR-10 and CIFAR- we found that fully random A creates large drops in clean accuracy, even when the network is trained to learn such distortions. As a result, for data sets with high spatial complexity like CIFAR-10 and CIFAR- we do not select A randomly. We choose A to be the identity matrix. Likewise for b' we only randomly generate 35% of the matrix values and leave the rest as O. For the randomly generated values, we choose them from uniform distribution from -0.5 to 0.5.
20

---

Under review as a conference paper at ICLR 2020
For datasets with less spatial complexity like FashionMNIST, we equate matrices =' A1 = A2 Ag and D' = bj = b2 by and select A' and as random matrices: The values of A'' and are selected from Gaussian distribution with μ and 0.1.
D EXPERIMENTAL RESULTS
We reported our experimental results for CIFAR-10 in Section 4 for runs (see Table 2) Here, one implemented by first choosing matrices A' and B' from the distribution corresponding the considered data set for each protected layer. Next the attacker's success rate and clean accuracy of the defense are simulated. For each next run, matrices A' and D' are again chosen anew.
As shown in Figure 5 for CIFAR-10, the average result for runs is not much different from that of I run. Moreover,
also report the result of the best case of the attack among runs in Table TO As one can see, different rums give very similar results. This shows that BUZz is not sensitive to the choice of A' and B' (worst and best cases are close one another).
Table 10: Mixed boxck-box attack on CIFAR-10 10- the best case
Vanilla
2-Networks
4-Networks
8-Networks
Clean Accuracy
0.8835
0.76
0.69
0.62
FGSM Targeted
0.831
0.96
0.99
0.99
iFGSM Targeted
0.766
0.98
0.99
FGSM Untargeted
0.297
0.85
0.94
0.97
IFGSM Untargeted
0.282
0.87
0.97
0.99
Fashion MNIST Papernot Attack
0.9
Accuracy
0.7
0.6
0.4
0.3
0.2
0.1
Vanilla
2-Networks
4-Networks
B-Networks
Clean Accuracy
0.9356
0.8537
0.8204
0.7779
FGSM Targeted
0.737
0.925
0.965
0.99
IFGSM Targeted
0.652
0.97
0.991
0.997
FGSM Untargeted
0.833
IFGSM Untargeted
0.164
0.78
0.878
0.96
Figure 4: Mixed Black- -ox Attack on FashionMNIST, single run.
For completeness, we also did experiments for pure black-box attacks. The results are in Figures 7 and for for single run. The numbers confirm the mixed black-box attack being stronger than the pure black-box attack.
We also mention that for the vanilla network for CIFAR-10 htttps://gtthu: com/ htppogggciittttttcciiioooee far clean accuracy of 92.6% which is more than the reported 88.4% our experiments this due to our limited training time (limited resources). For the vanilla network for CIFAR- 100 without data augmentation https: / gi thub. com/ SamKirkiiieeeeeree.a cirr100f shows a clean accuracy of 64% which similar to the clean accuracy of 63% reported here. For the vanilla network for FashionMNIST https: wwwww//www/..../ http://::/wwwwwpwppamppppap pyimagesearch. ccm/2019 2012000000000 /2/0///0 wwt.faashhonnnoaee fee.hoooo..iiittttt----eeaaaaaes
shows a
clean accuracy of 94% which is equal to the clean accuracy reported here.
Discussion. From the experiments on FashionMNIIST, CIFAR-10, CIFAR-100, 100, we see the success of untargeted mixed black-box attacks vanilla nets is significantly higher than that of the targeted
21

---

Under review as a conference paper at ICLR 2020
CIFAR 10 Papernot Attack
0.9
0.8
Accuracy
0.7
0.5
0.4
0.2
0.1
Vanilla
2-Networks
4-Networks
8-Networks
Clean Accuracy
0.8835
0.7599
0.6832
0.6091
FGSM Targeted
0.951
0.985
0.995
IFGSM Targeted
0.766
0.978
0.998
FGSM Untargeted
0.297
0.797
0.931
0.976
Untargeted
0.282
0.814
0.963
0.992
Figure 5: Mixed Black-Box Attack on CIFAR-10, single run.
CIFAR 100 Papernot Attack
0.8
Accuracy
0.7
0.5
0.4
0.3
0.2
0.1
Vanilla
2-Networks
4-Networks
B-Networks
Accuracy
0.6316
0.4613
0.3768
0.3066
FGSM Targeted
0.961
0.996
1
IFGSM Targeted
0.933
0.999
1
FGSM Untargeted
0,228
0.907
0.968
0.995
Untargeted
Figure 6: Mixed Black-Box Attack on CIFAR-100, single run.
attacks. Moreover, the success of targeted attacks on the vanilla network of CIFAR-100 is close to 1% which significantly poorer than that of CIFAR-10 10 or FashionMNIST (i.e., 16% 19%). We believe the main reason is that the clean accuracy of the vanilla network of CIFAR 100 is sig- nificantly lower than that of CIFAR- 10 or FashionMNIST. i.e. 63%, 88% and 93.5%, respectively. From this result, also believe that the resistance of network classifier against adversarial machine learning strongly depends on the clean accuracy of the network itself; the lower the clean accuracy, the lower the attacker's success as well (remember that the attacker's success rate is only measured over images that are accurately predicted by the network).
From the view of the designer, she/he should pay attention to the highest threat the most powerful attack, in this case untargeted mixed black box attacks. In other words, we should use 8, as a true measurement evaluate the resistance of given defense. Equipped with this argument, we can now see that BUZz has very good resistance against adversarial examples at the cost of clean accuracy. As we argued in Table surrent-state-of-thee to poor resistance against adversarial examples this is because their clean accuracy is nearly equal to that of the vanilla network (the defenses do not want to give up some the clean accuracy). A good lesson to gain security. game in that it is possible produce with significant probability adversarial examples with small noise to bypass the defense. Including 'buffer zones' means the designer has to give up some clean accuracy.
defenses generally have very large 8,, which is due
can learn from Table is is that we have to sacrifice something (here, clean accuracy) the defense does not have 'buffer zones', then the adversary always wins the
22

---

Under review as a conference paper at ICLR 2020
For Fashion MNIST, may want to use 8-network BUZz because it has the best trade-off between defense rate (one minus the attacker's succes rate) and the defense accuracy (the clean accuracy of the defense). Similarly, for CIFAR-10 and CIFAR-100, suggest 4-network BUZz and 2-network BUZz, respectively.
Fashion MNIST Pure Black-Box Attacks
0.8
0.5
0.4
0.2
0.1
Vanilla
2-Networks
4-Networks
8-Networks
Accuracy
0.9356
0.8537
0.8204
0.7779
FGSM Targeted
0.834
0.964
0.974
IFGSM Targeted
0.843
0.993
0.998
1
FGSM Untargeted
0.183
0,738
0.848
0.92
Untargeted
0.98
Figure 7: Pure Black-box Attack FashionMNIST, single run.
CIFAR 10 Pure Black-Box Attacks
0.9
0.8
Accuracy
0.5
0.2
Vanilla
2-Networks
4-Networks
8-Networks
Clean Accuracy
0.8835
0.7599
0.6832
0.6091
Targeted
0.863
0.969
0.986
0.993
IFGSM Targeted
0.812
0.991
FGSM Untargeted
0.421
0.872
0.948
0.98
IFGSM Untargeted
0.379
0.907
0.974
0.992
Figure 8: Pure Black-box Attack on CIFAR-10, single run.
23

---

Under review as a conference paper at ICLR 2020
CIFAR 100 Pure Black-Bor Box Attacks
0.9
0.8
Accuracy
0.7
0.6
0.4
0.3
Vanilla
2-Networks
4-Networks
8-Networks
Clean Accuracy
0.6316
0.4613
0.3768
0.3066
FGSM Targeted
0.991
0.998
0.999
1
IFGSM Targeted
0.986
0.998
1
FGSM Untargeted
0.978
0.991
IFGS Untargeted
0.586
0.915
0.98
0.999
Figure 9: Pure Black-box Attack on CIFAR-100, single run.
24


Reviews:
[{"review_id": "SkgjFhCcKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "First of all, I think the authors do not do enough literature research on the topic of adversarial examples:\n\n1. In Sec 2., the authors only mention the FGSM in White-box Attacks. It is widely accepted that when evaluating the white-box attacks, you should at least test PGD attacks and/or C&W attacks.\n\n2. When you try to propose an adversarial defense, you should report the performance under white-box adaptive attacks. Only claiming effectiveness under black-box attacks is not informative or convincing.\n\nThe experiment results are also weird. For example, in Figure 8, why the Vanilla clean accuracy on CIFAR-10 is only 88.35%? Besides, the clean accuracy on CIFAR-10 of 2-Networks Buzz is 75%, 8-Networks Buzz is 60%. This clean performance is not acceptable, no matter how robust is the model.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1xzzmnIFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a defense against black box adversarial attack. The authors train an ensemble of deep networks, and output a null label when the ensemble disagree. Success of adversarial attack is defined as the ensemble outputs an incorrect label that is not null. The authors experimentally show improved robustness to adversarial attack. \n\nThe idea is itself new, but very similar ideas are well known in the literature, and it is difficult to conclude that the proposed approach is superior. Several examples are:\n\nDefense by majority vote with ensembles has appeared several times in the literature (e.g. Pang et al 2019). The pro is that this paper proposes a novel way to create an ensemble by applying random linear transformations and rescaling to the input. But it is not clear this is superior compared to existing methods. \n\nRandomized smoothing (Cohen et al, 2019) guarantees smoothness of the classifier (and thus robustness to perturbation attack under certain norms). Note that randomized smoothing  provides certified guarantee against a stronger attack model (white box); it also guarantees the size of the margin (or buffer as the authors call it). The intuition of this paper is based on similar ideas so it seems necessary to at least compare with randomized smoothing. \n\nOutputting a null label is a major workhorse of adversarial defense. For example, previous work use a generative model to detect out of distribution samples; use a calibrated classifier to output null when low confidence. \n\nBecause the idea is only mildly interesting, good experimental support becomes crucial. However, I think there are several short-comings with the experiments:\n\nThe experiment contains only one (fairly old) attack method. Several recent alternatives such as SimBA (Guo et al 2019) can make the experiments more convincing. \n\nThe architecture is no longer the same for the target model (which is an ensemble with an additional random transformations) and surrogate model. It is unclear if the improvement is simply because of the difference in architecture. \n\nThe comparison to baselines seem unfair because it seems that the compared baselines do not have the option of outputting the null label. For example, a simple baseline of randomized smoothing + output the null label if the logit scores are below a threshold can make the story much stronger.\n\t\nMinor comments:\n\nSeveral suggestions on writing: the introduction contains much technical detail and even experimental results, and these are repeated again in later sections. The experimental section has many minor implementation details that could go into the appendix. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1gGCEmg5S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes the concept of buffer zones and suggests to use unanimous voting as a way to induce such buffer zones. To widen the buffer zone, they further propose to diversity the model. They then proposed a new metric for measuring defense and demonstrated that their method is effective.\n\nDecision: Weak Reject. This paper is fairly intuitive, but I am not sure about the fairness of the comparisons in the paper, and the level of rigor of the experiments.\n\nI think the conjecture that buffer zones are widened when the models are diverse deserve to be empirically tested. It is not clear to me a prior how exactly the buffer zones widen (even though the belief that they widen is intuitively appealing). I think one way to potentially characterize the buffer zone is by actually performing white-box attack experiments comparing:\nWhite-box attack vulnerability of unanimous voting vanilla models\nWhite-box attack vulnerability of unanimous voting “diversified” models\nI would also encourage the authors to think creatively about other ways to back up the the buffer zone claims put forth in the paper.\n\nI am also a little puzzled with the authors’ choice of the diversification procedure. The procedure c(x) = Ax + b will only linearly transform each column of the image, but not each row. This design choice feels rather ad hoc. Why did the authors settle on Ax + b in particular? Why not xA + b? Or AxC + b?\n\nRegarding the fairness of the experiments, do the models that the authors compare against have the luxury of returning a “Undecided” label? If not, then the problem formulation is fundamentally different, and I do not think the comparisons are necessarily fair. Are there any papers out there that also allow for an “Undecided” label? If so, they should be the baselines that one compares against. I have a rather hard time believing that this is the first paper to try unanimous voting across an ensemble.\n\nI am generally inclined to switch to weak accept so long as the other reviewers are willing to accept that the experiments are sufficient and the comparisons are fair. I am not opposed to the new problem setting, since I think the setting makes sense. I just want to know that the paper is doing due diligence regarding related work in this setting.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper formalizes the concept of buffer zones, and proposes a defense method based on a combination of deep neural networks and simple image transformations. The authors argue that the proposed method based on buffer zones is robust against state-of-the-art black box attacks methods.This paper, however, falls short of (1) unjustified claims (e.g., buffer zones are widened when the models are diverse); (2) incomplete literature survey and related work; (3) similar ideas are well-known in the literature, (4) unfair experimental evaluations and many others. Even after author response, it still does not gather support from the reviewers. Thus I recommend reject.
Decision: Reject