Thwarting finite difference adversarial attacks with output randomization
Authors: Haidar Khan
Dan Park
Azer Khan
Bülent Yener
Venue:  (2020)
Keywords: black box adversarial attacks
adversarial examples
defense
deep learning

Abstract:   Adversarial input poses a critical problem to deep neural networks (DNN). This problem is more severe in the "black box" setting where an adversary only needs to repeatedly query a DNN to estimate the gradients required to create adversarial examples. Current defense techniques against attacks in this setting are not effective. Thus, in this paper, we present a novel defense technique based on randomization applied to a DNN's output layer. While effective as a defense technique, this approach introduces a trade off between accuracy and robustness. We show that for certain types of randomization, we can bound the probability of introducing errors by carefully setting distributional parameters. For the particular case of finite difference black box attacks, we quantify the error introduced by the defense in the finite difference estimate of the gradient. Lastly, we show empirically that the defense can thwart three adaptive black box adversarial attack algorithms. 

Under review as a conference paper at ICLR 2020
THWARTING FINITE DIFFERENCE ADVERSARIAL ATTACKS WITH OUTPUT RANDOMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial input poses critical problem to deep neural networks (DNN). This problem is more severe in the "black box" " setting where an adversary only needs repeatedly query DNN to estimate the gradients required create adversarial examples. Current defense techniques against attacks in this setting are not effective. Thus, in this paper, we present a novel defense technique based on randomization applied to a DNN's output layer. While effective as a defense technique, this approach introduces a trade off between accuracy and robustness. We show that for certain types of randomization, we can bound the probability of introducing errors by carefully setting distributional parameters. For the particular case of finite difference black box attacks, we quantify the error introduced by the defense in the finite difference estimate of the gradient. Lastly, we show empirically that the defense can thwart three adaptive black box adversarial attack algorithms.
INTRODUCTION
The success of deep neural networks has led to scrutiny of the security vulnerabilities in deep neural network based models. One particular area concern is weakness to adversarial input: carefully crafted inputs that resist detection and can cause arbitrary errors in the model (I:2). This is especially highlighted in the domain of image classification, where an adversary creates an image that resembles natural image to human observer but easily fools deep neural network based image classifiers (3).
Different types of adversarial attacks exists throughout the lifecycle of a deep neural network model. For example, adversaries can attack model during training by injecting corrupting data into the training set or by modifying data in the training set. However, inference time attacks are more worrisome they represent the bulk of realistic attack surfaces 44 15 6).
The input created under an inference time attack is known as an adversarial example and methods for generating such examples have recently attracted much attention. In most cases, the adversarial example is created by perturbing a (correctly classified) input such that the model commits an error. The perturbation applied the "clean" " input is typically constrained to be undetectably small g (
Defending againsts adversarial attacks on deep neural networks is crucial and has seen relatively slow
progress compared to the sophistication and progress of adversarial attacks (10 [1] 12 13 15). This is because it is difficult to prove a defense can withstand the types of attacks it is designed for especially when the defense must prove capable of withstanding "adaptive" adversaries that have knowledge of the details of the defense (16).
defense against adversarial attacks is defined by the threat model it designed to defend against (16). The most permissive threat model makes the weakest assumptions about the adversary. One such threat model can be assuming the adversary has complete knowledge of the model, including architecture and parameters of the underlying network. This is known the "white-box" setting. Other threat models can also be useful, such as assuming the adversary has knowledge of the network architecture but not the parameters of the model. More restrictive threat models allow only called "black box" attacks, attacks that can create adversarial examples without having access to the model architecture weights and only accessing some form of output of the model.
The rest of the paper is organized as follows. In Section 2 we discuss the threat model considered. In Section 3 describe valid black box attacks under the threat model. Output randomization

---

Under review as a conference paper at ICLR 2020
defense is described in Section 4 We show empirical results in Section 5 cover related approaches in Section and conclude in Section 7
2 THREAT MODEL
Adversary goals:
The goal of the adversary is to force the classifier to commit an error within distortion limit, such that the example crafted by the adversary is similar to the original example. The adversary induces such an error in an untargeted or targeted attack. The goal of an untargeted attack misclassification of an input, whereas the goal of a targeted attack is misclassífication of an input as class specified by the adversary.
Adversary knowledge: The adversary has access to the model only at the input and output level, and has no knowledge of its architecture or parameters. This black-box adversary is aware of the details of the defenses protecting the model and the type of randomness associated with any defense but not the exact random numbers generated.
Adversary capability: The adversary only has access to the model by providing examples as input and observing the output probability vector generated by the model output. The adversary can modify aspects of the input in any way as long as it remains similar to the original input. This similarity is controlled by an I, distortion penalty the distortion penalty include I,, and lsc. We use the l2 perturbation penalty as this type of attack results in the strongest attacks (17). Attackers are allowed query the model up to a maximum limit, which we increase to our computational limit to strengthen attacks. In addition to a maximum query limit, attackers often use an early stopping parameter avoid wasting computation on unpromising direction during optimization. We also increase this parameter to test our defense against stronger attacks.
the adversarial image. Common choices for
3 EXISTING BLACK BOX ATTACKS
Black box attacks are called "gradient-free" attacks since they do not involve computing gradients of the input by backpropagation on the model under attack. Instead the gradients of the input estimated by using the finite difference estimate for each input feature.
Black box attacks can be categorized by the type of output the adversary is allowed observe: logit layer, class probabilitiesssotmax layer, or top k class probabilities/labels.
In general, designing successful black box attacks in the label only setting is much harder than the setting where the logit layer or softmax layer is available to the attacker. We consider the easiest setting for black box attacks, where all the class probabilities are available.
3.1 ZOO BLACK BOX ATTACK
The Zeroth Order Optimization based black-box (ZO0) attack (18) is method for creating adversarial examples for deep neural networks that only requires input and output access to the model. ZOO adopts similar iterative optimization based approach to adversarial example generation as other successful attacks, such the Carlini & Wagner (C&W) attack (17). The attack begins with correctly classified input image x, defines an adversarial loss function that scores perturbations applied to the input, and optimizes the adversarial loss function using gradient descent to find 8* that creates a succesful adversarial example. Specifically, gradient descent is used to find 8* such that:
f(r + 8*) 1ft
|| - (x +o++00 +8)1 Se
Namely, that the perturbed input x + 8* successfully fools the classifier f to predict the incorrect class yd and that the perturbed input is similar to the original input up some distortion limit e.
The primary (and strongest) adversarial loss used by the ZOO attack for targeted attacks is given by:
T(,, t) = max max (log fx) - log [ -K
(1)
isf
2

---

Under review as a conference paper at ICLR 2020
Where x is input image, is target class, and R is tuning parameter. Minimizing this loss function over the imput 2 causes the classifier to predict class t for the optimized input. For untargeted attacks, similar loss function is used:
((x) = max log f(x), max log f fz)) R 3/4
(2)
where i is the original label for the input 2. This loss function simply pushes to enter region of misclassification for the classifier f.
In order limit distortion of the original input, the adversarial loss function is combined with distortion penalty in the full optimization problem. This is given by:
min || no ll2 + c. L(x,t)
subject to x E 1]
In the white box setting, attackers can take advantage of the backprogation algorithm calculate the gradient the adversarial loss function with respect to the input coordinates (LL) and solve the optimization problem using gradient descent. In lieu this, ZOO uses "zeroth order stochastic coordinate descent" optimize input the adversarial loss directly. This is most easily understood as a finite difference estimate of the gradient of the input with the symmetric difference quotient (18):
åL
L(x he,) - L(x he,)
9:
dx:
2h
with e; as the basis vector for coordinate/pixel and h set a small constant. The ZOO attack uses this approximation to the gradients create adversarial example from the given input. Note that for an image with n pixels, computing an estimate of the gradient with respect to each pixel requires 2n queries to the model. Since this is usually prohibitive, the ZOO attack circumvents this by only estimating the gradients for a subset of coordinates at each step. ZOO also uses dimensionality reduction and hierarchical approach to further increase the efficiency of the attack and show empirically that these methods are effective (18)
3.2 QUERY LIMITED (QL) BLACK BOX ATTACK
similar approach ZO0 is adopted by (19) in query limited setting. Like ZOO, the QI attack estimates the gradients of the adversarial loss using finite difference based approach. However, the QL attack reduces the number of queries required estimate the gradients by employing search distribution. Natural Evolutionary Strategies (NES) (20) is used as a black box to estimate gradients from limited number of model evaluations. Projected Gradient Descent (PGD) (12) used to update the adversarial example using the estimated gradients. PGD uses the sign of the estimated gradients to perform an update: x² = r--1_ -17- sign (g:), with a step size v and the estimated gradient gr. The estimated gradient for the QI attack using NES is given by:
L(z o u)) u; L(x
u:) 24
9t
2mn
where is sampled from standard normal distribution with the same dimension as the input is the search variance, and m the number of samples used to estimate the gradient. The difference between this approach and ZOO is that 700 attempts to estimate the gradient with respect to one coordinate at time while this approach averages over perturbations to many coordinates to estimate the entire gradient directly.
In the next section, we show that applying a simple randomization function (that does not affect model accuracy) to the output of a model causes these types of attacks to fail even if the attack is adapted to the specific randomization function.
THWARTING BLACK BOX ATTACKS
The intuition behind output randomization is that model may deliberately make errors in its predictions in order to thwart potential attacker. This simple idea introduces a tradeoff between accurate predictions and the effectiveness of finite difference based black box adversarial attacks.
3

---

Under review as a conference paper at ICLR 2020
Output randomization for model that produces a probability distribution over class labels replaces the output the model p by stochastic function d(p). The function d must satisfy two conditions:
1. The probability of misclassiffing an input due applying d is bounded by
2. The vector d(p) prevents adversaries under the given threat model from generating adversar- ial examples.
The first condition ensures that the applied defense minimally impacts non-adversarial users of the model, such as users of an online image classification service. The effectiveness of the defense comes from satisfying the second condition as the introduced randomness must prevent an adversary (in the appropiate setting) from producing adversarial example.
the following two sections, we consider a simple noise-inducing function d(y) = = + where e random variable.
4.1 MISSCLASSIFICATTIN RATE
A simple function useful for defending a model is the gaussian noise function d(p) + where gaussian random variable with mean f and variance o² (N,, Ic)). In the black box setting, user querying the model with an input x receives the perturbed vector d(p) instead of the true probability vector p. Note that d(p) does not necessarily represent probability mass function like p.
To verify that this function satisfies the first condition above, we wish to know the probability that the class predicted by the undefended model is the same as the class predicted by the defended model. If the output the model for an input p. we will refer the maximum element p as Pm and the rest of the elements of decreasing order as P2, P3. PP...
Suppose the model correctly classifies the input x the vector p, we can express the probability that is misclassified in the vector d(p) as:
Σ P(d(p)) > d(pm))
We write P(d(p;) d(pm)) for 2,3, ...C
P(d(p:)) > d(pm)) = P(ps + 6 > Pm + 6m)
If we define 6; mean μr pm and variance of + on then we can write:
as shown in Figure Tal and since = E;
Em is itself a gaussian with
Mi Um
8; pi + pm
P(e: 0.) (lee s 6,) 1
o
of + o
Using the cumulative distribution function of standard gaussian distribution 4 write the misclassification probability as:
p4 + pmm
(- b1 pm
= Pdd(p)) > d(pm)) =
0² 02
o² 02
For the special case of gaussian noise function d(y) with mean and variance d² we would like to fix the probability of misclassification value K and compute the appropiate variance o², We can use the inverse of the standard gaussian cdf 4 or the probit function, to write this easily:
6
0²
2Φ 200-1(4 ((K)
Note that the desired misclassification rate K < 0.5 in any real case and the rhs will be positive. we consider ò, as the confidence of the model, then the allowable variance will be larger when the model is confident and smaller otherwise. We show the caleulations above for one class the misclassification probability (K) and level of noise (o²) can set for each class separately. In Figure Tb we show the maximum allowable variance for different misclassification rates.
4

---

Under review as a conference paper at ICLR 2020
K=5.0e03
0.20
K=1.0e02
K-1.6
0.12
K-2.0e-01
0
pribabitt
vallance
0.06
0.05
0.00
0.00
10
0.25
(a) Probability distribution over classes generated (b) Maximum output randomization 0² vs for mis- by a classification model. & represents the relative classification rates K {20%, 10%, 1%, 0.5%} confidence of the model's prediction.
Figure 1: Controlling misclassification caused by output randomization
4.2 FINITE DIFFERENCE GRADIENT ERROR
To verify the function d(p) satisfies the second condition, that it introduces error that prevents finite difference based black box attack, we show the effect of the output randomization on the gradient accessible the adversary.
Finite difference (FD) based approaches involve evaluating the adversarial loss at two points, x + he; and he; close (with small h and unit vector eş) and using the slope estimate the gradient of the loss with respect to pixel i of the input. For loss function L, the finite difference estimate of the gradient of pixel is given by:
L(f(x + he,)) - L(f(x he;))
2h
Here, we write the adversarial loss function (either the untargeted loss in Equation 2 or the targeted loss in Equation TT in terms of the output of the model to make explicit the dependence of L on the output vector of the network p f(-). needed compute the gradient estimate. When the network is defended using output randomization, function d() is applied to the output vector the network. Thus, the finite difference gradient computed the attacker is:
(-)- p and p' are used to distinguish between the two output vectors
7/ L(d(p)) L(d(p')
2h
The error in the FD gradient introduced by the defense is given by |g: 1 xxl When is function that adds noise e to the output of the network, the expected value of the error is:
T(p + e) Tlpl + 6')
|B 71] 9:
2h
This error term depends on the choice of the adversarial loss function L(-). Since the untargeted attack is generally considered easier than the targeted attack, consider how the gradient error of the defended model behaves under the untargeted adversarial loss function. For untargeted attacks, we simplify the loss function (p,(p) log(pe) log(p.) log( (e( where pe is the probability of the true class and p. the maximum probability assigned class other than the class of the input image.
Substituting the untargeted adversarial loss for L(-) we see:
Pc + EC) - log(
1/ + e p/, + c
Fggg -v]l =
log
2h
Pe + Co
5

---

Under review as a conference paper at ICLR 2020
We second order Taylor series approximation of Ellog(X)] = log(E[X]) mate the expectations. If we further assume is zero-mean with variance o², then Elp + p and the expectation of the defended gradient is approximately:
Var[X] 2 to approxi-
0² 4h
0
+
Po + Pe
[ -7i1 RJ
2' Pe P
2' PEPo
This approximation summarizes the suprising effect output randomization has on finite difference based black box attacks. Firstly, it is easy see that the error scales with the variance of (in the zero mean case). Even when the adversary adapts to the defense by averaging over the output randomization the variance is only reduced linearly by the number of samples. Secondly, even in expectation the error is never nenozero. This is because one of two cases must be true in order for the error be zero:
1. Pe Po and P'c P'o p'.
2. Pe P'c and Po
Case cannot occur because it implies the model is predicting two different classes simultaneously. Case will only occur if L(p) T(p') which means 9: O.
The reason for this behavior is mostly due to the log operation in the adversarial loss function L. As noted by the authors (18), the log is crucial to the success finite difference black box attacks as well trained models yield skewed distributions in the output f(x). In our experiments, we show that this behavior holds real world experiments on image classification datasets.
5 EMPIRICAL RESULTS
To evaluate the output randomization defense against black box attacks, we select three successful black box attacks (700 (18), QL (19), and BAND (2T) benchmark image classification datasets (MNIST (22), CIFARIO 10 23), and ImageNet (24)). For all defended models, we use ~ (O, o² Ic). Details of both the attacks and defenses can be found in our code roderane aadd the appendix. We follow the guidelines laid out in (16), most importantly we attempt to adapt the attacks to the proposed defense. In this case, this means the attacker is aware of the type of randomization applied the output of the network. Therefore, we adapt the attacks by allowing the attacker to average over the output randomization an attempt to bypass the defense. This type of adaptation has been shown to overcome certain types of input and model randomization defenses (25). In our results, we refer to this as the adaptive attack.
As sanity check, also measure the attack success rate of white box attacker (Carlini & Wagner L2 (17) attack) with randomized output. We find that the defense has some success in defending against this attack in the non-adaptive case. However the adaptive white box attacker is able to overcome the output randomization by averaging over a small number of samples. This is summarized in Figure 2
Our main set of experiments is shown in Figure and show the effects of output randomization on the non-adaptive and adaptive ZOO attacks. We show that the defense reduces the attack success rate significantly even in the adaptive attacker setting, where we average over randomness and double attack iterations. The effectiveness of the defense was affected by targeted or untargeted attacks. Table summarizes the results for three finite difference based black box attacks on ImageNet.
is important show how a defense affects the "normal" operating properties of model and this is typically demonstrated by comparing the test set accuracy of the defended model to the undefended model. Figure 2a and Figure 3a show the effect of increasing noise levels on test accuracy. Output randomization is an effective defense against black box attacks at noise levels as small as d² le-4 where model performance is identical to undefended models.
'Our attack code is based on httpt::: //tttpp.///ttb...../ http://c///////////tttt//. http::/tttssstttttt:e com/
aallackb------ attacks a Blackbxx-bandin for the 700, QI., and BAND attacks respectively.
and Intpps///githui www.h/...oommaaaaaa www.//adyo...a.bbbb..a..

---

Under review as a conference paper at ICLR 2020
34
3,0
Imagendt
V.
3.
Imagene:
GD-
64-
33-
Lo-a1 L.-2 6.0-9
1.0-20 6.032
rolie (81)
Imagenat
Imagenet
19-111 23 022-12
10-64 12 42-17
B00m
rolle
(a) Attack success rate (solid) and test set accuracy Attack success vs variance (groups) for adap- (dashed) vs variance for the non-adaptive attacker. Out- live attacker with increasing averaging (10, 50, 100 put randomization is not effective against a white box samples). Averaging allows the white box adaptive attacker at small noise levels.
attacker to overcome output randomization.
Figure 2: Carlini & Wagner (7) white box attack versus output randomization. Top row shows untargeted attacks, bottom row shows targeted attacks.
6.6
66-
68-
100
L0C
61
G2
1363k 1.3-3 636-3
3.044 196-22 62640
1ed4 1. 52632
(d')
6.6
60-
0--
Dak-
130
10e
62
12
62-
13-31 1. 626/0
3.e64 10 626.00
1.
52632
(a) Attack success rate (solid) and test set accuracy (b) Attack success rate vs variance (bar groups) for (dashed) ys variance for the non-adaptive attacker. Out-adaptive attacker with increasing averaging (10, 50, 100 put randomization blocks attacks even very small samples). Averaging does not improve attack success noise levels (o² < 1e-6).
rate of the black box attacker.
Figure 3: ZOO (88) black box attack versus output randomization. Top row shows untargeted attacks, bottom row shows targeted attacks. Compare this figure with the white box attack results in Figure
Defensive techniques without output randomization are still vulnerable black box attacks. We evaluated defensive distillation (IO) and input randomization (26) against the ZOO attack and found that these defenses did not reduce the attack success rate significantly as shown in Table Distillation is shown to be vulnerable to white box attacks in (17), we show it is also vulnerable to finite difference black box attacks. Input randomization (26) has limited success defending against ZOO (reducing attack success rate to 0.76), however it is not as effective as output randomization. This is because randomization applied at the input is not guaranteed to affect the finite difference gradient estimates. In addition, input randomization also does not allow fine control over model accuracy.
Table Output randomization black box attacks on 100 correctly classified ImageNet examples measured by attack success rate (fraction of examples misclassified)
Noise Variance
ZOO 18)
QL 19
BAND 21
(undefended)
0.69
1.00
0.92
1.00e-4
0.03
0.73
0.58
1.00e-2
0.00
0.02
0.07
5.76e-2
0.00
0.01
0.06
7

---

Under review as a conference paper at ICLR 2020
Table 2: ZOO black box attack success rate vs three defenses
Dataset
Ex. Type
Distillation(10)
Mitigation (26
OR (ours)
MNIST
Targeted
1.00
0.00
MNIST
Untargeted
0.99
0.01
CIFARIO
Targeted
1.00
0.011
CIFARIO
Untargeted
1.00
0.19
ImageNet
Untargeted
0.76
0.005
RELATED WORK
In this section, we discuss related gradient masking or obfuscated gradient defenses. We will focus on proactive defenses, which attempt to make a network robust, compared to reactive defenses, which attempt detect adversarial examples. (25) defined three ways obfuscate gradients: shattered gradients, exploding/vanishing gradients, and stochastic gradients.
Shattered gradients are non-differentiable defense that causes gradient to be nonexistent or incorrect. This can be done unintentionlly by introducing numeric instability. (27) and (28) proposed
shattered gradients defenses that introduce non-differentiable and non-linear discretization to model's input. These transformation are ineffective as black box attacks are agnostic of input randomization.
Exploding gradients make a network hard to train because of an extremely deep neural network. This is generally done by using an output of a neural network as the input of another. (29) and (30) both proposed defenses that utilize GANs. However, (25) shows that these defenses can be bypassed using transferability of adversarial attacks. The transferability property allows an attacker to use adversarial examples created using model (often surrogate model) to fool another model (11).
Although this a valid attack vector for even black box models, we do not consider this type of attack in this work.
Stochastic gradients randomize gradients by introducing some randomization to the network or randomly transforming the input to the network. (32) proposed stochastic gradients defense in which random subset of activations are pruned. (26) introduced randomness by randomly rescaling
input images. (25) showed that an adaptive attacker can bypass these defenses by computing the expected value over multiple queries. We test a similar approach on output randomization and show is not effective in the black box case (FigureBb)
Although other defense methods consider introducing randomness to the input model itself, this work is the first our knowledge consider randomizing the output of the model directly.
7 CONCLUSION
Black box attacks based on finite difference gradient estimates pose threat classification models without needing priveleged access to the model. In this paper, we show that this threat can averted by introducing simple types of randomization to the output of the model. Our empirical results show that even very small (o² = le-6) perturbations to the output can prevent these type of attacks.
Although work shows encouraging result for defending against black box attacks, show that output randomization (or other types of randomization) do not prevent white box attacks (25). Furthermore, output randomization only prevents query based black box attacks and does not address the problem of transfer attacks (33) 31). Other attacks, such as N Attack (34), utilize derivative-free optimization find adversarial examples bypassing the need for finite difference estimates. We leave defense against these types of attacks to future work.
REFERENCES
[1] C. Szegedy, W. 7aremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, "Intriguing properties of neural networks," pp. 1-10, 2013. [Online]. Available: htth://ariv.orraaa///122999999
8

---

Under review as a conference paper at ICLR 2020
[2] I. J. Goodfellow, Shlens, and C. Szegedy, "Explaining and Harnessing Adversarial Examples." pp. 1-11,
"Explaiini 2014. [Online]. Available: http://arii..ogg//////112222
[3] A. Kurakin, I. Goodfellow, and S. Bengio, " "Adversarial examples in the physical world," no. c, pp. 1-14, 2016. [Online]. A http://ankiiorrghhh///10...33333
[4] K. Grosse, N. Papernot, Manoharan, M. Backes, and P. McDaniel. "Adversarial Perturbations Against Deep Neural Networks for Malware Classification," jun 2016. [Online]. Available:
http://arsxiv.or//bb////c 6ttp://www..rr//b////666664333 1606.44435
[5] N. Narodytska and S. P. Kasiviswanathan, "Simple Black Adversarial Perturbations for Deep Networks," pp. 1-19, 2016. [Online]. Available: http://axii.orr//bb/lll22..999
[6] N. Carlini and D. Wagner, "Audio adversarial examples: Targeted attacks on spcech-to-text." in Proceedings 2018 IEEE Symposiun on Security and Privacy Workshops, SPW 2018. Institute of Electrical and Electronics Engineers Inc., aug 2018, pp. 1-7.
[7] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, "Practical Black-Box Attacks against Machine Learning," 2016. [Online]. Available: http:///arxiv.org/bb////10222
[8] M. Moosavi-Dezfooll A. Fawzi, O. Fawzi, and P. Frossard, "Universal Adversarial Perturbations," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 70, no. 5. IEEE, jul 2017. pp. 86 94. [Online]. Available: http://iecexplore.ieee..oo otgg//ooccmmmm//0009550000
[9] N. Carlini, G. Katz, C. Barrett, and D. L. Dill, "Provably Minimally-Distorred Adversarial Examples," sep 2017. [Onlinc]. Available: http://arxiiv.or//bbb/ 709. 10207
[10] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami, "Distillation Defense Adversarial Perturbations Against Deep Neural Networks," in 2016 IEEE Symposium on Security and Privacy (SP). IEEE, may 2016, pp. 582 597. [Online]. Available:
[11] F. Tramèr, A. Kurakin, N. Papernot, Goodfellow, D. Boneh, and P. McDaniel, "Ensemble Adversarial Training: Attacks and Defenses," pp. 1 1-20, 2017. [Online]. Available: http::////axaxivor//aaasa /0tp//////770000000000 705.07204
[12] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, "Towards Deep Learning Models Resistant to Adversarial Attacks," pp. 1-27, 2017. [Online]. Available: http:///ariiv.orr/aa//////...
[13] F. Croce, M. Andriushchenko, and M. Hein, "Provable Robustness of ReLU networks via Maximization of Linear Regions," oct 2018. [Online]. Available: lutp::/arxiv.orr//bbaa 180.07481
[14] L. Schott, J. Rauber, M. Bethge, and W. Brendel, "Towards the first adversarially robust neural network model on MNIST," may 2018. [Online], Available: http::/arxxiv.or//abb/ 805, 809.09190
[15] I. Rosenberg, A. Shabtai, Y. Elovici, and L. Rokach, "Defense Methods Against Adversarial Examples for Recurrent Neural Networks," jan 2019. [Online], Available: http:://arxiv.orr/aa///10000 901.///////99919999999
[16] N. Carlini, A. Athalye, Papernot, W. Brendel, Rauber, D. Tsipras, I. Goodfellow,
A. Madry, and A. Kurakin, "On Evaluating Adversarial Robustness," feb 2019. [Online]. Available:
http://arxiv.or//bb///s/s (1670
[17] N. Carlini and D. Wagner, "Towards Evaluating the Robustness of Neural Networks," Proceedings IEEE Symposium on Security and Privacy, pp. 39-57, 2017.
[18] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-. Hsieh, "ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models," 2017. [Online]. Available: http::/aaxxivorr//ab// wtwphh///70.00099990 (.9900000999... % J0Ahtp://dxx.oi..oro/. 10. 14573128572.31144444
[19] A. Iyas, L. Engstrom. A. Athalye, and Lin, "Black-box Adversarial Attacks with Limited Queries and Information," " 2018. [Online]. Available: http://rxxvvorr//bb/ wwwp://////brs////..0.00 884.08598
[20] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, and J. Schmidhuber. "Natural evolution strategies." " The Journal of Machine Learning Research, vol. 15, no. 1, pp. 989-980, 2014.
[21] A. Ilyas, L. Engstrom, and A. Madry, "Prior convictions: Black box adversarial attacks with bandits and priors," ICLR 2019, 2018. [Online]. Available: https///aaiivoor//b/////7777777
[22] Y. LeCun, C. Cortes, and C. Burges, "Mnist handwritten digit database," AT&T Labs [Online]. Available: htt:://ynnn. lecun. com/exdb/unist, vol. p. 18, 2010.

---

Under review as a conference paper at ICLR 2020
[23] A. Krizhevsky, V. Nair, and G. Hinton," "The cifar-10 dataset," online: http://www: toronto. edu/kriz/cifar html, vol. 55, 2014.
[24] J. Deng, W. Dong. R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in 2009 IEEE conference on computer vision and pattern recognition. Icee, 2009, pp. 248-22 248-255.
[25] A. Athalye, N. Carlini, and D. Wagner, "Obfuscated Gradients Give Security: Cireumventing Defenses Adversarial Examples," 2018. [Online]. Available: http Warxiv.org/abs/ hatps///8200000200
False Sense of
[26] C. Wang. Z. Zhang, Z. Ren, and A. Yuille, "Mitigatting Adversarial Effects Through Randomization," nov 2017. [Online], Available: httt://arxix.orr/b////11.111
[27] Buckman, A. Roy, C. Raffel, and I. Goodfellow, "Thermometer Encoding: One Hot Way Resist Adversarial Examples," ICLR, vol. 19, no. 1. pp. 92-97, 2018.
[28] C. Guo, Rana, Cisse, and van der Maaten. "Countering Adversarial Images using Input Transformations"" oct 2017. [Online]. Available: http://arrv.oor//ay/111.....1111
[29] Song. T. Kim, Nowozin. S. Ermon, N. Kushman, "PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples," oct 2017. [Online]. Available: http://aaii..org//////1110000000
[30] P. Samangouei, M. Kabkab, and R. Chellappa, "Defense-gad gan:: Protecting classifiers against adversarial attacks using generative models," arXiv preprint arXiv:1805.06005, 2018.
[31] N. Papernot, P. McDaniel, and I. Goodfellow, "Transferability in Machine Learning: from Phenomena to Black-Bos Box Attacks using Adversarial Samples," 2016. [Online]. Available: http://arxii.or//absse (ttps//0050002111 wrtp:/oorrrgaaa//0000777777
[32] G. S. Dhillon, K. Azizzadenesheli, Z. C. Lipton, J. Bernstein, J. Kossaifi, A. Khanna, and A. Anandkumar,
"Stochastic Activation Pruning for Robust Adversarial Defense," mar 2018. [Online]. Available: http://axki..or/aab///81......
[33] Liu, X. Chen, C. Liu, and D. Song. "Delving into Transferable Adversarial Examples and Black-box Attacks," no. pp. 1- 14, 2016. [Online]. Available: http://anxiv.orr/aas//611 1.000007
[34] Li, L. Li. L. Wang, T. Zhang, and B. Gong. "NATTACK: Learning the distributions of adversarial examples for improved black-box attack deep neural networks," in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Rescarch, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. Long Beach, California, USA: PMLR, 09-15 15 Jun 2019, pp. 3866-38663033 3876. [Online), Available: http://proceeinng..li.ppppp mlr press///9/l/s/
[35] D. Kingma and J. Ba, "Adam: method for stochastic optimization," arXiv preprint arXiv: 1412.6980, 2014.
10

---

Under review as a conference paper at ICLR 2020
APPENDIX
ATTACK DETAILS
For our evaluation, we trained models for MNIST, CIFARIO, and ImageNet that acheived 99%, 79%, and 72% test set accuracies respectively using the code provided by (18). Non-adaptive attacks were conducted using the parameters suggested by the attacks. For ZOO, these include using the ADAM (35) solver, binary search steps, and image resizing + reset ADAM ImageNet. The adaptive attackers (both white black) were modeled two ways. (i) We added averaging over randomness to the attack and (ii) the number iterations was doubled. For all of our experiments, we averaged the attack success rate over 100 images and report the value over 30 runs.
EXTENSIONS
Other randomization functions can be used to introduce randomness in the output instead of gaussian noise. As an extension, considered noise sampled from gaussian mixture model with random mixing coefficients and parameters. In theory, this type randomization should be harder for attackers average over and avoid. However, we saw that white box attacker could still average over the added noise with 100 samples and circumvent the defense. Since the simple gaussian noise was effective in the blackbox case, demonstrate results using gaussian noise.
We also experimented with randomization of logit layer and observed improvements over softmax layer randomization. However, we chose to present the more general softmax layer randomization for clarity.
Figure 4 shows the error between the finite difference gradients for the ZOO attack on a defended and undefended model at varying noise levels. As we expect, increased noise levels cause the overall error (measured by the of difference of the gradients) increase dramatically.
L2 Norm of gradient difference
o² 1.0e-06
1.0e-04
400000
o² 1.0c-02
llg Vil
200000
100000
0
25
50
75
100
125
150
175
200
iterations
Figure 4: L2 norm of difference between the true finite difference gradient calculated by ZOO the undefended model and the finite difference gradient calculated on the defended model with increasing variance, As expected the error significantly higher for noise with larger variance.
11


Reviews:
[{"review_id": "rklTFuDlcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for defending black box (in particular, finite difference based loss) adversary attacks by randomisation of the output of the network. \n\nThe idea appears to be somewhat novel considering that majority of existing methods consider randomize inputs or the model itself. \n\nA natural question that would be particularly interesting to me is how does such defence compare against the defence by  randomizing the input and the model. There is no such comparison in the paper, which, to me, is the main weakness of the paper. \n\nThe authors consider the randomization in terms of Gaussian distribution. How would this differ if other types of distributions are considered, e.g. non-Gaussian distributions?\n\nSection 4.2 considered the finite difference gradient error, and discussed the results for untargetted attacks. What bout targetted attacks?   The presentation of this section is also not very clear, e.g. the equation on page 6. \n\nThe citation style look odd to me, often you use either something like \"[2,3]\", or something like \"Dhillon et al. (2018)\", but not \"(2,3)\". In addition, the equations should be number for the convenience of references.\n\n\"... in our code \\superscript {1}\" - the links all point to other people's code, not your codes.\n\nThe notation for the output p is different from the notation in page 2 where you used y. Try to use consistent notations. \n\nRegarding the novelty of this paper, I was based on my judgement and experience of reading a few papers, not I never published papers on adversary attacks or defence. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkxkhdKrqS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes applying randomization to the output layer of a DNN to defend against query-based attacks based on finite difference estimates. Then some theoretical analysis is provided, showing that with perturbation of a suitable scale, the randomization layer will not affect the accuracy of the model, while causing a large estimation error of finite difference methods that prevents finite-difference based attacks. Empirical results verify that the proposed defense is still effective against adaptive attacks where the randomness is averaged.\n\nPros:\n\nThe proposed method is simple, straightforward, yet novel. Its working mechanism is easy to understand and analyze, so it should be useful against finite-difference based attacks.\n\nLimitation:\n\nThe proposed method is not useful to defense against white-box attacks and transfer-based attacks, since basically it does not change the predictive model. Some other randomization methods like [26], by contrast, change the predictive model, hence they may be useful against white-box attacks and transfer-based attacks.\n\nQuestions and suggestions:\n\nThis part is my main concern.\n\nIt seems that the experimental results are very good. For example, in Figure 3a, the defense is effective even if \\sigma^2<1e-6. However, by the analysis in Section 4.2 (the formula below Line 3, Page 6), when \\sigma^2=1e-6, |E[g_i-\\gamma_i]| should be rather small, hence it should not block finite-difference based attacks. I think more explanation is needed for the good performance in the experiments.\n\nFinite differences are extremely sensitive to small random perturbation of the function value when the spacing (step size) h is small. For example, g_i=\\frac{L(f(x+he_i))-L(f(x-he_i))}{2h}, when h is very small, f(x+he_i) and f(x-he_i) is very close, hence adding perturbation to them will change g_i a lot. To present stronger adaptive attacks to output randomization, my suggestion is that a larger h can be adopted. It will be better if the results are investigated against attacks with different values of h.\n\nA mistake:\n\nIn Section 4.1 on Page 4: \"we can express the probability that x is misclassified in the vector d(p) as: \\sum_{i=2}^C P(d(p_i)>d(p_m))\". I think this is wrong, since P(A or B happens)=P(A happens)+P(B happens) only when A and B are mutually exclusive. However, \"d(p_i)>d(p_m)\" and \"d(p_j)>d(p_m)\" are not mutually exclusive. Hence, the probability that x is misclassified in the vector should be less than or equal to that sum of probabilities.\n\nBy the way, the writing in Section 4.1 is not clear:\n\n- The overall misclassification probability is presented first, but after that K only represents the misclassification probability into a specific class. The connection between them is unclear.\n- At the beginning of Section 4.1, the distribution of \\epsilon is \\epsilon\\sim\\mathcal{N}(\\mu,\\sigma^2\\cdot\\mathbf{I}_C): a unique \\sigma is used. But after that, the variance of \\epsilon_i becomes \\sigma_i^2 instead of \\sigma^2.\n- In the second to the last line on Page 4, \"level of noise (\\sigma^2) can be set for each class separately\", but the authors did not explain how to set them, and in the experiments \\sigma^2 is set as the same scalar.\n- In Figure 1b, the line style of \"K=5.0e-3\" and \"K=1.0e-1\" in the legend is very similar. The line style of \"K=2.0e-01\" in the legend is not clear: I do not know whether it refers to \"-.-.-.\" or \"------\".\n\nTypos:\n\nSection 6, Page 8: \"unintentionlly\" => \"unintentionally\"\nSome missing spaces after punctuation:\n- Section 4.2, Page 5, \"... gradient estimate.When the ...\" => should add a space before \"When\"\n- Section 5, Page 7, \"In addition,input randomization ...\" => should add a space before \"input\"", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SkglhWCXcB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to introduce randomness in a classifier’s predictions to mitigate black-box attacks that rely on gradient estimation through finite differences. The intuition behind the defense is correct: finite differences rely on the outputs of the neural network being non-deterministic and accurate to estimate gradients near the test points being attacked.\n\nHowever, the threat model chosen in this paper is not well justified: the adversary cannot be forced to use a particular strategy. Unlike what is suggested in Section 3, estimating gradients through finite differences is not the only strategy available in the black-box threat model (this is later mentioned in Section 6). In this case, the adversary could for instance decide to adapt by instead mounting a black-box attack that relies on transferability. Because Figure 2 shows that the defense does not provide robustness in the white-box setting, this suggests that other forms of black-box attacks that either (a) rely on transferability or (b) are label-based only would still evade the model. This limitation should be addressed to understand how applicable the defense strategy is in a realistic deployment.\n\nPutting this aside, it is not clear from Figure 3 that an adaptive strategy was evaluated in the limited black-box setting that is considered here (the caption of Figure 3.b only describes a “white-box” adaptive adversary), or that the defense is effective. The attack success rates are high for many graphs and increase as the adversary averages over more runs. Moving forward, increasing further the highest number of runs would help appreciate the limitations of the approach: it is currently set to at most 100, which is low. \n\nAs far as organization is concerned, a lot of real estate is spent on background material, and few experimental results are presented to support claims made in the introduction. Addressing the above comments would probably require compressing background material a bit. \n\nPage-by-page details:\n\n1/ An attack is always adversarial by definition, “adversarial attack” is a tautology. \n\n2 / What do you mean by “successful attacks”?\n\n2/ What do you mean by “strongest” loss?\n\n2/ Having a perturbation limited to be small does not guarantee it won’t impact the semantics of the input, even in the vision domain. Have you verified that the perturbations that you chose left semantics unperturbed?\n\n2/ It is best to avoid making broad statements such as “We use the l2 perturbation penalty as this type of attack results in the strongest attacks” because they are unlikely to hold across datasets and models.\n\n7/ Figures are difficult to parse (e.g., does T and U stand for targeted and untargeted?)\n\n7/ Distillation was already shown to be vulnerable to black-box attacks in [7]\n\n8/ Gradient masking was introduced in [7] prior to [25].\n\n8/ It would be good to justify the following statement (see my comment above): “Although this is a valid attack vector for even black box models, we do not consider this type of attack in this work”", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyeCSCxhYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a randomization defense in the black-box threat model, where bounded l_2 norm perturbations are allowed. In the presented scheme, the defender adds Gaussian noise to every coordinate of the output probability vector before returning an inference result. \n\n\nThe paper suffers from an incomplete evaluation, and therefore I cannot recommend acceptance. \n\n\nDetails:\n* Completeness of evaluation. As shown by Ilyas et al [1] and Cheng et al [2] and Brendel et al [3], black-box attacks can succeed even when only information about the label is present. Thus, I suspect that an attacker can simply run any of these attack algorithms in order to fool the model. In addition, if we use enough samples in the NES-based “Query-Limited” algorithm of Ilyas et al (that is, with enough samples per step) then we should be able to perfectly mimic the white-box attack, which as shown in Figure 2 is effective. \n* Potential flaw in evaluation. BAND performs worse than QL on the undefended classifier in Table 1, which should not occur. I would check to make sure that the attacks are applied correctly here.\n* Lacking details in evaluation. I could not find how many samples are used to perform the attacks in Table 1. It is hard to evaluate the defense without knowing how many samples are used in each algorithm.\n\n\n[1] https://arxiv.org/abs/1804.08598\n[2] https://arxiv.org/abs/1807.04457\n[3] https://arxiv.org/abs/1712.04248\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a defense technique against query-based attacks based on randomization applied to a DNN's output layer. It further shows that for certain types of randomization, they can bound the probability of introducing errors by carefully setting distributional parameters. It has some valuable contributions; however, the rebuttal does not fully address the concerns.
Decision: Reject