A Perturbation Analysis of Input Transformations for Adversarial Attacks
Authors: Adam Dziedzic
Sanjay Krishnan
Venue:  (2020)
Keywords: adversarial examples
defenses
stochastic channels
deterministic channels
input transformations
compression
noise
convolutional neural networks

Abstract: The existence of adversarial examples, or intentional mis-predictions constructed from small changes to correctly predicted examples, is one of the most significant challenges in neural network research today. Ironically, many new defenses are based on a simple observation - the adversarial inputs themselves are not robust and small perturbations to the attacking input often recover the desired prediction. While the intuition is somewhat clear, a detailed understanding of this phenomenon is missing from the research literature. This paper presents a comprehensive experimental analysis of when and why perturbation defenses work and potential mechanisms that could explain their effectiveness (or ineffectiveness) in different settings.

Under review as a conference paper at ICLR 2020
A PERTURBATION ANALYSIS OF INPUT TRANSFORMATIONS FOR ADVERSARIAL ATTACKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Many defenses for Convolutional Neural Networks are based on a simple obser- vation that the adversarial examples are not robust and small perturbations to the attacking input often recover the desired prediction. Intuitively, the adversarial examples occupy a very small cone in the decision space which is surrounded by a large area corresponding to the correct class. While the intuition is simple, a detailed understanding of this phenomenon is missing from the research literature.
We identify a family of defense techniques that based the instability as- sumption. The defenses include deterministic lossy compression algorithms and randomized perturbations to the input that all lead similar gains robustness. We present comprehensive experimental analysis of when and why perturbation defenses work and potential mechanisms that could explain their effectiveness (or ineffectiveness) in different settings.
INTRODUCTION
The attacks on Convolutional Neural Networks, such as Carlini & Wanger (Carlini & Wagner 2017) or PGD (Madry et al. 2017), generate strategically placed modifications that can be easily dominated by different types of perturbations resulting in correct predictions (Dziugaite et al. 2016 Roth et al., 2019). This suggests that the standard adversarial examples are not robust. Many defense techniques explicitly leverage this property and can be retrospectively interpreted as perturbations of the input images. However, a detailed understanding of this phenomenon lacking from the research literature including: (1) what types of perturbations work and what is their underlying mechanism, (2) whether all attacks exhibit this property, and (3) possible perturbation defenses.
attackers can employ to defeat
We can interpret a large number of recent defenses as type of input perturbations, for example, feature squeezing (Xu al. [2017), frequency or JPEG compression (Dziugaite et al. 20166), randomized smoothing (Cohen et al. 2201)), and perturbation of network structure or the inputs randomly (Jafarnia Jahromi et al. 20188] 8 Zhang & Liang 2019; Guo al., 2017). The defense techniques exhibit very similar gains in robustness. To show it, we start with a simple model where every example is passed through a lossy channel (stochastic deterministic) prior model inference. This channel induces a small perturbation to the input. We optimize the perturbation to be small enough as not to affect the prediction accuracy on clean examples but large enough dominate any adversarial attack. We find that this trade-off off is surprisingly consistent across very different families of input perturbations, where the relationship between channel distortion (the L2 distance between channel input and output) and robustness is very similar.
Why are some state-of-the-an attacks are sensitive to perturbation- defenses? We find that many attacks execute optimization procedure that finds adversarial image that is very close to the original image in terms of of L1, L2. or Loe norm. The resultant optimum, i.c., the adversarial image, tends to exhibit a higher level of instability than natural examples, which we demonstrate from the perspective of a first-order and second-order analysis. By instability we mean that small perturbations of the example
affect the prediction confidences.
The unification of perturbation based defense also gives us some insight into how an attacker might
avoid them.

---

Under review as a conference paper at ICLR 2020
Our experiments suggest that all the perturbation based defenses are vulnerable to the same types of attack strategies. We argue that the optimization procedure in the attacker should find the smallest distance from the original image that closes the recovery window. In fact, can devise generic attacker that attacks a particularly strong lossy channel, based on the additive Laplace noise, and adaptive attacks designed this channel are often successful against other defenses. This result implies that for many input perturbation defenses the attacker need not be fully adaptive, i.e., they do not need know exactly what kind of transformation is used to defend the network.
2 RELATED WORK
Much of the community's current understanding of adversarial sensitivity in neural networks is based on the seminal work by Szegedy et al. (2014). Multiple contemporaneous works also studied different aspects of this problem. postulating linearity and wver-parametrization as possible expla- nations (Goodfellow et al. 2014 Biggio et al. 2013). Since the beginning of this line of work, the
connection between compression and adversarial robustness has been recognized. The main defense strategies include: the idea of defensive network distillation]] (Papernot et al., 2015), quantizing inputs using feature squeezing (Xu et al.) 2017), the thermometer encoding as another form of quantization (Buckman et al., 2018), JPEG compression harnessed by (Dziugaite et al. 2016: Guo
ctt al., 2017; Das et al.) 2017 2018 Aydemir et al., 2018; Liu et al. 2019). Other line of research leveraged connection between randomization and adversarial robustness: Pixel Deflection (Prakash
et al. 2018), random resizing and padding of the input image (Xie et al.) 2017). and total variance minimization (Guo et al.] 2017). In our work we unify the methods based on compression and randomization that are applied to the input images.
While all of the aforementioned defenses were later broken (Carlini & Wagner, 2017; Athalye et al. 2018). it is important to understand why these approaches afforded any form of robustness. The community actually lacks consensus on this point: [Szegedy al. (2014) suggest that neural networks have blind spots, Xu et al. (0177) suggest that quantization makes the adversarial search space smaller, Buckman et al.) (2018) suggest the linearity is the main culprit and quantized inputs break up linearity.
Zhang & Liang 2019) inject random Gaussian noise into an image and then discretize it. This method fits into the noisy channel framework with different definitions of C(z). They show improved performance of this combined model in the non-adaptive setting. Our experiments find that simply injecting Gaussian noise (or even Uniform or Laplace noise) is an equally effective defense method. We also show that the discretization is not essential to good performance if the level of noise is appropriately tuned. The imprecise channel defense in a neural network is also related to the idea of gradient masking or gradient obfuscation, i.c., a hard to differentiate layer (Papernot et al. 2007)). In this work, the backward pass computation is perturbed to make it difficult for a gradient- based attack to synthesize an adversarial image (while the forward pass is kept the same). We implement both non-adaptive attacks and adaptive that can observe the channel and take an approximate gradient through it. We further focus our study on the families of white-box attacks proposed by Carlini & Wagner (0177), and their adaptive variants.
Noise injection can be much more powerful than regularization or dataset augmentation method. The dropout algorithm can be seen as applying noise to the hidden units. The dropout randomiza- tion (Feinman et al., 2017) was used to create a defense that was not completely broken and required high distortion added to the adversarial examples (Carlini & Wagner 2017). Many new defenses propose randomization through noise injection without considering the adversarial training (Zhang & Liang 2019, Cohen et al.] 2019). The work on injection of noise into inputs and each of the layers of neural networks by Liu et al. (2018) is a strong heuristic that led to defenses with theoretical guarantees. The random smoothing provides certified robustness up to certain threshold of input distortion (and is not designed to be robust beyond the threshold) by utilizing inequalities from the differential privacy literature (Lecuyer al. 2018). Cohen al. (2019) improve the theoretical bounds of methods that randomly smooth the input examples. Recent work focuses on combination of randomized smoothing with adversarial training and achieves state of the art in terms of the provable robustness (Salman et al.) 2019).
'The distillation is form of compression, however, the defensive distillation does not result in smaller models.
2

---

Under review as a conference paper at ICLR 2020
3 LOSSY CHANNEL MODEL
We consider convolutional neural networks that take w x h (width times height) RGB digital images input, giving example space of X' (255)"xxx3 where (2) denotes the integer numbers from 0 to 2. We consider a discrete label space of classes represented as a confidence value y E [0, 1]". Neural networks are parametrized functions (by weight vector 0) between the example and label spaces f(z; 0) X + V.
An adversarial input Fado is perturbation of correctly predicted example that is incorrectly predicted by f.
f(x) * f(
The distortion is the la error between the original example and the adversarial one:
8ado = | Tadv 2
3.1 MODEL
Approximating f(-) with less precise version f()) can counter-intuitively make more robust (Dziugaite et al. 2016):
=xx)= f(#ado)
=
Intuitively, lossy version of introduces noise into prediction which dominates the strategic perturbations found by an adversarial attack procedure. It turns out that we can characterize a number of popular defense methodologies with this basic framework.
Let x be an example and f be a trained neural network. Precise evaluation means running fx) and observing the predicted label. Imprecise evaluation involves first transforming through deterministic or stochastic noise process C(z) = C(x æ]) and then evaluating the neural network
y= f(x') x' ~ C(z)
We think of C(x) as noisy channel (as in signal processing). The distortion of a C(x) is the expected lg reconstruction error:
5, = E(IC((> ==|2];
which is a measure of how much information lost passing the example through a channel.
This paper shows that there is subtle trade-off between ó, and 8ado- In particular, find 8 such that &, >> Sadt and / (x) = (C((aa))) We show that compression and randomization based techniques exhibit this property.
EXAMPLE OF DETERMINISTIC CHANNEL
When C(x) is deterministic it can be thought of as a lossy compression technique. Essentially, we run the following operation on each input example:
x' = compress(r)
One form of compression for CNNs is color-depth compression. Most common image classification neural network architectures convert the integer valued inputs into floating point numbers. We abstract this process with the no rm function that for each pixel E (255) maps it to real number E [0, 1] by normalizing the value and the corresponding denorn rm function that retrieves the original integer
value (where denotes the nearest integer function) +
norm(n) := 255
n
denorm(u) == 255 * e.
This process is normally reversible norm( (m((denommv)) (v)), but we can artificially make this process lossy. Consider a parametrized C(-) version of the color-depth compression function:
C(u, 6) = 2b - 1(2 - 1) * u
By decreasing b by A6 we reduce the fidelity of representing by a factor of 2 (for the bits of precision).
Mooe complex normalization schemes exist but for ease of exposition focus this simple process.
3

---

Under review as a conference paper at ICLR 2020
EXAMPLE OF STOCHASTIC PERTURBATION
The channel model is particularly interesting when C(x) is stochastic. Randomization has also been noted to play a big role in strong defenses in prior work (Madry et al. 2017; Zhang & Liang 2019
Cohen et al. 2019). For example, we could add independent random noise each input pixel:
= x +
We consider two schemes, Gaussian e ~ N(0, σ) and additive Uniform noise e ~~ U(-B, B) which add independent noise to each pixel.
One of the advantages of randomization that an adversary cannot anticipate how the particular channel C will transform input before prediction. However, there is another subtle advantage to randomization. Randomized approaches can partially recover their loss accuracy due to imprecision by averaging over multiple instances of the perturbation. In classification problems, we can take the most frequent label seen after perturbation trials:
f(x) arg max
Σ E5x 1...k
3.2 PERTURBATION ANALYSIS
While the intuition is that the channel's perturbations dominate strategically placed distortions in an adversarial example, the underlying mathematical mechanism of why recovery possible clear. We start with the hypothesis that synthesized adversarial examples have unstable predictions-meanigg that small perturbations to the input space can change confidence values drastically. How do we quantify instability?
Let fx) be function that maps an image to a single class confidence value (i.e., scalar output). We want to understand how f(x) changes if x is perturbed by We can apply Taylor expansion of around the given example x:
+ VV 2 (V(()) +
where fz) denotes the gradient of the function with respect to x and (x)(() denotes the Hessian of the function if with respect to æ. The magnitude of the change in confidence is governed by the Taylor series terms in factorially decreasing importance. ||e ||2 exactly the distortion measure 5, described at the beginning of Section 3.1 Thus, the expression is bounded in terms of the operator norm, or the maximal change in norm that could be induced, of each of the terms (see Appendix A):
1
Vv.f(( + 2zd (if((a)) + S 8, M1 (x) + 82 M2(x) +...
2
As (x) is vector, this is simply the familiar &a norm, and for the second order term this is the maximal eigenvalue:
(z = [V=f(x))lz M2(x) = Amar V2f(())
When Mj and M2 larger this means there is greater propensity to change the prediction for small perturbations. We will show experimentally that for certain types of attacks the M1 and M2 values around adversarial examples exhibit signs of instability compared to those around natural examples-suggestiig mathematical mechanism of why recovery possible.
4
EXPERIMENTS
Our experiments evaluate the efficacy of imprecision based defenses number of different adver- sarial problem settings.
4.1
EXPERIMENTAL SETUP
We run our experiments using ResNet-18 on CIFAR- 10 and ResNet-50 on ImageNet dataset using P- P-000 GPUs (16GB memory). We explore number of different attacks that are implemented in
4

---

Under review as a conference paper at ICLR 2020
the foolbox library (Rauber et al. 2017). In each experiment measure the test accuracy (%), the confidence of predictions, and distances between the original images and either their adversarial counterparts the recovered images after applying non-targeted attacks; if the adversary is successful it induces any misclassification. We experiment with many gradient-based attacks provided in the foolbox library that explore different optimization algorithms and distance measures, for instance:
of the defenses. We present results for
LBFGS minimizes the distance between the input image and the adversarial example as well as the cross-entropy between the predictions for the adversarial and the input image; introduced by (Szeedyy al. (2014) and further extended in Tabacof & Valle (2015).
Carlini- Warnin L2 (C&W I2) is generalization of the LBFGS attack that is devised after exhaustive search over possible space of: norms, loss functions, box optimization procedures, etc. (Carlini & Wagner 2017).
BIM L, is a modified version of the Basic Iterative Method that minimizes the I, dis- tance (Kurkkin et al. 2016).
FGSM adds the sign of the gradient to the image, gradually increasing the magnitude until the image misclassified Goodfellow et al. (2014).
PGD L the Projected Gradient Descent Attack that is an iterative version of the FGSM attack; use the version that minimizes the Lo distance (Madry et al. 2017).
We extend the Carlini- Carlini-Wagner T22 attack in its adaptive version so that the gradients are not obfuscated. We approximate the gradients for the backward pass on the compression layers as an identity function, similarly to (He et al., 2017 Athalyy et al. 2018). More details on the setup can be found in the
Appendix, Section C.13
1. PERTURBATION DEFENSES GIVE SIMILAR GAINS IN ROBUSTNESS
In the first experiment, select 1000 random images from ImageNet and CIFAR-10 datasets. For each of these images we generate an adversarial attack with popular white-box gradient-based methods. Then, we run the adversarial images through different channels: Frequency Compression (FC), Color Depth reduction (CD), Uniform noise (Unif), Gaussian Noise (Gauss), SVD-based compression (SVD), Identity (Iden). The identity channel just passes through the adversarial image with no modification. We measure the accuracy of f(C(x)), which indicates the ability of the imprecise channels to recover the original label. We present the results in Figure and also in the Supplement in Figure (for all images in the test CIFAR-10 and dev ImageNet sets) and in Table (for different channel parameters and five attacks).
When there is identity channel, the adversarial attack is always successful. However, each of the imprecise channels is able to recover a substantial portion of original labels from the adversarial examples. It is important to note that we evaluating these attacks in the setting, where any misscaassiiication is considered a success and the adversary is not aware of the defense.
Importantly, all the channels can be tuned to recover very similar maximum accuracy after attacks. For example, all the channels can achieve about 85% accuracy for CIFAR-10 after non- adaptive PGD C&W attacks. This suggests that any form of imprecision with the right error magnitude is effective at defending against these types of attacks. We observe that the attacks that incur higher distortions such as FGSM LBFGS decrease the accuracy more than the iterative attacks such as C&W L2 or PGD L This is because the iterative attacks find the adversarial images that are closer the original images in terms of the corresponding distance measure that they optimize for in the input space. The key is to ensure that the error introduced by the imprecise channels big enough to dominate the adversarial perturbations but small enough to generate valid predictions. Figure illustrates this relationship (see also detailed analysis for single image presented in Figure in the Appendix). For five different imprecise channels, we plot the channel distortion against the accuracy for CIFAR-10 10 and ImageNet datasets. We analyze FC, CD, Unif, Gauss, and SVD channels. The curves are qualitatively similar in the low noise regime, but they show more differences when higher distortions are incurred. The lowest accuracy across the datasets for high distortions is observed for content- content-preservee preserving FC and SVD compression channels. The Gaussian and Uniform channels have very similar trends. They outperform other channels ImageNet for
5

---

Under review as a conference paper at ICLR 2020
100
C&W Lg
CIFAR-10
100
CSW L2
ImageNet
100
PGD L. CIFAR-10
100
PGD L. ImageNet
100
FGSM ImageNet
Uni
Lnf
uni
uri
Urif
80
Gauss
80
Gauss
80
6suss
80
Gouss
80
Gouss
9WD
SVD
5VB
S/D
SVD
Accuracy
60
60
60
60
60
40
40
40
40
40
20
20
20
20
20
10
%
50 L2 distortion
100 %
5
10
50
100
50
100
L2 distortion
L2 distortion
L2 distortion
L2 distortion
We plot the channel distortion against test accuracy (%). The distortion of the imprecise channels has to be large enough to recover the correct label but not so large that it degrades model performance. The base test accuracy about 93.5% for CIFAR-10 and 83.5% for ImageNet on 1000 randomly chosen images (results for the full test CIFAR-10 10 set and the full dev ImageNet set can be found in the Appendix in Figure 77. The experiment is run for the C&W L2 attack with 100 iterations and the PGD attack with 40 iterations.
Figure 1:
Table 1: Transferability of the adversarial images created against given noisy channel denoted (adaptive attack specified in the first column) to the defense protected with a noisy channel denoted as D (the defense with a noisy channel specified in the first row). Each result represents a recovery (%) of the adversarial examples (generated for A) to correct labels after applying the defense (D). We use 30% FC compression, 50% SVD compression, bit values in CD, 0.03 noise level for Gauss and Laplace, and 0.04 noise level for the Uniform channel. We use 2000 images from the CIFAR-10 test set and 100 attack iterations with binary steps to find the e value (with initial e value set to 0.01) for the adaptive C&W Le attack. The baseline test accuracy is 93. 33.66% The test accuracy of the noisy channels on clean images is given in the first row denoted: Empry (an empty attack).
FC
CD
SVD
Gauss
Uniform
Laplace
A
Empty
93.32
93.01
93.12
92.53
91.6
91.35
FC
0.20
80.75
83.05
81.15
79.65
78.70
CD
3.85
0.70
43.60
47.30
60.45
62.35
SVD
1.99
47.96
0.77
46.52
62.87
65.75
Gauss
4.45
48.70
44.80
51.50
61.75
60.15
Uniform
3.45
30.30
30.60
30.15
48.05
$1.55
Laplace
3.05
23.35
24.60
23.80
39.15
46.70
large distortions but are less performant on low-resolution CIFAR-10 images, where the CD channel achieves higher accuracy.
2. ATTACKS ARE TRANSFERABLE
Many input transformation defenses are broken. If the attacker has full knowledge of the defense, it possible to construct an attack that is impervious to the defense. This is called the adaptive setting. Since the underlying mechanisms of input transformations are similar, we find that an attacker does not need to be fully adaptive. The attacker can assume a particular strong defense and that same adversarial input often transfers to other defenses. We narrow the attacker to single adaptive step (for details see Section I. the Appendix). Even in this weak adaptive setting, the deterministic channels are fully broken but the randomized channels retain relatively high accuracy above 23. 23.8%. We show in the Appendix in Figure 14 that the randomized defenses can also be broken when the adversary is given an unlimited number of adaptive steps.
Laplace attacked images transfer the best to other defenses. They decrease the accuracy of the defense models by at least 44.3% (for Laplace itself). Table shows that FC attacked images do not transfer well to other defenses; the maximum drop in accuracy of the model protected by other defenses is 12.26%. Most adversarial images (against given defense) transfer very well to the FC defense, i.c. an adversarial image against any defense (c.g. CD, SVD, Gauss, Uniform, or Laplace) is also adversarial against the FC defense. The adversarial images generated against the Uniform

---

Under review as a conference paper at ICLR 2020
defense show better transfer to other defenses in comparison to the adversarial images generated against the Gaussian defense. This is because the higher noise level is applied in the Uniform defense. We observe analogous trends for the ImageNet dataset and present the results in the supplement (Tables and 7
3. RECOVERABLE RANGE SHRINKS WITH HIGHER ATTACK DISTORTION
We use an example from the ImageNet dataset, the ResNet-50 model, and set the stochastic channel to the Gaussian noise. We start from an adversarial example generated with the Carlini & Wagner (non-adaptive) T22 attack and for consecutive subplots (in the left to right and top to bottom sequence), we increase the attack strength and incur higher distortion of the adversarial image from the original image, For a single plot, we increase the L2 distance of the output from the channel to the adversarial example by increasing the Gaussian noise (controlled by parameter σ). For L2 distances incurred by different noise levels, we execute 100 predictions. In Figure in the Appendix, we use the frequency count and report how many times the model predicts the original, adversarial other class. The plot shows what range of distances from the adversarial image reveal the correct class. For the adversarial examples that are very close to the original image (e.g. adversarial distance of 0.006 for the top-left figure), the window of recovery (which indicates which strengths of the random noise can recover the correct label) relatively wide, however, as we increase the distance of the adversarial image from the original image, the window shrinks and finally we are not enable to recover the correct label.
For attacks that incur more input distortion or the stronger white-box attacks, can resort to other statistics as proposed in Roth et al. (2019). They show that adversarial examples are much closer to the unperturbed sample than to any other neighbor and use the fact that the probability of the correct class increases faster than the probability of the highest other class when adding noise with a small to intermediate magnitude to the adversarial example.
4. COMPARISON WITH OTHER METHODS
Test accuracy [%
0.8
10a
Frequency count
Image type: original
0.6
Channel type:
oredy therel
102
adversarial
0.4
10
10a
0.2
0.0
10
10-3
10
10 103
100
200
300
400
param (strength of C&W L2 attack)
Eigenvalue magnitude
Non-adaptive attack and the recov- erable ranges in terms of the с parameter that controls the strength of the C& W I.2 attack. We systematically change the e parameter and keep the parameters for the channels unchanged. We use the VGG-16 network and all images from the CIFAR-10 test set. The test accuracy of the model without any noise layers and on the clean data is 85.23%.
Figure 2:
Figure 3: The top eigenvalues of the Hessians with respect to (w.r..) the input 1024 images from the CIFAR-10 dataset trained on the ResNet- 18 architecture. We plot the histogram that shows counts of magnitudes for the eigenvalues. Yao ct al. (2018) show analysis of the Hessian w.r.t. parameters and we extend it to analyze Hessian w.r.t. inputs.
In Figure we present the accuracy of different channels as the strength of the C& W attack is systematically increased. The strength is controlled by the tradeoff-paramett that is used to set the relative importance of distance and confidence.
We present related approach which is the RSE (Random Self-Ensemble) network with 0.2 noise level in the first layer and 0.1 noise level in the remaining layers (as recommended in Liu et al. (2018). This defense does better for lower distortion levels (c value below 0.1) than other noisy channels, but then its accuracy deterioration is faster for higher distortion levels. The Laplace channel gives the highest accuracy for high values of the e parameter (above 1.0). The CD, FC, SVD, Gauss, and
7

---

Under review as a conference paper at ICLR 2020
Uniform channels show similar trends. We also include very simple channel that reduces brightness of an image by subtracting an arbitrary value from each pixel. The comparison between very complex approaches and a simple input transformation is informative- as they largely follow the same trends.
5. ADVERSARIAL EXAMPLES ARE UNSTABLE
Original image
40 Adversarial: high confidence
Adversarial: low confidence
Gill,elxog Coalll
Edlla_exx elx.e Cuglll>
25
etxxx
20
15
recovered not recovered
recovered
recovered not recovered
recovered
%
10
30
%
%
10
20 Cxll Casilll
30
Ex Cas/llla
Figure 4: Comparison of magnitude of gradients for original and adversarial images.
First-order analysis. The key question is why adversarial examples are more sensitive to perturba- tions than natural inputs, when there is evidence that from an input perspective they are statistically indistinguishable. Our experiments suggest that this sensitivity arises from the optimization process that generates adversarial inputs. Based on the operator-norm analysis presented in Section 3.2 we plot in Figure 4 the L2 norm of gradients w.r.t. the original Worg and adversarial Wads images for original (correct) Corg and adversarial classes Cadv. The work by Himon--Gabres er al. (2019) also evaluates the norm of gradients of the network output with respect to its inputs. They show that the adversarial examples are primarily caused by large gradients of the classifier captured via the induced loss. At first-order approximation in c, an c-sized L-2 untargeted adversarial attack increases the loss L at point by < <|0,_ L(x, Corg) 12- Analogously, at first-order approximation in c, an e-sized L2 targeted adversarial attack decreases the loss L at point æ by el10,[(x,, Cadv)) In our experiments, the recovered original images have lower norms of the gradients for both original and adversarial classes than the not recovered images. Additionally, the gradient w.r.t. original image for the original class is much smaller than the gradient w.r.t. adversarial image for the adversarial class. We also observe that the magnitudes of gradients change smoothly as we systematically add more Gaussian noise to the original or adversarial images (Figures 15 T6p. For the adversarial image, the gradient w.r.t. adversarial image and for adversarial class decreases we increase the attack strength and the confidence of adversarial predictions.
Second-order analysis. Our results in Figure 3 show that the adversarial inputs lead to noticeably higher Hessian spectrum than the original inputs. This suggests that the model predictions for the adversarial inputs are less stable than for the original images. Thus, perturbations of the adversarial images with some form of noise can easily change the classification outcome while the prediction for the original images are much more robust and do not lead to such unstable predictions.
CONCLUSION, LIMITATIONS, AND FUTURE WORK
The non-adaptive attacks are not robust since small changes to the adversarial input often recover the original label. This is an obvious corollary to the very existence of adversarial examples that by definition are relatively close to correctly predicted examples in the input space. Random perturbations of the input can dominate the strategically placed perturbations synthesized by an attack. In fact, the results consistent across both deterministic and stochastic channels that degrade the fidelity of the input example. From the perspective of the attacker, the recovery window can be closed to make the perturbation based recovery techniques ineffective.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018.
Ayse Elvan Aydemir, Alptekin Temizel, and Tugha Taskaya-Temizel. The effects of JPEG and JPEG2000 compression on attacks using adversarial examples. CoRR, 104181088,, 2018. URL httptttpp/axxi..rrrt aws/ 10418
Battista Biggio, Igino Corona, Davide Maiorea, Blaine Nelson, Nedim Šmdićć Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference machine learning and knowledge discovery in databases, pp. 382-402. Springer, 2013.
Jacob Buckman, Aurko Roy, Colin Raffel, and Ian Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In International Conference on Learning Representations, 2018.
N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP). pp. 39 39-57, May 2017.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 3-14. ACM, 2017.
Jeremy M Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. arXiv preprint arXiv: 1902.02918, 2019.
Nilaksh Das, Madhuri Shanbhogue, Shang- Tse Chen, Fred Hohman, Li Chen, Michael E. Kounavis, and Duen Horng Chau. Keeping the bad guys out: Protecting and vaceinating deep learning with JPEG compression. CoRR, abs/1705.02900 2017. URL http: httt::///txxii/// http:://wxwvvv.....aaaa 02900
Nilaksh Das, Madhuri Shanbhogue, Shang-Tse Chen, Fred Hohman, Siwei Li, Li Chen, Michael E. Kounavis, and Duen Horng Chau. Shield: Fast, practical defense and vaccination for deep learning using JPEG compression. CoRR, abs/1802.068// 2018. URL http: /ttpxxxxiv... org.orgg//oo.... /abb/
1802. 06816
Gintare Karolina Dziugaite, Zoubin Ghahramani, and Daniel M Roy. A study of the effect of jpg compression on adversarial images. arXiv preprint arXiv: 1008..0880,,, 2016.
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint /703v: 00.700043110.. 10, 2017.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1rXiv:1412.665: 2014.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering Adversarial Images using Input Transformations. arXiv e-prints, art. arXiv:1711.001 Oct 2017.
Warren He, James Wei, Xinyun Chen, Nicholas Carlini, and Dawn Song. Adversarial example defense: Ensembles of weak defenses are not strong. In / Ithh USENIX Workshop Offensive Technologies (WOOT 2017.
Mehdi
Tasmin Chowdhury, Hsin- Tai Wu, and Sayandev Mukherjee. Ppd: Permuta- tion phase defense against adversarial examples deep learning. arXiv preprint 4rX1v:1812.10049, 2018.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. CoRR, bbss/667700233, 2016.
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified Ro- bustness to Adversarial Examples with Differential Privacy. arXiv e-prints, art. arXiv: 102.1110,, Feb 2018.

---

Under review as a conference paper at ICLR 2020
Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsich. Towards robust neural networks via random self-ensemble. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision - ECCV 2018. pp. 381-397, Cham, 2018. Springer International Publishing. ISBN 978-3-033--11234-..
Zihao Liu, Qi Liu, Tao Liu, Nuo Xu, Xue Lin, Yanzhi Wang, and Wujie Wen. Feature distillation: Dnn- oriented jpeg compression against adversarial examples. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1 06083, 2017.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation defense to adversarial perturbations against deep neural networks, arXiv preprint arXiv:/511.045000 2015.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp. 506-519, ACM, 2017.
Aaditya Prakash, Nick Moran, Solomon Garber, Antonella DiLillo, and James Storer. Deflecting adversarial attacks with pixel deflection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox to benchmark the robustness of machine learning models. arXiv preprint 1XXx:1707.041311 2017.
Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: statistical test for detecting adversarial examples. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5498-5507, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http:
htttp://proceedl.ngg mlr. press /v v97 wttp:ttttt........ html
Hadi Salman, Greg Yang. Jerry Li, Pengchuan Zhang, Huan Zhang, Ilya P. Razenshteyn, and Séhastien Bubeck. Provably robust deep learning via adversarially trained smoothed classifiers. CoRR, abs/1906.04584, 2019, URL /ttp: org/ abs/ 190./99066..00 00000.00484)
Carl-Johann Simon-Gabriel, Yann Ollivier, Leon Bottou, Bernhard Schölkopf, and David Lopez-Paz First-order adversarial vulnerability of neural networks and input dimension. Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, Long Beach, California, USA, 09-15 Jun 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. CoRR, abs/1312.6193 2014.
Pedro Tabacof and Eduardo Valle. Exploring the space of adversarial images. CoRR, 151.05530 2015. URL http: http///axvvvvv. orq/ abs 1111.11111.1 C5328
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. arXiv preprint 1r1iv:1711.01990 2017.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint arXiv:1704.01 2017.
Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. Hessian-based analysis of large batch training and robustness to adversaries. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. DesacBiinchh,, and R. Garnett (eds.), Advances Neural Information Processing Systems 31, pp. 4949 4959, Curran Associates, Inc., 2018
Yuchen Zhang and Perey Liang. Defending against whitebox adversarial attacks via randomized discretization. AISTATS, 2019.
10

---

Under review as a conference paper at ICLR 2020
A PERTURBATION ANALYSIS: ADDENDUM
1 82 M2(2) +...
Vv.(() + 2 +.. + 2-
(1 [+)(1)) M2(x) Maa (2)(
(1) vv)()) S d. M1(x)
From the Cauchy-Schward inequality: e" v2v((z) S ||ll12 M1
|/e||2 M1 dadv Mi (1)x)= S 8. M1(x) ( since dadv << 8c)
(2) (=f() 82 M2(x)
(2f(( Ee
From the definition of maximum eigenvalue Amax
e Vovffzz) = Sadu S oe Ammax
B COMPRESSION TECHNIQUES
B. FFT-BASED COMPRESSION
We apply compression in the frequency domain to reduce the precision of the input images. Let x be an input image, which has corresponding Fourier representation that re-indexes each tensor in the frequency domain:
F F(w] F(r(n))
This Fourier representation can be efficiently computed with an FFT. The mapping is invertible 1-1 (F)x)) Let My [w] be discrete indicator function defined follows:
1, f My[w] = 0, >
My[w] is mask that limits the F[W] to a certain band of frequencies. f represents how much of the frequency domain is considered. The band-limited spectrum is defined as, Flw]- M [w]], and the band-limited filter is defined as:
x' (F[w( My(w])
B.2 SVD- BASED COMPRESSION
Analogously the FaT-based method, we decompose an image with SVD transformation and reconstruct its compressed version with dominant singular values. The basis used in SVD are adaptive and determined by an image, as opposed to pre-selected basis used in FFT. This can result in higher quality for the same compression rate in case of SVD, however it is more computationally intensive than FFT-based compression.
C ADDITIONAL EXPERIMENTS FOR WHITE-BOX ATTACKS
C.I ACCURACY OF PERTURBATION DEFENSES ON CLEAN DATA
One pitfall the imprecise channel defense is that it introduces errors whether or not there are any adversarial examples. The enrors act upper-bound for the best possible test accuracy under adversarial perturbations. For frequency compression, color depth compression, and uniform noise injection, we compare the test accuracy for different levels of imprecision. Table shows the results for all test images from CIFAR- 10 on the ResNet- 18 architecture, for three of the imprecise channels, and for different noise settings.
get
11

---

Under review as a conference paper at ICLR 2020
Table 2: On CIFAR-10 with R8sNet-18, measure the test accuracy for imprecise channels without any adversarial perturbation. This signifies the amount of accuracy we sacrifice with respect the baseline test accuracy of the model (without any perturbations of the images); 93.56%.
FC (%)
Acc. (%)
CD (bits)
Acc. (%)
Uniform (t)
Acc. (%)
93.5
8
93.4
0.009
93.52
10
93.42
6
93.3
0.03
92.59
50
91.6
4
91.9
0.07
85.2
75
79.53
2
87.4
0.1
70.67
100
CIFAR-10
100
CIFAR-10
FC
CD
SVD
Geuss
Uniform
Laplace
90
90
80
80
Accuracy (%)
Accuracy (%)
70
70
40
40
30
38.00
20
40
60
80
0.02
0.04
0.06
0.08
0.10
Compression
(%)
Epsilon (strength nolse)
ImageNet
ImageNet
90
90
FC
Gauss
Uniformm
Laplace
80
80
70
70
60
60
Accuracy (%%
Accuracy (%)
40
30
30
20
20
10
10
%
20
40
60
80
8.00
0.02
0.04
0.06
0,08
0.10
Compression rate (%)
Epsilon (strength of the naise)
Figure 5: The test accuracy after passing clean images through six different noisy channels, where the added noise is controlled by the compression rate and epsilon parameters. We use full CIFAR-10 10 test set for ResNet-18, 18, and full ImageNet validation set for ResNet-50.
The test accuracy of the models can be increased by training with compression, e.g., by using FFT based convolutions with 50% compression in the frequency domain increases the accuracy to 92.32%.
We present the results for six different noisy channels; three of them are compression based: FC, CD, SVD, and other three add different type of noise: Gauss, Uniform, and Laplace. For each of the compression based channels, we increase the compression rate systematically from 0 to about 90% (in case of the CD channel, the compression rate is computed based on how many bits are used per value). For the noise based channels, we increase the strength of the noise by controlling the epsilon parameter é (in case of the Gaussian noise, it corresponds to the sigma parameter σ). The full result is presented in Figure
12

---

Under review as a conference paper at ICLR 2020
100
L2 C&W
L1 BMI
Linf FGSM
LBFGS
80
70
Test Accuracy (%)
50
40
20
10
Baseline acturacy
0.001
0.002
0.03
0.07
0.1
0.2
0.4
Epsilon strength the noise)
The test accuracy after four different attacks and the recovery via the uniform noise stochastic channel with different parameters e. For the Carlini-Wagner L2 attack, the best performing parameter 0.03. We run the experiment for ResNet-50 on ImageNet (1000 samples).
Figure
100
PGD (random start) L. CIFAR-10
100
C&W Lg CIFAR-10
100
C&W L2 ImageNet
FC
Gauss
FC
Gouss
FC
Gauss
Laplace
Laplace
80
80
80
Accuracy 1%)
60
60
20
20
8.0 2.5
0.0
5.0 L2 distortion
%
20
La distortion
L2 distortion
Figure 7: We plot the channel distortion against accuracy (%), analogously Figure The experiment run for the PGD attack with 40 iterations and C&W L2g attack with 100 iterations on all images from the test CIFAR- and the dev ImageNet datasets. The adversary is aware of the defense. The test accuracy on the full clean data is 93. 3656 and 76.13% for CIFAR-I0 and ImageNet, respectively. Interestingly enough, we observe that the C&W attack is on the sub-pixel level for ImageNet and rounding to the nearest bit integers in the CD channel has high recovery rate, while for bit integer the accuracy drops slightly due to imprecision.
6 presents the test accuracy (%) after four different attacks different norms) and the recovery via the uniform noise stochastic channel in more detail.
Figure
C.2 CHANNEL DISTORTION VS ACCURACY
We show channel distortion accuracy for the C&W L2 attack the full CIFAR- 10 test set and full ImageNet dev set in Figure 7
We further show the performance of the defenses across different attacks and 1000 images in Table
13

---

Under review as a conference paper at ICLR 2020
Table 3: Given adversarial input, pass the input through channel before prediction. We evaluate the accuracy (%) of the classifier over 1000 images, the best possible accuracy is listed in the Baseline column. For the channels, bits per value, and the strength of the attack (c). We use the attacks described in Section: 4.1
report in parentheses: compression used (%), number of
Attack
Data set
Baseline
FC (%)
CD (bits)
Uniform (e)
Gauss (e)
Iden
BIM IA
CIFARIO 10
93.5
86.2 (20)
85.1 (4)
82.8 (0.03)
82.1 (0.03)
LBFGS
CIFARIO 10
93.5
82.8 (50)
80.2 (4)
79.2 (0.04)
79.3 (0.05)
C&W L2
CIFARIO 10
93.5
85.2 (20)
84.4 (4)
84.3 (0.01)
84.8 (0.02)
O
FGSM
CIFARIO
93.5
79.0 (50)
44)
49.4
49.9 (0.03)
PGD L00
CIFAR10
93.5
88.6 (10)
84.
85.7 (0.01)
(0.9
BIM 4
ImageNet
83.5
81.5 (10)
(4)
81.2 (0.0099
81.2 (0.009)
LBFGS
ImageNet
83.5
71.7 (70)
(4)
76.4
76.5 (0.07)
C&W Le
ImageNet
83.5
78.7 (50)
80.9 (4)
81.4 (0.03)
80.4 (0.03)
FGSM
ImageNet
83.5
73.8 (50)
74)
75.4 (0.03)
75.4 (0.02)
PGD Lx
ImageNet
83.5
82.1 (5)
(2.9
82.0 (0.007)
80.9 (0.01)
C.3 CHANNEL ACCURACY ON CLEAN AND ADVERSARIAL EXAMPLES
We compare the accuracy of the perturbation channels on clean and adversarial examples in Figure g We plot the results for the whole test set from CIFAR- 10 and the whole validation set from ImageNet. We use two deterministic channels (FFT compression denoted by FC and SVD compression) and two noisy channels (Gaussian and Uniform noise). We use C&W and PGD attacks. The FC Clean label denotes that we pass clean images through the channel that applies FFT compression. The SVD C& w label denotes that we pass adversarial images found with the C& W attack through the channel that applies SVD compression. We tune the channels for given dataset across the attacks, and present the results in Table 4 The accuracy of the channels on clean data gives us the upper bound for the accuracy the adversarial examples. Thus, choose the channel parameter based on the highest accuracy on the adversarial images.
Table 4: Channel tuning. The best parameters for the perturbation channels when tuned on the PGD and C& W attacks.
Dataset
CIFAR-10
ImageNet
Channel
FC (%)
20
60
SVD (%)
40
70
Gauss (e)
0.015
0.04
Uniform (c)
0.025
0.04
C.4 NEIGHBORHOOD OF ADVERSARIAL EXAMPLES
Please, Figure 9 and description in Section 4.1
C.5 DISTRIBUTIONS OF DELTAS BETWEEN INPUTS & OUTPUTS FOR CHANNELS
We plot the distribution of deltas for six imprecise channels in Figure 10 We compute the deltas by subtracting an original image from the perturbed adversarial image and plot the histograms of differences. We were recovered. We use the C&W attack with 1000 iterations and the initial value == 0.01.
image from the ImageNet dataset. For all the examples, the correct labels
The CD channel resembles the Uniform distribution. The FFT and SVD compression methods belong to double-sided exponential distributions, thus they are more related to the Laplace distribution than to the Gaussian distribution.
14

---

Under review as a conference paper at ICLR 2020
CIFAR-10 C&W
CIFAR-10 10 PGD
100
100
80
80
Accuracy (%)
60
60
40
20
20
0
%
20
40
60
80
100
20
40
60
80
100
Compression (%)
Compression (%)
CIFAR-10 C&W
CIFAR- 10 PGD
100
100
80
80
Accuracy (%)
60
60
Gauss, Clear
Gauss C&W
Unif Clean
Unif C&W
40
40
20
20
Gauss Clean Unif Clean
Gauss PGD
Unif PGD
8.00
6.00
0.02
0.04
0.06
0.08
0.10
0.02
0.04
0.06
0.08
0.10
E (noise strength)
(noise strength)
ImageNet C&W
ImageNet C&W
100
100
FC Clean SVD Clean
FC C&W
Gauss Clean Unif Clean
..... Gauss C&W
SVD C&W
Unif C&WW
80
80
Accuracy (%)
60
60
40
40
20
20
%
8.00
20
40
60
80
100
0.02
0.04
0.06
0.08
0.10
Compression (%)
E (noise strength)
Comparison of accuracy of the perturbation channels on clean and adversarial images. We pass either clean adversarial images through the channels and measure their accuracy. The accuracy on clean inputs gives upper bound for the accuracy on the adversarial examples. The channels can be tuned based on their accuracy after different attacks.
Figure 8:
15

---

Under review as a conference paper at ICLR 2020
Carlnnitegrer (adv: dist: 0.006)
CarlinWWagnerr ZAttack tadv, dist: 0.016)
100
100
Frequency count
80
60
crigna
orglest
rdvensrrial
sther
20
10
20
10
10
10
161
10
10
distarce
12 distance
Carlinitagrert 2Attack dist: 0.033)
CarlinWWlgneet ZAttack (adv, dist: 0.104)
100
100
Trequency count
60
crggaal
orglest
advesarial
adeersaro
40
other
20
10
10
10
10
10'
10'
10
10
distarce
12 distance
CarliniWagnert Dattack dist: 0.297)
CarlinWalanee
1.590)
100
100
Frequency count
80
60
crgral
orglas
adveraris
adeerssro
40
sther
107
10'
10a
10
L2 distance
L2 distance
Figure 9: Frequency of model predictions for original, adversarial, and other classes as we increase the distance from an adversarial example using Gaussian noise.
16

---

Under review as a conference paper at ICLR 2020
CD (4 bits) L2 dist: 6.6261
FC (10%) L2 dist: 3.0144
Frequency
Frequency
20000
2000
0
-0.02
0.00
0.02
0.0
0.1
Value
Value
Uniform (0.03) L2 dist: 6.7100
Laplace (0.01) L2 dist: 3.1576
Frequency
Frequency
2000
2000
-0.02
0.00
0.02
-0.05
0.00
0.05
Value
Value
Gauss (0.03) L2 dist: 6.6962
SVD (60%) L2 dist: 1.8154
Frequency
Frequency
2000
2500
0
-0.05
0.00
0.05
-0.05
0.00
0.05
Value
Value
Figure 10: Distribution of deltas for imprecise channels.
C.6 VISUALIZATIONS OF ATTACKS AND IMPRECISE CHANNELS
Figure presents a sample image from ImageNet for the Carlini- Carlini-Wagner L2 attack.
Figure 12 shows the effect of the FC (frequency-based imprecise channel) in both spatial and Fourier
domains.
C.7 MULTIPLE TRIALS FOR STOCHASTIC CHANNELS
In addition to the robustness to non-adaptive attacks, another compelling reason to use a stochastic channel (like Uniform noise) defense is that it can be repeatedly in number of random trials. We show that doing so improves the efficacy of the defense. We randomly choose 1000 images from the CIFAR-10 test For each of these images, we generate adversarial attack. We then pass each image through the same stochastic channel multiple times. We take the most frequent prediction. Figure 1 illustrates the results.
Only 16 trials are needed to get relatively strong defense. We argue that this result is significant. Randomized defenses are difficult attack. The attacker cannot anticipate which particular pertur- bation to the model will happen. The downside is a potential of erratic predictions. We show that relatively small number of trials can greatly reduce this noise. Furthermore, the expense of running multiple trials of randomized defense is small relative to the expense of synthesizing an attack in first place.
C.8 WHITE-BoR ADAPTIVE ATTACK
We consider the problem setting when the adversary knows the defense method (i.e., has full knowledge of C())). We use the strategy deseribed in He et al. (2017) to construct attacks for each case. Not surprisingly. deterministic channels (CD, FC, and SVD) are easy for an adversary to fool when they known. However, such attacks incur higher distortion (distance to the original image)
17

---

Under review as a conference paper at ICLR 2020
when compared to attacks against unprotected networks. Intuitively, when the channel is deterministic and known, the adversary can account for the error introduced by the channel. Table illustrates the results.
Table The distortion and accuracy for the adaptive setting where the adversary knows the defense method. We report top-1 class, use 100 image samples, run 1000 iterations of C&W attack, = 0.04, apply a single random noise injection.
CIFAR-10
L Distortion
Acc. (%)
ImageNet
L2 Distortion
Acc. (%)
Iden
0.17
Iden
0.28
0
CD (b=5)
0.5
0
CD (b=5)
3.6
0
FC (c=30)
0.74
FC (c=30)
3.64
0
The stochastic channel is harder to attack with gradient-based adaptive methods. Unlike for deterministic compression, the adversary cannot anticipate the particular error pattern. We build an adaptive attack against the additive uniform noise channel. Our strategy is to send output from the adversarial algorithm through the stochastic channel and the network at least as many times as set in the defense. We mark the attack successful if the most frequent output label is different from the ground truth. The more passes through the noisy channel we optimize for, the stronger the attack. Furthermore, we many iterations of the attack to decrease the L2 distortion. An attack that always evades the noise injection defense is more difficult to generate because of randomization. The other randomized approach was introduced in dropout (Feinman et al., 2017). The attacks against randomized defenses require optimization of complex loss functions, incur higher distortion, and the attacks are not fully successful (Carlini & Wagner 2017).
In the Figure [4] we present the result of running attacks and defenses on CIFAR-10 data with single and many iterations. The defense with many trials can be drawn to 0% accuracy, however, the defense not fully optimized by the adversary (single noise injection) can result in about 40% or higher accuracy.
C.9 HYBRID APPROACHES
natural thought is whether these defenses can be made more effective by combining them. This is an approach that has been applied, for example, random discretization (Zhang & Liang] 2019). Our experiments contrast with the previous work and suggest that there is little benefit to hybrid approaches. We present an illustrative example for pair-wise combinations of Frequency Compression (FC), Color Depth (CD) compression, and Uniform (Unif) noise. Other combinations are possible but for brevity, we exclude them from the manuscript; we found no to very small benefit combining for properly tuned channels. We show the recovery rate, i.e. a fraction of original labels recovered. Table C.9 presents results for the 1.2 Carlini- Wagner attack on the MNIST, the whole test set of CIFAR-10, and the whole development set of ImageNet.
CD (bits)
FC (%)
Unif (e)
CD+FC
FC+CD
CD+Unif
FC+Unif
MNIST CIFAR-10
100
95
100
100
100
100
100
84.4(4)
85.2(20)
83.4(0.03)
86.0
83.6
85.18
86.45
ImageNet
80.9(4)
78.7(50)
80.3(0.03)
78.6
77.5
73.15
76.13
We believe these results suggest that the noisy channels do not exploit anything inherent to the images. Simply the addition of noise (through reconstruction error) is the mechanism for robustness. Composing two different schemes usually increases this noise.
C.10 TRANSFERABILITY OF THE ADVERSARIAL IMAGES
We present more results on the transferability of adversarial examples between different channels in Table and Table
18

---

Under review as a conference paper at ICLR 2020
Table 6: Transferability of the adversarial images that extend results from Table T for ImageNet dataset. We use 30% FC compression, 50% SVD compression, 4 bit values in CD, 0.03 noise level for Laplace, and 0.04 noise level for the Gauss and Uniform channels, We 3000 images from the ImageNet- 10 validation set and 100 attack iterations.
FC
CD
SVD
Gauss
Uniform
Laplace
A
FC
0.10
75.50
75.83
77.04
77.49
76.29
CD
0.17
1.16
6.77
62.60
62.04
65,46
SVD
12.02
72.33
0.46
72.79
72.52
73.09
Gauss
0.57
26.67
6.67
58.68
58.62
64.95
Uniform
0.50
26.71
6.99
58.48
59.06
64.59
Laplace 0.33
18.59
4.16
29.76
29.84
50.00
Table 7: Transferability of the adversarial images. The results are presented similarly to Figure but for different parameters. We use 50% FC compression, 50% SVD compression, 4 bit values in CD, 0.03 noise level for Laplace, and 0.04 noise level for the Gauss and Uniform channels. We use 3000 images from the CIFAR-10 validation set and 1000 attack iterations.
FC
CD
SVD
Gauss
Uniform
Laplace
A
FC
0.19
79.00
83.73
79.19
79.38
76.70
CD
6.74
0.93
47.57
62.98
63.04
65.22
SVD
78.89
47.04
0.50
59.85
63.42
67.94
Gauss
4.77
38.52
36.95
51.36
51,27
52,61
Uniform
4.65
38.14
36.35
50.14
51.08
53.21
Laplace
46.75
22.22
22.68
34.03
33,93
46.39
C.11 GRADIENT-BASE BASED ANALYSIS
We run the experiment on the ImageNet dataset. analyze only the clean images that were classified correctly.
We present how the gradient of the loss w.r.t the input image changes for correct class as we add the Gaussian noise to the original image in Figure 15
The norm of the gradient smoothly increases.
In Figure T6 we start from an adversarial image found with the default C& attack from the foolbox library. Then, systematically add Gaussian noise to the adversarial image and collect data the norm of gradients for the original and adversarial classes. The norm of the gradients for the adversarial class increases while the norm of the gradients for the original class decreases. We cross the decision boundary to the correct class very early and recover the correct labels for images. Then, we add more and more Gaussian noise, the predictions of the classifier become random and the norm of the gradients converge to single value.
In Figure 17 plot the gradients also for random class. We observe that for untargeted attack, the gradients for the original and adversarial classes are larger than for the other classes. The targeted attack decreases the loss for the target class and the gradients for the adversarial classes are lower when compared with gradients from the untargeted attacks, so fewer images can be recovered in the former case. The targeted attack causes a smaller increase of the norms gradients for the original class than the untargted attack. However, it is still higher than for random class.
C. HESSIAN-BASED ANALYSIS
We present the Hessian spectrum in Figure for top 20 eigenvalues and in Figure 19 the distribution of top eigenvalues of Hessians on ImageNet. For the adversarial images, the eigenvalues are clearly higher in both cases, which indicates higher instability and proclivity prediction changes.
19

---

Under review as a conference paper at ICLR 2020
C.13 DETAILS ON THE EXPERIMENTAL SETUP
We use the foolbox library (Rauber et al., 2017) in our experiments. We borrowed the nomenclature used for the attacks from the library. For example, the name for the attack initially proposed in Szegedy et al. (2014) and extended in 'Tabaoof & Valle (2013) is LBFGS. In most of our experiments, we also use the default foolbox parameters for the attacks. For example, for PGD the initial limit on the perturbation size epsilon is set to 0.3, step size to 0.01, default number of iterations is 40, For Carlini Wagner, we maximum number of iterations to 1000, learning rate to 0.005, initial value of the constant to 0.01. Note that for the Carlini & Wagner attack presented in 2 we use the parameter as described in (Carlini & Wagner, 2017) and the code from Liu et al. (2018). For the LBFGS attack, we use the epsilon parameter set to 0.00001 and up to 150 iterations. For the FGSM attack, different epsilons starting from 50 and up to 1000 are tried until the adversarial image is found. For the BIM L1 attack, we set epsilon to 0.3, step size to 0.05, and number of iterations to 10.
20

---

Under review as a conference paper at ICLR 2020
D ADDITIONAL EXPERIMENTS FOR BLACK-BOX ATTACKS
As a black box attack, we define an attack that does not need the knowledge about the gradient or the model.
D.1 DECISION-BASED ATTACKS
The attacks require neither gradients nor probabilities. They operate directly on the images.
D.1.1 ROBUSTNESS TO UNIFORM AND GAUSSIAN NOISE
We evaluate the robustness of band-limited CNNs. Specifically, models trained with more compres- sion discard part of the noise by removing the high frequency Fourier coefficients (FC channel). In Figure we show the test accuracy for input images perturbed with different levels of uniform and Gaussian noise, which is controlled systematically by the sigma parameter, fed into models trained with different compression levels (i.e., 0%. 50%, or 85%) and methods (i.e., band-limited vs. RPA-based Our results demonstrate that models trained with higher compression are more robust to the inserted noise. Interestingly, band-limited CNNs also outperform the RPA-based method and under-fitted models (c.g., via early stopping), which do not exhibit the robustness to noise.
Input test images are perturbed with uniform or Gaussian noise, where the sigma parameter is changed from to or 0 to 2, respectively. The more limdei model, the more robust is to the introduced noise.
D.1.2 CONTRAST REDUCTION ATTACK
This black-box attack gradually distorts all the pixels:
max + min
target
perturbed = (1 - e) image + e * target
where min and max values computed across all pixels of images in the dataset.
We can defend the attack with CD (Color Depth reduction) until certain value of epsilon, but then every pixel perturbed smooth way there are no gigh-frequeccy coefficients increased in the FFT domain of the image. The contrast reduction attack becomes low-frequency based attack when considered in the frequency domain. Another way to defend the attack is run a high-pass filter in the frequency domain instead of the low-pass filter.
We run the experiments for different models with CD and full spectra and no compression as well as model with 85% of compression with FC layers). The CD does defend the attack to some extent and the fewer pixels per channel (the stronger the CD in model), the more robust the model is against the contrast reduction attack.
sand-limited models (the model with
Test accuracy as a function of the contrast reduction attack for ResNet-18 on CIFAR-I0 (after 350 epochs) is plotted in Figure 20 We control the strength of the attack with parameter epsilon that changed systematically from 0.0 to 1.0. We use the whole test set for CIFAR-10, denotes the number of values used per channel (e.g., R=32 means that we use 32 values instead of standard 256).
D.1.3 MULTIPLE PIXELS ATTACK
The foolbox library supports single pixel attack, where given pixel is set to white or black A certain number of pixels (e.g., 1000) is chosen and each of them is checked separately it can lead to the misclassification of the image. The natural extension is to increase the number of pixels to be perturbed, in case where the single pixel attack does not succeed. We present results for the multiple pixel attack in Figure 20
'hhe Reduced Precision Arithmetic, where operations 16 bit floats used instead of 32 64 bit float numbers.
'The single and multiple pixels attacks are categorized as decision- based attacks since they treat the model oracle and require neither gradients nor probabilities.
21

---

Under review as a conference paper at ICLR 2020
D.2 SPATIAL-BASHE BASED ATTACKS
Spatial attacks apply adversarial rotations and translations that can be easily added to the data augmentation during training. However, these attacks are defended neither by removing the high frequency coefficients nor by quantization (CD) We separately apply rotation by changing its angle from 20 degrees and do the translations within horizontal and vertical limit of shifted pixels (Figure 20
D.3
SCORE-BASED ATTACKS
The score based attack require access to the model predictions and its probabilities (the inputs to the softmax) or the logits to estimate the gradients.
D.3.1 LOCAL SEARCH ATTACK
The local search attack estimates the sensitivity of individual pixels by applying extreme perturbations and observing the effect on the probability the correct class. Next, it perturbs the pixels to which the model is most sensitive. The procedure is repeated until the image is misclassified, searching for
additional critical pixels in the neighborhood of previously found ones. We run the experiments for the attack on 100 test images from CIFAR- 10, since the attack is relatively slow (Figure 20)..
22

---

Under review as a conference paper at ICLR 2020
Original label: Loafer
Adv. label: pencil box, pencil case confidence: 0.5466 L2 distance: 0.4032
confidence: 0.9561
L2 distance: 0.0
100
100
150
200
200
100
200
100
200
CD (32) label: Loafer confidence: 0.7438
FC (30) label: Loafer
confidence: 0.8995
L2 distance: 7.2698
L2 distance: 12.6384
50
50
100
100
150
150
200
200
100
200
100
200
Gauss (0.03) label: Loafer confidence: 0.9642
Uniform (0.03) label: Loafer
confidence: 0.966
L2 distance: 16.0992
L2 distance: 16.088
50
50
100
100
150
150
200
200
100
200
100
200
Figure 11: We plot sample image from the ImageNet dataset in its original state, after adversarial (white-box, non-adaptive) attack, and then after recovery imprecise channels: CD (color depth reduction with 32 bits), FC (30% compression in the frequency domain), Gaussian, and uniform noise (c 0.03). The order is from left to right, and from top to bottom.
23

---

Under review as a conference paper at ICLR 2020
Driginal label: Loader conficienc::
Adv. label pencil box, pepcil case
FC (30) label: Loafer confidence: 12.6384
25
50
50
75
40
40
172
20
20
202
150 200 200
200
402
The original image from the ImageNet dataset, its adversarial example, and the state of the image after recovery from the attack via the 30% compressed Fourier Channel (FC). The heat maps of magnitudes of Fourier coefficients are presented in logarithmic scale (dB) with linear interpolation and the max value is colored with white while the min value is colored with black. The black part of the (bottom-right) image represents the removed high-frequeng coefficients. The Fourier-ed representation plotted for single (0-th) channel. The lowest frequency coefficients are placed in the corners of the FFT maps (with the DC component in the top-left corner).
Figure 12:
90
Figure 13: For the CIFAR-10 dataset, we mul- tiple trials of the uniform noise channel and take the most frequent prediction. We further test multi- ple noise levels. The multiple trials improve over- all accuracy for different noise levels significantly. After 128 trials for the best setting we are within 3% of the overall model accuracy (of about 93.5%).
Accuracy
e=0.007
75
ea0.3
e=0.7
70
65
50
100
150
200
250
# Trials
100
many
Figure 14: For the CIFAR-I0 dataset, run multiple trials of the uniform noise channel and take the tion in the defense (many noise iterations). We also rum just single noise injection and return the predicated label. The attacker runs the same number of many uniform tri- als as the defender. The experiment is run on 100 images, with 100 C&W Le attack iterations.
0.03 single noise
0.03 many noise iterations
80
noise
many iterations
frequent predic-
Accuracy 1%)
60
40
20
00
22
2
25
2
2.00
212
2
of iterations
24

---

Under review as a conference paper at ICLR 2020
4
Exllxx[(x, Corg
3
gauss image
2
original image (baseline)
adv image (FGSM)
1
8.00
0.05
0.10
0.15
0.20
0.25
0.30
o (strength of the Gaussian noise)
Figure 15: The changes in the Le norm of the gradient of the loss w.r.t. the input image z for the correct class Corg as we add Gaussian noise to the original image. The experiment run on 1000 images from the ImageNet dataset.
gauss image org class gauss image adv class original image org class
15
Ex 2)2
10
adv image (C&W) adv class
5
0.000
0.005 0.010 0.015
0.020
0.025
0.030
a (strength of the Gaussian noise)
Figure 16: The changes in the L2 norm of the gradient for the correct class Corg and the adversarial class Cods as we add Gaussian noise to the adversarial image generated with C&W La attack (we label such image as gauss image xr). The experiment is run on 1000 images from the CIFAR-10 dataset.
25

---

Under review as a conference paper at ICLR 2020
15
gauss image org class gauss image adv class
(x c)l
10
gauss image random class
5
8.00
0.01 a (strength of the Gaussian noise)
0.02
0.03
0.04
0.05
Figure 17: The changes in the Le norm of the gradient of the loss for the correct class Corgs the adversarial class Cado. and random class Cran as we add Gaussian noise to the adversarial image generated with C& W L2 attack (we label such image gauss image 2)). The experiment is run on 1000 images from the CIFAR- dataset.
Highest eigenvalues Hessians wust inputs for $12 images from Imagehet
Image type:
Iracuency count
b0a
10 eigervalue
16
20
The spectrum of the Hessian with respect the original and adversarial inputs. We use the symmetric log scale on the axis and plot the min and 100 images from the ImageNet dataset trained the ResNet-50 architecture.
Figure 18:
50
sigeriallue magniade
1.00
150
200
250
Figure 19: The top eigenvalues of the Hessians ImageNet using ResNet-50.
ranges of the eigenvalues for
26

---

Under review as a conference paper at ICLR 2020
100
100
full spectra (FP
FP32-C=0%
FP16-C-0%
Test Test accuracy (%)
80
band-imitted (FP32-C=85%)
FP32-C=0%
early stopping
60
Test accuracy
60
FP32 C55%%
FP32-C-85%
20
20
8.0
0.2
0.4
0.6
0.8
1.0
8.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75 2.00
Level of uniform noise (sigma)
Level of Gaussian noise (sigma)
100
100
full spectra (FP32-C- C=0%)
Test accuracy (%)
Test accuracy (9))
band-lmited (FP32-C=85%)
60
60
R=2
R=64
R-128
R-8
R-256
R=16
full spectra
20
R=32
(FP (P32-C=0%)
20
imited (5532-C-85))
8.0
0.2
0.4
0.6
0.8
1.0
200
400
600
800
1000
Strength of contrast based attack (epsilon)
Number of perturbed pixels
multiple pixe! attack)
100
100
spectra
(FP32-C=0%)
Test accuracy (%)
80
band-imited (FP32-C=85%)
Test accuracy
60
60
full spectra (FP32-C=0%)
20
20
hand-limited (FP32CC-85%
8.0
0
10
15
20
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Number of perturbed pixels local search attack)
Angle of the rotation
degrees spatial
100
100
full spectra (FP32-C=0%)
full
(FF32-C=0%)
Test Test accuracy 19))
80
band-Imitted
(FP32-C-85%)
band-amited
(FP32-C-855)
Test accuracy
60
60
20
20
0
10
15
0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200 Strength the gradient sign attack (epsilon)
Limits of the herizontal and vertical translations (in pixels spatial attack)
Figure 20: Test accuracy function of the strenghts of the attacks for ResNet-18 CIFAR-10.
27


Reviews:
[{"review_id": "HyxyKWpnYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work is concerned with a very interesting question: \"robustness of the adversarial attacks\" and discusses the following: First, what are different ways of destabilizing a given adversarial attack (different perturbations) and how much each of them are effective. Secondly, what makes adversarial images particularly non-robust compared to natural images and is there empirical evidence for it? Thirdly, is it possible for an attacker to use this knowledge and use a rather universal model of perturbations to make its adversarial examples robust against such (deterministic or stochastic) perturbations?\n\nThe paper is quite well-justified and addresses a very interesting question by unifying the existing results of the literature in one place. First of all, comparing the accuracy-robustness trade-off of different methods are useful.  The idea of modeling perturbations as a general distorted communication channel to have a unification of these methods is very useful. The perturbation analyses along with the results in Figures 2, 3 are novel and can be useful for further theoretical investigation of the source of instability of adversarial images. To the best of my knowledge, the transferability of adaptive attacks against noisy channel defense methods has not been discussed in the literature. The paper is well-organized and well written (except for Section 5 both in sense of the writing and where it appears in the paper).\n\nA few questions and suggestions:\n\n* The discussion of results in Figure2 should be more extensive and more clear. \n\n* Why do you think the L2 of the gradient for adv images has such high variance for both orig and adv class? Is there a relationship between the magnitude and how effective is a perturbation-based defense?\n\n* It should be made clear in the text that methods like randomized smoothing are designed for the purpose of 'certified' accuracy and being robust against attack sizes more than the certification threshold was not part of the method's contributions.\n\n* The use of the term \"compression\" in Section 3 is not entirely correct as it is not \"technically\" correct to say that compression is what is necessarily happening in a general case of a deterministic C(x).\n\n* Fig 1 should be larger.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJg-Z7MpKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the robustness of adversarial attacks to transformations of their output. Specifically, for standard methods for crafting adversarial examples, the authors evaluate whether the crafted examples remain adversarial under (stochastic or deterministic) transformations such as Gaussian noise and SVD compression. The authors argue that different transformations have a similar impact on the perturbed inputs. Finally, they argue that the reason why these transformations can sometimes recover the correct label is due to the loss being more unstable at these points (from a first- and second-order pespective).\n\nFrom a conceptual point of view, I did not find the paper particularly impactful. In particular, I can identify three claimed contributions:\n\na) The L2 distortion introduced by these transformation might have more impact than the specific transformation used (figure 1).\nFirstly, I would argue that this effect is most prominent for small L2 distortions (distortions larger than 5/40 do exhibit noticeable differences). Secondly, I am not sure why one would expect these transformations to have fundamentally different impact. The adversarial attacks considered manipulate the input using first-order methods in input space. Since the transformations are model- and data-agnostic it seems expected that the primary mechanism behind their effect is the pixel-wise distortion of the image.\n\nb) Adversarial perturbations that are robust to one type of transformation tend to also be robust to other transformations (table 2).\nThis is probably the most interesting observation of the paper, indicating that attackers can bypass several of these transformation defenses by only aiming to be robust to a subset of them. At the same time, I am not sure what the impact of this observation is given that we already know how to bypass most of these defenses anyway.\n\nc) The instability of adversarial perturbations can be explained by a larger gradient norm and Hessian spectrum (figure 2, 3).\nFirst of all, I do not understand how this is considered a potential explanation of empirical behavior. If gradient norm is indicative of instability, then _natural_ images would be unstable (since the gradient wrt the wrong class is large). Furthermore, instability does not explain why adding noise leads to the _correct_ class as opposed to a random class. Perhaps most importantly, from what I understand (also skimming the code), Figure 2 plots the gradient of the _cross-entropy loss_ with respect to each class. However, the norm of the gradient is directly affected by the softmax probability of each class. Hence, for natural images, the probability of the correct class is high leading to a small norm while for the adversarial class it is low leading to large norm. Based on this reasoning, this plot does convey information about the classifier's stability but rather about the softmax probabilites assigned to each class.\n\nIn summary, after reading the paper, I am not sure how our understanding of transformation robustness has changed or what insight we have gained that will help us design future attacks and defenses (especially given prior work on how most of these defenses can be bypassed). I thus recommend rejection.\n\nComments to authors:\n-- Many of the attack details are missing (what is the epsilon allowed for PGD? what does the \"c\" parameter correspond to for CW attacks?) which prevented me from fully comprehending the experimental results. \n-- LBFGS is not a particular attack, it is a general optimization algorithm, https://en.wikipedia.org/wiki/Limited-memory_BFGS \n-- The fact that adversarial points are close to correctly classified points does not mean that a small amount of random noise will change the classifier prediction. In high dimension, the distance to a hyperplane can be epsilon but moving in a random direction requires movement of epsilon * sqrt(number of dimensions) .\n\n==============\n\nUPDATE: I appreciate the authors' response. As stated in my original review, I do recognize the performance of a non-adaptive adversary as a contribution. However, I still don't think that the impact of this finding is significant enough for publication at ICLR. I hence keep my original score.\n\nResponse to specific points:\n-- Unfortunately, the authors did not address my concerns about the gradient of the loss. The experiments and reasoning of the paper should be sufficient without citing other work in the author response. Moreover, it is unclear if the papers referenced in the response are showing fundamental properties of adversarial examples or are specific to the attacks considered.  For instance, there has been work challenging these papers (https://arxiv.org/abs/1907.12138).\n\n-- As mentioned in my original review, measuring the norm of the gradient of the softmax loss will inevitably take into account the predicted probability of each class. Hence in order to truly measure sensitivity it would be important to work with the logits instead of the softmax probabilities.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xDnB_aKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper studies the noise injection as defense methods against adversarial perturbations. It presents several experiments on the relationship between clean and robust accuracy. Conclusions of this study are (1-1) several defense methods have the same underlying mechanism (noise injection) and behave similarly against adversarial perturbations, (1-2) all of the defense methods can be attacked by the same black-box attack, and (1-3) the reason of the correct label recovery by noise injections might be the input instability on adversarial inputs.\n\nThis paper should be rejected because (2-1) some conclusions are not well-supported by the experiments, and (2-2) the paper fails to demonstrate the novelties and their importance of some conclusions.\n\nMajor comments:\n(3-1) Experiment 1 aims to highlight the similarity between the ``perturbations defenses''. However, the \"similarity\" is not defined, and the arguments are mostly subjective. For example, SVD and Gaussian noise appear to have different effectiveness (Figure 1). If the similarity means the observation that large distortions decrease the accuracy, it will not be surprising because all methods should achieve the chance rate accuracy when the distortions are extreme.\n(3-2) What is the point of Experiment 2? The phenomenon that strong defense methods decrease clean accuracy has been observed in many defense papers. The noise injection-based defense methods will not be exceptions. Remarking the existence of the trade-off will not be a strong contribution. The focus of the discussion should be on how to take a good balance or improve both of them.\n(3-3) To my best knowledge, experimentally showing that adversarial inputs have larger gradient norms (Section 4) is novel, and it is potentially interesting. I wonder how large the gradient norms will be for randomly perturbed images.\n\n===== UPDATE=====\nThank you for the response. After reading the response and other reviews, I mostly agree with Reviewer 2 and keep my score.\n\nThank you for the clarification on (3-1) in the initial review. I think it made the contribution of the paper clearer. However, as I and Reviewer 2 commented in their initial review, I did not find the observation significant.\nI also thank you for the additional experiments concerning (3-3). It addressed a concern that the larger gradient norms are not specific to adv. examples and appear everywhere around the natural images. However, concerns c) raised by Reviewer 2 seems critical. It is unclear whether the instability explains the observed phenomena and whether Experiment 3.3 is enough to confirm the existence of the instability. I think the paper is better to address the concerns before publication.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an analysis on different methods of noise injection in adversarial examples, using gaussian noise for example. There are important issues raised by reviewers 1 & 2 about some conclusions not being well supported by the experiments and the utility/importance of some conclusions. After a discussion among reviewers, as of now all 3 reviewers stand by the decision that substantial improvements, and analysis can be made in the paper. Thus, Im recommending a Rejection.
Decision: Reject