Dissecting Local Properties of Adversarial Examples
Authors: Lu Chen
Renjie Chen
Hang Guo
Yuan Luo
Quanshi Zhang
Yisen Wang
Venue: ICLR 2022 Submitted (2022)

Abstract: Adversarial examples have attracted significant attention over the years, yet a sufficient understanding is in lack, especially when analyzing their performances in combination with adversarial training. In this paper, we revisit some properties of adversarial examples from both frequency and spatial perspectives: 1) the special high-frequency components of adversarial examples tend to mislead naturally-trained models while have little impact on adversarially-trained ones, and 2) adversarial examples show disorderly perturbations on naturally-trained models and locally-consistent (image shape related) perturbations on adversarially-trained ones.  Motivated by these, we analyze the fragile tendency of models with the generated adversarial perturbations, and propose a connection with model vulnerability and local intermediate response. That is, a smaller local intermediate response comes along with better model adversarial robustness. To be specific, we demonstrate that: 1) DNNs are naturally fragile at least for large enough local response differences between adversarial/natural examples, 2) and smoother adversarially-trained models can alleviate local response differences with enhanced robustness. 

Under review as a conference paper at ICLR 2022
DISSECTING LOCAL PROPERTIES OF ADVERSARIAL EXAMPLES
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial examples have attracted significant attention over the years, yet suf- ficient understanding is in lack, especially when analyzing their performances combination with adversarial training. In this paper, revisit some properties of adversarial examples from both frequency and spatial perspectives: 1) the special high-frequency components of adversarial examples tend to mislead naturally- trained models while have little impact on adversarially-trainee ones, and 2) adversarial examples show disorderly perturbations naturally- -rrainne models and locally-consistent (image shape related) perturbations on adversarially-traie ones. Motivated by these, analyze the fragile tendency of models with the gen- erated adversarial perturbations, and propose a connection with model vulnerabil- ity and local intermediate response. That is, a smaller local intermediate response comes along with better model adversarial robustness. To be specific, we demon- strate that: 1) DNNs are naturally fragile at least for large enough local response differences between adversarial/natural examples, 2) and smoother adversarially- trained models can alleviate local response differences with enhanced robustness.
INTRODUCTION
Despite deep neural networks (DNNs) perform well in many fields (He et al. 2016 Devlin al. 2019), their
vulnerability attracts increasing attention. both for sufety-critice ap- plications (Sharif al., 2016)) and the black-box mechanism of DNNs (Fazlyab al.] 2019). DNNs have been found vulnerable to adversarial examples (Szegedy et al., 2014) Goodfellow et al. 2015), where small perturbations on the input can easily change the predictions of well-trained DNN with high confidence. In computer vision, adversarial examples exhibit their destructiveness both in the digital world and the physical world (Kurakin et al. 2017).
Since then, how to alleviate the vulnerability of DNN as narrow the performance gap between adversarial/natural examples is another key issue. Existing methods including defensive distilla- tion (Papernot al., (016)) and pixel denoising (Liao al.) 2018 have shown their limitations due to follow-up attack strategics (Carlini & Wagner (017)) or gradient masking (Athalye et al.) 2018). Amongst them, adversarial training (Goodfellow et al. 2015 Madry al.) (018)) and its variants (Zhang et al.] 2019 Wang et al. 2020b) indicate their reliable robustness and outperform. Moreover, as data augmentation method, adversarial training currently seems rely on additional data (Schmidt et al. 2018 Rebuffi et al.] 2021) to further improve robustness, while is sensitive to some basic model hyper-parameters, e.g., weight decay (Pang al. 202T). Apart from these, the effect of simply early stopping (Rice et al.
2020) even exceeds some promotion methods according
recent benchmarks (Croce Hein 2020 Chen & Gu 2020)). These studies arise our curiosity to further explore the relationship between adversarial examples and adversarial training, hoping to provide some new understanding.
Recalling that high-frequency components can be potentially linked to adversarial examples (Wang) ctt al., 2020a Yin et al. 2019: [arder et al., 2021), however, few explorations discuss the relation- ship between higu-eryy components and the destructiveness of adversarial examples. In this paper, we first demonstrate that feghffreyeency..c components of adversarial examples tend to mislead the standard DNNs. yet little impact the adversarially robust models. We further show that ad- versarial examples statistically have more high-frequency components than natural ones, indicating relatively drastic local changes among pixels of adversarial examples. Since adversarial examples

---

Under review as a conference paper at ICLR 2022
exhibit more semantically meaningful robust models (Tsipras et al. 2019), we further notice that adversarial examples show locally-come conass-consistent perturbations related to image shapes on adversarially- trained models, in contrast disorderly perturbations standard models. Both explorations on the frequency and spatial domain emphasize local properties of adversarial examples, and motivated by the local receptive field of the convolution kernels, we propose locally intermediate response perspective to rethink the vulnerability of DNNs. Different from the existing global activation per- spective (Bai et al., 2021 Xu et al. 2019). our local perspective reflects the joint effect of local features and the intermediate layers of the model. Based on the local perspective, we emphasize that large enough local response differences make it difficult for the network to treat an image and its potentially adversarial examples as one category, and demonstrate DNN models are naturally fragile at least attributed to it. Motivated by adversarially-trained models tend to have 'smooth' ker- nels (Wang et al., 2020a), we simply use the smoother kernels to alleviate local response differences on udversarially- trained models, which in turn affects the model robustness and reduces the robust overfitting (Rice et al.) 2020). To certain extent, this explains why weight decay effectively affects model robustness (Pang et al. 2021).
Our main contributions are summarized as follows:
We rirstle some properties of adversarial examples in the frequency and spatial domain: 1) the high- fretffrreueny components of adversarial examples tend to mislead naturally- trained DNNs, yet have little impact on weversarially-trainne models, and 2) adversarial examples have locally- consiscc perturbations on adversarially-trainee models, compared with dis- orderly local perturbations on naturally-trained models.
Then we introduce local response and emphasize its importance in the model adversarial robustness. That is, naturally-trained DNNs are often fragile, at least for non-ignorable lo- cal response differences through the same layer between potentially adversarial examples and natural ones. In contrast, adversarially trained models effectively alleviate the local re- sponse differences. And the smoother adversarially- trained models show better adversarial robustness as they can reduce local response differences.
Finally we empirically study local response with generated adversarial examples. We fur- ther show that, compared with failed attacks, adversarial examples (successful attacks) statistically show larger local response differences with natural examples. Moreover, com- pared with adversarial examples generated by the model itself, those transferred by other models show markedly smaller local response differences.
2 RELATED WORK
Understandings of model vulnerability. Since the discovery of adversarial examples (Szegedy
et al. 2014), number of understandings model vulnerability has been developed. For in-
stance, linear property of DNNs (Goodfellow et al. 2015), submanifold (Tanay & Griffin| 2016
and geometry of the manifold (Gilmer et al. 2018 were considered from the high-dimensional
perspective; the computing of Lipschitz constant (Szegedy et al.) 2014) Fazlyab et al.) 2019) and lower/upper bounds (Fawzi et al. 2018 Weng et al. 2018) were considered from the definition of model robustness; non-robust features (llyss et al. 2019), high-frequency components (Wang et al.) 2020a Yin et al.
2019) high-rank features (Jere et al. 2020) and high-order interactions among
2021 were explored adversarial examples from the different perspectives on im- ages, which imply y our local perspective; feature denosing (Xie et al. 2019), robust pruning (Madaan & Ju Hwang. 2020) and activation suppressing (Bai et al.) 2021 were focused on global intermedi- ate activations of models. On the other hand, taken adversarial training into consideration, [Tsipras) ct al. (2019), (Schmidt et al. (2018) and Zhang et al. (2019) explored the trade-off between robust- ness and accuracy; Wang et al. (2020a) found addversarially-tainne models tend to show smooth kernels: Tsipras et al. (2019) and Zhang & Zhu (2018) argued adversarially robust models learned more shape-biased representations.
pixels (Rnn et al.
Different from these studies, characterize the adversarial examples from both frequency and spa- tial domain to emphasize local properties of adversarial examples, and propose a locally intermediate response to rethink the vulnerability of DNNs.
2

---

Under review as a conference paper at ICLR 2022
Adversarial training. Adversarial training can be seen as min-max optimization problem:
min Σ max L(fo(z)). y:) <(B(())
where f denotes DNN model with parameters 6, and (a; ys) denotes pair of a natural example e and its ground-truth label y/- Given classification loss L, the inner maximization problem can be regarded as searching for suitable perturbations in boundary B to maximize loss, while the outer minimization problem is to optimize model parameters adversarial examples {2 generated from the inner maximization.
DIFFERENT PERTURBATIONS AFFECT THE VULNERABILITY OF THE MODEL
In this section, we first investigate adversarial examples in the frequency and spatial domain, and show the connections between models and their adversarial examples. Note that our threat model is a white-box model, and the fact that the adversarial example only defined by the misleading result under the legal threat model, our findings suggest that the properties of the sampled examples can broadly reflect the fragile tendency of the model.
Setup. We generate &xc. bounded adversarial examples by PGD-10 (maximum perturbation 8/255 and step size 2/255) with random start for the robust model (Madry et al. 2018). Specifically, use ResNet-18 (He et al. 2016) the backbone train the standard and ad cersarially-trainee models for 100 epochs on CIFAR-10 (Krizhevsky) 2009). Following (Pang et al.] (021). we use the SGD optimizer with momentum 0.9, weight decay 5 x 10-4 and initial learning rate 0.1, with a three-stage learning rate divided by 10 at 50 and 75 epoch respectively. The robustness of both models is evaluated by PGD-20 (step size 1/255).
3.1
THE DESTRUCTIVENESS OF HIGH-FREQUENCY COMPONENTS FROM ADVERSARIES
(a) filtered images (STD) (b) filtered images (ADV) (c) merged images (STD) (d) merged images (ADV)
Figure I: The destructiveness of frequfrry components from natural and adversarial examples on both standard (STD) and adversarially-traiee (ADV) models. Shown above are vell-trained models tested with (a)-(b) images through low-pass filter and (c)-(d) frequency-swapped images.
Inspired by (Wang et al. 2020a), we are naturally curious whether the adversarial examples cause considerable damage to the model mainly because of their high-frequency components. To answer this question, Figure Iillsttraess the trend of model performance and robustness on the test set with the high-frequency components increased (the increase of the filtering scale denotes that more high- frequency components are added to the filtered images). Figure ((a) shows that, for standard models, the high-frequency components of natural examples promote classification and reach the model per- formance (green line); on the contrary, the performance of the filtered adversarial examples first rises to get the highest accuracy 47.5%, and then drops rapidly to reach 0.0% (red line). Obviously, in the low-frequency range, the performance of natural and adversarial examples are quite close, yeí more high-frequency components widen the difference. That is, the special high-frequency components caused by adversaria perturbations exhibit clear destructive effeett standard models, and simply filter out them can effectively alleviate the destructiveness of adversaries even on standard models.
However, for robust models, we show that the prediction performance finally reaches robustness without a rapid drop in Figure [(b) But surprisingly, we find that the performance of filtered adver- sarial examples in some range exceeds the final robustness 47.5% (red line), reaching maximum of 51.2%. That is, although these high-frequened components do not exhibit a clear destructive effect, simply filtering out them has a positive impact alleviating robust overfitting (Rice et al.) 2020).
3

---

Under review as a conference paper at ICLR 2022
We then swap their Sugh-frequency components between both examples controlled by frequency threshold Figure [[]] I(d) For merged natural examples with high-frequency components from adversaries, the increase of the frequency threshold controls the accuracy increasing from the model robustness (red line) to the model performance (green line), the opposite occurs on merged adversar- ial examples. These clearly illustrate the boost effect of the high-frequency components from natural examples and the destructive effect of the
components from adversarial examples.
16
45
ori. images
(b) adv (STD)
(c) diff (STD)
(d) adv (ADV)
(c) diff (ADV)
Figure 2: The average logarithmic amplitude spectrum of (a) 1000 three-channel images and (b) their adversarial examples generated by the standard (STD) model, where the corners represent high-free tigh-freq range, and the colorbars represent the logarithmic amplitude log(1 [) (the redder the larger). And (c) denotes the difference between (b) and (a), log([adul) - log( |nat)) = log( [ad/l/|[aaa the write color of (c) represents equivalent, and the red represents |adul > |nat]. (d) and (e) are on the adversarially-trained (ADV) model, and (e) denotes the difference between (d) and (a).
where
To further illustrate, we find that statistically, the main difference the frequency domain between natural and adversarial examples is concentrated in the high-frequency region in Figure 2 We visu- alize the logarithmic amplitude spectrum of both examples. Figure 2(a) and [ show that, compare with natural examples', the high-frequency components of the adversaries emphasizes that adversarial examples markedly show more high- sigh-frequency components, indicating relatively drastic local changes among pixels. This statistical difference explains the high detection rate of using Magnitude Fourier Spectrum (Harder et al. 2021) to detect adversarial ex- amples. Furthermore, Figure 2(d) and [(c) show that the high-frequency components of adversarial examples generated by robust models are less than those from standard models, yet still more than natural examples'. Besides, the analysis of filtering out low-frequency components in Figure 6 (Appendix A) also emphasizes our statement. That is, compared with natural examples', the spe- cial figuefre components of adversarial examples show their serious misleading effects on standard models, yet are not enough to be fully responsible for the destructiveness to robust models.
hard to ignore. Figure
3.2 DIFFERENT PERTURBATION AND THE FRAGILE TENDENCY OF THE MODEL
(a)
(b)
(c)
(4)
(0)
(
(1)
(k)
(1)
(m)
Figure 3: Visualisation of adversarial perturbations (PGD) generated by Left: adversarially-traine
(ADV) model and Right: standard (STD) model. The columns from left to right are represented
by (a)-(m) respectively: (a) natural example; (b) adversarial example and (c)-(g) its perturbations (overall, average of channels, and three channels) generated by ADV model; (h) adversarial example and (i)-(m) its perturbations generated by STD model. Shown above are all successfully attacked, the first row is attacked as a bird from a dog, the second a cat from frog, the third a car from a ship.
To answer the different effects of highffreueecc components, we directly explore the adversarial examples themselves, e.g., in the spatial domain. We first get well-traines models above and visualize adversarial perturbations in Figure including the overall perturbations scaled to [0,255], the average perturbations of channels, and perturbations of three channels.
4

---

Under review as a conference paper at ICLR 2022
As shown in Figure Beguree[Rige Right,, the perturbations generated by the standard models tend to be locally inconsistent and disordered. This observation is different from Figure Left) that adversarial ex- amples show locally- coossst-c perturbations related to image shapes on the adversarially-tre trained models, that is, perturbations tend to be locally co-increasing or co-decreasing on each channel. In more detail, the perturbation of each pixel on a single channel tend reach the perturbation bound, i.c., +8/255 (the reddest), -8/255 (the bluest), which is counter-intuitive under the iterative attack of PGD-20 and naturally associated with the one-step attack FGSM in Figure 7 (Appendix B). In fact, both attacks show similar perturbations and similar model robustnes while the former produces more detailed perturbations. Besides, compared with failed attacks in Figure [8] (Appendix B), ad- versarial examples of successful attacks in Figure 3 show more misleading local perturbations, e-g.. the perturbations in the first row are more like a bird (bird wings are added), and the third like a car.
Perturbations in frequency vs. spatial domain. Similar to the different destructive effects of high-frequency components, the perturbations in the spatial domain exhibit a more intuitive differ- ence related to the models. Since more high-frequency components indicate images have relatively drastic local changes, we count that adversarial examples generated by adversarially-train show fewer high-frequency components mainly due their cocally-consistent perturbations. These perturbations with smooth local changes imply that simply filtering out sigh-ffreuency components has little promotion on robust models, while locally-disordered perturbations imply the effectiveness of filtering out high-frequency components on standard models.
models
Perturbations vs. fragile trend of models. For given model, optimization-baset attacks attempt to search for special perturbations to maximize the classification loss of adversarial examples. We note that for adversarially-traine models with smooth kernels, the perturbations that tend max- imize loss exhibit a locally-consistedr tendency, and for standard models with non-smooth kernels, the perturbations tend be locally disoodeddd maximize loss. This implies a potential connection between the fragile trend of models and their potentially adversarial examples.
4 LOCAL RESPONSES: FURTHER IMPACT OF PERTURBATIONS ON MODELS
In this section, motivated by local properties of adversarial examples and the local receptive field of convolution kernels, we first introduce a locally intermediate response perspective to rethink the vulnerability of the model, and then empirically show the relationship between local responses and the vulnerability of the model.
Different from the existing idea of searching for adversarial examples, we consider under what circumstances the potential examples, refer to all legal variants in boundary B regardless of whether destructive or not, would show their destructive effects on well-trained model, that is, the reason why some macro-similar potential examples present prediction results far away from their natural examples. Inspired by local properties of adversarial examples and kernels, we naturally consider whether the destructiveness of potential examples can be viewed from local responses, which reflects the combined effect of the local features and the model property.
Note that under ideal cireumstances, given a well-trained model, if the local responses of any ex- amples on a certain layer are completely consistent, then the final predictions of these examples exactly the same. To relax the condition in reality, we hypothesize Assumption 1): If macro-similar features through the same layer exhibit a sufficiently small differ- responses responses, with sufficiently small differences in the subsequent layers, then the final responses of the network are relatively close; otherwise, the large enough local differences make the network difficult to treat them as the same category. To illustrate this, we take the convolutional layer as an example.
follows (referred to as
ence in local
4.1 LOCAL RESPONSES
Macroscopically, we denote a DNN model f for classification, the (-th layer of activation feature maps f", and f to represent the input features. Macro-similar image and its potential example pass through the I-th layer to get fl and fl respectively. We also denote the (1+1) weight com-
'The adversarially-trin model (the hest checkpoint) reaches 51.9% robustness on the PGD-20 attack and 57.4% robustness on the FGSM attack.
5

---

Under review as a conference paper at ICLR 2022
ponent M I #xx**, where H, W, K represent height, weight and number of kernels spectively, From the locally intermediate response perspective, we first get one of the convolution kernels m R and then capture the (-th layer of local features centered at (i.j) position x and o4, corresponding to its local receptive field. Formally, for the difference local responses,
+ = 2ig @ m 1+1 =a, @ m [+ = xdd @ m +1
1++ where / denotes the (i.j)-h response difference of local features through the same kernel ml+1, and vd == oi, - denotes the difference of local feature maps between its potential example and natural example. The second equation is based on the linear property of the convolution operation.
Ideally, if the differences are equal zeros at any positions in the (1+1)-th layer, that is, the intermediate layer has the same utility for both examples, then the potential example after the ((+1) layer shows exactly the same responses as the natural example's. However, in most cases the local differences difficult to be exactly zeros everywhere, then based Assumption 1, our aim is to make the absolute difference of local responses | as small as possible to approach the model's cognition of potential example and natural example.
The difference local responses
A composed of xa and m²+1 can be further expressed as:
w Σ
H
xd 2 m 1+
m
141
where ctty and m represent the (i.))thh elements of xx and ml+1 respectively, thus the absolute is affected jointly by both local features and kernel parameters.
local difference | 1+1
b.j
Note that what we care about is under what cireumstances are more likely to produce large enough response differences. Though the specific impact of d, and m. we consider statistical circumstances as numerous convolution kernels and various local features are combined affect the response differences real networks. Statistically, the larger amplitude of ml with fixed x'a, or larger amplitude of xz' with fixed m²+1, tends produce larger | Besides, if a kernel +11 is relatively non-smooth, then non-smooth local features æd rather than likely to produce large enough local response differences.
A is quite complicated,
smooth local features,
4.2 FURTHER IMPACT OF LOCAL PERTURBATIONS ON MODELS
Recalling that local properties of advensarial examples in Section 3, we further explore how different local perturbations affect the models from the locally intermediate response perspective. To illustrate the effect of perturbations, we assume two convolution kernels with the same mean but different variances, one disordered, and the other smoother.
79055
8255
STD
Locally derodd
Disundeool kerne
Lo
0.11.255
itrage
8058
ADV
Locally consiteet
Smooth komel
Adversarial examples
Local perturbations
Kernels
Local response differences
Figure 4: Further impact of local perturbations on different convolutional kernels. Shown above is adversarial perturbations with same amplitude (single channel) act kernels with different smooth- ness and then get the local response differences of adversarial example and natural example.

---

Under review as a conference paper at ICLR 2022
Local responses vs. locally-disordered perturbations. Since adversarial perturbations generated by standard models tend to be disollyddiooreere,, we discuss these perturbations affect the local responses. Locally-disoodered perturbations indicate relatively drastic local changes among pixels of potential example, in other words, local input difference x.a" of potential example and natural ex- ample (also refers local perturbations) show its great variance. These perturbations on numerous kernels. As shown in Figure suppose convolution kernels with the same mean but perturbations are more likely to yield large enough abso-
convolved
greater variance, these rreally-disorders lute response differences | in the first layer, since the non-smooth kernels tend to emphasize the relationship between different pixels. On the other hand, suppose smoother kernels with rela- tively small variance, these -esldeed perturbations tend to show smaller absolute response differences since smoother kernels tend to focus on the average situation within their local receptive fields. That is, locally-disodded tend to produce different response differences, which can be accumulated in the subsequent layers.
perturbations pass through different types of convolution kernels
Local responses vs. locally-consisten perturbations. Since adversarial perturbations generated by adversarially robust models tend to be locally-consistent, we count that such perturbations are more destructive to smoother kernels, comparing with locally-disorders perturbations in Figure 4 coecssy-consistent perturbations xd reaching the perturbation bound tend to produce larger absolute response differences in the first layer since they tend show larger absolute average, yet locally- disordered perturbations show smaller response differences due to a smaller absolute average.
Both different local perturbations have different impacts on the local response differences of poten- tial examples and natural examples, implying the fragile trend of models and the transferability of adversarial examples.
4.3 EMPIRICAL UNDERSTANDING OF LOCAL RESPONSES
Motivated by deversarialll-trainee models that tend to show smoother kernels (Wang et al. 2020a), we first provide a further understanding of local responses and then give empirical understandings.
Local responses vs. model smoothness. Given a non-smooth convolution kernel m/+1 with great variance, it is more likely to cause huge impact on an absolute response difference (
4+1 within its local receptive field when fixing zz'. Considering numerous non-smooth kernels and various local features are combined in the same layer, is quite easy to yield large enough response differences. More complicated is the subsequent layers act on the current local response differences and get in- tricate accumulation effects. That is, for a standard model with non-smooth kernels, the model itself tends to amplify the local response differences between potential examples and natural examples. On the other hand, since an adversarially trained model show smoother kernels, the model has a ten- dency to weaken the local response differences and narrow the final responses of potential examples and natural examples, indicating a trade-off between model robustness and accuracy.
Setup. We get both standard and adversarially- traiied models in Section 3. Considering complex functional layers, including linear and nonlinear ones in DNNs, based on Assumption 1, we use the maximum and the total absolute differences of local responses in some layers to show their effects. Taking ResNet-18 18 as an example, we investigate some layers including the first convolutional layer as Conv 1, the feature maps before layer I as Layer and Layer I to Layer 4 respectivelyP
To verify above analysis, we first compare the local response differences of potential examples and natural examples between standard and robust models. As Table shows, the standard model ex- hibits significantly larger differences in local responses layer by layer, especially from the perspec- tive of total differences yielded by per pairs. These large enough response differences accumulate, making the model's 's cognition of potential examples far away from their natural ones and ultimately leading the model to express a high vulnerability. On the other hand, the adverrsaiillly-triin significantly shortens local response differences in corresponding layers, yet still some differences exist. In other words, how to further reduce the local response differences is a perspective of ap- proaching model robustness to performance. We also note that, due to the complex effect of nonlin- ear layers (c.g.. BatchNorm, ReLu), the maximum absolute difference of local responses does not strictly increase monotonically but increases in trend.
model
Feature map size of Conv 1, Layer O and Layer 1: 64 x 32 x 32, Layer 2: 128 x 16 16, Layer 3: 256 x 8 x 8, and Layer 4: 512 4
7

---

Under review as a conference paper at ICLR 2022
Table 1: The absolute difference of local responses (per image) between test set images and their
potentially adversarial examples on different layers. Left/Right denote standard/robust model.
Model
Conv
Layer 0
Layer 1
Layer
Layer
Layer 4
Max 0.12/0.05
0.30/0.15
1.92/0.52 4276/1246
1.46/0.36 2093/430
1.45/0.41
3.72/1.02
Total
370/556
443/489
581/164
2624/526
Local responses vs. destructive effect. Naturally, we wonder whether a clear difference between potential examples of successful and failed attacks on certain model exists. We first select the natural examples that are classified correctly, and count their potential examples of successful and failed attacks. As Table shows for the standard model, the successful ones show similar differ- ences with the failed ones in the front layers, but greater differences especially in Layer 4 close to the outputs, leading to finally destructive effects. However, for the adversarially- -rrinndd model in Table 4 (Appendix C), similar differences even in Layer 4 may be due to the smoother kernels and locally smoother perturbations, leading to the final responses of both close to natural examples'.
Table 2: Left/Right denote successfuully-attackooo//
adversarial examples on standard model.
Model
Cony 1
Layer 0
Layer
Layer 2
Layer 3
Layer 4
Max 0.16/0.15 Total
0.379/0.378 676/669
2.50/2.51
1.38/1.37
1.16/1.08 522/494
2.11/1.38 1481/757
558/554
5955/6001
2363/2339
Table 3: Left/Right denote original/transferred adversarial examples on adversarially-trraiee model.
Model
Conv
Layer O
Layer
Layer 2
Layer 3
Layer 4
Max
0.05/0.04
0.15/0.12
0.52/0.44
0.36/0.14 430/138
0.41/0.08
1.02/0.10
Total
556/321
489/302
1246/767
164/38
526/57
Local responses vs. transferability. Different local perturbations related to models imply the diffi- culty of adversarial examples' transferability, then we further explore whether the transferability can be understood from the locally intermediate response perspective. To verify the analysis of Section 4.2, we exchange the adversaries obtained from the standard and robust model. Table shows, locally-disordered perturbations, combined with a smoother ad iiversarially-trannee fairly small local response differences layer by layer, making the model's cognition close enough to the natural examples and leading to 83.6% robustness close enough to 84.9% model performance. Similar situations occur when locally-consistent perturbations combined with a non-smooth stan- dard model in Table (Appendix C). These results indicate that the searched perturbations related to models tend to amplify response differences as much as possible to enlarge the model's 's cognition of potential examples and natural examples, and the weak transferability of adversarial examples may be due to transferred perturbations that are difficult to enlarge the model's cogmition.
model. exhibit
4.4 SMOOTHER KERNELS: ALLEVIATE LOCAL RESPONSE DIFFERENCES
To further exhibit the effect of shortening local response differences, we simply show smoother ad- versarially robust models can alleviate local response differences and then improve their robustness. As shown in Figure [5] udversarially-tren -tranned models with different smoothness (i.c., larger weight decay parameters show the smaller magnitude of the kernels (Loshchilov & Hutter 2019)) show different local response differences. Figures ((b)) for the maximum differences and Figure 9 for the total differences (Appendix C) indicate that smoother kernels slightly weaken the local response differences between the potential examples and natural examples layer by layer, and finally tend to narrow the robustness and performance in Figure 5(h) This some extent explains adversarially- trained models are more sensitive to weight decay (Pang et al. 2021). On the other hand, we find that the increase of weight decay quite difficult to further reduce the magnitude of parameters and the local response differences in each layer, which may be one of the reasons for the current bottleneck in the robustness of the adversarial training. Besides, the increase weight decay can effectively weaken the robust overfitting (Rice et al. 2020) in Figure 5(h and Figure (gg)
Note that under the PGD-20 attack, the standard model hardly yields examples of failed attacks, then the FGSM attack illustrate the problem.
8

---

Under review as a conference paper at ICLR 2022
Moggen
2004
75
26
25
100
100
20
100
(a) Parameters
(b) Conv 1
(c) Layer 0
(d) Layer
Mecoin fift
Mecraer DISLCWO
1..
5
tpedk
(e) Layer 2
(1) Layer 3
gg) Layer 4
(h) Test Accuracy
Figure 5: The maximum local response differences on robust models with different smoothness. (a) shows different model smoothness affected by weight decay, (b)-(g) denote the impact of weight decay on the local response differences, and (h) shows the model robustness and performance.
4.5 DISCUSSION: LOCAL RESPONSES AND MODEL ROBUSTNESS
The above suggests that the model robustness is related to the model itself and the property of potential examples, while they are combined through the local responses. Due to enough differences local responses, some potential examples are not regarded as similar to natural examples (or their predictions are different from the natural ones with correct predictions), so they eventually show their destructive effects as adversarial examples. That is, if a model tends to weaken the local response differences of potential and natural examples, then the model exhibits great robustness as the final responses of the potential examples are more likely to be close to the natural ones.
On the other hand, though small differences of local responses make the network more inclined to treat both examples as the same categories, but the existing differences, especially the intricate dif- ferences after multi-layer accumulation emphasize the difficulty of upproaching model robustness to performance, and demonstrate that DNN models are naturally fragile. Essentially, these response differences come from whether a non-zero convolution kernel acts on all legal perturbations in boundary B to obtain differences that are all zeros or sufficiently small without further accumula- tion. In other words, it tells the model robustness is that, given legal potential examples from any attacks, the accumulated response differences can be alleviated or removed. For instance, feature denosing (Xie al. 2019) and activation suppressing (Bai al. 202T) can be viewed as shortening local response differences of both examples to improve the model robustness.
Besides. shortening local response differences does not directly mean an improvement of the model's robustness, in fact, it expresses the closeness of the model's cognition on both examples. That is, a prorly-trained model may also treat both as relatively close, but give a bad model performance. To further improve the model robustness, a more realistic idea is to find proper parameters (whether theoretical parameters to minimize the local response differences exist) or a new method (nonlinear layers, loss functions, model structures) to shorten the response differences much possible, while not overly weaken the classification performance of the model.
5 CONCLUSION
In this paper, we investigate the local properties of adversarial examples generated by different mod- els and rethink the vulnerability of DNNs from a novel locally intermediate response perspective. We find that the tighffrequency components of adversarial examples tend to mislead standard DNNs, but have little impact on diverssrialllytrainee models. Furthermore, docally-disordered perturbations are shown on standard models, but locally-consistent perturbations on adversarially-trained models. Both explorations emphasize the local perspective and the potential relationship between models and adversarial examples, then we explore how different local perturbations affect the models. We demonstrate DNN models are naturally fragile at least for large enough local response differences between potentially adversarial examples and natural examples, and empirically show smoother adversarialllytrriiee models can alleviate local response differences improve robustness.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In ICML. 2018.
Yang Bai, Yuyuan Zeng, Yong Jiang, Shu-Tao Xia, Xingjun Ma, and Yisen Wang. Improving adversarial robustness via channel-wise activation suppressing. In ICLR. 2021.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE
Symposium on Security and Privacy (SP). 2017.
Jinghui Chen and Quanquan Gu. Rays: A ray searching method for hard-label adversarial attack. In KDD. 2020.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICMI. 2020.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL. 2019.
Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Analysis of classifiers' robustness to adversarial perturbations. In Machine Learning. 2018
Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George J. Pappas. Effi- cient and accurate estimation of lipschitz constants for deep neural networks. In NeurIPS. 2019.
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S. Schoenholz, Maithra Raghu, Martin Watten- 02774. 2018.
berg, and Ian Goodfellow. Adversarial spheres. In arXiv preprint
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR. 2015.
Paula Harder, Franz-Josef Pfreundt, Margret Keuper, and Janis Keuper. Spectraldefense: Detecting adversarial attacks on cnns in the fourier domain. arXiv preprint arXiv:2103.03000 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR. 2016.
Andrew Nyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In NeurIPS. 2019.
Malhar Jere, Maghav Kumar, and Farinaz Koushanfar. A singular value perspective on model ro- bustness. In arXiv preprint arXiv:2012.035/1 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In ICLR. 2017.
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang. Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. CVPR. 2018
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In ICLR. 2019.
Divyam Madaan and Sung Ju Hwang. Adversarial neural pruning with latent vulnerability suppres- sion. In ICML. 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ICLR. 2018.
Tianyu Pang. Xiao Yang. Yinpeng Dong, Hang Su, and Jun Zhu. Bag tricks for adversarial training. In ICLR. 2021.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In IEEE Symposium on Security and Privacy (SP). 2016.
10

---

Under review as a conference paper at ICLR 2022
Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A. Calian, Florian Stimberg, Olivia Wiles, and Tim- othy Mann. Fixing data augmentation to improve adversarial robustness. In arxiv preprint arXiv:2103.01946. 2021.
Jie Ren, Die Zhang, Yisen Wang, Lu Chen, Zhanpeng Zhou, Yiting Chen, Xu Cheng, Xin Wang, Meng Zhou, Jie Shi, and Quanshi Zhang. Towards a unified game-theoretic view of adversarial perturbations and robustness. In arXiv preprint arXiv:2103.07364 2021.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In ICML. 2020.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Ad- versarially robust generalization requires more data. In NeurIPS. 2018.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. Accessorize to crime: Real and stealthy attacks on state-of-the-art face recognition. In SIGSAC. 2016.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR. 2014.
Thomas Tanay and Lewis Griffin. A boundary tilting persepective the phenomenon of adversarial examples. In arXiv preprint ar0iv 07690. 2016.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be odds with accuracy. In ICLR. 2019.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric Xing. High-frequency component helps explain the generalization of convolutional neural networks. In CVPR. 2020a.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR. 2020b.
Tsui- Wei Weng, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, and Luca Daniel. Evaluating the robustness of neural networks: An extreme value theory approach. In ICLR. 2018
Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In CVPR. 2019.
Kaidi Xu, Sijia Liu, Gaoyuan Zhang, Mengshu Sun, Pu Zhao, Quanfu Fan, Chuang Gan, and Xue Lin. Interpreting adversarial examples by activation promotion and suppression. In arXiv preprint arXiv:1904.02055 2019.
Dong Yin, Raphael Gontijo Lopes, Jonathon Shiens, Ekin D. Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. In NeurIPS. 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICMI. 2019.
Tianyuan Zhang and Zhanxing Zhu. Interpreting adversarially trained convolutional neural net- works. In ICML. 2018

---

Under review as a conference paper at ICLR 2022
A
THE DESTRUCTIVENESS OF ADVERSARIAL EXAMPLES ON FREQUENCY DOMAIN
02
10
02
10
(a) low-frequency components (STD)
(h) low-frequency components (ADV)
Figure 6: The destructiveness of only high-frequeney components from natural and adversarial amples on both standard (STD) and adversarially-traieed (ADV) models. Shown above are well- trained models tested with images through high-pass filter.
components to the destructive- ness of both examples. We get both standard and adversarially-trained models as above. Figure 6 illlstrrates the trend of model performance and robustness on the test set with the low-frequency components decreased (the increase of the filtering scale denotes that the less low-frequency com- ponents are added to the filtered images). As the increase of filtering scale, only natural examples on standard models keep certain classification performance and then decrease to reach 10% (the accu- racy of random classification). That is, the high-freque frequeeyyy components of natural examples to some extent can promote classification. On the other hand, the high-frequency components of adversarial examples on standard models show almost no promotion but destructive effect, and finally reach 10% as well. For robust models, the high-frequency components of both examples show similar but little performance.
Here, we further investigate the contribution of only
B THE LOCAL PROPERTIES OF ADVERSARIAL EXAMPLES ON SPATIAL DOMAIN
We further explore the local properties of adversarial examples generated by the FGSM attack. For adversarially-trained models, compared with the PGD attack Figure the adversarial examples generated by the FGSM attack in Figure 7 show similar perturbations and similar model robust- ness (51.9% robustness for PGD-20 and 57.4% robustness for FGSM), yet the latter produces less detailed perburbations. For standard models, though both attacks show locally-disorden tions, the latter exhibits less disordered perturbations since less perturbation values can be achieved, i.e., +8/255, 0, and -8/255.
perturba-
(b)
(0)
(0)
(h)
(
(k)
(1)
(m)
Figure 7: Visualisation of adversarial perturbations (FGSM) generated by Left: adversarially-trained (ADV) model and Right: standard (STD) model.
12

---

Under review as a conference paper at ICLR 2022
Similar to adversarial examples of successful attacks in Figure 3 the failed attacks in Figure show locally-consisten perturbations related to image shapes on the adversarially- trained models as well. The difference is that, these examples show less misleading local perturbations since the shape of their perturbations is closer to the shape of the original image.
(b)
(0)
(e)
(h)
(k)
(m)
Figure 8: Visualisation adversarial perturbations (PGD) generated by Left: adversarially-traine (ADV) model and Right: standard (STD) model. Shown above all unsuccessfully attacked on
ADV model.
C LOCAL RESPONSES
As mentioned in Section 4.3, we report destructive effects on the adversarially-trinn trained model in Table 4 We first select the natural examples that classified correctly, and count their potential examples of successful and failed attacks. Similar to the standard model, the successful ones show similar differences with the failed ones in the front layers, but greater differences in Layer 4 close to the outputs. The difference that, due to the smoother kernels and locally smoother perturbations, the robust model exhibits closer local response differences Layer between the successful ones and the failed ones. Besides, compared with the PGD attack, the adversarial examples generated by the FGSM attack show similar differences in the front layers, but less local response differences in Layer 4, leading to higher model robustness.
Table 4: The absolute difference of local responses (per image) between test set images and their potentially adversarial examples on different layers. Left/Right denote attesssfllly-tttaccc//llllddd adversarial examples on dddversarially-triine model respectively.
Model
Conv
Layer
Layer
Layer 2
Layer 3
Layer 4
Max (PGD-20)
0.048/0.047
0.15/0.15
0.51/0.53
0.36/0.37
0.40/0.42
1.10/0.99
Total (PGD-20)
557/555
488/490
1264/1242
445/420
169/160
578/491
Max (FGSM)
0.048/0.047
0.16/0.15
0.53/0.54
0.35/0.36
0.37/0.38
0.90/0.86
Total (FGSM)
580/576
510/511
1292/1272
436/414
161/153
478/428
We further report the transferability of adversarial examples the standard model, as shown in 55 We get the potentially adversarial examples obtained from the robust model to attack the standard model. That is, locally-consistent perturbations, combined with non-smooth standard model, exhibit small local response differences layer by layer, making the model's 's cognition close enough to the natural examples and leading 78.4% robustness close 94.4% model performance.
Table The absolute difference of local responses (per image) between test set images and their po- tentially adversarial versions different layers of ResNet-18. Left/Right denote original/transfereed adversarial examples on standard model respectively.
Model
Conv
Layer 0
Layer 1
Layer
Layer 3
Layer 4
Max Total
0.12/0.15 370/532
0.30/0.29 443/509
1.92/0.85
1.46/0.88
1.45/0.67
3.72/0.88
4276/2639
2093/1 157
581/280
2624/525
13

---

Under review as a conference paper at ICLR 2022
As mentioned in Section 4.4, Figure 9 is supplement to Figure from the total absolute local response differences perspective, which indicates that smaller local response differences tend to approach model robustness to performance as well.
4000
2000
2000
2000
1000
100
Conv
(b) Layer
(c) Layer
(d) Layer
Teal DiBucaae
45
18s
116
Spacch
Epeccs
(c) Layer
(f) Layer
(g) Robust Loss
Figure 9: The total absolute local response differences (per image) of some layers on adversarially- trained models with different smoothness. Among them, (a)-(f) denote the influence of different weight decay parameters on the local response differences in each layer, and (g) shows the robust loss of training and test set.
14


Reviews:
[{"review_id": "lwt_BFvpy-S", "reviewer": "Reviewer_Rh4T", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ This paper studies an interesting problem: the connection between adversarial perturbations and local properties of adversarial examples (how perturbations associate to image shape).\n\nWeakness:\n- The observation and evaluation of this paper lack novelty and are not so surprising to me. For example, adversarial examples containing high-frequency perturbations is also studied in [1]. Adv-trained models are more robust than naturally trained ones, so they are supposed to be more robust to these high-frequency components in adversarial perturbations.\n\n- The explanation of the relationship between frequency and spatial domain is not quite clear to me. Why do high-frequency components indicate images have relatively drastic local changes? How are the high-frequency components related to local changes in adversarial examples?\n\n- The paper argues that the proposed smoother adversarially-trained models can achieve better robustness, but there is no comparison to the SOTA adversarial training work to show the improvement in the robustness. A quick comparison could be made with some works listed in https://robustbench.github.io/.\n\n- The terminology, like \"local changes\" \"locally consistent\", has a high frequency in this paper, but I didn't find the definition or the explanation for what 'local' means. This makes the paper a bit vague to understand. Besides, some other words like \"fragile trend\" are also vague.\n\n[1] Yin, Dong, et al. \"A fourier perspective on model robustness in computer vision.\" arXiv preprint arXiv:1906.08988 (2019).", "summary_of_the_paper": "This paper studies the properties of adversarial examples from a spatial and frequency perspective and shows that naturally trained models are more vulnerable to high-frequency components in adversarial examples. Perturbations for naturally trained models are disordered, but perturbations for adv-trained models are image shape related. Based on these observations, the authors find that smoother adversarially-trained can achieve better robustness. ", "summary_of_the_review": "The results presented by the paper are not so surprising and lack some novelty. The evaluation also needs additional comparison experiments to better support the conclusion that smoother adversarially-trained models enhance robustness.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "S3O_Qgzaklb", "reviewer": "Reviewer_CGeY", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The empirical findings of the paper seem to have already been known broadly. In Section 3.1, the paper shows the difference between vulnerabilities of standard training and adversarial training in the frequency domain. However, similar analyses have already been shown in (Yin et al., 2019, Wang et al., 2020a, Tsuzuku and Sato 2019). Similarly, the difference of vulnerabilities in the spatial domain has been shown in (Tsipras et al. 2019). Thus, the contributions in Section 3 are weak.\n\nSection 4 discusses the relation between local responses and locally-disordered perturbations, locally consistent perturbation, and smoothness. However, since several terms are not well defined mathematically, the claims in this section might be ambiguous and it is difficult to obtain useful insights. For example, what is the rigorous difference between locally-disordered perturbations are locally-consistent perturbations? In addition, I think the experimental setup is not very sophisticated to answer the research question.\n\nTo obtain more useful insights, it might be helpful to narrow down the research questions and experimental setup and to develop new analyses based on the previous studies.\n\n(Tsuzuku and Sato 2019) \"On the structural sensitivity of deep convolutional networks to the directions of Fourier basis functions.\" CVPR 2019.", "summary_of_the_paper": "This paper investigates the properties of adversarial examples from frequency and spatial perspectives and claims that standard models are vulnerable to high-frequency perturbations and adversarially robust models are vulnerable to locally consistent perturbations. In addition, this paper investigates the relation between local intermediate response and adversarial robustness.", "summary_of_the_review": "- The experimental results and claims are not novel while this paper does not provide theoretical results and new algorithms. .\n- The paper's assumptions are not well defined and its claims and limitations are not clear.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ATSKbVn_yDX", "reviewer": "Reviewer_JTZC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength:**\n\nThe paper provides interesting experiments that can potentially contribute to the understanding of the community about what properties in adversarial examples enables them to sabotage DNNs, and how adversarial-training counteracts these properties. It also suggests interesting connections between the concept of local response differences and previously observed qualities of adversarial examples.\n\n**Concerns and Questions:**\n\n1- A major concern with the paper is that it does not read well, contains many confusing statements, and as such I think it needs a major editorial fix. \n\n2- There are also several ad hoc statements and arguments that must either be carefully justified mathematically or empirically or be properly referenced (for example this statement from section 4.2: “On the other hand, suppose smoother kernels with relatively small variance, these locally-disordered perturbations tend to show smaller absolute response differences since smoother kernels tend to focus on the average situation within their local receptive fields” or from section 3.4: “That is, for a standard model with non-smooth kernels, the model itself tends to amplify the local response differences between potential examples and natural examples” or from section 4.5: “On the other hand, though small differences of local responses make the network more inclined to treat both examples as the same categories, but the existing differences, especially the intricate differences after multi-layer accumulation emphasize the difficulty of approaching model robustness to performance, and demonstrate that DNN models are naturally fragile”\n\n3- The paper does not seem to add much more over the known results from prior works, for example Tsipras et al. (2019) has showed the difference between adversarial examples of adversarially-trained vs. standard DNNs, and the presence of spatially locally consistent patterns in the former, and high frequency patterns in the latter (Figure 3 and 2 of that paper); and Wang et al. (2020a) show the smoothness of the learnt kernel in adversarially-trained DNNs. A more detailed explanation of how this paper differs from these prior works, and what aspects of this work are novel and cannot be readily inferred from the prior works, would really help highlight the importance and relevance of this work.\n\n4- Are the results reported in Table 1,2,3 averaged? Over how many samples? What is the corresponding variance for each value?\n\n5- In section 4.4, how is the following statement inferred from Figure 5h: “and finally tend to narrow the robustness and performance in Figure 5(h)”? I cannot see this narrowness. (also the legends in Figure 5(h) must differentiate between plots with the same color).", "summary_of_the_paper": "This paper provides a set of empirical studies of the spectral and spatial properties of adversarial examples of deep neural nets (DNNs) classifiers. The studies illustrate that standard DNNs are much more sensitive to high frequency components of adversarial examples compared to adversarially-trained DNNs, and also that the adversarial examples corresponding to the latter exhibit more local consistency in the spatial domain. The paper then connects the effectiveness of an adversarial example to the concept of inducing larger differences in the local response of DNN layers compared to the corresponding natural example, denoted local response differences, and justifies this connection by empirically showing that: adversarially-trained DNNs have smaller local response differences on adversarial examples compared to standard DNNs; the smoother the DNN the better its robustness; successful attacks (adversarial examples) induce larger local response differences; and finally that adversarial examples transferred from other models, hence weaker examples, also induce smaller local response differences.", "summary_of_the_review": "As the paper does not read well, and also considering that its novelty compared to prior works is unclear to me, it might not be relevant and valuable to the community in its current form, and as such I cannot recommend accepting it. That said, I think the paper does contain some good experiments that, if explained with more clarity and better placed with respect to what already exists in the prior work, can be helpful to the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores geometric properties of image perturbations (e.g. frequency content and local consistency) and their impact on the adversarial response of networks.  The reviewers feel that the paper is at times unclear about the meaning of terminology (e.g. “local consistency”) that is not clearly defined.  Also, while the reviewers acknowledge that the paper contains a number of interesting ideas, it is not always clear how the paper’s discussions and contributions differ from existing papers (e.g. Dong 2019, Yin et al., 2019, Wang et al., 2020a, Tsuzuku and Sato 2019) that also discuss the frequency content and smoothness properties of adversarial perturbations.
Decision: Reject