Dissecting Local Properties of Adversarial Examples

1 INTRODUCTION . Despite deep neural networks ( DNNs ) perform well in many fields ( He et al. , 2016 ; Devlin et al. , 2019 ) , their counter-intuitive vulnerability attracts increasing attention , both for safety-critical applications ( Sharif et al. , 2016 ) and the black-box mechanism of DNNs ( Fazlyab et al. , 2019 ) . DNNs have been found vulnerable to adversarial examples ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) , where small perturbations on the input can easily change the predictions of a well-trained DNN with high confidence . In computer vision , adversarial examples exhibit their destructiveness both in the digital world and the physical world ( Kurakin et al. , 2017 ) . Since then , how to alleviate the vulnerability of DNN so as to narrow the performance gap between adversarial/natural examples is another key issue . Existing methods including defensive distillation ( Papernot et al. , 2016 ) and pixel denoising ( Liao et al. , 2018 ) have shown their limitations due to follow-up attack strategies ( Carlini & Wagner , 2017 ) or gradient masking ( Athalye et al. , 2018 ) . Amongst them , adversarial training ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ) and its variants ( Zhang et al. , 2019 ; Wang et al. , 2020b ) indicate their reliable robustness and outperform . Moreover , as a data augmentation method , adversarial training currently seems to rely on additional data ( Schmidt et al. , 2018 ; Rebuffi et al. , 2021 ) to further improve robustness , while is sensitive to some basic model hyper-parameters , e.g. , weight decay ( Pang et al. , 2021 ) . Apart from these , the effect of simply early stopping ( Rice et al. , 2020 ) even exceeds some promotion methods according to recent benchmarks ( Croce & Hein , 2020 ; Chen & Gu , 2020 ) . These studies arise our curiosity to further explore the relationship between adversarial examples and adversarial training , hoping to provide some new understanding . Recalling that high-frequency components can be potentially linked to adversarial examples ( Wang et al. , 2020a ; Yin et al. , 2019 ; Harder et al. , 2021 ) , however , few explorations discuss the relationship between high-frequency components and the destructiveness of adversarial examples . In this paper , we first demonstrate that high-frequency components of adversarial examples tend to mislead the standard DNNs , yet little impact on the adversarially robust models . We further show that adversarial examples statistically have more high-frequency components than natural ones , indicating relatively drastic local changes among pixels of adversarial examples . Since adversarial examples exhibit more semantically meaningful on robust models ( Tsipras et al. , 2019 ) , we further notice that adversarial examples show locally-consistent perturbations related to image shapes on adversariallytrained models , in contrast to disorderly perturbations on standard models . Both explorations on the frequency and spatial domain emphasize local properties of adversarial examples , and motivated by the local receptive field of the convolution kernels , we propose a locally intermediate response perspective to rethink the vulnerability of DNNs . Different from the existing global activation perspective ( Bai et al. , 2021 ; Xu et al. , 2019 ) , our local perspective reflects the joint effect of local features and the intermediate layers of the model . Based on the local perspective , we emphasize that large enough local response differences make it difficult for the network to treat an image and its potentially adversarial examples as one category , and demonstrate DNN models are naturally fragile at least attributed to it . Motivated by adversarially-trained models tend to have ‘ smooth ’ kernels ( Wang et al. , 2020a ) , we simply use the smoother kernels to alleviate local response differences on adversarially-trained models , which in turn affects the model robustness and reduces the robust overfitting ( Rice et al. , 2020 ) . To a certain extent , this explains why weight decay effectively affects model robustness ( Pang et al. , 2021 ) . Our main contributions are summarized as follows : • We first reveal some properties of adversarial examples in the frequency and spatial domain : 1 ) the high-frequency components of adversarial examples tend to mislead naturally-trained DNNs , yet have little impact on adversarially-trained models , and 2 ) adversarial examples have locally-consistent perturbations on adversarially-trained models , compared with disorderly local perturbations on naturally-trained models . • Then we introduce local response and emphasize its importance in the model adversarial robustness . That is , naturally-trained DNNs are often fragile , at least for non-ignorable local response differences through the same layer between potentially adversarial examples and natural ones . In contrast , adversarially-trained models effectively alleviate the local response differences . And the smoother adversarially-trained models show better adversarial robustness as they can reduce local response differences . • Finally we empirically study local response with generated adversarial examples . We further show that , compared with failed attacks , adversarial examples ( successful attacks ) statistically show larger local response differences with natural examples . Moreover , compared with adversarial examples generated by the model itself , those transferred by other models show markedly smaller local response differences . 2 RELATED WORK . Understandings of model vulnerability . Since the discovery of adversarial examples ( Szegedy et al. , 2014 ) , a number of understandings on model vulnerability has been developed . For instance , linear property of DNNs ( Goodfellow et al. , 2015 ) , submanifold ( Tanay & Griffin , 2016 ) and geometry of the manifold ( Gilmer et al. , 2018 ) were considered from the high-dimensional perspective ; the computing of Lipschitz constant ( Szegedy et al. , 2014 ; Fazlyab et al. , 2019 ) and lower/upper bounds ( Fawzi et al. , 2018 ; Weng et al. , 2018 ) were considered from the definition of model robustness ; non-robust features ( Ilyas et al. , 2019 ) , high-frequency components ( Wang et al. , 2020a ; Yin et al. , 2019 ) , high-rank features ( Jere et al. , 2020 ) and high-order interactions among pixels ( Ren et al. , 2021 ) were explored adversarial examples from the different perspectives on images , which imply our local perspective ; feature denosing ( Xie et al. , 2019 ) , robust pruning ( Madaan & Ju Hwang. , 2020 ) and activation suppressing ( Bai et al. , 2021 ) were focused on global intermediate activations of models . On the other hand , taken adversarial training into consideration , Tsipras et al . ( 2019 ) , Schmidt et al . ( 2018 ) and Zhang et al . ( 2019 ) explored the trade-off between robustness and accuracy ; Wang et al . ( 2020a ) found adversarially-trained models tend to show smooth kernels ; Tsipras et al . ( 2019 ) and Zhang & Zhu ( 2018 ) argued adversarially robust models learned more shape-biased representations . Different from these studies , we characterize the adversarial examples from both frequency and spatial domain to emphasize local properties of adversarial examples , and propose a locally intermediate response to rethink the vulnerability of DNNs . Adversarial training . Adversarial training can be seen as a min-max optimization problem : min θ 1 n n∑ i=1 max x′i∈B ( x ) L ( fθ ( x ′ i ) , yi ) where f denotes a DNN model with parameters θ , and ( xi , yi ) denotes a pair of a natural example xi and its ground-truth label yi . Given a classification loss L , the inner maximization problem can be regarded as searching for suitable perturbations in boundary B to maximize loss , while the outer minimization problem is to optimize model parameters on adversarial examples { x′i } ni=1 generated from the inner maximization . 3 DIFFERENT PERTURBATIONS AFFECT THE VULNERABILITY OF THE MODEL . In this section , we first investigate adversarial examples in the frequency and spatial domain , and show the connections between models and their adversarial examples . Note that our threat model is a white-box model , and the fact that the adversarial example is only defined by the misleading result under the legal threat model , our findings suggest that the properties of the sampled examples can broadly reflect the fragile tendency of the model . Setup . We generate ` ∞ bounded adversarial examples by PGD-10 ( maximum perturbation = 8/255 and step size 2/255 ) with random start for the robust model ( Madry et al. , 2018 ) . Specifically , we use ResNet-18 ( He et al. , 2016 ) as the backbone to train the standard and adversarially-trained models for 100 epochs on CIFAR-10 ( Krizhevsky , 2009 ) . Following ( Pang et al. , 2021 ) , we use the SGD optimizer with momentum 0.9 , weight decay 5 × 10−4 and initial learning rate 0.1 , with a three-stage learning rate divided by 10 at 50 and 75 epoch respectively . The robustness of both models is evaluated by PGD-20 ( step size 1/255 ) . 3.1 THE DESTRUCTIVENESS OF HIGH-FREQUENCY COMPONENTS FROM ADVERSARIES . Inspired by ( Wang et al. , 2020a ) , we are naturally curious whether the adversarial examples cause considerable damage to the model mainly because of their high-frequency components . To answer this question , Figure 1 illustrates the trend of model performance and robustness on the test set with the high-frequency components increased ( the increase of the filtering scale denotes that more highfrequency components are added to the filtered images ) . Figure 1 ( a ) shows that , for standard models , the high-frequency components of natural examples promote classification and reach the model performance ( green line ) ; on the contrary , the performance of the filtered adversarial examples first rises to get the highest accuracy 47.5 % , and then drops rapidly to reach 0.0 % ( red line ) . Obviously , in the low-frequency range , the performance of natural and adversarial examples are quite close , yet more high-frequency components widen the difference . That is , the special high-frequency components caused by adversarial perturbations exhibit a clear destructive effect on standard models , and simply filter out them can effectively alleviate the destructiveness of adversaries even on standard models . However , for robust models , we show that the prediction performance finally reaches robustness without a rapid drop in Figure 1 ( b ) . But surprisingly , we find that the performance of filtered adversarial examples in some range exceeds the final robustness 47.5 % ( red line ) , reaching a maximum of 51.2 % . That is , although these high-frequency components do not exhibit a clear destructive effect , simply filtering out them has a positive impact on alleviating robust overfitting ( Rice et al. , 2020 ) . We then swap their high-frequency components between both examples controlled by a frequency threshold in Figure 1 ( c ) -1 ( d ) . For merged natural examples with high-frequency components from adversaries , the increase of the frequency threshold controls the accuracy increasing from the model robustness ( red line ) to the model performance ( green line ) , the opposite occurs on merged adversarial examples . These clearly illustrate the boost effect of the high-frequency components from natural examples and the destructive effect of the high-frequency components from adversarial examples . To further illustrate , we find that statistically , the main difference in the frequency domain between natural and adversarial examples is concentrated in the high-frequency region in Figure 2 . We visualize the logarithmic amplitude spectrum of both examples . Figure 2 ( a ) and 2 ( b ) show that , compare with natural examples ’ , the high-frequency components of the adversaries are hard to ignore . Figure 2 ( c ) further emphasizes that adversarial examples markedly show more high-frequency components , indicating relatively drastic local changes among pixels . This statistical difference explains the high detection rate of using Magnitude Fourier Spectrum ( Harder et al. , 2021 ) to detect adversarial examples . Furthermore , Figure 2 ( d ) and 2 ( e ) show that the high-frequency components of adversarial examples generated by robust models are less than those from standard models , yet still more than natural examples ’ . Besides , the analysis of filtering out low-frequency components in Figure 6 ( Appendix A ) also emphasizes our statement . That is , compared with natural examples ’ , the special high-frequency components of adversarial examples show their serious misleading effects on standard models , yet are not enough to be fully responsible for the destructiveness to robust models .

Summary: This paper provides a set of empirical studies of the spectral and spatial properties of adversarial examples of deep neural nets (DNNs) classifiers. The studies illustrate that standard DNNs are much more sensitive to high frequency components of adversarial examples compared to adversarially-trained DNNs, and also that the adversarial examples corresponding to the latter exhibit more local consistency in the spatial domain. The paper then connects the effectiveness of an adversarial example to the concept of inducing larger differences in the local response of DNN layers compared to the corresponding natural example, denoted local response differences, and justifies this connection by empirically showing that: adversarially-trained DNNs have smaller local response differences on adversarial examples compared to standard DNNs; the smoother the DNN the better its robustness; successful attacks (adversarial examples) induce larger local response differences; and finally that adversarial examples transferred from other models, hence weaker examples, also induce smaller local response differences.