Dissecting Local Properties of Adversarial Examples

1 INTRODUCTION . Despite deep neural networks ( DNNs ) perform well in many fields ( He et al. , 2016 ; Devlin et al. , 2019 ) , their counter-intuitive vulnerability attracts increasing attention , both for safety-critical applications ( Sharif et al. , 2016 ) and the black-box mechanism of DNNs ( Fazlyab et al. , 2019 ) . DNNs have been found vulnerable to adversarial examples ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) , where small perturbations on the input can easily change the predictions of a well-trained DNN with high confidence . In computer vision , adversarial examples exhibit their destructiveness both in the digital world and the physical world ( Kurakin et al. , 2017 ) . Since then , how to alleviate the vulnerability of DNN so as to narrow the performance gap between adversarial/natural examples is another key issue . Existing methods including defensive distillation ( Papernot et al. , 2016 ) and pixel denoising ( Liao et al. , 2018 ) have shown their limitations due to follow-up attack strategies ( Carlini & Wagner , 2017 ) or gradient masking ( Athalye et al. , 2018 ) . Amongst them , adversarial training ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ) and its variants ( Zhang et al. , 2019 ; Wang et al. , 2020b ) indicate their reliable robustness and outperform . Moreover , as a data augmentation method , adversarial training currently seems to rely on additional data ( Schmidt et al. , 2018 ; Rebuffi et al. , 2021 ) to further improve robustness , while is sensitive to some basic model hyper-parameters , e.g. , weight decay ( Pang et al. , 2021 ) . Apart from these , the effect of simply early stopping ( Rice et al. , 2020 ) even exceeds some promotion methods according to recent benchmarks ( Croce & Hein , 2020 ; Chen & Gu , 2020 ) . These studies arise our curiosity to further explore the relationship between adversarial examples and adversarial training , hoping to provide some new understanding . Recalling that high-frequency components can be potentially linked to adversarial examples ( Wang et al. , 2020a ; Yin et al. , 2019 ; Harder et al. , 2021 ) , however , few explorations discuss the relationship between high-frequency components and the destructiveness of adversarial examples . In this paper , we first demonstrate that high-frequency components of adversarial examples tend to mislead the standard DNNs , yet little impact on the adversarially robust models . We further show that adversarial examples statistically have more high-frequency components than natural ones , indicating relatively drastic local changes among pixels of adversarial examples . Since adversarial examples exhibit more semantically meaningful on robust models ( Tsipras et al. , 2019 ) , we further notice that adversarial examples show locally-consistent perturbations related to image shapes on adversariallytrained models , in contrast to disorderly perturbations on standard models . Both explorations on the frequency and spatial domain emphasize local properties of adversarial examples , and motivated by the local receptive field of the convolution kernels , we propose a locally intermediate response perspective to rethink the vulnerability of DNNs . Different from the existing global activation perspective ( Bai et al. , 2021 ; Xu et al. , 2019 ) , our local perspective reflects the joint effect of local features and the intermediate layers of the model . Based on the local perspective , we emphasize that large enough local response differences make it difficult for the network to treat an image and its potentially adversarial examples as one category , and demonstrate DNN models are naturally fragile at least attributed to it . Motivated by adversarially-trained models tend to have ‘ smooth ’ kernels ( Wang et al. , 2020a ) , we simply use the smoother kernels to alleviate local response differences on adversarially-trained models , which in turn affects the model robustness and reduces the robust overfitting ( Rice et al. , 2020 ) . To a certain extent , this explains why weight decay effectively affects model robustness ( Pang et al. , 2021 ) . Our main contributions are summarized as follows : • We first reveal some properties of adversarial examples in the frequency and spatial domain : 1 ) the high-frequency components of adversarial examples tend to mislead naturally-trained DNNs , yet have little impact on adversarially-trained models , and 2 ) adversarial examples have locally-consistent perturbations on adversarially-trained models , compared with disorderly local perturbations on naturally-trained models . • Then we introduce local response and emphasize its importance in the model adversarial robustness . That is , naturally-trained DNNs are often fragile , at least for non-ignorable local response differences through the same layer between potentially adversarial examples and natural ones . In contrast , adversarially-trained models effectively alleviate the local response differences . And the smoother adversarially-trained models show better adversarial robustness as they can reduce local response differences . • Finally we empirically study local response with generated adversarial examples . We further show that , compared with failed attacks , adversarial examples ( successful attacks ) statistically show larger local response differences with natural examples . Moreover , compared with adversarial examples generated by the model itself , those transferred by other models show markedly smaller local response differences . 2 RELATED WORK . Understandings of model vulnerability . Since the discovery of adversarial examples ( Szegedy et al. , 2014 ) , a number of understandings on model vulnerability has been developed . For instance , linear property of DNNs ( Goodfellow et al. , 2015 ) , submanifold ( Tanay & Griffin , 2016 ) and geometry of the manifold ( Gilmer et al. , 2018 ) were considered from the high-dimensional perspective ; the computing of Lipschitz constant ( Szegedy et al. , 2014 ; Fazlyab et al. , 2019 ) and lower/upper bounds ( Fawzi et al. , 2018 ; Weng et al. , 2018 ) were considered from the definition of model robustness ; non-robust features ( Ilyas et al. , 2019 ) , high-frequency components ( Wang et al. , 2020a ; Yin et al. , 2019 ) , high-rank features ( Jere et al. , 2020 ) and high-order interactions among pixels ( Ren et al. , 2021 ) were explored adversarial examples from the different perspectives on images , which imply our local perspective ; feature denosing ( Xie et al. , 2019 ) , robust pruning ( Madaan & Ju Hwang. , 2020 ) and activation suppressing ( Bai et al. , 2021 ) were focused on global intermediate activations of models . On the other hand , taken adversarial training into consideration , Tsipras et al . ( 2019 ) , Schmidt et al . ( 2018 ) and Zhang et al . ( 2019 ) explored the trade-off between robustness and accuracy ; Wang et al . ( 2020a ) found adversarially-trained models tend to show smooth kernels ; Tsipras et al . ( 2019 ) and Zhang & Zhu ( 2018 ) argued adversarially robust models learned more shape-biased representations . Different from these studies , we characterize the adversarial examples from both frequency and spatial domain to emphasize local properties of adversarial examples , and propose a locally intermediate response to rethink the vulnerability of DNNs . Adversarial training . Adversarial training can be seen as a min-max optimization problem : min θ 1 n n∑ i=1 max x′i∈B ( x ) L ( fθ ( x ′ i ) , yi ) where f denotes a DNN model with parameters θ , and ( xi , yi ) denotes a pair of a natural example xi and its ground-truth label yi . Given a classification loss L , the inner maximization problem can be regarded as searching for suitable perturbations in boundary B to maximize loss , while the outer minimization problem is to optimize model parameters on adversarial examples { x′i } ni=1 generated from the inner maximization . 3 DIFFERENT PERTURBATIONS AFFECT THE VULNERABILITY OF THE MODEL . In this section , we first investigate adversarial examples in the frequency and spatial domain , and show the connections between models and their adversarial examples . Note that our threat model is a white-box model , and the fact that the adversarial example is only defined by the misleading result under the legal threat model , our findings suggest that the properties of the sampled examples can broadly reflect the fragile tendency of the model . Setup . We generate ` ∞ bounded adversarial examples by PGD-10 ( maximum perturbation = 8/255 and step size 2/255 ) with random start for the robust model ( Madry et al. , 2018 ) . Specifically , we use ResNet-18 ( He et al. , 2016 ) as the backbone to train the standard and adversarially-trained models for 100 epochs on CIFAR-10 ( Krizhevsky , 2009 ) . Following ( Pang et al. , 2021 ) , we use the SGD optimizer with momentum 0.9 , weight decay 5 × 10−4 and initial learning rate 0.1 , with a three-stage learning rate divided by 10 at 50 and 75 epoch respectively . The robustness of both models is evaluated by PGD-20 ( step size 1/255 ) . 3.1 THE DESTRUCTIVENESS OF HIGH-FREQUENCY COMPONENTS FROM ADVERSARIES . Inspired by ( Wang et al. , 2020a ) , we are naturally curious whether the adversarial examples cause considerable damage to the model mainly because of their high-frequency components . To answer this question , Figure 1 illustrates the trend of model performance and robustness on the test set with the high-frequency components increased ( the increase of the filtering scale denotes that more highfrequency components are added to the filtered images ) . Figure 1 ( a ) shows that , for standard models , the high-frequency components of natural examples promote classification and reach the model performance ( green line ) ; on the contrary , the performance of the filtered adversarial examples first rises to get the highest accuracy 47.5 % , and then drops rapidly to reach 0.0 % ( red line ) . Obviously , in the low-frequency range , the performance of natural and adversarial examples are quite close , yet more high-frequency components widen the difference . That is , the special high-frequency components caused by adversarial perturbations exhibit a clear destructive effect on standard models , and simply filter out them can effectively alleviate the destructiveness of adversaries even on standard models . However , for robust models , we show that the prediction performance finally reaches robustness without a rapid drop in Figure 1 ( b ) . But surprisingly , we find that the performance of filtered adversarial examples in some range exceeds the final robustness 47.5 % ( red line ) , reaching a maximum of 51.2 % . That is , although these high-frequency components do not exhibit a clear destructive effect , simply filtering out them has a positive impact on alleviating robust overfitting ( Rice et al. , 2020 ) . We then swap their high-frequency components between both examples controlled by a frequency threshold in Figure 1 ( c ) -1 ( d ) . For merged natural examples with high-frequency components from adversaries , the increase of the frequency threshold controls the accuracy increasing from the model robustness ( red line ) to the model performance ( green line ) , the opposite occurs on merged adversarial examples . These clearly illustrate the boost effect of the high-frequency components from natural examples and the destructive effect of the high-frequency components from adversarial examples . To further illustrate , we find that statistically , the main difference in the frequency domain between natural and adversarial examples is concentrated in the high-frequency region in Figure 2 . We visualize the logarithmic amplitude spectrum of both examples . Figure 2 ( a ) and 2 ( b ) show that , compare with natural examples ’ , the high-frequency components of the adversaries are hard to ignore . Figure 2 ( c ) further emphasizes that adversarial examples markedly show more high-frequency components , indicating relatively drastic local changes among pixels . This statistical difference explains the high detection rate of using Magnitude Fourier Spectrum ( Harder et al. , 2021 ) to detect adversarial examples . Furthermore , Figure 2 ( d ) and 2 ( e ) show that the high-frequency components of adversarial examples generated by robust models are less than those from standard models , yet still more than natural examples ’ . Besides , the analysis of filtering out low-frequency components in Figure 6 ( Appendix A ) also emphasizes our statement . That is , compared with natural examples ’ , the special high-frequency components of adversarial examples show their serious misleading effects on standard models , yet are not enough to be fully responsible for the destructiveness to robust models .

Summary: This paper studies the properties of adversarial examples from a spatial and frequency perspective and shows that naturally trained models are more vulnerable to high-frequency components in adversarial examples. Perturbations for naturally trained models are disordered, but perturbations for adv-trained models are image shape related. Based on these observations, the authors find that smoother adversarially-trained can achieve better robustness.