A frequency domain analysis of gradient-based adversarial examples

1 INTRODUCTION . Recently , deep neural networks ( DNN ) have achieved great success in the field of image processing , but it was found that DNNs are vulnerable to some synthetic data called adversarial examples ( ( Szegedy et al. , 2013 ) , ( Kurakin et al. , 2016 ) ) . Adversarial examples are natural samples plus adversarial perturbations , and the perturbations between natural samples and adversarial examples are imperceptible to human but able to fool the model . Typically , generating an adversarial example can be considered as finding an example in an -ball around a natural image that could be misclassified by the classifier . Recent studies designed Fast Gradient Sign Method ( FGSM , ( Goodfellow et al. , 2014 ) ) , Fast Gradient Method ( FGM , ( Miyato et al. , 2016 ) ) , Projected Gradient Descent ( PGD , ( Madry et al. , 2017 ) ) and other algorithms ( Carlini & Wagner ( 2017 ) , ( Su et al. , 2019 ) , ( Xiao et al. , 2018 ) , ( Kurakin et al. , 2016 ) , ( Chen et al. , 2017 ) ) to attack the model . Since the phenomenon of adversarial examples was discovered , many related works have made progress to study why they exist . Several works studied this phenomenon from the perspective of feature representation . Ilyas et al . ( 2019 ) divided the features into non-robust ones that are responsible for the model ’ s vulnerability to adversarial examples , and robust ones that are close to human perception . Further , they showed that adversarial vulnerability arises from non-robust features that are useful for correct classification . Another way to characterize adversarial examples is to investigate them in the frequency domain via Fourier transform . Wang et al . ( 2020a ) divided an image into low-frequency component ( LFC ) and high-frequency component ( HFC ) and they empirically showed that human can only perceive LFC , but convolutional neural networks can obtain useful information from both LFC and HFC . Yin et al . ( 2019 ) filters out the input data with low-pass or high-pass filters to study the sensitivity of the additive noise with different frequencies . Wang et al . ( 2020b ) claimed that existing adversarial attacks mainly concentrate in the high-frequency part . Sharma et al . ( 2019 ) found that when perturbations are constrained to the low-frequency subspace , they are generated faster and are more transferable , and will be effective to fool the defended models , but not for clean models . All of these works showed that spectrum in frequency domain is an reasonable way to study the adversarial examples . However , there is a lack of theoretical understanding about the dynamics of the adversarial perturbations in frequency domain along the training process of the model parameters . In this work , we focus on the frequency domain of the adversarial perturbations to explore the spectral properties of the adversarial examples generated by FGM ( Miyato et al. , 2016 ) and PGD ( Madry et al. , 2017 ) . In this work , we give a theoretical analysis in frequency domain of natural images for the adversarial examples . • For a two-layer neural network with non-linear activation function , we prove that adversarial perturbations by FGM and l2-PGD attacks gradually increase the concentration in the low-frequency part of the spectrum during the training process over model parameters . • Meanwhile , the log-spectrum difference of the adversarial examples ( the definition will be clarified in Section 2.2 ) will be more concentrated in the high-frequency part than the low-frequency part . • Furthermore , we show that the ratio of the high-frequency and the low-frequency part in the adversarial perturbation is much larger than that in the corresponding clean image . Empirically , • we design several experiments on the two-layer model and Resnet-32 with CIFAR-10 to verify the above findings . • Based on these phenomena , we filter out the high-frequency part of the adversarial examples before feeding them to the model to improve the robustness . Compared with the adversarially trained model with the same architecture , our method achieves comparable robustness with the similar computational cost of normal training and almost no loss of accuracy . The rest of the paper is organized as follows : In Section 2 , we present preliminaries and some theoretical analysis on the calculation of the spectrum and log-spectrum . Then we provide our main results about the frequency domain analysis of gradient-based adversarial examples in Section 3 . Furthermore , some experiments for supporting our theoretical findings are shown in Section 4 . Finally we conclude our work and conduct a discussion about the future work in Section 5 . All the details about the proof and experiments are shown in the Appendix . 2 BACKGROUND . 2.1 PRELIMINARIES . Notations We use { 0 , 1 , ... , d } to denote the set of all integers between 0 and d and use ‖ · ‖p to denote lp norm . Specifically , we denote by ‖ · ‖ the l2 norm . For a d-dimensional vector x , we use xµ to denote its µ-th component with index starting at 0 . For a scalar function f ( x ) : Rd 7→ R , ∇xf and ∂µf denote the gradient vector and its µ-th component . We let sgn ( x ) = 1 for x > 0 and −1 for x < 0 . Normal training refers to training on the original training data for learning the optimal weights for the neural networks . x̃ = F ( x ) denotes the Discrete Fourier Transform ( DFT ) of x. Discrete Fourier Transform The k-th frequency component g̃ [ k ] of one-dimensional DFT of a vector g is defined by g̃ [ k ] = F ( g ) [ k ] = d−1∑ µ=0 gµe ik 2πd µ , where k ∈ { −d/2 , −d/2 + 1 , ... , d/2 } if d is even and k ∈ { − ( d − 1 ) /2 , ... , ( d − 1 ) /2 } if d is odd . For convenience , we always consider an odd d and one can easily generalize the case to even dimensions . For an integer cut-off ( cutoff frequency ) kr ∈ ( 0 , ( d− 1 ) /2 ) , the Low Frequency Components ( LFC ) and High Frequency Components ( HFC ) of g̃ are denoted by g̃l [ k ] = { g̃ [ k ] if − kc ≤ k ≤ kc , 0 otherwise g̃h [ k ] = { 0 if − kc ≤ k ≤ kc , g̃ [ k ] otherwise . Then the frequency space Rd can be decomposed as Sl ⊕ Sh where g̃l ∈ Sl and g̃h ∈ Sh . LetNg̃ =√∑d/2 k=−d/2 |g̃ [ k ] |2 , then the ratio of a frequency component with respect to the whole frequency spectrum is denoted by τ ( g̃ [ k ] ) = |g̃ [ k ] |2 N2g̃ and the ratio of LFC is denoted by τ ( g̃ ∈ Sl ) = kc∑ k=−kc |g̃ [ k ] |2 N2g̃ . Setup In this paper , we consider a two-layer neural network f : Rd 7→ R f ( x , θ ) = m∑ r=1 arσ ( w T r x ) ( 1 ) where σ is an activation function , a = ( a1 , a2 , ... , am ) T is an m-dimensional vector and wr ’ s are d-dimensional weight vectors . Van der Schaaf & van Hateren ( 1996 ) showed that natural images obey a power law in the frequency domain . Therefore , to make our settings closer to widely studied image related tasks , we also assume our input x obey the power law |x̃ [ k ] | ∝ k−α to imitate LFC-concentrated images , where the constant α ≥ 1 and |x̃ [ 0 ] | = α0 is another constant . Besides , we choose the cut-off kc in such a way that kc∑ k=1 1 k2α + α20 > ( d−1 ) /2∑ k=kc+1 1 k2α , ( 2 ) which indicates that τ ( x̃ ∈ Sl ) > τ ( x̃ ∈ Sh ) and can be easily satisfied for a small kc . Gradient-based adversarial attack For a loss function ` , we find lp-norm -bounded adversarial perturbations by solving the optimization problem max ‖δ‖p≤ ` ( f ( x+ δ ) , y ) , where ( x , y ) ∈ Rd × R is d-dimensional input and output following the joint distribution D. To solve the above optimization problem , Goodfellow et al . ( 2014 ) proposed FGSM , an attack for an l∞-bounded adversary , and computed an adversarial example as xadv = x + sgn ( ∇x ` ( θ , x , y ) ) . Miyato et al . ( 2016 ) proposed FGM with an l2-bounded adversary , xadv = x+ ∇x ` ( x , y ) ‖∇x ` ( x , y ) ‖2 . Madry et al . ( 2017 ) introduced PGD which training adversarial examples in an iterative way that xt+1 = Πx+S ( x t + α sgn ( ∇x ` ( θ , x , y ) ) ) or xt+1 = Πx+S ( xt + α ∇x ` ( x , y ) ‖∇x ` ( x , y ) ‖2 ) To simplify the analysis , we make the following assumptions on the normal training process : Assumption 1 There exist β′ , β > 0 such that β′ ≤ |∂ ` /∂f | ≤ β . Assumption 2 If ` = 12 ( y − f ( x , θ ) ) 2 , |∂ ` /∂f | should be a small quantity such that |∂ ` /∂f | = 1+v if the network is well trained in the end stage of normal training , where 0 < v < 1 . Assumption 3 There exist λ , γ > 0 such that 0 ≤ σ′ ≤ λ and 0 ≤ σ′′ ≤ γ1 . Assumption 4 ar ’ s are i.i.d drawn from a distribution2 and will not be updated during the training . 1e.g . λ = 1 and γ = 1 4 if σ = ln ( 1 + ex ) . 2e.g . Bernoulli distribution . 2.2 EXPLORING THE LOG-SPECTRUM AND SPECTRUM . Wang et al . ( 2020a ) considered the log-spectrum of an image : 20 ∗ log ( |F ( x ) | ) , where x is a 2ddimensional matrix representing an image . One way to calculate the ” log-spectrum difference of the adversarial examples ” is to consider the difference between the log-spectrum of the adversarial and natural image and as follows , 20 ∗ |log ( |F ( x+ δ ) | ) − log ( |F ( x ) | ) | ≈ 20 ∗ ∣∣∣∣log ( 1 + |F ( δ ) ||F ( x ) | ) ∣∣∣∣ ≈ 20 ∗ |F ( δ ) ||F ( x ) | . ( 3 ) We can observe that this quantity is approximately proportional to the ratio of the perturbation ’ s over the natural image ’ s frequency . Moreover , the average Relative Change in discrete cosine Transforms ( RCT ) in ( Wang et al. , 2020b ) ) approximately equals to it . Fig . 5 empirically shows that this ratio is large for high frequency component but small for low frequency one . However , Van der Schaaf & van Hateren ( 1996 ) claimed that the LFC of F ( x ) is much bigger than its HFC , and the right side of Eq . 3 is inversely proportional to F ( x ) . The denominator of the right side of Eq . ( 3 ) in low-frequency part is much larger than it in high frequency part , so it may be imprudent to consider the frequency distribution of the adversarial perturbations only based on the ratio in the right side of Eq. ( 3 ) . Yin et al . ( 2019 ) provided another spectral measurement for adversarial examples . They considered the ” spectrum of the adversarial perturbation ” |F ( δ ) | , which directly reflects the distribution of adversarial perturbations in the frequency domain . Unfortunately , they claimed that ‘ The adversarial perturbations for the normally trained model are uniformly distributed across frequency components ’ without sufficient empirical and theoretical justification . In this work , we provide a totally different theoretical finding that the adversarial perturbations for the normally trained model are gradually concentrated in the low-frequency domain , further verified by extensive experiments . Fig . 1 visually shows the calculation methods of the spectrum of perturbations F ( δ ) and the logspectrum difference of adversarial examples | log ( |F ( x+ δ ) | ) − log ( |F ( x ) | ≈ |F ( δ ) ||F ( x ) | . 3 MAIN RESULTS In Section 3.1 , we show that , for a well trained two-layer neural network described in Eq . ( 1 ) with LFC-concentrated inputs , l2norm adversarial perturbations generated by FGM ( Theorem 1 ) prefer to concentrate in the low frequency domain . Section 3.2 theoretically proves the HFC-concentration of the logspectrum difference of adversarial examples . Section 3.3 attempts to demonstrate the effectiveness of masking the HFC of the adversarial examples to improve model ’ s robustness despite that perturbations can concentrate in the low frequency domain . All technical proofs are deferred to Appendix A . Results on l2 PGD perturbations are presented in Appendix B . In this work , we consider the two-layer neural networks and the loss function ` = 1 2 ( y − f ( x , θ ) ) 2 . For any x in this setting , we have ∂f ∂ar = σ ( wTr x ) , ∂f ∂wr = a′rx , ∇xf = ∑ r a′rwr , where a′r = arσ ′ ( wTr x ) and a ′ = ( a′1 , a ′ 2 , ... , a ′ m ) . At the ( t+ 1 ) -th step of gradient descent in the normal training process of the classifier , weight w is updated as w ( t+1 ) r = w ( t ) r − η ∂ ` ∂f ( t ) a′ ( t ) r x . ( 4 ) We use Lx and Hx to denote the LFC and HFC of the clean input Lx = kc∑ k=0 |x̃ [ k ] |2 and Hx = ( d−1 ) /2∑ k=kc+1 |x̃ [ k ] |2 . ( 5 ) 3.1 SPECTRAL TRAJECTORIES OF l2-NORM FGM PERTURBATIONS Consider the l2-norm FGM perturbation δ and its DFT δ̃ [ k ] δ = ∂ ` ∂f ∇xf ‖ ∂ ` ∂f ∇xf‖ , δ̃ [ k ] = sgn ( ∂ ` ∂f ) 1 ‖∇xf‖ ∇̃f [ k ] , ( 6 ) where ∇̃f [ k ] = F ( ∇xf ) [ k ] . Since the coefficient before ∇̃f [ k ] in ( 6 ) is same for different frequencies k , we only need to analyze the spectrum of ∇̃f [ k ] to investigate the ratio of certain frequency component over the whole frequency spectrum , i.e . τ ( δ̃ [ k ] ) = τ ( ∇̃f [ k ] ) . ( 7 ) In this subsection , we explore the evolving of τ ( δ̃ [ k ] ) along the normal training process . For a randomly initialized network , the FGM-attack perturbation will not have bias towards either HFC or LFC . We attack this model with FGM at each step of the normal training to study the trajectory of these perturbations in the frequency domain . According to Eq . ( 4 ) , ∇xf is updated to the order of η at the ( t+ 1 ) -th step as ∇xf ( t+1 ) = m∑ r=1 ( 1− η ( t ) r ) a′ ( t ) r w ( t ) r − η̃ ( t ) x+O ( η2 ) , where η ( t ) r = ηar‖x‖2 ∂ ` ∂f ( t ) σ′′ ( t ) ( w ( t ) T r x ) and η̃ ( t ) = η ∂ ` ∂f ( t ) ‖a′ ( t ) ‖2 are composite learning rates for easing our notation . For ReLU activation function σ ( x ) = max ( 0 , x ) , we have that η ( t ) r = Ei∈ { 1,2 , ... , m } [ η ( t ) i ] , therefore , considering that Softplus has similar shape as ReLU , we derive that η ( t ) r = η̄ ( t ) + O ( η2 ) , where η̄ ( t ) = η‖x‖2 ∂ ` ∂f ( t ) Er∈ { 1,2 , ... , m } [ arσ ′′ ( t ) ( w ( t ) T r x ) ] . In this way , the 1d-DFT of ∇xf ( t+1 ) has the following form , ∇̃f ( t+1 ) [ k ] = ( 1− η̄ ( t ) ) ∇̃f ( t ) [ k ] − η̃ ( t ) x̃ [ k ] . ( 8 ) We are now ready to study the trajectory of τ ( δ̃ [ k ] ) along the training step t. Let 4ϕ ( t ) k denote the difference of phases between ∇̃f ( t ) [ k ] and x̃ [ k ] for frequency k at the t-th step . And for ease of notation , denote 4τ ( t ) l , −2 kc∑ k=0 |∇̃f ( t ) [ k ] | |x̃ [ k ] | cos ( 4ϕ ( t ) k ) , 4τ ( t ) h , −2 ( d−1 ) /2∑ k=kc+1 |∇̃f ( t ) [ k ] | |x̃ [ k ] | cos ( 4ϕ ( t ) k ) , where η̃ ( t ) 4τ ( t ) l and η̃ ( t ) 4τ ( t ) h are approximately changed amounts for LFC and HFC of |∇̃f ( t ) |2 at the t-th step of the normal training of the network , respectively . Since the network is randomly initialized and the clean inputs x are highly concentrated in the low frequency domain , we always study the case where τ ( δ̃ ( 0 ) ∈ Sl ) < τ ( x̃ ∈ Sl ) . We now , taking Eq . ( 7 ) into consideration , present our main theorem about l2 norm FGM perturbations as follows . Theorem 1 ( The spectral trajectory of l2 FGM perturbation ) During the training process of the two-layer neural network f ( x ) in ( 1 ) , the l2 norm FGM adversarial perturbation will change its ratio of LFC , τ ( δ̃ ∈ Sl ) , at the ( t+ 1 ) -th step as follows , τ ( δ̃ ( t+1 ) ∈ Sl ) = τ ( δ̃ ( t ) ∈ Sl ) + η̃ ( t ) 4τ ( t ) l τ ( δ̃ ( t ) ∈ Sh ) −4τ ( t ) h τ ( δ̃ ( t ) ∈ Sl ) ∑ ( d−1 ) /2 k=0 |∇̃f ( t+1 ) [ k ] |2 ; ( 9 ) there will be a t1 s.t . ∀t ≥ t1 , we have τ ( δ̃ ( t+1 ) ∈ Sl ) > τ ( δ̃ ( t ) ∈ Sl ) . ( 10 ) Remark According to theorem 1 , during the normal training process for a randomly initialized network with LFC-concentrated input , there will be a t0 s.t . ∀t ≥ t0 , we have 4τ ( t ) l > 4τ ( t ) h , and τ ( δ̃ ( t+1 ) ∈ Sl ) > τ ( δ̃ ( t ) ∈ Sl ) if τ ( δ̃ ( t ) ∈ Sl ) < τ ( δ̃ ( t ) ∈ Sh ) . For ReLU activa- tion function , starting with τ ( δ̃ ( 0 ) ∈ Sl ) = 1/2 − ζ and 4τ ( 0 ) l = 4τ ( 0 ) h + bζ , we have t0 = max { 0 , − bζnηβ′ ( Lx−Hx ) } , where n = mint∈ [ 0 , t0 ] ‖a′ ( t ) ‖2 . Besides , given such a net- work trained with at least t0 steps , there will be a t1 > t0 such that 4τ ( t1 ) l τ ( δ̃ ( t1 ) ∈ Sh ) ≥ 4τ ( t1 ) h τ ( δ̃ ( t1 ) ∈ Sl ) and FGM perturbation will increase its ratio of LFC for all t > t1 no matter whether it is smaller than 1/2 . Finally , if τ ( δ̃ ( t2 ) ∈ Sl ) > τ ( δ̃ ( t2 ) ∈ Sh ) for some t2 > t0 , this relation will hold for any t ≥ t2 .

Summary: This submission deals with understanding the gradient based adversarial examples. For this means, it analyzes the adversarial examples in the frequency domain, where it identifies that the ratio of high-frequency and low frequency parts is quite large in adversarial examples compared with the natural ones. As a result, it proposes to apply a low-pass filtering to the data that can significantly improve the model robustness. Some experiments for CIFAR-10 classification are performed to support the main conclusions.