A frequency domain analysis of gradient-based adversarial examples

1 INTRODUCTION . Recently , deep neural networks ( DNN ) have achieved great success in the field of image processing , but it was found that DNNs are vulnerable to some synthetic data called adversarial examples ( ( Szegedy et al. , 2013 ) , ( Kurakin et al. , 2016 ) ) . Adversarial examples are natural samples plus adversarial perturbations , and the perturbations between natural samples and adversarial examples are imperceptible to human but able to fool the model . Typically , generating an adversarial example can be considered as finding an example in an -ball around a natural image that could be misclassified by the classifier . Recent studies designed Fast Gradient Sign Method ( FGSM , ( Goodfellow et al. , 2014 ) ) , Fast Gradient Method ( FGM , ( Miyato et al. , 2016 ) ) , Projected Gradient Descent ( PGD , ( Madry et al. , 2017 ) ) and other algorithms ( Carlini & Wagner ( 2017 ) , ( Su et al. , 2019 ) , ( Xiao et al. , 2018 ) , ( Kurakin et al. , 2016 ) , ( Chen et al. , 2017 ) ) to attack the model . Since the phenomenon of adversarial examples was discovered , many related works have made progress to study why they exist . Several works studied this phenomenon from the perspective of feature representation . Ilyas et al . ( 2019 ) divided the features into non-robust ones that are responsible for the model ’ s vulnerability to adversarial examples , and robust ones that are close to human perception . Further , they showed that adversarial vulnerability arises from non-robust features that are useful for correct classification . Another way to characterize adversarial examples is to investigate them in the frequency domain via Fourier transform . Wang et al . ( 2020a ) divided an image into low-frequency component ( LFC ) and high-frequency component ( HFC ) and they empirically showed that human can only perceive LFC , but convolutional neural networks can obtain useful information from both LFC and HFC . Yin et al . ( 2019 ) filters out the input data with low-pass or high-pass filters to study the sensitivity of the additive noise with different frequencies . Wang et al . ( 2020b ) claimed that existing adversarial attacks mainly concentrate in the high-frequency part . Sharma et al . ( 2019 ) found that when perturbations are constrained to the low-frequency subspace , they are generated faster and are more transferable , and will be effective to fool the defended models , but not for clean models . All of these works showed that spectrum in frequency domain is an reasonable way to study the adversarial examples . However , there is a lack of theoretical understanding about the dynamics of the adversarial perturbations in frequency domain along the training process of the model parameters . In this work , we focus on the frequency domain of the adversarial perturbations to explore the spectral properties of the adversarial examples generated by FGM ( Miyato et al. , 2016 ) and PGD ( Madry et al. , 2017 ) . In this work , we give a theoretical analysis in frequency domain of natural images for the adversarial examples . • For a two-layer neural network with non-linear activation function , we prove that adversarial perturbations by FGM and l2-PGD attacks gradually increase the concentration in the low-frequency part of the spectrum during the training process over model parameters . • Meanwhile , the log-spectrum difference of the adversarial examples ( the definition will be clarified in Section 2.2 ) will be more concentrated in the high-frequency part than the low-frequency part . • Furthermore , we show that the ratio of the high-frequency and the low-frequency part in the adversarial perturbation is much larger than that in the corresponding clean image . Empirically , • we design several experiments on the two-layer model and Resnet-32 with CIFAR-10 to verify the above findings . • Based on these phenomena , we filter out the high-frequency part of the adversarial examples before feeding them to the model to improve the robustness . Compared with the adversarially trained model with the same architecture , our method achieves comparable robustness with the similar computational cost of normal training and almost no loss of accuracy . The rest of the paper is organized as follows : In Section 2 , we present preliminaries and some theoretical analysis on the calculation of the spectrum and log-spectrum . Then we provide our main results about the frequency domain analysis of gradient-based adversarial examples in Section 3 . Furthermore , some experiments for supporting our theoretical findings are shown in Section 4 . Finally we conclude our work and conduct a discussion about the future work in Section 5 . All the details about the proof and experiments are shown in the Appendix . 2 BACKGROUND . 2.1 PRELIMINARIES . Notations We use { 0 , 1 , ... , d } to denote the set of all integers between 0 and d and use ‖ · ‖p to denote lp norm . Specifically , we denote by ‖ · ‖ the l2 norm . For a d-dimensional vector x , we use xµ to denote its µ-th component with index starting at 0 . For a scalar function f ( x ) : Rd 7→ R , ∇xf and ∂µf denote the gradient vector and its µ-th component . We let sgn ( x ) = 1 for x > 0 and −1 for x < 0 . Normal training refers to training on the original training data for learning the optimal weights for the neural networks . x̃ = F ( x ) denotes the Discrete Fourier Transform ( DFT ) of x. Discrete Fourier Transform The k-th frequency component g̃ [ k ] of one-dimensional DFT of a vector g is defined by g̃ [ k ] = F ( g ) [ k ] = d−1∑ µ=0 gµe ik 2πd µ , where k ∈ { −d/2 , −d/2 + 1 , ... , d/2 } if d is even and k ∈ { − ( d − 1 ) /2 , ... , ( d − 1 ) /2 } if d is odd . For convenience , we always consider an odd d and one can easily generalize the case to even dimensions . For an integer cut-off ( cutoff frequency ) kr ∈ ( 0 , ( d− 1 ) /2 ) , the Low Frequency Components ( LFC ) and High Frequency Components ( HFC ) of g̃ are denoted by g̃l [ k ] = { g̃ [ k ] if − kc ≤ k ≤ kc , 0 otherwise g̃h [ k ] = { 0 if − kc ≤ k ≤ kc , g̃ [ k ] otherwise . Then the frequency space Rd can be decomposed as Sl ⊕ Sh where g̃l ∈ Sl and g̃h ∈ Sh . LetNg̃ =√∑d/2 k=−d/2 |g̃ [ k ] |2 , then the ratio of a frequency component with respect to the whole frequency spectrum is denoted by τ ( g̃ [ k ] ) = |g̃ [ k ] |2 N2g̃ and the ratio of LFC is denoted by τ ( g̃ ∈ Sl ) = kc∑ k=−kc |g̃ [ k ] |2 N2g̃ . Setup In this paper , we consider a two-layer neural network f : Rd 7→ R f ( x , θ ) = m∑ r=1 arσ ( w T r x ) ( 1 ) where σ is an activation function , a = ( a1 , a2 , ... , am ) T is an m-dimensional vector and wr ’ s are d-dimensional weight vectors . Van der Schaaf & van Hateren ( 1996 ) showed that natural images obey a power law in the frequency domain . Therefore , to make our settings closer to widely studied image related tasks , we also assume our input x obey the power law |x̃ [ k ] | ∝ k−α to imitate LFC-concentrated images , where the constant α ≥ 1 and |x̃ [ 0 ] | = α0 is another constant . Besides , we choose the cut-off kc in such a way that kc∑ k=1 1 k2α + α20 > ( d−1 ) /2∑ k=kc+1 1 k2α , ( 2 ) which indicates that τ ( x̃ ∈ Sl ) > τ ( x̃ ∈ Sh ) and can be easily satisfied for a small kc . Gradient-based adversarial attack For a loss function ` , we find lp-norm -bounded adversarial perturbations by solving the optimization problem max ‖δ‖p≤ ` ( f ( x+ δ ) , y ) , where ( x , y ) ∈ Rd × R is d-dimensional input and output following the joint distribution D. To solve the above optimization problem , Goodfellow et al . ( 2014 ) proposed FGSM , an attack for an l∞-bounded adversary , and computed an adversarial example as xadv = x + sgn ( ∇x ` ( θ , x , y ) ) . Miyato et al . ( 2016 ) proposed FGM with an l2-bounded adversary , xadv = x+ ∇x ` ( x , y ) ‖∇x ` ( x , y ) ‖2 . Madry et al . ( 2017 ) introduced PGD which training adversarial examples in an iterative way that xt+1 = Πx+S ( x t + α sgn ( ∇x ` ( θ , x , y ) ) ) or xt+1 = Πx+S ( xt + α ∇x ` ( x , y ) ‖∇x ` ( x , y ) ‖2 ) To simplify the analysis , we make the following assumptions on the normal training process : Assumption 1 There exist β′ , β > 0 such that β′ ≤ |∂ ` /∂f | ≤ β . Assumption 2 If ` = 12 ( y − f ( x , θ ) ) 2 , |∂ ` /∂f | should be a small quantity such that |∂ ` /∂f | = 1+v if the network is well trained in the end stage of normal training , where 0 < v < 1 . Assumption 3 There exist λ , γ > 0 such that 0 ≤ σ′ ≤ λ and 0 ≤ σ′′ ≤ γ1 . Assumption 4 ar ’ s are i.i.d drawn from a distribution2 and will not be updated during the training . 1e.g . λ = 1 and γ = 1 4 if σ = ln ( 1 + ex ) . 2e.g . Bernoulli distribution . 2.2 EXPLORING THE LOG-SPECTRUM AND SPECTRUM . Wang et al . ( 2020a ) considered the log-spectrum of an image : 20 ∗ log ( |F ( x ) | ) , where x is a 2ddimensional matrix representing an image . One way to calculate the ” log-spectrum difference of the adversarial examples ” is to consider the difference between the log-spectrum of the adversarial and natural image and as follows , 20 ∗ |log ( |F ( x+ δ ) | ) − log ( |F ( x ) | ) | ≈ 20 ∗ ∣∣∣∣log ( 1 + |F ( δ ) ||F ( x ) | ) ∣∣∣∣ ≈ 20 ∗ |F ( δ ) ||F ( x ) | . ( 3 ) We can observe that this quantity is approximately proportional to the ratio of the perturbation ’ s over the natural image ’ s frequency . Moreover , the average Relative Change in discrete cosine Transforms ( RCT ) in ( Wang et al. , 2020b ) ) approximately equals to it . Fig . 5 empirically shows that this ratio is large for high frequency component but small for low frequency one . However , Van der Schaaf & van Hateren ( 1996 ) claimed that the LFC of F ( x ) is much bigger than its HFC , and the right side of Eq . 3 is inversely proportional to F ( x ) . The denominator of the right side of Eq . ( 3 ) in low-frequency part is much larger than it in high frequency part , so it may be imprudent to consider the frequency distribution of the adversarial perturbations only based on the ratio in the right side of Eq. ( 3 ) . Yin et al . ( 2019 ) provided another spectral measurement for adversarial examples . They considered the ” spectrum of the adversarial perturbation ” |F ( δ ) | , which directly reflects the distribution of adversarial perturbations in the frequency domain . Unfortunately , they claimed that ‘ The adversarial perturbations for the normally trained model are uniformly distributed across frequency components ’ without sufficient empirical and theoretical justification . In this work , we provide a totally different theoretical finding that the adversarial perturbations for the normally trained model are gradually concentrated in the low-frequency domain , further verified by extensive experiments . Fig . 1 visually shows the calculation methods of the spectrum of perturbations F ( δ ) and the logspectrum difference of adversarial examples | log ( |F ( x+ δ ) | ) − log ( |F ( x ) | ≈ |F ( δ ) ||F ( x ) | . 3 MAIN RESULTS In Section 3.1 , we show that , for a well trained two-layer neural network described in Eq . ( 1 ) with LFC-concentrated inputs , l2norm adversarial perturbations generated by FGM ( Theorem 1 ) prefer to concentrate in the low frequency domain . Section 3.2 theoretically proves the HFC-concentration of the logspectrum difference of adversarial examples . Section 3.3 attempts to demonstrate the effectiveness of masking the HFC of the adversarial examples to improve model ’ s robustness despite that perturbations can concentrate in the low frequency domain . All technical proofs are deferred to Appendix A . Results on l2 PGD perturbations are presented in Appendix B . In this work , we consider the two-layer neural networks and the loss function ` = 1 2 ( y − f ( x , θ ) ) 2 . For any x in this setting , we have ∂f ∂ar = σ ( wTr x ) , ∂f ∂wr = a′rx , ∇xf = ∑ r a′rwr , where a′r = arσ ′ ( wTr x ) and a ′ = ( a′1 , a ′ 2 , ... , a ′ m ) . At the ( t+ 1 ) -th step of gradient descent in the normal training process of the classifier , weight w is updated as w ( t+1 ) r = w ( t ) r − η ∂ ` ∂f ( t ) a′ ( t ) r x . ( 4 ) We use Lx and Hx to denote the LFC and HFC of the clean input Lx = kc∑ k=0 |x̃ [ k ] |2 and Hx = ( d−1 ) /2∑ k=kc+1 |x̃ [ k ] |2 . ( 5 ) 3.1 SPECTRAL TRAJECTORIES OF l2-NORM FGM PERTURBATIONS Consider the l2-norm FGM perturbation δ and its DFT δ̃ [ k ] δ = ∂ ` ∂f ∇xf ‖ ∂ ` ∂f ∇xf‖ , δ̃ [ k ] = sgn ( ∂ ` ∂f ) 1 ‖∇xf‖ ∇̃f [ k ] , ( 6 ) where ∇̃f [ k ] = F ( ∇xf ) [ k ] . Since the coefficient before ∇̃f [ k ] in ( 6 ) is same for different frequencies k , we only need to analyze the spectrum of ∇̃f [ k ] to investigate the ratio of certain frequency component over the whole frequency spectrum , i.e . τ ( δ̃ [ k ] ) = τ ( ∇̃f [ k ] ) . ( 7 ) In this subsection , we explore the evolving of τ ( δ̃ [ k ] ) along the normal training process . For a randomly initialized network , the FGM-attack perturbation will not have bias towards either HFC or LFC . We attack this model with FGM at each step of the normal training to study the trajectory of these perturbations in the frequency domain . According to Eq . ( 4 ) , ∇xf is updated to the order of η at the ( t+ 1 ) -th step as ∇xf ( t+1 ) = m∑ r=1 ( 1− η ( t ) r ) a′ ( t ) r w ( t ) r − η̃ ( t ) x+O ( η2 ) , where η ( t ) r = ηar‖x‖2 ∂ ` ∂f ( t ) σ′′ ( t ) ( w ( t ) T r x ) and η̃ ( t ) = η ∂ ` ∂f ( t ) ‖a′ ( t ) ‖2 are composite learning rates for easing our notation . For ReLU activation function σ ( x ) = max ( 0 , x ) , we have that η ( t ) r = Ei∈ { 1,2 , ... , m } [ η ( t ) i ] , therefore , considering that Softplus has similar shape as ReLU , we derive that η ( t ) r = η̄ ( t ) + O ( η2 ) , where η̄ ( t ) = η‖x‖2 ∂ ` ∂f ( t ) Er∈ { 1,2 , ... , m } [ arσ ′′ ( t ) ( w ( t ) T r x ) ] . In this way , the 1d-DFT of ∇xf ( t+1 ) has the following form , ∇̃f ( t+1 ) [ k ] = ( 1− η̄ ( t ) ) ∇̃f ( t ) [ k ] − η̃ ( t ) x̃ [ k ] . ( 8 ) We are now ready to study the trajectory of τ ( δ̃ [ k ] ) along the training step t. Let 4ϕ ( t ) k denote the difference of phases between ∇̃f ( t ) [ k ] and x̃ [ k ] for frequency k at the t-th step . And for ease of notation , denote 4τ ( t ) l , −2 kc∑ k=0 |∇̃f ( t ) [ k ] | |x̃ [ k ] | cos ( 4ϕ ( t ) k ) , 4τ ( t ) h , −2 ( d−1 ) /2∑ k=kc+1 |∇̃f ( t ) [ k ] | |x̃ [ k ] | cos ( 4ϕ ( t ) k ) , where η̃ ( t ) 4τ ( t ) l and η̃ ( t ) 4τ ( t ) h are approximately changed amounts for LFC and HFC of |∇̃f ( t ) |2 at the t-th step of the normal training of the network , respectively . Since the network is randomly initialized and the clean inputs x are highly concentrated in the low frequency domain , we always study the case where τ ( δ̃ ( 0 ) ∈ Sl ) < τ ( x̃ ∈ Sl ) . We now , taking Eq . ( 7 ) into consideration , present our main theorem about l2 norm FGM perturbations as follows . Theorem 1 ( The spectral trajectory of l2 FGM perturbation ) During the training process of the two-layer neural network f ( x ) in ( 1 ) , the l2 norm FGM adversarial perturbation will change its ratio of LFC , τ ( δ̃ ∈ Sl ) , at the ( t+ 1 ) -th step as follows , τ ( δ̃ ( t+1 ) ∈ Sl ) = τ ( δ̃ ( t ) ∈ Sl ) + η̃ ( t ) 4τ ( t ) l τ ( δ̃ ( t ) ∈ Sh ) −4τ ( t ) h τ ( δ̃ ( t ) ∈ Sl ) ∑ ( d−1 ) /2 k=0 |∇̃f ( t+1 ) [ k ] |2 ; ( 9 ) there will be a t1 s.t . ∀t ≥ t1 , we have τ ( δ̃ ( t+1 ) ∈ Sl ) > τ ( δ̃ ( t ) ∈ Sl ) . ( 10 ) Remark According to theorem 1 , during the normal training process for a randomly initialized network with LFC-concentrated input , there will be a t0 s.t . ∀t ≥ t0 , we have 4τ ( t ) l > 4τ ( t ) h , and τ ( δ̃ ( t+1 ) ∈ Sl ) > τ ( δ̃ ( t ) ∈ Sl ) if τ ( δ̃ ( t ) ∈ Sl ) < τ ( δ̃ ( t ) ∈ Sh ) . For ReLU activa- tion function , starting with τ ( δ̃ ( 0 ) ∈ Sl ) = 1/2 − ζ and 4τ ( 0 ) l = 4τ ( 0 ) h + bζ , we have t0 = max { 0 , − bζnηβ′ ( Lx−Hx ) } , where n = mint∈ [ 0 , t0 ] ‖a′ ( t ) ‖2 . Besides , given such a net- work trained with at least t0 steps , there will be a t1 > t0 such that 4τ ( t1 ) l τ ( δ̃ ( t1 ) ∈ Sh ) ≥ 4τ ( t1 ) h τ ( δ̃ ( t1 ) ∈ Sl ) and FGM perturbation will increase its ratio of LFC for all t > t1 no matter whether it is smaller than 1/2 . Finally , if τ ( δ̃ ( t2 ) ∈ Sl ) > τ ( δ̃ ( t2 ) ∈ Sh ) for some t2 > t0 , this relation will hold for any t ≥ t2 .

Summary: This paper analyzes the frequency spectrum of adversarial perturbations during normal training. The authors show that the low frequency component (LFC) of adversarial perturbation is increasing during training, but it is not increasing fast enough, so the LFC of adversarial perturbation is not as dense as the input natural images which obey the power law. Therefore, the log-spectrum difference of adversarial examples will express a HFC-concentrated phenomenon. The authors used theory and experiments to demonstrate this point.