Smooth Kernels Improve Adversarial Robustness and Perceptually-Aligned Gradients

Recent research has shown that CNNs are often overly sensitive to high-frequency textural patterns . Inspired by the intuition that humans tend to be more sensitive to lower-frequency ( larger-scale ) patterns , we design a regularization scheme that penalizes large differences between adjacent components within each convolutional kernel . We apply our regularization onto several popular training methods , demonstrating that the models with the proposed smooth kernels enjoy improved adversarial robustness . Further , building on recent work establishing connections between adversarial robustness and interpretability , we show that our method appears to give more perceptually-aligned gradients . 1 INTRODUCTION . In recent years , deep learning models have demonstrated remarkable capabilities for predictive modeling in computer vision , leading some to liken their abilities on perception tasks to those of humans ( e.g. , Weyand et al. , 2016 ) . However , under closer inspection , the limits of such claims to the narrow scope of i.i.d . data become clear . For example , when faced with adversarial examples ( Szegedy et al. , 2013 ; Goodfellow et al. , 2015 ) or even in non-adversarial domain-agnostic cross-domain evaluations ( Wang et al. , 2019a ; b ; Carlucci et al. , 2019 ) , performance collapses , dispelling claims of human-like perceptive capabilities and calling into doubt more ambitious applications of this technology in the wild . A long line of recent research has investigated the robustness of neural networks , including investigations of the high-dimension nature of models ( Fawzi et al. , 2018 ) , enlarging the gaps between decision boundaries ( Zhang et al. , 2019a ) , training the models with augmented examples through attack methods ( Madry et al. , 2018 ) , and even guaranteeing the robustness of models within given radii of perturbation ( Wong & Kolter , 2018 ; Cohen et al. , 2019 ) . Compared to earlier methods , these recent works enjoy stronger robustness both as assessed via theoretical guarantees and empirically via quantitative performance against strong attacks . However , despite the success of these techniques , vulnerabilities to new varieties of attacks are frequently discovered ( Zhang et al. , 2019b ) . In this paper , we aim to lessen the dependency of neural networks on high-frequency patterns in images , regularizing CNNs to focus on the low-frequency components . Therefore , the main argument of this paper is that : by regularizing the CNN to be most sensitive to the low-frequency components of an image , we can improve the robustness of models . Interestingly , this also appears to lead to more perceptually-aligned gradients . Further , as Wang et al . ( 2019c ) explicitly defined the low ( or high ) -frequency components as images reconstructed from the low ( or high ) -end of the image frequency domain ( as is frequently discussed in neuroscience literature addressing human recognition of shape ( Bar , 2004 ) or face ( Awasthi et al. , 2011 ) ) , we continue with this definition and demonstrate that a smooth kernel can filter out the high-frequency components and improve the models ’ robustness . We test our ideas and show the empirical improvement over popular adversarial robust methods with standard evaluations and further use model interpretation methods to understand how the models make decisions and demonstrate that the regularization helps the model to generate more perceptually-aligned gradients . 2 RELATED WORK . Adversarial examples are samples with small perturbations applied that are imperceptible to humans but can nevertheless induce misclassification in machine learning models ( Szegedy et al. , 2013 ) ) . The discovery of adversarial examples spurred a torrent of research , much of it consisting of an arm race between those inventing new attack methods and others offering defenses to make classifiers robust to these sorts of attacks . We refer to survey papers such as ( Akhtar & Mian , 2018 ; Chakraborty et al. , 2018 ) and only list a few most relevant works about applying regularizations to the networks to improve the adversarial robustness , such as regularizations constraining the Lipschitz constant of the network ( Cisse et al. , 2017 ) ( Lipschitz smoothness ) , regularizing the scale of gradients ( Ross & Doshi-Velez , 2018 ; Jakubovitz & Giryes , 2018 ) ( smooth gradients ) , regularizing the curvature of the loss surface ( Moosavi-Dezfooli et al. , 2019 ) ( smooth loss curvature ) , and promoting the smoothness of the model distribution ( Miyato et al. , 2015 ) . These regularizations also use the concept of “ smoothness , ” but different from ours ( small differences among the adjacent weights ) . Recently , adversarial training ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ) has become one of the most popular defense methods , based on the simple idea of augmenting the training data with samples generated through attack methods ( i.e. , threat models ) . While adversarial training excels across many evaluations , recent evidence exposes its new limitations ( Zhang et al. , 2019b ) , suggesting that adversarial robustness remains a challenge . Key differences : In this paper , we present a new technique penalizing differences among the adjacent components of convolutional kernels . Moreover , we expand upon the recent literature demonstrating connections between adversarial robustness and perceptually-aligned gradients . 3 SMOOTH KERNEL REGULARIZATION . Intuition . High-frequency components of images are those reconstructed from the high-end of the image frequency-domain through inverse Fourier transform . This definition was also verified previously by neuroscientists who demonstrated that humans tend to rely on the low-frequency component of images to recognize shapes ( Bar , 2004 ) and faces ( Awasthi et al. , 2011 ) . Therefore , we argue that the smooth kernel regularization is effective because it helps to produce models less sensitive to high-frequency patterns in images . We define a smooth kernel as a convolutional kernel whose weight at each position does not differ much from those of its neighbors , i.e. , ( wi , j −wh , k∈N ( i , j ) ) 2 is a small number , where w denotes the convolutional kernel weight , i , j denote the indices of the convolutional kernel w , and N ( i , j ) denotes the set of the spatial neighbors of i , j . We note two points that support our intuition . 1 . The frequency domain of a smooth kernel has only negligible high-frequency components . This argument can be shown with Theorem 1 in ( Platonov , 2005 ) . Roughly , the idea is to view the weight matrix w as a function that maps the index of weights to the weights : w ( i , j ) → wi , j , then a smooth kernel can be seen as a Lipschitz function with constant α . As pointed out by Platonov ( 2005 ) , Titchmarsh ( 1948 ) showed that when 0 < α < 1 , in the frequency domain , the sum of all the high frequency components with a radius greater than r will converge to a small number , suggesting that the high-frequency components ( when r is large ) are negligible . 2 . The kernel with negligible high-frequency components will weigh the high-frequency components of input images accordingly . This argument can be shown through Convolution Theorem ( Bracewell , 1986 ) , which states w~x = F−1 ( F ( w ) F ( x ) ) , where F ( · ) stands for Fourier transform , ~ stands for convolution operation , and stands for point-wise multiplication . As the theorem states , the convolution operation of images is equivalent to the element-wise multiplication of image frequency domain . Therefore , roughly , if w has negligible high-frequency components in the frequency domain , it will weigh the high-frequency components of x accordingly with negligible weights . Naturally , this argument only pertains to a single convolution , and we rely on our intuition that repeated applications of these smooth kernels across multiple convolution layers in a nonlinear deep network will have some cumulative benefit . Formally , we calculate our regularization term R0 ( w ) as follows : R0 ( w ) = ∑ i , j ∑ h , k∈N ( i , j ) ( wi , j −wh , k ) 2 , We also aim to improve this regularization by trying a few additional heuristics : • First , we notice that directly appending R0 ( w ) will sometimes lead to models that achieve the a small value of R0 ( w ) by directly scaling down the every coefficient of w proportionally , without changing the fluctuation pattern of the weights . To fix this problem , we directly subtract the scale of w ( i.e. , ∑ i , j w 2 i , j ) after R0 ( w ) . • Another heuristic to fix this same problem is to directly divide R0 ( w ) by the scale of w. Empirically , we do not observe significant differences between these two heuristics . We settle with the first heuristic because of the difficulty in calculating gradient when a matrix is the denominator . • Finally , we empirically observe that the regularization above will play a significant role during the early stage of training , but may damage the overall performances later when the regularization pulls towards smoothness too much . To mitigate this problem , we use an exponential function to strengthen the effects of the regularization when the value is big and to weaken it when the value is small . Overall , our final regularization is : R ( w ) = exp ∑ i , j ∑ h , k∈N ( i , j ) ( wi , j −wh , k ) 2 − ∑ i , j w2i , j In practice , the convolutional kernel is usually a 4-dimensional tensor , while our method only encourages smoothness over the two spatial dimensions corresponding to the 2D images . Thus , we only regularize through these two dimensions broadcasting the operation through the channels . Because a repeated calculation of each kernel component ’ s distance with its neighbors will double count some pairs , our implementation instead enumerates over all pairs of neighbors , counting each squared difference only once towards the total penalty . We can directly append the regularization λR ( w ) to most loss functions , where λ is a tuning hyperparameter . In the following experiments , we append λR ( w ) to the vanilla loss function ( crossentropy loss ) , Trades loss ( Zhang et al. , 2019a ) , adversarial training loss ( Madry et al. , 2018 ) , and a variation of logit pairing loss ( Kannan et al. , 2018 ) , as introduced in the following paragraphs . Adversarial training works by fitting the model using adversarial examples generated on the fly at train time by the threat model . Trades loss fits the model with clean examples while regularizing the softmax of augmented adversarial examples to be close to that produced for corresponding clean examples , a natural alternative is to fit the model with augmented adversarial examples while regularizing the softmax of clean examples to be close to that of the corresponding adversarial examples , which is related to logit pairing . However , to make the comparison consistent , we use a variation of logit pairing , penalizing the KL divergence of softmax ( rather than ` 2 distance over logits ) , following the Trades loss , which also uses KL divergence over softmax as the distance metric . To be specific , with the standard notations such as 〈X , Y〉 denoting a data set , 〈x , y〉 denoting a sample , the logit pairing loss is formalized as : minE〈x , y〉∼〈X , Y〉l ( f ( x′ ; θ ) ; y ) + γk ( fl ( x′ ; θ ) , fl ( x ; θ ) ) where x′ = argmax d ( x′ , x ) ≤ l ( f ( x′ ; θ ) ; y ) where d ( · , · ) and k ( · , · ) are distance functions , fl ( · ; · ) denotes the model f ( · ; · ) but outputs the softmax instead of a prediction , l ( · , · ) is a cost function , γ is a tuning hyperparameter , and is the upper bound of perturbation . In our following experiments , we consider d ( · , · ) as ` ∞ norm following popular adversarial training set-up and k ( · , · ) as KL divergence following standard Trades loss . Intuitively , our usage of KL divergence in logit pairing loss is argued to be advantageous because Pinsker ’ s inequality suggests that KL divergence upper-bounds the total variation ( TV ) distance ( e.g. , Csiszar & Körner , 2011 ) , the usage of KL divergence can be seen as a regularization that limits the hypothesis space to the parameters that yield small TV distance over perturbations of samples , which is linked to the robustness of an estimator , a topic that has been studied by the statistics community for over decades ( e.g. , see ( Diakonikolas et al. , 2019 ) and references within ) .

Summary: The authors propose a method for learning smoother convolutional kernels with the goal of improving robustness and human alignment. Specifically, they propose a regularizer penalizing large changes between consecutive pixels of the kernel with the intuition of penalizing the use of high-frequency input components. They evaluate the impact of their method on the adversarial robustness of various models and class visualization methods.