Extreme Values are Accurate and Robust in Deep Networks

1 INTRODUCTION . Convolutional neural networks ( CNNs ) evolve very fast ever since AlexNet ( Krizhevsky & Hinton , 2012 ) makes a great breakthrough on ImageNet image classification challenge ( Deng et al. , 2009 ) in 2012 . Various network architectures have been proposed to further boost classification performance since then , including VGGNet ( Simonyan & Zisserman , 2015 ) , GoogleNet ( Szegedy et al. , 2015 ) , ResNet ( He et al. , 2016 ) , DenseNet ( Huang et al. , 2017 ) and SENet ( Hu et al. , 2018 ) , etc . Recently , people even introduce network architecture search to automatically learn better network architectures ( Zoph & Le , 2017 ; Liu et al. , 2018 ) . However , state-of-the-art CNNs are challenged by their robustness , especially vulnerability to adversarial attacks based on small , human-imperceptible modifications of the input ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) . Su et al . ( 2018 ) thoroughly study the robustness of 18 well-known ImageNet models using multiple metrics , and reveals that adversarial examples are widely existent . Many methods are proposed to improve network robustness , which can be roughly categorized into three perspectives : ( 1 ) modifying input or intermediate features by transformation ( Guo et al. , 2018 ) , denoising ( Liao et al. , 2018 ; Jia et al. , 2019 ) , generative models ( Samangouei et al. , 2018 ; Song et al. , 2018 ) ; ( 2 ) modifying training by changing loss functions ( Wong & Kolter , 2018 ; Elsayed et al. , 2018 ; Zhang et al. , 2019 ) , network distillation ( Papernot et al. , 2016 ) , or adversarial training ( Goodfellow et al. , 2015 ; Tramer et al. , 2018 ) ( 3 ) designing robust network architectures ( Xie et al. , 2019 ; Svoboda et al. , 2019 ; Nayebi & Ganguli , 2017 ) and possible combinations of these basic categories . For more details of current status , please refer to a recent survey ( Akhtar & Mian , 2018 ) . Although it is known that adversarial examples are widely existent ( Su et al. , 2018 ) , some fundamental questions are still far from being well studied like what causes it , and how the factor impacts the performance , etc . One of the interesting findings in ( Su et al. , 2018 ) is that model architecture is a more critical factor to network robustness than model size ( e.g . number of layers ) . Some recent works start to explore much deeper nature . For instance , both ( Geirhos et al. , 2019 ; Baker et al. , 2018 ) show that CNNs are trained to be strongly biased towards textures so that CNNs do not distinguish objects contours from other local or even noise edges , thus perform poorly on shape dominating object instances . On the contrary , there are no statistical difference for human behaviors on both texture rich objects and global shape dominating objects in psychophysical trials . Ilyas et al . ( 2019 ) further analyze and show that deep convolutional features can be categorized into robust and non-robust features , while non-robust features may even account for good generalization . However , non-robust features are not expected to have good model interpretability . It is thus an interesting topic to disentangle robust and non-robust features with certain kinds of human priors in the network designing or training process . In fact , human priors have been extensively used in handcraft designed robust visual features like SIFT ( Lowe , 2004 ) . SIFT detects scale-space ( Lindeberg , 1994 ) extrema from input images , and selects stable extrema to build robust descriptors with refined location and orientation , which achieves great success for many matching and recognition based vision tasks before CNN being reborn in 2012 ( Krizhevsky & Hinton , 2012 ) . The scale-space extrema are efficiently implemented by using a difference-of-Gaussian ( DoG ) function to search over all scales and image locations , while the DoG operator is believed to biologically mimic the neural processing in the retina of the eye ( Young , 1987 ) . Unfortunately , there is ( at least explicitly ) no such scale-space extrema operations in all existing CNNs . Our motivation is to study the possibility of leveraging good properties of SIFT to renovate CNN networks architectures towards better accuracy and robustness . In this paper , we borrow the scale-space extrema idea from SIFT , and propose extreme value preserving networks ( EVPNet ) to separate robust features from non-robust ones , with three novel architecture components to model the extreme values : ( 1 ) parametric DoG ( pDoG ) to extract extreme values in scale-space for deep networks , ( 2 ) truncated ReLU ( tReLU ) to suppress noise or non-stable extrema and ( 3 ) projected normalization layer ( PNL ) to mimic PCA-SIFT ( Ke et al. , 2004 ) like feature normalization . pDoG and tReLU are combined into one block named EVPConv , which could be used to replace all k × k ( k > 1 ) conv-layers in existing CNNs . We conduct comprehensive experiments and ablation studies to verify the effectiveness of each component and the proposed EVPNet . Figure 1 illustrates a comparison of responses for standard convolution + ReLU and EVPConv in ResNet-50 trained on ImageNet , and shows that the proposed EVPConv produces less noises and more responses around object boundary than standard convolution + ReLU , which demonstrates the capability of EVPConv to separate robust features from non-robust ones . Our major contribution are : • To the best of our knowledge , we are the first to explicitly separate robust features from non-robust ones in deep neural networks from an architecture design perspective . • We propose three novel network architecture components to model extreme values in deep networks , including parametric DoG , truncated ReLU , and projected normalization layer , and verify their effectiveness through comprehensive ablation studies . • We propose extreme value preserving networks ( EVPNets ) to combine those three novel com- ponents , which are demonstrated to be not only more accurate , but also more robust to a set of adversarial attacks ( FGSM , PGD , etc ) even for clean model without adversarial training . 2 RELATED WORK . Robust visual features . Most traditional robust visual feature algorithms like SIFT ( Lowe , 2004 ) and SURF ( Bay et al. , 2006 ) are based on the scale-space theory ( Lindeberg , 1994 ) , while there is a close link between scale-space theory and biological vision ( Lowe , 2004 ) , since many scalespace operations show a high degree of similarity with receptive field profiles recorded from the mammalian retina and the first stages in the visual cortex . For instance , DoG computes the difference of two Gaussian blurred images and is believed to mimic the neural processing in the retina ( Young , 1987 ) . SIFT is one such kind of typical robust visual features , which consists of 4 major stages : ( 1 ) scale-space extrema detection with DoG operations ; ( 2 ) Keypoints localization by their stability ; ( 3 ) Orientation and scale assignment based on primary local gradient direction ; ( 4 ) Histogram based keypoint description . We borrow the scale-space extrema idea from SIFT , and propose three novel and robust architecture components to mimic key stages of SIFT . Robust Network Architectures . Many research efforts have been devoted to network robustness especially on defending against adversarial attacks as summarized in Akhtar & Mian ( 2018 ) . However , there are very limited works that tackle this problem from a network architecture design perspective . A major category of methods ( Liao et al. , 2018 ; Xie et al. , 2019 ) focus on designing new layers to perform denoising operations on the input image or the intermediate feature maps . Most of them are shown effective on black-box attacks , while are still vulnerable to white-box attacks . Non-local denoising layer proposed in Xie et al . ( 2019 ) is shown to improve robustness to white-box attack to an extent with adversarial training ( Madry et al. , 2018 ) . Peer sample information is introduced in Svoboda et al . ( 2019 ) with a graph convolution layer to improve network robustness . Biologically inspired protection ( Nayebi & Ganguli , 2017 ) introduces highly non-linear saturated activation layer to replace ReLU layer , and demonstrates good robustness to adversarial attacks , while similar higher-order principal is also used in Krotov & Hopfield ( 2018 ) . However , these methods still lack a systematic architecture design guidance , and many ( Svoboda et al. , 2019 ; Nayebi & Ganguli , 2017 ) are not robust to iterative attack methods like PGD under clean model setting . In this work , inspired by robust visual feature SIFT , we are able to design a series of innovative architecture components systematically for improving both model accuracy and robustness . We should stress that extreme value theory is a different concept to scale-space extremes , which tries to model the extreme in data distribution , and is used to design an attack-independent metric to measure robustness of DNNs ( Weng et al. , 2018 ) by exploring input data distribution . 3 PRELIMINARY . Difference-of-Gaussian . Given an input image I and Gaussian kernel G ( x , y , σ ) as below G ( x , y , σ ) = 1 2πσ2 e− ( x 2+y2 ) /2σ2 , ( 1 ) where σ denotes the variance . Also , difference of Gaussian ( DoG ) is defined as D ( x , y , σ ) = G ( x , y , σ ) ⊗ I1 −G ( x , y , σ ) ⊗ I0 , ( 2 ) where ⊗ is the convolution operation , and I1 = G ( x , y , σ ) ⊗ I0 . Scale-space DoG repeatedly convolves input images with the same Gaussian kernels , and produces difference-of-Gaussian images by subtracting adjacent image scales . Scale-space extrema ( maxima and minima ) are detected in DoG images by comparing a pixel to its 26 neighbors in 3×3 grids at current and two adjacent scales ( Lowe , 2004 ) . Adversarial Attacks . We use h ( · ) to denote the softmax output of classification networks , and hc ( · ) to denote the prediction probability of class c. Then given a classifier h ( x ) = y , the goal of adversarial attack is to find xadv such that the output of classifier deviates from the true label y : maxi h i ( xadv ) 6= y while closing to the original input : ||x− xadv|| ≤ . Here || · || refers to a norm operator , i.e . L2 or L∞ . Attack Method . The most simple adversarial attack method is Fast Gradient Sign Method ( FGSM ) ( Goodfellow et al. , 2015 ) , a single-step method which takes the sign of the gradient on the input as the direction of the perturbation . L ( · , · ) denotes the loss function defined by cross entropy . Specifically , the formation is as follows : xadv = x+ · sign ( ∇xL ( h ( x ) , y ) ) , ( 3 ) where x is the clean input , y is the label . is the norm bound ( ||x − xadv|| ≤ , i.e . -ball ) of the adversarial perturbation . Projected gradient descent ( PGD ) iteratively applies FGSM with a small step size αi ( Kurakin et al. , 2017a ; Madry et al. , 2018 ) with formulation as below : xadvi+1 = Proj ( x+ α · sign ( ∇xL ( h ( x adv i ) , y ) ) ) , ( 4 ) where i is the iteration number , α = /T with T being the number of iterations . ‘ Proj ’ is the function to project the image back to -ball every step . Some advanced and complex attacks are further introduced in DeepFool ( Moosavi-Dezfooli et al. , 2016 ) , CW ( Carlini & Wagner , 2017 ) , MI-FGSM ( Dong et al. , 2018 ) . Adversarial Training aims to inject adversarial examples into training procedure so that the trained networks can learn to classify adversarial examples correctly . Specifically , adversarial training solves the following empirical risk minimization problem : arg min h∈H E ( x , y ) D [ max x∗∈A ( x ) L ( h ( x∗ ) , y ) ] , ( 5 ) where A ( x ) denotes the area around x bounded by L∞/L2 norm , and H is the hypothesis space . In this work , we employ both FGSM and PGD to generate adversarial examples for adversarial training .

Summary: This paper presents a SIFT-feature inspired modification to the standard convolutional neural network (CNN). Specifically the authors propose three innovations: (1) a differences of Gaussians (DoG) convolutional filter; (2) a symmetric ReLU activation function (referred to as a truncated ReLU; and (3) a projected normalization layer. The paper makes the claim that the proposed CNN variant (referred to as the EVPNet) demonstrates superior performance as well as improved robustness to adversarial attacks.