Extreme Values are Accurate and Robust in Deep Networks

1 INTRODUCTION . Convolutional neural networks ( CNNs ) evolve very fast ever since AlexNet ( Krizhevsky & Hinton , 2012 ) makes a great breakthrough on ImageNet image classification challenge ( Deng et al. , 2009 ) in 2012 . Various network architectures have been proposed to further boost classification performance since then , including VGGNet ( Simonyan & Zisserman , 2015 ) , GoogleNet ( Szegedy et al. , 2015 ) , ResNet ( He et al. , 2016 ) , DenseNet ( Huang et al. , 2017 ) and SENet ( Hu et al. , 2018 ) , etc . Recently , people even introduce network architecture search to automatically learn better network architectures ( Zoph & Le , 2017 ; Liu et al. , 2018 ) . However , state-of-the-art CNNs are challenged by their robustness , especially vulnerability to adversarial attacks based on small , human-imperceptible modifications of the input ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) . Su et al . ( 2018 ) thoroughly study the robustness of 18 well-known ImageNet models using multiple metrics , and reveals that adversarial examples are widely existent . Many methods are proposed to improve network robustness , which can be roughly categorized into three perspectives : ( 1 ) modifying input or intermediate features by transformation ( Guo et al. , 2018 ) , denoising ( Liao et al. , 2018 ; Jia et al. , 2019 ) , generative models ( Samangouei et al. , 2018 ; Song et al. , 2018 ) ; ( 2 ) modifying training by changing loss functions ( Wong & Kolter , 2018 ; Elsayed et al. , 2018 ; Zhang et al. , 2019 ) , network distillation ( Papernot et al. , 2016 ) , or adversarial training ( Goodfellow et al. , 2015 ; Tramer et al. , 2018 ) ( 3 ) designing robust network architectures ( Xie et al. , 2019 ; Svoboda et al. , 2019 ; Nayebi & Ganguli , 2017 ) and possible combinations of these basic categories . For more details of current status , please refer to a recent survey ( Akhtar & Mian , 2018 ) . Although it is known that adversarial examples are widely existent ( Su et al. , 2018 ) , some fundamental questions are still far from being well studied like what causes it , and how the factor impacts the performance , etc . One of the interesting findings in ( Su et al. , 2018 ) is that model architecture is a more critical factor to network robustness than model size ( e.g . number of layers ) . Some recent works start to explore much deeper nature . For instance , both ( Geirhos et al. , 2019 ; Baker et al. , 2018 ) show that CNNs are trained to be strongly biased towards textures so that CNNs do not distinguish objects contours from other local or even noise edges , thus perform poorly on shape dominating object instances . On the contrary , there are no statistical difference for human behaviors on both texture rich objects and global shape dominating objects in psychophysical trials . Ilyas et al . ( 2019 ) further analyze and show that deep convolutional features can be categorized into robust and non-robust features , while non-robust features may even account for good generalization . However , non-robust features are not expected to have good model interpretability . It is thus an interesting topic to disentangle robust and non-robust features with certain kinds of human priors in the network designing or training process . In fact , human priors have been extensively used in handcraft designed robust visual features like SIFT ( Lowe , 2004 ) . SIFT detects scale-space ( Lindeberg , 1994 ) extrema from input images , and selects stable extrema to build robust descriptors with refined location and orientation , which achieves great success for many matching and recognition based vision tasks before CNN being reborn in 2012 ( Krizhevsky & Hinton , 2012 ) . The scale-space extrema are efficiently implemented by using a difference-of-Gaussian ( DoG ) function to search over all scales and image locations , while the DoG operator is believed to biologically mimic the neural processing in the retina of the eye ( Young , 1987 ) . Unfortunately , there is ( at least explicitly ) no such scale-space extrema operations in all existing CNNs . Our motivation is to study the possibility of leveraging good properties of SIFT to renovate CNN networks architectures towards better accuracy and robustness . In this paper , we borrow the scale-space extrema idea from SIFT , and propose extreme value preserving networks ( EVPNet ) to separate robust features from non-robust ones , with three novel architecture components to model the extreme values : ( 1 ) parametric DoG ( pDoG ) to extract extreme values in scale-space for deep networks , ( 2 ) truncated ReLU ( tReLU ) to suppress noise or non-stable extrema and ( 3 ) projected normalization layer ( PNL ) to mimic PCA-SIFT ( Ke et al. , 2004 ) like feature normalization . pDoG and tReLU are combined into one block named EVPConv , which could be used to replace all k × k ( k > 1 ) conv-layers in existing CNNs . We conduct comprehensive experiments and ablation studies to verify the effectiveness of each component and the proposed EVPNet . Figure 1 illustrates a comparison of responses for standard convolution + ReLU and EVPConv in ResNet-50 trained on ImageNet , and shows that the proposed EVPConv produces less noises and more responses around object boundary than standard convolution + ReLU , which demonstrates the capability of EVPConv to separate robust features from non-robust ones . Our major contribution are : • To the best of our knowledge , we are the first to explicitly separate robust features from non-robust ones in deep neural networks from an architecture design perspective . • We propose three novel network architecture components to model extreme values in deep networks , including parametric DoG , truncated ReLU , and projected normalization layer , and verify their effectiveness through comprehensive ablation studies . • We propose extreme value preserving networks ( EVPNets ) to combine those three novel com- ponents , which are demonstrated to be not only more accurate , but also more robust to a set of adversarial attacks ( FGSM , PGD , etc ) even for clean model without adversarial training . 2 RELATED WORK . Robust visual features . Most traditional robust visual feature algorithms like SIFT ( Lowe , 2004 ) and SURF ( Bay et al. , 2006 ) are based on the scale-space theory ( Lindeberg , 1994 ) , while there is a close link between scale-space theory and biological vision ( Lowe , 2004 ) , since many scalespace operations show a high degree of similarity with receptive field profiles recorded from the mammalian retina and the first stages in the visual cortex . For instance , DoG computes the difference of two Gaussian blurred images and is believed to mimic the neural processing in the retina ( Young , 1987 ) . SIFT is one such kind of typical robust visual features , which consists of 4 major stages : ( 1 ) scale-space extrema detection with DoG operations ; ( 2 ) Keypoints localization by their stability ; ( 3 ) Orientation and scale assignment based on primary local gradient direction ; ( 4 ) Histogram based keypoint description . We borrow the scale-space extrema idea from SIFT , and propose three novel and robust architecture components to mimic key stages of SIFT . Robust Network Architectures . Many research efforts have been devoted to network robustness especially on defending against adversarial attacks as summarized in Akhtar & Mian ( 2018 ) . However , there are very limited works that tackle this problem from a network architecture design perspective . A major category of methods ( Liao et al. , 2018 ; Xie et al. , 2019 ) focus on designing new layers to perform denoising operations on the input image or the intermediate feature maps . Most of them are shown effective on black-box attacks , while are still vulnerable to white-box attacks . Non-local denoising layer proposed in Xie et al . ( 2019 ) is shown to improve robustness to white-box attack to an extent with adversarial training ( Madry et al. , 2018 ) . Peer sample information is introduced in Svoboda et al . ( 2019 ) with a graph convolution layer to improve network robustness . Biologically inspired protection ( Nayebi & Ganguli , 2017 ) introduces highly non-linear saturated activation layer to replace ReLU layer , and demonstrates good robustness to adversarial attacks , while similar higher-order principal is also used in Krotov & Hopfield ( 2018 ) . However , these methods still lack a systematic architecture design guidance , and many ( Svoboda et al. , 2019 ; Nayebi & Ganguli , 2017 ) are not robust to iterative attack methods like PGD under clean model setting . In this work , inspired by robust visual feature SIFT , we are able to design a series of innovative architecture components systematically for improving both model accuracy and robustness . We should stress that extreme value theory is a different concept to scale-space extremes , which tries to model the extreme in data distribution , and is used to design an attack-independent metric to measure robustness of DNNs ( Weng et al. , 2018 ) by exploring input data distribution . 3 PRELIMINARY . Difference-of-Gaussian . Given an input image I and Gaussian kernel G ( x , y , σ ) as below G ( x , y , σ ) = 1 2πσ2 e− ( x 2+y2 ) /2σ2 , ( 1 ) where σ denotes the variance . Also , difference of Gaussian ( DoG ) is defined as D ( x , y , σ ) = G ( x , y , σ ) ⊗ I1 −G ( x , y , σ ) ⊗ I0 , ( 2 ) where ⊗ is the convolution operation , and I1 = G ( x , y , σ ) ⊗ I0 . Scale-space DoG repeatedly convolves input images with the same Gaussian kernels , and produces difference-of-Gaussian images by subtracting adjacent image scales . Scale-space extrema ( maxima and minima ) are detected in DoG images by comparing a pixel to its 26 neighbors in 3×3 grids at current and two adjacent scales ( Lowe , 2004 ) . Adversarial Attacks . We use h ( · ) to denote the softmax output of classification networks , and hc ( · ) to denote the prediction probability of class c. Then given a classifier h ( x ) = y , the goal of adversarial attack is to find xadv such that the output of classifier deviates from the true label y : maxi h i ( xadv ) 6= y while closing to the original input : ||x− xadv|| ≤ . Here || · || refers to a norm operator , i.e . L2 or L∞ . Attack Method . The most simple adversarial attack method is Fast Gradient Sign Method ( FGSM ) ( Goodfellow et al. , 2015 ) , a single-step method which takes the sign of the gradient on the input as the direction of the perturbation . L ( · , · ) denotes the loss function defined by cross entropy . Specifically , the formation is as follows : xadv = x+ · sign ( ∇xL ( h ( x ) , y ) ) , ( 3 ) where x is the clean input , y is the label . is the norm bound ( ||x − xadv|| ≤ , i.e . -ball ) of the adversarial perturbation . Projected gradient descent ( PGD ) iteratively applies FGSM with a small step size αi ( Kurakin et al. , 2017a ; Madry et al. , 2018 ) with formulation as below : xadvi+1 = Proj ( x+ α · sign ( ∇xL ( h ( x adv i ) , y ) ) ) , ( 4 ) where i is the iteration number , α = /T with T being the number of iterations . ‘ Proj ’ is the function to project the image back to -ball every step . Some advanced and complex attacks are further introduced in DeepFool ( Moosavi-Dezfooli et al. , 2016 ) , CW ( Carlini & Wagner , 2017 ) , MI-FGSM ( Dong et al. , 2018 ) . Adversarial Training aims to inject adversarial examples into training procedure so that the trained networks can learn to classify adversarial examples correctly . Specifically , adversarial training solves the following empirical risk minimization problem : arg min h∈H E ( x , y ) D [ max x∗∈A ( x ) L ( h ( x∗ ) , y ) ] , ( 5 ) where A ( x ) denotes the area around x bounded by L∞/L2 norm , and H is the hypothesis space . In this work , we employ both FGSM and PGD to generate adversarial examples for adversarial training .

Summary: This paper proposes a network model named EVPNet, inspired by the idea scale-space extreme value from SIFT, to improve network robustness to adversarial pertubations over textures. To achieve better robustness, EVPNet separates outliers (non-robust) from robust examples by extenting DoG to parametric DoG, utilising truncated ReLU, and then applying a projected normalisation layer to mimic PCA-SIFT like feature normalisation, which are the three novelties that the authors claim in this paper. In the experiments, FGSM and PGD are used to provide adversarial attacks, and experiments conducted on CIFAR-10 and SVHN reveal that EVPNet enhances network robustness.