Radial Basis Feature Transformation to Arm CNNs Against Adversarial Attacks

1 INTRODUCTION . It has been shown that deep convolutional neural networks ( CNNs ) can be highly vulnerable to adversarial perturbations Szegedy et al . ( 2013 ) ; Goodfellow et al . ( 2015 ) produced by two main groups of attack mechanisms : white- and back-box , which refer to having access to the victim model parameters and architecture or not , respectively . In order to mitigate the effect of adversarial attacks , two main categories of defense techniques have been proposed : data-level and algorithmic-level . Data-level methods include adversarial training Szegedy et al . ( 2013 ) ; Goodfellow et al . ( 2015 ) , pre-/post-processing methods ( e.g . feature squeezing ) Xu et al . ( 2017 ) , pre-processing using basis functions Shaham et al . ( 2018 ) , and noise removal Hendrycks & Gimpel ( 2016 ) ; Meng & Chen ( 2017 ) . Algorithmic-level methods Kolter & Wong ( 2017 ) ; Samangouei et al . ( 2018 ) ; Folz et al . ( 2018 ) ; Samangouei et al . ( 2018 ) modify the deep model or training procedure by reducing the magnitude of gradients Papernot et al . ( 2015 ) or blocking/masking gradients Buckman et al . ( 2018 ) ; Guo et al . ( 2017 ) ; Song et al . ( 2017 ) . However , these approaches are not completely effective against several different white- and black-box attacks Samangouei et al . ( 2018 ) ; Tramèr et al . ( 2017 ) ; Meng & Chen ( 2017 ) and pre-processing based methods might deteriorate an un-attacked model performance . Generally , most of these defense strategies cause a drop in standard accuracy on clean data Tsipras et al . ( 2018 ) . For more details on adversarial attacks and defenses , we refer readers to Yuan et al . ( 2018 ) . Gradient masking has shown sub-optimal performance against different types of adversarial attacks Papernot et al . ( 2017 ) ; Tramèr et al . ( 2017 ) . Athalye et al . ( 2018 ) identified obfuscated gradients , a special case of gradient masking that leads to a false sense of security in defenses against adversarial perturbations . They showed that 7 out of 9 recent white-box defenses relying on this phenomenon ( Buckman et al . ( 2018 ) ; Ma et al . ( 2018 ) ; Guo et al . ( 2017 ) ; Dhillon et al . ( 2018 ) ; Xie et al . ( 2017a ) ; Song et al . ( 2017 ) ; Samangouei et al . ( 2018 ) ) are vulnerable to single step or nongradient based attacks . They finally suggested several symptoms of defenses that rely on obfuscated gradients . As explored in the literature , adversarial examples can be mainly the results of models being too linear Goodfellow et al . ( 2015 ) in high dimensional manifolds when the decision boundary is close to the manifold of the training data Tanay & Griffin ( 2016 ) and/or because of the models ’ low flexi- bility Fawzi et al . ( 2015 ) . To boost the non-linearity of a model , Goodfellow et al . ( 2015 ) explored a variety of methods involving utilizing quadratic units and including shallow and deep radial basis function ( RBF ) networks . They achieved reasonably good performance against adversarial perturbations with shallow RBF networks , however , they found it difficult to train deep RBF models , leading to a high training error using stochastic gradient decent . Fawzi et al . ( 2018 ) showed that support vector machines with RBF kernels can effectively resist adversarial attacks . Typically , a single RBF network layer takes a vector of x ∈ Rn as input and output a scalar function of the input vector f ( x ) : Rn → R computed as f ( x ) = ∑N i=1 wie −βiD ( x , ci ) , where N is the number of neurons in the hidden layer , ci is the center vector for neuron i , D ( x , ci ) measures the distance between x and ci , and wi weights the output of neuron i . The Gaussian basis functions , commonly used in RBF networks , are local to the center vectors , i.e . lim‖x‖→∞D ( x , ci ) = 0 , which in turn implies that changing the parameters of a neuron has an increasing smaller effect on the output as the sample input x is farther from center of that neuron ( i.e . x is a sample that the RBF network does not ‘ understand ’ Goodfellow et al . ( 2015 ) ) . When such a distant sample is in the response domain of an RBF , the change in the response caused by a perturbation added to the sample is negligible . Traditional RBF networks are normally trained in two sequential steps . First , an unsupervised method , e.g . k-means clustering , is applied to find RBF centers Wu et al . ( 2012 ) and , second , a linear model with coefficients wi is fit to the desired outputs . To tackle the linearity issue of the current models ( i.e . stacking several linear units ) , particularly deep CNNs , which results in vulnerability to adversarial perturbations , we arm CNNs with radial basis feature transformation . Further , to add more flexibility to the models , radial basis functions with Euclidean distance might not be effective as the activation of each neuron depends on the Euclidean distance between a pattern and the neuron center . However , since the activation function is constrained to be symmetrical , all attributes are considered equally relevant . This limitation can be addressed by applying non-symmetrical quadratic distance functions , such as a Mahalanobis distance , in the activation function in order to take into account the variability of the attributes and their correlations . However , computed this distance directly from the variance-covariance matrix of training data is sensitive to outliers and does not consider the accuracy of the learning algorithm . In this paper , we propose a non-linear radial basis convolutional feature transformation method based on Mahalanobis distance . In contrast to traditional Mahalanobis formulation , in which a constant , pre-defined covariance matrix is adopted , we propose to learn such ” transformation ” matrix T ∈ Rn×n . To enforce T ’ s positive semi-definiteness , using the eigenvalue decomposition , it can be decomposed into T ′T . All other RBF parameters , i.e . centers C = { c1 , c2 , · · · , cN } and βi ( width of Gaussian ) , along with all CNN parameters Ω , are also learned end-to-end using back-propagation . This approach causes the local RBF centers to be adjusted optimally as they are updated based on the whole network parameters . Therefore , we define a loss function L encoding the classification error in the transformed space and seek T ∗ , β∗ , C∗ , and Ω∗ that minimize L : T ∗ , C∗ , β∗ , Ω∗ = argminP , C , β , ΩL ( P , C , β , Ω ) . ( 1 ) 2 RADIAL BASIS FEATURE TRANSFORMATION . Given a convolutional feature map F of size n ×m × k the goal is to map the features onto a new space G of size n×m×o where classes are dense ( tightly clustered ) and well separated . To achieve this , we leverage a RBF projector that takes as input feature vectors of Fi , j , k and transforms them to a new space by learning a transformation matrix under Mahalanobis distance formulation as follows : φk = e −βD ( Fi , j , K , ck ) ( 2 ) where i = 1 : n , j = 1 : m , φk is kth activation function , ck is kth learnable center , trained β controls the width of the Gaussian function and D ( . ) refers to Mahalanobis distance between a convolutional feature vector and a center computed as : D ( Fi , j , K , ck ) = √ ( Fi , j , K − ck ) T P ( Fi , j , K − ck ) ( 3 ) where P refers to learnable transformation matrix of size n×m and its size varies for different size inputs in each layer . Finally , the projected feature vector G ( Fi , j , k ) is computed as : G ( Fi , j , K ) = k∑ i=1 ( wk · φk ) + bk ( 4 ) The detailed diagram of the transformation step is shown in Figure 2 . Finally , in each layer l of network , output feature maps of each convolutional block are concatenated with projected feature maps ( Figure 1 ) . 3 DATA . We conduct two sets of experiments : image ( i ) classification and ( ii ) segmentation . ( i ) For the image classification experiments , we use MNIST and ChestX-ray14 dataset ( Wang et al . ( 2017 ) ) , which comprises 112,120 gray-scale images with 14 disease labels and one ‘ no ( clinical ) finding ’ label . We treat all the disease classes as positive and formulate a binary classification task . We randomly selected 90,000 images for training , 45,000 images with “ positive ” label and the remaining 45,000 with “ negative ” label . The validation set comprised 13,332 images with 6,666 images of each label . We randomly picked 200 unseen images as the test set , with 93 images with positive and 107 having negative labels . These clean ( test ) images are used for carrying out different adversarial attacks and the models trained on clean images are evaluated against them . ( ii ) For the image segmentation task experiments , we used the 2D RGB skin lesion dataset from the 2017 IEEE ISBI International Skin Imaging Collaboration ( ISIC ) Challenge Codella et al . ( 2017 ) . We trained on a set of 2,000 images and tested on an unseen set of 150 images . 4 EXPERIMENTS AND RESULTS . In this section , we report the results of several experiments for two tasks of classification and segmentation . We first start with MNIST as it has commonly been used for evaluating adversarial attacks and defences . Next we show how the proposed method is scalable to another classification dataset and segmentation task . 4.1 EVALUATION ON CLASSIFICATION TASK . In this section , we analyze the proposed method on two different classifications datasets MNIST and of X-chest14 . In Table 1 , we report the results of the proposed method on MNIST dataset when attacked by different targeted and un-targeted attacks i.e . fast gradient sign method ( FGSM ) Kurakin et al . ( 2016a ) , basic iterative method ( BIM ) Kurakin et al . ( 2016b ) , projected gradient descent ( PGD ) Madry et al . ( 2017 ) , Carlini & Wagner method ( C & W ) Carlini & Wagner ( 2017 ) , momentum iterative method ( MIM ) Dong et al . ( 2018 ) winner of NIPS 2017 adversarial attacks competition . The proposed method ( i.e . PROP ) successfully resists all the attacks for which the 3-layers CNN ( i.e . ORIG ) network almost completely fails e.g . for the strongest attack ( i.e . MIM ) the proposed method achieves 64.25 % accuracy while the original CNN network obtains almost zero ( 0.58 % ) accuracy . Further , we test the proposed method with a non-gradient based attack i.e . Gaussian additive noise ( GN ) Rauber et al . ( 2017 ) to show that the robustness of the method is not because of gradient masking . To ensure that the proposed method robustness is not due to masked/obfuscated gradient , as suggested by Athalye et al . ( 2018 ) , we test the proposed feature transformation method based on several characteristic behaviors of defenses which cause obfuscated gradients to occur . a ) As reported in Table 1 , one-step attacks ( e.g . FGSM ) did not perform better than iterative attacks ( e.g . BIM , MIM ) ; b ) According to Tables 4 and 5 , black-box attacks did not perform better than white-box ones ; c ) as shown in Figure 3 ( left and middle ) , larger distortion factors monotonically increase the attack success rate . The right plot in Figure 3 also indicates that the robustness of the proposed method is not because of numerical instability of gradients . Next , to quantify the compactness and separability of different clusters/classes , we evaluate the features produced ORIG and PROP methods with clustering evaluation techniques such as mutual information based score Vinh et al . ( 2010 ) , homogeneity and completeness Rosenberg & Hirschberg ( 2007 ) , Silhouette coefficient Rousseeuw ( 1987 ) , and Calinski-Harabaz index Caliński & Harabasz ( 1974 ) . Both Silhouette coefficient and Calinski-Harabaz index quantify how well clusters are separated from each other and how compact they are without taking into account the ground truth labels , while mutual information based score , homogeneity , and completeness scores evaluate clusters based on labels . As reported in Table 2 , when the original CNN network applies radial basis feature transformation it achieves considerably higher scores ( the higher values for all the metrics the better ) . As both original and the proposed method achieved high classification test accuracy i.e . ∼ 99 % , the difference in scores for ground truth label based metrics , i.e . mutual information based , homogeneity , and completeness , scores are small . In Figure 4 , we illustrate feature spaces of each layer in a simple 3-layer CNN network using tSNE and PCA methods by reducing the high dimensional feature space into two dimensions . As can be seen , the proposed radial basis feature transformation helps reduce intra-class and increase inter-class distances . t-SNE 1st layer t-SNE 2nd layer t-SNE 3rd layer PCA 3rd layer To test the robustness of the proposed method on the X-chest14 dataset we follow the strategy of Taghanaki et al . ( 2018 ) . We select Inception-ResNet-v2 and modify it by the proposed radial basis function blocks . According to the study done by Taghanaki et al . ( 2018 ) , we focus on the most effective attacks in term of imperceptibility and power i.e . gradient-based attacks ( basic iterative method Kurakin et al . ( 2016a ) : BIM and L1-BIM ) . We also compare the proposed method with two defense strategies : Gaussian data augmentation ( GDA ) Zantedeschi et al . ( 2017 ) and feature squeezing ( FSM ) Xu et al . ( 2017 ) . GDA is a data augmentation technique that augments a dataset with copies of the original samples to which Gaussian noise has been added . FSM method reduces the precision of the components of the original input by encoding them on a smaller number of bits . In Table 3 , we report the classification accuracy of different attacks and defences ( including PROP ) on X-chest14 dataset .

Summary: The authors propose a new defense against adversarial examples based on radial basis features. Prior work has suggested that the linearity of standard convolutional networks may be a factor contributing to their vulnerability against adversarial examples, and that radial basis functions may help alleviate this weakness. The current paper builds on this idea and proposes a concrete way to add radial basis features to existing convnet architectures.