Radial Basis Feature Transformation to Arm CNNs Against Adversarial Attacks

1 INTRODUCTION . It has been shown that deep convolutional neural networks ( CNNs ) can be highly vulnerable to adversarial perturbations Szegedy et al . ( 2013 ) ; Goodfellow et al . ( 2015 ) produced by two main groups of attack mechanisms : white- and back-box , which refer to having access to the victim model parameters and architecture or not , respectively . In order to mitigate the effect of adversarial attacks , two main categories of defense techniques have been proposed : data-level and algorithmic-level . Data-level methods include adversarial training Szegedy et al . ( 2013 ) ; Goodfellow et al . ( 2015 ) , pre-/post-processing methods ( e.g . feature squeezing ) Xu et al . ( 2017 ) , pre-processing using basis functions Shaham et al . ( 2018 ) , and noise removal Hendrycks & Gimpel ( 2016 ) ; Meng & Chen ( 2017 ) . Algorithmic-level methods Kolter & Wong ( 2017 ) ; Samangouei et al . ( 2018 ) ; Folz et al . ( 2018 ) ; Samangouei et al . ( 2018 ) modify the deep model or training procedure by reducing the magnitude of gradients Papernot et al . ( 2015 ) or blocking/masking gradients Buckman et al . ( 2018 ) ; Guo et al . ( 2017 ) ; Song et al . ( 2017 ) . However , these approaches are not completely effective against several different white- and black-box attacks Samangouei et al . ( 2018 ) ; Tramèr et al . ( 2017 ) ; Meng & Chen ( 2017 ) and pre-processing based methods might deteriorate an un-attacked model performance . Generally , most of these defense strategies cause a drop in standard accuracy on clean data Tsipras et al . ( 2018 ) . For more details on adversarial attacks and defenses , we refer readers to Yuan et al . ( 2018 ) . Gradient masking has shown sub-optimal performance against different types of adversarial attacks Papernot et al . ( 2017 ) ; Tramèr et al . ( 2017 ) . Athalye et al . ( 2018 ) identified obfuscated gradients , a special case of gradient masking that leads to a false sense of security in defenses against adversarial perturbations . They showed that 7 out of 9 recent white-box defenses relying on this phenomenon ( Buckman et al . ( 2018 ) ; Ma et al . ( 2018 ) ; Guo et al . ( 2017 ) ; Dhillon et al . ( 2018 ) ; Xie et al . ( 2017a ) ; Song et al . ( 2017 ) ; Samangouei et al . ( 2018 ) ) are vulnerable to single step or nongradient based attacks . They finally suggested several symptoms of defenses that rely on obfuscated gradients . As explored in the literature , adversarial examples can be mainly the results of models being too linear Goodfellow et al . ( 2015 ) in high dimensional manifolds when the decision boundary is close to the manifold of the training data Tanay & Griffin ( 2016 ) and/or because of the models ’ low flexi- bility Fawzi et al . ( 2015 ) . To boost the non-linearity of a model , Goodfellow et al . ( 2015 ) explored a variety of methods involving utilizing quadratic units and including shallow and deep radial basis function ( RBF ) networks . They achieved reasonably good performance against adversarial perturbations with shallow RBF networks , however , they found it difficult to train deep RBF models , leading to a high training error using stochastic gradient decent . Fawzi et al . ( 2018 ) showed that support vector machines with RBF kernels can effectively resist adversarial attacks . Typically , a single RBF network layer takes a vector of x ∈ Rn as input and output a scalar function of the input vector f ( x ) : Rn → R computed as f ( x ) = ∑N i=1 wie −βiD ( x , ci ) , where N is the number of neurons in the hidden layer , ci is the center vector for neuron i , D ( x , ci ) measures the distance between x and ci , and wi weights the output of neuron i . The Gaussian basis functions , commonly used in RBF networks , are local to the center vectors , i.e . lim‖x‖→∞D ( x , ci ) = 0 , which in turn implies that changing the parameters of a neuron has an increasing smaller effect on the output as the sample input x is farther from center of that neuron ( i.e . x is a sample that the RBF network does not ‘ understand ’ Goodfellow et al . ( 2015 ) ) . When such a distant sample is in the response domain of an RBF , the change in the response caused by a perturbation added to the sample is negligible . Traditional RBF networks are normally trained in two sequential steps . First , an unsupervised method , e.g . k-means clustering , is applied to find RBF centers Wu et al . ( 2012 ) and , second , a linear model with coefficients wi is fit to the desired outputs . To tackle the linearity issue of the current models ( i.e . stacking several linear units ) , particularly deep CNNs , which results in vulnerability to adversarial perturbations , we arm CNNs with radial basis feature transformation . Further , to add more flexibility to the models , radial basis functions with Euclidean distance might not be effective as the activation of each neuron depends on the Euclidean distance between a pattern and the neuron center . However , since the activation function is constrained to be symmetrical , all attributes are considered equally relevant . This limitation can be addressed by applying non-symmetrical quadratic distance functions , such as a Mahalanobis distance , in the activation function in order to take into account the variability of the attributes and their correlations . However , computed this distance directly from the variance-covariance matrix of training data is sensitive to outliers and does not consider the accuracy of the learning algorithm . In this paper , we propose a non-linear radial basis convolutional feature transformation method based on Mahalanobis distance . In contrast to traditional Mahalanobis formulation , in which a constant , pre-defined covariance matrix is adopted , we propose to learn such ” transformation ” matrix T ∈ Rn×n . To enforce T ’ s positive semi-definiteness , using the eigenvalue decomposition , it can be decomposed into T ′T . All other RBF parameters , i.e . centers C = { c1 , c2 , · · · , cN } and βi ( width of Gaussian ) , along with all CNN parameters Ω , are also learned end-to-end using back-propagation . This approach causes the local RBF centers to be adjusted optimally as they are updated based on the whole network parameters . Therefore , we define a loss function L encoding the classification error in the transformed space and seek T ∗ , β∗ , C∗ , and Ω∗ that minimize L : T ∗ , C∗ , β∗ , Ω∗ = argminP , C , β , ΩL ( P , C , β , Ω ) . ( 1 ) 2 RADIAL BASIS FEATURE TRANSFORMATION . Given a convolutional feature map F of size n ×m × k the goal is to map the features onto a new space G of size n×m×o where classes are dense ( tightly clustered ) and well separated . To achieve this , we leverage a RBF projector that takes as input feature vectors of Fi , j , k and transforms them to a new space by learning a transformation matrix under Mahalanobis distance formulation as follows : φk = e −βD ( Fi , j , K , ck ) ( 2 ) where i = 1 : n , j = 1 : m , φk is kth activation function , ck is kth learnable center , trained β controls the width of the Gaussian function and D ( . ) refers to Mahalanobis distance between a convolutional feature vector and a center computed as : D ( Fi , j , K , ck ) = √ ( Fi , j , K − ck ) T P ( Fi , j , K − ck ) ( 3 ) where P refers to learnable transformation matrix of size n×m and its size varies for different size inputs in each layer . Finally , the projected feature vector G ( Fi , j , k ) is computed as : G ( Fi , j , K ) = k∑ i=1 ( wk · φk ) + bk ( 4 ) The detailed diagram of the transformation step is shown in Figure 2 . Finally , in each layer l of network , output feature maps of each convolutional block are concatenated with projected feature maps ( Figure 1 ) . 3 DATA . We conduct two sets of experiments : image ( i ) classification and ( ii ) segmentation . ( i ) For the image classification experiments , we use MNIST and ChestX-ray14 dataset ( Wang et al . ( 2017 ) ) , which comprises 112,120 gray-scale images with 14 disease labels and one ‘ no ( clinical ) finding ’ label . We treat all the disease classes as positive and formulate a binary classification task . We randomly selected 90,000 images for training , 45,000 images with “ positive ” label and the remaining 45,000 with “ negative ” label . The validation set comprised 13,332 images with 6,666 images of each label . We randomly picked 200 unseen images as the test set , with 93 images with positive and 107 having negative labels . These clean ( test ) images are used for carrying out different adversarial attacks and the models trained on clean images are evaluated against them . ( ii ) For the image segmentation task experiments , we used the 2D RGB skin lesion dataset from the 2017 IEEE ISBI International Skin Imaging Collaboration ( ISIC ) Challenge Codella et al . ( 2017 ) . We trained on a set of 2,000 images and tested on an unseen set of 150 images . 4 EXPERIMENTS AND RESULTS . In this section , we report the results of several experiments for two tasks of classification and segmentation . We first start with MNIST as it has commonly been used for evaluating adversarial attacks and defences . Next we show how the proposed method is scalable to another classification dataset and segmentation task . 4.1 EVALUATION ON CLASSIFICATION TASK . In this section , we analyze the proposed method on two different classifications datasets MNIST and of X-chest14 . In Table 1 , we report the results of the proposed method on MNIST dataset when attacked by different targeted and un-targeted attacks i.e . fast gradient sign method ( FGSM ) Kurakin et al . ( 2016a ) , basic iterative method ( BIM ) Kurakin et al . ( 2016b ) , projected gradient descent ( PGD ) Madry et al . ( 2017 ) , Carlini & Wagner method ( C & W ) Carlini & Wagner ( 2017 ) , momentum iterative method ( MIM ) Dong et al . ( 2018 ) winner of NIPS 2017 adversarial attacks competition . The proposed method ( i.e . PROP ) successfully resists all the attacks for which the 3-layers CNN ( i.e . ORIG ) network almost completely fails e.g . for the strongest attack ( i.e . MIM ) the proposed method achieves 64.25 % accuracy while the original CNN network obtains almost zero ( 0.58 % ) accuracy . Further , we test the proposed method with a non-gradient based attack i.e . Gaussian additive noise ( GN ) Rauber et al . ( 2017 ) to show that the robustness of the method is not because of gradient masking . To ensure that the proposed method robustness is not due to masked/obfuscated gradient , as suggested by Athalye et al . ( 2018 ) , we test the proposed feature transformation method based on several characteristic behaviors of defenses which cause obfuscated gradients to occur . a ) As reported in Table 1 , one-step attacks ( e.g . FGSM ) did not perform better than iterative attacks ( e.g . BIM , MIM ) ; b ) According to Tables 4 and 5 , black-box attacks did not perform better than white-box ones ; c ) as shown in Figure 3 ( left and middle ) , larger distortion factors monotonically increase the attack success rate . The right plot in Figure 3 also indicates that the robustness of the proposed method is not because of numerical instability of gradients . Next , to quantify the compactness and separability of different clusters/classes , we evaluate the features produced ORIG and PROP methods with clustering evaluation techniques such as mutual information based score Vinh et al . ( 2010 ) , homogeneity and completeness Rosenberg & Hirschberg ( 2007 ) , Silhouette coefficient Rousseeuw ( 1987 ) , and Calinski-Harabaz index Caliński & Harabasz ( 1974 ) . Both Silhouette coefficient and Calinski-Harabaz index quantify how well clusters are separated from each other and how compact they are without taking into account the ground truth labels , while mutual information based score , homogeneity , and completeness scores evaluate clusters based on labels . As reported in Table 2 , when the original CNN network applies radial basis feature transformation it achieves considerably higher scores ( the higher values for all the metrics the better ) . As both original and the proposed method achieved high classification test accuracy i.e . ∼ 99 % , the difference in scores for ground truth label based metrics , i.e . mutual information based , homogeneity , and completeness , scores are small . In Figure 4 , we illustrate feature spaces of each layer in a simple 3-layer CNN network using tSNE and PCA methods by reducing the high dimensional feature space into two dimensions . As can be seen , the proposed radial basis feature transformation helps reduce intra-class and increase inter-class distances . t-SNE 1st layer t-SNE 2nd layer t-SNE 3rd layer PCA 3rd layer To test the robustness of the proposed method on the X-chest14 dataset we follow the strategy of Taghanaki et al . ( 2018 ) . We select Inception-ResNet-v2 and modify it by the proposed radial basis function blocks . According to the study done by Taghanaki et al . ( 2018 ) , we focus on the most effective attacks in term of imperceptibility and power i.e . gradient-based attacks ( basic iterative method Kurakin et al . ( 2016a ) : BIM and L1-BIM ) . We also compare the proposed method with two defense strategies : Gaussian data augmentation ( GDA ) Zantedeschi et al . ( 2017 ) and feature squeezing ( FSM ) Xu et al . ( 2017 ) . GDA is a data augmentation technique that augments a dataset with copies of the original samples to which Gaussian noise has been added . FSM method reduces the precision of the components of the original input by encoding them on a smaller number of bits . In Table 3 , we report the classification accuracy of different attacks and defences ( including PROP ) on X-chest14 dataset .

Summary: The proposed method is too simplistic, the model being succinctly described in less than one page with many errors in the given math expressions. Only the model is given. The optimization problem, as given in (1) is not explained. the authors need to stud the optimization problem, to derive its resolution, and to describe the obtained algorithm.