Calibration of neural network logit vectors to combat adversarial attacks

1 INTRODUCTION AND RELATED WORK . Adversarial examples are specially crafted input instances generated by adversarial attacks . The term was introduced by Szegedy et al . ( 2013 ) in the context of image classification . These attacks generate , or manipulate data , to achieve poor performance when classified by neural networks , which poses existential questions about their usage in high stakes security critical applications . Since they were introduced , there have been many papers that have introduced novel attack methods and other papers that attempt to combat those attacks . For instance , Goodfellow et al . ( 2014 ) introduces the fast gradient sign method ( FGSM ) , and Papernot et al . ( 2016c ) proposes a method based on modifying the gradient of the softmax function as a defence . Adversarial attacks can be identified into various classes such as white box and black box , where in the former , the attack has full knowledge of all model parameters . Examples created by these attacks can be false positives or false negatives . In the case of images , they can be nonsensical data ( e.g . noise classified as a road sign ) or clear cut ( e.g . a visually clear cat , classified as a road sign ) . These attacks can be non-targeted or targeted such that the classifier chooses a specific class for the adversarial example . Various adversarial defences exist , some based on deep learning techniques and others on purely distributional techniques . Similar work on adversarial defences has been done by Grosse et al . ( 2017 ) , in which the network is trained on specific attack types and parameters with an additional outlier class for adversarial examples . A multi-dimensional statistical test over the maximum mean discrepancy and the energy distance on input features is then used to classify instances as adversarial . Other work has been done by Bradshaw et al . ( 2017 ) , where Gaussian Processes are placed on top of conventional convolutional neural network architectures , with radial basis kernels , imbuing the neural network with a way of understanding its own perceptual limits . The authors find that the network becomes more resistant to adversarial attack . The work that follows continues in a similar vein to both of these methods . Some methods such as Metzen et al . ( 2017 ) use sub-units of deep learning architectures to detect adversarial instances . Calibration is a technique of converting model scores , normally , through application of a post processing function , to probability estimates . Background Check is a method to yield probability estimates , via a set of explicit assumptions , in regions of space where no data has been observed . In this work , Background Check is useful in producing calibrated probabilities for adversarial data that often exists in regions where no training and test data has been seen . Reliable probability estimates can then be measured by calibration and refinement loss . Various calibrating procedures exist such as binning , logistic regression , isotonic regression and softmax . Jordan et al . ( 1995 ) demonstrates the logistic function is optimal when the class-conditional densities are Gaussians with unit variance . Softmax extends this to multi-variate Gaussian densities with unit variance . Calibration of neural network models has been performed by Guo et al . ( 2017 ) , using a method called Temperature Scaling , that modifies the gradient of the softmax function allowing softmax to calibrate densities with non-unit variance . The authors perform this calibration after noticing that calibration loss for neural networks has increased in recent years . When adversarial attacks against neural networks are brought into perspective , a problem arises for existing calibration techniques , which is the question of mapping adversarial logit scores to reliable probability estimates ( which should be zero for a successful adversarial attack ) . In this work , a method is demonstrated that uses Background Check to identify adversarial attacks . 2 CALIBRATION . A classifier is said to be well calibrated , if , as the number of predictions approaches infinity , the proportion of outcomes given probability p , occur p fraction of the time . Denoting x1 , x2 , ... , xn as data-set instances and y1 , y2 , ... , yn as their corresponding ground truth class labels , a scoring classifier s = f ( xi ) has a calibrating function µ applied to it yielding µ ( f ( xi ) ) . Perfect calibration is defined as the expectation si = f ( xi ) such that si = E [ Y |f ( X ) = si ] where random variables X , Y denote the features and class label of a uniformly randomly drawn instance from the dataset respectively , such that Y = 1 , Y = 0 represent an individual positive and negative instance , respectively . To visualize calibration performance of a classifier , Murphy & Winkler ( 1977 ) plots the observed frequency of an event against the predicted frequency yielding calibration curves . Calibration curves plot the observed relative frequency against the predicted probability for all test data . Perfect calibration occurs if the calibration curve exactly fits the identity line . Refinement loss measures the difference between a probability estimate and zero or one . This can be combined with a frequency distribution , giving an indication of spread , which is useful if there are few events associated with a particular probability . The notion of refinement is also useful when considering calibration . By considering the crude constant classifier , which predicts the probability corresponding to the class distribution for all inputs , it is clear that this calibration estimate is perfectly calibrated . However , an intuitively more valuable calibration estimate , is one which predicts a value closer to either zero or one . For this reason , DeGroot & Fienberg ( 1981 ) suggests measuring refinement loss , which measures the distance of the classifiers probability estimates to either zero or one . Together , calibration and refinement loss make up the Brier Score . Kull & Flach ( 2015 ) defines calibration and refinement loss . Calibration loss = E [ d ( S , C ) ] is the loss due to the expected difference between the model score S and the proportion of positives among instances ( observed relative frequency ) with the same score . Refinement loss = E [ d ( C , Y ) ] is the loss due to the presence of instances from multiple classes among instances with the same estimate S. In the worst case , this clearly reduces to the crude constant classifier mentioned above . An instance of recent work related to calibration is Beta calibration . Beta calibration Kull et al . ( 2017 ) is based on the beta distribution which includes functions such as the logit , sigmoid and identity . This allows it to calibrate scores produced by models such as naive bayes , which biases its scores towards extremities when the assumption of feature independence is not met , using the inverse sigmoid or logit function , a function which is not in softmax ’ s repertoire . 3 ADVERSARIAL ATTACKS . In the context of adversarial attacks , the optimization problem that is often formulated to construct adversarial examples is shown . argmin δx ||δx||s.t.F ( X + δx ) = Y ∗ An adversarial example X∗ is constructed by adding a perturbation vector δx , where Y ∗ is the desired adversarial output and X + δx ∈ [ 0 , 1 ] where [ 0 , 1 ] is the upper and lower bound of a well formed example , subject to the constraints above . Intuitively , these constraints coerce the network into mis-classifying each example with a minimal perturbation vector . Example distance metrics that measure the size of the perturbation include Lp metrics , as well as the PASS score Rozsa et al . ( 2016 ) , a metric designed to better reflect notions of psycho-physical similarity than Lp metrics . Adversarial attack methods include L-BFGS from Szegedy et al . ( 2013 ) , which uses linear trial and error to find a c for each data instance such that cmultiplied by an arbitrary small perturbation vector mis-classifies the instance . FGSM builds on the L-BFGS method , replacing expensive linear search with gradient descent to find the perturbation vector . This uses the following optimization strategy to create the perturbation vector η , such that x is the input to the model , y is the predicted class , θ are the model parameters and J ( θ , x , y ) is the cost function . Ox indicates the derivative of the cost function is taken with respect to the input x . The sign function takes the sign of the resulting derivative . The adversarial example is then constructed as x′ = x+ η. η = sign ( Ox , J ( θ , x , y ) ) A momentum term can be added to the gradient descent process to yield the work of Dong et al . ( 2018 ) . The BIM attack by Kurakin et al . ( 2016 ) , uses a smaller noise vector produced by FGSM , applied iteratively , before a clip operation is applied to the resulting image , keeping it within the maximum image pixel values after each iteration . JSMA is a forward derivative approach , by Papernot et al . ( 2016b ) , which uses a Jacobian matrix , to produce an adversarial saliency map , which indicate the features that when ( positively or negatively ) perturbed , most efficiently achieve a desired network output . DeepFool ( Moosavi Dezfooli et al. , 2016 ) finds an image that is at a minimal distance to the decision boundary from the proposed example , to another target class which isn ’ t the source class , treating multi-class classifiers as combinations of binary affine classifiers . 4 BACKGROUND CHECK . Background Check , introduced by Perello-Nieto et al . ( 2016 ) , is the calibrating procedure that our proposed method uses to defend neural networks from adversarial attacks . Background Check takes as input , the scores from logit vectors and maps them to probabilities , replacing softmax after training . Background Check provides a framework to classify regions where no previous data has been seen as background regions , that in the context of the proposed method will be classified as regions where adversarial data may lie . Background Check also provides a framework to resolve ambiguity inherent in a single value representing probability . More specifically , Background Check provides two values to represent a single probability value of a data instance . One value represents distance from the data density , and another represents the certainty of a particular class . This approach avoids overloading the meaning of a single number representing probability . More specifically , an uncertainty of 1/n for all classes , could represent an instance very close to the decision boundary or very far away from training data . On the other hand , an output of zero , could represent a point very far away from training data or a classification that the data is definitely not that particular class . 4.1 BACKGROUND CHECK IN MODELLING SPARSE DATA . Background Check introduces an additional outlier background class , b , representing regions of space where data is sparse or non-existent . Then , a foreground class is introduced , f . This represents regions of space where data is plentiful or dense , i.e . data from any class apart from the background class , is abundant . b is introduced as an additional class whilst f is kept as a reference class . Every instance x necessarily belongs to either f or b. P ( b|x ) = 0 and P ( f |x ) = 1 refers to absolute certainty that the instance belongs to one of the classes with sufficient training data , where P ( C|x ) is a conditional probability measure . The ratio of the two conditional measures defines the reliability factor r ( x ) . r ( x ) = P ( f |x ) P ( b|x ) = P ( f1|x ) + ... + P ( fn|x ) P ( b|x ) = P ( f , x ) P ( x ) P ( b , x ) P ( x ) = P ( x , f ) P ( x , b ) If r ( x ) ≤ 1 , the classification that the reliability factor indicates is b , else if r ( x ) > 1 then the classification is f . P ( x , f ) and P ( x , b ) are referred to as the foreground and background densities . Furthermore , the relative foreground and background densities can be defined , qf ( x ) and qb ( x ) . qf ( x ) = P ( x , f ) maxx P ( x , f ) , qb ( x ) = P ( x , b ) maxx P ( x , f ) Intuitively , the relative density outputs the proportion of f or b , at the point in space corresponding to the instance x being evaluated . Simple dividing qf ( x ) by qb ( x ) yields r ( x ) . 4.1.1 JUSTIFICATION OF qb ( x ) To construct qb ( x ) , four inductive biases , in increasing strength , are given . This work only uses the third inductive bias . 1 . Inductive bias 1 : qb ( x ) is a function of qf ( x ) . This is justified , by the idea that with no other information , there is no reason to assign different background densities to points with the same foreground density . The domain knowledge informs the function used µ : [ 0 , 1 ] −→ [ 0 , ∞ ) . 2 . Inductive bias 2 : monotonicity of µ , that is , when moving to a region with higher foreground density the background increases or decreases . 3 . Inductive bias 3 : an affine bias , i.e . µ ( x ) = ax + b or by replacing a and b : µ ( x ) = ( 1− x ) µ ( 0 ) + xµ ( 1 ) . 4 . Inductive bias 4 : constant background ie µ ( 0 ) , µ ( 1 ) = 0.5 .

Summary: This paper addresses adversarial detection through the absolute-value difference between the two logit vector values of a DNN binary classifier, with one class associated with normal data and the other with adversarial data. Assignment of examples to an "adversarial" class is problematic in that adversarial examples are typically generated in regions for which training data is very sparse. To cope with this, the authors propose use of the Background Check calibration techniques recently proposed by Perello-Nieto et al. (ICDM 2016).