Prior Networks for Detection of Adversarial Attacks

1 INTRODUCTION . Neural Networks ( NNs ) have become the dominant approach to addressing computer vision ( CV ) ( Girshick , 2015 ; Simonyan & Zisserman , 2015 ; Villegas et al. , 2017 ) , natural language processing ( NLP ) ( Mikolov et al. , 2013b ; a ; 2010 ) , speech recognition ( ASR ) ( Hinton et al. , 2012 ; Hannun et al. , 2014 ) and bio-informatics ( Caruana et al. , 2015 ; Alipanahi et al. , 2015 ) tasks . However , as observed by ( Szegedy et al. , 2013 ) , they are susceptible to adversarial attacks - small perturbations to the input which are almost imperceptible to humans , yet which drastically affect the predictions of the neural network . It was found that adversarial attacks have several properties which make them a serious security concern . Firstly , adversarial attacks are transferable - an adversarial attack computed on one network may be able to successfully attack a different network ( Szegedy et al. , 2013 ; Goodfellow et al. , 2015 ) . Secondly , there exists a plethora of adversarial attacks which are quite easy to construct ( Goodfellow et al. , 2015 ; Kurakin et al. , 2016 ; Dong et al. , 2018 ; Carlini & Wagner , 2016 ; Papernot et al. , 2016a ; b ; Liu et al. , 2016 ) . Thirdly , it is possible to craft adversarial attacks within the physical world , such as putting stickers on paper ( Kurakin et al. , 2016 ) . To compound the issue , it was found that adversarial attacks are hard to defend against ( Carlini & Wagner , 2017 ) . Specifically , it was found that while adversarial training ( Szegedy et al. , 2013 ) and adversarial distillation ( Papernot et al. , 2016c ) are able to improve the robustness of a network , it is still possible to craft successful adversarial attacks against these networks . Indeed , currently , there are far more methods of successfully attacking networks than there are of defending networks ( Carlini & Wagner , 2017 ; 2016 ) . Altogether , this raises serious concerns about how safe it is to deploy neural networks for high-stakes applications , such as autonomous vehicle control , financial and medical applications . While much work focuses on constructing neural networks which are robust to adversarial attacks , ( Carlini & Wagner , 2017 ) investigate detection of adversarial attack and shows that adversarial attacks can be detectable using a range of approaches . However , it turns out that attacks can then be crafted to fool the proposed detection schemes . However , ( Carlini & Wagner , 2017 ) singles out detection of adversarial attacks using uncertainty measures derived from Monte-Carlo dropout as being successful . Detection of adversarial attack using Monte-Carlo dropout was further investigated in ( Smith & Gal , 2018 ) . Work by ( Smith & Gal , 2018 ) suggests that adversarially modified inputs can be interpreted as inputs which lie off the manifold of natural images . In other words , adversarial samples can be seen as a special type of out-of-domain or out-of-distribution input which can be detected by measures of uncertainty derived via Monte-Carlo Dropout . Recently , ( Malinin & Gales , 2018 ) proposed Prior Networks - a new approach to modelling uncertainty which has been shown to outperform Monte-Carlo dropout on a range of tasks . Prior Network parameterize a distribution over output distributions which allows them to separately model data uncertainty and distributional uncertainty . Furthermore , they are explicitly taught to learn decision boundaries between an in-domain and an out-of-domain region , unlike Monte-Carlo dropout , where it is not possible to specify the behaviour of measures of uncertainty in different regions of the input space explicitly . Finally , Prior Networks allow measures of uncertainty to be obtained analytically , without the need for expensive Monte-Carlo sampling . Thus , this work investigates the detection of adversarial attacks using measures of uncertainty derived from Prior Networks . The contributions of this work are as follows : • Measures for assessing the success of adversarial attacks in the context of detection are proposed . • It is shown that whitebox and blackbox adversarial attacks with no knowledge of the detection scheme can be successfully detected based on measures of uncertainty . • It is shown that whitebox detection evading adversarial attacks are difficult to construct for Prior Networks , and detection evading attack fail entirely against Prior Networks . 2 UNCERTAINTY FOR DEEP LEARNING . In ( Smith & Gal , 2018 ) adversarially modified inputs are interpreted as inputs which lie off the manifold of natural images - the stronger the adversarial perturbation , the further is the input from the manifold . Thus , adversarial samples can be seen as out-of-distribution inputs . Out-of-distribution inputs are associated with distributional uncertainty , which arises due to mismatch between the training and test distributions - in other words distributional uncertainty arises when the test data is ’ out-of-distribution ’ relative to the training data . Standard DNNs model data uncertainty ( uncertainty due to class overlap and noise in the data ) , as described in appendix B , but fail to capture distributional uncertainty . In order to capture distributional uncertainty it is necessary to use approaches such as Monte-Carlo dropout or Prior Networks . Thus , this section describes how Monte-Carlo Dropout ( Gal & Ghahramani , 2016 ) and Prior Networks ( Malinin & Gales , 2018 ) model distributional uncertainty . 2.1 UNCERTAINTY ESTIMATION VIA MONTE-CARLO DROPOUT . The essence of Bayesian approaches is to treat model parameters θ as random variables and place a prior distribution p ( θ ) over them to compute a posterior distribution p ( θ|D ) over model parameters given the training data D via Bayes ’ rule . Uncertainty in the model parameters induces a distribution over predictive distributions P ( y|x∗ , θ ) for each observation x∗ - each set of model parameters parameterizes a conditional distribution over class labels . The expected distribution P ( y|x∗ , D ) is obtained by marginalizing out the parameters : P ( y|x∗ , D ) = ∫ P ( y|x∗ , θ ) p ( θ|D ) dθ ( 1 ) Unfortunately , both the integral in eq . 1 and calculation of the model posterior are intractable for neural networks . Typically the model posterior distribution is approximated using either an implicit or explicit variational approximation q ( θ ) and the integral is approximated via sampling ( eq . 2 ) , using approaches such as Monte-Carlo dropout ( Gal & Ghahramani , 2016 ) : P ( y|x∗ , D ) ≈ 1 M M∑ i=1 P ( y|x∗ , θ ( i ) ) , θ ( i ) ∼ q ( θ ) ( 2 ) Bayesian approaches model distributional uncertainty through model uncertainty . By selecting an appropriate approximate inference scheme and model prior p ( θ ) Bayesian approaches aim to craft a model posterior p ( θ|D ) such that the ensemble { P ( ωc|x∗ , θ ( i ) ) } Mi=1 is consistent in-domain and becomes increasingly diverse the further away x∗ is from the region of training data . The entropy of the expected distribution P ( ωc|x∗ , D ) will indicate the total uncertainty in predictions . Measures of the diversity of the ensemble , such as Mutual Information , assess uncertainty in predictions due to model uncertainty , which yields distributional uncertainty . MI [ y , θ|x∗ , D ] ︸ ︷︷ ︸ Model Uncertainty = H [ Ep ( θ|D ) [ P ( y|x∗ , θ ) ] ] ︸ ︷︷ ︸ Total Uncertainty − Ep ( θ|D ) [ H [ P ( y|x∗ , θ ) ] ] ︸ ︷︷ ︸ Expected Data Uncertainty ( 3 ) In practice , however , for deep , distributed models with tens of million parameters , such as DNNs , it is difficult to select an appropriate approximate inference scheme to craft a model posterior which induces a distribution over distributions with the desired properties . 2.2 UNCERTAINTY ESTIMATION VIA PRIOR NETWORKS . Unlike Bayesian approaches , which indirectly specify a conditional distribution over output distributions , a Prior Network p ( π|x∗ ; θ̂ ) , proposed by ( Malinin & Gales , 2018 ) directly parametrizes a prior distribution over categorical output distributions . In this work the Dirichlet distribution ( eqn 5 ) is chosen due to its tractable analytic properties . p ( π|x∗ ; θ̂ ) = Dir ( π|α ) α = f ( x∗ ; θ̂ ) ( 4 ) A Dirichlet distribution is parameterized by its concentration parameters α , where α0 , the sum of all αc , is called the precision of the Dirichlet distribution . Higher values of α0 lead to sharper , more confident distributions : Dir ( π|α ) = Γ ( α0 ) ∏K c Γ ( αc ) K∏ c=1 παc−1c , αc > 0 , α0 = K∑ c αc ( 5 ) The predictive distribution is given by the expected categorical distribution under the Dirichlet prior : P ( ωc|x∗ ; θ̂ ) = ∫ P ( ωc|π ) p ( π|x∗ ; θ̂ ) dπ = αc α0 ( 6 ) The desired behaviors of the Prior Network can be visualized on a simplex ( fig 1 ) , where figure 1 : a describes confident behavior ( low-entropy prior focused on low-entropy output distributions ) , figure 1 : b describes uncertainty due severe class overlap and figure 1 : c describes the behaviour for an out-of-distribution input . A Prior Network is trained to display these behaviors by minimizing the following multi-task loss functions , as described in ( Malinin & Gales , 2018 ) . A flat Dirichlet is chosen as the out-of-distribution target distribution pout ( π ) , as it is the maximum entropy Dirichlet distribution . The target in-domain distribution pin ( π|x ) is constructed by specifying the mean and precision of the Dirichlet , as described in ( Malinin & Gales , 2018 ) . In order to train with this loss function , it is necessary to have out-of-distribution training data . One way of doing this is to chose , in addition to the target training data , an ’ out-of-distribution ’ dataset . For example , if a model is trained on CIFAR-10 ( Krizhevsky , 2009 ) , it is possible to use CIFAR-100 as the out-of-distribution dataset , as they don ’ t have overlapping classes . L ( θ ) = Epin ( x ) [ DKL [ pin ( π|x ) ||p ( π|x ; θ ) ] ] + Epout ( x ) [ DKL [ pout ( π ) ||p ( π|x ; θ ) ] ] ( 7 ) Given a trained Prior Network it is possible to calculate the Mutual Information using an expression similar to equation 3 : MI [ y , π|x∗ ; θ̂ ] ︸ ︷︷ ︸ Distributional Uncertainty = H [ Ep ( π|x∗ ; θ̂ ) [ P ( y|π ) ] ] ︸ ︷︷ ︸ Total Uncertainty − Ep ( π|x∗ ; θ̂ ) [ H [ P ( y|π ) ] ] ︸ ︷︷ ︸ Expected Data Uncertainty ( 8 ) This expression , as in the Bayesian case , allows total uncertainty , given by the entropy of the expected distribution , to be decomposed into data uncertainty and distributional uncertainty 3 MEASURES OF PERFORMANCE . In order to investigate detection of adversarial attacks , it is necessary to discuss how to assess the effectiveness of an adversarial attack in a scenario where detection of the attack is possible . Previous work on detection of adversarial examples ( Gong et al. , 2017 ; Grosse et al. , 2017 ; Metzen et al. , 2017 ; Carlini & Wagner , 2017 ; Smith & Gal , 2018 ) assesses the performance of detection methods separately from whether an adversarial attack was successful , and use the standard measures of adversarial success and detection performance . However , in a real deployment scenario , an attack can only be considered successful if it both affects the predictions and evades detection . In this section we develop a measure of performance to assess this . An adversarial input xadv will be defined as the output of an adversarial attack generation process Aadv applied to a natural input x : Aadv ( x , t ) = arg min x̃∈RK { L ( P ( y|x̃ ; θ̂ ) , t ) + δ ( x , x̃ ) } ( 9 ) This process is typically an optimization problem which tries to minimize a loss L ( · ) , which is a function of the input x , the model P ( y|x̃ ; θ̂ ) and a target class t , with respect to the input while also minimizing some distance δ ( · , · ) between the original image and the generated adversarial perturbation . This loss is typically the negative log-likelihood of a particular target class , in case of targeted adversarial attacks , or the ( positive ) log-likelihood of the predicted class , in case of an untargeted attack : Ltgt ( P ( y|x̃ ; θ̂ ) , t ) = − ln P ( ωt|x̃ ; θ̂ ) Lutgt ( P ( y|x̃ ; θ̂ ) , t ) = ln P ( ωt|x̃ ; θ̂ ) , ωt = arg max ω { P ( y = ω|x ; θ̂ ) } ( 10 ) The distance δ ( · , · ) , typically the L1 , L2 or L∞ norm , is minimized so that the adversarial attack is still perceived to be a natural input to a human observer . The best adversarial attack is one which minimizes the chosen loss and has the minimal deviation from the original image x . There are multiple ways in which this optimization problem can be solved ( Szegedy et al. , 2013 ; Goodfellow et al. , 2015 ; Kurakin et al. , 2016 ; Dong et al. , 2018 ) . While the process Aadv will always generate some kind of perturbation , it will not always yield a successful attack , as it may fail to affect the prediction . For the purposes of this discussion the adversarial generation process Aadv will be defined to either yield a successful adversarial attack xadv or an empty set ∅ . Aadv ( x , t ) ∈ { xadv , ∅ } ( 11 ) It is necessary to point out that here we take the perspective of the attacker - a successful attack , in the case of targeted attacks , is one which yields the target class . However , consider the perspective of the system designer who does not know whether the attacker attempts a targeted attack or not . From the point of view of the system designer , any attack which is able to affect the predictions of the network may be considered successful , regardless of whether the attack produced the attacker ’ s desired prediction . In the case of non-targeted attacks , however , the attacker ’ s and defender ’ s definition of success are the same . In a standard scenario , where there is no detection , the efficacy of an adversarial attack on a model1 can be summarized via the success rate S of the attack : S = 1 N N∑ i=1 I ( Aadv ( xi , t ) ) , I ( x ) = { 1 , x 6= ∅ 0 , x = ∅ ( 12 ) Typically S is plotted against the total maximum perturbation from the original image , measured as either the L1 , L2 or L∞ distance from the original image . Now consider using a threshold-based detection scheme where a sample is labelled ’ positive ’ if some measure of uncertainty , such as entropyH ( x ) , is less than a threshold T and ’ negative ’ if it is higher than a threshold : IT ( x ) = { 1 , T > H ( x ) 0 , T ≤ H ( x ) ( 13 ) The performance of such a scheme can be evaluated at every threshold value using the true positive rate tp ( T ) and the false positive rate fp ( T ) : tp ( T ) = 1 N N∑ i=1 IT ( xi ) fp ( T ) = 1 N N∑ i=1 IT ( Aadv ( xi , t ) ) ( 14 ) The whole range of such trade offs can be visualized using a Receiver-Operating-Characteristic ( ROC ) and the quality of the trade-off can be summarized using area under the ROC curve . However , a standard ROC curve does not give credit to the system for being robust to adversarial attacks - it doesn ’ t account for situations where the process Aadv ( · ) fails to produce a successful attack . In fact , if an adversarial attack is made against a system which has a detection scheme , it can only be considered successful if it both affects the predictions and evades detection . This condition can be summarized in the following indicator function : ÎT ( x ) = 1 , T > H ( x ) 0 , T ≤ H ( x ) 0 , x = ∅ ( 15 ) Given this indicator function , a new false positive rate f̂P ( T ) can be defined as : f̂p ( T ) = 1 N N∑ i=1 ÎT ( Aadv ( xi , t ) ) ( 16 ) This false positive rate can now be seen as a new Joint Success Rate which measures how many attacks were both successfully generated and evaded detection , given the threshold of the detection scheme . The Joint Success Rate can be plotted against the standard true positive rate on an ROC curve to visualize the possible trade-offs . This ROC curve will now give credit to the system when the attacker fails to generate a succesful attack . One possible operating point is where the false positive rate is equal to the false negative rate , also known as the Equal Error-Rate point : f̂P ( TEER ) = 1− tP ( TEER ) ( 17 ) Throughout the remainder of this work the EER false positive rate will be quoted as the Joint Success Rate .

Summary: This paper proposed the use of uncertainty measure evaluated by the prior network framework in (Malinin and Gales 2018) to detect adversarial inputs. Empirically, the best detector against three L_infinity based attacks (FGSM, BIM and MIM), is a prior network that is adversarially trained with FGSM, in both white-box and black-box settings. The results also showed superior performance over a detector based on Monte Carlo Dropout methods (MCDP). Although the idea is interesting and the presented results seem promising, there are some key experiments lacking that may prevent this work from making its claims on robustness and detectability. The detailed comments are as follows.