Detecting Adversarial Examples Via Neural Fingerprinting

Deep neural networks are vulnerable to adversarial examples : input data that has been manipulated to cause dramatic model output errors . To defend against such attacks , we propose NeuralFingerprinting : a simple , yet effective method to detect adversarial examples that verifies whether model behavior is consistent with a set of fingerprints . These fingerprints are encoded into the model response during training and are inspired by the use of biometric and cryptographic signatures . In contrast to previous defenses , our method does not rely on knowledge of the adversary and can scale to large networks and input data . In this work , we 1 ) theoretically analyze NeuralFingerprinting for linear models and 2 ) show that NeuralFingerprinting significantly improves on state-of-the-art detection mechanisms for deep neural networks , by detecting the strongest known adversarial attacks with 98-100 % AUCROC scores on the MNIST , CIFAR-10 and MiniImagenet ( 20 classes ) datasets . In particular , we consider several threat models , including the most conservative one in which the attacker has full knowledge of the defender ’ s strategy . In all settings , the detection accuracy of NeuralFingerprinting generalizes well to unseen test-data and is robust over a wide range of hyperparameters . 1 INTRODUCTION . Deep neural networks ( DNNs ) are highly effective pattern-recognition models for a wide range of tasks , e.g. , computer vision ( He et al. , 2015 ) , speech recognition ( Xiong et al. , 2016 ) and sequential decision-making ( Silver et al. , 2016 ) . However , DNNs are vulnerable to adversarial examples : an attacker can add ( small ) perturbations to input data that are imperceptible to humans , and can drastically change the model ’ s output , introducing catastrophic errors ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ) . A key challenge then is to make DNNs robust to adversarial attacks , so DNNs can be reliably used in noisy environments or mission-critical applications , e.g. , in autonomous vehicles . There are two broad classes of solution approaches : making robust predictions and detecting adversarial examples . A major bottleneck with robust predictions can be the inherently larger sample complexity associated with robust learning ( Schmidt et al. , 2018 ) . Hence , we focus on the latter and propose NeuralFingerprinting ( NeuralFP ) : a fast , secure and effective method to detect adversarial examples . The core idea of NeuralFP is to encode fingerprint patterns into the response of a neural network around real ( i.e. , non-adversarial ) data . These patterns characterize the model ’ s expected behavior around real data and can thus be used to detect adversarial examples , around which the model outputs are not consistent with the expected fingerprint outputs ( Figure 1 ) . This approach is attractive as encoding fingerprints is feasible and simple to implement during training , and evaluating fingerprints is computationally inexpensive . Furthermore , NeuralFP is agnostic of the adversary ’ s attack mechanism , and differs from recent methods ( Meng & Chen , 2017 ; Ma et al. , 2018 ; Lee et al. , 2018 ) that rely on auxiliary classifiers for detecting adversarial examples . In this work , we extensively analyze and evaluate NeuralFP under various settings and threat models . We theoretically analyze the feasibility and effectiveness of NeuralFP for linear models . Furthermore , for DNNs , we experimentally validate under multiple ( including the most conservative ) threat models that 1 ) NeuralFP achieves almost perfect detection AUC-ROC scores against state-of-the-art adversarial attacks on various datasets and 2 ) adaptive attackers with knowledge of the fingerprints fail to craft successful attacks . To summarize , our key contributions are : 1.1 RELATED WORK . Several forms of defense to adversarial examples have been proposed , primarily in the setting of robust prediction , including adversarial training , detection and reconstructing images using adversarial networks ( Meng & Chen , 2017 ) . Instead , our focus is on detecting adversarial attacks . Robust Prediction . Raghunathan et al . ( 2018 ) ; Kolter & Wong ( 2017 ) train on convex relaxations of the network to maximize robustness and are able to formally certify robustness for small perturbations . These methods do not yet scale to large models or high-resolution inputs . Several other defenses attempt to make robust predictions : by relying on randomization ( Xie et al. , 2018 ) , introducing nonlinearity that is not differentiable ( Buckman et al. , 2018 ) and by relying on Generative Adversarial Networks ( Song et al. , 2018 ; Pouya Samangouei , 2018 ) for denoising images . However , recent work has shown that several of these defenses – including ( Buckman et al. , 2018 ; Meng & Chen , 2017 ; Song et al. , 2018 ) , amongst others – are not secure , when subject to a stronger adversary ( Athalye et al. , 2018 ; Uesato et al. , 2018 ; Carlini & Wagner , 2017a ; b ) . The reliance of these defenses on obfuscated gradients and obscurity is used to render the defenses inviable . Madry et al . ( 2017 ) employs robust-optimization techniques to minimize the maximal loss that can be achieved through first-order attacks , and has been shown to not cause obfuscated gradients ( Athalye et al. , 2018 ) . Robust Detection . Amongst defenses that study the detection of adversarial examples , Ma et al . ( 2018 ) detects adversarial samples using an auxiliary classifier trained to use an expansion-based measure , local intrinsic dimensionality ( LID ) . Similar detection methods based on Kernel Density ( KD ) , Bayesian-Uncertainty ( BU ) ( Feinman et al. , 2017 ) and the Mahalonobis Distance ( MD ) ( Lee et al. , 2018 ) using artifacts from trained networks have been considered . ( Carlini & Wagner , 2017a ; Athalye et al. , 2018 ) showed LID , KD and BU to be ineffective against stronger attacks . In contrast , NeuralFP does not depend on auxiliary classifiers and performs significantly better than LID , KD and BU : 1 ) when the attack mechanism is unknown and 2 ) is robust to stronger attacks that break LID , KD and BU . A recent defense , ( Lee et al. , 2018 ) , remains to be extensively studied under a stronger threat model and for signs of gradient obfuscation and security through obscurity ( Athalye et al. , 2018 ; Uesato et al. , 2018 ) , where the dynamic attacker is aware of the defense mechanism.2 1Code to reproduce experiments and the model weights : https : //www.dropbox.com/sh/iq0yub74gquz1od/ AADZXUVabMvZasPrt-6-c-Wpa ? dl=0 . 2 Due to unavailable implementation code , we were not able to baseline against this defense . Further , all of the above mentioned defenses ( including MD ) rely on auxiliary models for detection , which often adds to the vulnerability of the defense . Adversarial Examples in Other Domains . We evaluate NeuralFP on computer vision tasks , a domain in which DNNs have proved to be effective and adversarial examples have been extensively studied . NeuralFP could potentially be employed to secure DNNs against attacks in domains such as speech recognition ( Carlini & Wagner , 2018 ) and text comprehension ( Jia & Liang , 2017 ) . Adversarial attacks have also been studied in domains such as detection of malware ( Grosse et al. , 2017 ) , spam ( Nelson et al. , 2008 ) and intrusions ( Wagner & Soto , 2002 ) . Data-poisoning ( Alfeld et al. , 2016 ) is another form of attack where maliciously crafted data is injected during training . 2 FINGERPRINTING FOR DETECTION OF ADVERSARIAL EXAMPLES . We consider supervised classification , where we aim to learn a model f ( x ; θ ) from real data { ( xi , y∗i ) } i∈I , where x ∈ R l is an input example ( e.g. , an image ) and y∗ is a 1-hot label vector y∗ ∈ { 0 , 1 } K over K classes . Here , we assume the data is sampled from a data distribution Pdata ( x , y ) . For example , a neural network model f predicts class probabilities P ( y|x ; θ ) as : f ( x ; θ ) j = P ( yj |x ; θ ) = exph ( x ; θ ) j∑ l exph ( x ; θ ) l , ( 1 ) where h ( x ; θ ) ∈ RK are called logits and the most likely class is chosen . The optimal θ∗ can be learned by minimizing a loss function L ( x , y ; θ ) , e.g. , cross-entropy loss . In this setting , an attacker attempts to construct adversarial examples x′ , such that ŷ = argmaxl P ( yl|x′ ; θ ) is an incorrect class prediction ( i.e. , Pdata ( x′ , ŷ ) = 0 ) . In this work , we focus on bounded adversarial attacks , which produce small perturbations η that cause mis-classification . This is a standard threat model , for an extensive review see ( Akhtar & Mian , 2018 ) . More generally , a bounded adversarial example causes a large change in model output , i.e . for δ , ρ > 0 : ‖η‖ ≤ δ , ‖f ( x+ η ) − f ( x ) ‖ > ρ , such that the class predicted by the model changes : argmax j f ( x+ η ) j 6= argmax j f ( x ) j . An example of a bounded attack is the Fast-Gradient Sign Method ( FGSM ) Goodfellow et al . ( 2014 ) which uses an input-space gradient : η ∝ sign ∂L ( x , y ; θ ) ∂x . Since the perturbation η is bounded and small , if x is an image , x′ can be indistinguishable from x but still cause very different predictions . Hence , our goal in this work is to efficiently detect and separate real from adversarial examples . Algorithm 1 NeuralFP 1 : Input : example x , comparison function D ( see Eqn 3 ) , threshold τ > 0 , χi , j ( see Eqn 2 ) , model f . 2 : Output : accept / reject . 3 : if ∃j : D ( x , f , χi , j ) ≤ τ then 4 : Return : accept # x is real 5 : else 6 : Return : reject # x is not real 7 : end if Neural Fingerprinting . To defend DNNs against adversarial attacks , we propose NeuralFP : a method that detects whether an input example x is real or adversarial . This algorithm is summarized in Algorithm 1 and Figure 1 . The key idea of NeuralFP is to detect adversarial examples by using a form of consistency check of the model output around the input . More precisely , a defender using NeuralFP chooses ( a set of ) input perturbation ( s ) ∆x around x and checks whether the model output on x+ ∆x changes according to a chosen ∆y . These chosen output-changes are encoded into the network during training . We will discuss strategies for choosing the fingerprints ( ∆x , ∆y ) in Section 2.2 . Formally , we define a fingerprint χ as the tuple χ , ( ∆x , ∆y ) . For K-class classification , we define a set of N fingerprints : χi , j = ( ∆xi , ∆yi , j ) , i = 1 , . . . , N , j = 1 , . . . , K , ( 2 ) where χi , j is the ith fingerprint for class j . As noted before , the ∆xi ( ∆yi , j ) are chosen by the defender . Note that we use the same directions ∆xi for each class j = 1 , 2 . . . , K , and that ∆yi , j can be either discrete or continuous depending on f ( x ; θ ) . To characterize sensitivity , we define the function F ( x , ∆xi ) to measure the change in model output . A simple choice could be F ( x , ∆xi ) = f ( x+∆xi ) −f ( x ) ( although we will use variations hereafter ) . To compare F ( x , ∆xi ) with the reference output-perturbation ∆yi , we use a comparison function D : D ( x , f , χ· , j ) , 1 N N∑ i=1 ‖F ( x , ∆xi ) −∆yi , j‖2 . ( 3 ) Hence , the goal of a defender is to minimize D ( x , f , χi , j ) for real data x. Detecting Adversarial Examples . NeuralFP classifies a new input x′ as real if the change in model output is close to the ∆yi , j for some class j , for all i . Here , we use a comparison function D and threshold τ > 0 to define the level of agreement required , i.e. , we declare x′ real when D is below a threshold τ . x′ is real⇔ ∃j : D ( x′ , f , χ· , j ) ≤ τ . ( 4 ) Hence , the NeuralFP test is defined by the data : NFP = ( { χi , j } i=1 ... N , j=1 ... K , D , τ ) . Encoding Fingerprints . Once a defender has constructed a set of desired fingerprints χ , the chosen fingerprints can be embedded into the network ’ s response by adding a fingerprint regression loss during training . Given a classification model , the fingerprint loss is : Lfp ( x , y , χ ; θ ) = N∑ i=1 ‖F ( x , ∆xi ) −∆yi , k‖22 , ( 5 ) where k is the ground truth class for example x and ∆yi , k are the fingerprint outputs . Note that we only train on the fingerprints for the ground truth class . The total training objective then is : min θ ∑ ( x , y ) ( L0 ( x , y ; θ ) + αLfp ( x , y , χ ; θ ) ) , where L0 is a loss function for the task ( e.g . cross-entropy loss for classification ) and α a positive scalar . In practice , we choose α such that it balances the task and fingerprint losses . The procedure trains the model so that the function D has low values around the real data , e.g. , the train-set . We then exploit this characterization to detect adversarial test-set examples using D. Complexity . Extra computation comes from Algorithm 1 which requires O ( NK ) forward passes to compute the differences F ( x , ∆xi ) . A straightforward implementation is to check ( 4 ) iteratively for all classes , and stop whenever an agreement is seen or all classes have been exhausted . However , this can be parallelized and performed in minibatches for real-time applications . 2.1 THREAT MODEL ANALYSIS We study NeuralFP under various threat models ( Table 1 ) , where the attacker has varying levels of knowledge of NFP and model f ( x ; θ ) . In the partial-whitebox-attack ( PWA ) setting , the attacker has access to θ , can query f ( x ; θ ) and its derivatives , but does not know NFP . We evaluate under this setting in Section 3 . This setting is the most commonly studied setting , and most defenses reported in Section 1 , including ( Ma et al. , 2018 ; Song et al. , 2018 ; Lee et al. , 2018 ) , study attacks under this threat model . The PWA setting is relevant , for instance , when the attacker has a copy of the model , but the fingerprints are kept private , in accordance with Kerchhoff ’ s principle ( Shannon , 1949 ) ( e.g . when the NeuralFP test is run on a cloud ) . Although PWA is not the strongest threat scenario , we study it because of its widespread study in recent literature ( Ma et al. , 2018 ; Song et al. , 2018 ) and it provides a direct comparison of NeuralFP against established baselines . Reverse engineering the NFP by brute-force search can be combinatorially hard . To see this , consider a simple setting where only the ∆yi.j are unknown , and that the attacker knows that each fingerprint ± ) de- is discrete , i.e . each component ∆yi.jk = ±1 . Then the attacker would have to search over combinatorially ( O ( 2NK ) ) many ∆y to find the subset of ∆y that satisfies the detection criterion in equation ( 4 ) . Further , smaller τs reduce the volume of inputs accepted as real . However , the strongest threat model assumes the attacker has full information about the defender , for e.g. , when the attacker can either reverse-engineer or has access to the fingerprint data NFP , so that stronger attacks could be possible . Hence , we study this scenario extensively as well . In the adaptive-whitebox-attack ( AWA ) setting , the adversary has perfect knowledge of NFP in addition to the information available under PWA . For this setting , first , in Section 2.2 we analyze NeuralFP for binary classification with linear models ( SVMs ) , by characterizing the region of inputs that will be judged as real for a given set of fingerprints χi.j and the correspondence with the data distribution Pdata ( x , y ) ( see Theorem 1 ) . Second , we empirically show the robustness of NeuralFP for DNNs to adaptive attacks in Section 3.2 . Ma et al . ( 2018 ) ; Xu et al . ( 2018 ) are other recent defenses that investigate the robustness to such adaptive attacks . The blackbox-attack setting is the weakest setting , where the adversary has no knowledge of the model parameters or NFP . We evaluate NeuralFP under this setting in Appendix F. Additionally , we define the notion of whitebox-defense ( the defender is aware of the attack mechanism ) and blackbox-defense ( defender has no knowledge about attacker ) . As such , NeuralFP is a blackboxdefense . There has been considerable progress in the blackbox-attack and whitebox-defense settings ( e.g . ( Liao et al. , 2017 ) ) . However , progress in the blackbox-defense threat model is relatively limited .

Summary: This paper proposes a method for the detection of adversarial examples via what the authors term "neural fingerprinting" (NeuralFP). Essentially, a reference collection of perturbations are applied to the training data so as to learn the effects on the classification decision. The premise here is that on average, normal examples from a common class would have similar changes in the classification decision when reference perturbations are applied, whereas adversarial examples (particularly those off the local submanifold) may have a markedly different set of changes from what was expected for the targeted class. These reference perturbations as well as the anticipated output perturbations together form the "fingerprints".