Title: DETECTING ADVERSARIAL EXAMPLES VIA NEURAL FINGERPRINTING

Abstract: Deep neural networks are vulnerable to adversarial examples: input data that has been manipulated to cause dramatic model output errors. To defend against such attacks, we propose NeuralFingerprinting: a simple, yet effective method to detect adversarial examples that verifies whether model behavior is consistent with a set of fingerprints. These fingerprints are encoded into the model response during training and are inspired by the use of biometric and cryptographic signatures. In contrast to previous defenses, our method does not rely on knowledge of the adversary and can scale to large networks and input data. In this work, we 1) theoretically analyze NeuralFingerprinting for linear models and 2) show that NeuralFingerprinting significantly improves on state-of-the-art detection mechanisms for deep neural networks, by detecting the strongest known adversarial attacks with 98-100% AUC-ROC scores on the MNIST, CIFAR-10 and MiniImagenet (20 classes) datasets. In particular, we consider several threat models, including the most conservative one in which the attacker has full knowledge of the defender's strategy. In all settings, the detection accuracy of NeuralFingerprinting generalizes well to unseen test-data and is robust over a wide range of hyperparameters.

1. INTRODUCTION:

Deep neural networks (DNNs) are highly effective pattern-recognition models for a wide range of tasks, e.g., computer vision (He et al., 2015) , speech recognition (Xiong et al., 2016) and sequential decision-making (Silver et al., 2016) . However, DNNs are vulnerable to adversarial examples: an attacker can add (small) perturbations to input data that are imperceptible to humans, and can drastically change the model's output, introducing catastrophic errors (Szegedy et al., 2013; Goodfellow et al., 2014) . A key challenge then is to make DNNs robust to adversarial attacks, so DNNs can be reliably used in noisy environments or mission-critical applications, e.g., in autonomous vehicles. There are two broad classes of solution approaches: making robust predictions and detecting adversarial examples. A major bottleneck with robust predictions can be the inherently larger sample complexity associated with robust learning (Schmidt et al., 2018) . Hence, we focus on the latter and propose NeuralFingerprinting (NeuralFP): a fast, secure and effective method to detect adversarial examples.

The core idea of NeuralFP is to encode fingerprint patterns into the response of a neural network around real (i.e., non-adversarial) data. These patterns characterize the model's expected behavior around real data and can thus be used to detect adversarial examples, around which the model outputs are not consistent with the expected fingerprint outputs (Figure 1 ). This approach is attractive as encoding fingerprints is feasible and simple to implement during training, and evaluating fingerprints is computationally inexpensive. Furthermore, NeuralFP is agnostic of the adversary's attack mechanism, and differs from recent methods (Meng & Chen, 2017; Ma et al., 2018; Lee et al., 2018 ) that rely on auxiliary classifiers for detecting adversarial examples.

In this work, we extensively analyze and evaluate NeuralFP under various settings and threat models. We theoretically analyze the feasibility and effectiveness of NeuralFP for linear models. Furthermore, for DNNs, we experimentally validate under multiple (including the most conservative) threat models that 1) NeuralFP achieves almost perfect detection AUC-ROC scores against state-of-the-art adversarial attacks on various datasets and 2) adaptive attackers with knowledge of the fingerprints fail to craft successful attacks. To summarize, our key contributions are: • We present NeuralFP: a simple and secure method to detect adversarial examples that does not rely on knowledge of the attack mechanism.

• We formally characterize the effectiveness of NeuralFP for linear classification.

• We empirically show on vision tasks that NeuralFP achieves state-of-the-art near-perfect AUC-ROC scores on detecting and separating unseen test data and the strongest known adversarial attacks.

• We empirically show that the performance of NeuralFP is robust to the choice of fingerprints and is effective for a wide range of choices of hyperparameters.

• Finally, we show that NeuralFP can be robust to known adaptive-whitebox-attacks, where an attacker has full knowledge of the fingerprint data 1 .

1.1. RELATED WORK:

Several forms of defense to adversarial examples have been proposed, primarily in the setting of robust prediction, including adversarial training, detection and reconstructing images using adversarial networks (Meng & Chen, 2017) . Instead, our focus is on detecting adversarial attacks.

Robust Prediction. Raghunathan et al. (2018) ; Kolter & Wong (2017) train on convex relaxations of the network to maximize robustness and are able to formally certify robustness for small perturbations. These methods do not yet scale to large models or high-resolution inputs. Several other defenses attempt to make robust predictions: by relying on randomization (Xie et al., 2018) , introducing nonlinearity that is not differentiable (Buckman et al., 2018) and by relying on Generative Adversarial Networks (Song et al., 2018; Pouya Samangouei, 2018) for denoising images. However, recent work has shown that several of these defenses -including (Buckman et al., 2018; Meng & Chen, 2017; Song et al., 2018) , amongst others -are not secure, when subject to a stronger adversary (Athalye et al., 2018; Uesato et al., 2018; Carlini & Wagner, 2017a; b) . The reliance of these defenses on obfuscated gradients and obscurity is used to render the defenses inviable. Madry et al. (2017) employs robust-optimization techniques to minimize the maximal loss that can be achieved through first-order attacks, and has been shown to not cause obfuscated gradients (Athalye et al., 2018) .

Robust Detection. Amongst defenses that study the detection of adversarial examples, Ma et al. (2018) detects adversarial samples using an auxiliary classifier trained to use an expansion-based measure, local intrinsic dimensionality (LID). Similar detection methods based on Kernel Density (KD), Bayesian-Uncertainty (BU) (Feinman et al., 2017) and the Mahalonobis Distance (MD) (Lee et al., 2018) using artifacts from trained networks have been considered. (Carlini & Wagner, 2017a; Athalye et al., 2018) showed LID, KD and BU to be ineffective against stronger attacks. In contrast, NeuralFP does not depend on auxiliary classifiers and performs significantly better than LID, KD and BU: 1) when the attack mechanism is unknown and 2) is robust to stronger attacks that break LID, KD and BU. A recent defense, (Lee et al., 2018) , remains to be extensively studied under a stronger threat model and for signs of gradient obfuscation and security through obscurity (Athalye et al., 2018; Uesato et al., 2018) , where the dynamic attacker is aware of the defense mechanism. 2Further, all of the above mentioned defenses (including MD) rely on auxiliary models for detection, which often adds to the vulnerability of the defense.

Adversarial Examples in Other Domains. We evaluate NeuralFP on computer vision tasks, a domain in which DNNs have proved to be effective and adversarial examples have been extensively studied. NeuralFP could potentially be employed to secure DNNs against attacks in domains such as speech recognition (Carlini & Wagner, 2018 ) and text comprehension (Jia & Liang, 2017) . Adversarial attacks have also been studied in domains such as detection of malware (Grosse et al., 2017) , spam (Nelson et al., 2008) and intrusions (Wagner & Soto, 2002) . Data-poisoning (Alfeld et al., 2016) is another form of attack where maliciously crafted data is injected during training.

2. FINGERPRINTING FOR DETECTION OF ADVERSARIAL EXAMPLES:

We consider supervised classification, where we aim to learn a model f (x; θ) from real data (x i , y * i ) i∈I , where x ∈ R l is an input example (e.g., an image) and y * is a 1-hot label vector y * ∈ {0, 1} K over K classes. Here, we assume the data is sampled from a data distribution P data (x, y). For example, a neural network model f predicts class probabilities P (y|x; θ) as:

EQUATION

where h(x; θ) ∈ R K are called logits and the most likely class is chosen. The optimal θ * can be learned by minimizing a loss function L(x, y; θ), e.g., cross-entropy loss. In this setting, an attacker attempts to construct adversarial examples x , such that ŷ = argmax l P (y l |x ; θ) is an incorrect class prediction (i.e., P data (x , ŷ) = 0). In this work, we focus on bounded adversarial attacks, which produce small perturbations η that cause mis-classification. This is a standard threat model, for an extensive review see (Akhtar & Mian, 2018) . More generally, a bounded adversarial example causes a large change in model output, i.e. for δ, ρ > 0:

η ≤ δ, f (x + η) -f (x) > ρ,

such that the class predicted by the model changes: Return: reject # x is not real 7: end if Neural Fingerprinting. To defend DNNs against adversarial attacks, we propose NeuralFP: a method that detects whether an input example x is real or adversarial. This algorithm is summarized in Algorithm 1 and Figure 1 . The key idea of NeuralFP is to detect adversarial examples by using a form of consistency check of the model output around the input. More precisely, a defender using NeuralFP chooses (a set of) input perturbation(s) ∆x around x and checks whether the model output on x + ∆x changes according to a chosen ∆y. These chosen output-changes are encoded into the network during training. We will discuss strategies for choosing the fingerprints (∆x, ∆y) in Section 2.2.

argmax j f (x + η) j = argmax j f (x) j .

Formally, we define a fingerprint χ as the tuple χ (∆x, ∆y). For K-class classification, we define a set of N fingerprints:

EQUATION

where χ i,j is the i th fingerprint for class j. As noted before, the ∆x i (∆y i,j ) are chosen by the defender. Note that we use the same directions ∆x i for each class j = 1, 2 . . . , K, and that ∆y i,j can be either discrete or continuous depending on f (x; θ).

To characterize sensitivity, we define the function F (x, ∆x i ) to measure the change in model output.

A simple choice could be F (x, ∆x i ) = f (x+∆x i )-f (x) (although we will use variations hereafter).

To compare F (x, ∆x i ) with the reference output-perturbation ∆y i , we use a comparison function D:

D(x, f, χ •,j ) 1 N N i=1 F (x, ∆x i ) -∆y i,j 2 .

(3)

Hence, the goal of a defender is to minimize D(x, f, χ i,j ) for real data x.

Detecting Adversarial Examples. NeuralFP classifies a new input x as real if the change in model output is close to the ∆y i,j for some class j, for all i. Here, we use a comparison function D and threshold τ > 0 to define the level of agreement required, i.e., we declare x real when D is below a threshold τ .

x is real ⇔ ∃j :

EQUATION

Hence, the NeuralFP test is defined by the data: NFP = χ i,j i=1...N,j=1...K , D, τ .

Encoding Fingerprints. Once a defender has constructed a set of desired fingerprints χ, the chosen fingerprints can be embedded into the network's response by adding a fingerprint regression loss during training. Given a classification model, the fingerprint loss is:

EQUATION

)

where k is the ground truth class for example x and ∆y i,k are the fingerprint outputs. Note that we only train on the fingerprints for the ground truth class. The total training objective then is:

min θ (x,y) (L 0 (x, y; θ) + αL fp (x, y, χ; θ)) ,

where L 0 is a loss function for the task (e.g. cross-entropy loss for classification) and α a positive scalar. In practice, we choose α such that it balances the task and fingerprint losses. The procedure trains the model so that the function D has low values around the real data, e.g., the train-set. We then exploit this characterization to detect adversarial test-set examples using D.

Complexity. Extra computation comes from Algorithm 1 which requires O(N K) forward passes to compute the differences F (x, ∆x i ). A straightforward implementation is to check (4) iteratively for all classes, and stop whenever an agreement is seen or all classes have been exhausted. However, this can be parallelized and performed in minibatches for real-time applications.

2.1. THREAT MODEL ANALYSIS:

θ NFP × × blackbox-attack × partial-whitebox-attack adaptive-whitebox-attack

Table 1 : Threat models: attacker knows θ and / or NFP.

We study NeuralFP under various threat models (Table 1 ), where the attacker has varying levels of knowledge of NFP and model f (x; θ).

In the partial-whitebox-attack (PWA) setting, the attacker has access to θ, can query f (x; θ) and its derivatives, but does not know NFP. We evaluate under this setting in Section 3. This setting is the most commonly studied setting, and most defenses reported in Section 1, including (Ma et al., 2018; Song et al., 2018; Lee et al., 2018) , study attacks under this threat model. The PWA setting is relevant, for instance, when the attacker has a copy of the model, but the fingerprints are kept private, in accordance with Kerchhoff's principle (Shannon, 1949 ) (e.g. when the NeuralFP test is run on a cloud). Although PWA is not the strongest threat scenario, we study it because of its widespread study in recent literature (Ma et al., 2018; Song et al., 2018) and it provides a direct comparison of NeuralFP against established baselines.

Reverse engineering the NFP by brute-force search can be combinatorially hard. To see this, consider a simple setting where only the ∆y i.j are unknown, and that the attacker knows that each fingerprint is discrete, i.e. each component ∆y i.j k = ±1. Then the attacker would have to search over combinatorially (O(2 N K )) many ∆y to find the subset of ∆y that satisfies the detection criterion in equation ( 4). Further, smaller τ s reduce the volume of inputs accepted as real.

However, the strongest threat model assumes the attacker has full information about the defender, for e.g., when the attacker can either reverse-engineer or has access to the fingerprint data NFP, so that stronger attacks could be possible. Hence, we study this scenario extensively as well.

In the adaptive-whitebox-attack (AWA) setting, the adversary has perfect knowledge of NFP in addition to the information available under PWA. For this setting, first, in Section 2.2 we analyze NeuralFP for binary classification with linear models (SVMs), by characterizing the region of inputs that will be judged as real for a given set of fingerprints χ i.j and the correspondence with the data distribution P data (x, y) (see Theorem 1). Second, we empirically show the robustness of NeuralFP for DNNs to adaptive attacks in Section 3.2. Ma et al. (2018) ; Xu et al. (2018) are other recent defenses that investigate the robustness to such adaptive attacks.

The blackbox-attack setting is the weakest setting, where the adversary has no knowledge of the model parameters or NFP. We evaluate NeuralFP under this setting in Appendix F. Additionally, we define the notion of whitebox-defense (the defender is aware of the attack mechanism) and blackbox-defense (defender has no knowledge about attacker). As such, NeuralFP is a blackboxdefense. There has been considerable progress in the blackbox-attack and whitebox-defense settings (e.g. (Liao et al., 2017) ). However, progress in the blackbox-defense threat model is relatively limited.

2.2. CHOOSING AND CHARACTERIZING FINGERPRINTS: LINEAR MODELS:

We first analyze NeuralFP on binary classification with data x i , y * i i∈I and linear model (SVM):

f (x) = w, x + b, ŷ = sign f (x) ∈ {-1, 1} , on inputs x i ∈ R n , where n

1 (e.g., n = 900 for MNIST). The SVM defines a hyperplane f (x) = 0 to separate positive (ŷ = +1) from negative examples (ŷ = -1). We will assume that the positive and negative examples are linearly separable by a hyperplane defined by a normal ŵ = w w . We define the minimal and maximal distance from the examples to the hyperplane along ŵ as:

δ min ± = min i:y i =±1 x i , ŵ , δ max ± = max i:y i =±1 x i , ŵ .

In this setting, the set of x i classified as real by fingerprints is determined by the geometry of f (x).

Here, for detection, we measure the exact change in predicted class using

F (x, ∆x) = ∆y = sign ( w, x + ∆x + b) -sign ( w, x + b) ∈ {-2, 0, 2} .

Theorem 1 (Fingerprint Detection for SVM). Consider an SVM with ŵ = w w and separable data, and the following criteria:

EQUATION

EQUATION

EQUATION

EQUATION

)

An adversarial input x = x ± + η for which one of the following holds:

EQUATION

will satisfy one of the above listed criteria. Here, d(x ) = x ,w +b w represents the signed distance of x from the separating hyperplane.

The proof for two fingerprints is shown in Figure 2 . For the full proof, see the Appendix. Theorem 1 by itself does not prevent attacks parallel to the decision boundary. An adversary could push a negative example x -across the boundary to a region outside the data distribution (P data (x -+ η, y) = 0), but within distances δ min + and δ max + of the boundary. This would still be judged as real by using fingerprints. However, such examples could still be detected by also checking the distance of x -+ η to the nearest x + in the dataset.

2.3. CHOOSING AND CHARACTERIZING FINGERPRINTS: DNNS:

In contrast to the linear setting, in general NeuralFP utilizes a softer notion of fingerprint matching by checking whether the model outputs match changes in normalized-logits. Specifically, for classification models f (x; θ) with logits h(x; θ) (see Eqn 1), where F is defined as:

F (x, ∆x i ) ϕ(x + ∆x i ) -ϕ(x), ϕ(x) h(x; θ) h(x; θ) ,

where ϕ are the normalized logits. The logits are normalized so the DNN does not fit the ∆y by making the weights arbitrarily large. Here, we use D(x, ∆x i ) as in (3). Note that here ∆y i,j ∈ R K .

Choosing ∆y For our experiments, we choose the ∆y so that the normalized-logit of the true class either increases or decreases along the ∆x i (analogous to the linear case). For e.g., for a 10-class classification task, if x is in class k we choose ∆y of the form:

EQUATION

where α, β ∈ R + . We found that reasonable choices are α = 0.25 and β = 0.75, and that the method is not sensitive to these specific choices. Further, we experimented with randomizing the signs of ∆y k and found that NeuralFP's performance is robust to this randomization as well.3 

Choosing ∆x For nonlinear models (e.g., DNNs), the best fingerprint-direction choice is not obvious. To overcome this, we propose a straightforward extension from the linear case, using randomly sampled fingerprints. Randomization minimizes structural assumptions that may make NeuralFP exploitable. We empirically found that this provides effective detection.

For all experiments, we sampled the fingerprint directions ∆x i from a uniform distribution (∆x i ∼ U(-ε, ε) l ), where l is the input dimension, and each pixel is uniformly randomly sampled in the range [-ε, ε]. Our experiments (see Figure 6 ) suggest that NeuralFP is not sensitive to the random values sampled. This also suggests that the ∆x i could be chosen based on a different approach.

Visualizing Fingerprints. To understand the behavior of fingerprinted non-linear models we trained a neural network (two hidden layers with 200 ReLU nodes each) to distinguish between two Gaussian balls in a 2D space (Figure 3 , left). Without fingerprints, the model learns an almost linear boundary separating the two balls (compare with Figure 2 ). When we train to encode the fingerprints (negative of ∆ys in ( 11)), we observe that NeuralFP causes the model to learn a highly non-linear boundary (Figure 3 , center) forming pockets of low fingerprint-loss characterizing the datadistribution (Figure 3 ,right). In this simple setting, NeuralFP learns to delineate the data-distribution, where the darker regions are accepted as real and the rest is rejected (Figure 4 ). 

3. EVALUATING NeuralFP ON DETECTION OF ADVERSARIAL ATTACKS:

We now empirically validate the effectiveness of NeuralFP on a number of vision data-sets, as well as analyze the behavior and robustness of NeuralFP under the various threat models. We evaluate NeuralFP on distinguishing between unseen real images and adversarial images, for data and models of varying scales in the three threat-models discussed earlier (PWA, AWA, BA). Further, we study the sensitivity of NeuralFP to varying hyperparameters. The study of the blackbox-attack setting is deferred to Appendix F. We empirically find that:

• the defense is robust under all the three threat models, and • using NeuralFP does not diminish test accuracy.

3.1. DETECTION OF PARTIAL-WHITEBOX ATTACKS:

We report the AUC-ROC of NeuralFP on MNIST, CIFAR-10 and MiniImagenet-20 against four state-of-the-art partial-whitebox attacks (Table 2 ):

• Fast Gradient Method (FGSM) Goodfellow et al. (2014) and Basic Iterative Method (BIM) Kurakin et al. (2016) are both gradient based attacks with BIM being an iterative variant of FGSM. We consider both BIM-a (iterates until misclassification has been achieved) and BIM-b (iterates 50 times).

• Jacobian-based Saliency Map Attack (JSMA) Papernot et al. (2015) perturbs pixels using a saliency map.

• Carlini-Wagner Attack (CW-L 2 ): an optimization-based attack, is one of the strongest known attacks (Carlini & Wagner, 2016; Carlini & Wagner, 2017a) , and optimizes to minimize the perturbation needed for misclassification.

Following Ma et al. (2018) , for each dataset we consider a randomly sampled pre-test-set of unseen 1328 test-set images, and discard misclassified pre-test images. For the test-set of remaining images, we generate adversarial perturbations by applying each of the above mentioned attacks. We report AUC-ROC on sets composed in equal parts of the test-set and test-set adversarial samples. The AUC-ROC is computed by varying the threshold τ . See Appendix for model and dataset details.

The baselines are LID Ma et al. (2018) , a recent detection based defense; KD; BU Feinman et al. (2017) ; all trained on FGSM, as in Ma et al. (2018) . The FGSM, BIM-a, BIM-b and JSMA attacks are untargeted. We use published code for the attacks and code from Ma et al. (2018) for the baselines.

MNIST. We trained a 5-layer ConvNet to 99.2 ± 0.1% test-accuracy. The set of ∆x i ∈ R 28×28 is chosen at random, with each pixel perturbation chosen uniformly in [-ε, ε] . For each ∆x i , if x is of label-class k, ∆y i,k ∈ R 10 is chosen to be such that ∆y i,k l =k = -0.235 and ∆y i,k l=k = 0.73, with ∆y 2 = 1. The AUC-ROCs for the best N and ε using grid-search are reported in CIFAR-10. For CIFAR-10, we trained a 7-layer ConvNet (similar to (Carlini & Wagner, 2016) ) to 85 ± 1% accuracy. The ∆x i and ∆y i,j are chosen similarly as for MNIST. Across attacks, NeuralFP outperforms LID on average by 11.77% and KD+BU, KD, BU even more substantially (Table 4 ).

∞ ≤ 0.4 BIM-a 0.00% η ∞ ≤ 0.4 BIM-b 0.00% η ∞ ≤ 0.4 JSMA 1.73% CW-L 2 0.00% CIFAR FGSM 11.39 % η ∞ ≤ 0.05 BIM-a 0.00% η ∞ ≤ 0.05 BIM-b 0.00% η ∞ ≤ 0.05 JSMA 13.33% CW-L 2 0.00% MiniImagenet FGSM 100% η ∞ ≤ 16/255 BIM-b 100% η ∞ ≤ 16/255

Even compared to LID-whitebox (where LID is aware of the attackers mechanism but NeuralFP is not), NeuralFP outperforms LID-whitebox on average by 8.0% (Appendix, Table 10 ). MiniImagenet-20. We also test on MiniImagenet-20 with 20 classes randomly chosen from the 100 MiniImagenet classes (Vinyals et al., 2016) and trained an AlexNet network on 10,600 images (not downsampled) with 91.1% top-1 accuracy. We generated test-set adversarial examples using BIM-b with 50 steps (NIP) and FGSM. NeuralFP achieves AUC-ROCs of > 99.5% (Table 5 ). We could not get results for JSMA and CW-L 2 , which require too much computation for tasks of this size. Results for other defenses are not reported due to time constraints & unavailable implementations.

Visualization. Figure 4 shows that fingerprint-loss differs significantly for most test and adversarial samples (across the 5 attacks in Table 2 ), enabling NeuralFP to achieve close to 100% AUC-ROC.

3.2. DETECTION OF ADAPTIVE-WHITEBOX ATTACKS:

The strongest threat-model, AWA, is one where the adversary has access to the parameters of NeuralFP. To evaluate whether NeuralFP is robust in this setting, or if it relies on gradient obfuscation or obscurity (for a detailed analysis, see Section 4), we consider adaptive variants of FGSM, BIM-b, CW-L 2 , and SPSA (Uesato et al., 2018) . Under AWA, the attacker tries to find an adversarial example (Song et al., 2018; Liao et al., 2017) , including the baselines KD and BU, fail under adaptive-attacks (< 10% accuracy). For MNIST, the NeuralFP parameters for FGSM, SPSA are (ε, N ) = (0.1, 10) and (ε, N ) = (0.05, 20) for the BIM-b, CW-L 2 attacks. For CIFAR-10, the parameters are set at (ε, N ) = (0.003, 30) across attacks.

x that also minimizes the fingerprint-loss (5), while attacking the model trained with NeuralFP. We find that NeuralFP is robust across all AW-attacks, achieving AUC-ROCs of 96-100% (See Table 6 )

Adaptive-FGSM, Adaptive-BIM-b, Adaptive-SPSA For the FGSM, BIM-b and SPSA (untargeted) attacks we mount an adaptive attack with a modified optimization objective as in (Uesato et al., 2018) . Specifically, for SPSA, the loss function to minimize is:

J adv θ (x , y * , θ) + γL fp (x , y * , χ; θ) ,

where J adv θ is the original adversarial objective from (Uesato et al., 2018) . For the gradient-based FGSM and BIM-B attacks, we use gradients of the following loss function:

L CE (x, y * , θ) -γL fp (x, y * , χ; θ) ,

where L CE (x, y * ) is the cross-entropy loss. For each of the attacks and for each data-point, we choose the largest γ ∈ [10 -3, 10 4 ] that results in a successful attack with a bisection search over γnote that larger γ values increase the priority for minimizing L f p . For the three adaptive attacks, the perturbation bounds are η ∞ ≤ 0.4 for MNIST and η ∞ ≤ 0.05 for CIFAR-10.

Adaptive-CW-L 2 We consider two adaptive variants of the CW-L 2 attack. The first variant we consider is with the modified objective function:

EQUATION

Here, y * is the label-vector, γ 1 ∈ [10 -3 , 10 6 ] and γ 2 ∈ [10 -3 , 10 4 ] are scalar coefficients, L fp is the fingerprint-loss we trained on and L CW is an objective encouraging misclassification. To find γ 1 and γ 2 we do a bisection search, first decreasing γ 1 (as in Carlini & Wagner (2017a) ) and then increasing γ 2 gradually in a similar manner. Note that increasing γ 2 increases the importance given to minimizing L fp . The successful attack with the smallest L fp during our search is chosen.

The second variant is similar to the one considered in (Carlini & Wagner, 2017a) . Here γ 2 is held at 1.0 and the successful attack with the smallest x -x 2 is chosen during a bisection search over γ 1 .

4. DISCUSSION AND ANALYSIS OF EMPIRICAL RESULTS:

Robustness Across Attacks. In addition to being robust to blackbox attacks, we find that NeuralFP is robust across a set of adaptive attacks. The fingerprint loss L f p is differentiable, and accordingly, the gradient-free adaptive attack (SPSA) does not perform better than the gradient based ones (CW-L 2 , BIM-b, FGSM). Other recent defenses such as (Song et al., 2018; Liao et al., 2017) , including the baselines KD and BU, were shown as not viable (< 10% accuracy) under adaptive attacks similar to the ones we consider in our work (Carlini & Wagner, 2017a; Uesato et al., 2018) .

Adversarial training and robust optimization based defenses (Madry et al., 2017; Kannan et al., 2018) that perform robust prediction show decaying accuracy with increasing adversarial distortion. However, NeuralFP shows robust performance across varying adversarial distortion (See Figure 5 ). A possible explanation is that NeuralFP functions by rejecting regions that are further away from the data-distribution (Figure 3 ). Empirically, NeuralFP is less sensitive to small perturbations and more robust to larger distortions, and this is shown by the degradation in performance against the adaptive BIM & CW-L 2 attacks (which produce significantly smaller distortions relative to SPSA & FGSM). Obfuscating Gradients and Obscurity. Athalye et al. (2018) argued that several previous defenses are vulnerable as they mask the true gradients ("gradient obfuscation"). A similar phenomenon was observed in Uesato et al. (2018) , where such defenses were successfully attacked (i.e., <10% defense success rate) by exploiting their reliance on obscurity and masked gradients. In contrast, our experiments suggest that NeuralFP does not rely on either obfuscating gradients or obscurity. First, Athalye et al. (2018) argue that defenses that rely on obfuscating gradients likely perform worse on iterative attacks (e.g., BIM) compared to non-iterative ones (e.g., FGSM). However, NeuralFP is robust to both types of attack (Tables 4, 10 ). Second, NeuralFP is robust against gradient-free (e.g., SPSA) adaptive attacks, which suggests that it does not only rely on obscured gradients. Gradientfree attacks can be used to accurately measure robustness in the presence of obscured gradients. Finally, Carlini & Wagner (2017a) ; Uesato et al. (2018) argue that defenses that rely on obscurity are vulnerable to adaptive attacks. However, NeuralFP is robust against a variety of adaptive attacks.

Sensitivity and Efficiency Analysis Next, we study the effect of changing N (number of fingerprint directions) and ε (magnitude of fingerprint-perturbation ∆x) on the AUC-ROC for CIFAR-10 (for analysis on MNIST, see the Appendix). Figure 7 and 9 show that NeuralFP performs well across a wide range of hyperparameters and is robust to variation in the hyperparameters for PWAs. With increasing ε, the AUC-ROC for CW-L 2 decreases. As discussed before, a possible explanation is that CW-L 2 produces smaller adversarial perturbations than other attacks, and for larger fingerprintdistortions ε, the fingerprints are less sensitive to those small adversarial perturbations. However, the degradation in performance is not substantial (∼ 4 -8%) as we increase ε over an order of magnitude. With increasing N , the AUC-ROC generally increases across attacks. We conjecture that larger sets of fingerprints can detect perturbations in more directions and results in better detection.

Figure 6 shows that NeuralFP achieves mean AUC-ROC of 98% -100% against all PWA, with standard deviation < 1%. This suggests that NeuralFP is not very sensitive to the chosen fingerprints.

5. CONCLUSION:

Our experiments suggest that NeuralFP is an effective method for detecting the strongest known stateof-the-art adversarial attacks, and the high AUC-ROC scores indicate that the fingerprints generalize well to the test-set, but not to adversarial examples. Open questions are if there are attacks that can fool NeuralFP or if it is provably robust to certain attacks. Learning the fingerprints during training, and studying NeuralFP within a robust optimization framework are other interesting directions. 

APPENDIX E NeuralFP VS WHITEBOX-LID:

We also compare LID against NeuralFP in the setting where LID is aware of the attack mechanism and trains against the specific attack mechanism, while NeuralFP still does not use any attacker information(Table 9 ). In this unfair setting, we see LID performs comparably on CW-L2, and NeuralFP outperforms LID against all other attacks. Athalye et al. (2018) provides a possible explanation for this behavior for LID, where defenses relying on obfuscated gradients perform better against stronger attacks (CW-L 2 ) compared to one-step weaker attacks.

APPENDIX F EVALUATING NeuralFP AGAINST BLACKBOX ATTACKS:

In Athalye et al. (2018) , the authors indicate that testing against blackbox attacks is useful to gauge if the defense is relying on gradient masking. To explore this aspect of our defense, we construct adversarial examples for models trained without fingerprinting and then test the ability of fingerprinted models to distinguish these from unseen test-data. We find that the performance does not degrade from the whitebox-attack setting, in contrast to other defenses evaluated in Athalye et al. (2018) , where the performance degrades against blackbox-attacks. 

APPENDIX G EVALUATED ATTACK HYPERPARAMETERS:



:

Due to unavailable implementation code, we were not able to baseline against this defense.

Using ∆y i,k l =k = -α • (2p -1), ∆y i,k l=k = β • (2p -1), p ∼ Bern 1 2 ,NeuralFP achieves AUC-ROC > 95% on PWAs. Here p ∈ {0, 1} is a Bernoulli random variable that is resampled for each ∆x i .

Figures and Tables:

Figure 1: Detecting adversarial examples using NeuralFP with N = 2 fingerprints, for K-class classification. ϕ(x) is the model output. NeuralFP separates real data x (top) from adversarial data x = x + η (bottom) by comparing the sensitivity of the model to certain predefined perturbations around unseen inputs with a reference sensitivity encoded around the manifold of real images during training.

Figure 2: Geometry of fingerprints for SVMs with linearly separable data. Let d(x) be the distance of x to the decision boundary (see Thm 1). δ max ± (δ min ± ) denote the maximal (minimal) distances of the positive (x + ) and negative (x -) examples to the separating hyperplane w, x + b = 0. The fingerprint ∆x 1 with ∆x 1 , ŵ = δ min will have f (x -+ ∆x) < 0 and f (x -) < 0 for all x -in the data distribution (red region). Hence, ∆x 1 will flag all x in the regions -δ min -< d(x ) < 0 as not real (d(x ) is the signed distance of x from the boundary), since for those x it will always see a change in predicted class. Similarly, ∆x 2 with ∆x 2 , ŵ = δ max always sees a class change for real x -, flagging all x : d(x ) < -δ max as not real.

Figure 3: Left: decision boundary without fingerprints. Center: with fingerprints, red arrows indicate fingerprint-directions. The decision boundary is significantly more non-linear. Right: contour plot of fingerprint loss. NeuralFP detects dark regions as "real", while lighter regions are "fake" (tunable through τ ). Fingerprinting create valleys of low-loss delineating the data-distribution from outliers.

Detection AUC-ROC of Neu-ralFP vs partial-whitebox-attacks on MiniImagenet-20, N = 20, ε = 0.05.

Figure 7: AUC-ROC for different hyperparameters (left, middle) and ROC curves (right) on CIFAR-10 for partial-whitebox attacks. For analysis on MNIST, see Appendix. NeuralFP is robust across attacks & hyperparameters with an AUC-ROC between 95 -100%. Increasing N improves performance, indicating more fingerprints are harder to fool. Increasing the magnitude ε decreases AUC on CW-L 2 only, suggesting that as adversarial perturbations become of smaller magnitude, NeuralFP requires ε.

Figure 8: Geometry of fingerprints for SVMs with linearly separable data. Let d(x) be the distance of x to the decision boundary (see Thm 1). δ max ± (δ min ± ) denote the maximal (minimal) distances of the positive (x + ) and negative (x -) examples to the separating hyperplane w, x + b = 0. The fingerprint ∆x 1 with ∆x 1 , e = δ min will have f (x -+ ∆x) < 0 and f (x -) < 0 for all x -in the data distribution (red region). Hence, ∆x 1 will flag all x in the regions -δ min -< d(x ) < 0 as "fake", since for those x it will always see a change in predicted class. Similarly, ∆x 2 with ∆x 2 , e = δ max always sees a class change for real x -, thus flagging all x with d(x ) < -δ max

Figure9: AUC-ROC performance for different hyperparameter settings (left, middle) and ROC curves (right) on MNIST. We see that the performance of NeuralFP is robust across attacks and hyperparameters, with the AUC-ROC between 90 -100% for most settings. The AUC-ROC is lowest versus CW-L 2 , which is one of the strongest known attack.

Figure 10: Histograms depicting the distribution of losses for randomly sampled adversarial examples and test-data. Randomly sampled adversarial examples are well separated from unseen test examples. For NeuralFP, hyperparameters are (ε, N ) = (0.1, 10) and (ε, N ) = (0.003, 30) for MNIST and CIFAR-10 respectively.



Table content: <table><tr><td/><td>An example</td></tr><tr><td colspan="2">of a bounded attack is the Fast-Gradient Sign Method (FGSM) Goodfellow et al. (2014) which uses</td></tr><tr><td colspan="2">an input-space gradient: η ∝ sign ∂L(x,y;θ) ∂x image, x Algorithm 1 NeuralFP . Since the perturbation η is bounded and small, if x is an</td></tr><tr><td colspan="2">1: Input: example x, comparison function</td></tr><tr><td/><td>D (see Eqn 3), threshold τ &gt; 0, χ i,j</td></tr><tr><td/><td>(see Eqn 2), model f .</td></tr><tr><td colspan="2">2: Output: accept / reject.</td></tr><tr><td colspan="2">3: if ∃j : D(x, f, χ i,j ) ≤ τ then</td></tr><tr><td>4:</td><td>Return: accept # x is real</td></tr><tr><td colspan="2">5: else</td></tr><tr><td>6:</td><td/></tr></table>

We see that NeuralFP achieves near-perfect detection with AUC-ROC of 99 -100% across all attacks.

Table content: <table><tr><td>Data</td><td>Attack</td><td>Test</td><td>Bound on Adversarial</td></tr><tr><td/><td/><td>Accuracy</td><td>Perturbation η</td></tr><tr><td>MNIST</td><td>FGSM</td><td>11.87%</td><td>η</td></tr></table>

Parameters and model test-accuracy on PWattacks for different datasets (without NeuralFP test). CW-L 2 and JSMA attacks are unbounded. The bounds are relative to images with pixel intensities in [0, 1].

Table content: <table><tr><td/><td>1.4</td><td/></tr><tr><td>Fingerprint-Loss</td><td>0.6 0.8 1.0 1.2</td><td>Test Adversarial</td></tr><tr><td/><td>0.4</td><td/></tr><tr><td/><td>0.2</td><td/></tr><tr><td colspan="3">Figure 4: Fingerprint losses (mean over</td></tr><tr><td colspan="3">N fingerprints) on 100 random test</td></tr><tr><td colspan="3">(blue) and adversarial (red) CIFAR-10</td></tr><tr><td colspan="3">images. We see a clear separation in</td></tr><tr><td colspan="3">loss, illustrating that NeuralFP is effec-</td></tr><tr><td colspan="3">tive across many thresholds τ .</td></tr></table>



Table content: <table/>

Detection AUC-ROC for adaptive attacks on datasets MNIST and CIFAR-10. Other defenses such as

Table content: <table><tr><td>Data</td><td>Method</td><td colspan="5">Adaptive-FGSM Adaptive-BIM-b Adaptive-CW-L 2 Adaptive-CW-L 2 (γ 2 = 1) Adaptive-SPSA</td></tr><tr><td>MNIST</td><td>NeuralFP</td><td>99.91</td><td>99.37</td><td>95.04</td><td>99.17</td><td>99.94</td></tr><tr><td colspan="2">CIFAR-10 NeuralFP</td><td>99.99</td><td>99.92</td><td>97.19</td><td>97.56</td><td>99.99</td></tr></table>

Detection AUC-ROC for NeuralFP,whitebox-LID against whitebox-attackers (know model f (x; θ), but not fingerprints; see Section 2.1), on MNIST, CIFAR-10 tasks on test-set ("real") and corresponding adversarial ("fake") samples (1328 pre-test samples each). NeuralFP outperforms the baselines (LID, KD, BU) on MNIST and CIFAR-10 across all attacks, except CW-L2 where it performs comparably. A possibly explanation for LID's improved performance against stronger, iterative attacks is gradient maskingAthalye et al. (2018).

Table content: <table><tr><td/><td/><td/><td>Data</td><td>Method</td><td>FGM JSMA BIM-a BIM-b CW-L2</td></tr><tr><td/><td/><td/><td>MNIST</td><td colspan="2">LID NeuralFP 100.0 99.97 99.68 98.67</td><td>99.61 99.94</td><td>99.90 99.98</td><td>99.55 99.74</td></tr><tr><td/><td/><td/><td>CIFAR-10</td><td colspan="2">LID NeuralFP 99.96 99.91 82.38 95.87</td><td>82.30 99.91</td><td>99.78 99.95</td><td>98.94 98.87</td></tr><tr><td colspan="6">APPENDIX D SENSITIVITY ANALYSIS</td></tr><tr><td/><td>1.00</td><td/><td/><td/></tr><tr><td>AUC-ROC score</td><td>0.92 0.94 0.96 0.98</td><td>fgsm cw-l2 bim-a bim-b jsma</td><td/><td/></tr><tr><td/><td>0.03</td><td>0.06</td><td>0.1</td><td>0.3</td></tr></table>

summarizes the hyperparameter settings corresponding to the experiments in Tables 4,6.

Table content: <table/>

Peer review (iclr2019):
 This paper proposes a new technique for detecting adversarial examples by introducing "fingerprints" into the landscape while training, and exploiting the fingerprints at test time to detect adversarial examples. The idea is novel and the paper is well-written, but concerns about gradient masking prevent me from recommending acceptance just yet. Positives: The paper is extremely well-written, and the approach is clear and presented well. The authors also clearly put significant effort into the evaluation, and accurately/consistently describe threat models that they consider. The approach is also clearly novel, and is interesting. Concerns: My biggest concern is that this detection mechanism masks gradients in its loss function. The two reasons I strongly believe this is the case are (a) Figure 5 and (b) the authors themselves state that their loss is highly non-convex and that no gradient-based method may be able to find a solution. This, however, does not guarantee robustness (see [1] for why such “unfriendly” landscapes can usually be circumvented) Some concrete evaluation concerns and experiments that the authors can run to alleviate them: - Figure 5 shows adversarial robustness even against eta = 0.25---at this value of epsilon, the attacker should be able to create realistic images of other classes (even without PGD), so this suggests that the loss is somehow making examples hard to find rather than removing them. The authors should address this issue. - Showing that at a sufficiently high eta attacks start to succeed is also useful - Running SPSA, CarliniL2-FP, and PGD for *many* more iterations and using *many* more steps for binary search (right now it looks like the binary search is looking in a space of size 10^6 with 10 steps, which only has a granularity of about 5k, which means you never see any value < 5k in a bisection search, which casts into doubt all of these results) - The AUC should monotonically degrade with eta (this is another indication the attacks might not be running for long enough) - The method does not seem to be specific to L-infinity constraint. To this end, a version of Figure 5 in the L2 case would be extremely useful in understanding the detection method. I also apologize if some of these concerns about gradient masking seem unsubstantiated; that said, I tried to run the code given in the paper, but got several OOM and other errors (utils modules not found, and PyTorch deprecations), even on a machine with 8 12GB-memory GPUs. If the authors can provide instructions for running the code I will be happy to test it and alleviate some of my own concerns. I also tried to reimplement the approach, but did not manage to finish before the review deadline. If I am able to reimplement the approach I will update my review accordingly. Some smaller comments on the paper: A consolidated set of tables for attack parameters in an appendix is needed - Page 4 last paragraph line 4 find the subset that “satisfies” instead of “satisfy” - Page 5 paragraph 1 line 1 defender,for needs a space before for - Page 5 paragraph right before theorem 1 last line Here, for detection needs a , after detection - Page 7 paragraph 1 line 2 (2 hidden layers the 2 should be written two - Page 7 second last paragraph line 2 “is chosen” instead of “are chosen” - Page 7 last paragraph line 2 “across attacks” needs a , after - Page 9 table 6 label line 2 “does not shown” should be “show” instead of “shown” - Page 9 last line “measure of robustness” remove "of" [1] https://arxiv.org/pdf/1802.00420.pdf