DETECTING ADVERSARIAL EXAMPLES VIA NEURAL FINGERPRINTING
Year: 2018

Abstract:
Deep neural networks are vulnerable to adversarial examples: input data that has been manipulated to cause dramatic model output errors. To defend against such attacks, we propose NeuralFingerprinting: a simple, yet effective method to detect adversarial examples that verifies whether model behavior is consistent with a set of fingerprints. These fingerprints are encoded into the model response during training and are inspired by the use of biometric and cryptographic signatures. In contrast to previous defenses, our method does not rely on knowledge of the adversary and can scale to large networks and input data. In this work, we 1) theoretically analyze NeuralFingerprinting for linear models and 2) show that NeuralFingerprinting significantly improves on state-of-the-art detection mechanisms for deep neural networks, by detecting the strongest known adversarial attacks with 98-100% AUCROC scores on the MNIST, CIFAR-10 and MiniImagenet (20 classes) datasets. In particular, we consider several threat models, including the most conservative one in which the attacker has full knowledge of the defender’s strategy. In all settings, the detection accuracy of NeuralFingerprinting generalizes well to unseen test-data and is robust over a wide range of hyperparameters.

text: Deep neural networks are vulnerable to adversarial examples: input data that has been manipulated to cause dramatic model output errors. To defend against such attacks, we propose NeuralFingerprinting: a simple, yet effective method to detect adversarial examples that verifies whether model behavior is consistent with a set of fingerprints. These fingerprints are encoded into the model response during training and are inspired by the use of biometric and cryptographic signatures. In contrast to previous defenses, our method does not rely on knowledge of the adversary and can scale to large networks and input data. In this work, we 1) theoretically analyze NeuralFingerprinting for linear models and 2) show that NeuralFingerprinting significantly improves on state-of-the-art detection mechanisms for deep neural networks, by detecting the strongest known adversarial attacks with 98-100% AUCROC scores on the MNIST, CIFAR-10 and MiniImagenet (20 classes) datasets. In particular, we consider several threat models, including the most conservative one in which the attacker has full knowledge of the defender’s strategy. In all settings, the detection accuracy of NeuralFingerprinting generalizes well to unseen test-data and is robust over a wide range of hyperparameters.

heading: 1 INTRODUCTION
text: Deep neural networks (DNNs) are highly effective pattern-recognition models for a wide range of tasks, e.g., computer vision (He et al., 2015), speech recognition (Xiong et al., 2016) and sequential decision-making (Silver et al., 2016). However, DNNs are vulnerable to adversarial examples: an attacker can add (small) perturbations to input data that are imperceptible to humans, and can drastically change the model’s output, introducing catastrophic errors(Szegedy et al., 2013; Goodfellow et al., 2014). A key challenge then is to make DNNs robust to adversarial attacks, so DNNs can be reliably used in noisy environments or mission-critical applications, e.g., in autonomous vehicles. There are two broad classes of solution approaches: making robust predictions and detecting adversarial examples. A major bottleneck with robust predictions can be the inherently larger sample complexity associated with robust learning (Schmidt et al., 2018). Hence, we focus on the latter and propose NeuralFingerprinting (NeuralFP): a fast, secure and effective method to detect adversarial examples.
The core idea of NeuralFP is to encode fingerprint patterns into the response of a neural network around real (i.e., non-adversarial) data. These patterns characterize the model’s expected behavior around real data and can thus be used to detect adversarial examples, around which the model outputs are not consistent with the expected fingerprint outputs (Figure 1). This approach is attractive as encoding fingerprints is feasible and simple to implement during training, and evaluating fingerprints is computationally inexpensive. Furthermore, NeuralFP is agnostic of the adversary’s attack mechanism, and differs from recent methods (Meng & Chen, 2017; Ma et al., 2018; Lee et al., 2018) that rely on auxiliary classifiers for detecting adversarial examples.
In this work, we extensively analyze and evaluate NeuralFP under various settings and threat models. We theoretically analyze the feasibility and effectiveness of NeuralFP for linear models. Furthermore, for DNNs, we experimentally validate under multiple (including the most conservative) threat models that 1) NeuralFP achieves almost perfect detection AUC-ROC scores against state-of-the-art adversarial attacks on various datasets and 2) adaptive attackers with knowledge of the fingerprints fail to craft successful attacks. To summarize, our key contributions are:

heading: 1.1 RELATED WORK
text: Several forms of defense to adversarial examples have been proposed, primarily in the setting of robust prediction, including adversarial training, detection and reconstructing images using adversarial networks (Meng & Chen, 2017). Instead, our focus is on detecting adversarial attacks.
Robust Prediction. Raghunathan et al. (2018); Kolter & Wong (2017) train on convex relaxations of the network to maximize robustness and are able to formally certify robustness for small perturbations. These methods do not yet scale to large models or high-resolution inputs. Several other defenses attempt to make robust predictions: by relying on randomization (Xie et al., 2018), introducing nonlinearity that is not differentiable (Buckman et al., 2018) and by relying on Generative Adversarial Networks (Song et al., 2018; Pouya Samangouei, 2018) for denoising images. However, recent work has shown that several of these defenses – including (Buckman et al., 2018; Meng & Chen, 2017; Song et al., 2018), amongst others – are not secure, when subject to a stronger adversary (Athalye et al., 2018; Uesato et al., 2018; Carlini & Wagner, 2017a;b). The reliance of these defenses on obfuscated gradients and obscurity is used to render the defenses inviable. Madry et al. (2017) employs robust-optimization techniques to minimize the maximal loss that can be achieved through first-order attacks, and has been shown to not cause obfuscated gradients (Athalye et al., 2018).
Robust Detection. Amongst defenses that study the detection of adversarial examples, Ma et al. (2018) detects adversarial samples using an auxiliary classifier trained to use an expansion-based measure, local intrinsic dimensionality (LID). Similar detection methods based on Kernel Density (KD), Bayesian-Uncertainty (BU) (Feinman et al., 2017) and the Mahalonobis Distance (MD) (Lee et al., 2018) using artifacts from trained networks have been considered. (Carlini & Wagner, 2017a; Athalye et al., 2018) showed LID, KD and BU to be ineffective against stronger attacks. In contrast, NeuralFP does not depend on auxiliary classifiers and performs significantly better than LID, KD and BU: 1) when the attack mechanism is unknown and 2) is robust to stronger attacks that break LID, KD and BU. A recent defense, (Lee et al., 2018), remains to be extensively studied under a stronger threat model and for signs of gradient obfuscation and security through obscurity (Athalye et al., 2018; Uesato et al., 2018), where the dynamic attacker is aware of the defense mechanism.2
1Code to reproduce experiments and the model weights: https://www.dropbox.com/sh/iq0yub74gquz1od/ AADZXUVabMvZasPrt-6-c-Wpa?dl=0.
2 Due to unavailable implementation code, we were not able to baseline against this defense.
Further, all of the above mentioned defenses (including MD) rely on auxiliary models for detection, which often adds to the vulnerability of the defense.
Adversarial Examples in Other Domains. We evaluate NeuralFP on computer vision tasks, a domain in which DNNs have proved to be effective and adversarial examples have been extensively studied. NeuralFP could potentially be employed to secure DNNs against attacks in domains such as speech recognition (Carlini & Wagner, 2018) and text comprehension (Jia & Liang, 2017). Adversarial attacks have also been studied in domains such as detection of malware (Grosse et al., 2017), spam (Nelson et al., 2008) and intrusions (Wagner & Soto, 2002). Data-poisoning (Alfeld et al., 2016) is another form of attack where maliciously crafted data is injected during training.

heading: 2 FINGERPRINTING FOR DETECTION OF ADVERSARIAL EXAMPLES
text: We consider supervised classification, where we aim to learn a model f(x; θ) from real data{ (xi, y∗i) } i∈I , where x ∈ R l is an input example (e.g., an image) and y∗ is a 1-hot label vector y∗ ∈ {0, 1}K over K classes. Here, we assume the data is sampled from a data distribution Pdata(x, y). For example, a neural network model f predicts class probabilities P (y|x; θ) as:
f(x; θ)j = P (yj |x; θ) = exph(x; θ)j∑ l exph(x; θ)l , (1)
where h(x; θ) ∈ RK are called logits and the most likely class is chosen. The optimal θ∗ can be learned by minimizing a loss function L(x, y; θ), e.g., cross-entropy loss. In this setting, an attacker attempts to construct adversarial examples x′, such that ŷ = argmaxl P (yl|x′; θ) is an incorrect class prediction (i.e., Pdata(x′, ŷ) = 0). In this work, we focus on bounded adversarial attacks, which produce small perturbations η that cause mis-classification. This is a standard threat model, for an extensive review see (Akhtar & Mian, 2018). More generally, a bounded adversarial example causes a large change in model output, i.e. for δ, ρ > 0:
‖η‖ ≤ δ, ‖f(x+ η)− f(x)‖ > ρ,
such that the class predicted by the model changes: argmax j f(x+ η)j 6= argmax j f(x)j . An example of a bounded attack is the Fast-Gradient Sign Method (FGSM) Goodfellow et al. (2014) which uses an input-space gradient: η ∝ sign ∂L(x,y;θ)∂x . Since the perturbation η is bounded and small, if x is an image, x′ can be indistinguishable from x but still cause very different predictions. Hence, our goal in this work is to efficiently detect and separate real from adversarial examples.
Algorithm 1 NeuralFP 1: Input: example x, comparison function D (see Eqn 3), threshold τ > 0, χi,j (see Eqn 2), model f .
2: Output: accept / reject. 3: if ∃j : D(x, f, χi,j) ≤ τ then 4: Return: accept # x is real 5: else 6: Return: reject # x is not real 7: end if Neural Fingerprinting. To defend DNNs against adversarial attacks, we propose NeuralFP: a method that detects whether an input example x is real or adversarial. This algorithm is summarized in Algorithm 1 and Figure 1. The key idea of NeuralFP is to detect adversarial examples by using a form of consistency check of the model output around the input. More precisely, a defender using NeuralFP chooses (a set of) input perturbation(s) ∆x around x and checks whether the model output on x+ ∆x changes according to a chosen ∆y. These chosen output-changes are encoded into the network during training. We will discuss strategies for choosing the fingerprints (∆x,∆y) in Section 2.2.
Formally, we define a fingerprint χ as the tuple χ , (∆x,∆y). For K-class classification, we define a set of N fingerprints:
χi,j = (∆xi,∆yi,j), i = 1, . . . , N, j = 1, . . . ,K, (2)
where χi,j is the ith fingerprint for class j. As noted before, the ∆xi (∆yi,j) are chosen by the defender. Note that we use the same directions ∆xi for each class j = 1, 2 . . . ,K, and that ∆yi,j can be either discrete or continuous depending on f(x; θ).
To characterize sensitivity, we define the function F (x,∆xi) to measure the change in model output. A simple choice could be F (x,∆xi) = f(x+∆xi)−f(x) (although we will use variations hereafter). To compare F (x,∆xi) with the reference output-perturbation ∆yi, we use a comparison function D:
D(x, f, χ·,j) , 1
N N∑ i=1 ‖F (x,∆xi)−∆yi,j‖2. (3)
Hence, the goal of a defender is to minimize D(x, f, χi,j) for real data x.
Detecting Adversarial Examples. NeuralFP classifies a new input x′ as real if the change in model output is close to the ∆yi,j for some class j, for all i. Here, we use a comparison function D and threshold τ > 0 to define the level of agreement required, i.e., we declare x′ real when D is below a threshold τ .
x′ is real⇔ ∃j : D(x′, f, χ·,j) ≤ τ. (4)
Hence, the NeuralFP test is defined by the data: NFP = ({ χi,j } i=1...N,j=1...K , D, τ ) .
Encoding Fingerprints. Once a defender has constructed a set of desired fingerprints χ, the chosen fingerprints can be embedded into the network’s response by adding a fingerprint regression loss during training. Given a classification model, the fingerprint loss is:
Lfp(x, y, χ; θ) = N∑ i=1 ‖F (x,∆xi)−∆yi,k‖22, (5)
where k is the ground truth class for example x and ∆yi,k are the fingerprint outputs. Note that we only train on the fingerprints for the ground truth class. The total training objective then is:
min θ ∑ (x,y) (L0(x, y; θ) + αLfp(x, y, χ; θ)) ,
where L0 is a loss function for the task (e.g. cross-entropy loss for classification) and α a positive scalar. In practice, we choose α such that it balances the task and fingerprint losses. The procedure trains the model so that the function D has low values around the real data, e.g., the train-set. We then exploit this characterization to detect adversarial test-set examples using D.
Complexity. Extra computation comes from Algorithm 1 which requires O(NK) forward passes to compute the differences F (x,∆xi). A straightforward implementation is to check (4) iteratively for all classes, and stop whenever an agreement is seen or all classes have been exhausted. However, this can be parallelized and performed in minibatches for real-time applications.
2.1 THREAT MODEL ANALYSIS
We study NeuralFP under various threat models (Table 1), where the attacker has varying levels of knowledge of NFP and model f(x; θ).
In the partial-whitebox-attack (PWA) setting, the attacker has access to θ, can query f(x; θ) and its derivatives, but does not know NFP. We evaluate under this setting in Section 3. This setting is the most commonly studied setting, and most defenses reported in
Section 1, including (Ma et al., 2018; Song et al., 2018; Lee et al., 2018), study attacks under this threat model. The PWA setting is relevant, for instance, when the attacker has a copy of the model, but the fingerprints are kept private, in accordance with Kerchhoff’s principle (Shannon, 1949) (e.g. when the NeuralFP test is run on a cloud). Although PWA is not the strongest threat scenario, we study it because of its widespread study in recent literature (Ma et al., 2018; Song et al., 2018) and it provides a direct comparison of NeuralFP against established baselines.
Reverse engineering the NFP by brute-force search can be combinatorially hard. To see this, consider a simple setting where only the ∆yi.j are unknown, and that the attacker knows that each fingerprint
± ) de-
is discrete, i.e. each component ∆yi.jk = ±1. Then the attacker would have to search over combinatorially (O(2NK)) many ∆y to find the subset of ∆y that satisfies the detection criterion in equation (4). Further, smaller τs reduce the volume of inputs accepted as real.
However, the strongest threat model assumes the attacker has full information about the defender, for e.g., when the attacker can either reverse-engineer or has access to the fingerprint data NFP, so that stronger attacks could be possible. Hence, we study this scenario extensively as well.
In the adaptive-whitebox-attack (AWA) setting, the adversary has perfect knowledge of NFP in addition to the information available under PWA. For this setting, first, in Section 2.2 we analyze NeuralFP for binary classification with linear models (SVMs), by characterizing the region of inputs that will be judged as real for a given set of fingerprints χi.j and the correspondence with the data distribution Pdata(x, y) (see Theorem 1). Second, we empirically show the robustness of NeuralFP for DNNs to adaptive attacks in Section 3.2. Ma et al. (2018); Xu et al. (2018) are other recent defenses that investigate the robustness to such adaptive attacks.
The blackbox-attack setting is the weakest setting, where the adversary has no knowledge of the model parameters or NFP. We evaluate NeuralFP under this setting in Appendix F. Additionally, we define the notion of whitebox-defense (the defender is aware of the attack mechanism) and blackbox-defense (defender has no knowledge about attacker). As such, NeuralFP is a blackboxdefense. There has been considerable progress in the blackbox-attack and whitebox-defense settings (e.g. (Liao et al., 2017)). However, progress in the blackbox-defense threat model is relatively limited.

heading: 2.2 CHOOSING AND CHARACTERIZING FINGERPRINTS: LINEAR MODELS
text: We first analyze NeuralFP on binary classification with data {( xi, y∗i )} i∈I and linear model (SVM):
f(x) = 〈w, x〉+ b, ŷ = sign f(x) ∈ {−1, 1} ,
on inputs xi ∈ Rn, where n 1 (e.g., n = 900 for MNIST). The SVM defines a hyperplane f(x) = 0 to separate positive (ŷ = +1) from negative examples (ŷ = −1). We will assume that the positive and negative examples are linearly separable by a hyperplane defined by a normal ŵ = w‖w‖ . We define the minimal and maximal distance from the examples to the hyperplane along ŵ as:
δmin± = min i:yi=±1 ∣∣〈xi, ŵ〉∣∣ , δmax± = max i:yi=±1 ∣∣〈xi, ŵ〉∣∣ . In this setting, the set of xi classified as real by fingerprints is determined by the geometry of f(x). Here, for detection, we measure the exact change in predicted class using

heading: F (x,∆x) = ∆y = sign (〈w, x+ ∆x〉+ b)− sign (〈w, x〉+ b) ∈ {−2, 0, 2} .
text: Theorem 1 (Fingerprint Detection for SVM). Consider an SVM with ŵ = w‖w‖ and separable data, and the following criteria:
(∆x1 = δmin− ŵ,∆y 1,− = 0), (6) (∆x2 = δmax− ŵ,∆y 2,− = +2), (7)
(∆x3 = −δmax+ ŵ,∆y3,+ = −2), (8) (∆x4 = −δmin+ ŵ,∆y4,+ = 0). (9)

heading: An adversarial input x′ = x± + η for which one of the following holds:
text: d(x′) > δmax+ , 0 < d(x ′) < δmin+ , d(x ′) < −δmax− , −δmin− < d(x′) < 0, (10)
will satisfy one of the above listed criteria. Here, d(x′) = 〈x ′,w〉+b ‖w‖ represents the signed distance of x′ from the separating hyperplane.
The proof for two fingerprints is shown in Figure 2. For the full proof, see the Appendix. Theorem 1 by itself does not prevent attacks parallel to the decision boundary. An adversary could push a negative example x− across the boundary to a region outside the data distribution (Pdata(x− + η, y) = 0), but within distances δmin+ and δ max + of the boundary. This would still be judged as real by using fingerprints. However, such examples could still be detected by also checking the distance of x− + η to the nearest x+ in the dataset.

heading: 2.3 CHOOSING AND CHARACTERIZING FINGERPRINTS: DNNS
text: In contrast to the linear setting, in general NeuralFP utilizes a softer notion of fingerprint matching by checking whether the model outputs match changes in normalized-logits. Specifically, for classification models f(x; θ) with logits h(x; θ) (see Eqn 1), where F is defined as:
F (x,∆xi) , ϕ(x+ ∆xi)− ϕ(x), ϕ(x) , h(x; θ) ‖h(x; θ)‖ ,
where ϕ are the normalized logits. The logits are normalized so the DNN does not fit the ∆y by making the weights arbitrarily large. Here, we use D(x,∆xi) as in (3). Note that here ∆yi,j ∈ RK .
Choosing ∆y For our experiments, we choose the ∆y so that the normalized-logit of the true class either increases or decreases along the ∆xi (analogous to the linear case). For e.g., for a 10-class classification task, if x is in class k we choose ∆y of the form:
∆ykl 6=k = −α, ∆ykl=k = β, k = 1, . . . , 10, (11)
where α, β ∈ R+. We found that reasonable choices are α = 0.25 and β = 0.75, and that the method is not sensitive to these specific choices. Further, we experimented with randomizing the signs of ∆yk and found that NeuralFP’s performance is robust to this randomization as well. 3
Choosing ∆x For nonlinear models (e.g., DNNs), the best fingerprint-direction choice is not obvious. To overcome this, we propose a straightforward extension from the linear case, using randomly sampled fingerprints. Randomization minimizes structural assumptions that may make NeuralFP exploitable. We empirically found that this provides effective detection.
For all experiments, we sampled the fingerprint directions ∆xi from a uniform distribution (∆xi ∼ U(−ε, ε)l), where l is the input dimension, and each pixel is uniformly randomly sampled in the range [−ε, ε]. Our experiments (see Figure 6) suggest that NeuralFP is not sensitive to the random values sampled. This also suggests that the ∆xi could be chosen based on a different approach.
Visualizing Fingerprints. To understand the behavior of fingerprinted non-linear models we trained a neural network (two hidden layers with 200 ReLU nodes each) to distinguish between two Gaussian balls in a 2D space (Figure 3, left). Without fingerprints, the model learns an almost linear boundary separating the two balls (compare with Figure 2). When we train to encode the fingerprints (negative of ∆ys in (11)), we observe that NeuralFP causes the model to learn a highly non-linear boundary (Figure 3, center) forming pockets of low fingerprint-loss characterizing the datadistribution (Figure 3,right). In this simple setting, NeuralFP learns to delineate the data-distribution, where the darker regions are accepted as real and the rest is rejected (Figure 4).
3 Using ∆yi,kl 6=k = −α · (2p − 1),∆y i,k l=k = β · (2p − 1), p ∼ Bern ( 1 2 ) , NeuralFP achieves AUC-ROC
> 95% on PWAs. Here p ∈ {0, 1} is a Bernoulli random variable that is resampled for each ∆xi.
3 EVALUATING NeuralFP ON DETECTION OF ADVERSARIAL ATTACKS
We now empirically validate the effectiveness of NeuralFP on a number of vision data-sets, as well as analyze the behavior and robustness of NeuralFP under the various threat models. We evaluate NeuralFP on distinguishing between unseen real images and adversarial images, for data and models of varying scales in the three threat-models discussed earlier (PWA, AWA, BA). Further, we study the sensitivity of NeuralFP to varying hyperparameters. The study of the blackbox-attack setting is deferred to Appendix F. We empirically find that:
• the defense is robust under all the three threat models, and • using NeuralFP does not diminish test accuracy.

heading: 3.1 DETECTION OF PARTIAL-WHITEBOX ATTACKS
text: We report the AUC-ROC of NeuralFP on MNIST, CIFAR-10 and MiniImagenet-20 against four state-of-the-art partial-whitebox attacks (Table 2):
• Fast Gradient Method (FGSM)Goodfellow et al. (2014) and Basic Iterative Method (BIM) Kurakin et al. (2016) are both gradient based attacks with BIM being an iterative variant of FGSM. We consider both BIM-a (iterates until misclassification has been achieved) and BIM-b (iterates 50 times).
• Jacobian-based Saliency Map Attack (JSMA) Papernot et al. (2015) perturbs pixels using a saliency map.
• Carlini-Wagner Attack (CW-L2): an optimization-based attack, is one of the strongest known attacks (Carlini & Wagner, 2016; Carlini & Wagner, 2017a), and optimizes to minimize the perturbation needed for misclassification.
Following Ma et al. (2018), for each dataset we consider a randomly sampled pre-test-set of unseen 1328 test-set images, and discard misclassified pre-test images. For the test-set of remaining images, we generate adversarial perturbations by applying each of the above mentioned attacks. We report AUC-ROC on sets composed in equal parts of the test-set and test-set adversarial samples. The AUC-ROC is computed by varying the threshold τ . See Appendix for model and dataset details.
The baselines are LID Ma et al. (2018), a recent detection based defense; KD; BU Feinman et al. (2017); all trained on FGSM, as in Ma et al. (2018). The FGSM, BIM-a, BIM-b and JSMA attacks are untargeted. We use published code for the attacks and code from Ma et al. (2018) for the baselines.
MNIST. We trained a 5-layer ConvNet to 99.2± 0.1% test-accuracy. The set of ∆xi ∈ R28×28 is chosen at random, with each pixel perturbation chosen uniformly in [−ε, ε]. For each ∆xi, if x is of label-class k, ∆yi,k ∈ R10 is chosen to be such that ∆yi,kl 6=k = −0.235 and ∆y i,k l=k = 0.73, with ‖∆y‖2 = 1. The AUC-ROCs for the best N and ε using grid-search are reported in Table 4. We see that NeuralFP achieves near-perfect detection with AUC-ROC of 99− 100% across all attacks.
Data Attack Test Bound on Adversarial Accuracy Perturbation η MNIST FGSM 11.87% ‖η‖∞ ≤ 0.4BIM-a 0.00% ‖η‖∞ ≤ 0.4 BIM-b 0.00% ‖η‖∞ ≤ 0.4 JSMA 1.73% CW-L2 0.00% CIFAR FGSM 11.39 % ‖η‖∞ ≤ 0.05BIM-a 0.00% ‖η‖∞ ≤ 0.05 BIM-b 0.00% ‖η‖∞ ≤ 0.05 JSMA 13.33% CW-L2 0.00% MiniImagenet FGSM 100% ‖η‖∞ ≤ 16/255BIM-b 100% ‖η‖∞ ≤ 16/255
Table 2: Parameters and model test-accuracy on PWattacks for different datasets (without NeuralFP test). CWL2 and JSMA attacks are unbounded. The bounds are relative to images with pixel intensities in [0, 1].
CIFAR-10. For CIFAR-10, we trained a 7-layer ConvNet (similar to (Carlini & Wagner, 2016)) to 85± 1% accuracy. The ∆xi and ∆yi,j are chosen similarly as for MNIST. Across attacks, NeuralFP outperforms LID on average by 11.77% and KD+BU, KD, BU even more substantially (Table 4). Even compared to LID-whitebox (where LID is aware of the attackers mechanism but NeuralFP is not), NeuralFP outperforms LID-whitebox on average by 8.0% (Appendix, Table 10).
MiniImagenet-20. We also test on MiniImagenet-20 with 20 classes randomly chosen from the 100 MiniImagenet classes (Vinyals et al., 2016) and trained an AlexNet network on 10,600 images (not downsampled) with 91.1% top-1 accuracy. We generated test-set adversarial examples using BIM-b with 50 steps (NIP) and FGSM. NeuralFP achieves
AUC-ROCs of > 99.5% (Table 5). We could not get results for JSMA and CW-L2, which require too much computation for tasks of this size. Results for other defenses are not reported due to time constraints & unavailable implementations.
Visualization. Figure 4 shows that fingerprint-loss differs significantly for most test and adversarial samples (across the 5 attacks in Table 2), enabling NeuralFP to achieve close to 100% AUC-ROC.

heading: 3.2 DETECTION OF ADAPTIVE-WHITEBOX ATTACKS
text: The strongest threat-model, AWA, is one where the adversary has access to the parameters of NeuralFP. To evaluate whether NeuralFP is robust in this setting, or if it relies on gradient obfuscation or obscurity (for a detailed analysis, see Section 4), we consider adaptive variants of FGSM, BIM-b, CW-L2, and SPSA (Uesato et al., 2018). Under AWA, the attacker tries to find an adversarial example
x′ that also minimizes the fingerprint-loss (5), while attacking the model trained with NeuralFP. We find that NeuralFP is robust across all AW-attacks, achieving AUC-ROCs of 96-100% (See Table 6)
Adaptive-FGSM, Adaptive-BIM-b, Adaptive-SPSA For the FGSM, BIM-b and SPSA (untargeted) attacks we mount an adaptive attack with a modified optimization objective as in (Uesato et al., 2018). Specifically, for SPSA, the loss function to minimize is:
Jadvθ (x ′, y∗, θ) + γLfp (x ′, y∗, χ; θ) ,
where Jadvθ is the original adversarial objective from (Uesato et al., 2018). For the gradient-based FGSM and BIM-B attacks, we use gradients of the following loss function:
LCE(x, y ∗, θ)− γLfp (x, y∗, χ; θ) ,
where LCE(x, y∗) is the cross-entropy loss. For each of the attacks and for each data-point, we choose the largest γ ∈ [10−3, 104] that results in a successful attack with a bisection search over γ – note that larger γ values increase the priority for minimizing Lfp. For the three adaptive attacks, the perturbation bounds are ‖η‖∞ ≤ 0.4 for MNIST and ‖η‖∞ ≤ 0.05 for CIFAR-10.
Adaptive-CW-L2 We consider two adaptive variants of the CW-L2 attack. The first variant we consider is with the modified objective function:
min x′ ‖x− x′‖2 + γ1 (LCW (x′) + γ2Lfp (x′, y∗, χ; θ)). (12)
Here, y∗ is the label-vector, γ1 ∈ [10−3, 106] and γ2 ∈ [10−3, 104] are scalar coefficients, Lfp is the fingerprint-loss we trained on and LCW is an objective encouraging misclassification. To find γ1 and γ2 we do a bisection search, first decreasing γ1 (as in Carlini & Wagner (2017a)) and then increasing γ2 gradually in a similar manner. Note that increasing γ2 increases the importance given to minimizing Lfp. The successful attack with the smallest Lfp during our search is chosen.
The second variant is similar to the one considered in (Carlini & Wagner, 2017a). Here γ2 is held at 1.0 and the successful attack with the smallest ‖x− x′‖2 is chosen during a bisection search over γ1.

heading: 4 DISCUSSION AND ANALYSIS OF EMPIRICAL RESULTS
text: Robustness Across Attacks. In addition to being robust to blackbox attacks, we find that NeuralFP is robust across a set of adaptive attacks. The fingerprint loss Lfp is differentiable, and accordingly, the gradient-free adaptive attack (SPSA) does not perform better than the gradient based ones (CW-L2, BIM-b, FGSM). Other recent defenses such as (Song et al., 2018; Liao et al., 2017), including the baselines KD and BU, were shown as not viable (< 10% accuracy) under adaptive attacks similar to the ones we consider in our work (Carlini & Wagner, 2017a; Uesato et al., 2018).
Adversarial training and robust optimization based defenses (Madry et al., 2017; Kannan et al., 2018) that perform robust prediction show decaying accuracy with increasing adversarial distortion. However, NeuralFP shows robust performance across varying adversarial distortion (See Figure 5). A possible explanation is that NeuralFP functions by rejecting regions that are further away from the data-distribution (Figure 3). Empirically, NeuralFP is less sensitive to small perturbations and more robust to larger distortions, and this is shown by the degradation in performance against the adaptive BIM & CW-L2 attacks (which produce significantly smaller distortions relative to SPSA & FGSM).
Obfuscating Gradients and Obscurity. Athalye et al. (2018) argued that several previous defenses are vulnerable as they mask the true gradients (“gradient obfuscation”). A similar phenomenon was observed in Uesato et al. (2018), where such defenses were successfully attacked (i.e., <10% defense success rate) by exploiting their reliance on obscurity and masked gradients. In contrast, our experiments suggest that NeuralFP does not rely on either obfuscating gradients or obscurity. First, Athalye et al. (2018) argue that defenses that rely on obfuscating gradients likely perform worse on iterative attacks (e.g., BIM) compared to non-iterative ones (e.g., FGSM). However, NeuralFP is robust to both types of attack (Tables 4, 10). Second, NeuralFP is robust against gradient-free (e.g., SPSA) adaptive attacks, which suggests that it does not only rely on obscured gradients. Gradientfree attacks can be used to accurately measure robustness in the presence of obscured gradients. Finally, Carlini & Wagner (2017a); Uesato et al. (2018) argue that defenses that rely on obscurity are vulnerable to adaptive attacks. However, NeuralFP is robust against a variety of adaptive attacks.
Sensitivity and Efficiency Analysis Next, we study the effect of changing N (number of fingerprint directions) and ε (magnitude of fingerprint-perturbation ∆x) on the AUC-ROC for CIFAR-10 (for analysis on MNIST, see the Appendix). Figure 7 and 9 show that NeuralFP performs well across a wide range of hyperparameters and is robust to variation in the hyperparameters for PWAs. With increasing ε, the AUC-ROC for CW-L2 decreases. As discussed before, a possible explanation is that CW-L2 produces smaller adversarial perturbations than other attacks, and for larger fingerprintdistortions ε, the fingerprints are less sensitive to those small adversarial perturbations. However, the degradation in performance is not substantial (∼ 4−8%) as we increase ε over an order of magnitude. With increasing N , the AUC-ROC generally increases across attacks. We conjecture that larger sets of fingerprints can detect perturbations in more directions and results in better detection.
Figure 6 shows that NeuralFP achieves mean AUC-ROC of 98% − 100% against all PWA, with standard deviation < 1%. This suggests that NeuralFP is not very sensitive to the chosen fingerprints.

heading: 5 CONCLUSION
text: Our experiments suggest that NeuralFP is an effective method for detecting the strongest known stateof-the-art adversarial attacks, and the high AUC-ROC scores indicate that the fingerprints generalize well to the test-set, but not to adversarial examples. Open questions are if there are attacks that can fool NeuralFP or if it is provably robust to certain attacks. Learning the fingerprints during training, and studying NeuralFP within a robust optimization framework are other interesting directions.

heading: APPENDIX A PROOF FOR THEOREM 1
text: Proof. Also see Figure 8. Consider any perturbation η = λe that is positively aligned with w, and has 〈η, e〉 = δmin− . Then for any negative example (x−,−1) (except for the support vectors that lie exactly δmin− from the hyperplane), adding the perturbation η does not change the class prediction:
sign f(x−) = −1, sign f(x− − η) = −1. (13)
The fingerprint in (6) is an example of such an η. However, if λ is large enough, that is:
〈η, e〉 = δmax− , (14)
(e.g. the fingerprint in (7)), for all negative examples (x−,−1) the class prediction will always change (except for the x− that lie exactly δmax− from the hyperplane):
sign f(x−) = −1, sign f(x− + η) = +1, (15)
Note that if η has a component smaller (or larger) than δmin± , it will exclude fewer (more) examples, e.g. those that lie closer to (farther from) the hyperplane. Similar observations hold for fingerprints (8) and (9) and the positive examples x+. Hence, it follows that for any x that lies too close to the hyperplane (closer than δmin± ), or too far (farther than δ max ± ), the model output after adding the four fingerprints will never perfectly correspond to their behavior on examples x from the data distribution. For instance, for any x that is closer than δmin+ to the hyperplane, (9) will always cause a change in class, while none was expected. Similar observations hold for the other regions in (10). Since the SVM is translation invariant parallel to the hyperplane, the fingerprints can only distinguish examples based on their distance perpendicular to the hyperplane. Hence, this choice of λs is optimal.

heading: APPENDIX B RANDOMIZED FINGERPRINTS
text: Instead of simple ∆yi,j , we can encode more complex fingerprints that are harder to guess for an adversary. For instance, we were able to train a network on CIFAR-10 using random ∆yi,j : for each ∆xi, if x is in class k:
∆yi,kl 6=k = −0.235p,∆y i,k l=k = 0.7p. (16)
Here p is a random variable with Pr(p = 1) = 0.5 and Pr(p = −1) = 0.5 that is resampled for each ∆xi, making it prohibitively hard for a brute-force attacker to guess. For this NFP, we achieve AUC-ROCs of > 95% across attacks with N = 30, ε = 0.05, without extensive tuning. The high model-capacity of neural networks allows to learn such complex patterns, that can be hard to reverse engineer. This also indicates that the proposed approach is robust and the specific choice of fingerprints or the distributions they are sampled from do not actually influence the detection performance to a large extent.

heading: APPENDIX C MODELS FOR EVALUATION
text: Note: Code for CW-adaptive is based on code fromhttps://github.com/carlini/nn_ robust_attacks. Code for the other attacks was obtained from the paper (Ma et al., 2018).
C.1 MNIST
For MNIST, we use the model described in Table 8.
C.2 CIFAR-10
For CIFAR-10, we use the model described in Table 9
C.3 MINIIMAGENET-20
MiniAlexNet Model We use a model similar to AlexNet for MiniImagenet-20. The model used is described in Table 7
MiniImagenet-20 classes We use images from the following 20 ImageNet classes for our experiments: n01558993, n02795169, n03062245, n03272010, n03980874, n04515003 n02110063, n02950826, n03146219, n03400231, n04146614, n04612504, n02443484, n02981792, n03207743, n03476684, n04443257, n07697537

heading: APPENDIX D SENSITIVITY ANALYSIS
text: APPENDIX E NeuralFP VS WHITEBOX-LID
We also compare LID against NeuralFP in the setting where LID is aware of the attack mechanism and trains against the specific attack mechanism, while NeuralFP still does not use any attacker information(Table 9). In this unfair setting, we see LID performs comparably on CW-L2, and NeuralFP outperforms LID against all other attacks. Athalye et al. (2018) provides a possible explanation for this behavior for LID, where defenses relying on obfuscated gradients perform better against stronger attacks (CW-L2) compared to one-step weaker attacks.
APPENDIX F EVALUATING NeuralFP AGAINST BLACKBOX ATTACKS
In Athalye et al. (2018), the authors indicate that testing against blackbox attacks is useful to gauge if the defense is relying on gradient masking. To explore this aspect of our defense, we construct adversarial examples for models trained without fingerprinting and then test the ability of fingerprinted models to distinguish these from unseen test-data. We find that the performance does not degrade from the whitebox-attack setting, in contrast to other defenses evaluated in Athalye et al. (2018), where the performance degrades against blackbox-attacks.

heading: APPENDIX G EVALUATED ATTACK HYPERPARAMETERS
text: Table 12 summarizes the hyperparameter settings corresponding to the experiments in Tables 4,6.

heading: APPENDIX H ADAPTIVE-PGD
text: To further investigate NeuralFP’s dependence on gradient obfuscation, we evaluate the detection performance on l0 and l2 variants of an adaptive attack with projected gradient descent (PGD) (Madry et al., 2017) as the optimization method. The adaptive attack is set-up in a similar manner to adaptiveFGSM. PGD attack fails across all sets of hyperparameters explored (See Table 13). Even at distortion bounds of ‖η‖∞ ≤ 1.0 and ‖η‖2 ≤ 60.0 PGD does not succeed in degrading the performance of NeuralFP. Obviously, at these large distortion bounds (unbounded), adversarial examples exist as part of the training dataset itself. This indicates that the optimization procedure is not performing optimally, and it remains an open question if optimization procedures can be developed to render NeuralFP vulnerable.
APPENDIX I ADAPTIVE-CW-L2 - MORE ITERATIONS, MORE BISECTION STEPS
As a sanity check to see if the defense only succeeds because of the limited number of iteration steps during our preliminary investigation, we run Adaptive-CW-L2 for 20000 iteration steps with 20 bisection search steps for the value of γ1 and 10 bisection search steps for the value of γ2. For a batch of 16 samples, this takes approximately 10 hours on an NVIDIA Tesla K80 for CIFAR-10 with N = 30. We find that even with these large number of iterations/bisection steps, NeuralFP is quite robust and the AUC-ROC does not degrade substantially. We evaluate Adaptive-CW-L2 with these hyperparameters for two sets of NeuralFP hyperparameters. Table 14 summarizes the results from these experiments. Note that since the number of samples for these evaluations is quite small, the
AUC-ROC often fluctuates a few percent depending on the set of samples randomly chosen from the unseen test-set. Increasing the number of iterations causes a drop in AUC-ROC of about 2-4% when moving from 1000 iteration steps to 20000 steps. For NeuralFP with (N = 50, ε = 0.003), the AUC-ROC corresponding to the settings in Table 12 is 96.2%. An interesting observation is that the perturbation size (mean l2 distortion) seems to grow with the number of increased iteration steps. For the vanilla CW-L2 attack (non-adaptive), the mean l2 distortion is in the range 0.15− 0.25. For further comparison, the defense in (Madry et al., 2017) has a robust-prediction accuracy of < 5% at a l2 distortion of 0.45 when evaluated against PGD.

heading: APPENDIX J ADAPTIVE-SPSA WITH 1000 ITERATIONS
text: For SPSA we investigate if increasing the number of iterations degrades performance for 3 sets of hyperparameters (See Table 15) with randomly sampled 112 unseen test-samples. We find that even with increased iterations the detection performance largely remains unaffected.

heading: APPENDIX K RANDOMIZED SAMPLING TO CHECK FOR GRADIENT
text: OBFUSCATION
A test suggested in (Athalye et al., 2018) to check for gradient-masking is sampling randomly in -balls around unseen test-data to find adversarial examples. They recommend sampling 105 points or more, and checking if any of these points can pass through the defense. Accordingly, for 100 MNIST test samples, we sample 106 points uniformly from the l∞-ball with ‖η‖∞ ≤ 0.3 and for 50 CIFAR10 test samples, we sample 5× 105 points uniformly from the l∞-ball with ‖η‖∞ ≤ 0.25. Amongst these randomly sampled points for each test point, we compute the smallest min
i (Lfp (x, i, ξ; θ)) (let
us call this value L∗fp) across points that are adversarial (i.e. cause a misclassification). Note that L ∗ fp for each test point is the minimum fingerprint-loss over all adversarial samples sampled around the point and across all labels. For CIFAR-10, we are able to find adversarial examples for 47 points and for MNIST, we are able to find adversarial examples for 52 points. Figure 10 shows the distribution of Lfp across the test points and L∗fp across the corresponding randomly sampled adversarial points. We observe that for MNIST, one single test sample has a high fingerprint-loss while all adversarial samples have high fingerprint-losses. The remaining test-samples have fingerprint-losses that are roughly 3− 30 times smaller than the adversarial examples. For CIFAR-10, the test and adversarial points are well separated, with most test samples having losses significantly smaller (roughly 10 times smaller) than the randomly sampled adversarial examples. This indicates that it is likely that NeuralFP does not simply function by misleading the gradient based attacks.

Review summaries:
This paper proposes a method for the detection of adversarial examples via what the authors term "neural fingerprinting" (NeuralFP). Essentially, a reference collection of perturbations are applied to the training data so as to learn the effects on the classification decision. The premise here is that on average, normal examples from a common class would have similar changes in the classification decision when reference perturbations are applied, whereas adversarial examples (particularly those off the local submanifold) may have a markedly different set of changes from what was expected for the targeted class. These reference perturbations as well as the anticipated output perturbations together form the "fingerprints".