Deep Mining: Detecting Anomalous Patterns in Neural Network Activations with Subset Scanning
Authors: Skyler Speakman
Celia Cintas
Victor Akinwande
Srihari Sridharan
Edward McFowland III
Venue:  (2020)
Keywords: anomalous pattern detection
subset scanning
node activations
adversarial noise

Abstract: This work views neural networks as data generating systems and applies anomalous pattern detection techniques on that data in order to detect when a network is processing a group of anomalous inputs.  Detecting anomalies is a critical component for multiple machine learning problems including detecting the presence of adversarial noise added to inputs. More broadly, this work is a step towards giving neural networks the ability to detect groups of out-of-distribution samples.  This work introduces ``Subset Scanning methods from the anomalous pattern detection domain to the task of detecting anomalous inputs to neural networks.  Subset Scanning allows us to answer the question: "``Which subset of inputs have larger-than-expected activations at which subset of nodes?"  Framing the adversarial detection problem this way allows us to identify systematic patterns in the activation space that span multiple adversarially noised images.  Such images are ``"weird together".  Leveraging this common anomalous pattern, we show increased detection power as the proportion of noised images increases in a test set.   Detection power and accuracy results are provided for targeted adversarial noise added to CIFAR-10 images on a 20-layer ResNet using the Basic Iterative Method attack. 

Under review as a conference paper at ICLR 2020
DEEP MINING: DETECTING ANOMALOUS PATTERNS IN NEURAL NETWORK ACTIVATIONS WITH SUBSET SCANNING
Anonymous authors Paper under double-blind review
ABSTRACT
This work views neural networks as data generating systems and applies anoma- lous pattern detection techniques that data in order to detect when a network is processing a group anomalous inputs. Detecting anomalies is a critical compo- nent for multiple machine learning problems including detecting the presence of adversarial noise added to inputs. More broadly, this work is a step towards giving neural networks the ability detect groups of out-of-distribution samples. This work introduces "Subset Scanning" methods from the anomalous pattern detec- tion domain the task of detecting anomalous inputs neural networks. Subset Scanning allows us to answer the question: "Which subset of inputs have larger- than-expected activations which subset of nodes?" Framing the adversarial de- tection problem this way allows us to identify systematic patterns in the activation space that span multiple adversarially noised images. Such images are "weird together"" Leveraging this common anomalous pattern we show increased detec- tion power as the proportion of noised images increases in a test set. Detection power and accuracy results are provided for targeted adversarial noise added CIFAR-10 images on a 20-layer ResNet using the Basic Iterative Method attack.
INTRODUCTION
The vast majority of data in the world can be thought of as created by unknown, and possibly complex, normal behavior data generating systems. But what happens when data is generated by an alternative system instead? Fraudulent records, disease outbreaks, cancerous cells on pathology slides, or adversarial noised images are all examples of data that does not come from the original, normal system. These are the interesting data points worth studying. The goal of anomalous pattern detection is to quantify, detect, and characterize the data that are generated under these alternative systems. Furthermore, subset scanning extends these ideas to consider groups of data records that may only appear anomalous when viewed together (as a subset) due to the assumption that they generated by the same alternative system. Neural networks may be viewed data generating systems. The activations are source of high- -igeeniinnii data that can be mined discover anomalous patterns. Mining activation data has implications for interpretable machine learning as well as more objective tasks such as detecting groups of
of these
samples.
This paper addresses the question: "Which of the exponentially many subset of inputs (images) have activations at which of the exponentially many subset of nodes in hidden layer of neural network?" We treat this scenario as a search problem with the goal of finding "high-scoring" subset of images x nodes by efficiently maximizing nonparametric scan statistics in the activation space of neural networks.
The primary contribution of this work is to demonstrate that nonparametric scan statistics, efficiently optimized over node-activations X multiple inputs (images), are able to quantifyy the anomalousness of a subset of those inputs (images) into a real-valued "score", This definition of anomalousness is with respect to a set of clean "background" inputs (images) that are assumed to generate normal expected patterns in the activation space of the network. Our method measures the deviance between the activations a subset of inputs (images) under evaluation and the activations generated by the background inputs. The challenging aspect of measuring deviances in the activation space

---

Under review as a conference paper at ICLR 2020
of neural networks is dealing with nigh-dimensional data, on the order of the number of nodes in a hidden layer x the number of inputs (images) under consideration. Therefore, the measure of anomalousness must be effective in capturing systematic (yet potentially subtle) deviances in high-dimensional subspace and be computationally tractable. Subset scanning meets both of these requirements (see Sec
The reward for addressing this difficult problem is an unsupervised, anomalous-input detector that can be applied to any input and any type of neural network architecture. Neural networks univer- sally rely on their activation space to encode the features of their inputs and therefore quantifying deviations from expected behavior in the activation space has broad appeal and potential beyond detecting anomalous patterns in groups of images. Furthermore, an additional output of subset scan- ning not fully explored in this paper is the subset of nodes which the subset of inputs (images) activations. These may be used to characterize the anomalous pattern
had the that is affecting the inputs.
The second contribution of this work focuses on detection of targeted adversarial noise added to inputs in order to change the labels to a target class Szegedy et al. (2013); Goodfellow et al. (2014); Papernot& & McDaniel (2016). Our critical insight to this problem the ability to detect the presence of noise (i.e. an anomalous pattern) across multiple images simultaneously. This view is grounded by the idea that targeted attacks will create subtle, but systematic, anomalous pattern of activations across multiple noised images. Therefore, during a realistic attack on a machine learning system, expect subset of the inputs to be anomalous together by sharing higher-than-expected acti- vations at similar nodes. Empirical results show that detection power drastically increases when targeted images compose 8% 10% of the data under evaluation. Detection power is near I when the proportion reaches 14%. In summary, this is the first work to apply subset scanning techniques to data generated from neural networks in order detect anomalous patterns of activations that span multiple inputs (images). To the best of our knowledge, this is the first topic to address adversarial noise detection by considering images as group rather than individually.
SUBSET SCANNING
Subset scanning treats pattern detection as a search for the "most anomalous" subset of observations in the data where anomalousness is quantified by a scoring function, F(S) (typically a log-
likelihood ratio). Therefore, we wish to efficiently identify S = arg maxy F(S) over all relevant subsets of the data S. Subset scanning has been shown to succeed where other heuristic approaches may fail (Neill 2012). "Top-down" methods look for globally interesting patterns and then identi- fies sub-partitions find smaller anomalous groups of records. These approaches may fail when the true anomaly is not evident from global aggregates. Similarly. "Bottom-up" methods look for indi- vidually anomalous data points and attempt to aggregate them into clusters. These methods may fail when the pattern is only evident by evaluating a group of data points collectively. Treating the de- tection problem as a subset scan has desirable statistical properties for maximizing detection power but the exhaustive search is infeasible for even moderately sized data sets. However, a large class of scoring functions satisfy the "Linear Time Subset Scanning" (LTSS) property which allows for exact, efficient maximization over all subsets of data without requiring exhaustive search Neill (2012). The following sub-sections highlight a class of functions that satisfy LTSS and describe how the efficient maximization process works for scanning over activations.
2.1 NONPARAMETRIC SCAN STATISTICS
This work uses nonparametric scan statistics (NPSS) that have been used in other pattern detection methods Neill & Lingwail (2007): McFowland III et al. (2013); McFowland et al. (2018); Chen & Neill (2014). These scoring functions make parametric assumptions about how activations are distributed at a node in neural network. They do, however, require baseline or background data to inform their data distribution under the null hypothesis Hu of no anomaly present.
The first step of NPSS is to compute empirical p-values for the evaluation input (e.g. images poten- tially affected with adversarial noise) by comparing it to the empirical baseline distribution gener- ated from the background inputs that are "natural" inputs known to be free of an anomalous pattern. NPSS then searches for subsets of data S in the evaluation inputs that contain the most evidence for
2

---

Under review as a conference paper at ICLR 2020
Distribution Activations for Multiple Images Single Node
Distribution of Empirical p-values for Multiple Images Single Node
06
Clean images
Clean images
Images
Noised Images
05-
Background Images
15-
Density
01
00
00
02 04
04 Empirical p-value
30
Activation Value
Figure 1: (a) Distribution of activations from three different image types at a single node under consideration. Noised images tend to have higher activations at this node than the background im- ages. Clean images have activations similar to the background images. (b) Distribution of empirical p-values for clean and noised images at the same node considered in the left panel. Higher activa- tions from noised images (as compared to the background) create larger number of smaller p- p-values. Activations from clean images (as compared to the background) create near-uniform distribution of p-values as expected under a null hypothesis.
not having been generated under Ho. This evidence is quantified by an unexpectedly large number of low empirical p-values generated by the evaluation inputs. See Figure
In our specific context, we have evaluation data D a collection of N [D] images X,, which are fed through neural network (defined by a set of nodes O = {O1..... O, producing an activation A.j. at each node Oy. The challenge we face is detecting which subset S = Sx So, if any, of the evaluation images Sx C D is anomalous as evidenced from their activation at some nodes So C O. To approach this challenge, NPSS uses as baseline data DH, of known, clean CIFAR-10 images.
Each of the M = D background images X,, also generates an activation Alf at each network node Oj-
Given D, our N x J matrix of activations for each evaluation image at each network node, and D DH corresponding M J matrix of activations for each of the background images, we obtain an empirical p-value for each Aiy: a means to measure of how anomalous the activation value of (potentially contaminated) image X; is at node Oj. This p-value Pij is the proportion of activations from the background images, Ho that are larger than the activation from the evaluation images Ag at node Oj. We note that [McFowland III et al. (2013) extend this notion to p-value ranges such that Pij is uniformly distributed between pram and Ps This current work makes a simplifying
Ex,epm (0))>=Ag)++
assumption here to only consider range by its upper bound, Pij
M+1
The matrix of activations AG is now converted into a matrix of p-values P.z. Intuitively, if an evaluation image X, "natural" (its activations are drawn from the same distribution as the baseline images) then few of the p-values generated by image X; across the network nodes-will be extreme. The key assumption for subset scanning approaches that under the alternative hypothesis of an anomaly present in the activation data then at least some subset of the activations, for the effected subset of images, will systematically appear extreme. The goal to identify this "high-scoring" subset through an efficient search procedure that maximizes a nonparametric scan statistic.
The matrix of p-values P.j from evaluation images is processed by a nonparametric scan statistic in order to identify the subset of evaluation images Sx C D whose activations at some subset of nodes So C O maximizes the scoring function maxs=Sx So F(S), where = Sx x So represents a submatrix of P.j. as this is the subset with the most statistical evidence for having been effected by an anomalous pattern.
The general form of the NPSS score function is F(S) max, (a max, 0(a, (S), N(S) where NSS) represents the number of empirical valuue contained in subset and N, (S) is the number of p-values less than (significance level) contained in subset S.
3

---

F(Sx So), where Under review as a conference paper at ICLR 2020
This generalizes submatrix, S Sx x So, intuitively.
(S) = _x,esx Eo,eso (p:jaur < a:) and N(S) = Ex,esx Σo,Eso 1.
There are well-known goodness-of-fh statistics that can be utilized in NPSS McFowland et al. (2018), the most popular is the Colmogorov-Smimoo test Kolmogorov (1933). Another option is Higher-Criticism Donoho & Jin (2004). In this work we use the Berk-Jones test statistic [Berk] & Jones (979): (a, Na, N) = N * KL aa).). where KL is the Kullback-Licbeer diver- I- gence K (1r v) = x log y + (1 x) log I-y between the observed and expected proportions of significant p-values. Berk-Jones can be interpreted as the log- log-likelihood ratio for testing whether the p-values are uniformly distributed on [0, 1] as compared following piece-wise constant step function alternative distribution, and has been shown to fulfill several optimality properties and has greater power than any weighted Kolmogorov statistic.
2.2 EFFICIENT MAXIMIZATION OF NPSS
Although NPSS provides a means to evaluate the anomalousness of a subset of node activations So for given subset of evaluation images Sx, discovering which of the 2 2' possible subsets (S = Sx x So) provides the most evidence of an anomalous pattern is computationally infeasible for moderatly sized subsets of images and nodes. However, NPSS has been shown satisfy the linear-time subset scanning (LTSS) property Neill (2012). which allows for an efficient and exact maximization over subsets of data.
For pair of functions F(S) and G(X,) representing the score of given subset S of data and the "priority" of a data record X, respectively, we have guarantee that the subset maximizing the score will be one consisting only of the top-k highest priority records, for some k between 1 and N. If we consider a data record to be an image X,, then our goal is to max Sw. F(Sx x So). for given subset of nodes So. The corresponding G(X,) function to measure the priority of an image, (x)Xxx5o) Therefore, from LTSS, is the proportion of its p-values that are less than a:: GX,),) = N(X.85))
know that m XX) F(Sx So) (X.)).... X(k) is the htth priority image.
Thus far we have described how to find the most anomalous subset of images for given subset of nodes. Because F(S) operates on a submatrix of p-value ranges, we can reorient the same process to identify an anomalous subset of nodes So for given subset of images. The goal is then to maxs. (O). F(Sx x So), for a given subset of images Sx- The corresponding G(O,)) function measure the priority of an node, is the proportion of its p-value ranges that are less than a: (O,) = N5 x0,) Therefore, from LTSS, we know that max Ci. F(SxxSo) =
x,)x
F(Sx x So), where ((k)) the ht priority node.
Given the two efficient optimization steps described above (optimizing over all subsets of images for a given subset of nodes, and optimizing over all subsets of nodes for a given subset of images), we are able compute an efficient local maximum of max ((X)...XXXX. N..... 1.0, via an iterative ascent procedure. To do so, first choose subset of attributes So S (O)...O,) uniformly at random. We then iterate between the two LTSS-enabled optimization steps described above, until convergence, at which point we have reached conditional maximum of the score func- tion (Sx is conditionally optimal given So, and So is conditionally optimal given Sx). Moreover. we can perform multiple random restarts to approach the global optimum. LTSS enabled efficient optimization of NPSS has been shown to reach the global maximum with high probability empir- ically and theoretical conditions have been provided the guarantee exact identification of the truly affected subset of data Lata|MoFowland III et al. (2013).
(Sx So),
DETECTING ADVERSARIAL NOISE WITH SUBSET SCANNING
Machine Learning models are susceptible to adversarial perturbations their input data that can cause the input to be misclassified SZegedy et al. (0013); Goodfellow et al. (2014); Kurakin et al. (2016a); Dalvi et al. (2004). There are a variety of methods to make neural networks more robust to adversarial noise, Some require retraining with altered loss functions so that adversarial images must
4

---

Under review as a conference paper at ICLR 2020
have higher perturbation in order be successful Papernot et al. (2015): Papernot & McDaniel (2016). Other detection methods rely on a supervised approach and treat the problem as classifica- tion by training on labeled noised examples Grosse et al. (2017); Gong et al. (2017); Huang et al. (2015) Another supervised approach is to use activations from hidden layers as features used by the
detector. Metzen et al. (217))
In contrast, our work treats the problem as anomalous pattern detection and operates in an unsu- pervised manner without apriori knowledge of the attack or labeled examples. We also do rely on training data augmentation or specialized training techniques. Furthermore, our work is compli- mentary to many of the defenses mentioned above. For example, if one defense type requires the noising process to makes more extreme perturbations to change the class label, then those patterns should more easily detected by subset scanning methods. A defense in Feinman al. (2017) is more similar to our work. They build kernel density estimate over background activations from the nodes in only the last hidden layer and report when an image falls in a low density part of the density estimate. This works well on MNIST, but performs poorly CIFAR-10 Carlini & Wagner (017). Our novel subset scanning approach looks at anomalousness at the node-level and across multiple inputs (images) simultaneously in order to detect patterns that span altered inputs (images).
3.1 TRAINING AND EXPERIMENT SETUP
We trained a ResNet20 (v1) residual neural network He et al. (015)) on 50,000 CIFAR-10 training images that had their mean pixel values subtracted. The test accuracy of the model was 0.9183. Future work will explore the effect of model classification accuracy the ability to detect anoma- lous patterns within its activations. For this paper we chose Resnet20 for its relatively small size, popularity, and classification accuracy.
We focus our subset scanning methods on the final convolutional layer of the last residual block. This layer contains 64 filters each containing 8x8 nodes. Therefore our analysis is the activations produced at these 4096 nodes. We do not use the spatial locations or filter membership of the nodes in the scanning process, however both of these could be useful extensions. For our adversarial experiments, we took M = [DH,1 = 9000 of the 10000 validation images and used them to generate the background activation distribution (DH,) at each of the 4096 nodes. These images form our expectation of "normal' activation behavior for the network. The remaining 1000 images were used to form groups: "Clean" (C) and "Adversarial" (A,) with Adversarial containing targeted noised versions of the 1000 images in Clean, repeated for each targeted class label / E (0...9).
We used the Basic Iterative Method (BIM) adversarial attack Kurakin et al.] (2016b) which is an improved version of the original Fast Gradient Sign Method Goodfellow et al. (2014). BIM has an E parameter which controls how far a pixel is allowed to change from its original value when noise is being added to the image. We used a value of e 0.02 in the scaled [0,1] pixel space. We also allowed the method to reach its final noised state over 10 steps with each of size 0.002. Smaller values of make the pattern subtler and harder to detect, but also less likely for the attacks to succeed in changing the class label to the target. All attacks were generated with the CleverHans package Papernot et al. (2016) 2018).
The images in A, were additionally reduced to remove any images that were originally classified as t (and therefore the noise had no effect). The attacks had near 100% success rates. Target classes 1, 4, and 7 each had I failed attack that were also removed from their respective group. Ar. The end result is each A, group contains approximately 900 images that were successfully noised from an original predicted label to label t. The 1000 images in group C' are natural and have all class labels represented (nearly) equally.
We now proceed to generating tests sets that are composed of 500 images with varying percentages for clean images and adversarial images drawn randomly from group C and A, without replacement, respectively. The percentages of adversarial images used in our test sets are 6%, 8%, 10%, 12%, and 14% for each of the target class labels. We additionally consider a special case where instead of drawing from single group A,, we uniformly draw from all A, groups. This is a particularly difficult test case because we do not expect there to be a common pattern consistent within images adversarialy targeted to separate classes. Each case is then repeated 100 times. Using p = 0.10 as an example, there would be 100 test sets of size 500 images composed of randomly selected 450 clean images (from all classes) and randomly selected 50 images noised to class 0.
0 and
5

---

Under review as a conference paper at ICLR 2020
Distribution Subset Scores Target
Receiver Operator Characteristic Curve
Cleen Images
Target 6%
Target 12%
00-
True Positive Rate
04-
Target O. 6% AUC 0.794 Target 12% AUC 0.939
6000 3000
30300
12600 34600 00 02
06
Subset Score
False Positive Rate
Figure 2: (a) Distribution of subset scores for test sets of 500 images. Test sets containing all natural images had lower than scores than test sets containing noised images. Higher proportion of noised images resulted in higher scores. (b) ROC curves for each of the noised cases as compared to the scores from test sets containing all natural images. The AUROC values for all cases reported in Table 1.
The tests sets are scanned by our method in order to determine the highest-scoring (most anomlaous) subset of the 500 images. Each test set was evaluated with 5 random restarts where random subset of nodes was generated as the seed for the iterative ascent. The most anomlaous subset returned by the scan is compared with the true state of the image (clean noised). We then report Detection Accuracy as measured by precision and recall of the most anomalous subset. These results are shown in Table 1.
In order to measure Detection Power we must model the null hypothesis of no adversarial noise also create 100 test sets containing 500 clean images only; noised images. present. To do Scanning these sets generates a distribution scores under the null hypothesis. We then compare the distribution of scores generated when noised samples of scores when scanning over subset of 500 clean images. This analysis results in an ROC curve for each case of target & and proportion p. The area under the ROC curve then measures detection power of our method for that particular case. Figure 2 demonstrates this process for target class and proportions 6% and 12%. Detection Power results are reported in Table 1.
present in the test set to the distribution
Finally, we consider experiments where images are scanned individually instead of as larger group of 500 images. The process is identical to the one describe above except the test set size is image instead of 500. We do not consider varying proportions in the individual case each set is either p = or p = We also not report precision or recall as there is no subset returned in this restricted case. We are able report detection power by comparing the distribution of scores of an individual anomalous image to the distribution scores created by scanning clean images individually.
3.2 RESULTS AND DISCUSSION
The top panel of Table provides detection power for our experiments, Detection power is measured by Area-Under-ROC curves demonstrated in Figure The ability detect targeted noise on individual images varies by class with moderate results and can be viewed as a performance floor. The focus of this work however, is the ability to detect adversarial noise across multiple images simultaneously. To that end, we show how detection power increases as the proportion of noised images increases the test sets. At proportion of 10% detection power is higher as a group than it is for individual image across all target classes. Detection is nearly perfect for all classes at 12% and above. This suggests our scanning method is identifying a subtle anomalous pattern of activations that persists across multiple noised images targeting single class.
We now focus on the "All" category which considers the test sets containing targeted examples from each of the 10 class labels. Detection power lags behind any single target class. This is because in the single target cases, our scanning method exploiting an anomalous activation pattern that consistent across multiple images. This pattern is less consistent when targeting different class

---

Under review as a conference paper at ICLR 2020
BIM (e = 0.02)
Detection Power (AUROC)
Proportion of noised images in
Target Class
Ind.
the 500 image test
6%
8%
10%
12%
14%
0.856
0.794
0.885
0.978
0.999
1.000
0.764
0.797
0.895
0.963
0.989
0.999
2
0,884
0.394
0.816
0.991
1.000
1.000
0.916
0.865
0.993
1.000
1.000
1.000
4
0.902
0.675
0.937
0.994
1.000
1.000
5
0.810
0.490
0.748
0.941
0.994
1.000
0.806
0.389
0.660
0.953
1.000
1.000
7
0.783
0.609
0.794
0.932
0.987
0.999
8
0.753
0.497
0.638
0.769
0.915
0.968
0.777
0.910
0.976
0.995
0.999
1.000
All
0.507
0.353
0.699
0.6044
0.769
BIM
Detection Accuracy: Precision
Detection Accuracy: Recall
(c ===00 0.02)
Proportion noised images
Proportion noised images int the 500 image test set
Target Class
the 500
image test set
6%
8%
10%
12%
14%
6%
8%
10%
12%
14%
0
0.196
0.275
0.364
0.494
0.620
0.828
0.903
0.923
0.941
0.946
0.216
0.286
0.352
0.415
0.480
0.800
0.831
0.862
0.866
0.875
0.228
0.363
0.638
0.794
0.852
0.894
0.950
0.971
0.973
0.974
0.267
0.470
0.618
0.726
0.772
0.947
0.969
0.966
0.966
0,962
0.232
0.341
0.505
0.642
0.746
0.889
0.920
0.941
0.940
0.941
0.196
0,290
0.401
0.541
0.654
0.836
0.899
0.913
0.935
0.944
0.180
0.349
0.561
0.673
0.725
0.733
0.896
0.955
0.963
0.964
7
0.200
0.271
0.358
0.442
0.519
0.822
0.845
0.878
0.894
0.896
0.166
0.238
0.299
0.367
0.436
0.678
0.763
0.803
0.848
0.869
0.256
0.342
0.410
0.477
0.527
0.845
0.860
0.882
0.878
0.886
All
0.109
0.160
0.240
0.255
0.337
0.403
0.484
0.625
0.566
0.702
Table 1: Detection Power and Accuracy for targeted adversarial noise added to CIFAR-10 images by the Basic Iterative Method attack. Results provided for detecting individual images and subsets of 500 images where the number of noised images varies from 6% to 14%.
labels in the same test set. This suggests that targetted noise is activating the same set of nodes despite the original images coming from different classes.
In addition to Detection Power, Table provides precision and recall measurements for the subsets of images identified by our scanning method. Precision is consistently lower than recall. We attribute this to two reasons. The first is that the 500 image test set contains targeted noised examples of single class label, as well as natural images of that same class. Therefore, we believe the subset of anomalous images is likely to include the noised images and the natural images belonging to the target class, which decreases precision. Another reason for relatively low precision is due static setting of parameter to the scanning function, O'mar- For simplicity, this value was set to 0.5 for all runs and may be interpreted as assuming up to half of the data may affected by the anomalous pattern. This is an inflated value which can be lowered if investigators had an apriori belief on the prevalence of the affected subsets in their data (i.c. the 6%--1%% used our experiments). Lowering this value would almost certainly increase precision (and lower recall).
We now consider recall measurements located in the bottom panel of Table 1. Recall is exceptionally high in our experiments. Similar to the argument for low precision, the high recall values are due to a large, static amaz value. A parammer search is feasible in supervised settings. Instead these experiments were conducted in an unsupervised form with Omaz set arbitrarily at 0.5. We
7

---

Under review as a conference paper at ICLR 2020
now highlight more subtle strength of our method with regards to recall. All things being equal, increasing the number of the noised images should decrease the recall rate as there are more noised images to miss. However, in almost all target classes we observe steady trend or increase. This demonstrates subset scanning's s innate adaptability by maintaining strong recall despite the number of noised images more than doubling (from 6% to 14%).
4 CONCLUSION
This work uses the Adversarial Noise domain as an effective narrative device to demonstrate that anomalous patterns in the activation space of neural networks can be efficiently quantified and de- tected across subset of inputs (images).
The primary contribution of this work to the data mining and deep learning literature is a novel, unsupervised anomaly detector that can be applied to any pre-trained, off-the-shelf neural network model. The method based on subset scanning which treats the detection problem as a search for the highest scoring (most anomalous) subset of node activations x inputs (images) as measured by nonparametric scan statistics. This is the first work to apply subset scanning methods to neural network activations and represents novel contribution both domains.
Nonparametric scan statistics applied to neural network activations operate on three levels of anoma- lousness. The first level is at single activation generated by an input at node. The anomalousness of this activation is quantified by its empirical p-value that reflects how large this activation is com- pared "background" of activations from known, natural images the same node. Of course, not every input that has a large activation at this node is anomalous.
We therefore must consider the second level measured by NPSS: the anomalousness of subxet of activations for single image (or equivalently, a single node). This level identifies the most anoma- lous subset of empirical p-values from a single image (or a single node). Despite the exponentially many subsets to consider, this optimization can be done exactly by only considering a linearly- -any number of subsets of activations Neil (012)). An image (or node) that has large number of small p-values is considered to be more anomalous. However, the large activations that make one im- age anomalous may occur different nodes than an image that equally anomalous with high activations at a different subet of nodes.
This consideration brings us to the third and highest level of anomalousness for NPSS applied to neural network activations: identifying a subset of inputs (images) that have saneee--thn-eppectte activations (i.e. large number of low empirical p-values) at a subset of nodes. This search pro- cedure uses the same efficient optimization step to iteratively ascend between identifying the most anomalous subset of nodes (for a given, fixed subset images) and identifying the most anomalous subset of images (for a given, fixed subset of nodes). In practice, this scanning method is able to identify a high scoring subset of images x nodes from a search space of 500 images and 4096 nodes in 3.8 seconds on average.
Efficient optimization is important in the large search space of neural network activations. However, this work also demonstrated that the subset identified by the scanning procedure relevant for an anomalous pattern of interest. The second contribution of this paper is providing empirical results that subset scanning can detect the presence of targeted adversarial noise. Furthermore the detection power, precision, and recall increase when images with targeted noise are considered together as group. To the best of our knowledge. this the first work consider adversarial noise detection across group of images. Most adversarial noise defenses only provide detection results for indi- vidual images as they fail to scale to detecting at the group level. In practical settings, if neural network is under targeted attack there will be systematic differences across the affected images. Our method is capable of detecting these subtle, but systematic, patterns at node activations across multiple images.
We also highlight that the adversarial noise detection task was performed completely unsupervised and orthogonal to the original goal of the trained ResNet: to attain high classification accuracy. This suggests that subset scanning over neural network activations will be relevant in a broad range of neural network applications.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
R. H. Berk and D. H. Jones. Goodness-of-fit test statistics that dominate the Kolmogorov statistics. Zeitschrift fär Wahhscheiniichkeesstheeeeee und Verwandte Gebiete, 47:47-59, 1979.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. CoRR, abs/1705.07263, 2017. URL httpp://axxvv.orgg////
07263
Feng Chen and Daniel B. Neill. Non-parametric statistics for event detection and forecasting in heterogeneous social media graphs. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '44, pp. 1166-1175, 2014.
Nilesh Dalvi, Pedro Domingos, Mausam, Sumit Sanghai, and Deepak Verma. Adversarial classi- fication. In ACM SIGKOD International Conference Knowledge Discovery and Data Mining (KDD), KDD '04, pp. 99-108, New York, NY, USA, 2004, ACM. ISBN 1-58113-888-11 doi: 10. 1145/1014052.101400 URL http: //oii acm. org /10 wtw.:/................. 11l...11111111150000001 1014006
David Donoho and Jiashun Jin. Higher criticism for detecting sparse heterogeneous mixtures. An- nals of Statistics, 32(3):962-99) 994,99944999999 2004.
Reuben Feinman, Ryan R. Curtin, Saurabh Shintre, and Andrew B. Gardner. Detecting adversarial samples from artifacts. CoRR, 1703.00410, 2017. URL
00410
Zhitao Gong, Wenlu Wang, and Wei-Shinn Ku. Adversarial and clean data are not twins. CoRR, abs/1704. 2017. URL httt/: hwtp://///aa//////aa// 04960
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harmessing adversarial examples. CoRR, abs/1412.6372, 2014.
6572)
Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick D. McDaniel. On the (statistical) detection of adversarial examples. CoRR, abs/1702.00688, 2017. (tRttt:::
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. redraaa for image recog-
nition. CoRR, abs/1512.03385, 2015. URL htt.:cppp//axxi...rrr//////1111.
Ruitong Huang, Bing Xu, Dale Schuurmans, and Csaba Szepesvári. Leamiin with strong adver- sary. CoRR, 1.000334 2015. URL hta.l/tppp//aaa....or/////bb....
Andrej N Kolmogorov. Sulla determinazione empirica di una legge di distribuzione. na, 1933.
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial machine learning at scale. CoRR, abs/1611.01236, 2016a.
01236
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. axddverrarial in the physical world. CoRR, abs/1607 0025333, 2016b.
E. McFowland, III. S. Somanchi, and D. B. Neill. Efficient Discovery of Heterogeneous Treatment Effects in Randomized Experiments via Anomalous Pattern Detection. ArXiv e-prints, March 2018.
Edward McFowland III, Skyler D Speakman, and Daniel B Neill. Fast generalized subset sean for anomalous pattern detection. The Journal of Machine Learning Research, 14(1):1533-1561, Jun 2013.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversar- ial perturbations. CoRR, abs/1702.044677, 2017. URL htt.:t/ss//aaa....rgg/a/////////
0 04267
Daniel B. Neill. Fast subset scan for spatial pattern detection. Journal of the Royal Statistical Society (Series B: Statistical Methodology). 74(2):337-360, 2012.

---

Under review as a conference paper at ICLR 2020
Daniel B. Neill and J. Lingwall. A nonparametric sean statistic for multivariate disease surveillance. Advances in Disease Surveillance, 4:106, 2007.
Nicolas Papernot and Patrick D. McDaniel. On the effectiveness of defensive distillation. CoRR, abs/1607.05111 2016. http:///aaax wwtp///i.o.oooooooooo...oooooere abs wttp/bb///....5.//.
Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. CoRR, abs/151 11111555000 .04508, 2015. URL
04508
Nicolas Papernot, Ian Goodfellow, Ryan Sheatsley, Reuben Feinman, and Patrick McDanicl. clev- erhans v1.0.0: adversarial machine learning library. arXiv preprint arXiv: 16100006100 2016.
Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Ku- rakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy. Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan. Zhishuai Zhang. Yi-Lin Juang. Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jonathan Uesato, Willi Gierke, Yinpeng Dong, David Berthelot, Paul Hendricks, Jonas Rauber, and Rujun Long. Technical report on the cleverhans v2.1.0 adversarial examples library. arXiv preprint orXiv:1610.00768, 2018.
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. CoRR, abs/1312.6199 2013, URL org/ abw/ wttp:/wwwwww.ro/////b/22 6199
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
This appendix is meant to provide implementation details that are relevant to readers wishing to implement their own version of our experiments. Direct code provided anonymously on Github https://githhee com wwt.:hwww...oo./aaaaaa.aaaaaaaaa http:///wwwi...aaaaaaaarrraaaaaa
We also provide code examples and psuedo code here well.
A.1
CODE SNIPPETS
This paper has focus reproducible results and has used as many vanilla settings possible. Our resnet training was done in Keras and was taken directly from htttps://gith/i change steps per epoch to
resseeeeerre__eeeeeee py We had to
model. finegeretttttrr Sieps_per_epae x x_train. shape / batch_size
Our noised attacks were generated by CleverHans and on the final 1000 images of the test set. The first 9000 were kept clean and used to form our baselines of expected activations under Ho.
from cleverhans attacks import BasicIterati iveeerrtt from cleverhans _krass import derasModelWrne Wrapper #Baasic Iterative Method
targs np. ap.array 1p,.2rr sess K.get_session!
wrap KerasModelW:aapeer (model)
bim BasicIreratiiveMethodlttttt (wrap, sess=sess)
bim params = ('eps': 0.02,
'nb_iter' 10,
'eps_iter': 0.002,
'y_arret': targs]
adv_x bim. bim.gemeratee_np(x_teer. (xt00 ;, *sbim_
Extracting activations from the network.
import keras backend as K
def get_layer (inp, out, layer_array):
return K. function [laaeerarrray layers [inp] input],
Sayer_arrry_.. www.raaaayyyaaaaaaaaaaaaaa [ayers[out] output)) #Sample extraction
f1 fi_func get_layer (0, 1, model)
#get acts from first layer
fl_act = fl_func [x_test])
#get acts from all images
The activations from the 9000 background images sorted offline. Then runtime np.searchsorted to determine where an evaluation activation would fall among the sorted background activations each node. This was used to efficiently calculate p-value ranges.
11

---

Under review as a conference paper at ICLR 2020
A.2 PSEUDOCODE
Inputs:
evaluation dataset, validation dataset, training dataset, O'max Y.
Train Neural Network from the training dataset.
2. For each image X; and each Node Oj. in both validation and evaluation datasets, compute the activation As given the network.
3. Compute the p-value range Pij = [Pmin(Ps)) Pmux (Piy)l corresponding to each activation Aij in the evaluation dataset.
4. Iterate the following steps Y times. Record the maximum value F of F(S), and the corresponding subsets of images X and nodes O over all such iterations:
(a) Initialize So + random subset of nodes.
(b) Repeat until convergence:
i. Maximize F(S)
maxasame (Sx So) over subsets of images Sx S (X1,..., for the current subset of nodes So. and set Sx f arg mx XNX F(Sx x So).
ii Maximize F(S) maxosam. Fa (Sx So) over all subsets of nodes So, for the current subset of images Sx, and set So + arg ((O...... F(8x So).
5. Output S Sx x So as well as F(S*).
12


Reviews:
[{"review_id": "rkx6zcYyqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors apply linear time subset scanning to find groups of anomalous inputs and network activations. They further use this method to detect effects of the same adversarial perturbation algorithm over a set of images.\nMajor comments:\nOverall this is quite interesting work, and seems like a promising method to detect groups of anomalies. However, this work is not complete without comparison to existing methods, the lack of which makes it impossible to evaluate its usefulness. See [1,2].\n\nMinor questions/comments:\nThe results seem to be applied to ReLU activation networks and anomalous signals are described as having higher activation that the background or the clean images. Could you clear up whether this method works for all activations or just ReLU networks?\nI’m envisioning a case where the anomalous images are all anomalous because they are all very similar. If I put 100 of the same (or very similar images) would this be something that this method could detect?\nYour results seem to suggest that adversarial perturbation methods produce adversarial images with activations that are more extreme than natural images. This doesn’t seem immediately obvious to me and I would be interested in further exploration of this direction.\nA value of \\epsilon=0.02 was used in experiments for BIM. While I agree that smaller values of \\epsilon are harder to detect it would be useful to have an evaluation of performance over smaller values of \\epsilon even if these did not perform as well.\n[1] Xiong, L., P ́oczos, B., Schneider, J., Connolly, A., VanderPlas, J.: Hierarchical probabilistic models for group anomaly detection. In: AISTATS 2011 (2011)\n[2] Chalapathy R., Toth E., Chawla S. (2019) Group Anomaly Detection Using Deep Generative Models. In: ECML PKDD 2018. Lecture Notes in Computer Science, vol 11051. Springer, Cham", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "rkletIhsYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a scheme to detect the presence of anomalous inputs, such as samples designed adversarially for deep learning tasks, that is based on a \"subset scanning\" approach to detect anomalous activations in the deep learning network. The paper is considering a very interesting problem and provides the suitable application of an approach previously developed for pattern detection. The approach is motivated by p-value statistics of the activation patterns in the deep learning network under the \"null hypothesis\" of a non-anomalous input.\n\nMy rating is \"weak reject\" because the explanation of the subset scanning approach is not clear. The paper describes two functionals F and G that are defined over subsets of the data and activations. Section 2.2 mentions a method that maximizes F over data and activation subsets by maximizing F over the data subsets under a fixed activation subset and vice versa, and iterating over these two. The section also discusses the function G that measures the priority of an image, but does not describe how the priority is known to provide an optimal solution for the former optimization over F. Another function with the same name is defined to measure the priority of an activation node, and again it is not clear why this will provide an optimal solution for the latter optimization. It would have been helpful to establish (or at least instantiate) the optimality results for this approach; only a citation is provided.\n\nThe applicability of the approach is limited to the requirement that multiple adversarial samples be present for accurate detection. It would appear that other approaches to anomaly detection do not require this condition, particularly if they are designed to operate on individual samples. Furthermore, the requirement for the \"same system\" to design the anomalous samples limits the applicability of the anomaly detection approach to cases like the single adversarial design detection setting studied here.\n\nSome questions for the authors:\nIs there a reason why the figures (Fig. 2) show results only for a single class?\nIs there a reason why no other anomaly detection algorithms for the activation patterns were used in comparisons? How about other adversarial noise detection algorithms?\nIs there intuition behind the difference in performance when all target classes vs. a single target class is used? Does this mean that a multi-class adversary generation would be too diverse for the proposed approach to detect it effectively?\n\nMinor comments\n\"Weird together\" - is too informal, and it's not clear why this phrasing is needed. Consider replacing or explaining.\nTypos \"anomlaous\" multiple times.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Hyx_XCKQcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is the first paper, in my knowledge, that introduces the problem of identifying anomalous (or corrupted) subset of data input to a neural network. The corrupted inputs are identified vis-a-vis a set of “clean” background set (e.g., the training/ validation data set). Experimental evaluation is performed on the problem of identifying the subset of noisy CIFAR-10 images created by adding targeted adversarial perturbations.\n\nTo achieve this, the problem is posed as that of subset detection and subset scanning approaches are adapted for the same. The activation of a node under consideration is converted to a p-value (how extreme the activation is, for that input data, w.r.t. the background set). \n\nA goodness of fit is then defined for any subset using the Berk-Jones test statistic which is used to create an NPSS scoring function to be maximized over the set of all subsets of the product set – (set of nodes x set of input images). The authors suggest that this combinatorial problem can be addressed efficiently as NPSS has been shown before to have the linear-time subset scanning (LTSS) property (Neill 2012). \n\nHowever, since the search is now over a product subspace and the LTSS property holds for individual subspaces (selection of activation nodes or input images while the other is held fixed), the LTSS property doesn’t trivially extend to the product space. The authors suggest an algorithm which alternatively iterates between the two LTSS steps. Since the above algorithm is not guaranteed to obtain the global optimum, multiple random restarts are suggested. This is the weakest part of the paper – neither is a formal proof of convergence provided, nor is the efficiency of the propped algorithm demonstrated empirically. Also, the performance of the algorithm (accuracy) can only be weakly used to judge the goodness of the local optimum obtained as it confounds the power of NPSS with the gap between the local and global optimum.\n\nContributions\n\nIn summary, the main contribution of the paper is the introduction of the problem of the identification of anomalous subsets of adversarially corrupted examples on deep neural networks and the first approach to address the problem via a NPSS scoring function that (partially) satisfies the LTSS property suggesting the existence of efficient (and exact?) algorithms for solving the problem. The work has incremental novelty as it seems to be largely based on prior art (not on DNNs). Initial results look promising.  \n\nNegatives \n\nHowever, I have the following concerns:\n\n- (Clarity) Mathematical formulation for the NPSS score function and the optimization problem to be solved should be clearly stated (and not inside paragraphs of text).\n\n- The proposed algorithm is ad hoc - an obvious extension of the standard algorithm that exploits the LTSS property to the product of two set spaces. The algorithm may not find a good optimum. No proof, analysis or study of its properties – optimality gap, convergence, and, efficiency is presented. \n\n- At deployment, the attacks may be non-targeted or the target may be different for different corrupted inputs. This will dilute the signature as the set of nodes where the ‘extreme values’ appear may not be the same across the corrupted examples. It is not clear if it is the case if the target is the same. In this case, subset scanning may not work. This assumption should’ve been directly tested.\n\n- ‘Berk-Jones can be interpreted as the log-likelihood ratio for testing whether p-values are uniformly distributed on [0,1] vs. … alternate distribution’. Why should this be the test? P-values on clean, background data can be directly obtained and whatever that distribution can be empirically tested or uniformity can be tested against after a whitening transformation?\n\n- Comparison with reasonable ‘baselines’ – for example, with Feinman et al. 2017? Comparison with current state of the art defenses on CIFAR10?\n\n- Why is the BIM attack (Kurakin et al. 2016b) used? There are more powerful attacks available now?\n\n- The NPSS scoring function is supposed to maximize over significance values \\alpha. What range was used and how was it decided? In the discussion on Detection Power reported in Table 1, it is suggested that different \\alpha_max will trade off precision vs recall. This has not been experimented with. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the use of the subset scanning to the detection of anomalous patterns in the input to a neural network. The paper has received mixed reviews (one positive and two negatives). The reviewers agree that the idea is interesting, has novelty, and is worth investigating. At the same time they raise issues about the clarity and the lack of comparisons with baselines. Despite a very detailed rebuttal, both of the negative reviewers still feel that addressing their concerns through paper revision would be needed for acceptance.
Decision: Reject