Deep Mining: Detecting Anomalous Patterns in Neural Network Activations with Subset Scanning

1 INTRODUCTION . The vast majority of data in the world can be thought of as created by unknown , and possibly complex , normal behavior of data generating systems . But what happens when data is generated by an alternative system instead ? Fraudulent records , disease outbreaks , cancerous cells on pathology slides , or adversarial noised images are all examples of data that does not come from the original , normal system . These are the interesting data points worth studying . The goal of anomalous pattern detection is to quantify , detect , and characterize the data that are generated under these alternative systems . Furthermore , subset scanning extends these ideas to consider groups of data records that may only appear anomalous when viewed together ( as a subset ) due to the assumption that they were generated by the same alternative system . Neural networks may be viewed as one of these data generating systems . The activations are a source of high-dimensional data that can be mined to discover anomalous patterns . Mining activation data has implications for interpretable machine learning as well as more objective tasks such as detecting groups of out-of-distribution samples . This paper addresses the question : ” Which of the exponentially many subset of inputs ( images ) have higher-than-expected activations at which of the exponentially many subset of nodes in a hidden layer of a neural network ? ” We treat this scenario as a search problem with the goal of finding a “ high-scoring ” subset of images × nodes by efficiently maximizing nonparametric scan statistics in the activation space of neural networks . The primary contribution of this work is to demonstrate that nonparametric scan statistics , efficiently optimized over node-activations×multiple inputs ( images ) , are able to quantify the anomalousness of a subset of those inputs ( images ) into a real-valued “ score ” . This definition of anomalousness is with respect to a set of clean “ background ” inputs ( images ) that are assumed to generate normal or expected patterns in the activation space of the network . Our method measures the deviance between the activations of a subset of inputs ( images ) under evaluation and the activations generated by the background inputs . The challenging aspect of measuring deviances in the activation space of neural networks is dealing with high-dimensional data , on the order of the number of nodes in a hidden layer × the number of inputs ( images ) under consideration . Therefore , the measure of anomalousness must be effective in capturing systematic ( yet potentially subtle ) deviances in a high-dimensional subspace and be computationally tractable . Subset scanning meets both of these requirements ( see Section 2 ) . The reward for addressing this difficult problem is an unsupervised , anomalous-input detector that can be applied to any input and to any type of neural network architecture . Neural networks universally rely on their activation space to encode the features of their inputs and therefore quantifying deviations from expected behavior in the activation space has broad appeal and potential beyond detecting anomalous patterns in groups of images . Furthermore , an additional output of subset scanning not fully explored in this paper is the subset of nodes at which the subset of inputs ( images ) had the higher-than-expected activations . These may be used to characterize the anomalous pattern that is affecting the inputs . The second contribution of this work focuses on detection of targeted adversarial noise added to inputs in order to change the labels to a target class Szegedy et al . ( 2013 ) ; Goodfellow et al . ( 2014 ) ; Papernot & McDaniel ( 2016 ) . Our critical insight to this problem is the ability to detect the presence of noise ( i.e . an anomalous pattern ) across multiple images simultaneously . This view is grounded by the idea that targeted attacks will create a subtle , but systematic , anomalous pattern of activations across multiple noised images . Therefore , during a realistic attack on a machine learning system , we expect a subset of the inputs to be anomalous together by sharing higher-than-expected activations at similar nodes . Empirical results show that detection power drastically increases when targeted images compose 8 % -10 % of the data under evaluation . Detection power is near 1 when the proportion reaches 14 % . In summary , this is the first work to apply subset scanning techniques to data generated from neural networks in order to detect anomalous patterns of activations that span multiple inputs ( images ) . To the best of our knowledge , this is the first topic to address adversarial noise detection by considering images as a group rather than individually . 2 SUBSET SCANNING . Subset scanning treats pattern detection as a search for the “ most anomalous ” subset of observations in the data where anomalousness is quantified by a scoring function , F ( S ) ( typically a loglikelihood ratio ) . Therefore , we wish to efficiently identify S∗ = argmaxS F ( S ) over all relevant subsets of the data S. Subset scanning has been shown to succeed where other heuristic approaches may fail ( Neill , 2012 ) . “ Top-down ” methods look for globally interesting patterns and then identifies sub-partitions to find smaller anomalous groups of records . These approaches may fail when the true anomaly is not evident from global aggregates . Similarly , “ Bottom-up ” methods look for individually anomalous data points and attempt to aggregate them into clusters . These methods may fail when the pattern is only evident by evaluating a group of data points collectively . Treating the detection problem as a subset scan has desirable statistical properties for maximizing detection power but the exhaustive search is infeasible for even moderately sized data sets . However , a large class of scoring functions satisfy the “ Linear Time Subset Scanning ” ( LTSS ) property which allows for exact , efficient maximization over all subsets of data without requiring an exhaustive search Neill ( 2012 ) . The following sub-sections highlight a class of functions that satisfy LTSS and describe how the efficient maximization process works for scanning over activations . 2.1 NONPARAMETRIC SCAN STATISTICS . This work uses nonparametric scan statistics ( NPSS ) that have been used in other pattern detection methods Neill & Lingwall ( 2007 ) ; McFowland III et al . ( 2013 ) ; McFowland et al . ( 2018 ) ; Chen & Neill ( 2014 ) . These scoring functions make no parametric assumptions about how activations are distributed at a node in a neural network . They do , however , require baseline or background data to inform their data distribution under the null hypothesis H0 of no anomaly present . The first step of NPSS is to compute empirical p-values for the evaluation input ( e.g . images potentially affected with adversarial noise ) by comparing it to the empirical baseline distribution generated from the background inputs that are “ natural ” inputs known to be free of an anomalous pattern . NPSS then searches for subsets of data S in the evaluation inputs that contain the most evidence for not having been generated under H0 . This evidence is quantified by an unexpectedly large number of low empirical p-values generated by the evaluation inputs . See Figure 1 . In our specific context , we have evaluation data D a collection of N = |D| images Xi , which are fed through a neural network ( defined by a set of nodesO = { O1 , . . . , OJ } ) producing an activation Aij , at each node Oj . The challenge we face is detecting which subset S = SX × SO , if any , of the evaluation images SX ⊆ D is anomalous as evidenced from their activation at some nodes SO ⊆ O . To approach this challenge , NPSS uses as baseline data DH0 of known , clean CIFAR-10 images . Each of the M = |DH0 | background images Xz , also generates an activation A H0 zj at each network node Oj . GivenD , ourN×J matrix of activations for each evaluation image at each network node , andDH0 , our corresponding M × J matrix of activations for each of the background images , we can obtain an empirical p-value for each Aij : a means to measure of how anomalous the activation value of ( potentially contaminated ) image Xi is at node Oj . This p-value pij is the proportion of activations from the background images , AH0zj , that are larger than the activation from the evaluation images Aij at node Oj . We note that McFowland III et al . ( 2013 ) extend this notion to p-value ranges such that pij is uniformly distributed between pminij and p max ij . This current work makes a simplifying assumption here to only consider a range by its upper bound , pmaxij = ∑ Xz∈DH0 I ( Azj > =Aij ) +1 M+1 . The matrix of activations Aij is now converted into a matrix of p-values Pij . Intuitively , if an evaluation imageXi is “ natural ” ( its activations are drawn from the same distribution as the baseline images ) then few of the p-values generated by image Xi across the network nodes–will be extreme . The key assumption for subset scanning approaches is that under the alternative hypothesis of an anomaly present in the activation data then at least some subset of the activations , for the effected subset of images , will systematically appear extreme . The goal is to identify this “ high-scoring ” subset through an efficient search procedure that maximizes a nonparametric scan statistic . The matrix of p-values Pij from evaluation images is processed by a nonparametric scan statistic in order to identify the subset of evaluation images SX ⊆ D whose activations at some subset of nodes SO ⊆ O maximizes the scoring function maxS=SX×SO F ( S ) , where S = SX × SO represents a submatrix of Pij , as this is the subset with the most statistical evidence for having been effected by an anomalous pattern . The general form of the NPSS score function is F ( S ) = maxα Fα ( S ) = maxα φ ( α , Nα ( S ) , N ( S ) ) where N ( S ) represents the number of empirical p-values contained in subset S and Nα ( S ) is the number of p-values less than ( significance level ) α contained in subset S. This generalizes to a submatrix , S = SX × SO , intuitively . Nα ( S ) = ∑ Xi∈SX ∑ Oj∈SO I ( p max ij < α ) and N ( S ) = ∑ Xi∈SX ∑ Oj∈SO 1 . There are well-known goodness-of-fit statistics that can be utilized in NPSS McFowland et al . ( 2018 ) , the most popular is the Kolmogorov-Smirnov test Kolmogorov ( 1933 ) . Another option is Higher-Criticism Donoho & Jin ( 2004 ) . In this work we use the Berk-Jones test statistic Berk & Jones ( 1979 ) : φBJ ( α , Nα , N ) = N ∗ KL ( Nα N , α ) , where KL is the Kullback-Liebler divergence KL ( x , y ) = x log xy + ( 1 − x ) log 1−x 1−y between the observed and expected proportions of significant p-values . Berk-Jones can be interpreted as the log-likelihood ratio for testing whether the p-values are uniformly distributed on [ 0 , 1 ] as compared to following a piece-wise constant step function alternative distribution , and has been shown to fulfill several optimality properties and has greater power than any weighted Kolmogorov statistic .

Summary: The paper proposed a scheme to detect the presence of anomalous inputs, such as samples designed adversarially for deep learning tasks, that is based on a "subset scanning" approach to detect anomalous activations in the deep learning network. The paper is considering a very interesting problem and provides the suitable application of an approach previously developed for pattern detection. The approach is motivated by p-value statistics of the activation patterns in the deep learning network under the "null hypothesis" of a non-anomalous input.