Title: ANOMALOUS PATTERN DETECTION IN ACTIVATIONS AND RECONSTRUCTION ERROR OF AUTOENCODER

Abstract: In real-world machine learning applications, large outliers and pervasive noise are commonplace, and access to clean training data as required by standard deep autoencoders is unlikely. Reliably detecting anomalies in a given set of images is a task of high practical relevance for visual quality inspection, surveillance, or medical image analysis. Autoencoder neural networks learn to reconstruct normal images, and hence can classify those images as anomalous if the reconstruction error exceeds some threshold. In this paper, we proposed an unsupervised method based on subset scanning over autoencoder activations. The contributions of our work are threefold. First, we propose a novel method combining detection with reconstruction error and subset scanning scores to improve the anomaly score of current autoencoders without requiring any retraining. Second, we provide the ability to inspect and visualize the set of anomalous nodes in the reconstruction error space that make a sample noised. Third, we show that subset scanning can be used for anomaly detection in the inner layers of the autoencoder. We provide detection power results for several untargeted adversarial noise models under standard datasets.

1. INTRODUCTION:

Neural networks generate a large amount of activation data when processing an input. This work applies anomalous pattern detection techniques on this activation data in order to determine if the input is anomalous. Examples of an anomalous input can be noised samples by an adversary (Szegedy et al., 2013; Goodfellow et al., 2014; Kurakin et al., 2016a; Dalvi et al., 2004a) , human annotation errors (Klebanov et al., 2008) , etc. The goal of anomalous pattern detection is to quantify, detect, and characterize the data that are generated by an alternative process. Since anomalies are rare and come from diverse sources, it is not feasible to obtain labeled datasets of all possible anomalies/attacks. If an observation deviates from the learned model, it is classified as an anomaly (Chandola et al., 2009) . In real-world problems, large outliers and pervasive perturbations are commonplace, and one may not have access to clean training data as required by standard deep denoising autoencoders (Beggel et al., 2019) due to reasons such as human annotation errors (Klebanov et al., 2008) and poisoning techniques (Dalvi et al., 2004b) .

Autoencoders differ from classical classifier networks such as Convolutional Neural Networks (CNNs) (LeCun & Bengio, 1998) . Autoencoders do not require labels because the expected output is the input data. The autoencoder is trained to minimize the reconstruction error L(x, x ). During the prediction step, anomaly detection can be performed by looking at the distribution of mean reconstruction error L(w, d(e(w))) when w ∈ X clean and L(w , d(e(w ))) when w ∈ X adv (Frosst et al., 2018) . An example of both, clean and noise reconstruction error distribution can be seen in Figure 4 (b). Using this type of anomaly detection with autoencoders assumes that the autoencoder is properly trained with clean data. Otherwise, this manifold can be used advantageously by training the autoencoder with corrupted samples that are mapped to clean samples. As a result, the autoencoder will learn an underlying vector field that points in the direction of the manifold in which the clean samples lie. Thus, upon the introduction of a perturbation, the magnitude of each arrow in the vector field will indicate the direction in which the data must be moved to map the sample to its clean representation (Sahay et al., 2019) . Further detail on the autoencoder architecture and training setup for the experiments can be found in the Section A.4. Subset scanning frames the detection problem as a search over subsets of data in order to find a subset that maximizes a scoring function F (S), typically a likelihood ratio. Subset scanning exploits a property of these scoring functions that allow for efficient maximization over the exponentially large search space (Neill, 2012) .

In this paper, we show how subset scanning methods can enhance the anomaly detection power of autoencoders in an unsupervised manner and without a retraining step. We treat this anomaly detection approach as a search for a subset of node activations that are higher than expected. This is formally quantified as the subset with the highest score according to a non-parametric scan statistic.

The contributions of our work are threefold. First, we propose a novel approach combining detection with reconstruction error and subset scanning scores to improve the anomaly score of current autoencoders without requiring any retraining. Second, we provide the ability to identify and visualize the set of anomalous nodes in the reconstruction error space that make noised samples. Third, we show that subset scanning can be used for anomaly detection in the inner layers of the autoencoder.

Figure 1 : Example of subset scanning score distributions across layers of an autoencoder for adversarial BIM noise = 0.01. In the top of the graph we can see subset score distributions per nodes in a layer. The distributions of subset scanning scores are shown in blue for clean images (C) (expected distribution), and in orange for noised samples A t . Higher AUCs are expected when distributions are separated from each other and lower AUCs when they overlap. The purple structure corresponds to convolutional layers at the Encoder, while the red structure corresponds to the convolution layers for the Decoder. The computed AUC for the subset score distributions can be found in Table 1 . The highest mutual information exchange with the adversarial input happens on the first layers (convolutional and maxpooling). This is why the greatest divergence in both C and A t subset scores distributions is seen. In the latent space, due to properties described in Section 4, the autoencoder abstracts basic representations of the images, losing subset scanning power due to the autoencoder mapping the new sample to the expected distribution. This can be seen as an almost perfect overlap of distribution in conv 2d 7.

2. RELATED WORK AND BACKGROUND INFORMATION:

Machine learning models are susceptible to adversarial perturbations of their input data that can cause the input to be misclassified (Szegedy et al., 2013; Goodfellow et al., 2014; Kurakin et al., 2016a; Dalvi et al., 2004a) . There are a variety of methods to make neural networks more robust to adversarial noise. Some require retraining with altered loss functions so that adversarial images must have a higher perturbation in order to be successful (Papernot et al., 2015; Papernot & McDaniel, 2016) .

Our work treats the problem as anomalous pattern detection and operates in an unsupervised manner without a priori knowledge of the attack or labeled examples. We also do not rely on training data augmentation or specialized training techniques. These constraints make it a more difficult problem, but more realistic in the adversarial noise domain as new attacks are constantly being created. Before introducing our approach in the next section, we explain related work in two parts. First, we provide a quick overview of Autoencoders as anomaly detectors and second, we discuss different adversarial attacks models used in this paper.

2.1. AUTOENCODERS AS ANOMALY DETECTORS:

Several approaches have been used for anomaly detection with autoencoders. Since autoencoders can model training data distribution, these neural networks are an interesting option for anomaly detection. Most of the methods found in the literature require that the training data only consist of normal examples such as denoising autoencoders (Meng & Chen, 2017; Xie et al., 2012) , but this alone is no guarantee for anomalies to have a large reconstruction error. Beggel et al. (2019) present a robust Anomaly Detection with ITSR (Iterative Training Set Refinement) and Adversarial Autoencoders. Their work uses the capabilities of adversarial autoencoders to address the shortcoming of conventional autoencoders in the presence of anomalies samples during training. They also propose a combined criterion of reconstruction error and likelihood in the latent space, as well as a retraining method to increase the separation in both latent and image space. Zhai et al. (2016) use deep structured energy-based models, showing that a criterion based on an energy score leads to better results than the reconstruction error criterion. Zhou & Paffenroth (2017) present an extension of denoising autoencoders that can work with corrupted data. During training, the network uses an anomaly regularizing penalty based on L p -norms.

Most of the approaches for anomaly detection with autoencoders require the training data to consist of clean examples or use complex autoencoder architectures and special training. In this work, we propose subset scanning applied to autoencoders. This is an unsupervised anomaly detector that can be applied to any pre-trained, off-the-shelf autoencoder network. We use, as a baseline, the detection capabilities based on mean autoencoder reconstruction error distributions (Sakurada & Yairi, 2014) and One-SVM (Schölkopf et al., 2001) for the autoencoder reconstruction error space analysis.

2.2. ADVERSARIAL ATTACKS:

Several attack models have been used to target classifiers in this study, we focus on untargeted attacks with Basic Iterative Method (BIM) (Kurakin et al., 2016b) , Fast Gradient Signal Method (FGSM) (Goodfellow et al., 2014) , and DeepFool (DF) (Moosavi-Dezfooli et al., 2016) . The idea behind these attacks is to find a perturbation to be included in the original sample X, generating an adversarial sample X adv .

Fast Gradient Sign Method (FGSM) FGSM (Goodfellow et al., 2014) was designed to be extremely fast rather than optimal. It simply uses the sign of the gradient at every pixel to determine the direction with which to change the corresponding pixel value. Given an image x and its corresponding true label y, the FGSM attack sets the perturbation δ to:

EQUATION

Basic Iterative Method (BIM) BIM (Kurakin et al., 2016b ) is a straightforward extension of FGSM where adversarial noise is applied multiple times iteratively with small step size:

X adv 0 = X, X adv N +1 = Clip X,k X adv N + α sign ∇ X J X adv N , y true (2)

DeepFool (DF):

The DF algorithm presented by Moosavi-Dezfooli et al. (2016) computes the optimal adversarial perturbation to perform a misclassification. In a binary classifier, the robustness of the model f for an input X 0 is equal to the distance the input to the hyper-plane that separates both classes. So the minimal perturbation to change the classifier decision is the orthogonal projection defined as:

- f (X 0 ) w 2 2 * w (3)

3. SUBSET SCANNING FOR ANOMALOUS PATTERN DETECTION:

Subset scanning treats the pattern detection problem as a search for the "most anomalous" subset of observations in the data. Herein, anomalousness is quantified by a scoring function, F (S) which is typically a log-likelihood ratio statistic. Therefore, the goal is to efficiently identify S * = arg max S F (S) over all relevant subsets of node activations within an autoencoder that is processing an image at runtime. The particular scoring functions F (S) used in this work are covered in the next sub-section.

Heuristic alternatives to subset scanning include "top-down" and "bottom-up" methods. Topdown approaches detect globally interesting patterns and then identify sub-partitions to find smaller anomalous groups of records. These may fail to detect small-scale patterns that are not evident from global aggregate statistics. Similarly, bottom-up approaches that identify individually anomalous data points and aggregates them into clusters may fail when the pattern is only evident by evaluating a group of data points collectively (Neill, 2012) .

Treating the detection problem as a subset scan has desirable statistical properties. However, the exhaustive search over groups quickly becomes computationally infeasible due to the exponential number of subsets of records. Fortunately, a large class of scoring functions used in subset scanning satisfy the "Linear Time Subset Scanning" (LTSS) property that allows for exact, efficient maximization over all subsets of data without requiring an exhaustive search (Neill, 2012) . The LTSS property essentially reduces the search space from 2 N to N for a dataset with N records, while guaranteeing that the highest-scoring subset of records is identified.

3.1. NON-PARAMETRIC SCAN STATISTICS:

This work uses non-parametric scan statistics (NPSS) that have been used in other pattern detection methods (Neill & Lingwall, 2007; McFowland III et al., 2013; McFowland et al., 2018; Chen & Neill, 2014) . Although subset scanning can use parametric scoring functions (i.e. Gaussian, Poisson), the distribution of activations within particular layers are highly skewed and in some cases bi-modal. See Figure 9 . Therefore, this work uses non-parametric scan statistics that makes minimal assumptions on the underlying distribution of node activations.

The intuition behind the role of non-parametric scan statistics is best explained in a simple example. Consider 100 p-values that are supposed to be uniformly distributed between 0 and 1 under the null hypothesis of no anomaly present in the data. A larger-than-expected activation at a node results in a lower p-value for that node. What if we observe 30 (out of 100) p-values all under a threshold value of 0.10? Is that more or less anomalous than finding 20 (out of 100) p-values all under a threshold of 0.075? Non-parametric scan statistics quantify these situations. This same example can be used to highlight why subset scanning is appropriately paired with non-parametric scan statistics. A single p-value of 0.1 is not interesting when viewed by itself. However, if there are 29 other p-values in the same data set that are also 0.1 (or lower), then the observations are now more interesting when considered together, as a group. Subset scanning efficiently identifies the combination of p-values and thresholds in order to maximize the non-parametric scan statistic.

There are three steps to appropriately use non-parametric scan statistics on neural network activation data. The first is to form a distribution of "expected" activations at each node. This is done by letting the autoencoder process images that are known to be clean from anomalies (sometimes referred to as "background" images) and recording the activations at each node. The second step involves a test image that may be clean or noised and needs to be scored. We record the activations induced by the test image and compare it to the baseline activations created in the first step. This comparison results in a p-value at each node. The third step is to quantify the anomalousness of the resulting p-values by finding the subset of nodes that maximize the non-parametric scan statistic.

We now formalize these three steps. Let there be M background images X z included in D H0 . These images generate activations A H0 zj at each node O j . Let X i (not in D H0 ) be a test image under evaluation. This image creates activations A ij at each node O j . The p-value, p ij , is the proportion of background activations A H0 zj greater than the activation induced by the test image A ij at node O j . (We note that McFowland III et al. (2013) extend this notion to p-value ranges such that p ij is uniformly distributed between p min ij and p max ij ). This current work makes a simplifying assumption to only consider a range by its upper bound,

p ij = Xz ∈D H 0 I(Azj >=Aij )+1 M +1 .

A test image X i is now converted to a vector of p-values p ij of length J = |O|, the number of nodes in the network under consideration. Intuitively, if a test image is "natural" (its activations are drawn from the same distribution as the baseline images) then few of the p-values will be extreme. The key assumption is that under the alternative hypothesis of an anomaly present in the activation data, then at least some subset of the activations S O ⊆ O will systematically appear extreme. We now turn to non-parametric scan statistics to identify and quantify this set of p-values.

The general form of the NPSS score function is

EQUATION

where N (S) represents the number of empirical p-values contained in subset S and N α (S) is the number of p-values less than (significance level) α contained in subset S.

Moreover, it has been shown that for a subset S consisting of McFowland III et al., 2013) . We assume an anomalous process will create some S where the observed significance is higher than the expected, N α (S) > N (S)α, for some α.

N (S) empirical p-values, E [N α (S)] = N (S)α (

There are well-known goodness-of-fit statistics that can be utilized in NPSS (McFowland et al., 2018) , the most popular is the Kolmogorov-Smirnov test (Kolmogorov, 1933) . Another option is Higher-Criticism (Donoho & Jin, 2004) . In this work we use the Berk-Jones test statistic (Berk & Jones, 1979) :

φ BJ (α, N α , N ) = N * KL Nα N , α

, where KL is the Kullback-Liebler divergence KL(x, y) = x log x y +(1-x) log 1-x 1-y between the observed and expected proportions of significant p-values. Berk-Jones can be interpreted as the log-likelihood ratio for testing whether the p-values are uniformly distributed on [0, 1] as compared to following a piece-wise constant alternative distribution, and has been shown to fulfill several optimality properties and has greater power than any weighted Kolmogorov statistic.

3.2. EFFICIENT MAXIMIZATION OF NPSS:

Although NPSS provides a means to evaluate the anomalousness of a subset of node activations S O discovering which of the 2 J possible subsets provides the most evidence of an anomalous pattern is computationally infeasible for moderately sized data sets. However, NPSS has been shown to satisfy the linear-time subset scanning (LTSS) property (Neill, 2012) , which allows for an efficient and exact maximization over subsets of data.

The LTSS property uses a priority function G(O j ) to rank nodes and then proves that the highestscoring subset consists of the "top-k" priority nodes for some k in 1 . . . J. The priority of a node for NPSS is the proportion of p-values that are less than α. However, because we are scoring a single image and there is only one p-value at each node, the priority of a node is either 1 (when the p-value is less than α) or 0 (otherwise). Therefore, for a fixed, given α threshold, the most anomalous subset is all and only nodes with p-values less than alpha.

In order to maximize the scoring function over α we first sort the O j nodes by their p-values. Let S (k) be the subset containing the k nodes with with the smallest p-values. Let α k be the largest pvalue among these k nodes. The LTSS property guarantees that the highest-scoring subset (over all α thresholds) will be one of these J subsets S (1) , S (2) , . . . S (J) with their corresponding α k threshold. Any subset of nodes that does not take this form (or uses an alternate α k ) is provably sub-optimal and not considered. Critically, this drastically reduced search space still guarantees identifying the highest-scoring subset of nodes for a test image under evaluation.

Figure 2 shows how the optimal α threshold (and subset size) can vary for different test images under consideration. The leftmost panel shows the distributions of the size of the most anomalous subset of nodes in both clean and noised images. We note that noised images tend to return a larger subset of nodes than clean images. The middle panel shows the optimal α threshold value that maximized the non-parametric scan statistic for clean and noised images. We note that noised images tend to have lower thresholds than clean images. When an image induces a larger number of smaller p-values, the resulting score of the image is higher. This is demonstrated in the right-most panel where noised test images have higher scores than clean test images.

4. DETECTING ADVERSARIAL ATTACKS ON AUTOENCODERS:

A conventional autoencoder (Bengio et al., 2007) learns the underlying manifold of the training data, which is used to reconstruct the input (x) as the output (x ). The general architecture of any autoencoder involves an encoder and a decoder. The encoder (e : X → Z) is composed of one or more layers that perform nonlinear dimensionality reduction from the high dimensional input space into a low-dimensional latent representation (z = e(x)), while the decoder (d : Z → X) reconstructs the original sample from the latent representation. Both functions compute x = d(e(x)). The autoencoder is optimized by minimizing the reconstruction error L(x, x ).

Anomalous pattern detection can be performed on a trained autoencoder, by looking at the distributions of mean reconstruction error L(w, d(e(w))) when w ∈ X clean and L(w , d(e(w ))) when w ∈ X adv . Due to the inherent properties of the autoencoder for anomaly detection, we propose two experiments or applications of subset scanning. First, we are interested in subset scanning scores distributions along the layers of the encoder. During the untangling phase (z = e(x)) of information reduction from the input space to the latent representation (z), we want to observe until which layer we're able to discriminate the input (clean and noised) to the distribution learnt by the autoencoder. Second, we apply subset scanning methods on the reconstruction error space, to understand if reconstruction error criterion suffices for detection in training autoencoder based anomaly detectors.

Figure 2 : The connection between the number of nodes in a subset, α value that maximizes the non-parametric scan statistic, and the resulting subset score. These results are for Fashion-MNIST examples with activations coming from the first layer of the autoencoder. Under the presence of BIM adversarial noise, we observe a larger number of nodes that have smaller p-values. This combination results in a higher subset score than the clean images. Critically, the LTSS property allows α to be efficiently chosen to maximize the score for each individual image. The subset size is all nodes with p-values less than the α threshold. We enforce a α max = 0.5 constraint on the search.

5. EXPERIMENTAL SETUP:

In this section, we describe the baselines methods used as comparison, as well as the datasets, evaluation metric, adversarial noise generation and autoencoder architecture we used. For generating the attacks a standard CNN model LeCun et al. (1998) was trained for both datasets. The test accuracies for these models are 0.992 for MNIST and 0.921 for Fashion-MNIST.

We trained an autoencoder network (Bengio et al., 2007) on MNIST (LeCun et al., 1998) and Fashion-MNIST (Xiao et al., 2017) (detailed in Section 5.1). The architecture of the autoencoder is depicted in Figure 8 , and further details on the training setup can be found in Appendix A.4. The test reconstruction error of the model was 0.284 for Fashion-MNIST and 0.095 for MNIST. In real-world applications, clean training datasets cannot always be guaranteed due to factors such as human annotation errors (Klebanov et al., 2008) , and poisoning techniques (Dalvi et al., 2004b) . Consequently, we trained the autoencoder with different levels of data poisoning. We trained autoencoders with 100% of clean samples, 1% of adversarial samples, and 9% of adversarial samples. For this experiment, we used BIM as the attack and Fashion-MNIST as the dataset. We evaluated subset scanning over two experiments. First, we applied our subset scanning method on the reconstruction error calculated over the input data and the last layer of the autoencoder (conv2d 7). This layer has 1 filter containing 784 nodes. For more information, refer to Section 4. Second, we studied subset scanning patterns across adversarial attacks and datasets, to see if we have some common subset scanning behaviors. For this, we applied subset scanning across all layers of the autoencoder (convolutional, max pooling and up-sampling) and analyzed the detection power in each case.

5.1. DATASETS:

For our adversarial experiments, we took M = |D H0 | = 7000 of the 10000 validation images and used them to generate the background activation distribution (D H0 ) at each of the 784 nodes (28×28) for the reconstruction error space and the activations nodes per each inner layer. These 7000 images were not used again. These images form our expectation of "normal" activation behavior for the network. The remaining 3000 images were used to form a "Clean" (C = 1500) sample and an "Adversarial" (A t = 1500) noised sample. For the experiments we only kept the successful attacks for DF, FGSM and BIM, so we only preserve noised samples that were incorrectly classified by the model.

We evaluated anomaly detection with subset scanning on the classical MNIST (LeCun et al., 1998) dataset and more complex dataset Fashion-MNIST (Xiao et al., 2017) . We present a quick overview of both datasets:

• MNIST (LeCun et al., 1998): The training set has 60000 images and the test set has 10000 images of handwritten digits. Each digit has been normalized and centered to 28 × 28. • Fashion-MNIST (Xiao et al., 2017) : a relatively new dataset comprising 28 × 28 grayscale images of 70.000 fashion products from 10 categories, with 7000 images per category.

The training set has 60000 images and the test set has 10000 images. As an alternative to MNIST, it has the same image size, data format and validation splits, with the digits from MNIST replaced with 10 products of clothes and accessories.

5.2. ADVERSARIAL NOISE SAMPLE GENERATION:

Several adversarial attacks for the subset scanning experiments were implemented, briefly introduced in Section 2.2. Specifically, we describe in this section the hyperparameter selection for Basic Iterative Method (BIM) adversarial attack (Kurakin et al., 2016b) , Fast Gradient Signal Method (FGSM) (Goodfellow et al., 2014) and DeepFool (DF) (Moosavi-Dezfooli et al., 2016) .

BIM and FGSM have an parameter which controls how far a pixel is allowed to change from its original value when noise is added to the image. We used a value of = 0.01 in the scaled [0, 1] pixel space. We also allowed the method to reach its final noised state over 100 steps with each of size 0.002. Smaller values of make the pattern subtler and harder to detect, but also less likely for the attacks to succeed in changing the class label to the target. For DeepFool, we used standard = 1e -06 and 100 iterations. Example of generated adversarial samples for both datasets are depicted in Figure 7 . All untargeted attacks were generated with the Adversarial Robustness Toolbox (Nicolae et al., 2018) 1 . The set A t only contains images that were successfully noised by each type of adversarial attack. This means that those samples were misclassified from an original predicted label. The 1500 images in group C are natural and have all class labels represented equally.

6. RESULTS:

We adopted the following metric to measure the effectiveness of subset scanning over an autoencoder to distinguishing different types of adversarial attacks images under the activation and reconstruction error space. The Detection Power is measured by AUROC, the Area Under the Receiver Operating Characteristic curve, which is also a threshold independent metric (Davis & Goadrich, 2006) . The ROC curve depicts the relationship between true positive rate (TPR) and false positive rate (FPR). Results shown in Figure 6 for reconstruction error and activations space in the first convolutional layer for Figure 3 . 1 : Detection power for individual subset scanning over all layers (convolutional, max pooling and up-sampling) for both datasets under three different adversarial attacks. The noised columns refer to the autoencoder being trained with 1% and 9% BIM noised samples. Under different datasets and attacks, the same initial layers hold the highest detection power.

In Table 1 , we can observe that across different datasets, noise attacks models, and two proportion of noised samples during training, the first layers (conv 2d 1 and max pooling 2d 1) maintain a high performance regarding detection power (between 0.96 to 1.0 depending on dataset and noise attack).

The ROC curves and subset scores distribution for the BIM and FGSM attacks under Fashion-MNIST for the layer conv 2d 1 are shown in Figure 3 . Furthermore, Table 1 shows that in the cases where 1% and 9% of the samples are noised during training stage of the autoencoder, the detection power of subset scanning still performs correctly, above 0.82.

Table 2 shows the behavior of subset scanning over the reconstruction error space and the detection power in detail for both datasets and different adversarial attacks. We can observe a difference of performance of our method over the Fashion-MNIST dataset. One hypothesis would be that this is due to the autoencoder performance (Loss for Fashion-MNIST 0.284 and MNIST 0.095). To test this idea, we performed preliminary experiments that show a relationship between the decrease in the loss of the trained autoencoder and the increase in the detection power of subset scanning methods under the reconstruction error space. A poorly-trained autoencoder will have a higher loss, while a well-trained autoencoder will have a lower loss. If an autoencoder's loss is high, it is more difficult to separate between clean and noised samples in the reconstruction space. Nonetheless, subset scanning has higher detection power than Mean Reconstruction Error distributions under clean and noise samples (see Figure 4 ) and Unsupervised outlier detection methods such as One-SVM (Schölkopf et al., 2001) . Furthermore, subset scanning under the reconstruction error space is an interesting technique to explore and introspect what nodes or portions of the input image look anomalous. With this information we can not only point out which image looks anomalous, but also indicate which nodes make the input a noised sample, an example of this is depicted in Figure 5 . (Sakurada & Yairi, 2014) and One-SVM over reconstruction error of the AE (Schölkopf et al., 2001) .

Datasets:



7. CONCLUSION AND FUTURE WORK:

In this work, we proposed a novel unsupervised method for adversarial noise detection with off-theshelf autoencoders and subset scanning. We have successfully demonstrated how subset scanning can be used to gain detection strength against multiple adversarial attacks on images across several datasets, without requiring any retraining or complex deep autoencoder network structures.

Furthermore, we tested subset scanning over the reconstruction error space and observed significant variations depending on the dataset, autoencoder architecture, and training setup. We performed preliminary experiments that yielded a relation between a decrease in the loss of the trained autoencoder and an increase in the detection power of subset scanning under the reconstruction error space. Nonetheless, applying our method under this space provides introspection capabilities that allow us to identify the nodes or portions of the input image look anomalous. Consequently, we are able to not only point out which image looks anomalous but also characterize the nodes that make the input a noised sample. We also evaluated the performance of applying subset scanning over the autoencoder's activations. We observed a consistent and high detection power results across noise attacks, datasets, autoencoders architectures and different noised training levels in the initial layers (Convolutional and MaxPooling layers).

Due to versatile properties of subset scanning under neural network activation analysis it may be used for several other studies, including unsupervised classification in the latent space of an autoencoder. We would expect that same class images will identify as a subset of inputs (images) that have higher-than-expected activations (i.e. large number of low empirical p-values) at a subset of nodes. Subset scanning applied to autoencoders activations is a novel, unsupervised anomaly detector that can be applied to any pre-trained, off-the-shelf neural network, previously only used in classifier neural networks such as CNNs and ResNet (Speakman et al., 2018) . Several examples of successful noised samples from both datasets and the three different attacks are depicted in Figure 7 .

A.4 ARCHITECTURE AND TRAINING SETUP FOR AUTOENCODER:

We train the same autoencoder architecture (4385 parameters) for both Fashion-MNIST and MNIST, A diagram of the network is shown in the Figure 8 . The encoder comprises three (3) convolutional layers with 16, 8, 8 filters respectively, a kernel size of three (3), each with relu activations, and a maxpooling layer with a pool size of two (2) after every convolutional layer. The decoder comprises four (4) convolutional layers with 8, 8, 16, 1 filters respectively, a kernel size of three (3), each with relu activations except the final layer which uses a sigmoid. Each consecutive pair of convolutional layer is interspersed with an upsampling layer with a size of two (2). We train the autoencoder by minimizing the binary cross-entropy of the decoder output and the original input image using an adadelta optimizer (citep) for 100 epochs taking 128 records per batch.

A.5 ACTIVATION VALUES DISTRIBUTION ACROSS ALL LAYERS:

Although subset scanning can use parametric scoring functions (i.e. Gaussian, Poisson), the distribution of activations within particular layers are highly skewed and in some cases bi-modal. See Figure 9 . Therefore, this work uses non-parametric scan statistics that makes minimal assumptions on the underlying distribution of node activations.

Furthermore we only consider 1-tailed p-values (in the greater direction). This is due to nuances of the ReLu activation function. Alternative activation functions such as tanh and signmoid would allow an "extreme" activation to be considered as either larger or smaller than expected with a pvalue coming from a 2-tailed calculation.

A.6 NON-PARAMETRIC SCAN STATISTICS NPSS can be viewed as a second-order test statistic that operate on (by aggregating information across) p-values (i.e., the first order test statistics) to evaluate the the evidence for violations of H 0 in a given subset S. NPSS is operationalized with a given score (test) function; each test is powered for different alternatives, and therefore, NPSS's detection power is linked to preferences of the selected score function. This work used the Berk-Jones scoring function (Berk & Jones, 1979) .

EQUATION

Where KL is the Kullback-Liebler divergence KL(x, y) = x log x y + (1 -x) log 1-x 1-y between the observed and expected proportions of significant p-values. Berk-Jones can be interpreted as the log-likelihood ratio for testing whether the p-values are uniformly distributed on [0, 1] as compared to following a piece-wise constant alternative distribution, and has been shown to fulfill several optimality properties.

A more commonly known scoring function that also satisfies the LTSS property is the Kolmogoorv-Smirnov test statistic

EQUATION

which is known to be more sensitive to deviations in the center of a distribution. https://www. jstor.org/stable/2958837 and http://www.jstor.org/stable/2958836.

Another test is Higher-Criticism Donoho & Jin (2004) :

EQUATION

which can be interpreted as the test statistic of a Wald test for the amount of significant p-values given that N α is binomially distributed with parameters N α and α. Because Higher-Criticism normalizes by the standard-deviation of N α , it tends to be more sensitive to small subsets with very extreme p-values. Most values are accumulated around 0 due to ReLu activations. The large skew and sometimes bi-modal distribution of activations motivated the use of non-parametric scan statistics to quantify what it means for an activation to be larger-than-expected.

:

https://github.com/IBM/adversarial-robustness-toolbox

Figures and Tables:

Figure 3: (a) ROC curves for each of the noised cases as compared to the scores from test sets containing all natural images for layer Conv2d 1. (b) Distribution of subset scores for test sets of images over Conv2d 1.Test sets containing all natural images had lower than scores than test sets containing noised images. Higher proportion of noised images resulted in higher scores.

Figure 4: Reconstruction error baseline visualization. (a) Baseline mean reconstruction error samples for clean images and BIM noised samples processed by the autoencoder. We can observe that in clean samples reconstruction error only appears on the contours of the number, while noise samples have lower reconstruction error values distributed throughout the image. (b) Mean reconstruction error distribution for clean (blue distribution) and noise samples (orange samples).

Figure 5: Anomalous nodes visualization. Overlap of anomalous nodes (white) and reconstruction error (darker blue) per sample. (a) Noised samples with BIM. We can observe that nodes outside the contour will make the sample be classified as noised. (b) Whereas clean we expect the anomalous nodes will be along the contour of the figure.

Figure 7: Successful noised samples from MNIST and Fashion-MNIST generated with Adversarial Robustness Toolbox (Nicolae et al., 2018). (a) BIM noised samples (b) DeepFool samples (c) FGSM. Parameters used for each attack are detailed at Section 5.2.

Figure 8: Autoencoder architecture diagram. Same architecture was implemented for both datasets and for different proportions of noised samples.

Figure 9: Activation distribution values across all layers.

Test sets containing all natural images had lower than scores than test sets containing noised images. Higher proportion of noised images resulted in higher scores.

Table content: <table><tr><td>Layers</td><td/><td colspan="2">Clean Training</td><td/><td/><td colspan="2">Noised (1%) Noised (9%)</td></tr><tr><td/><td>F-MNIST</td><td/><td/><td>MNIST</td><td/><td>F-MNIST</td><td>F-MNIST</td></tr><tr><td/><td>BIM FGSM</td><td>DF</td><td colspan="2">BIM FGSM</td><td>DF</td><td>BIM</td><td>BIM</td></tr><tr><td>conv2d 1</td><td colspan="2">0.964 0.974 0.965</td><td>1.0</td><td>1.0</td><td>0.999</td><td>0.909</td><td>0.823</td></tr><tr><td colspan="3">max pool 1 0.972 0.979 0.965</td><td>1.0</td><td>1.0</td><td>0.999</td><td>0.928</td><td>0.850</td></tr><tr><td>conv2d 2</td><td colspan="5">0.519 0.530 0.686 0.975 0.941 0.953</td><td>0.441</td><td>0.700</td></tr><tr><td colspan="6">max pool 2 0.500 0.513 0.634 0.855 0.809 0.837</td><td>0.424</td><td>0.693</td></tr><tr><td>conv2d 3</td><td colspan="5">0.500 0.507 0.481 0.382 0.384 0.443</td><td>0.470</td><td>0.469</td></tr><tr><td colspan="6">max pool 3 0.473 0.478 0.479 0.374 0.373 0.423</td><td>0.451</td><td>0.450</td></tr><tr><td>conv2d 4</td><td colspan="5">0.403 0.406 0.483 0.270 0.271 0.261</td><td>0.472</td><td>0.410</td></tr><tr><td colspan="6">up sampl 1 0.403 0.406 0.483 0.270 0.271 0.261</td><td>0.472</td><td>0.410</td></tr><tr><td>conv2d 5</td><td colspan="5">0.413 0.419 0.474 0.228 0.228 0.193</td><td>0.356</td><td>0.388</td></tr><tr><td colspan="6">up sampl 2 0.413 0.419 0.474 0.228 0.228 0.193</td><td>0.346</td><td>0.388</td></tr><tr><td>conv2d 6</td><td colspan="5">0.342 0.350 0.483 0.259 0.261 0.285</td><td>0.306</td><td>0.323</td></tr><tr><td colspan="6">up sampl 3 0.342 0.350 0.483 0.259 0.261 0.285</td><td>0.306</td><td>0.323</td></tr><tr><td>conv2d 7</td><td colspan="5">0.594 0.597 0.506 0.693 0.688 0.848</td><td>0.613</td><td>0.603</td></tr><tr><td>Table</td><td/><td/><td/><td/><td/><td/></tr></table>

Detection power for individual subset scanning over reconstruction error space for both dataset under three different adversarial attacks, two baselines for reconstruction error over AE

Table content: <table><tr><td/><td>Attacks</td><td/><td>Clean Training</td><td/></tr><tr><td/><td/><td colspan="3">Subset Scanning Mean AE Rec. Error One-SVM</td></tr><tr><td>Fashion-MNIST</td><td>BIM</td><td>0.698</td><td>0.641</td><td>0.478</td></tr><tr><td/><td>FGSM</td><td>0.672</td><td>0.630</td><td>0.497</td></tr><tr><td/><td>DF</td><td>0.599</td><td>0.477</td><td>0.534</td></tr><tr><td>MNIST</td><td>BIM</td><td>0.998</td><td>0.751</td><td>0.624</td></tr><tr><td/><td>FGSM</td><td>0.983</td><td>0.725</td><td>0.624</td></tr><tr><td/><td>DF</td><td>0.992</td><td>0.574</td><td>0.637</td></tr></table>

Examples of Nonparametric Scan Statistics that satisfy the Linear Time Subset Scanning property.

Table content: <table><tr><td>Name</td><td>φ(α, N α , N )</td></tr><tr><td>Higher Criticism</td><td>|Nα-N α| √ N α(1-α)</td></tr><tr><td>Berk-Jones</td><td>N  *  KL Nα N , α</td></tr><tr><td>Kolmogorov Smirnov</td><td>|Nα-N α| √ N</td></tr></table>

Peer review (iclr2020):
This paper explores the idea of using subset scanning [7] over the hidden activations of an autoencoder (AE) for the task of unsupervised anomaly detection (AD). The proposed method combines the standard reconstruction error with subset scanning scores and does not require any modification/retraining of the standard AE. The paper further inspects how units of the AE that detect anomalous patterns from such subset scanning scores might be used for anomaly visualization. Finally, the paper concludes with experiments on MNIST and Fashion-MNIST that indicate an improvement in detection performance over the standard AE reconstruction error and OC-SVM baselines. This paper should be rejected in my opinion due to the following four main reasons: (i) The technical quality of the paper is poor and the main idea not well explained; (ii) The experimental evaluation considers rather simple datasets (MNIST, Fashion-MNIST) and only includes two baselines (vanilla AE, OC-SVM), but not any major competitors ; (iii) The work is not well placed in the literature and major related work is missing; (iv) The overall presentation is poor; (i) I find that subset scanning [7], seemingly the main component the approach, is not well defined and explained in the paper. Section 3 introduces the method with an example that to me reads like an explanation of p-hacking (Section 3.1, second paragraph). Inconsistent notation and the use of undefined expressions (see examples in comments below) make it hard to follow and understand what exactly is being done. After reading, I would somewhat summarize the idea as “conducting multiple hypothesis testing over the hidden network activation statistics of an autoencoder for unsupervised anomaly detection”. This idea might be valid and indeed lead to significant improvements over the baseline of just using the AE reconstruction error. However, the specific method proposed is not rigorously defined and clearly explained in this manuscript. The explanation of the method further leaves important questions unanswered: (ia) Why use the Berk-Jones over the Kolmogorov-Smirnov test? (ib) How is the Linear Time Subset Scanning (LTSS) property exactly defined and why is it satisfied for AEs? (ii) I find the experimental evaluation in the paper not convincing. Only two baselines (vanilla AE using reconstruction error and OC-SVM) are considered, but there are many deep competitors (VAEs, GAN-based AD, Deep SVDD, etc.) [1, 10, 4, 9, 5, 6] that have significantly improved detection performance over the AE baseline. MNIST and Fashion-MNIST are also rather simple datasets. Adding more complex datasets like CIFAR-10 or MVTec [2] would be insightful (also see experimental evaluations in [9, 5, 6]). (iii) The paper does not properly place the work into context with existing literature. Major works on deep anomaly detection and out-of-distribution detection are not included [1, 10, 4, 9, 5, 6, 8, 3]. (iv) The overall presentation of the paper is poor. The writing is wordy, terms are imprecisely and inconsistently used, and the text contains many grammatical errors. Figures are unpolished and references are scattered over the paper. See comments below for examples. Going over the 8 page limit is not justified in my opinion. I listed some suggestions and ideas to improve this work below. #################### *Additional Feedback* *Ideas for Improvement* 1. Explain your method rigorously and clearly. I believe that using the activation statistics of AEs (via testing or otherwise) should lead to an improvement over the reconstruction error baseline. 2. Plots as in Figure 9 for the activation distributions of normal vs. anomalous samples would be insightful. In which layers may anomalies be detected? 3. It might be interesting to specifically look into different activation functions and choose/design specific suitable tests. Might using some form of normalization (e.g. batch normalization) be beneficial? 4. Going further into the direction of deep AD interpretability and improving over the reconstruction heatmap baseline might be worthwhile since this is an important problem with only little prior work. 5. Include major related works as outlined above and clearly state the contribution of your work in context of the existing literature. *Specific comments* 6. Imprecise use of terms and language: • Abstract: “large outliers” What should large outliers be exactly? Be specific. • Section 1: “... improve the anomaly score of current autoencoders ...” You improve the anomaly detection performance. A score is just a function. • Section 1: “... reconstruction error space ...” The reconstruction error finally is a one-dimensional loss. Be specific what you mean. • Figure 1, caption: “... the highest mutual information exchange ...” Exchange? • Section 3 on subset scanning jumps between subsets of data samples and subsets of network activations. 7. Inconsistent notation: • Section 1, second paragraph: Jumping between x and w for samples. • Function in Eq. 4 not defined. • Section 2.2, definition of BIM: Clip function not explained. Readers might guess, but rather explain your notation and definitions. 8. Writing is repetitive and redundant: • Section 1, first paragraph: Repetitions of applications with the same references already (“... human annotation errors ...”). • The autoencoder is introduced and explained three times: Section 1, Section 2.1, Section 4. 9. Abstract: “In this paper, we proposed ...” » “In this paper, we propose ...” Simple present is the primary writing tense. Many more examples like this in the main text. 10. Closely place Figures where they are referenced in the text and vice versa. The overall first reference in the Introduction, is to Figure 4 (b) on page 9. 11. Section 1, second paragraph: “Further detail on the autoencoder architecture and training setup for the experiments can be found in the Section A.4” A reference like this only should first appear in the experimental section. 12. Section 1: “This is formally quantified as the subset with the highest score according to a non-parametric scan statistic.” Passive voice is bad writing style. Write in active voice. Many more examples like this in the main text. Section 2.2: “Several attack models have been used to target classifiers in this study, ...” etc. 13. Section 2.1: “... to address the shortcoming of conventional autoencoders in the presence of anomalies samples during training.” Name them. Be specific! 14. Section 3: “Treating the detection problem as a subset scan has desirable statistical properties.” No reference or explanation. Be specific! #################### *References* [1] J. An and S. Cho. Variational autoencoder based anomaly detection using reconstruction probability. Technical report, SNU Data Mining Center, Seoul, South Korea, 2015. [2] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9592–9600, 2019. [3] R. Chalapathy and S. Chawla. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407, 2019. [4] H. Choi, E. Jang, and A. A. Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392, 2018. [5] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In NIPS, 2018. [6] D. Hendrycks, M. Mazeika, and T. G. Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019. [7] E. McFowland, S. Speakman, and D. B. Neill. Fast generalized subset scan for anomalous pattern detection. The Journal of Machine Learning Research, 14(1):1533–1561, 2013. [8] E. Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, and B. Lakshminarayanan. Do deep generative models know what they don’t know? In ICLR, 2018. [9] L. Ruff, R. A. Vandermeulen, N. Go?rnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Mu?ller, and M. Kloft. Deep one-class classification. In International Conference on Machine Learning, pages 4393–4402, 2018. [10] T. Schlegl, P. Seebo?ck, S. M. Waldstein, U. Schmidt-Erfurth, and G. Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In Proceedings International Conference on Information Processing in Medical Imaging, pages 146–157. Springer, 2017.