Title: RAPP: NOVELTY DETECTION WITH RECONSTRUC-TION ALONG PROJECTION PATHWAY

Abstract: We propose RAPP, a new methodology for novelty detection by utilizing hidden space activation values obtained from a deep autoencoder. Precisely, RAPP compares input and its autoencoder reconstruction not only in the input space but also in the hidden spaces. We show that if we feed a reconstructed input to the same autoencoder again, its activated values in a hidden space are equivalent to the corresponding reconstruction in that hidden space given the original input. We devise two metrics aggregating those hidden activated values to quantify the novelty of the input. Through extensive experiments using diverse datasets, we validate that RAPP improves novelty detection performances of autoencoder-based approaches. Besides, we show that RAPP outperforms recent novelty detection methods evaluated on popular benchmarks.

1. INTRODUCTION:

How can we characterize novelty when only normality information is given? Novelty detection is the mechanism to decide whether a data sample is an outlier with respect to the training data. This mechanism is especially useful in situations where a proportion of detection targets is inherently small. Examples are fraudulent transaction detection (Pawar et al., 2014; Porwal & Mukund, 2018) , intrusion detection (Lee, 2017; Aoudi et al., 2018) , video surveillance (Ravanbakhsh et al., 2017; Xu et al., 2015b) , medical diagnosis (Schlegl et al., 2017; Baur et al., 2018) and equipment failure detection (Kuzin & Borovicka, 2016; Zhao et al., 2017; Beghi et al., 2014) . Recently, deep autoencoders and their variants have shown outstanding performances in finding compact representations from complex data, and the reconstruction error has been chosen as a popular metric for detecting novelty (An & Cho, 2015; Vasilev et al., 2018) . However, this approach has a limitation of measuring reconstruction quality only in an input space, which does not fully utilize hierarchical representations in hidden spaces identified by the deep autoencoder.

In this paper, we propose RAPP, a new method of detecting novelty samples exploiting hidden activation values in addition to the input values and their autoencoder reconstruction values. While ordinary reconstruction-based methods carry out novelty detection by comparing differences between input data before the input layer and reconstructed data at the output layer, RAPP extends these comparisons to hidden spaces. We first collect a set of hidden activation values by feeding the original input to the autoencoder. Subsequently, we feed the autoencoder reconstructed input to the autoencoder to calculate another set of activation values in the hidden layers. This procedure does not need additional training of the autoencoder. In turn, we quantify the novelty of the input by aggregating these two sets of hidden activation values. To this end, we devise two metrics. The first metric measures the total amount of reconstruction errors in input and hidden spaces. The second metric normalizes the reconstruction errors before summing up. Note that RAPP falls back to the ordinary reconstruction-based method if we only aggregate input values before the input layer and the reconstructed values at the output layer. Also, we explain the motivations that facilitated the development of RAPP. We show that activation values in a hidden space obtained by feeding a reconstructed input to the autoencoder are equivalent to the corresponding reconstruction in that hidden space for the original input. We refer the latter quantity as a hidden reconstruction of the input. Note that this is a natural extension of the reconstruction to the hidden space. Unfortunately, we cannot directly compute the hidden reconstruction as in the computation of the ordinary reconstruction because the autoencoder does not impose any correspondence between encoding-decoding pairs of hidden layers during the training. Nevertheless, we show that it can be computed by feeding a reconstructed input to the autoencoder again. Consequently, RAPP incorporates hidden reconstruction errors as well as the ordinary reconstruction error in detecting novelty.

With extensive experiments, we demonstrate using diverse datasets that our method effectively improves autoencoder-based novelty detection methods. In addition, we show by evaluating on popular benchmark datasets that RAPP outperforms competing methods recently developed.

Our contributions are summarized as follows.

• We propose a new novelty detection method by utilizing hidden activation values of an input and its autoencoder reconstruction, and provide aggregation functions for them to quantify novelty of the input.

• We provide motivation that RAPP extends the reconstruction concept in the input space into the hidden spaces. Precisely, we show that hidden activation values of a reconstructed input are equivalent to the corresponding hidden reconstruction of the original input.

• We demonstrate that RAPP improves autoencoder-based novelty detection methods in diverse datasets. Moreover, we validate that RAPP outperforms recent novelty detection methods on popular benchmark datasets.

2. RELATED WORK:

Various novelty detection methods with deep neural networks rely on the reconstruction error (Sakurada & Yairi, 2014; Hoffmann, 2007; An & Cho, 2015) , because discriminative learning schemes are not suitable for highly class-imbalanced data which is common in practice. Unsupervised and semi-supervised learning approaches handle such imbalance by focusing on the characterization of normality and detecting samples out of the normality.

Variational Autoencoders (VAE) (Kingma & Welling, 2014) were reported to outperform vanilla autoencoders for novelty detection based on reconstruction error (An & Cho, 2015) . To carry out the novelty detection outlined in this approach, an autoencoder needs to be trained only with normal data. The autoencoder encodes the training data, which comprises of only normal data in this case, into a lower-dimensional space and decodes them to the input space. To test novelty, an input value is fed to the autoencoder to produce a reconstructed value and calculate the distance between the input and reconstructed values. This distance is the reconstruction error. A higher reconstruction error means that the input value cannot be encoded onto the lower-dimensional space that represents normal data. Therefore, the input value can be marked as a novelty if its reconstruction error exceeds a certain threshold.

Instead of autoencoders, Generative Adversarial Networks (GAN) have been also suggested to model a distribution of normal data (Sabokrou et al., 2018; Schlegl et al., 2017) . Despite the same purpose of discovering a simpler, lower-dimensional representation, the training criterion for GAN is focusing on the quality of data generation rather than the reconstruction quality of training data.

Recently, several pieces of research have combined autoencoders and adversarial learning to meet both criteria in dimension reduction and data generation (Haloui et al., 2018; Pidhorskyi et al., 2018; Zenati et al., 2018) . One limitation of these methods based on the ordinary reconstruction error is that they do not exploit all the information available along the projection pathway of deep autoencoders. We will explain how to leverage this information for novelty detection in the next section.

From the viewpoint of the diversity and ratio of the normal data in novelty detection, there are two cases available. The first case is when a small fraction of classes are normal. This case has been studied in a one-class classification context, and usually evaluated by organizing training data into a collection of samples belonging to a small number of normal classes (Ruff et al., 2018; Perera & Patel, 2018; Sabokrou et al., 2018; Golan & El-Yaniv, 2018) . The second case is when a majority of classes are assigned as normal (An & Cho, 2015; Schlegl et al., 2017; Haloui et al., 2018; Zenati et al., 2018) . In this case, normal data is more diverse, and the training data is consist of samples of a relatively large number of normal classes: e.g., nine digits of MNIST. One setup does not dominate the other, but depending on applications, either can be more suitable than the other. Different methods may perform differently in both cases. In this paper, we evaluate RAPP and other competing methods with experiments in both setups.

3. PROPOSED METHOD: RAPP:

In this section, we describe the proposed novelty detection method RAPP based on an autoencoder. The main idea is to compare hidden activations of an input and its hidden reconstructions along the projection pathway of the autoencoder. To be precise, we project the input and its autoencoder reconstruction onto the hidden spaces to obtain pairs of activation values, and aggregate them to quantify the novelty of the input. For the aggregation, we present two metrics to measure the total amount of difference within each pair.

3.1. RECONSTRUCTION BASED NOVELTY DETECTION:

An autoencoder A is a neural network consisting of an encoder g and a decoder f , responsible for dimension reduction and its inverse mapping to the original input space, respectively: i.e. A = f • g.

For the purpose, training the autoencoder aims to minimize difference between its input x and output A(x). The space that the encoder g constitutes is called the latent space, and provides more concise representation for data than the input space.

Due to this unsupervised representation learning property, the autoencoder has been widely used for novelty detection. Specifically, training an autoencoder on normal data samples, novelty of a test sample x is measured by the following reconstruction error :

= x -A(x) 2 .

The test sample x is more likely to be novel as the error (x) becomes larger, because it means that x is farther from the manifold that the autoencoder describes.

Although this approach has shown promising results in novelty detection, the reconstruction error alone does not fully exploit information provided by a trained autoencoder especially when its architecture is deep. In other words, hierarchical information identified by the deep architecture is being ignored. This is rather unfortunate because hierarchical representation learning is one of the most successfully proven capabilities of deep neural networks.

To fully leverage that capability, below we will describe the way to exploit hidden spaces to capture the difference between normal and novel samples in more detail.

3.2. RECONSTRUCTION ERROR IN HIDDEN SPACES:

Let A = f • g be a trained autoencoder where g and f are an encoder and a decoder, and be the number of hidden layers of g.

Namely, g = g • • • • • g 1 .

We define partial computation of g as follows:

g :i = g i • • • • • g 1 , for 1 ≤ i ≤ .

Let x be an input vector, and x be its reconstruction by A: i.e., x = A(x). In addition to comparing x and x in the input space, as the ordinary approach does, we examine them in hidden spaces along a projection pathway of A. More precisely, feeding x and x into A, we obtain pairs (h i , ĥi ) of their hidden representations where

h i (x) = g :i (x), ĥi (x) = g :i (x) = g :i (A(x)).

Figure 1a illustrates the procedure of computing h i and ĥi . As a result, novelty of the sample x is quantified by aggregating

H(x) = {(h i (x), ĥi (x)) : 1 ≤ i ≤ }.

The overall procedure of RAPP is summarized in Algorithm 1. To clearly state the required variables to construct H, we write the algorithm with the for loop in Lines 3-5, but in practice, all of them The quantity that RAPP computes, the hidden activation of the reconstruction input, is equivalent to the hidden reconstruction of the input. If f = f , computing ĥ 2 (x) = ĥ2 (x) does not require explicitly evaluating fi but only g i and f = f .

EQUATION

Algorithm 1: RAPP to compute a novelty score.

Input : Sample x, trained autoencoder A = f • g, the number of layers , and aggregation s.

Output: Novelty score S.

x = A(x). H = ∅. foreach i in 1 to do H = H ∪ {(g :i (x), g :i (x))}. end S = s(H).

can be computed by feed-forwarding one time each of x and x to g. Note that RAPP is indeed a generalization of the ordinary reconstruction method with defining g 0 as the identity function and s ord as follows.

s ord (H(x)) = h 0 (x) -ĥ0 (x) 2 2 , where h 0 (x) = g 0 (x) = x and ĥ0 (x) = g 0 (x) = x.

In this paper, we provide two metrics s SAP and s N AP which more extensively utilize H than s ord . Those are especially suited when no prior knowledge exists for the selection of layers to derive a novelty metric, which commonly happens when modeling with deep neural networks. Note that, however, more elaborate metrics can be designed if we have knowledge on or can characterize the spaces.

3.2.1. SIMPLE AGGREGATION ALONG PATHWAY (SAP):

This is the most straightforward metric that one can define on H. For a data sample x, SAP is defined by summing the square of Euclidean distances for all pairs in H:

s SAP (x) = i=0 h i (x) -ĥi (x) 2 2 = h(x) -ĥ(x) 2 2 ,

where h(x) and ĥ(x) are the concatenations of

[h 0 (x), • • • , h (x)

] and [ ĥ0 (x); • • • ; ĥ (x)], respectively.

3.2.2. NORMALIZED AGGREGATION ALONG PATHWAY (NAP):

Although SAP is intuitive, it does not consider properties of hidden spaces; distance distributions of pairs in H may be different depending on the individual hidden spaces. For instance, the magnitude of distances can depend on layers, or there may exist correlated neurons even across layers which are unintentionally emphasized in SAP. To capture clearer patterns, we propose to normalize the distances via two steps: orthogonalization and scaling.

Let d(x) = h(x) -ĥ(x); given a training set X, let D be a matrix whose i-th row corresponds to d(x i ) for x i ∈ X, and D be the column-wise centered matrix of D. For the normalization, we compute D = U ΣV , SVD of D, to obtain its singular values Σ and right singular vectors V . For a given data sample x, we define s N AP as follows:

s N AP (x) = (d(x) -µ X ) V Σ -1 2 2 ,

where µ X is the column-wise mean of D, and d(x) is expressed as a column vector. Note that s N AP is equal to the Mahalanobis distance with the covariance matrix V ΣΣV . Although SVD computation time is quadratic in the number of columns of the target matrix, we observe that its impact is relatively small in practical setups. See Appendix A for more details.

4. MOTIVATION OF RAPP:

One natural question in using the ordinary reconstruction method is as follows: why do we investigate only the input space? Or, why do we not use information in hidden spaces? While the reconstruction error in the input space is extensively employed, any similar concept does not exist in hidden spaces. One reason is that the corresponding encoding and decoding layers are not guaranteed to express the same space: e.g. permuted dimensions. This is because the autoencoder objective does not have any term involving activations from intermediate hidden layers. As a result, f :i+1 (g(x)) cannot be considered a reconstruction of g :i (x), except for i = 0 with which they become the ordinary reconstruction of and input to an autoencoder, respectively.

Nevertheless, in this section, we will show that there is an indirect way to compute the hidden reconstruction. Precisely, we will show that ĥi (x) = g :i (A(x)) is indeed equivalent to a reconstruction of g :i (x). The overall mechanism is depicted in Figure 1b .

4.1. COMPUTATION OF HIDDEN RECONSTRUCTION:

Let A = f • g be a trained autoencoder, and M 0 = {A(x) : x ∈ R n } be the low dimensional manifold that A describes (Pidhorskyi et al., 2018) : i.e., ∀x ∈ M 0 , x = A(x). Defining M i = {g :i (x) : x ∈ M 0 }, which is the low dimensional image of M 0 defined by g :i , g and f restricted on M 0 and M , respectively, are inverse functions of each other.

Quantifying Hidden Reconstruction We first assume that there exists a decoder f = f1

EQUATION

EQUATION

The second condition makes f :i+1 a proper decoder corresponding to g i+1: , and thus, f enables to define the i-th hidden reconstruction ĥ i (x) as follows:

ĥ i (x) = ( f :i+1 • g i+1: )(h i (x)).

Finally, we conclude that ĥ i (x) is equal to ĥi (x) for x ∈ M 0 as follows. ĥ

i (x) = ( f :i+1 • g i+1: )(h i (x)) = ( f :i+1 • g)(x) = (g :i • f • g)(x) (by equation 2) = (g :i • A)(x) = h i (x) = ĥi (x).

(by equation 1)

where we do not need fi for computing ĥ i (x), but only g i and f . Note that for x ∈ M 0 already on the manifold, its i-th hidden reconstruction ĥ i (x) becomes equal to its corresponding hidden input h i (x) = ĥi (x) for every 1 ≤ i ≤ : i.e. h i (x) = ĥ i (x) as x = A(x). For x / ∈ M 0 , its hidden reconstruction ĥ i (x) will differ from the input h i (x). Existence of f Since x = A(x) for x ∈ M 0 , g i and f i are one-to-one functions from M i-1 and M i , respectively. Let us define fi = g -1 i for M i ; then it also holds f = g -1 for M . This implies x = ( f • g)(x) for x ∈ M 0 , and consequently, f = f on M . This definition of fi satisfies the two conditions above, and as discussed, we are able to compute hidden reconstructions given an input x, through computing the i-th hidden activation of the reconstructed input: i.e. ĥ i (x) = (g

:i • A)(x) = ĥi (x).

Existence of f with Neural Networks Given g i , if the symmetric architecture for fi is used, we may not be able to learn fi = g -1 i . Neural networks are, however, highly flexible frameworks in which we can deal with models of arbitrary function forms by adjusting network architecture. This property enables us to design a layer capable of representing fi . For instance, even if fi is too complicated to be represented with a single fully connected layer, we can still approximate fi by stacking multiple layers. Hence, given g i , fi can be represented by neural networks.

5. EVALUATION:

In this section, we evaluate RAPP in comparison to existing methods. To this end, we tested the methods on several benchmarks and diverse datasets collected from Kaggle and the UCI repository which are suitable for evaluating novelty detection methods.

5.1. DATASETS AND PROBLEM SETUPS:

The datasets from Kaggle and the UCI repository are chosen from problem sets of anomaly detection and multi-class classification, summarized in Table 1 . We note that MI-F and MI-V share the same feature matrix, but are considered to be different datasets because their labels normal and abnormal are assigned by different columns: i.e. machine completed and pass visual inspection, respectively. We use these datasets to compare RAPP with standard autoencoder-based methods described in Section 5.2.

To compare RAPP with novelty detection methods in recent literatures, we also use popular benchmark datasets for evaluating deep learning techniques: MNIST (LeCun & Cortes, 2010) and F-MNIST (Xiao et al., 2017) . For theses datasets, we do not take pre-split training and test sets, but instead merge them for post-processing.

Novelty detection detects novel patterns by focusing on deviations from model-learned normal patterns. Thus, training sets contain only normal samples and test sets contain both normal and anomaly samples in our evaluation setups. Precisely, if a dataset contains an anomaly label, we assign all samples with that label to the test set for detection. If a dataset does not have any anomaly labels, we consider the following two setups.

• Multimodal Normality: A single class is chosen to be the novelty class and the remaining classes are assigned as the normal class. This setup is repeated to produce sub-datasets with all possible novelty assignments. For instance, MNIST results in a set of datasets with 10 different novelty classes.

• Unimodal Normality: In contrast to the multimodal normality setup, we take one class for normality, and the others for novelty. For instance, MNIST results in a set of datasets with 10 different normal classes.

We applied these two setups to STL, OTTO, SNSR, MNIST, and F-MNIST datasets.

5.2. COMPARISON METHOD:

We compare RAPP and the other methods using Area Under Receiver Operating Characteristic (AUROC). Note that we do not employ thresholding-based metrics such as F 1 score because access to abnormal samples is only allowed in testing time. Hence, we focus on the separability of models for novelty with AUROC.

For the datasets in Table 1 , we compare the effectiveness of the reconstruction error, SAP and NAP for three models: Autoencoder (AE), Variational Autoencoder (VAE), Adversarial Autoencoder (AAE) (Makhzani et al., 2016) . For the benchmark datasets, recent approaches including OCNN (Chalapathy et al., 2018) , GPND (Pidhorskyi et al., 2018) , DSVDD (Ruff et al., 2018) and GT (Golan & El-Yaniv, 2018) are available. To obtain the performances of the existing approaches, we downloaded their codes and applied against our problem setups.

For MNIST and F-MNIST, we create test sets of novelty ratios 35% for the multimodal setup and 50% for the unimodal setup, where novelty samples in the test sets are randomly selected from given novelty classes. For the other datasets, we take all samples in given novelty classes to create test sets. Note that the expectation value of AUROC is invariant to the novelty ratio.

5.3. IMPLEMENTATION DETAILS:

We use symmetric architecture with fully-connected layers for the three base models, AE, VAE, and AAE. Each encoder and decoder has 10 layers with different bottleneck size. For the Kaggle and UCI datasets, we carry out PCA for each dataset first. The minimum number of principal components that explain at least 90% of the variance is selected as the bottleneck size of the autoencoders.

We set bottleneck size to 20 for benchmark datasets. Leaky-ReLU (Xu et al., 2015a) activation and batch normalization (Ioffe & Szegedy, 2015) layers are appended to all layers except the last layer.

We train AE, VAE and AAE with Adam optimizer (Kingma & Ba, 2015) , and select the model with the lowest validation loss as the best model. For training stability of VAE, 10 Monte Carlo samples were averaged in the reparamterization trick (Kingma & Welling, 2014) to obtain reconstruction from the decoder. In the calculation of SAP and NAP, we excluded reconstructions in the input space for MNIST and F-MNIST.

5.4. RESULTS:

Each AUROC score is obtained by averaging AUROC scores from multiple trials to reduce the random errors in training neural networks: 5 trials for MNIST and F-MNIST, and 20 trials for the other datasets. More results are provided in Appendix: standard deviations in Appendix B, comparison to baselines other than autoencoder variants C, and the effect of varying hidden layers involved in RAPP computation in Appendix D.

5.4.1. COMPARISON WITH BASELINES:

Table 2 summarizes the result of our performance evaluation; the best score for each model is in bold, and the best score for each dataset with an underline. Since STL, OTTO, SNSR, MNIST, and F-MNIST do not have anomaly labels, their scores are averaged over all possible anomaly class assignments. For instance, the AUROC value for OTTO in the unimodal normality setup is the average of 9 AUROC values with different novelty class assignments.

In Table 2 , RAPP shows the highest AUROC scores for most of the cases. If we examine the performance for each dataset, RAPP achieves the best for 10 cases out of 15 (see the underlines). Except for the unimodal F-MNIST setup, NAP outperforms all competing methods regardless of base model choice. Notably, NAP combined with VAE always shows the best performance, which is even higher than that of GT relying on image-specific data transformations for all cases.

6. CONCLUSION:

In this paper, we propose a novelty detection method which utilizes hidden reconstructions along a projection pathway of deep autoencoders. To this end, we extend the concept of reconstruction in the input space to hidden spaces found by an autoencoder and present a tractable way to compute the hidden reconstructions, which requires neither modifying nor retraining the autoencoder. Our experimental results show that the proposed method outperforms other competing methods in terms of AUROC for diverse datasets including popular benchmarks.

A SVD COMPUTATION TIME:

We compare running times of training an autoencoder and computing SVD for NAP. We choose two packages for the SVD computation: Pytorch SVD and fbpca provided in https://fbpca. readthedocs.io/en/latest/.

Since the time complexity of SVD is linear in the number of data samples1 , we mainly focus on the performance of SVD with varying the number of columns of the input matrix that SVD is applied.

To obtain variable sizes of the columns, we vary the depth and bottleneck size of autoencoders.

The result is shown below. Notably, Pytorch SVD utilizing GPU is at least 47x faster than training neural networks. Even, fbpca running only on CPU achieves at least 2.4x speedup. The detailed setups to obtain the matrices for the experiment are given in the We investigate the performance of NAP while increasing the number of hidden layers involved in the NAP computation. Specifically, we consider two ways for the increment: 1) adding hidden layers one by one from the input layer (forward addition), and 2) adding hidden layers one by one from the bottleneck layer (backward addition). Experimental results on three datasets are shown below. For most cases, more hidden layers tend to result in higher performance. The values are obtained from one trial without averaging results from multiple trials. 

:

This is common in practice where the number of data samples is larger than the number of features

Figures and Tables:

Figure1: (a) Computation of h i (x) and ĥi (x). The reconstruction x is fed to the same autoencoder that produced itself. (b) Motivation of RAPP. The quantity that RAPP computes, the hidden activation of the reconstruction input, is equivalent to the hidden reconstruction of the input. If f = f , computing ĥ 2 (x) = ĥ2 (x) does not require explicitly evaluating fi but only g i and f = f .

Description of datasets used in our evaluation.

Table content: <table><tr><td>Name</td><td colspan="3"># Samples # Features # Class</td><td>Domain</td><td>Novelty Target</td></tr><tr><td>MI-F</td><td>25,286</td><td>58</td><td>2</td><td colspan="2">CNC milling Machine not completed</td></tr><tr><td>MI-V</td><td>23,125</td><td>58</td><td>2</td><td>CNC milling</td><td>Workpiece out-of-spec</td></tr><tr><td>EOPT</td><td>90,515</td><td>20</td><td>2</td><td>Storage system</td><td>System failures</td></tr><tr><td>NASA</td><td>4,687</td><td>33</td><td>2</td><td>Astronomy</td><td>Hazardous asteroids</td></tr><tr><td>RARM</td><td>20,221</td><td>6</td><td>2</td><td>Robotics</td><td>Malfunctions</td></tr><tr><td>STL</td><td>1,941</td><td>27</td><td>7</td><td>Steel</td><td>Surface defects</td></tr><tr><td>OTTO</td><td>61,878</td><td>93</td><td>9</td><td>E-commerce</td><td>Types of products</td></tr><tr><td>SNSR</td><td>58,509</td><td>48</td><td>11</td><td>Electric Currents</td><td>Defective conditions</td></tr><tr><td>MNIST</td><td>70,000</td><td>784</td><td colspan="2">10 Hand written digits</td><td>Digits</td></tr><tr><td>F-MNIST</td><td>70,000</td><td>784</td><td>10</td><td>Fashion articles</td><td>Articles</td></tr></table>

AUROC of RAPP and the baselines.

Table content: <table><tr><td/><td>AE</td><td/><td>VAE</td><td/><td>AAE</td><td/></tr><tr><td>Dataset</td><td>Recon SAP</td><td>NAP</td><td>Recon SAP</td><td>NAP</td><td>Recon SAP</td><td>NAP</td></tr><tr><td/><td/><td/><td colspan="2">Multimodal Normality</td><td/><td/></tr><tr><td>STL</td><td colspan="2">0.723 0.703 0.711</td><td colspan="2">0.700 0.675 0.711</td><td colspan="2">0.726 0.711 0.724</td></tr><tr><td>OTTO</td><td colspan="2">0.617 0.616 0.665</td><td colspan="2">0.630 0.631 0.649</td><td colspan="2">0.617 0.618 0.665</td></tr><tr><td>SNSR</td><td colspan="2">0.613 0.611 0.614</td><td colspan="2">0.608 0.620 0.658</td><td colspan="2">0.612 0.608 0.609</td></tr><tr><td>MNIST</td><td colspan="2">0.825 0.881 0.899</td><td colspan="2">0.900 0.965 0.965</td><td colspan="2">0.847 0.911 0.929</td></tr><tr><td colspan="3">F-MNIST 0.712 0.725 0.734</td><td colspan="2">0.725 0.728 0.755</td><td colspan="2">0.721 0.710 0.727</td></tr><tr><td/><td/><td/><td colspan="2">Unimodal Normality</td><td/><td/></tr><tr><td>MI-F</td><td colspan="2">0.607 0.670 0.705</td><td colspan="2">0.591 0.572 0.678</td><td colspan="2">0.632 0.692 0.704</td></tr><tr><td>MI-V</td><td colspan="2">0.897 0.898 0.907</td><td colspan="2">0.845 0.833 0.903</td><td colspan="2">0.895 0.891 0.904</td></tr><tr><td>EOPT</td><td colspan="2">0.610 0.607 0.625</td><td colspan="2">0.675 0.634 0.596</td><td colspan="2">0.606 0.603 0.634</td></tr><tr><td>NASA</td><td colspan="2">0.719 0.702 0.692</td><td colspan="2">0.738 0.714 0.719</td><td colspan="2">0.712 0.695 0.688</td></tr><tr><td>RARM</td><td colspan="2">0.687 0.674 0.686</td><td colspan="2">0.587 0.565 0.648</td><td colspan="2">0.664 0.675 0.675</td></tr><tr><td>STL</td><td colspan="2">0.870 0.856 0.830</td><td colspan="2">0.850 0.824 0.792</td><td colspan="2">0.875 0.861 0.830</td></tr><tr><td>OTTO</td><td colspan="2">0.829 0.829 0.832</td><td colspan="2">0.831 0.835 0.833</td><td colspan="2">0.828 0.828 0.831</td></tr><tr><td>SNSR</td><td colspan="2">0.979 0.982 0.990</td><td colspan="2">0.975 0.979 0.964</td><td colspan="2">0.979 0.983 0.991</td></tr><tr><td>MNIST</td><td colspan="2">0.972 0.980 0.979</td><td colspan="2">0.980 0.987 0.989</td><td colspan="2">0.972 0.966 0.977</td></tr><tr><td colspan="3">F-MNIST 0.924 0.928 0.933</td><td colspan="2">0.926 0.926 0.942</td><td colspan="2">0.922 0.905 0.928</td></tr></table>

AUROC on benchmark datasets.

Table content: <table><tr><td>Dataset</td><td colspan="3">OCNN GPND DSVDD</td><td>GT</td><td colspan="3">NAP AE NAP VAE NAP AAE</td></tr><tr><td/><td colspan="5">Multimodal Normality (Novelty Ratio: 35%)</td><td/><td/></tr><tr><td>MNIST</td><td>0.600</td><td>0.501</td><td>0.622</td><td>0.893</td><td>0.899</td><td>0.965</td><td>0.929</td></tr><tr><td colspan="2">F-MNIST 0.609</td><td>0.691</td><td>0.610</td><td>0.725</td><td>0.734</td><td>0.755</td><td>0.727</td></tr><tr><td/><td/><td colspan="4">Unimodal Normality (Novelty Ratio: 50%)</td><td/><td/></tr><tr><td>MNIST</td><td>0.927</td><td>0.971</td><td>0.922</td><td>0.974</td><td>0.979</td><td>0.989</td><td>0.977</td></tr><tr><td colspan="2">F-MNIST 0.915</td><td>0.917</td><td>0.923</td><td>0.935</td><td>0.933</td><td>0.942</td><td>0.928</td></tr><tr><td colspan="4">5.4.2 COMPARISON WITH COMPETITORS</td><td/><td/><td/><td/></tr></table>

summarizes the comparison of RAPP to recent novelty detection methods. As in Table2, AUROC values are calculated by averaging results from 10 cases with different anomaly class assignments for both datasets.

Table content: <table/>

We provide the standard deviations of the result in Table2. Given a dataset and a setup, multimodal or unimodal, AUROC values are first averaged over multiple cases with different novelty class assignments; then, the standard deviations are calculated for those averaged values over multiple trials. The number of cases and the number of trials depend on datasets, and we refer to Section 5 for more details. COMPARING RAPP TO BASELINES OTHER THAN NEURAL NETWORKS The table below shows the result of comparing RAPP and baselines not based on neural networks. Note that the baselines are run with hyperparameters or hyperparameter search spaces provided inRuff et al. (2018), while NAP is not further tuned.

Table content: <table><tr><td colspan="11">B STANDARD DEVIATIONS OF EXPERIMENTAL RESULTS D NAP PERFORMANCE OVER INCREASING INVOLVED HIDDEN LAYERS</td></tr><tr><td/><td colspan="2">Dataset</td><td colspan="7">OCSVM ISOF PCA kPCA NAP AE NAP VAE NAP AAE</td></tr><tr><td/><td/><td/><td>AE</td><td/><td colspan="3">VAE Multimodal Normality</td><td/><td>AAE</td></tr><tr><td>Dataset</td><td colspan="2">Recon STL</td><td>SAP 0.608</td><td colspan="3">NAP 0.635 0.700 0.693 Recon SAP</td><td>NAP 0.711</td><td>Recon 0.711</td><td>SAP 0.724</td><td>NAP</td></tr><tr><td/><td colspan="2">OTTO</td><td>0.628</td><td colspan="4">Multimodal Normality 0.548 0.572 0.644 0.665</td><td>0.649</td><td>0.665</td></tr><tr><td>STL</td><td colspan="2">0.723 SNSR</td><td>0.703 0.524</td><td colspan="3">0.711 0.526 0.536 0.580 0.700 0.675</td><td>0.711 0.614</td><td>0.726 0.658</td><td>0.711 0.606</td><td>0.724</td></tr><tr><td/><td colspan="7">(0.005) (0.007) (0.006) MNIST 0.566 0.562 0.698 0.701 (0.012) (0.018) (0.012) 0.899</td><td colspan="3">(0.009) (0.011) (0.016) 0.965 0.929</td></tr><tr><td>OTTO</td><td colspan="2">0.617 F-MNIST</td><td>0.616 0.571</td><td colspan="3">0.665 0.650 0.670 0.676 0.630 0.631</td><td>0.649 0.734</td><td>0.617 0.755</td><td>0.618 0.727</td><td>0.665</td></tr><tr><td>SNSR</td><td colspan="7">(0.002) (0.001) (0.004) 0.613 0.611 0.614 (0.002) (0.003) (0.004) MI-F 0.777 0.826 0.546 0.838 (0.003) (0.005) (0.003) 0.608 0.620 0.658 Unimodal Normality (0.001) (0.002) (0.003) 0.705</td><td colspan="3">(0.001) (0.003) (0.004) 0.612 0.608 0.609 (0.005) (0.005) (0.005) 0.678 0.704</td></tr><tr><td colspan="8">MNIST F-MNIST MI-V 0.825 (0.001) (0.007) (0.008) 0.881 0.899 0.840 0.839 0.874 0.888 0.900 0.965 (0.004) (0.005) (0.003) 0.965 0.907 EOPT 0.597 0.595 0.552 0.693 0.625 0.712 0.725 0.734 0.725 0.728 0.755 NASA 0.573 0.664 0.765 0.699 0.692 (0.001) (0.004) (0.004) (0.003) (0.002) (0.006) RARM 0.749 0.752 0.587 0.734 0.734 STL 0.879 0.871 0.883 0.880 0.880</td><td colspan="3">0.847 0.903 0.596 (0.005) (0.006) (0.005) 0.911 0.904 0.929 0.634 0.721 0.710 0.719 0.688 0.727 0.648 0.675 (0.001) (0.005) (0.005) 0.792 0.830</td></tr><tr><td/><td colspan="2">OTTO</td><td>0.821</td><td colspan="4">Unimodal Normality 0.699 0.799 0.823 0.823</td><td>0.833</td><td>: 0.831</td></tr><tr><td>MI-F</td><td colspan="2">0.607 SNSR</td><td>0.670 0.959</td><td colspan="3">0.705 0.893 0.894 0.970 0.591 0.572</td><td>0.678 0.970</td><td>0.632 0.964</td><td>0.692 0.991</td><td>0.704</td></tr><tr><td/><td colspan="7">MNIST (0.023) (0.041) (0.018) MNIST 0.894 0.919 0.903 0.906 OTTO (0.013) (0.019) (0.047) 0.979</td><td colspan="3">SNSR (0.028) (0.042) (0.024) 0.989 0.977</td></tr><tr><td>MI-V</td><td colspan="7">Depth Bottleneck Size 0.897 0.898 0.907 (0.010) (0.005) (0.005) F-MNIST 0.884 0.854 0.950 0.943 Depth Bottleneck Size 0.845 0.833 0.903 (0.021) (0.032) (0.010) 0.933</td><td colspan="3">Depth Bottleneck Size 0.895 0.891 0.904 (0.005) (0.006) (0.005) 0.942 0.928</td></tr><tr><td>1 EOPT</td><td/><td>20 0.610</td><td>0.607</td><td>100 0.625</td><td>20 0.675</td><td>0.634</td><td>40 0.596</td><td>20 0.606</td><td>0.603</td><td>90 0.634</td></tr><tr><td>2</td><td/><td colspan="3">18 (0.015) (0.014) (0.008) 100</td><td colspan="3">18 (0.017) (0.007) (0.002) 40</td><td colspan="3">18 (0.011) (0.013) (0.010) 90</td></tr><tr><td>3 4 NASA 5 6 RARM 7</td><td/><td colspan="3">16 14 0.719 12 (0.009) (0.015) (0.016) 100 100 0.702 0.692 100 10 100 0.687 0.674 0.686 8 100 (0.038) (0.031) (0.030)</td><td colspan="3">16 14 0.738 12 (0.009) (0.015) (0.009) 40 40 0.714 0.719 40 10 40 0.587 0.565 0.648 8 40 (0.021) (0.029) (0.022)</td><td colspan="3">16 14 0.712 12 (0.016) (0.027) (0.024) 90 0.695 0.688 90 90 10 0.664 0.675 0.675 90 8 90 (0.015) (0.023) (0.040)</td></tr><tr><td>8 STL 9</td><td/><td colspan="3">6 0.870 4 (0.007) (0.007) (0.007) 100 0.856 0.830 100</td><td colspan="3">6 0.850 4 (0.006) (0.003) (0.034) 40 0.824 0.792 40</td><td colspan="3">6 0.875 4 (0.003) (0.004) (0.008) 0.861 0.830 90 90</td></tr><tr><td>10 OTTO</td><td/><td>2 0.829</td><td>0.829</td><td>100 0.832</td><td>2 0.831</td><td>0.835</td><td>40 0.833</td><td>2 0.828</td><td>0.828</td><td>90 0.831</td></tr><tr><td>11</td><td/><td colspan="3">2 (0.001) (0.001) (0.001) 80</td><td colspan="3">2 (0.002) (0.003) (0.004) 30</td><td colspan="3">2 (0.002) (0.001) (0.002) 70</td></tr><tr><td>12 SNSR</td><td/><td>2 0.979</td><td>0.982</td><td>60 0.990</td><td>2 0.975</td><td>0.979</td><td>20 0.964</td><td>2 0.979</td><td>0.983</td><td>50 0.997</td></tr><tr><td>13</td><td/><td colspan="3">2 (0.001) (0.001) (0.001) 40</td><td colspan="3">2 (0.003) (0.002) (0.043) 10</td><td colspan="3">2 (0.001) (0.002) (0.001) 30</td></tr><tr><td>14 MNIST</td><td/><td>2 0.972</td><td>0.980</td><td>20 0.979</td><td>0.980</td><td>0.987</td><td>0.989</td><td>2 0.972</td><td>0.966</td><td>10 0.977</td></tr><tr><td/><td/><td colspan="3">(0.001) (0.001) (0.000)</td><td colspan="3">(0.002) (0.002) (0.001)</td><td colspan="3">(0.001) (0.004) (0.001)</td></tr><tr><td colspan="2">F-MNIST</td><td>0.924</td><td>0.928</td><td>0.933</td><td>0.926</td><td>0.926</td><td>0.942</td><td>0.922</td><td>0.905</td><td>0.928</td></tr><tr><td/><td/><td colspan="3">(0.002) (0.002) (0.002)</td><td colspan="3">(0.001) (0.001) (0.000)</td><td colspan="3">(0.001) (0.002) (0.002)</td></tr></table>

Peer review (iclr2020):
UPDATE: I acknowledge that I‘ve read the author responses as well as the other reviews. I appreciate the improvements and clarifications the authors have made, especially adding an ablation study to see the benefits of adding additional layers. I updated my score to Weak Accept (6). #################### This paper considers deep autoencoders (AEs) for the unsupervised novelty/anomaly detection task and proposes to extend the standard AE anomaly score, given by the reconstruction error between the input and output in the original data space, to also utilize the reconstruction errors of the hidden activations in the AE network. The proposed method, Reconstruction along Projection Pathway (RaPP), specifically compares the hidden activations of all encoder layers given by the original input with the activations of the same units given by feeding the reconstruction back into the AE. Thus RaPP compares the activation statistics of the original input and its reconstruction along the encoder projection pathway from original data space to latent code space. Two ways for aggregating those reconstruction errors to a final anomaly score are presented: (1) Simple Aggregation Along Pathway (SAP) which simply computes the sum of reconstruction errors, and (2) Normalized Aggregation Along Pathway (NAP) which computes the sum of reconstruction errors after normalization via Singular Value Decomposition (SVD). The paper conclusively presents experiments on eight datasets from various domains, in which SAP and NAP are compared to the reconstruction error baseline for vanilla AE, VAE, and AAE, as well as experiments on MNIST and Fashion-MNIST in which NAP is compared to state-of-the-art deep anomaly detectors. Though this work is well presented and indicates promising results, I think the paper should not yet be accepted due to the following main reasons: (i) The experimental evaluation indicates promising, but not yet convincing results; (ii) The computational complexity of NAP seems to be a major limitation of RaPP which is not addressed in the text; (iii) The added value/insights from the theoretical Section 4 (Motivation of RaPP) are not clear. (i) I think the experimental section shows promising, but not yet convincing results. To judge the significance of results, I think the paper should address the following: (ia) The experiments on the eight non-image datasets should include other baselines (e.g. OC-SVM, Isolation Forest) besides the standard AE reconstruction error. One should expect SAP and NAP to improve over the standard AE since both methods include the original data space reconstruction errors as well. Moreover, the advantage of deep approaches on such non-image datasets is less clear [7] why a comparison to well-known baselines should be given. (ib) The main motivation for deep approaches to anomaly detection are large and complex datasets [6, 5, 4, 2]. I think the comparison to recent, state-of-the-art deep competitors should at least include another dataset more complex than MNIST or Fashion-MNIST, e.g. CIFAR-10 as reported in the previous works or MVTec [1]. (ic) I think the proposed method begs for an ablation study of subsequently adding the reconstruction errors of additional layers. This would clearly demonstrate the potential benefits of adding the hidden reconstructions. (ii) The experiments indicate that a proper normalization of the hidden activation reconstruction errors is crucial for improving detection performance. NAP shows consistent improvements, whereas SAP often performs similar to the AE baseline. However, the current SVD normalization procedure on a matrix with dimensions number of samples × number of hidden encoder units seems extremely costly to me and appears to be a major limitation towards larger datasets or networks. Could you comment on this since this is not yet addressed in the manuscript. Have you tried using Batch Normalization (after activation) together with per-layer averaging? To me, this seems the natural first choice to normalize unit scores and to account for different layer widths. Do you apply SVD on mini-batches? (iii) The additional insights from the theoretical Section 4 are not clear to me. I think the presented reconstruction property for the hidden layers follows somewhat directly per definition for symmetrically constructed deep autoencoders (specifically if weights would be shared in addition). For a theoretical contribution, on the other hand, the proof and proposition should be fully rigorous in my mind, i.e. stating all the necessary assumptions on the function class (e.g. you implicitly assume invertibility and thus some smoothness of the *s which Conv+ReLU modules do not satisfy for instance). As of now, I think this section does not add to intuition, but on the other hand is not completely rigorous. Maybe I am missing something? The overall presentation of the paper is good (clear writing and structure, polished Figures and Tables). The work is well motivated and properly placed in the literature. Maybe since the approach is rather simple (which I don’t find negative), the author felt the need to add some rigor to the paper, which I think would not be necessary for a significant contribution if the experimental results hold up against the additional baselines and more complex datasets as described in (i). #################### *Additional Feedback* *Positive Highlights* 1. Simple idea that does not require autoencoder modification or retraining that indicates improved anomaly detection results. 2. The work is well placed in the literature. The related work includes all relevant and recent major works on the subject matter. 3. I appreciate the evaluation on both anomaly/novelty detection setups, unimodal and multimodal. 4. Comparison to recent OC-NN [3], GPND [5], Deep SVDD [6], and GT [4]. 5. The writing, structure and overall presentation is good. *Ideas for Improvement* 6. Include additional baselines and more complex datasets as described in (i). 7. Address the computational complexity of RaPP as in described in (ii). 8. Maybe cut the methodical/theoretical parts in Section 3.2 and Section 4 a bit. I think they are rather straightforward. Maybe combine Figures 1+2 as well. Extend the experimental evaluation instead. 9. Report the AUROC standard deviations over the trials as well to better infer statistical significance of the results (defer to appendix if space is a constraint). *Minor comments* 10. Section 2: “Unsupervised and semi-supervised learnings” » “Unsupervised and semi-supervised learning approaches”. 11. Section 2: “Variational Autoencoders (VAE) was reported ...” » “Variational Autoencoders (VAE) were reported ...” 12. Section 3.1: “Due to this representation learning property, the autoencoder has been widely used for novelty detection.” » emphasis on unsupervised learning property, specifically. 13. Section 3.1: “Although this approach has shown a promising result in novelty detection ...” » “Although this approach has shown promising results in novelty detection ...” 14. Section 3.1, last sentence: “... in more details.” » “... in more detail.” 15. Section 3.2: “Those are especially suited for the case of zero-knowledge to interpret identified hidden spaces, which commonly happens when modeling with deep neural networks.” Zero-knowledge case? Reference? 16. In Section 5.1: “Further setups are described in Section 5.1”? 17. Section 5.4.1: “Also, we showed the best score ...” » “Also, we show the best score ...”. 18. Section 5.2: “... maintaining novelty ratios to 35% for the multimodal and 50% for the unimodal normality setups, respectively.” Why use different ratios? #################### *References* [1] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9592–9600, 2019. [2] R. Chalapathy and S. Chawla. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407, 2019. [3] R. Chalapathy, A. K. Menon, and S. Chawla. Anomaly detection using one-class neural networks. arXiv preprint arXiv:1802.06360, 2018. [4] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In NIPS, 2018. [5] S. Pidhorskyi, R. Almohsen, and G. Doretto. Generative probabilistic novelty detection with adversarial autoencoders. In NeurIPS, pages 6822–6833, 2018. [6] L. Ruff, R. A. Vandermeulen, N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In International Conference on Machine Learning, pages 4393–4402, 2018. [7] L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694, 2019.