RaPP: Novelty Detection with Reconstruction along Projection Pathway
Authors: Ki Hyun Kim
Sangwoo Shim
Yongsub Lim
Jongseob Jeon
Jeongwoo Choi
Byungchan Kim
Andre S. Yoon
Venue:  (2020)
Keywords: Novelty Detection
Anomaly Detection
Outlier Detection
Semi-supervised Learning

Abstract: We propose RaPP, a new methodology for novelty detection by utilizing hidden space activation values obtained from a deep autoencoder.
Precisely, RaPP compares input and its autoencoder reconstruction not only in the input space but also in the hidden spaces.
We show that if we feed a reconstructed input to the same autoencoder again, its activated values in a hidden space are equivalent to the corresponding reconstruction in that hidden space given the original input.
In order to aggregate the hidden space activation values, we propose two metrics, which enhance the novelty detection performance.
Through extensive experiments using diverse datasets, we validate that RaPP improves novelty detection performances of autoencoder-based approaches.
Besides, we show that RaPP outperforms recent novelty detection methods evaluated on popular benchmarks.


Published as a conference paper at ICLR 2020
RAPP: NOVELTY DETECTION WITH RECONSTRUC- TION ALONG PROJECTION PATHWAY
Ki Hyun Kim, Sangwoo Shim, Yongsub Lim, Jongseob Jeon, Jeongwoo Choi, Byungchan Kim, Andre S. Yoon
MakinaRocks
fkhkim, sangwoo, yongsub, jongseob. jeon [ f narocks.. [jeongwoo, kbc8894, andre (aaeee)mmmiiaarrer
ABSTRACT
We propose RAPP, a new methodology for novelty detection by utilizing hid- den space activation values obtained from deep autoencoder. Precisely, RAPP compares input and its autoencoder reconstruction not only the input space but also in the hidden spaces. We show that if we feed reconstructed input to the same autoencoder again, its activated values in a hidden space are equivalent to the corresponding reconstruction in that hidden space given the original input. We devise two metrics aggregating those hidden activated values to quantify the nov- elty of the input. Through extensive experiments using diverse datasets, we val- idate that RAPP improves novelty detection performances autoencoder- based approaches. Besides, we show that RAPP outperforms recent novelty detection methods evaluated on popular benchmarks.
INTRODUCTION
How can we characterize novelty when only normality information is given? Novelty detection is the mechanism to decide whether data sample is an outlier with respect the training data. This mechanism is especially useful in situations where proportion of detection targets is inherently
small. Examples are fraudulent transaction detection (Pawar et al. 2014] Porwal & Mukund 2018).
intrusion detection (Lee 2017 Aoudi et al. 2018). video surveillane lance (Ravanbakhsh et al. 2017
Xu al. 2015b), medical diagnosis (Schlegl et 2017 Baur al. 2018) and equipment failure
detection Kuzin & Borovicka 2016 Zhao et al.. 2017 Beghi et al. 2014). Recently, deep autoen- coders and their variants have shown outstanding performances in finding compact representations from complex data, and the reconstruction error has been chosen ing novelty (An & Cho 2015) Vasilev et al.) 2018). However, this approach has a limitation of measuring reconstruction quality only in input space, which does not fully utilize hierarchical representations in hidden spaces identified by the deep autoencoder.
popular metric for detect-
In this paper, we propose RAPP, tivation values addition to the input values and their autoencoder reconstruction values. While ordinary reconstruction-bast tween input data before the input layer and reconstructed data at the output layer, RAPP extends these comparisons hidden spaces. We first collect of hidden activation values by feeding the original input to the autoencoder. Subsequently, we feed the autoencoder reconstructed input to the autoencoder calculate another set of activation values in the hidden layers. This procedure does not need additional training of the autoencoder. In turn, we quantify the novelty of the input by aggregating these two sets of hidden activation values. To this end, we devise two metrics. The first metric measures the total amount of reconstruction errors in input and hidden spaces. The second metric normalizes the reconstruction errors before summing up. Note that RAPP falls back to the ordinary reconstruction-bu based method if we only aggregate input values before the input layer and the reconstructed values at the output layer.
method of detecting novelty samples exploiting hidden
methods carry novelty detection by comparing differences be-
Also, we explain the motivations that facilitated the development of RAPP. We show that activation values a hidden space obtained by feeding a reconstructed input to the autoencoder are equivalent to the corresponding reconstruction in that hidden space for the original input. We refer the latter

---

Published as a conference paper at ICLR 2020
quantity hidden reconstruction of the input. Note that this is natural extension of the recon- struction to the hidden space. Unfortunately, we cannot directly compute the hidden reconstruction as in the computation of the ordinary reconstruction because the autoencoder does not impose any correspondence between encoding-decoding pairs of hidden layers during the training. Neverthe- less, we show that it be computed by feeding reconstructed input the autoencoder again. Consequently, RAPP incorporates hidden reconstruction errors as well as the ordinary reconstruc- tion error detecting novelty.
With extensive experiments, we demonstrate using diverse datasets that our method effectively im- proves uutoencoder--assee novelty detection methods. In addition, we show by evaluating on popular benchmark datasets that RAPP outperforms competing methods recently developed.
Our contributions are summarized follows.
We propose new novelty detection method by utilizing hidden activation values of an imput and its autoencoder reconstruction, and provide aggregation functions for them to quantify novelty of the input.
We provide motivation that RAPP extends the reconstruction concept in the input space into the hidden spaces. Precisely, we show that hidden activation values of reconstructed input are equivalent to the corresponding hidden reconstruction of the original input.
We demonstrate that RAPP improves autoencoder-basse novelty detection methods in di- verse datasets. Moreover, we validate that RAPP outperforms recent novelty detection methods on popular benchmark datasets.
2 RELATED WORK
Various novelty detection methods with deep neural networks rely the reconstruction error (Saku- rada & Yairi 2014; Hoffmann] [207] An Cho (015). because discriminative learning schemes are not suitable for highly class-imbalanced data which is common in practice. Unsupervised and semi-supervised learning approaches handle such imbalance by focusing on the characterization of normality and detecting samples out of the normality.
Variational Autoencoders (VAE) (Kingma & Welling 2014) were reported to outperform vanilla autoencoders for novelty detection based on reconstruction error (An & Cho (0155). To carry out the novelty detection outlined in this approach, an autoencoder needs to be trained only with normal data. The autoencoder encodes the training data, which comprises of only normal data this case, into a lower-dimensional space and decodes them to the input space. To test novelty, an input value fed to the autoencoder to produce a reconstructed value and calculate the distance between the input and reconstructed values. This distance is the reconstruction error. A higher reconstruction error means that the input value cannot be encoded onto the sewer-dimessioall space that represents normal data. Therefore, the input value can be marked novelty if its reconstruction error exceeds a certain threshold.
Instead of autoencoders, Generative Adversarial Networks (GAN) have been also suggested to model a distribution of normal data (Sabokrou et al. 2018 Schlegl et al. 2017). Despite the same purpose of discovering simpler, lower-dimensional representation, the training criterion for GAN is focusing on the quality of data generation rather than the reconstruction quality of training data. Recently, several pieces of research have combined autoencoders and adversarial learning to meet both criteria in dimension reduction and data generation (Haloui et al.) 2018 Pidhorskyi et al., 2018 Zenati et al., 2018). One limitation of these methods based on the ordinary reconstruction error is that they do not exploit all the information available along the projection pathway of deep autoen- coders. We will explain how to leverage this information for novelty detection in the next section.
From the viewpoint of the diversity and ratio of the normal data in novelty detection, there are two cases available. The first case is when a small fraction of classes are normal. This case has been studied in a one-class classification context, and usually evaluated by organizing training data into a collection of samples belonging to a small number of normal classes (Ruff et al., 2018 Perera & Patel 2018, Sabokrou al. 2018 Golan & El- Yanyy 2018). The second case is when majority of classes are assigned as normal (An & Chol (015] Schlegl et al. 2017:; Haloui et al. 2018 Zenati
et al. 2018). In this case, normal data is more diverse, and the training data is consist samples of
2

---

Published as a conference paper at ICLR 2020
relatively large number of normal classes: e.g., nine digits of MNIST. One setup does not dominate the other, but depending on applications, either can more suitable than the other. Different meth- ods may perform differently in both cases. In this paper, we evaluate RAPP and other competing methods with experiments both setups.
PROPOSED METHOD: RAPP
In this section, we describe the proposed novelty detection method RAPP based on an autoencoder. The main idea is to compare hidden activations of an input and its hidden reconstructions along the projection pathway the autoencoder. To be precise, we project the input and its autoencoder reconstruction onto the hidden spaces obtain pairs of activation values, and aggregate them to quantify the novelty of the input. For the aggregation, we present two metrics to measure the total amount of difference within each pair.
3.1 RECONSTRUCTION BASED NOVELTY DETECTION
An autoencoder A is a neural network consisting of an encoder and decoder f, responsible for dimension reduction and its inverse mapping the original imput space, respectively; i.e. f For the purpose, training the autoencoder aims to minimize difference between its input æ and output A(x). The space that encoder y constitutes called the latent space, and provides more concise representation for data than the input space.
Due to this unsupervised representation learning property, the autoencoder has been widely used for novelty detection. Specifically, training autoencoder normal data samples, novelty of test sample x is measured by the following reconstruction error e:
c= c= -()))22
The test sample is more likely to be novel as the error e(z) becomes larger, because it means that x is farther from the manifold that the autoencoder describes.
Although this approach has shown promising results in novelty detection, the reconstruction error alone does not fully exploit information provided by trained autoencoder especially when its archi- tecture is deep. In other words, hierarchical information identified by the deep architecture is being ignored. This is rather unfortunate because hierarchical representation learning is one of the most successfully proven capabilities of deep neural networks.
To fully leverage that capability, below we will describe the way to exploit hidden spaces to capture the difference between normal and novel samples in more detail.
3.2 RECONSTRUCTION ERROR IN HIDDEN SPACES
Let A f be trained autoencoder where g and f are encoder and decoder, and é be the number of hidden layers of Namely, g gr o ... 91- We define partial computation of g as follows:
9+ = 9: og1
for 1 sisé.
Let æ be an input vector, and f be its reconstruction by A: i.c., f = A(x). In addition to comparing and in the input space, as the ordinary approach does, examine them in hidden spaces along a projection pathway of More precisely, feeding x and & into A, we obtain pairs (he, hs) of their hidden representations where
h1(2) gx(
hx(z) g.((t) g,((A(z)).
Figure la illustrates the procedure of computing h; and h4 As result, novelty of the sample is quantified by aggregating H(x)) ((h, h,(x)) :1 15 f).
The overall procedure RAPP is summarized in Algorithm To clearly state the required variables construct H, write the algorithm with the for loop in Lines 3-5, but in practice, all of them

---

Published as a conference paper at ICLR 2020
hidden activation Input
hg (x)
(x) (x(x)
22
81
82.
fa
91
(:) 82(x)
R2(x)= (ʃ2 g)(
Midden activiation of reconstracted input
hiddees reconstraation inpurt (not computable directly)
(x) (x(x)
(a) Hidden activations of input and reconstruction
(b) Indirect computation of hidden reconstruction
Figure I: (a) Computation of (z) and (x). The reconstruction is fed to the same autoencoder that produced itself. (b) Motivation of RAPP. The quantity that RAPP computes, the hidden acti- vation of the reconstruction input, is equivalent the hidden reconstruction of the input. f f. computing (x) (1)) does not require explicitly evaluating fi but only yi and f = f.
Algorithm 1: RAPP to compute a novelty score.
Input Sample x, trained autoencoder A = I' the number of layers E, and aggregation Output: Novelty score
i =A(r).
H Ø.
foreach i in to l do
H HU ((),())
end
S= sHH.
can be computed by feed-forwarding one time each of and & g. Note that RAPP is indeed generalization of the ordinary reconstruction method with defining 9u as the identity function and Sord as follows.
Sora(H())) (1h((1) (o)
where ho(x)= = go(x)= and ho(x) (0 = a.
In this paper, we provide two metrics SSAP and SNAP which more extensively utilize H than Sond- Those are especially suited when no prior knowledge exists for the selection of layers to derive novelty metric, which commonly happens when modeling with deep neural networks. Note that, however, more elaborate metrics can be designed if we have knowledge on or can characterize the spaces.
3.2.1 SIMPLE AGGREGATION ALONG PATHWAY (SAP)
This is the most straightforward metric that define H. For data sample SAP is defined by summing the square of Euclidean distances for all pairs in H:
$SAP(x)
(m)()) (,)2))1 [h(x:) f(())]l.
where hix)) and hi(z)) are the concatenations of (ho) (x) and [ho());-.. he(x))) respec- tively.
3.2.2 NORMALIZED AGGREGATION ALONG PATHWAY (NAP)
Although SAP is intuitive, it does not consider properties of hidden spaces; distance distributions of pairs in H may be different depending the individual hidden spaces. For instance, the magnitude
4

---

(A(x) x [R²) be the low dimensional Published as a conference paper at ICLR 2020
of distances can depend layers, there may exist correlated neurons across layers which are unintentionally emphasized in SAP. To capture clearer patterns, we propose to normalize the distances via two steps: orthogonalization and scaling.
Let (xx) (xx) h(z);; given training set X, let D be matrix whose i-th row corresponds to (a;; for x; E X, and D be the column-wise centered matrix of D. For the normalization, we compute D UEVT, SVD of D, obtain its singular values Σ and right singular vectors V. For a given data sample æ, we define ANAP as follows:
8NAP(2) = (d(x) - px ) VE VE- 112
where ax is the column-wise mean of D, and d(xr) is expressed SN equal to the Mahalanobis distance with the covariance matrix VΣEVT. Although SVD computation time is quadratic in the number of columns of the target matrix, observe that its impact is relatively small in practical setups. See Appendix [A] for more details.
column vector. Note that
MOTIVATION OF RAPP
One natural question in using the ordinary reconstruction method is follows: why do in- vestigate only the input space? Or, why do we not use information in hidden spaces? While the reconstruction eπrоr in the input space is extensively employed, any similar concept does not ist in hidden spaces. One reason is that the corresponding encoding and decoding layers are not guaranteed to express the same space: e.g. permuted dimensions. This is because the autoencoder objective does not have any term involving activations from intermediate hidden layers. As a re- sult, fea+1 (g(x)) cannot be considered reconstruction of gx), except for 0 with which they become the ordinary reconstruction of and input to an autoencoder, respectively.
Nevertheless, in this section, we will show that there is an indirect way to compute the hidden recon- struction. Precisely, we will show that hy (x) 9:: 9:: (A(x) is indeed equivalent a reconstruction of (z) The overall mechanism is depicted in Figure lb
4.1 COMPUTATION OF HIDDEN RECONSTRUCTION
Let A = ff g be trained autoencoder, and Mo manifold that A describes (Pidhorskyi et al.
2018 i.e.,
Vo Mo, x (zx).
Defining M, = (z)(x)) x E Mo). which is the low dimensional image of Mo defined by J.s. 9 and restricted My and Me, respectively, inverse functions of each other.
Quantifying Hidden Reconstruction such that
We first assume that there exists decoder f f= o...0
Vx E Me, f(z) = f(x),
(1)
Va E M,, (g: (a).
(2)
The second condition makes fes+1 proper decoder corresponding g++1. and thus, f enables to define the i-th hidden reconstruction (4(x)) as follows:
[((x) = (Jest1 (44)))
Finally, we conclude that (((x) is equal (z(2)) for x E Mo as follows.
R((x) (Jeap o 9).1)((h,x))) (frrm o (x)
(9: o f o (z)
(by equation
(0. A)(x) h₂(8) i,(x).
(by equation
where we do not need F. for computing i((x), but only 9; and f. Note that for x Mo already on the manifold, its i-th hidden reconstruction M((x)) becomes equal to its corresponding hidden input
(x) = hz(z) for every 1 < i < & i.e. hx(x)) = h((x) as A(x). For & Mo, its hidden reconstruction (x) will differ from the input h.(x))
5

---

Published as a conference paper at ICLR 2020
Table 1: Description of datasets used in our evaluation.
Name
# Samples
Features
# Class
Domain
Novelty Target
MI F
25,286
58
CNC milling CNC milling
Machine not completed
MI
23,125
58
Workpiece out-of- spee
EOPT
90,515
20
Storage system
System failures
NASA
4,687
33
Astronomy
Hazardous asteroids Malfunctions
RARM
20,221
6
Robotics
STL
1,941
27
Steel
Surface defects
OTTO
61,878
93
E-commerce Electric Currents
Types of products Defective conditions
SNSR
58,509
48
11
MNIST
70,000
10
Hand written digits
Digits Articles
F-NNIST
70,000
784
10
Fashion articles
Existence of/ f Since x = A(x) for x E Mo. 9: and fi are one-to-one functions from M,-1 and M, respectively. Let us define fi - 9; for M;; then it also holds f f= g- for My. This implies (f (z) for Ma. and consequently, f = f on My. This definition of f. satisfies the two conditions above, and as discussed, we are able to compute hidden reconstructions given an input x, through computing the i-th hidden activation of the reconstructed input: i.e. M((x) = (g.s Az) = (x).
Existence of f with Neural Networks Given 9r. if the symmetric architecture for ft is used, we may not be able to learn f. 9; Neural networks are. however, highly flexible frameworks in which we can deal with models arbitrary function forms by adjusting network architecture. This property enables us to design layer capable of representing fi- For instance, even if fi is too complicated be represented with single fully connected layer, we can still approximate fü by stacking multiple layers. Hence, given gt. ft can be represented by neural networks.
EVALUATION
In this section, we evaluate RAPP in comparison existing methods. To this end, we tested the methods on several benchmarks and diverse datasets collected from Kaggle and the UCI repository which are suitable for evaluating novelty detection methods.
5.1 DATASETS AND PROBLEM SETUPS
The datasets from Kaggle and the UCI repository chosen from problem sets of anomaly detection and multi-class classification, summarized in Table We note that MI-F and MI-V share the same feature matrix, but are considered to be different datasets because their labels normal and abnormal are assigned by different columns: i.c. machine completed and pass visual inspection, respectively. We use these datasets to compare RAPP with standard autoencoder-basse methods described in Section 5.2
To compare RAPP with novelty detection methods in recent literatures, we also use popular bench- mark datasets for evaluating deep learning techniques: MNIST (LeCun & Cortes 2010) and F- MNIST (Xiao et al. 2017). For theses datasets, we do not take pre-split training and test sets, but instead merge them for post-processing.
Novelty detection detects novel patterns by focusing on deviations from model-learned normal pat- terns. Thus, training sets contain only normal samples and test sets contain both normal and anomaly samples in our evaluation setups. Precisely, a dataset contains an anomaly label, we assign all sam- ples with that label to the test set for detection. If dataset does not have any anomaly labels, we consider the following two setups.
Multimodal Normality: single class is chosen to be the novelty class and the remaining classes are assigned the normal class. This setup is repeated to produce sub-datasets

---

Published as a conference paper at ICLR 2020
with all possible novelty assignments. For instance, MNIST results in set of datasets with 10 different novelty classes.
Unimodal Normality: In contrast to the multimodal normality setup, we take one class for normality, and the others for novelty. For instance, MNIST results in set of datasets with 10 different normal classes.
We applicd these two setups to STL, OTTO, SNSR, MNIST, and F-MNIST datasets.
5.2 COMPARISON METHOD
We compare RAPP and the other methods using Area Under Receiver Operating Characteristic (AUROC). Note that do not employ thresholding-basee baaeed metrics such as F'l score because access to abnormal samples is only allowed testing time. Hence, we focus on the separability of models for novelty with AUROC.
For the datasets in Table T we compare the effectiveness of the reconstruction error, SAP and NAP for three models: Autoencoder (AE), Variational Autoencoder (VAE), Adversarial Autoen- coder (AAE) (Makhzani et al.) 2016). For the benchmark datasets, recent approaches including OCNN (Chalapathy al. 2018), GPND (Pidhorskyi et al., 2018), DSVDD (Ruff al., (018) and GT (Golan & Yaniv (2018) are available. To obtain the performances of the existing approaches, downloaded their codes and applied against our problem setups.
For MNIST and F-MNIST, we create test sets of novelty ratios 35% for the multimodal setup and 50% for the unimodal setup, where novelty samples in the test sets are randomly selected from given novelty classes. For the other datasets, we take all samples in given novelty classes to create test sets. Note that the expectation value of AUROC is invariant to the novelty ratio.
5.3 IMPLEMENTATION DETAILS
We use symmetric architecture with fully-connected layers for the three base models, AF, VAE, and AAE. Each encoder and decoder has 10 layers with different bottleneck size. For the Kaggle and UCI datasets, we carry out PCA for each dataset first. The minimum number of principal compo- nents that explain at least 90% of the variance is selected the bottleneck size of the autocncoders. We set bottleneck size to 20 for benchmark datasets. Leaky-ReL. (Xu et al. 2015a) activation and batch normalization (loffe & Szegedy 2015) layers appended to all layers except the last layer.
We train AE, VAE and AAE with Adam optimizer (Kingma & Ba 2015), and select the model with the lowest validation loss the best model. For training stability of VAE, 10 Monte Carlo samples were averaged in the reparamterization trick (Kingma & Welling 2014) to obtain reconstruction from the decoder. In the calculation of SAP and NAP, excluded reconstructions in the input space for MNIST and F-MNIST.
5.4 RESULTS
Each AUROC score is obtained by averaging AUROC scores from multiple trials to reduce the random errors training neural networks: 5 trials for MNIST and F-MNIST, and 20 trials for the other datasets. More results are provided in Appendix: standard deviations in Appendix B comparison to baselines other than autoencoder variants C and the effect of varying hidden layers involved in RAPP computation in Appendix
5.4.1 COMPARISON WITH BASELINES
Table 2 summarizes the result of our performance evaluation; the best score for each model is in bold, and the best score for each dataset with an underline. Since STL, OTTO, SNSR, MNIST, and F-MNIST do not have anomaly labels, their scores averaged over all possible anomaly class assignments. For instance, the AUROC value for OTTO the unimodal normality setup is the average of AUROC values with different novelty class assignments.
In Table 2, RAPP shows the highest AUROC scores for most of the cases. If we examine the performance for each dataset, RAPP achieves the best for 10 cases out of 15 (see the underlines).

---

Published as a conference paper at ICLR 2020
Table 2: AUROC of RAPP and the basclines.
AE
VAE
AAE
Dataset
Recon
SAP
NAP
Recon
SAP
NAP
Recon
SAP
NAP
Multimodal Normality
STL
0.723
0.703
0.711
0.700
0,675
0.711
0.726
0.711
0.724
ОТТО
0.617
0.616
0.665
0.630
0.631
0.649
0.617
0.618
0.665
SNSR
0.613
0.611
0.614
0.608
0.620
0.658
0.612
0.608
0.609
MNIST
0.825
0.881
0.899
0,900
0.965
0.965
0.847
0.911
0.929
F-MNIST
0.712
0.725
0.734
0.725
0.728
0.755
0.721
0.710
0.727
Unimodal Normality
MI-F
0.607
0,670
0.705
0.591
0.572
0.678
0.632
0.692
0.704
MI-V
0.897
0.898
0.907
0.845
0.833
0.903
0.895
0.891
0.904
HOPT
0.610
0.607
0.625
0.675
0.634
0.596
0.606
0.603
0.634
NASA
0.719
0.702
0.692
0.738
0.714
0.719
0.712
0.695
0.688
RARM
0.687
0.674
0.686
0,587
0.565
0.648
0,664
0.675
0.675
STL
0.870
0.856
0.830
0.850
0.824
0.792
0.875
0.861
0.830
OTTO
0.829
0.829
0.832
0.831
0.835
0.833
0.828
0.828
0.831
SNSR
0.979
0.982
0.990
0.975
0.979
0.964
0.979
0.983
0.991
MNIST
0.972
0.980
0.979
0.980
0.987
0.989
0.972
0.966
0.977
F-MNIST
0.924
0.928
0.933
0.926
0.926
0.942
0.922
0.905
0.928
Table 3: AUROC on benchmark datasets.
Dataset
OCNN
GPND
DSVDD
GT
NAPAE NAPVAE NAPAAE
Multimodal Normality (Novelty Ratio: 35%)
MNIST
0,600
0.501
0.622
0.893
0.899
0.965
0.929
F-MNIST
0.609
0.691
0.610
0.725
0.734
0.755
0.727
Unimodal Normality (Novelty Ratio: 50%)
MNIST
0.927
0.971
0.922
0.974
0.979
0.989
0.977
F-NNIST
0.915
0.917
0.923
0.935
0.933
0.942
0.928
5.4.2 COMPARISON WITH COMPETITORS
Table summarizes the comparison of RAPP to recent novelty detection methods. As in Table AUROC values are calculated by averaging results from cases with different anomaly class as- signments for both datasets.
Except for the unimodal F-MNIST setup, NAP outperforms all competing methods regardless of base model choice. Notably, NAP combined with VAE always shows the best performance, which even higher than that GT relying on image-specific data transformations for cases.
CONCLUSION
this paper, we propose novelty detection method which utilizes hidden reconstructions along a projection pathway of deep autoencoders. To this end, we extend the concept of reconstruction in the input space to hidden spaces found by an autoencoder and present a tractable way to compute the hidden reconstructions, which requires neither modifying retraining the autoencoder. Our experimental results show that the proposed method outperforms other competing methods in terms of AUROC for diverse datasets including popular benchmarks.
8

---

Published as a conference paper at ICLR 2020
REFERENCES
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruc- tion probability. SNUDM-TR, Mar 2015.
Wissam Aoudi, Mikel Iturbe, and Magnus Almgren. Truth will out: Departure- process- level detection of stealthy attacks control systems. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. ACM, 2018.
Christoph Baur, Benedikt Wiestler, Shadi Albarqouni, and Nassir Navab. Deep autoencoding models for unsupervised anomaly segmentation in brain mr images. MICCAI, 2018.
Alessandro Beghi, Luca Cecchinato, C Corazzol, Mirco Rampazzo, F Simmini, and Gian Antonio Susto. A one-class sym based tool for machine learning novelty detection in hvac chiller systems. IFAC Proceedings Volumes (IFAC-PuperSOnlin
19: 19:1953-1958, 2014.
Raghavendra Chalapathy, Aditya Krishna Menon, and Sanjay Chawla. Anomaly detection using one-class neural networks. arXiv preprint /80X.vv:800.0000 2018.
EOPT. http://wwww..aa....oo..ooo.ooooo hwwp://ww............dodooooooom
F-MNIST. https//gghhaa.ooommaaddooeeerest
Izhak Golan and Ran Elniv..v. Deep anomaly detection using geometric transformations. NIPS, 2018.
llyass Haloui, Jayant Sen Gupta, and Vincent Feuillard. Anomaly detection with Wasserstein GAN. arXiv e-prints, art. arXiv:1812.02466 Dee 2018.
Heiko Hoffmann. Kernl pea for novelty detection. Pattern recognition, 40(3):863-88)) 2007.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In /CML, volume 37, pp. 448-456, 2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015,
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Tomás Kuzin and Tomás Borovicka. Early failure detection for predictive maintenance of sensor parts. In ITAT, 2016.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URLhttp:
lecun. hwtpm/exxb//ooe/ wwxdb/mmnis////
D. Lee. Anomaly detection multivariate non-stationary time series for automatic dbms diagnosis. In ICMLA, 2017.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and Ian Goodfellow. Adversarial autoencoders. In International Conference on Learning Representations, 2016. URL
hts/1511.05644
MI. http:///www.kkggooooooooooooooor MNIST. http://aamm.eccun.ccoooddd////s.
NASA. http:///wwwaaaeecsttttttoooooooa
OTTO. http:///wwwa.lgggccccooooooooocc
Anuradha Vijay Pawar, Prakash N. Kalavadekar, and Ms. Swapnali N. Tambe. survey on outlier detection techniques for credit card fraud detection. IOSR Journal of Computer Engineering, 16: 44 48, 2014.
Pramuditha Perera and Vishal M. Patel. Learning deep features for one-class classification. CoRR, abs/1801 801.05365, 2018.

---

Published as a conference paper at ICLR 2020
Stanislav Pidhorskyi, Ranya Almohsen, and Gianfranco Doretto. Generative probabilistic novelty detection with adversarial autoencoders. In NeurIPS, pp. 6823 6834, 2018.
Utkarsh Porwal and Smruthi Mukund. Credit card fraud detection in e-commerce: An outlier detec- tion approach. CoRR, abs/1811..21191 2018.
RARM. https://giitab.ccm/maaaaaaaa/// anomaly detector.
M. Ravanbakhsh, M. Nabi, E. Sangineto, L.. Marcenaro, Regazzoni, and Sebe. Abnormal event detection in videos using generative adversarial nets. In 2017 IEEE International Conference Image Processing (ICIP), 2017.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In ICMI,, 2018.
Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, and Ehsan Adeli. Adversarially learned one-class classifier for novelty detection. In CVPR, pp. 3379-3388, 2018.
Mayu Sakurada and Takehisa Yairi. Anomaly detection using autoencoders with nonlinear dimen- sionality reduction. In MLSDA, 2014.
Thomas Schlegl. Philipp Seeböck, Sebastian M. Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In IPMI, 2017.
SNSR. http://wwww....g.ccccmmmmmmmllee
STL.
Aleksei Vasilev, Vladimir Golkov, Mare Meissner, Ilona Lipp, Eleonora Sgarlata, Valentina Tomassini, Derek K. Jones, and Daniel Cremers. q-Space Novelty Detection with Variational Autoencoders. arXiv, Jun 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnis a novel image dataset for benchmark- ing machine learning algorithms. CoRR, abs/1708.00744 2017. URL http:://axxi...rettt
abs/1708 07747
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv, hess/505500055. 2015a.
Dan Xu, Elisa Ricci, Yan Yan, Jingkuan Song, and Nicu Sebe. Learning deep representations of appearance and motion for anomalous event detection. In BMVC, 2015b.
Houssam Zenati, Chuan Sheng Foo, Bruno Lecouat, Gaurav Manek, and Vijay Ramaseshan Chan- drasekhar. Efficient GAN-based anomaly detection. arXiv, abs/1802.00622, 2018
Pushe Zhao, Masaru Kurihara, Junichi Tanaka, Tojiro Noda, Shigeyoshi Chikuma, and Tadashi
Suzuki. Advanced correlation-based anomaly detection method for predictive maintenance. In ICPHM, 2017.
10

---

Published as a conference paper at ICLR 2020
A SVD COMPUTATION TIME
We compare running times of training an autoencoder and computing SVD for NAP. We choose Iwo packages for the SVD computation: Pytorch SVD and fbpca provided in https:// fbpca. readthedocs wttes/ioooenn//aaaaaeeeeee
Since the time complexity of SVD is linear in the number of data samples mainly focus the performance of SVD with varying the number of columns of the input matrix that SVD is applied. To obtain variable sizes of the columns, we vary the depth and bottleneck size of autoencoders.
The result is shown below. Notably, Pytorch SVD utilizing GPU is at least 47x faster than training neural networks. Even, fbpca running only on CPU achieves least 2.4x speedup.
MNIST
OTTO
SNSR
- Terch SVD
Torch SVD
Running Time (S)
10a
fbpcca
fopca
Training
Training
10a
100
Tarch Тbpсa Training
10
10'
10a
10'
Number Columns Input
Number
Input
Number Input
The detailed setups to obtain the matrices for the experiment are given in the table below:
MNIST
OTTO
SNSR
Depth
Bottleneck Size
Depth
Bottleneck Size
Depth
Bottleneck Size
1
20
100
20
40
20
90
2
100
18
40
18
90
3
16
100
16
40
16
90
4
14
100
14
40
14
90
5
12
100
12
40
12
90
6
10
100
10
40
10
90
7
8
100
8
40
90
8
6
100
6
40
6
90
9
100
4
40
90
10
2
100
2
40
2
90
80
2
30
70
12
2
60
2
20
2
50
13
40
2
30
14
2
20
2
10
TThis common in practice where the number of data samples is larger than the numher of features
11

---

Published as a conference paper at ICLR 2020
B STANDARD DEVIATIONS OF EXPERIMENTAL RESULTS
We provide the standard deviations of the result in Table 2 Given a dataset and a setup, multimodal or unimodal, AUROC values are first averaged over multiple cases with different novelty class as- signments; then, the standard deviations are calculated for those averaged values over multiple trials. The number of cases and the number of trials depend on datasets, and we refer Section for more details.
AF
VAE
AAE
Dataset
Recon
SAP
NAP
Recon
SAP
NAP
Recon
SAP
NAP
Multimodal Normality
STL
0.723
0.703
0.711
0,700
0.675
0.711
0.726
0.711
0.724
(0.005)
(0.007)
(0.006)
(0.012)
(0.018)
(0.012)
(0.009)
(0.011)
(0.016)
OTTO
0.617
0,616
0.665
0,630
0.631
0.649
0.617
0.618
0.665
(0.002)
(0.001)
(0.004)
(0.003)
(0.005)
(0.003)
(0.001)
(0.003)
(0.004)
SNSR
0.613
0.611
0.614
0.608
0.620
0.658
0.612
0.608
0.609
(0.002)
(0.003)
(0.004)
(0.001)
(0.002)
(0.003)
(0.005)
(0.005)
(0.005)
MNIST
0.825
0.881
0.899
0,900
0.965
0,965
0.847
0.911
0,929
(0.001)
(0.007)
(0.008)
(0.004)
(0.005)
(0.003)
(0.005)
(0.006)
(0.005)
F-MNIST
0.712
0.725
0.734
0.725
0.728
0.755
0.721
0.710
0.727
(0.001)
(0.004)
(0.004)
(0.003)
(0.002)
(0.006)
(0.001)
(0.005)
(0.005)
Unimodal Normality
MI-F
0.607
0.670
0.705
0.591
0.572
0.678
0.632
0.692
0.704
(0.023)
(0.041)
(0.018)
(0.013)
(0.019)
(0.047)
(0.028)
(0.042)
(0.024)
MI-V
0.897
0.898
0.907
0.845
0.833
0.903
0.895
0.891
0.904
(0.010)
(0.005)
(0.005)
(0.021)
(0.032)
(0.010)
(0.005)
(0.006)
(0.005)
EOPT
0.610
0,607
0.625
0.675
0.634
0.596
0,606
0.603
0.634
(0.015)
(0.014)
(0.008)
(0.017)
(0.007)
(0.002)
(0.011)
(0.013)
(0.010)
NASA
0.719
0.702
0.692
0.738
0.714
0.719
0.712
0.695
0.688
(0.009)
(0.015)
(0.016)
(0.009)
(0.015)
(0.009)
(0.016)
(0.027)
(0.024)
RARM
0.687
0.674
0.686
0.587
0.565
0.648
0.664
0.675
0.675
(0.038)
(0.031)
(0.030)
(0.021)
(0.029)
(0.022)
(0.015)
(0.023)
(0.040)
STL
0.870
0.856
0,830
0.850
0.824
0.792
0.875
0.861
0.830
(0.007)
(0.007)
(0.007)
(0.006)
(0.003)
(0.034)
(0.003)
(0.004)
(0.008)
OTTO
0.829
0.829
0.832
0.831
0.835
0.833
0.828
0.828
0.831
(0.001)
(0.001)
(0.001)
(0.002)
(0.003)
(0.004)
(0.002)
(0.001)
(0.002)
SNSR
0.979
0.982
0.990
0.975
0.979
0.964
0.979
0.983
0.997
(0.001)
(0.001)
(0.001)
(0.003)
(0.002)
(0.043)
(0.001)
(0.002)
(0.001)
MNIST
0.972
0.980
0.979
0.980
0.987
0.989
0.972
0.966
0.977
(0.001)
(0.001)
(0.000)
(0.002)
(0.002)
(0.001)
(0.001)
(0.004)
(0.001)
F-MNIST
0.924
0.928
0,933
0,926
0,926
0.942
0.922
0.905
0.928
(0.002)
(0.002)
(0.002)
(0.001)
(0.001)
(0.000)
(0.001)
(0.002)
(0.002)
12

---

Published as a conference paper at ICLR 2020
C COMPARING RAPP TO BASELINES OTHER THAN NEURAL NETWORKS
The table below shows the result of comparing RAPP and baselines not based on neural networks. Note that the baselines are run with hyperparameters or hyperparameter search spaces provided in Ruff et al. 2018), while NAP is not further tuned.
Dataset
OCSVM
ISOF
PCA
kPCA
NAPAE
NAPvAE
NAPAAE
Multimodal Normality
STL
0.608
0.635
0.700
0.693
0.711
0.711
0.724
OTTO
0.628
0.548
0.572
0.644
0.665
0.649
0.665
SNSR
0.524
0.526
0.536
0.580
0.614
0.658
0.606
MNIST
0.566
0.562
0.698
0.701
0.899
0.965
0.929
F-MNIST
0.571
0.650
0.670
0.676
0.734
0.755
0.727
Unimodal Normality
MI-F
0.777
0.826
0.546
0.838
0.705
0.678
0.704
MI-V
0.840
0,839
0.874
0.888
0.907
0.903
0.904
EOPT
0.597
0.595
0.552
0.693
0.625
0.596
0.634
NASA
0.573
0.664
0.765
0.699
0.692
0.719
0.688
RARM
0.749
0.752
0.587
0.734
0.734
0.648
0.675
STL
0.879
0.871
0.883
0.880
0.880
0.792
0.830
OTTO
0.821
0.699
0.799
0.823
0.823
0.833
0.831
SNSR
0.959
0.893
0.894
0.970
0.970
0.964
0.991
MNIST
0.894
0.919
0.903
0.906
0.979
0.989
0.977
F-MNIST
0.884
0.854
0.950
0.943
0.933
0.942
0.928
13

---

Published as a conference paper at ICLR 2020
D NAP PERFORMANCE OVER INCREASING INVOLVED HIDDEN LAYERS
We investigate the performance of NAP while increasing the number of hidden layers involved the NAP computation. Specifically, we consider two ways for the increment: 1) adding hidden layers one by one from the imput layer (forward addition), and 2) adding hidden layers one by one from the bottleneck layer (backward addition). Experimental results on three datasets are shown below. For most cases, more hidden layers tend to result in higher performance. The values are obtained from one trial without averaging results from multiple trials.
(Forward)
NAP (Forward)
SAP (Backward)
(Backward)
Rccon
VAE
AAE
0.95
0.90
AUROC
0.75
0.70
10 # Hidden Layers Used for NAP
of Hidden Layers Used for NAP
# of Hidden Layers Used for NAP
(a) MNIST
NAP (Forward)
SAP (Forward)
NAP (Backward)
- SAP (Backward)
Recon
AE
VAE
0.65
AUROC
0.60
0.55
10 of Hidden Layers Used for NAP
10
10
of Hidden Layers Used for NAP
of Hidden Layers Used for NAP
(b) OTTO
NAP (Forward)
SAP (Forward)
(Backward)
SAP (Backward)
Recon
VAE
AAE
0.70
AUROC
0.65
G
0.60
10
10
10
of Hidden Layers Used for NAP
# of Hidden Layers Used for NAP
# of Hidden Layers Used for NAP
(c) STEEL
14


Reviews:
[{"review_id": "S1lmbNX0cB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novelty detection method by utilizing latent variables in auto-encoder. Based on this, this paper proposes two metrics to quantifying the novelty of the input. Their main contribution is the NAP metric based on SVD. Their method is empirically demonstrated on several benchmark datasets, and they compare their proposed metrics with other competing methods using AUROC and experiments results are encouraging.  \n\nThe metrics proposed in this paper are intuitive and interesting. The experiments shown in Table2 is very convincing, and it could be better to extend Table3 to include other datasets (STL,OTTO, etc. )\n\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HylFcgFRFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I have read the reviews and the comments.\n\nI appreciate the effort of the authors. I feel positive about the paper and I think it should be accepted.\n\nI confirm my rating.\n\n=================\nThe paper proposes a new method for novelty detection that is based on measuring the reconstruction error in latent space between layer of the encoder. \nThe reconstructed sample is fed back to the encoder and activations of the hidden layers of the encoder are compared with the activations that occurred when the original sample was fed into it.\n\nTo aggregate the reconstruction error from all layers of the encoder, two methods are proposed SAP (simple aggregation along pathway) and NAP (normalized aggregation along pathway). SAP is simply the sum of the squared norm, while NAP performs decorrelation and normalization of the magnitude.\n\nThe idea is novel, well motivated and explained.\n\nIt is said in the paper that NAP performs distance normalization by doing orthogonalization and scaling. The way it is described seems to be equivalent to PCA whitening. Thus, the computed distance should be a Mahalanobis distance.\n\nIt is not clear why for the VAE case 10 samples are averaged, instead of just using the mean component given by the encoder and passing it to decoder. It is typical to use reparametrization only during training.\n\nComparison with other state of the art methods is somewhat weak, since only two similar datasets are used (MNIST and F-MNIST).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1x9jGHjFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "UPDATE: \nI acknowledge that I‘ve read the author responses as well as the other reviews. \n\nI appreciate the improvements and clarifications the authors have made, especially adding an ablation study to see the benefits of adding additional layers. I updated my score to Weak Accept (6). \n\n####################\n\nThis paper considers deep autoencoders (AEs) for the unsupervised novelty/anomaly detection task and proposes to extend the standard AE anomaly score, given by the reconstruction error between the input and output in the original data space, to also utilize the reconstruction errors of the hidden activations in the AE network. The proposed method, Reconstruction along Projection Pathway (RaPP), specifically compares the hidden activations of all encoder layers given by the original input $x$ with the activations of the same units given by feeding the reconstruction $\\hat{x}$ back into the AE. Thus RaPP compares the activation statistics of the original input $x$ and its reconstruction $\\hat{x}$ along the encoder projection pathway from original data space to latent code space. Two ways for aggregating those reconstruction errors to a final anomaly score are presented: (1) Simple Aggregation Along Pathway (SAP) which simply computes the sum of reconstruction errors, and (2) Normalized Aggregation Along Pathway (NAP) which computes the sum of reconstruction errors after normalization via Singular Value Decomposition (SVD). The paper conclusively presents experiments on eight datasets from various domains, in which SAP and NAP are compared to the reconstruction error baseline for vanilla AE, VAE, and AAE, as well as experiments on MNIST and Fashion-MNIST in which NAP is compared to state-of-the-art deep anomaly detectors.\n\nThough this work is well presented and indicates promising results, I think the paper should not yet be accepted due to the following main reasons: \n(i) The experimental evaluation indicates promising, but not yet convincing results; \n(ii) The computational complexity of NAP seems to be a major limitation of RaPP which is not addressed in the text; \n(iii) The added value/insights from the theoretical Section 4 (Motivation of RaPP) are not clear.\n\n(i) I think the experimental section shows promising, but not yet convincing results. To judge the significance of results, I think the paper should address the following:\n(ia) The experiments on the eight non-image datasets should include other baselines (e.g. OC-SVM, Isolation Forest) besides the standard AE reconstruction error. One should expect SAP and NAP to improve over the standard AE since both methods include the original data space reconstruction errors as well. Moreover, the advantage of deep approaches on such non-image datasets is less clear [7] why a comparison to well-known baselines should be given.\n(ib) The main motivation for deep approaches to anomaly detection are large and complex datasets [6, 5, 4, 2]. I think the comparison to recent, state-of-the-art deep competitors should at least include another dataset more complex than MNIST or Fashion-MNIST, e.g. CIFAR-10 as reported in the previous works or MVTec [1]. \n(ic) I think the proposed method begs for an ablation study of subsequently adding the reconstruction errors of additional layers. This would clearly demonstrate the potential benefits of adding the hidden reconstructions.\n\n(ii) The experiments indicate that a proper normalization of the hidden activation reconstruction errors is crucial for improving detection performance. NAP shows consistent improvements, whereas SAP often performs similar to the AE baseline. However, the current SVD normalization procedure on a matrix with dimensions number of samples × number of hidden encoder units seems extremely costly to me and appears to be a major limitation towards larger datasets or networks. Could you comment on this since this is not yet addressed in the manuscript. Have you tried using Batch Normalization (after activation) together with per-layer averaging? To me, this seems the natural first choice to normalize unit scores and to account for different layer widths. Do you apply SVD on mini-batches?\n\n(iii) The additional insights from the theoretical Section 4 are not clear to me. I think the presented reconstruction property for the hidden layers follows somewhat directly per definition for symmetrically constructed deep autoencoders (specifically if weights would be shared in addition). For a theoretical contribution, on the other hand, the proof and proposition should be fully rigorous in my mind, i.e. stating all the necessary assumptions on the function class (e.g. you implicitly assume invertibility and thus some smoothness of the $g_i$'s which Conv+ReLU modules do not satisfy for instance). As of now, I think this section does not add to intuition, but on the other hand is not completely rigorous. Maybe I am missing something?\n\nThe overall presentation of the paper is good (clear writing and structure, polished Figures and Tables). The work is well motivated and properly placed in the literature. Maybe since the approach is rather simple (which I don’t find negative), the author felt the need to add some rigor to the paper, which I think would not be necessary for a significant contribution if the experimental results hold up against the additional baselines and more complex datasets as described in (i).\n\n\n####################\n*Additional Feedback*\n\n*Positive Highlights*\n1. Simple idea that does not require autoencoder modification or retraining that indicates improved anomaly detection results.\n2. The work is well placed in the literature. The related work includes all relevant and recent major works on the subject matter.\n3. I appreciate the evaluation on both anomaly/novelty detection setups, unimodal and multimodal.\n4. Comparison to recent OC-NN [3], GPND [5], Deep SVDD [6], and GT [4].\n5. The writing, structure and overall presentation is good.\n\n*Ideas for Improvement*\n6. Include additional baselines and more complex datasets as described in (i).\n7. Address the computational complexity of RaPP as in described in (ii).\n8. Maybe cut the methodical/theoretical parts in Section 3.2 and Section 4 a bit. I think they are rather straightforward. Maybe combine Figures 1+2 as well. Extend the experimental evaluation instead.\n9. Report the AUROC standard deviations over the trials as well to better infer statistical significance of the results (defer to appendix if space is a constraint).\n\n*Minor comments*\n10. Section 2: “Unsupervised and semi-supervised learnings” » “Unsupervised and semi-supervised learning approaches”.\n11. Section 2: “Variational Autoencoders (VAE) was reported ...” » “Variational Autoencoders (VAE) were reported ...”\n12. Section 3.1: “Due to this representation learning property, the autoencoder has been widely used for novelty detection.” » emphasis on unsupervised learning property, specifically.\n13. Section 3.1: “Although this approach has shown a promising result in novelty detection ...” » “Although this approach has shown promising results in novelty detection ...”\n14. Section 3.1, last sentence: “... in more details.” » “... in more detail.”\n15. Section 3.2: “Those are especially suited for the case of zero-knowledge to interpret identified hidden spaces, which commonly happens when modeling with deep neural networks.” Zero-knowledge case? Reference?\n16. In Section 5.1: “Further setups are described in Section 5.1”?\n17. Section 5.4.1: “Also, we showed the best score ...” » “Also, we show the best score ...”.\n18. Section 5.2: “... maintaining novelty ratios to 35% for the multimodal and 50% for the unimodal normality setups, respectively.” Why use different ratios?\n\n\n####################\n*References*\n[1] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9592–9600, 2019.\n[2] R. Chalapathy and S. Chawla. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407, 2019.\n[3] R. Chalapathy, A. K. Menon, and S. Chawla. Anomaly detection using one-class neural networks. arXiv preprint arXiv:1802.06360, 2018.\n[4] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In NIPS, 2018.\n[5] S. Pidhorskyi, R. Almohsen, and G. Doretto. Generative probabilistic novelty detection with adversarial autoencoders. In NeurIPS, pages 6822–6833, 2018.\n[6] L. Ruff, R. A. Vandermeulen, N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In International Conference on Machine Learning, pages 4393–4402, 2018.\n[7] L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694, 2019.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to extend the autoencoder loss in a deep generative model to include per-latent-layer loss terms.  Two variants are proposed: SAP (simple aggregation along pathway) and NAP (normalized aggregation along pathway). SAP is simply the sum of the squared norm, while NAP performs decorrelation and normalization of the magnitude.  This was viewed as novel by the reviewers, and the experiments supported the proposed approach.

In the post rebuttal phase, the inclusion of an ablation study has led to an upgrade in the reviewer recommendation.  As a result, there was a unanimous opinion that the paper is suitable for publication at ICLR.
Decision: Accept (Poster)