Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a novelty detection method by utilizing latent variables in auto-encoder. Based on this, this paper proposes two metrics to quantifying the novelty of the input. Their main contribution is the NAP metric based on SVD. Their method is empirically demonstrated on several benchmark datasets, and they compare their proposed metrics with other competing methods using AUROC and experiments results are encouraging.  

The metrics proposed in this paper are intuitive and interesting. The experiments shown in Table2 is very convincing, and it could be better to extend Table3 to include other datasets (STL,OTTO, etc. )



review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: I have read the reviews and the comments.

I appreciate the effort of the authors. I feel positive about the paper and I think it should be accepted.

I confirm my rating.

=================
The paper proposes a new method for novelty detection that is based on measuring the reconstruction error in latent space between layer of the encoder. 
The reconstructed sample is fed back to the encoder and activations of the hidden layers of the encoder are compared with the activations that occurred when the original sample was fed into it.

To aggregate the reconstruction error from all layers of the encoder, two methods are proposed SAP (simple aggregation along pathway) and NAP (normalized aggregation along pathway). SAP is simply the sum of the squared norm, while NAP performs decorrelation and normalization of the magnitude.

The idea is novel, well motivated and explained.

It is said in the paper that NAP performs distance normalization by doing orthogonalization and scaling. The way it is described seems to be equivalent to PCA whitening. Thus, the computed distance should be a Mahalanobis distance.

It is not clear why for the VAE case 10 samples are averaged, instead of just using the mean component given by the encoder and passing it to decoder. It is typical to use reparametrization only during training.

Comparison with other state of the art methods is somewhat weak, since only two similar datasets are used (MNIST and F-MNIST).
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: UPDATE: 
I acknowledge that I‘ve read the author responses as well as the other reviews. 

I appreciate the improvements and clarifications the authors have made, especially adding an ablation study to see the benefits of adding additional layers. I updated my score to Weak Accept (6). 

####################

This paper considers deep autoencoders (AEs) for the unsupervised novelty/anomaly detection task and proposes to extend the standard AE anomaly score, given by the reconstruction error between the input and output in the original data space, to also utilize the reconstruction errors of the hidden activations in the AE network. The proposed method, Reconstruction along Projection Pathway (RaPP), specifically compares the hidden activations of all encoder layers given by the original input $x$ with the activations of the same units given by feeding the reconstruction $\hat{x}$ back into the AE. Thus RaPP compares the activation statistics of the original input $x$ and its reconstruction $\hat{x}$ along the encoder projection pathway from original data space to latent code space. Two ways for aggregating those reconstruction errors to a final anomaly score are presented: (1) Simple Aggregation Along Pathway (SAP) which simply computes the sum of reconstruction errors, and (2) Normalized Aggregation Along Pathway (NAP) which computes the sum of reconstruction errors after normalization via Singular Value Decomposition (SVD). The paper conclusively presents experiments on eight datasets from various domains, in which SAP and NAP are compared to the reconstruction error baseline for vanilla AE, VAE, and AAE, as well as experiments on MNIST and Fashion-MNIST in which NAP is compared to state-of-the-art deep anomaly detectors.

Though this work is well presented and indicates promising results, I think the paper should not yet be accepted due to the following main reasons: 
(i) The experimental evaluation indicates promising, but not yet convincing results; 
(ii) The computational complexity of NAP seems to be a major limitation of RaPP which is not addressed in the text; 
(iii) The added value/insights from the theoretical Section 4 (Motivation of RaPP) are not clear.

(i) I think the experimental section shows promising, but not yet convincing results. To judge the significance of results, I think the paper should address the following:
(ia) The experiments on the eight non-image datasets should include other baselines (e.g. OC-SVM, Isolation Forest) besides the standard AE reconstruction error. One should expect SAP and NAP to improve over the standard AE since both methods include the original data space reconstruction errors as well. Moreover, the advantage of deep approaches on such non-image datasets is less clear [7] why a comparison to well-known baselines should be given.
(ib) The main motivation for deep approaches to anomaly detection are large and complex datasets [6, 5, 4, 2]. I think the comparison to recent, state-of-the-art deep competitors should at least include another dataset more complex than MNIST or Fashion-MNIST, e.g. CIFAR-10 as reported in the previous works or MVTec [1]. 
(ic) I think the proposed method begs for an ablation study of subsequently adding the reconstruction errors of additional layers. This would clearly demonstrate the potential benefits of adding the hidden reconstructions.

(ii) The experiments indicate that a proper normalization of the hidden activation reconstruction errors is crucial for improving detection performance. NAP shows consistent improvements, whereas SAP often performs similar to the AE baseline. However, the current SVD normalization procedure on a matrix with dimensions number of samples × number of hidden encoder units seems extremely costly to me and appears to be a major limitation towards larger datasets or networks. Could you comment on this since this is not yet addressed in the manuscript. Have you tried using Batch Normalization (after activation) together with per-layer averaging? To me, this seems the natural first choice to normalize unit scores and to account for different layer widths. Do you apply SVD on mini-batches?

(iii) The additional insights from the theoretical Section 4 are not clear to me. I think the presented reconstruction property for the hidden layers follows somewhat directly per definition for symmetrically constructed deep autoencoders (specifically if weights would be shared in addition). For a theoretical contribution, on the other hand, the proof and proposition should be fully rigorous in my mind, i.e. stating all the necessary assumptions on the function class (e.g. you implicitly assume invertibility and thus some smoothness of the $g_i$'s which Conv+ReLU modules do not satisfy for instance). As of now, I think this section does not add to intuition, but on the other hand is not completely rigorous. Maybe I am missing something?

The overall presentation of the paper is good (clear writing and structure, polished Figures and Tables). The work is well motivated and properly placed in the literature. Maybe since the approach is rather simple (which I don’t find negative), the author felt the need to add some rigor to the paper, which I think would not be necessary for a significant contribution if the experimental results hold up against the additional baselines and more complex datasets as described in (i).


####################
*Additional Feedback*

*Positive Highlights*
1. Simple idea that does not require autoencoder modification or retraining that indicates improved anomaly detection results.
2. The work is well placed in the literature. The related work includes all relevant and recent major works on the subject matter.
3. I appreciate the evaluation on both anomaly/novelty detection setups, unimodal and multimodal.
4. Comparison to recent OC-NN [3], GPND [5], Deep SVDD [6], and GT [4].
5. The writing, structure and overall presentation is good.

*Ideas for Improvement*
6. Include additional baselines and more complex datasets as described in (i).
7. Address the computational complexity of RaPP as in described in (ii).
8. Maybe cut the methodical/theoretical parts in Section 3.2 and Section 4 a bit. I think they are rather straightforward. Maybe combine Figures 1+2 as well. Extend the experimental evaluation instead.
9. Report the AUROC standard deviations over the trials as well to better infer statistical significance of the results (defer to appendix if space is a constraint).

*Minor comments*
10. Section 2: “Unsupervised and semi-supervised learnings” » “Unsupervised and semi-supervised learning approaches”.
11. Section 2: “Variational Autoencoders (VAE) was reported ...” » “Variational Autoencoders (VAE) were reported ...”
12. Section 3.1: “Due to this representation learning property, the autoencoder has been widely used for novelty detection.” » emphasis on unsupervised learning property, specifically.
13. Section 3.1: “Although this approach has shown a promising result in novelty detection ...” » “Although this approach has shown promising results in novelty detection ...”
14. Section 3.1, last sentence: “... in more details.” » “... in more detail.”
15. Section 3.2: “Those are especially suited for the case of zero-knowledge to interpret identified hidden spaces, which commonly happens when modeling with deep neural networks.” Zero-knowledge case? Reference?
16. In Section 5.1: “Further setups are described in Section 5.1”?
17. Section 5.4.1: “Also, we showed the best score ...” » “Also, we show the best score ...”.
18. Section 5.2: “... maintaining novelty ratios to 35% for the multimodal and 50% for the unimodal normality setups, respectively.” Why use different ratios?


####################
*References*
[1] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9592–9600, 2019.
[2] R. Chalapathy and S. Chawla. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407, 2019.
[3] R. Chalapathy, A. K. Menon, and S. Chawla. Anomaly detection using one-class neural networks. arXiv preprint arXiv:1802.06360, 2018.
[4] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In NIPS, 2018.
[5] S. Pidhorskyi, R. Almohsen, and G. Doretto. Generative probabilistic novelty detection with adversarial autoencoders. In NeurIPS, pages 6822–6833, 2018.
[6] L. Ruff, R. A. Vandermeulen, N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In International Conference on Machine Learning, pages 4393–4402, 2018.
[7] L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694, 2019.
review_title: Official Blind Review #3

Meta-review: The paper proposes to extend the autoencoder loss in a deep generative model to include per-latent-layer loss terms.  Two variants are proposed: SAP (simple aggregation along pathway) and NAP (normalized aggregation along pathway). SAP is simply the sum of the squared norm, while NAP performs decorrelation and normalization of the magnitude.  This was viewed as novel by the reviewers, and the experiments supported the proposed approach.

In the post rebuttal phase, the inclusion of an ablation study has led to an upgrade in the reviewer recommendation.  As a result, there was a unanimous opinion that the paper is suitable for publication at ICLR.
Decision: Accept (Poster)