Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes an improved extension of the Wasserstein auto-encoder for anomaly detection. The novelty is in proposing a weighted reconstruction error that penalizes the mapping of data with high reconstruction errors (mostly anomalies) into high probability regions. The idea being that an outlier would have a higher reconstruction error, and hence should be mapped to low-probability region of the latent distribution. 

Experimental Results:  As a distribution mapping auto-encoder model, OP-DMA outperforms the deep learning based state-of-the-art models in the same domain. 

Overall Assessment: The authors have a nice idea of forcing the latent mappings of inputs to correlate with their reconstruction error. Overall, the method is promising, but I have the following concerns:

* Using the reconstruction error as an anomaly score has been explored many years ago (check replicator neural networks), the novelty here is to enforce that on the latent space in the context of a variational auto-encoder. I am not sure if, from anomaly detection perspective, this is any better than simply using the reconstruction score. Why go the VAE route at all? 
* Is there a possibility that assuming a single multi-variate Gaussian, as a prior, too restrictive? Could it result in a high false alarm rate as well? I guess this could be answered by more experimental results on richer data sets (even synthetic is fine). 
 
* In most score based algorithms, the anomaly score is computed without assuming any prior knowledge about the contamination proportion. However, in the case of OP-DMA, the contamination parameter is used to train the auto-encoder that scores the data. This might result in an optimization that is very specific to the parameter setting. I strongly recommend a sensitivity analysis to study the robustness of the model against different values of contamination parameter.

* Performance on synthetic data-set has not been presented. The set H in theorem 3 has not been defined. 

* Additional comparison with other non-distribution mapping state-of-the-art models such as LOF, oc-SVM, KNN would give a clearer idea of the performance. This is important, because in my past experience, non-deep learning methods give much better results on the benchmark data sets that the authors have evaluated their method on. In fact, a comparative analysis (See - https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152173) gives a very nice comparison. However, since the authors provide results using Avg F1-score, instead of AUC curve, it was not possible to compare them myself.

* In figure 3, in the training process, the authors have describe to add the divergence between the latent and prior distribution to the loss function, however, nothing like this is clearly shown in the figure.The references of figures in the text are either out of place or incorrect. Figure 1(a) and (b) in reference to the text are incorrect. Figure 2 is the misleading figure as it doesn't illustrate the anomaly detection process. Figure 3 has not been mentioned anywhere in the text. The authors have mentioned the comparison of their method with Wasserstien and variational auto-encoders in the text, while in table 2 and 4, AAE has also been shown as one of the method for comparison, which is never mentioned or described in the text.

* Typo in caption of Figure 1 and the first line of section 3.3

Overall, I am hesitant to recommend the paper before cross-checking the issue with contamination proportion and learning more about how a VAE framework is indeed important for anomaly detection.

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This work proposes an outlier detection method based on WAE framework. WAE is trained to ensure that 1) latent distribution follows a prior distribution 2) weighted reconstruction error is low where prior PDF is used to weight the reconstruction error.

Positives
------------
1.I liked the intuition behind the proposed method and I felt its worth exploring. Paper points out that in previous works, there is no mechanism to prevent outliers from getting mapped to high probability areas in the model [21]. Authors claim that their method will over-come this issue. I believe this is the core contribution of the paper.

2.I agree with authors point that WAE is a better choice than VAE for outlier detection because, former "encourages the latent representations as a whole to match the prior ". 

3.I agree training with a distributional divergence loss along with a weighed reconstruction loss will be helpful for learning a robust representation (as outliers in the training dataset would be assigned a lower weight). 

4.Authors have compared the performance of their method on OOD dataset where they have  compared against three other baseline methods, where they obtain better performance in majority of cases.


Negatives
--------------
A. Outlier detection and anomaly/novelty detection are two very different problems.  Outliers are 'bad eggs' coming from the same class as normal data. On the other hand, anomaly/novelty are unexpected data possibly coming from other classes. This is the taxonomy followed by majority of works. In my understanding this work is about 'outlier detection'. I hope authors will use the term 'outlier detection' consistency through the paper.

B. Authors have not done a good survey on existing outlier detection methods.  Eg:
    I. Chong You, Rene Vidal, Provable Self-Representation Based Outlier Detection in a Union of Subspaces, CVPR 17
    II. Yan Xia, Xudong Cao, Fang Wen, Gang Hua, Jian Sun, Learning Discriminative Reconstructions for Unsupervised Outlier Removal, ICCV 15
   III. Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, Ehsan Adeli, Adversarially Learned One-Class Classifier for Novelty Detection, CVPR 18. (they have experiments on outlier detection)

C. Although existing methods have not explicitly stated the problem identified in (1) above, their proposals are indirectly solving this problem. Therefore, authors should have compared with papers listed in (B) to demonstrate the effectiveness of their method for a meaningful comparison.

D. This is the first time I'm seeing the OOD dataset used in the paper. Have other works published their results on this dataset? Can they be included in your paper?. If not, consider reporting results on standard datasets used in papers (B). I believe reporting results on at least two datasets is necessary to demonstrate the generalizability of the method.

Other comments
------------------------
a. What is the dimensionality used in the latent space? I believe a larger latent space may be required in modeling more complex data such as images. Is the weighting mechanism effective when a very large latent space is used due to the curse of dimensionality. 

b. I don't think the synthetic dataset experiment is giving any interesting insights. This space is better used if an additional dataset is used instead.

In conclusion, I like the idea presented in this paper; however, I believe experimental results needs to be improved significantly to demonstrate the effectiveness of the proposed method. I cannot recommend to accept the paper in its present condition. 


Post Rebuttal:
Authors have partially addressed my concerns. In light of new experiments provided, I'm changing my decision to weak accept.
review_title: Official Blind Review #5

final_score: rating: 6: Weak Accept
review_content: This paper proposes a novel outlier detection approach, based on Wasserstein auto encoders. 

Unfortunately, I cannot comment on the overall scientific contribution of the paper, as I simply do not possess the expertise to judge it accurately. I will rely on the judgement of the other reviewers, whom I hope will have more experience and will better know the literature. I will report on a few issues with aspects related to the presentation below.

In fig. 1, the (a) and (b) should probaby appear below each diagram. "on trained" is repeated twice in the caption. 

In fig. 2, the WAE acronym is defined only much later in the text. 

Fig. 2 is also a bit confusing, since it seems like the order in which the diagrams appear should be swapped. Indeed, the text also refers to fig. 2 (b) before (a). The text just below fig. 2 also refers to Figure 1, but I think it should be 2? 

In sec. 4.2, the text mentions table 2 when it should really be table 1. Also, table 1 should appear before table 2 in the body. 

It looks as if the symbols () and [] are inverted? All references are referred to with () and text within parentheses (e.g. references to figures) have []. 

review_title: Official Blind Review #3

Meta-review: This paper proposes an outlier detection method that maps outliers to low probability regions of the latent space. The novelty is in proposing a weighted reconstruction error penalizing the mapping of outliers into high probability regions. The reviewers find the idea promising.
They have also raised several questions. It seems the questions are at least partially addressed in the rebuttal, and as a result one of our expert reviewers (R5) has increased their score from WR to WA. But since we did not have a champion for this paper and its overall score is not high enough, I can only recommend a reject at this stage.
Decision: Reject