Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents a novel deep anomaly detection model. It combines two existing models: B-VAE and t-SNE. The B-VAE is trained unsupervised and learns an encoder and decoder which provide both an embedding and a reconstruction. Using t-SNE to reduce its dimensionality, the embedding is projected into a 2 dimensional space. An anomaly score function is defined that combines the reconstruction error and the distance in t-SNE space to the K nearest neighbor(s). Experiments are conducted with several image datasets (MNIST,FMNIST,CIFAR10,SmallNORB) and one timeseries dataset (Arrhythmia). For the image sets, the B-VAE model is implemented with a CNN, while for timeseries, a TCN is used. Comparisons are conducted showing the approach to be beat other SOT unsupervised methods, AnoGAN and ADGAN, by 63% and 22% respectively for MNIST and 8% and 2% for FMNIST (in terms of error reduction). For CIFAR-10 and FMNIST it is even demonstrated to beat a supervised SOT method CapsNET. Another experiment shows that t_SNE dramatically improves the performance over B-VAE alone. For the timeseries, the approach is not compared to other SOT approaches as the authors only provide an experiment showing that TCN beats CNN and LSTM for the implementation of the B-VAE. In addition the authors study the effect of the various parameters of the system, in particular the effect of the B in B-VAE and of alpha, the mixing factor between reconstruction error and kNN distance in t_SNE. 3D plots give a good idea on how to select optimal values for the various datasets. The impact of B is also shown on the t-SNE map for MNIST. Finally an ablation studies compares on MNIST the performance of the approach with t-SNE alone, reconstruction alone, and latent distance. On average over 4 digits taken as anomaly, the proposed approach dramatically outperforms the others.

PROS:

* The proposed approach improves over the SOT of competitive recent methods for anomaly detection on four image datasets.
* The authors make an effort to abstract the approach into a framework where other deep learning models and dimensionality reduction techniques can be used. They illustrate this by using a TCN instead of a CNN for the timeseries example.
* The parameter studies and ablation studies are informative and answer many of the questions i had as i read the paper.
* The paper is relatively clearly written (at least sufficiently to easily understand the technical details).


CONS:

* The novelty of the paper is limited as it is mostly a combination of 2 existing methods.
* The timeseries dataset is not compared to SOT methods (although the authors claim SOT in the conclusion).
* A pseudo-code algorithm is not provided, making it unlikely someone can reproduce the method.
* There are many typos an grammatical errors
* The paper could have been shortened. 10 pages is too long.

Overall, because of the good performance and thoughtful ablation studies, and despite the limited novelty, I think the paper makes a good contribution to anomaly detection.


review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes to combine beta-VAE and t-SNE for anomaly detection.
Although the problem and the proposed approach is relevant, I have the following concerns.

- The problem setting is not well explained.
    In particular, it is not clear whether the setting is unsupervised or not.
    It seems that the proposed method is for unsupervised anomaly detection.
    However, the authors mention that beta-VAE is trained on normal data, which means that it is not unsupervised.
    Please clarify this point.
- The originality and the technical quality of the proposed method is not high as it is a straightforward combination of two existing method.
    If the proposed combination has some theoretical advantage for anomaly detection, the paper becomes more interesting.
    However, there is no theoretical analysis of the proposed method, hence the significance of the contribution is hot high in its current state.
- Experimental results are not convincing.
    * The authors argue that the t-SNE step is important as the proposed method is better than the naive beta-VAE.
      However, Figure 2 shows that in most cases the score becomes better as \alpha gets larger and shows the best score if alpha = 1.
      From Equation (2), this means that the t-SNE step does not contribute to the anomaly detection performance.
      This inconsistency should be carefully discussed.
    * Also, Figure 2 shows that beta should be small and the best score is achieved when beta = 0 in most cases.
      This means that representation learning is not meaningful and the raw representation (feature vectors) may be already effective for anomaly detection.
      Hence the significance of the proposed method is not convincing.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In a paper a new way to compute anomality score (for a test point) is suggested. A paper is purely experimental, based on existing techniques to dimension reduction (beta-VAE and t-SNE). Given trained beta-VAE, latent vectors, obtained for training set, are feed into t-SNE algorithm. The overall anomality score for a test point is combined from 1-NN distances on t-SNE plot and reconstruction error of beta-VNE. 

There is substantive question naturally appears from the application of t-SNE to obtained latent vectors. By construction, beta-VAE tries to make latent vector to be distributed according to N(0,I). By definition, it is very hard to project such a distribution on a plane, even by non-linear methods, such as t-SNE. Yet at the second step of paper's approach, this vectors are feed into 2-dimensional t-SNE. 

This aspect makes me to think that optimal alpha in paper's anomality score should be close to 1. This would imply that t-SNE step is not needed at all. So, I am curious what actual optimal alpha was in experiments? 

Or, how would results change if t-SNE mapping would be set to Identity tranformation (into space whose dimension is the same as latent space), but formula for anomality remains the same?
review_title: Official Blind Review #2

Meta-review: The paper presents AnoDM (Anomaly detection based on unsupervised Disentangled representation learning and Manifold learning) that combine beta-VAE and t-SNE for anomaly detection. Experiment results on both image and time series data are shown to demonstrate the effectiveness of the proposed solution. 

The paper aims to attack a challenging problem. The proposed solution is reasonable. The authors did a job at addressing some of the concerns raised in the reviews. However, two major concerns remain: (1) the novelty in the proposed model (a combination of two existing models) is not clear; (2) the experiment results are not fully convincing. While theoretical analysis is not a must for all models, it would be useful to conduct thorough experiments to fully understand how the model works, which is missing in the current version. 

Given the two reasons above, the paper did not attract enough enthusiasm from the reviewers during the discussion. We hope the reviews can help improve the paper for a better publication in the future. 




Decision: Reject