Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes a robust novelty detection method ("MAW") to model the distribution of the training data in the presence of high fraction (corruption ratios up to 30\%) of outliers. The method add new features to the variational autoencoder (VAE), to detect and isolate the outlier so that the learned distribution only represent the inlier distribution:
1. Uses a carefully designed dimension reduction component to extracts latent lower-dimensional features of the latent distribution.
2. Model the distribution of latent representation as a mixture of Gaussian low-rank inliers and full-rank outliers, both using full covariances instead of diagonal covariances as commonly used in previous VAE-based methods for novelty detection.
3. Penalizes the Wasserstein-1 distance between the data distribution and the latent distribution from the prior distribution. Under a special setting, it theoretically proves that using the Wasserstein-1 metric for regularization yields outliers-robust estimation and is suitable to the low-rank modeling of inliers, while the commonly used Kullback-Leibler (KL) divergence does not.
4. Using the least absolute deviation error for reconstruction.
Experiments on popular anomaly detection datasets demonstrate state-of-the-art results of MAW on standard benchmarks for novelty detection.

================== After rebuttal ==================
Several typos:
In Figure 1, to be consistent the X and Z may be written in lower case.
In Eq. (6), the transpose should be applied to the right $U_1^{(i)}$ instead of the left one.
In Appendix C.1, ``It seems that MAW seems to learn" should be ``It seems that MAW learns" or ``MAW seems to learn".

In Appendix D.3:
In line 2 above Proposition D.2, ``the ill-posedness of (11) with $\mathcal{R}= W_2$": The $\mathcal{R}= W_2$ should be $\mathcal{R}= KL$.
Please rephrase ``the KL divergence fails is unsuitable for low-rank covariance modeling", by e.g., removing ``fails" or inserting ``and" between ``fails is".
review_title: This is a solid piece of work, very well written. The problem is well motivated/defined with respect to previous works, which are comprehensively reviewed. The methods and theories are technically sound. Contributions are multi-fold and significant.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This study proposes a novel method that can work well even the training data is corrupted by partial data from the unknown domain. Though it deals with the well-known problem called 'Noisy data/label', its approach is not the same thing as the previous works as it focuses on variational autoencoder on the task of novelty detection. And its arguments and statistical assumptions are followed by mathematical proofs. 

Overall, it is an interesting approach and I believe it would give a good way to ML practitioners who are struggling with noisy datasets in real-world applications. However, there some questions/comments about the article which may make the study more consolidate:

Questions
- In the description of the proposed method, MAW, Discriminator generates Loss(Lw1) by comparing between Zgen and Zhyp. And Zhyp is unimodal distribution while Zgen follows MoG. I wonder whether there is a risk that inlier and outlier distributions are mixed(combined) as the loss makes the generator generates just the same mu/sigma regardless of the domains. If so, is there any equilibrium trick required so that the generator would not be strong too much?

- Though it is hard to pre-estimate how the outlier distribution looks like, it is more common to assume the outlier distribution has multi-modal than uni-modal. However, the proposed method approximates the outlier distribution as unimodal Gaussian distribution. Is it possible to model the outliers as multi-modal distribution such as MoG?

- In the experiment with the multiclass dataset, the number of possible inlier domains is the same as the number of classes in the dataset. And the characteristic of 'training data' may be different by each combination. I wonder the experiment of this study covered all possible sets. 
 And the corrupted data is sampled randomly from the other classes. Is there any deviation in the performance by each sampling?

- This study aims to generate the model to be robust to corrupted training data. However, in the result, it is not clear that the proposed method is more robust than others as the AUC/AP from MAW falls (maybe greater than others) as the outlier ratio increases. The authors may give explanations about the result in detail. 


Additional Comments
- The readability of Figure 2, 3 is not good. How about showing them on the tables?
- This study shows the superiority from four datasets (image, non-image). However, there is more dataset widely used for novelty detection such as (Fashion) MNIST or MVTech. The authors may consider doing the same experiments on the other dataset.
-The authors may compare the method not only to the novelty detection methods, but many previous works which also aims to be robust to noisy data(or label) in the training process. 

review_title: Presenting a robust method to the noisy training dataset for novelty detection by modeling mixture of Gaussian with outlier and inlier distribution in the latent space

final_score: rating: 6: Marginally above acceptance threshold
review_content: Compared with conventional VAE, this paper incorporates the following strategies to improve VAE for novelty detection from corrupted training data. 1) By considering that outliers tend to have more complex structures, this paper assumes inliers and ourliers lie on Gaussian distributions with different ranks, and proposes a Gaussian mixture model for posterior q(z|x) including two component: a low-rank multivariate Gaussian distribution component for modeling inliers and a full-rank multivariate
Gaussian distribution component for outliers.  2) Applying the Wasserstein distance (W distance) between q(z|x) and p(z) instead of the KL divergence in the original VAE. The paper also proves the superiority of using W distance over KL divergence, by showing that the minimized value exists in W distance between a Gaussian mixture distribution q(z|x) and Gaussian prior p(z). 3) 3)For Rrconstruction loss, this model adopts ||x-D(z)||_2 as described in (3) instead of ||x-D(z)||^2_2, which is used in conventional VAE. This practice helps to alleviate the problem that the loss item in conventional VAE being too small when data point deviates from the center of the Gaussian distribution.
The experiments show that proposed method achieves good result among 4 datasets, which domonstrate the effectiveness of the three strategies.
Advantages of this paper: although Gaussian mixture model has been used in previous work of VAE, this model innovatively propose Gaussian components with different ranks for modeling inlier and outliers to improve the performance for training data with corrupted samples. This work also gives a theoretical guarantee for the advantage of W distance minimization in given setting.
Potential drawbacks: Problem setting is confusing. Outliers/novelties are supposed to be rare in the dataset, while the paper assumes that a nontrivial fraction of outliers existing in the data. In addition, more recent work is not compared, such as Ruff, L., Vandermeulen, R. A., Görnitz, N., Binder, A., Müller, E., Müller, K. R., Kloft, M. (2019). Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694
review_title: This paper propose a robust VAE model for novelty detection (semi-supervised anomaly detection), which allows a nontrivial fraction of corrupted samples (outliers) within the training set.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper seeks to address the problem of novelty detection under the circumstance of having high corruptions in the training data. This is different from most previous work, which often assumes that training dataset is pure. To address this issue, a VAE-based approach is adopted in this paper, with several modifications made to the vanilla VAE to promote the robustness of VAE in detecting outliers in the corruption circumstance. Among the modifications, the paper assumes the posterior is approximated by a two-component Gaussian mixture distribution, with each having a low-rank and full-rank covariance matrix, respectively. The paper hopes that the posterior of inliers (normal data points) can be represented by the low-rank covariance matrix, while that of outliers cannot. Another notable modification is that the Wasserstein-1 regularization is used to replace the KL-regularization in the ELBO, which is claimed to be more suitable to the low-rank modeling. Some experiments are conducted to evaluate the outlier detection performance of the proposed method under corrupted circumstance.

Strength:
1. The problem of detecting outliers under highly corrupted environment is of practical importance and is not investigated under the extremely corrupted circumstance. 

2. The idea of proposing to use low-rank and full-rank geometry characteristics to separate inliers and outliers is interesting. 


Weakness:
1. Although the idea of using low/full-rank geometric characteristic to detect outliers is interesting, the paper barely states how to implement this idea, that is, how to enforce the inliers resides in the low-rank covariance matrix, while the outlier will not. Without any specific design, we cannot believe this will be realized automatically.

2. The theoretical result (Section 3) established to argue the superiority of Wasserstein regularization in robustness is better than the original KL regularization in vanilla VAE is not convincing, or is not practically meaningful. That is because the result only reveals that under a very special case, the Wasserstein may learn the true distribution under corrupted environment. We cannot see this result has any implication under a broader or more general circumstance.

3. I guess the true reason to choose W_1 over KL distance is that because a Gaussian mixture posterior is employed, the KL cannot be evaluated in close-form. By resorting to W_1 distance, the distance can be estimated with the samples, but the KL distance cannot be estimated in this way. Overall, I cannot buy the argument that the authors proposed to use the W_1 distance/regularization here.

4. I also have some concerns over the choosing of hyper-parameters. The paper sets the dimension of latent representation (d) to be 2, which, I think, is too small. But if it is set to be an appropriate value (e.g. 100), the posterior with a full-rank covariance matrix will be computationally expensive. Moreover, I also think that it is not a good idea to set the mixture coefficient $\eta$ to be a fixed value. Generally, this should be learned from data, because you cannot know the ratio of different components in advance.

5. I also have some doubts over the experiment settings. At the training stage, the training data is corrupted by a fraction of outliers. But if the data is corrupted by ‘outliers’, that means the model has already made use of the information of outliers. So, if the ratio of outlier in the training dataset is higher, it may be more favorable to the proposed method. For example, if the ratio is large, e.g. c=0.3, this problem under this setting is more like a clustering problem. So, this kind of settings may be not fair to the comparing methods. Maybe, the better setting should be let the training data corrupted by some data that is not in both the training and testing datasets. 
review_title: Interesting problem, but the method is not convincing

Meta-review: The paper proposes a novelty detection method when training data is itself noisy. A VAE-based approach is developed that promotes robustness of the VAE. The paper assumes that the encoder a two-component Gaussian mixture distribution, individual components denoting inliers and outliers.

The paper hopes that the posterior of the inliers (normal data points) can be represented by a low-rank covariance matrix, while the outliers need a full covariance. Another notable modification is that the Wasserstein-1 regularization is used to replace the KL-regularization in the ELBO, which is claimed to be more suitable to the low-rank modeling. 

While this is a relevant problem, and the idea is perhaps interesting, some concerns have been raised.
* The details how to fit the model with the desired mixture posterior in practice is unclear.
* The arguments of section 3 to illustrate the superiority of Wasserstein were found unconvincing, with limiting/unclear assumptions
* The ultra-low latent space dimension (2) is not sufficiently justified 
* The experimental section and the selected datasets are small scale, it would be good to include a free larger scale datasets (at least cifar10).
* Comparisons to the open set recognition, or out-of-distribution (OOD) detection would have been a plus.

Overall, this is an OK paper but not yet of sufficient quality.
Decision: Reject