Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:
The authors propose augmenting VAEs with an additional latent variable to allow them to detect out-of-distribution (OOD) data. They propose several measures based on this model to distinguish between inliers and outliers, and evaluate the model empirically, finding it successful.

Unfortunately, the method in this paper is developed unclearly and incorrectly. Although their experiments are somewhat successful, the problems with the text and method are severe enough to justify rejection.

Specifically, the authors' method proposes adding a term to the loss of the VAE that encourages the variational posterior (q) to distribute latent codes (z) for inliers and outliers differently. The equation which defines their new objective is unclear -- specifically, it is not clear whether the added KL term is computed for inliers and outliers both, or whether it is only computed for outliers. If it is the former, then the method does not make sense. If it is the latter, then the equation is incorrect or at the very least not clear in the extreme.

Furthermore, the term is added without consideration of whether or not the method is still optimizing a sensible variational lower bound. The authors attempt to justify the objective by writing out a variational lower bound for a VAE with a mixture prior where inliers and outliers are generated from different mixture components. However, their equations are incorrect -- the equation that is called the log likelihood is not the log likelihood, and the ELBO is similarly wrong.

Their empirical evaluation is reasonable, although the measures they propose to distinguish between inliers and outliers (i.e. the kl from the approximate posterior to the prior) is not thoroughly justified.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposes to counteract OOD problem in VAE by adding a regularization term to the ELBO. The regularizer is defined as the Kullback-Leibler divergence between a variational posterior for a negative sample and a marginal distribution over latents for negative data. The authors present experiments on MNIST and MNIST-like datasets, and CIFAR10 with SVHN. Unfortunately, I do not find the paper especially interesting. The motivation for adding the regularization term is not convincing. The experiments are insufficiently discussed.

Remarks:
- The paper proposes to ad a regularization to ELBO, namely, the Kullback-Leibler divergence between a variational distribution for a negative sample and a marginal distribution over latent variables for negative samples. I do not fully understand the motivation given on page 3. The authors show that including the negative data yields a new objective that is a sum of two log-lihelihood functions for "real" and negative data. However, later they propose to skip a (negative) reconstruction error term for the negative data. As a result, the authors obtain the objective they proposed. This explanation is very vague and I do not see what it adds to the story. Contrary, it causes new questions about their model and whether it is properly formulated.
I suggest to look into the following paper to see whether the model could be re-formulated:
Hu, Z., Yang, Z., Salakhutdinov, R., & Xing, E. P. (2017). On unifying deep generative models. arXiv preprint arXiv:1706.00550.

- I do not understand why the authors used Bernoulli distribution to model color and gray-scale images. The Bernoulli distribution could be used only for binary random variables. This is obviously flawed.

- In general, the results seem to partially confirm claims of the paper, however, they are quite vague. First, utilizing a wrong distribution is demotivating (see my previous remark). Second, I miss a better description of models and, in general, experiments' setup. Third, all results are explained in a laconic manner (e.g., "The other results in the table (...) confirm the assymetric behaviour of the phenomenon (...)"). There is neither deeper understanding nor discussion provided.

- Why there are no samples for CIFAR or SVHN provided?

======== AFTER REBUTTAL ========
I would like to thank the authors for their rebuttal. I really appreciate that the paper is updated and some concerns are solved. After reading the updated paper again, I tend to agree that the proposed idea is interesting for the problem of OOD detection using generative models. Therefore, I decide to update my score.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper discusses the detection of out-of-distribution (OOD) samples for variational autoencoders (VAE).
The idea is to train the encoder such that its output variational distribution q(z|\bar{x}) is pushed away from the prior of latent z. 
I think the paper needs more clarification and investigation for being published in the conference. 
My major concern is that more empirical investigation is necessary since the formulation provides a minor novelty. 
Specific points are given below. 

1) Weak novelty in terms of model design. 
The objective function consists of the standard (negative) ELBO term and additional KL term to modify the variational posterior of negative samples. 
This modification can be regarded as a form of outlier exposure (Hendrycks et al. 2018) specialized for VAE. 
The choice of \bar{p} is not much investigated. 
Any discussion if we use a more sophisticated model such as VampPrior* for stronger modeling capacity. 
* J. Tomczak and M. Welling, VAE with a VampPrior, AISTATS 2018.

2) The use generated samples as negative samples is interesting but mysterious. 
The authors conjecture that this works because the generated samples come from near the data manifold, but in-distribution samples and negative samples can be indistinguishable when the generative model is very well trained. 
What happens if, for example, the negative samples are generated by data augmentation techniques (such as cropping, rotation, mirroring, though mirroring and much rotation may be unsuitable for text images)? 
This can also produce near-manifold points. 
A deeper analysis why generated samples can improve the OOD detection performance is necessary. 
Furthermore, why does not this approach impact much for color images in Table 4?

3) More details of experimental procedures. 
3-1) How was data points are generated from VAE as negative samples? 
Possible ways are:
* sample z ~ p(z), then draw from the decoder x ~ p(x|z).
* use negative prior z ~ \bar{p}(z), then draw from the decoder x ~ p(x|z).
* this seems weird: use variational posterior z ~ q(z|x), then x ~ p(x|z).

3-2) Latent dimension of 10 for grayscale images seems small. 
Does the size affect the OOD detection performance when the size is 50 or 100 to make the model richer. 

3-3) How was the variance obtained when the decoder uses the Gaussian likelihood?
* fixed value?
* learned for each pixel?
* output from the decoder?

4) If we have access to diverse negative datasets, can the ODD detection perform better? 
Mixing multiple datasets or using both available dataset and generated samples can improve the performance while the test OOD samples are kept unseen. 
For example, train VAE on MNIST while using KMNIST and EMNIST as the negative sets to detect Fashion-MNIST as ODD. 
review_title: Official Blind Review #3

Meta-review: This paper proposes to improve VAEs' modeling of out-of-distribution examples, by pushing the latent representations of negative examples away from the prior.  The general idea seems interesting, at least to some of the reviewers and to me.  However, the paper seems premature, even after revision, as it leaves unclear some of the justification and analysis of the approach, especially in the fully unsupervised case.  I think that with some more work it could be a very compelling contribution to a future conference.
Decision: Reject