Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper studies the problem of out-of-distribution data detection, which is an important problem in machine learning. The authors propose to use Bayesian variational autoencoder which applies SGHMC to get samples of the weights of the encoder and the decoder. The proposed method is tested on two benchmarks to demonstrate effectiveness.

The proposed Bayesian variational autoencoder appears to be technically sound. When applying it to OoD detection, effective sample size is used to quantify how much the posterior changes given the new data. The authors claim that ESS will be large when the data is in-distribution since all samples explain the data equally well. First, I’m not sure this is true that all samples from the posterior should explain the data equally well even if it is in-distribution. Second, if the data is out of distribution, it is likely that all samples explain the data equally bad which also results in high ESS. In practice, it is very likely that p(x*|theta) are low for all the theta when x* is out-of-distribution. Am I missing something here?

How to determine whether a data is out-of-distribution or not based on ESS? Is the threshold of ESS a hyperparameter to tune?

For the experiments, I wonder why the authors put Gamma hyper priors for BVAE which was not used in the previous work that use SGHMC. Is there any reason for doing this? Again, it is unclear to me how the authors decide whether a data is out-of-distribution or not based on ESS.

It seems like simply applying SGHMC for the decoder parameters is sufficient, as the other treatments only improve the results incrementally but adding large computational and storage cost. I’m not familiar with the literature enough to tell whether the results of previous methods are reasonable or not. By looking at the table, it seems that the proposed method achieves some gain over the previous methods. 

In the experiments, BVAE only keeps the most recent 10 samples. Aren’t the samples very similar? Since the thinning interval is only 1 epoch.

It would make the paper stronger if the authors are able to demonstrate the usefulness of detecting OoD in latent space through experiments.   

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: After reading all the reviews, the comments, and the additional work done by the Authors, I have decided to confirm my rating.

==================

This paper leverage probabilistic inference techniques to maintain a posterior distribution over the parameters of a variational autoencoder (VAE). This results in a Bayesian VAE (BVAE) model, where instead of fitting a point estimate of the decoder parameters via maximum likelihood, they estimate their posterior distribution using samples generated via stochastic gradient Markov chain Monte Carlo (MCMC).
The informativeness of an unobserved input x* / latent z* is then quantified by measuring the (expected) change in the posterior over model parameters after having observed x* / z*.  The motivation is clear, when considered inputs which are uninformative about the model parameters, they are likely similar to the data points already in the training set. In contrast, inputs which are very informative about the model parameters are likely different from everything in the training data.

The contributions are:
- A Bayesian VAE model which uses state-of-the-art Bayesian inference techniques to estimate a posterior distribution over the decoder parameters.
- A description of how this model can be used to detect outliers both in input space and in the model’s latent space.
- Results showing that this approach outperforms state-of-the-art outlier detection methods.

The paper is well written, and the proposed ideas are well motivated.
However, the experiment section is too limited. The authors should at least use one more dataset such as CIFAR10. They just use FashionMNIST vs MNIST FashionMNIST (held-out).
In addition, it would strengthen the paper if the authors could show at least initial result about how the model performs to detect out of distribution in the latent space, given that it is considered as part of the contribution.

The paper lacks some references such as:
- Predictive uncertainty estimation via prior networks, NEURIPS 2018.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper advocates to use information gain to detect whether a sample is out of distribution. To that end, a Bayesian VAE is introduced for which that quantity is tractable. The experiments show a solid improvement over previous methods.

I like the paper, but I have a few questions. I am more than willing to increase my score from weak reject to weak or strong accept if these are addressed properly.

What is the relationship of information gain to the marginal likelihood of the data? Since both can be expressed in entropies, I can see a very strong relationship, but would enjoy the authors opinion here–what exactly is it that gives the edge?

The experiments report results on the likelihood based score. Were these results taken from previous publications or obtained from exactly the same pipeline?

Why is the "outlier in latent space" section included even though it is not experimentally verified? I think it should go, as conducting experiments is cheap in ML. On the other hand, if we cannot come up with an experiment to conduct, then what is hypothesis is tested? I think the section needs to be removed and be revisited in future work.

Is the method really principled? Where is the connection from the assumption that the score should be high for out of distribution and low for in distribution? If a method is called "principled" I want to see a rigorous derivation of how a method derives from what principles exactly and how it is approximated.

Since only the 10 most recent samples are kept to represent the posterior, I am worried about their diversity. I think the authors should back up that this is sufficient to represent the posterior.

What happens in the non-parametric limit, where the posterior will collapse to a point? Does the method not rely on an insufficiently inferred model?
review_title: Official Blind Review #3

Meta-review: This paper tackles the problem of detection out-of-distribution (OoD) samples. The proposed solution is based on a Bayesian variational autoencoder. The authors show that information-theoretic measures applied on the posterior distribution over the decoder parameters can be used to detect OoD samples. The resulting approach is shown to outperform baselines in experiments conducted on three benchmarks (CIFAR-10 vs SVNH and two based on FashionMNIST).

Following the rebuttal, major concerns remained regarding the justification of the approach. The reason why relying on active learning principles should allow for OoD detection would need to be clarified, and the use of the effective sample size (ESS) would require a stronger motivation. Overall, although a theoretically-informed OoD strategy is indeed interesting and relevant, reviewers were not convinced by the provided theoretical justifications. I therefore recommend to reject this paper.
Decision: Reject