Negative Sampling in Variational Autoencoders
Authors: Adrián Csiszárik
Beatrix Benkő
Dániel Varga
Venue:  (2020)
Keywords: Variational Autoencoder
generative modelling
out-of-distribution detection

Abstract: We propose negative sampling as an approach to improve the notoriously bad out-of-distribution likelihood estimates of Variational Autoencoder models. Our model pushes latent images of negative samples away from the prior. When the source of negative samples is an auxiliary dataset, such a model can vastly improve on baselines when evaluated on OOD detection tasks. Perhaps more surprisingly, we present a fully unsupervised variant that can also significantly improve detection performance: using the output of the generator as a source of negative samples results in a fully unsupervised model that can be interpreted as adversarially trained.


Under review as a conference paper at ICLR 2020
NEGATIVE SAMPLING IN VARIATIONAL AUTOENCODERS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose negative sampling as an approach to improve the notoriously bad -utto--diitributtin likelihood estimates of Variational Autoencoder models. Our model pushes latent images of negative samples away from the prior. When the source negative samples is an auxiliary dataset, such a model can vastly improve on baselines when evaluated on OOD detection tasks. Perhaps more surprisingly, we present fully unsupervised version of employing negative sampling in VAEs: when the generator is trained in an adversarial manner, using the generator's own outputs as negative samples can also significantly improve the robustness of OOD likelihood estimates.
INTRODUCTION
Learning semantically meaningful and useful representations for downstream tasks in an unsuper- vised manner is big promise of generative modeling. While plethora of work demonstrates the effectiveness of deep generative models in this regard, recent work of Nalisnick et al. (2019a) and Choi al. (2018) show that these models often fail even at task that supposed to be close their original goal of learning densities. Variational Autoencoders, PixeICNN and flow-based models cannot distinguish common objects like cats and dogs from house numbers. That is, when trained e.g., on CIFAR-10, the models consistently assign higher likelihoods to the elements of the SVHN test set than for the elements of the CIFAR-IO test set or even the elements of the CIFAR-10 train set. As generative models are becoming more and more ubiquitous due to the massive progress in this area in recent years, it is of fundamental importance to understand these phenomena.
In this work we study Variational Autoencoder (VAE) models, and besides the likelihood, also investigate to what extent the latent representation of a data point can be used to identify out-of- distribution (OOD) samples (points that not from the true data distribution). For this purpose, we consider the KL divergence between the prior and the posterior distribution of a data point as a score to distinguish inliers and outliers. Our contributions
summarized as follows:
We demonstrate empirically that the extent of this notorious phenomenon of bad out- of-distribution likelihood estimates - present in VAEs largely depends on the observation model of the VAE. In particular, our experiments show that it diminishes when a Gaussian noise model is considered (with a reasonably sized fixed or learned variance) instead of a Bernoulli. Mcanwhile, when examining only the KL divergence between the prior and the posterior distributions in the latent space (instead of the full likelihood), the weak separating capability more consistently prevails between inliers and outliers.
We propose negative sampling in Variational Autoencoders above weaknesses of the model family. this method, we introduce an additional prior distribution p(z) in the latent space, where the representations of negative samples meant to be mapped by the inference model of the VAE machinery. Negative samples can be obtained from an auxiliary dataset, setting - from a generative model trained on the ground truth distribution itself.
approach to alleviate the
to remain completely in the unsupervised
We present empirical evidence that utilizing negative samples either from an auxiliary dataset from adversarial training scheme (using the adversarially trained generative model itself to provide the negative samples) significantly and consistently improves the discriminative power of VAE models regarding dustof-distributtin samples.

---

Under review as a conference paper at ICLR 2020
The general intuition behind our approach is that if the posterior distribution of each and every point is pulled towards the prior then it is rather natural to expect that the system will map out- of-distribution samples close to the prior, well. This viewpoint suggests that providing negative signals throughout the learning process would be heneficial to enhance the OOD discriminative power of the system.
Hendrycks et al. (2199) demonstrate that utiliizigg auxiliary datasets as OOD examples (as a super- vised signal) significantly improves the performance of existing anomaly detection models on image and text data. First, we study how this approach can be employed in the VAE setting. Beyond that, we also propose a method which remains completely in the unsupervised learning paradigm (with- out using an auxiliary dataset for supervised signal). The core idea of this unsupervised approach is to use generative model to provide near-manifold negative samples throughout the training process for which the model is either implicitly explicitly encouraged to give low likelihood estimates. In our proposed method, these negative samples are obtained from the currently trained VAE model itself by utilizing the generated samples.
2 BACKGROUND
The generative modeling task aims to model a ground truth data density (xx) on a space X' by learning to generate samples from the corresponding distribution. The learning is done in an unsu- pervised manner with sampled observables X = {x(()) J==1 training points assumed to be drawn independently from p" (ar). where N is the sample size. In latent variable models, the observables modeled together with hidden variables on which a prior distribution p(z) is imposed.
The Variational Autoencoder (VAE) Kingma & Welling 2013) is a latent variable model that takes the maximum likelihood approach and maximizes a lower bound of the sample data log likelihood E log fogro(x()) where 0 are the generator parameters. The utilized lower bound L(O, db x()) (called the ELBO) comes from a variational approximation (z(x((()) of the intractable posterior ())) where & are the variational parameters:
logpo(x(")) = loog / [=(x((==))(2) >
E (= log po (x(()) DKL (xl((()) ll p(x)) & x(0)),
Reconstruction term
divergence term
In the VAE model the parametrized distributions pe and g6 are modeled with neural networks and are trained jointly to maximize C with some variant of the SGD. The prior is often chosen to be the multivariate standard normal distribution, and a Bernoulli or Gaussian noise model is considered in the observable space to define the likelihood.
To give likelihood estimates for unseen data points test time, one can use the trained inference model go (=(2)(()) (also referred
encoder) and generative model po po [z) (also referred as decoder) to estimate the ELBO, thus giving a lower bound of the likelihood. Throughout our paper, we are considering these ELBO estimates to measure the likelihood of data points.
NEGATIVE SAMPLING IN VARIATIONAL AUTOENCODERS
To incorporate negative samples in the VAE training process, we introduce an additional prior dis- tribution p(z) (called the negative prior) on the latent variables z into which the representations of negative samples X = { are meant to be mapped by the inference model. This encour- aged in the training process by adding to the regular ELBO new loss term: the KL divergence of the posterior distributions of negative samples to this negative prior. Thus the joint loss function (to be minimized) is as follows:
2

---

Under review as a conference paper at ICLR 2020
L(0 0, x0,x x(1) - L(0, x ) + (ex || [(())
(xxx((x)x(0) log po (x (2) DKL (20 p(z)) DKL(qo(z/2 ((()).
(1)
-1- ELBO for x(1)
Kl.term for negative sample x
Motivating our loss function The loss function defined in equation I is still upper bound of the negative data log likelihood (for the positive samples) as the added loss term is non-negative. The new loss term explicitly imposes the discriminative task for the inference model: to distinguish inliers and outliers in the latent space. With these two components, while still preserving the aim of maximizing the likelihood for inliers, we also expect the implicit behavior of reducing likelihood estimates for outliers. For the outliers, a trained inference model produces latent representations that are close the negative prior p(z). thus, supposedly far from the prior p(z). Also, the system is not encouraged to learn to generate from the vicinity of the negative prior, therefore not only the KL. term the likelihood, but the reconstruction part a negative sample is affected when inferring the likelihood estimate of an outlier.
3.1 THE CHOICE OF THE NEGATIVE PRIOR
One has numerous options to choose the positive and negative priors. In this paper, we simply choose use a standard normal for the positive prior, and shifted standard normal for the nega- tive prior. With a rotationally symmetric posterior distribution, the distance between the two priors would be the only unspecified hyperparameter of such a model. The assumption of diagonal co- variance matrix posterior breaks rotational symmetry in principle, but our exploratory experiments have demonstrated that the magnitude of the shift is a more significant modeling choice than the direction/sparsity of the shift.
The role of DKL (p(=) p(z)) The magnitude KL divergence between the negative and the positive prior plays important role. Larger (x || p(=)) values result in larger DKL (gx(((2))(() || p(z)) terms when evaluating the KL divergence term of the likelihood in trained model, and also result in heavier weighted KL divergence terms during the optimization pro- E.g., with farther shifted negative prior mean, larger penalty is given for wrong inference.
The role of the latent dimension The above argument gives rise to an interesting side effect: increasing the latent dimension also increases (KL(f(z) || p(=)), thus resulting in larger weight of the discriminative KL terms.
With the above simple choice of the shifted normal for the negative prior, our experiments already demonstrate the effectiveness of our proposed method. One possible direction for further improve- ment would be to explore positive-negative prior pairs that better reflect the inlier-outlier structure of different datasets. We leave this investigation for future work.
3.2 SOURCE OF NEGATIVE SAMPLES
Negative samples can also obtained in different ways. The task our models to generalize from the negative samples much possible all possible out-of-distribution samples, so that they can push down likelihood estimates of those. Depending on the source of negative samples, this generalization can be easier or harder. We conduct experiments with several variants:
samples from an auxiliary dataset,
the data with isotropic Gaussian noise added,
generated samples from the trained model itself,
generated samples utilizing an adversarial training scheme.
Negative samples that very far from the data manifold do not facilitate generalization. Noise added to data points is simple and principled way to sample from the vicinity of the data manifold, but will see, it does not provide good generalization. We argue that the reason for this is that
3

---

Under review as a conference paper at ICLR 2020
discriminating between noisy and noiseless points is too easy for the encoder, "semantically" the noisy versions are far from the data manifold. In contrast, utilizing samples produced by a generative model (which could be the trained generative model itsclf) is manifold negative samples, as we will experimentally demonstrate.
suitable way to acquire near-
Why using generated data as negative samples could help? An immediate counterargument against utilizing generated samples as negatives could be the following: for a well-trained model, the generated images are indistinguishable from the ground truth images, so training the model to discriminate them is nonsensical. There are several reasons why such an unsupervised method could still work. First, in practice, trained generative model is typically not perfect. True data samples and generated samples can be distinguished even for fully trained models. Second, even assuming a perfect generator at the end, during the training process, the generated samples might still help to guide the model toward equilibrium that promotes a lower likelihood for OOD samples. Moreover, when utilizing auxiliary datasets, we have to choose the auxiliary dataset (or multiple datasets) carefully to wedge in between the training set and potential nut-of-distributin data point, otherwise the weak separating capability could prevail. In contrast, learning to discriminate the generated near-manifold examples from the ground truth data is a harder task, and could result in discriminating a more diverse set of potential out-of-distribution samples.
Our experiments also confirm this. When utilizing adversarial training scheme (to be introduced later in this section), generated images not only facilitate our discriminative training procedure, but even achieve higher level of generalization in the following sense: utilizing generated images im- proves on the baseline in all permutations of the roles for the grayscale datasets when considering discrimination in the latent space, while utilizing auxiliary datasets fails to achieve notable improve- ment in some cases. (See the results in Table in rows with AUC KL, and more details in the experiments section.)
In our preliminary experiments, we observed that in some examined cases, utilizing simply the gen- erated images of the currently trained VAE model fail to provide a good signal for the discriminative task. We achieved greater success when we augmented our model with adversarial training. We hypothesize that the reason behind this is that obtained negative samples are richer features and semantically more meaningful for the task. (See Lee et al. (2018) for an incarnation of this idea in the context of classification and generative adversarial networks.)
The utilized adversarial training scheme When experimenting with generated samples as nega- tives, we utilize an adversarial training scheme where the generator (and only additional gradient signal through the encoder map the randomly generated images into the prior. This is encouraged via the following additional loss term:
the generator) gets an
DKL (2)(x()))) p(=)).
where ( denotes a generated image obtained from the generator (x(()) 2), where z is sampled from the prior p(z). Together with the fact that the encoder also gets the generated images as negative samples, this results in adversarial training procedure. In this setup, the separate loss functions of the encoder and generator are:
I T (0, Φ (1) & (())
T( (0) ())).
I Lody (0, 6. x()) x (0) L(0, Φ (0) () + DKL DKL (2)(2 || p(z)).
Our utilized scheme is simple yet effective. However, it is just of the options. Another choice would be to use separate generative model with the specific task to provide negative samples. We invite the research community to develop methods that can provide near-manifold examples that can facilitate the training of models with better OOD likelihood properties.
4 EXPERIMENTAL RESULTS
The general setup Our main concern is the discriminative power of VAE models regarding out- of-distribution samples. Following the conventions of related work, the general experimental setup in this section is as follows: we train a model on a train set of dataset (c.g. train set of Fashion- MNIST) and then require the model discriminate between the test set of the train dataset (e.g. test
4

---

Under review as a conference paper at ICLR 2020
set of Fashion-MNISTT -NIST) and the test set of an out-of-distribution dataset (e.g. test set of MNIST). During the training phase, the models do not encounter examples from the OOD dataset, only at test time are they expected to able to distinguish between inliers and out-of-distribution samples.
Quantitative assessment For quantitative assessment, we use the threshold independent AUC metric calculated with the biss-per-dimension score (denoted by AUC BPD) and also with the KL di- vergence of the posterior distribution of a data point to the prior (denoted by AUC KL). All reported numbers this section are averages of 5 runs with standard deviations denoted in parentheses.
Datasets and experimental details We conduct experiments two sets of datasets: color im- ages of size 32x32 (CIFAR-IO, 10, SVHN, downscaled ImageNet) and grayscale images of size 28x28 (MNIST, Fashion- -MNIST, Kuzushiji-MNN -MNIST, EMNIST-Leeters). For both cases, the (positive) prior chosen to be standard normal, and the latent dimension is set to 100 for color images, and to 10 for grayscale images. For more detailed description of the utilized datasets, models, and training methodology, see Appendix AA] We present generated samples from the models in Appendix [ The samples demonstrate that the models preserve their generative capability even after adding the extra loss terms.
The choice of the negative prior In our experiments, the negative prior is standard normal with shifted mean. For color images it is centered at 25 1, for grayscale images it is centered at 8 1. The magnitude of the shift is set based on parameter sweep, which was evaluated using Fashion- MNIST and MNIST in the range of {2, 4, 6, 8, 10) for grayscale images, and using CIFAR- 10 and SVHN in the range of {5, 10, 15, 20, 25, 30} for color images. After observing a clear trend, we have chosen the mode.
4.1 THE EFFECTIVENESS OF NEGATIVE SAMPLING
To demonstrate the effectiveness of negative sampling we present two different sets of experiments: first we incorporate negative samples from an auxiliary dataset, second we explore the use of adver- sarially generated negative samples.
Almost perfect discrinination with auxiliary datasets The AUC scores Table D show that using the auxiliary dataset source of negative samples in most cases proved to result in models that are capable to distinguish nearly perfectly between inliers and OOD samples. This is also the case with color images, experimental results in Table 2 show.
Failure modes with auxiliary datasets One can observe in Table 1 that despite the above mentioned improvements there cases when utilizing an auxiliary dataset fails improve on the OOD separating capability. One example for this is when the inlier set is the LEMISSTLettters, the OOD test set is MNIST, and the utilized auxliary dataset is MeshiOn-MNIST (the results for this setup are in the last row of Table T. Showing skirts and boots for the model in training time does not help discriminating between letters and numbers test time. We hypothesize, that this as an example of the case, when the auxiliary dataset (regarding its features) does not wedge in between the inlier and the outlier test set. One possible way of improvement in this regard to utilize several auxiliary datasets to present a more diverse set of examples for possible out-of-distributie samples in terms of features and semantic content.
Of course, the most beneficial would be to train the system distinguish between the inliers and every possible outlier data points instead of just learning to separate only one or a specific set of auxiliary dataset. This motivates experiments utilizing generated samples as negative, with the idea that learning to separate from near-manifold examples could facilitate a better generalization in terms of OOD detection.
improvements in all permutations in AUC KL In the case of the grayscale images, the last column in Table shows the effectiveness of the fully unsupervised ap- proach: regardless of whether using Gaussian and Bernoulli noise mode!"] the trained models
Unsupervised method:
Even though the Berouuli noise model might not be particularly good choice for modeling grayscale or color images, here follow the literature when considering baseline.
5

---

Under review as a conference paper at ICLR 2020
Table I: Comparing the out-of-distributinn discriminative power of baseline VAE models and VAE models with negative sampling on grayscale images. Numbers for all permutations with the different possible roles of the three datasets (MINST, Fashion-MNIIST and EMNIST-Letters) are reported. When an auxiliary dataset is utilized, the auxiliary dataset is the one out of the three that is not utilized neither as inlier nor for OOD testing purposes.
Baseline VAE
Negative:
Negative:
Inlier
OOD
Noise model
(no negative)
auxiliary
adversarial
Fashion-Min MNIST
MNIST
Bernoulli
0.46 (0.05)
1.00 (0.00)
0.70 (0.13)
Fashion-MNIST
MNIIT
Gaussian
0.98 (0.00)
1.00 (0.00)
0.80 (0.04)
Fashion-MNY -NNIST
Letters
Bernouli
0.61 (0.01)
0.99 (0.00)
0.78 (0.07)
Fashion-MNIST
Letters
Gaussian
0.97 (0.00)
1.00 (0.00)
0.85 (0.04)
AUC BPD
MNIST
Fashion-MNIST
Bernoulli
1.00 (0.00)
1.00 (0.00)
1.00 (0.00)
MNIST
Fashion-MNIST
Gaussian
0.97 (0.00)
1.00 (0.00)
0.98 (0.01
MNIST
Letters
Bernoulli
0.99 (0.00)
0.99 (0.00)
0.98 (0.00)
MNIST
Letters
Gaussian
0.78 (0.14)
0.93 (0.08)
0.79 (0.04)
Letters
Pashion-MNIST
Bernoulli
0.98. (0.00)
0.98 (0.00)
0.99 (0.00)
Letters
Fashion-MNIST
-NNIST
Gaussian
0.80 (0.07)
0.76 (0.08)
0.93 (0.04)
Letters
MNIST
Bernoulii
0.58 (0.02)
0.58 (0.02)
0.73 (0.07)
Letters
MNIST
Gaussian
0.67 (0.17)
0.58 (0.20)
0.65 (0.04)
Fashion-MNIST
MNIST
Bernoulli
0.61 (0.09)
1.00 (0.00)
0.88 (0.07)
Fashion-MNIST
MNIST
Gaussian
0.26 (0.03)
1.00 (0.00)
0.74 (0.05)
Fashion-MNIST
Letters
Bernoulli
0.68 (0.07)
1.00 (0.00)
0.84 (0.04)
Fashion-MNIST
Letters
Gaussian
0.38 (0.04)
0.99 (0.00)
0.79 (0.05)
AUC KL
MNIST
Fashion- MNIITT
Bernoulli
0.73 (0.14)
1.00 (0.00)
0.94 (0.10)
MNIST
Mashion-MNISS
Gaussian
0.71 (0.04)
1.00 (0.00)
0.98 (0.01)
MNIST
Letters
Bernoulli
0.64 (0.03)
0.76 (0.03)
0.89 (0.02)
MNIST
Letters
Gaussian
0.54 (0.07)
0.75 (0.08)
0.74 (0.04)
Letters
MNshiTn-MNIST
Bernoulli
0.66 (0.14)
0.54 (0.09)
0.98 (0.00)
Letters
-MIIST
Gaussian
0.54 (0.10)
0.49 (0.23)
0.91 (0.05)
Letters
MNIST
Bernoulli
0.37 (0.05)
0.45 (0.03)
0.75 (0.06)
Letters
MNIST
Gaussian
0.36 (0.08)
0.43 (0.10)
0.64 (0.04)
achieve higher AUC KL scores than the baseline in all permutations. The method also shows better AUC BPD scores than the baseline in most of the cases where the baseline fails (i.e., baselines with below 0.6 AUC BPD scores). One can observe that when the train set is EMNIST-Letters and the OOD set is MNIST, the separation is still not achieved with this method either. The possible reason behind this is that the visual features of these two datasets are very close to each other and it is hard task to switch the default relation between them (note that when these two datasets switch roles, the likelihood estimates correct). Table 2 shows that in the case of color images, the unsupervised method also achieves notable discriminative performance improving on the baseline.
Random noise and additive isotropic Gaussian noise does not help We also investigated how the choice of negative samples influences the performance of the trained model. We conducted further experiments with the following negative samples: 1) Kuzushiji-MNIST)] (KMNIST) as another auxiliary dataset, 2) random noise (in which we sample each pixel intensity from the uniform distribution on [0, 1--1-_11 modeling a dataset with less structure), 3) with additive isotropic Gaussian noise added to the inlier dataset.
The results in Table 3 show that utilizing either KMNIST or MNIST-Letters results in perfect sep- aration of the inliers (Fashion-MNIITT) and outliers (MNIST). The weak results with random noise as negative samples show the significance of the choice of negative samples. We also experimented with utilizing the training set itself with an additive isotropic Gaussian noise as negative samples rather natural choice to provide near-mann manifold examples. With an additive noise of a 0.25,
"EMNIST-Letteer Kuzushij-MNIST and Fashion-MNIST are datasets that can be utilized drop-in re- placements for MNIST.

---

Under review as a conference paper at ICLR 2020
Table 2: Comparing baseline VABS and VA with negative sampling with Bernoulli, Gaussian, and
Quantized Gaussian (Q. Gaussian) noise models on color image datasets.
Baseline VAE
Negative: auxiliary
Negative: adversarial
Inlier
OOD
Noise model
(no negative)
CIFAR- 10
SVHN
Bernoulli
0.59 (0.00)
0.90 (0.05)
0.81 (0.04)
AUC BPD
CIFAR-10
SVHN
Gaussian
0.25 (0.02)
0.93 (0.01)
0.84 (0.03)
CIFAR-10 10
SVHN
Q. Gaussian
0.19 (0.00)
0.92 (0.03)
0.82 (0.03)
SVHN
CIFAR-I0
Bernoulli
0.51 (0.00)
1.00 (0.00) 0.70 (0.03)
SVHN
CIFAR-10
Gaussian
0.92 (0.00)
1.00 (0.00)
0.75 (0.11)
CIFAR-10
SVHN
Bernoulli
0.29 (0.00)
0.90 (0.06)
0.81 (0.04)
AUC KL
CIFAR-10 10
SVHN
Gaussian
0.25 (0.01)
0.93 (0.01)
0.84 (0.03)
CIFAR-10
SVHN
Q. Gaussian
0.28 (0.01)
0.92 (0.03)
0.82 (0.03)
SVHN
CIFAR-10 10
Bernoulli
0.87 (0.00)
1.00 (0.00)
0.70 (0.03)
SVHN
CIFAR-10
Gaussian
0.74 (0.01)
1.00 (0.00)
0.74 (0.11)
Table 3:
Comparing baseline model and negative sampling with different sources for negatives. Columns correspond to different sources for negative samples. Results for the baseline (i.c., VAE without negative sampling) are indicated again in the first column for comparison. Samples from the different data sets are also depicted in the last row to show their general visual characteristics.
Inlier
OOD
Auxiliary dataset as negative
Negative:
Reshion-MNIST
MNIST
Random
KMNIST
Letters
Adversarial
AUC BPD
0.46 (0.05)
0.47 (0.05)
1.00 (0.00)
1.00 (0.00)
0.70 (0.13)
AUC KL
0.61 (0.09)
0.56 (0.08)
1.00 (0.00)
1.00 (0.00)
0.88 (0.07)
Test BPD
0.30 (0.00)
0.30 (0.00)
0.30 (0.00)
0.30 (0.00)
0.47 (0.09)
OOD BPD
0.35 (0.08)
0.32 (0.04)
1.10 (0.09)
1.44 (0.20)
10 (1029)
6
3
2' 700
2
7
X
2
9
0
9
0
0
6
5
17
0 C 0-
/
3
/
9
2 3
KR
11
2N X 1 X ^
P
0
5 3
8
/ 7
2
7 3
6
4 ~)イ
38 7/
M
0
4
D
1 622
the results for the AUC BPD metric is 0.44 (0.01) and 0.70 (0.09) for the AUC KL, showing weak discriminative power.
4.2 THE EFFECT OF THE NOISE MODEL
Examining the results for baseline VAE models (i.e., models without negative sampling) in Table and Table 2 we can observe great variability in the OOD detection performance.
The noise model greatly influences the phenomenon The results suggest that the intriguing phe- nomenon in VAEs discussed by Nalisnick et al. (2019a) and [hoi et al. (018) is highly depen- dent on modelling choices. the case of grayscale images, when changing the noise model from Bernoulli to Gaussian (and otherwise remaining in the same experimental setting as Nalisnick et al. (2019a)), the issue assigning higher likelihood estimates to OOD samples simply does not occur. However, one can observe that discrimination between inliers and OOD samples based on the KL divergence between approximate posterior and prior is hardly feasible, with below- 2 AUC scores. Mcanwhile, with Bernoulli noise model (also used in Nalisnick al. (2019a)) both the likelihood- estimates and the KL divergences fail to discriminate. The other results in the table (where models trained MNIST) confirm the asymmetric behaviour already described by Nalisnick et al.
7

---

Under review as a conference paper at ICLR 2020
2019a). that is, switching the roles of the inlier and outlier dataset affects the presence of the phe- nomenon. Concerning experiments with color images, the corresponding rows of Table 2 again show the importance of modelling choices. When CIFAR-10 is the training set, the phenomenon persistently occurs with Berouulli, Gaussian and Quantized Gaussian noise models. When SVHN the training set, one can observe again a great variability in the AUC scores.
5 RELATED WORK
Our investigations are mostly inspired by and related to recent work on the evaluation of generative models on OOD data (Shafaci al. | 2019). These works report that despite intuitive expectations, generative models including but not limited VAEs consistently fail at distinguishing OOD data from the training data, yielding higher likelihood estimates on unseen OOD samples.
Nalisnick et al. (2019| Choi al. 2018: Hendryeks et al.
Nalisnick et al. (2019a) examine the phenomenon in detail, focusing on finding the cause of by analyzing flow-based models that allow exact likelihood calculation. Choi et al. (2018) also notice the above-mentiones phenomenon, while they address the task of OOD sample detection with Generative Ensembles. They decrease the weight of the KI. divergence term in the ELBO (contrarily to what is promoted by the 3-VAE loss function) to encourage a higher distortion penalty during training. resulting in better performing model. This observation also confirms the importance of the noise model and the balance between the KL. and the reconstruction term.
The ominous observation is presented also by HHendrycks al. (2019), but they concentrate on improving the OOD data detection with Outlier Exposure. Their work demonstrates that utilizing samples from auxiliary data set as OOD examples, i.e., training models to discriminate between training and auxiliary samples, significantly improves on the performance of existing OOD detection models on image and text data. However, they do not investigate the VAE model, and their general setup always requires an auxiliary dataset. Our work also sheds light on an issue with this approach: one should choose the auxiliary datasets carefully to obtain robust OOD detection.
Within the context of uncertainty estimation, Lee et al. (018) demonstrate that adversarially gener- ated samples improve the confidence of classifiers their correct predictions. They train classifier simultaneously with a GAN and require it to have lower confidence on GAN samples. For each class distribution, they tune the classifier and GAN using samples from that OOD dataset. Their method of utilizing generated samples of GANs is closest to our approach of using generated data points as Lee
negative samples, but Lee et al. 2018) work within a classification setting.
Nalisnick et al. (2019b) propose solution that can alleviate the issue without modifying existing generative models, but the issue they aim to address (distributional shift) is very different from the standard concerns of OOD sample detection. Their model works by using the likelihood estimates coming from likelihood- them raw OOD sample detectors. The model operates under the assumption that at evaluation time, samples come in batches, and thus can be the inputs of statistical tests differentiating be- tween likelihood estimates for inlier datasets and likelihood estimates for evaluation datasets. In the limiting case where the evaluation dataset has batch-size 1, the performance of this model can be compared meaningfully with our unsupervised models.
models as inputs to detect distributional shift, as opposed to using
6 CONCLUSIONS
In this work, we studied Variational Autoencoder models and investigated to what extent the la- tent representations of data points or the likelihood estimates given by the model can be used to identify out-of-distribution samples. We demonstrated empirically that the extent of the notorious phenomenon of wrong out-of-distribution likelihood estimates present in VAEs is highly depen- dent on the observation model. We introduced negative sampling above weakness of the Variational Autoencoder model family. We presented empirical evidence that utilizing negative samples either from auxiliary dataset from an adversarial training scheme significantly and consistently improves the discriminative power of VAE models regarding out-of- distribution samples.
approach to alleviate the
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Hyunsun Choi, Eric Jang, and Alexander A Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810001392, 2018.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature, 2.18.
Gregory Cohen, Saced Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. 2017 International Joint Conference on Neural Networks (IJCNN), 2017. doi: 10. 2..1009/jemm..017776666
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019. URL https://
openreview hwtp:/ee//ooooe feret//foum?idee wow.mmmiidtyxxxxxicrr...
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann. lecun. com/exdlbnunist, 2, 2010.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibraeed classifiers for
detecting out-of-disttribttion samples. In International Conference on Learning Representations, 2018. URL
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In CoRR, volume abs/1802.05957, 2018
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? In International Conference on Learning Representations, 2019a. URL /tLL:tttp///oeeeeeeeee
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of-distribution inputs to deep generative models using test for typicality. arXiv preprint arXiv:1906.02994, 2019b.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning, Advances in Neural Information Processing Systems (NIPS), 2011.
Mihaela Rosca, Balaji Lakshminarayanan, and Shakir Mohamed. Distribution matching in varia- tional inference. arXiv preprint arXiv: /00.:00022..77 2018.
Alireza Shafaei, Mark Schmidt, and James J Little. Does your model know the digit 6 is not a cat?
less biased evaluation of" outlier" detectors. arXiv preprint trXhv:1809.04729, 2018.
Aäron van den Oord, Nal Kalehbrenner, and Koray Kavukeuoglu. Pixel recurrent neural networks. CoRR, 0/////601067550 2016.
06759
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint
2017.

---

Under review as a conference paper at ICLR 2020
A EXPERIMENTAL DETAILS
A.1 DATASETS AND PREPROCESSING
We conduct experiments with two types of data set: color images of size 32x32 and grayscale images of size 28x28. The utilized datasets are listed below.
Datasets of grayscale images of size 28x28:
MNIST (LeCun al. 2010
28x28x1, 60.000 train + 10.000 test, 10 classes
Fashion-MNIST (Xiao al.
2017
28x28x1, 60.000 train 10.000 test, 10 classes
Kuzushiji-MIST (Clanwwat et al.
2018): 28x28x1, 60.000 train + 10.000 test, 10 classes
EMNIST-Letters (Cohen et al.
2017):
28x28x1, 60.000 train + 10.000 test, 10 classes
Datasets of color images of size 32x32:
CIFAR-I0 (Krizhessky 2009 32x32x3 images, 50.000 train 10.000 test, 10 classes
SVHN (cropped) (Netzer et al., 2011): 32x32x3 images, 73.257 train 26,032 test (+ 531.131 extra unlabeled), classes
Downsampled ImageNet (van den Oord et al. 2016): 32x32x3 images, 1.281.149 train 49.99 .999 validation, 1000 classes
We apply no preprocessing step other than normalizing the input images to [0, 1].
A.2 NETWORK ARCHITECTURE AND TRAINING DETAILS
Details for grayscale images Following Nalisnick al. (011a)). for grayscale images, we use the encoder architecture described in Rosca et al. (2018) in appendix K table 4. Also, as in Rosca et al. (2018), all of the models are trained with the RMSProp optimizer with learning rate set 10-4 We train the models for 100 epochs with mini-batch size of 50. We update the parameters of the encoder and decoder network an alternating fashion.
Details for color images For color images BatchNorm-ReLL. modules for both the encoder and the decoder. The size of the kemels are x 4, and the number of filters 32, 64, 128 for the encoder; and 128, 64, for the decoder. All of the models are trained with the Adam optimizer (3) = 0.9, 82 = 0.999) for 100 epochs with mini-batch size 50. The learning rate is set 10-4 We update the parameters of the encoder and decoder network in an alternating fashion. When generated images are used as negative samples, we employ spectral normalization (Miyato et al., 2018) for the convolutional weights of the encoder in order to stabilize and enhance the performance of the respective models, and in this case the models are trained for 300 epochs.
DCGAN-style CNN architecture with Conv-
10

---

Under review as a conference paper at ICLR 2020
B RECONSTRUCTION OF NEGATIVES
Table 4: Comparing the discriminative power VAE models with negative sampling with Bernoulli noise model, with -NNIST and MNIST inlier and OOD datasets, respectively, when reconstruction of negative samples from EMNIST--etters also taken into account, with a weight. One can observe that the models are able to reconstruct OOD samples, while the discriminative power does not diminish. The same was observed when using Gaussian noise model.
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
AUC BPD
0.999
0.999
0.998
0.998
0.998
0.998
0.998
0.997
0.997
0.998
0.99 (0.01)
Test BPD
0.305
0.304
0.305
0.306
0.307
0.307
0.308
0.308
0.308
0.309
0.31 (0.00)
OOD BPD
1.521
0.681
0.636
0.639
0.641
0.632
0.638
0.634
0.638
0.645
0.59 (0.03)
AUC
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.00 (0.00)
Test KI
15.12
15.73
15.26
15.85
16.38
16.53
16.59
16.66
16.94
17.04
15.86 (0.23)
OOD KL
325.1
325.4
325.5
340.2
350.4
343.0
354.3
350.1
353.6
363.2
321.9 (19.9)
Inlier
OOD
Generated samples
Reconstructed train samples
Reconstructed Reconstructed
Beshion-MNIST
MNIST
test samples
OOD samples
3
7 7
3
4
5
6
9
2
O 6
9
0
0.0
0 3
7
8
7
2
3
3
O
Y
O
7 7 0
9
2 0 3
9
6
0.1
3
8 2 O
7
3
Y
O
7 7
9
0
9
0
6
3
/ 0
9
0.5
4
O
2 1 3 Y
3
7
C
O
O
7
4
2 3 9
7 0 6 / 0 2
A Z
9
9
2
0
4
3 P
9
4 O
1 S
1.0
O 3 7
3
> 0 2
3
0 4
O
i 0
11

---

Under review as a conference paper at ICLR 2020
C INCREASING THE LATENT DIMENSION
Table 5: Comparing the discriminative performance of baseline VAE models with different latent dimension sizes, trained Fashion-MNIST, and MNIST used as OOD dataset, with Bernoulli noise model. First column corresponds to our default setup. Reconstructed training samples and generated samples from the models are also provided. Our exploratory experiments indicate that simply increasing the latent dimension size does not help to overcome the problem of assigning higher likelihoods OOD data, and even the generative performance is diminishing.
Latent dimension
10
50
100
250
500
AUC BPD
0.46 (0.05)
0.35
0.35
0.39
0.34
AUC KL
0.61 (0.09)
0.76
0.73
0.42
0.64
Test BPD
0.30 (0.00)
0.31
0.32
0.33
0.34
OOD BPD
0.35 (0.08)
0.27
0.27
0.29
0.29
Test KL
15.61 (0.55)
16.06
16. 14
18.02
17.65
OOD KL
31.91 (16.89)
19.37
18.70
17.43
18.75
Reconstruction
Generated samples
12

---

Under review as a conference paper at ICLR 2020
D GENERATED SAMPLES
Table 6: Generated samples from models trained on grayscale and color images.
Trained on
VAE with negative sampling
VAE with negative sampling
Bascline VAE
Rashion-MNIST
negative:
EMNIST-Leetters
negative: adversarial
Bernoulli
Gaussian
Bernoulli
Gaussian
Bernoulli
Gaussian
Trained on MNIST
VAE with negative sampling
VAE with negative sampling
Baseline VAE
negative: EMNIST-Letters
negative: adversarial
Bernoulli
Gaussian
Bernoulli
Gaussian
Bernoulli
Gaussian
6
2 3
7
2
S 5
7
3
2
2 9
2
2 6 3
0 / 6
4 0 6 4
9
/
2
6
8
f
4 7 ~ 3 4 2 0 5 1
2
6
9
a
9 6 6
8 3
2
0
4 3
6
{ 9
5
6
9
3
9
5
l 6 7
J 2 ]
2
9 3
3
6
1
7
6
2
6 /
5
3
99 0
8
9
0
1
6
3
9,
2
3
/ 3 0
1
9
7
0 7
9
4 /
w
/
4
9
6
9
9
2
8
0
9
5
9
0
O
5
4
8
7
8
4
0
2 2
5
7
6
7
3
/ 6
9 7
5
6
9
9
8 4 /
7 7
4 8 2
3
3
4
0
7
2
4
9
2
0
Q 4
3
9
7
6
[
3
9
8
3
6
3
7
4
6
9
6
f
5
7
8
8
2
1
7
5
8 1
7
8
5
7
0
3
9
0
0
4
1
O
3
5
7
6
0
7
/
0
7
4
0
9 5
O 8
Trained
VAE with negative sampling
VAE with negative sampling
Baseline VAE
CIFAR-10
negative: Ds. ImageNet
negative: adversarial
Bernoulli
Gaussian
Bernoulli
Gaussian
Bernoulli
Gaussian
Trained
VAE with negative sampling
VAE with negative sampling
Baseline VAE
SVHN
negative: Ds. ImageNet
negative: adversarial
Bernoulli
Gaussian
Beroulli
Gaussian
Bernouli
Gaussian
5 58
6
5
62
43( 48
22s (6
124 22
2:30 50r
6
10
12 4188
4504
1
18113107
13

---

Under review as a conference paper at ICLR 2020
E PLOTS FROM THE LATENT SPACE
Table First two coordinates of the latent space of baseline VAE and VAE with negative sampling, with Bernoulli noise model, trained
and MNIST used as OOD dataset.
Epoch
Epoch
Epoch 100
Marrt ci1ii
Baseline VAE
Epoch
Epoch 10
Epoch 100
VAE with negative sampling
negative: EMNIST-L.c MINIST-Leetters Letters
(01 latonn corpeient
Bbrug calorn reryeilentted
Epoch
Epoch 10
Epoch 100
Antervrillyy (ee.sintieeeeeee
VAE with negative sampling, negative: adversarial generated
Epoch 45
Epoch 95
Epoch 100
Atweerrt pencened
14


Reviews:
[{"review_id": "S1leviI0YB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose augmenting VAEs with an additional latent variable to allow them to detect out-of-distribution (OOD) data. They propose several measures based on this model to distinguish between inliers and outliers, and evaluate the model empirically, finding it successful.\n\nUnfortunately, the method in this paper is developed unclearly and incorrectly. Although their experiments are somewhat successful, the problems with the text and method are severe enough to justify rejection.\n\nSpecifically, the authors' method proposes adding a term to the loss of the VAE that encourages the variational posterior (q) to distribute latent codes (z) for inliers and outliers differently. The equation which defines their new objective is unclear -- specifically, it is not clear whether the added KL term is computed for inliers and outliers both, or whether it is only computed for outliers. If it is the former, then the method does not make sense. If it is the latter, then the equation is incorrect or at the very least not clear in the extreme.\n\nFurthermore, the term is added without consideration of whether or not the method is still optimizing a sensible variational lower bound. The authors attempt to justify the objective by writing out a variational lower bound for a VAE with a mixture prior where inliers and outliers are generated from different mixture components. However, their equations are incorrect -- the equation that is called the log likelihood is not the log likelihood, and the ELBO is similarly wrong.\n\nTheir empirical evaluation is reasonable, although the measures they propose to distinguish between inliers and outliers (i.e. the kl from the approximate posterior to the prior) is not thoroughly justified.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gQtiyvtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses the detection of out-of-distribution (OOD) samples for variational autoencoders (VAE).\nThe idea is to train the encoder such that its output variational distribution q(z|\\bar{x}) is pushed away from the prior of latent z. \nI think the paper needs more clarification and investigation for being published in the conference. \nMy major concern is that more empirical investigation is necessary since the formulation provides a minor novelty. \nSpecific points are given below. \n\n1) Weak novelty in terms of model design. \nThe objective function consists of the standard (negative) ELBO term and additional KL term to modify the variational posterior of negative samples. \nThis modification can be regarded as a form of outlier exposure (Hendrycks et al. 2018) specialized for VAE. \nThe choice of \\bar{p} is not much investigated. \nAny discussion if we use a more sophisticated model such as VampPrior* for stronger modeling capacity. \n* J. Tomczak and M. Welling, VAE with a VampPrior, AISTATS 2018.\n\n2) The use generated samples as negative samples is interesting but mysterious. \nThe authors conjecture that this works because the generated samples come from near the data manifold, but in-distribution samples and negative samples can be indistinguishable when the generative model is very well trained. \nWhat happens if, for example, the negative samples are generated by data augmentation techniques (such as cropping, rotation, mirroring, though mirroring and much rotation may be unsuitable for text images)? \nThis can also produce near-manifold points. \nA deeper analysis why generated samples can improve the OOD detection performance is necessary. \nFurthermore, why does not this approach impact much for color images in Table 4?\n\n3) More details of experimental procedures. \n3-1) How was data points are generated from VAE as negative samples? \nPossible ways are:\n* sample z ~ p(z), then draw from the decoder x ~ p(x|z).\n* use negative prior z ~ \\bar{p}(z), then draw from the decoder x ~ p(x|z).\n* this seems weird: use variational posterior z ~ q(z|x), then x ~ p(x|z).\n\n3-2) Latent dimension of 10 for grayscale images seems small. \nDoes the size affect the OOD detection performance when the size is 50 or 100 to make the model richer. \n\n3-3) How was the variance obtained when the decoder uses the Gaussian likelihood?\n* fixed value?\n* learned for each pixel?\n* output from the decoder?\n\n4) If we have access to diverse negative datasets, can the ODD detection perform better? \nMixing multiple datasets or using both available dataset and generated samples can improve the performance while the test OOD samples are kept unseen. \nFor example, train VAE on MNIST while using KMNIST and EMNIST as the negative sets to detect Fashion-MNIST as ODD. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BkgalbL3FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to counteract OOD problem in VAE by adding a regularization term to the ELBO. The regularizer is defined as the Kullback-Leibler divergence between a variational posterior for a negative sample and a marginal distribution over latents for negative data. The authors present experiments on MNIST and MNIST-like datasets, and CIFAR10 with SVHN. Unfortunately, I do not find the paper especially interesting. The motivation for adding the regularization term is not convincing. The experiments are insufficiently discussed.\n\nRemarks:\n- The paper proposes to ad a regularization to ELBO, namely, the Kullback-Leibler divergence between a variational distribution for a negative sample and a marginal distribution over latent variables for negative samples. I do not fully understand the motivation given on page 3. The authors show that including the negative data yields a new objective that is a sum of two log-lihelihood functions for \"real\" and negative data. However, later they propose to skip a (negative) reconstruction error term for the negative data. As a result, the authors obtain the objective they proposed. This explanation is very vague and I do not see what it adds to the story. Contrary, it causes new questions about their model and whether it is properly formulated.\nI suggest to look into the following paper to see whether the model could be re-formulated:\nHu, Z., Yang, Z., Salakhutdinov, R., & Xing, E. P. (2017). On unifying deep generative models. arXiv preprint arXiv:1706.00550.\n\n- I do not understand why the authors used Bernoulli distribution to model color and gray-scale images. The Bernoulli distribution could be used only for binary random variables. This is obviously flawed.\n\n- In general, the results seem to partially confirm claims of the paper, however, they are quite vague. First, utilizing a wrong distribution is demotivating (see my previous remark). Second, I miss a better description of models and, in general, experiments' setup. Third, all results are explained in a laconic manner (e.g., \"The other results in the table (...) confirm the assymetric behaviour of the phenomenon (...)\"). There is neither deeper understanding nor discussion provided.\n\n- Why there are no samples for CIFAR or SVHN provided?\n\n======== AFTER REBUTTAL ========\nI would like to thank the authors for their rebuttal. I really appreciate that the paper is updated and some concerns are solved. After reading the updated paper again, I tend to agree that the proposed idea is interesting for the problem of OOD detection using generative models. Therefore, I decide to update my score.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to improve VAEs' modeling of out-of-distribution examples, by pushing the latent representations of negative examples away from the prior.  The general idea seems interesting, at least to some of the reviewers and to me.  However, the paper seems premature, even after revision, as it leaves unclear some of the justification and analysis of the approach, especially in the fully unsupervised case.  I think that with some more work it could be a very compelling contribution to a future conference.
Decision: Reject