From Variational to Deterministic Autoencoders
Authors: Partha Ghosh
Mehdi S. M. Sajjadi
Antonio Vergari
Michael Black
Bernhard Scholkopf
Venue:  (2020)
Keywords: Unsupervised learning
Generative Models
Variational Autoencoders
Regularization

Abstract:  Variational Autoencoders (VAEs) provide a theoretically-backed and popular framework for deep generative models. However, learning a VAE from data poses still unanswered theoretical questions and considerable practical challenges. In this work, we propose an alternative framework for generative modeling that is simpler, easier to train, and deterministic, yet has many of the advantages of the VAE. We observe that sampling a stochastic encoder in a Gaussian VAE can be interpreted as simply injecting noise into the input of a deterministic decoder. We investigate how substituting this kind of stochasticity, with other explicit and implicit regularization schemes, can lead to an equally smooth and meaningful latent space without having to force it to conform to an arbitrarily chosen prior. To retrieve a generative mechanism to sample new data points, we introduce an ex-post density estimation step that can be readily applied to the proposed framework as well as existing VAEs, improving their sample quality. We show, in a rigorous empirical study, that the proposed regularized deterministic autoencoders are able to generate samples that are comparable to, or better than, those of VAEs and more powerful alternatives when applied to images as well as to structured data such as molecules. 

Published as a conference paper at ICLR 2020
FROM VARIATIONAL
TO DETERMINISTIC AUTOENCODERS
Partha Ghoshi"
Mehdi S. M. Sajjadij
Antonio Vergarij
Michael Black
Bernhard Schölkopfi
i Max Planck Institute for Intelligent Systems, Tübingen, Germany fpghosh, msaj j black, futue .mp. de
$ University of California, Los Angeles, USA averêcs. ucla.edu
ABSTRACT
Variational Autoencoders (VAEs) provide theoretically-backed and popular framework for deep generative models. However, learning a VAE from data poses still unanswered theoretical questions and considerable practical challenges. this work, we propose an alternative framework for generative modeling that is simpler, easier train, and deterministic, yet has many of the advantages VAFs. We observe that sampling stochastic encoder in Gaussian VAE can be inter- preted as simply injecting noise into the input of deterministic decoder. We investigate how substituting this kind of stochasticity, with other explicit and im- plicit regularization schemes, can lead to an equally smooth and meaningful latent space without forcing it to conform to an arbitrarily chosen prior. To retrieve generative mechanism to sample new data, we introduce an ex-post density es- timation step that can be readily applied also existing VAEs, improving their sample quality. We show. in rigorous empirical study, that the proposed regular- ized deterministic autoencoders are able to generate samples that are comparable to, or better than, those of VAEs and more powerful alternatives when applied to images as well as structured data such as molecules.
INTRODUCTION
Generative models lie at the core of machine learning. By capturing the mechanisms behind the data generation process, one can reason about data probabilistically, access and traverse the low- dimensional manifold the data is assumed to live and ultimately generate fore not surprising that generative models have gained momentum in applications such as computer vision (Sohn et al., 2015; Brock al., 2019), NLP (Bowman et al., 2016; Severyn et al., 2017), and chemistry (Kusner et al., 2017; Jin et al., 2018; Gómez-Bombarede et al., 2018).
data. It is there-
Variational Autoencoders (VAEs) (Kingma & Welling, 2014; Rezende al., 2014) cast learning rep- resentations for high-dimensional distributions as a variational inference problem. Learning a VAE amounts the optimization of an objective balancing the quality of samples that autoencoded through a stochastic encoder-decoder pair while encouraging the latent space to follow fixed prior distribution. Since their introduction, VAEs have become one of the frameworks of choice among the different generative models. VAEs promise theoretically well-founded and more stable training than Generative Adversarial Networks (GANs) (Goodfellow al., 2014) and more efficient sam- pling mechanisms than autoregressive models (Larochelle & Murray, 2011; Germain et al., 2015).
However, the VAE framework still far from delivering the promised generative mechanism, as there are several practical and theoretical challenges yet to be solved. A major weakness of VAEs is
Equul contribution.
An implementation is available at: autteencoder rs- RAK-

---

Published as a conference paper at ICLR 2020
the tendency to strike unsatisfying compromise between sample quality and reconstruction qual- ity. In practice, this has been attributed to overly simplistic prior distributions (Tomczak & Welling, 2018: Dai & Wipf, 2019) or alternatively, to the inherent over-regularization induced by the KL divergence term in the VAE objective (Tolstikhin et al., 2017). Most importantly, the VAE objective itself poses several challenges as it admits trivial solutions that decouple the latent space from the input (Chen et al., 2017: Zhao et al., 2017), leading to the posterior collapse phenomenon in conjunc- tion with powerful decoders (van den Oord et al., 2017). Furthermore, due to its variational formula- tion, training a VAE requires approximating expectations through sampling at the cost of increased variance in gradients (Burda et al., 2015; Tucker et al., 2017), making initialization, validation, and annealing of hyperparameters essential in practice (Bowman et al., 2016; Higgins et al., 2017; Bauer & Mnih, 2019). Lastly, even after a satisfactory convergence of the objective, the learned aggregated posterior distribution rarely matches the assumed latent prior in practice (Kingma et al., 2016; Bauer & Mnih, 2019; Dai & Wipf, 2019), ultimately hurting the quality of generated samples. All in all, much of the attention around VAEs is still directed towards "fixing" the aforementioned drawbacks associated with them.
In this work, we take different route: we question whether the variational framework adopted by VAEs is necessary for generative modeling and, in particular, to obtain a smooth latent space. We propose to adopt simpler, deterministic version of VAEs that scales better, is simpler to optimize, and, most importantly, still produces a meaningful latent space and equivalently good or better sam- ples than VAEs or stronger alternatives, c.g., Wasserstein Autoencoders (WAEs) (Tolstikhin al., 2017). We do so by observing that, under commonly used distributional assumptions, training a stochastic encoder-decoder pair in VAEs does not differ from training a deterministic architecture where noise is added to the decoder's input. We investigate how to substitute this noise injection mechanism with other regularization schemes in the proposed deterministic Regularized Autoen- coders (RAEs), and we thoroughly analyze how this affects performance. Finally, we equip RAEs with generative mechanism via simple ex-post density estimation step on the learned latent space.
In summary, our contributions are as follows: i) we introduce the RAE framework for generative modeling as a drop-in replacement for many common VAE architectures; ii) we propose an ex-post density estimation scheme which greatly improves sample quality for VAEs, WAEs and RAEs with- out the need to retrain the models; iii) we conduct a rigorous empirical evaluation to compare RAEs with VAEs and several baselines standard image datasets and on more challenging structured domains such as molecule generation (Kusner et al., 2017; Bombarlll et al., 2018).
2 VARIATIONAL AUTOENCODERS
For general discussion, we consider a collection of high-dimensional i.i.d. samples X {xx;; ) drawn from the true data distribution (x) over a random variable X taking values in the input space. The aim of generative modeling is to learn from X' a mechanism to draw new samples Xnew Pdata- Variational Autoencoders provide powerful latent variable framework to infer such a mechanism. The generative process of the VAE is defined as
Znew p pZ), Nnew ~ po X Z= knew)
where p(Z) is a fixed prior distribution over a low-dimensional latent space Z. A stochastic decoder
Dø(z)
p6(x z2) p(X (z))
(2)
links the latent space to the input space through the likelihood distribution po, where go is an expres- sive non-linear function parameterized by 0.2 As a result, a VAE estimates pdata(x) as the infinite
mixture model (x()) f pe (x z)p(z)dz. At the same time, the input space mapped the latent
space via a stochastic encoder
(x) = z~
2 (z x) = 100000((220 (Z(((x)))
(3)
where (z x) is the posterior distribution given by a second function fs parameterized by Φ. Computing the marginal log-likelihood log (x() is generally intractable. One therefore follows variational approach, maximizing the evidence lower bound (ELBO) for a sample x:
log xx) 2 ELBO( 6, x) Ex x) log Po (x 2) (gx x))))())
(4)
With slight abuse of notation, use lowercase letters for both random variables and their realizations, c.g. (x [z) instead of p(X IZ z)., when it is clear to discriminate hetween the two.
2

---

Published as a conference paper at ICLR 2020
Maximizing Eq. 4 over data X' w.r.t. model parameters o, 8 corresponds to minimizing the loss
arg min Exc-puas
LELBO = Expppss PREC + LKL
(5)
0.0
where LREC and LKL are defined for a sample x as follows:
RECC - -ozzzgg x) log po 2))
LKL (1o ((|x))]l(()))
(6)
Intuitively, the reconstruction loss PREC takes into account the quality of autoencoded samples through (E(E'( (x)). while the KL-divergence term LKL encourages 4s x) to match the prior p(z) for each z which acts regularizer during training (Hoffman & Johnson, 2016).
2.1 PRACTICE AND SHORTCOMINGS OF VAES
To fit VAE to data through Eq. 5 one has specify the parametric forms for p(z). g0 x), Po(x z), and hence the deterministic mappings fa and go- In practice, the choice for the above distributions is guided by trading off computational complexity with model expressiveness. In the most commonly adopted formulation of the VAE, (o(z x) and po(x 2) are assumed to be Gaussian:
Ex(x)
(x(Ex(x)) A A (z),,, diag(oo(z)))
with means Ma. p40 and covariance parameters 0a. given by So and go. In practice, the covariance of the decoder is set to the identity matrix for all z, i.c., (z = 1 (Dai Wipf, 2019). The expec- tation of LREC in Eq. 6 must be approximated via k Monte Carlo point estimates. It is expected that the quality of the Monte Carlo estimate, and hence convergence during learning and sample quality increases for larger (Burda al., 2015). However, only 1-sample approximation is generally carried out (Kingma& & Welling, 2014) since memory and time requirements are prohibitive for large k. With the I-sample approximation, LREC can be computed the mean squared error between input samples and their mean reconstructions ss0 by decoder that is deterministic in practice:
CREC = ||x no (x)
Gradients w.r.t. the encoder parameters & are computed through the expectation of LrEC in Eq. via the reparametrization trick (Kingma & Welling, 2014) where the stochasticity of E.g. is relegated to an auxiliary random variable which does not depend d:
Fx(() = (x) + (x) O 6, 6 (~N(0.1)
where denotes the Hadamard product. An additional simplifying assumption involves fixing the prior p(z) to be a d-dimensional isotropic Gaussian AZ 0, I). For this choice, the KL-divergence for sample x is given in closed form: 2CKL
(x) + d Σ ox(x): log a (x);
While the above assumptions make VAEs easy to implement, the stochasticity in the encoder and decoder still problematic in practice (Makhzani et al., 2016; Tolstikhin al., 2017; Dai & Wipf, 2019). In particular, one has to carefully balance the trade-off between the LKL term and LREC during optimization (Dai & Wipf, 2019; Bauer & Mnih, 2019). too-large weight on the CKL term can dominate LELBO, having the effect of over-regularization. As this would smooth the latent space, it can directly affect sample quality in a negative way. Heuristics avoid this include manually fine- tuning or gradually annealing the importance of LKL during training (Bowman et al., 2016; Bauer & Mnih, 2019). We also observe this trade-off in practical experiment Appendix
Even after employing the full array of approximations and "tricks" to reach convergence of Eq. 5 for a satisfactory set of parameters, there no guarantee that the learned latent space is distributed according to the assumed prior distribution. In other words, the aggregated posterior distribution (z) EEx-par q(2 [z]x x) has been shown not to conform well to p(z) after training (Tolstikhin et al., 2017; Bauer & Mnih, 2019; Dai & Wipf, 2019). This critical issue severely hinders the generative mechanism of VAEs (cf. Eq. 1) since latent codes sampled from p(z) (instead of q(z)) might lead to regions of the latent space that previously unseen to D3 during training. This results in generating nut-of-distribution samples. We refer the reader to Appendix for a visual demonstration this phenomenon the latent space of VAEs. We analyze solutions this problem in Section 4.
3

---

Published as a conference paper at ICLR 2020
2.2 CONSTANT- VARIANCE ENCODERS
Before introducing our fully-deterministic take on VAES, it is worth investigating intermediate fla- vors of VAEs with reduced stochasticity. Analogous to what is commonly done for decoders as discussed in the previous section, one can fix the variance of gø(2 x) to be constant for all x. This simplifies the computation of Eg from Eq. 9
FCC (x) = Ma (x) +
<~ <~~ N0,N oI)
(10)
where a is a fixed scalar. Then, the KL loss term in Gaussian VAE simplifies (up to a constant)
to CRY = [x)=(x)] [|. We name this variant Constant- Variance VAEs (CV-VAEs). While CV-VAEs
have been adopted in some applications such variational image compression (Ballé et al., 2017) and adversarial robustness (Ghosh et al., 2019), to the best of our knowledge, there is no systematic study of them in the literature. We will fill this gap in experiments in Section 6. Lastly, note that now a in Eq.10 10 is not learned along the encoder as in Eq. 9. Nevertheless, it can still be fitted as an hyperparameter, e.g., by velidavai,,,,,,, to maximise the model likelihood. This highlights the possibility to estimate a better parametric form for the latent space distribution after training, or in a outer-loop including training. We address this provide a more complex and flexible solution to deal with the prior structure over Z via ex ex-post density estimation in Section 4.
3 DETERMINISTIC REGULARIZED AUTOENCODERS
Autoencoding in VAEs is defined in a probabilistic fashion: Es and Dg map data points not a single point, but rather to parameterized distributions (cf. Eq. 7). However, common implementa- tions of VAFs as discussed in Section 2 admit a simpler, deterministic view for this probabilistic mechanism. A glance at the autoencoding mechanism of the VAE is revealing.
The encoder deterministically maps data point x to mean (x(x) and variance (x) in the latent space. The input to D3 is then simply the mean na (x) augmented with Gaussian noise scaled by (x) via the reparametrization trick (cf. Eq. 9). the CV-VAE, this relationship is even more obvious, the magnitude of the noise is fixed for all data points (ef. Eq. 10). In this light, VAE can be seen as a deterministic autoencoder where (Gaussian) noise is added to the decoder's input.
We argue that this noise injection mechanism is key factor in having a regularized decoder. Using random noise injection to regularize neural networks is a well-known technique that dates back sev- eral decades (Sietsma & Dow, 1991; An, 1996). It implicitly helps to smooth the function learned by the network at the price of increased variance in the gradients during training. In turn, decoder regularization is key component in generalization for VAEs, it improves random sample quality and achieves a smoother latent space. Indeed, from a generative perspective, regularization is moti- vated by the goal to learn a smooth latent space where similar data points x are mapped to similar latent codes z, and small variations in Z lead to reconstructions by D, that vary only slightly.
We propose to substitute noise injection with an explicit regularization scheme for the decoder. This entails the substitution of the variational framework in VAEs, which enforces regularization on the encoder posterior through LKL, with deterministic framework that applies other flavors of decoder regularization. By removing noise injection from a VA-, we are effectively left with a deterministic autoencoder (AE). Coupled with explicit regularization for the decoder, we obtain a Regularized Autoencoder (RAE). Training RAE thus involves minimizing the simplified loss
LRAE = CREC + B + >LREG
(11)
where CRGG represents the explicit regularizer for Dg (discussed in Section 3.1) and C
1/2|||| (resulting from simplifying <FV is equivalent to constraining the size of the learned la- tent space, which is still needed to prevent unbounded optimization. Finally, B and À are two hyper parameters that balance the different loss terms.
Note that for RAES, no Monte Carlo approximation is required to compute PREC. This relieves the need for more samples from (z x) to achieve better image quality (cf. Appendix A). Moreover, by abandoning the variational framework and the LKL term, there is no need in RAEs for fixed prior distribution over Z. Doing so however loses a clear generative mechanism for RAEs to sample from Z. We propose method to regain random sampling ability in Section 4 by performing density estimation on ex- ex-post, a step that is otherwise still needed for VAEs to alleviate the posterior mismatch issue.
4

---

Published as a conference paper at ICLR 2020
3.1 REGULARIZATION SCHEMES FOR RAES
Among possible choices for CREG. first obvious candidate is Tikhonov regularization (Tikhonov & Arsenin, 1977) since is known to be related to the addition of low-magnitude input noise (Bishop. 2006). Training a RAE within this framework thus amounts to adopting LreG Li, - =1012 which effectively applies weight decay on the decoder parameters
Another option comes from the recent GAN literature where regularization is hot topic (Kurach ct al., 2018) and where injecting noise to the input of the adversarial discriminator has led to im- proved performance in technique called instance noise (Sønderby et al., 2017). To enforce Lips- chitz continuity on adversarial discriminators, weight clipping has been proposed (Arjovsky et al., 2017), which is however known to significantly slow down training. More successfully, a gradient penalty on the discriminator can be used similar to Guirajani et al. (2017); Mescheder et al. (2018), yielding the objective LreG LGP (Ea((()x)) which bounds the gradient norm of the decoder w.r.t. its input.
Additionally, spectral normalization (SN) has been successfully proposed as an alternative way to bound the Lipschitz norm of an adversarial discriminator (Miyato et al., 2018). SN normalizes each weight matrix Og in the decoder by an estimate of its largest singular value: 0SN 0t/s(0e) where s(Oe) is the current estimate obtained through the power method.
In light of the recent successes of deep networks without explicit regularization (Zagoruyko Ko- modakis, 2016; Zhang et al., 2017), it is intriguing to question the need for explicit regularization of the decoder in order to obtain a meaningful latent space. The assumption here is that techniques such as dropout (Srivastava et al., 2014), batch normalization (loffe & Szegedy, 2015), adding noise during training (An, 1996) implicitly regularize the networks enough. Therefore, as a natural base- line to the LRAE objectives introduced above, we also consider the RAE framework without LrEG and CRAE, i.., a standard deterministic autoencoder optimizing CREC only.
To complete "autopsy" of the VAE loss, additionally investigate deterministic autoencoders with decoder regularization, but without the CRAE term, as well as possible combinations different regularizers in our RAF framework (cf. Table 3 in Appendix I).
Lastly, it is worth questioning it is possible to formally derive our RAE framework from first principles. We answer this affirmatively, and show how to augment the ELBO optimization problem of a VAE with an explicit constraint, while not fixing parametric form for qo (z x). This indeed leads to a special case of the RAE loss in Eq. 11. Specifically, we derive a regularizer like LGp for a deterministic version of the CV-VAE. Note that this derivation legitimates bounding the decoder's gradients and as such justifies the spectral norm regularizer as well since the latter enforces the decoder's 's Lipschitzness. We accommodate the full derivation in Appendix B.
4 EX-PoST DENSITY ESTIMATION
By removing stochasticity and ultimately. the KL divergence term LKL from RAEs, we have sim- plified the original VAE objective at the cost of detaching the encoder from the prior p(z) over the latent space. This implies that i) we cannot ensure that the latent space Z is distributed according to simple distribution (e.g.. isotropic Gaussian) anymore and consequently, ii) we lose the simple mechanism provided by p(z) to sample from Z as in Eq. 1.
As discussed in Section 2.1. issue i) is compromising the VAE framework in any case, as reported in several works (Hoffman & Johnson, 2016; Rosca et al., 2018; Dai & Wipf, 2019). To fix this, some works extend the VAE objective by encouraging the aggregated posterior to match p(z) (Tolstikhin et al., 2017) by utilizing more complex priors (Kingma et al., 2016; Tomezak & Welling, 2018; Bauer & Mnih, 2019).
To overcome both i) and ii), we instead propose to employ ex-post density estimation over Z. We fit a density estimator denoted as q 2) to {z = (x) E X). This simple approach not only fits our RAE framework well, but it can also be readily adopted for any VAE variants thereof such as the WAE as a practical remedy to the aggregated posterior mismatch without adding any computational overhead to the costly training phase.
5

---

Published as a conference paper at ICLR 2020
The choice of qá (z) needs to trade-off expressiveness to provide good fit of arbitrary space for Z with simplicity, to improve generalization. For example, placing a Dirac distribution on each latent point would allow the decoder to output only training sample reconstructions which have high quality, but do not generalize. Striving for simplicity, we employ and compare a full covariance multivariate Gaussian with 10-component Gaussian mixture model (GMM) in our experiments.
RELATED WORKS
Many works have focused on diagnosing the VAF framework, the terms in its objective (Hoffman & Johnson, 2016; Zhao al., 2017: Alemi et al., 2018), and ultimately augmenting it to solve optimization issues (Rezende & Viola, 2018; Dai & Wipf, 2019). With RAE, we argue that simpler deterministic framework can be competitive for generative modeling.
Deterministic denoising (Vincent et al., 2008) and contractive autoencoders (CAEs) (Rifai et al., 2011) have received attention the past for their ability to capture smooth data manifold. Heuristic attempts to equip them with a generative mechanism include MCMC schemes (Rifai et al., 2012; Bengio et al., 2013). However, they are hard to diagnose for convergence, require considerable effort in tuning (Cowles & Carlin, 1996), and have not scaled beyond MNIST, leading to them being superseded by VAEs. While computing the Jacobian for CAEs (Rifai et al., 2011) is close in spirit to CGP for RAEs, the latter is much more computationally efficient.
Approaches to cope with the aggregated posterior mismatch involve fixing a more expressive form for p(z) (Kingma et al., 2016; Bauer & Mnih, 2019) therefore altering the VAE objective and quiring considerable additional computational efforts. Estimating the latent space of a VAE with a second VAE (Dai & Wipf, 2019) reintroduces many of the optimization shortcomings discussed for VAEs and is much more expensive in practice compared to fitting a simple (z) after training.
Adversarial Autoencoders (AAE) (Makhzani et al., 2016) add discriminator encoder-decoder pair, leading to sharper samples at the expense of higher computational overhead and the introduction of instabilities caused by the adversarial nature of the training process.
deterministic
RECONSTRUCTIONS
RANDOM SAMPLES
INTERPOLATIONS
GT
VAE
CV-VAE
WAE
2s
RAE-GP
RAE-L2
RAE-SN
RAE
AE
Figure I: Qualitative evaluation of sample quality for VAEs, WAEs, 2sVAEs, and RAEs on CelebA. RAE provides slightly sharper samples and reconstructions while interpolating smoothly in the latent space. Corresponding qualitative overviews for MNIST and CIFAR-10 are provided in Appendix F.
Wasserstein Autoencoders (WAE) (Tolstikhin et al., 2017) have been introduced as a generaliza- tion of AAEs by casting autoencoding as optimal transport (OT) problem. Both stochastic and deterministic models can be trained by minimizing a relaxed OT cost function employing either an adversarial loss term or the maximum mean discrepancy score between p(z) and (z) reg-

---

Published as a conference paper at ICLR 2020
ularizer in place of CKL- Within the RAE framework, we look at this problem from different perspective: instead of explicitly imposing a simple structure on Z that might impair the ability to fit high-dimensional data during training, propose to model the latent space by ex-post density estimation step.
The most successful VAE architectures for images and audio so far are variations of the VQ- VAE (van den Oord et al., 2017; Razavi et al., 2019). Despite the name, VQ- VAES are neither similar RAEs
stochastic, nor variational, but they are deterministic autoencoders. VQ-VAEs in that they adopt ex-post density estimation. However, VQ-VAEs necessitates complex discrete autoregressive density estimators and a training loss that is non-differentiable due quantizing Z.
Lastly, RAEs share some similarities with GLO (?). However, differently from RAEs, GLO can be interpreted as a deterministic AE without and encoder, and when the latent space built "on- demand" by optimization. On the other hand, RAEs augment deterministic decoders as in GANs with deterministic encoders.
6 EXPERIMENTS
Our experiments designed to answer the following questions: Q1: Are sample quality and latent space structure in RAES comparable to VAEs? Q2: How do different regularizations impact RAE performance? Q3: What is the effect of ex-post density estimation on VAEs and its variants?
CIFAR
CELEBA
SAMPLES
SAMPLES
SAMPLES
REC.
REC.
REC.
GMM
Interp.
N
GMM
Interp.
NN
GMM
Interp.
VAE
18.26
19.21
17.66
18.21
57.94
106.
103.78
88 88.62
39.12
48.12
45.52
44.49
CV-VAE
15.15
33.79
17.87
25.12
WAE
10.03
20.42
9.39
14.34
35.97
117.44 .44
93.53
76.89
34.81
53.67
42.73
40.93
2sVAE
20.31
18.81
18.35
62.54
109.77
49.70
RAE GP
14.04
22.21
11.54
15.32
32.17
83.05
76.33
64. 08
39.71
116.30
45.63
47.00
RAE-L2
10.53
22.22
8.69
14.54
32.24
80.80
74.16
62.54
43.52
51.13
47.97
45.98
RAE-SN
15.65
19.67
11.74
15.15
27.61
84.25
75.30
63.62
36.01
44.74
40.95
39.53
11.67
23.92
14.67
29.
83.87
76.28
48.20
AE
12.95
58.73
10.66
17.12
30.52
84.74
76.47
61.57
40.79
127.85
45.10
50.94
AB-L2
11.19
315.15
9.36
17.15
34.35
247.48 48
75.40
61.09
44.72
346.29
48.42
56.16
Table 1: Evaluation of all models by FID (lower is better, best models in bold). We evaluate each model by REC.: test sample reconstruction; M: random samples generated according to the prior distribution p(z) (isotropic Gaussian for VAE / WAE, another VAE for 2SVAE) or by fitting Gaus- sian to qæ(z) (for the remaining models); GMM: random samples generated by fitting mixture of 10 Gaussians in the latent space; Interp.: mid-point interpolation between random pairs of test reconstructions. The RAE models are competitive with or outperform previous models throughout the evaluation. Interestingly, interpolations do not suffer from the lack of explicit priors on the latent space in our models.
6.1 RAES FOR IMAGE MODELING
We evaluate all regularization schemes from Section 3.1: RAE-GP, RAE-L2, and RAE-SN. For a thorough ablation study, we also consider only adding the latent code regularizer CRAE to LREC (RAE), and an autoencoder without any explicit regularization (AE). We check the effect of applying one regularization scheme while not including the LBA term in the AE-L2 model.
As baselines, employ the regular VAE, constant-variance VAE (CV-VAB), Wasserstein Au- toencoder (WAE) with the MMD loss as a state-of-the-at method, and the recent 2-stage VAE (2sVAE) (Dai & Wipf, 2019) which performs a form of ex-post density estimation via another VAE. For a fair comparison, we use the same network architecture for all models. Further details about the architecture and training given in Appendix C.
We measure the following quantities: held-out sample reconstruction quality, random sample qual- ity, and interpolation quality. While reconstructions give us lower bound on the best quality
7

---

Published as a conference paper at ICLR 2020
achievable by the generative model, random sample quality indicates how well the model gener- alizes. Finally, interpolation quality sheds light on the structure of the learned latent space. The evaluation of generative models is nontrivial research question (Theis et al., 2016; Sajjadi et al., 2017: Lucic et al., 2018). We report here the ubiquitous Fréchet Inception Distance (FID) (Heusel et al., 2017) and we provide precision and recall scores (PRD) (Sajjadi et al., 2018) in Appendix E.
Table summarizes our main results. All of the proposed RAE variants are competitive with the VAE, WAE and 2s w.r.t. generated image quality in all settings. Sampling RAEs achieve the best FIDs across all datasets when a modest 10-component GMM is employed for ex-post density estimation. Furthermore, even when N is considered as qs (z), RAEs rank first with the exception of MNIST. where it competes for the second position with a VAE. Our best RAE FIDs are lower than the best results reported for VA in the large scale comparison of (Lucic et al., 2018), challenging even the best scores reported for GANs. While we are employing a slightly different architecture than theirs, our models underwent only modest finetuning instead of an extensive hyperparameter search. A comparison of the different regularization schemes for RAEs (Q2) yields no clear winner across all settings as all perform equally well. Striving for a simpler implementation, one may prefer RAE-L2 over the GP and SN variants.
For completeness, we investigate applying multiple regularization schemes to our RAE models. We report the results of all possible combinations in Table 3, Appendix I. There, no significant boost of performance can be spotted when comparing to singly regularized RAES.
Surprisingly, the implicitly regularized RAE and AE models are shown be able to score impressive FIDs when go (z) is fit through GMMs. FIDs for AEs decrease from 58.73 to 10.66 on MNIST and from 127.85 to 45.10 on CelebA value close the state of the art. This is a remarkable result that follows a long series of recent confirmations that neural networks are surprisingly smooth by design (Neyshabur et al., 2017). is also surprising that the lack of an explicitly fixed structure on the latent space of the RAE does not impede interpolation quality. This is further confirmed by the qualitative evaluation on CelebA as reported in Fig. I and for the other datasets in Appendix F, where RAE interpolated samples seem sharper than competitors and transitions smoother.
Our results further confirm and quantify the effect of the aggregated posterior mismatch. In Table 1, ex-post density estimation consistently improves sample quality across all settings and models. A 10-component GMM halves FID scores from 20 to 100 for WAE and RAE models on MNIST and from 116 to 46 on CelebA. This is especially striking since this additional step is much cheaper and simpler than training a second-stage -tage VAE as in 2sV (Q3). In summary, the results strongly sup- port the conjecture that the simple deterministic RAE framework can challenge VAEs and stronger alternatives (Q1).
6.2 GRAMMARRAE: MODELING STRUCTURED INPUTS
We now evaluate RAEs for generating complex structured objects such as molecules and arithmetic expressions. We do this with a twofold aim: i) to investigate the latent space learned by RAE for more challenging input spaces that abide some structural constraints, and ii) quantify the gain of replacing the VAE in state-of-th oh-the-rrt generative model with a RAE.
To this end, we adopt the exact architectures and experimental settings of the GrammarVAE (GVAE) (Kusner et al., 2017), which has been shown to outperform other generative alternatives such the CharacterVAE (CVAE) (Gómez-Bommarelli et al., 2018). As in Kusner al. (2017), we are interested in traversing the latent space learned by our models to generate samples (molecules or expressions) that best fit some downstream metric. This done by Bayesian optimization (BO) by considering the log(1 + MSE) (lower is better) for the generated expressions w.r.t. some ground truth points, and the water-octanol partition coefficient (log P) (Pyzer- Knapp et al., 2015) (higher is better) in the case of molecules. A well-behaved latent space will not only generate molecules or expressions with better scores during the BO step, but it will also contain syntactically valid ones, i..., , samples abide to grammar of rules describing the problem.
Figure 2 summarizes our results over 5 trials of BO. Our GRAEs (Grammar RAE) achieve better average scores than CVAEs and GVAEs in generating expressions and molecules. This is visible also for the three best samples and their scores for all models, with the exception of the first best expression of GVAE. We include in the comparison also the GCVVAE, the equivalent of CV-VAE
8

---

Published as a conference paper at ICLR 2020
MODEL
1ST
2ND
3RD
PROBLUM
MODEL
% VALID
AVG. SCORE
GRAF
1.00 +: 0.00 0.00
3.22 4 0.03
EXPRESSIONS
0.99 0.01
0.08
GRAE
GVAE
0.99
3.26 0.20
CVAB
0.82 0.07
4.74
0.25
SCORE
3.74
3.52
GRAE
0.09
-5.62 + 0.71
GVAE
0.28 1: 0.04
-7,89
1 1.90
GCVVAE
0.04
-25
6.35
3.22
2.83
MODEL
EXPRESSION
SCORE
sin(3) +
0.39
GRAE
1/ exp(1) x
0.39
GVAB
2 2inin 1
0.43
SCORE
3.13
3.10
2.37
++ sin(3) +1
0.39
GCVVAE
x/2/3 sin(exp(cxxp1))) x
0.43
CVAE
sin(x) sin x))
GVAE
1/2 sin( x)
0.46
SCORE
2.75
0.82
0.63
z/2 sin(1) (z/2)
x + sim(2) 3in(3 + x)
0.45
CVAE
z/1 sin(1) 22 2]
0.48
1/1 (x) + s/n(1
0.61
Generating structured objects by GVAE, CVAE and GRAE. (Upper left) Percentage of valid samples and their average mean score (see text, Section 6.2). The three best expressions (lower left) and molecules (upper right) and their scores reported for all models.
Figure 2:
structured objects, as an additional baseline. We can observe that while GCVVAE delivers better average scores for the simpler task of generating equations (even though the single three best equations are on par with GRAE), when generating molecules GRAEs deliver samples associated to much higher scores.
More interestingly, while GRAEs are almost equivalent to GVAEs for the easier task generat- ing expressions, the proportion of syntactically valid molecules for GRAEs greatly improves over GVAEs (from 28% to 72%).
7 CONCLUSION
While the theoretical derivation of the VAE has helped popularize the framework for generative modeling, recent works have started to expose some discrepancies between theory and practice. We have shown that viewing sampling in VAEs as noise injection to enforce smoothness can enable one to distill deterministic autoencoding framework that is compatible with several regularization techniques to learn a meaningful latent space. We have demonstrated that such an autoencoding framework generate comparable better samples than VAEs while getting around the practical drawbacks tied to a stochastic framework. Furthermore, we have shown that our solution of fitting a simple density estimator the learned latent space consistently improves sample quality both for the proposed RAE framework as well as for VAEs, WAEs, and 2sVAEs which solves the mismatch between the prior and the aggregated posterior in VAEs.
ACKNOWLEDGEMENTS
We would like to thank Anant Raj, Matthias Bauer, Paul Rubenstein and Soubhik Sanyal for fruitful discussions.

---

Published as a conference paper at ICLR 2020
REFERENCES
Alexander Alemi, Ben Poole, lan Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing broken ELBO. In ICML, 2018.
Guozhong An. The effects of adding noise during backpropagation training on generalization performance. In Neural computation, 1996.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML, 2017.
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. In ICLR, 2017.
M. Bauer and A. Mnih. Resampled priors for variational autoencoders. AISTATS, 2019.
Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto-encoders as generative models. In NewrlPS, 2013.
Christopher M Bishop. Pattern recognition and machine learning. Springer, 2006.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from a continuous space. In CoNLL, 2016.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2019.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv 2015.
preprint
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In ICLR, 2017.
Mary Kathryn Cowles and Bradley P Carlin. Markov chain Monte Carlo convergence diagnosties: a comparative review. Journal of the American Statistical Association, 1996.
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. ICLR, 2019.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International Conference on Machine Learning, pp. 881-889, 2015.
Partha Ghosh, Arpan Losalka, and Michael J Black. Resisting adversarial attacks using Gaussian mixture variational autoencoders. In AAAI, 2019.
Rafael Gómez-Bombarelli, Jennifer Wei, David Duvenaud, José Miguel Hernández-Looato, Benjamín Sánchez-Lengeling Dennis Sheberla, Jorge Ryan P Adams, and Alán Aspuru- Guzik, Automatic chemical design using a data-driven contin- uous representation of molecules. In ACS central science, 2018.
Timothy D Hirzel,
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of Wasserstein GANs. In NeurlPS, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to Nash equilibrium. In NewrlPS, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. Beta-VAE: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way carve up the vari- ational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NeurlPS, 2016.
10

---

Published as a conference paper at ICLR 2020
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In JCML, 2015.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. arXiv preprint arXiv: 04064, 2018.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. In ICLR, 2014.
Diederik Kingma, Tim Salimans, Rafal Jozefowicz, Chen, Ilya Sutskever, and Max Welling- Improving variational inference with inverse autoregressive flow. In NeurIPS, 2016.
Alex Krizhevsky and Geoffrey Hinton. Learning Multiple Layers of Features from Tiny Images, 2009.
Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. The GAN land- scape: Losses, architectures, regularization, and normalization. arXiv preprint arXiv: 1807.04700 2018.
Matt J Kusner, Brooks Paige, and José Miguel Hemúáneez-Lobtto. Grammar variational autoen- coder. In ICML, 2017.
Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. In AISTATS, 2011.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In IEEE, 1998.
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep Learning Face Attributes in the Wild. In ICCV, 2015.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are GANs created equal? A large-scale study. In NeurIPS, 2018.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. In ICLR, 2016.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for GANs do actually converge? In ICML, 2018
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
Behnam Neyshabur, Ryota Tomioka, Ruslan Salakhutdinov, and Nathan Srebro. Geometry of opti- mization and implicit regularization in deep learning. arXiv preprint arXiv:1705.03071, 2017.
Edward O Pyzer-Knapp Changwon Suh, Rafael Bombarlll Jorge Aguilera-pparaguirre and Alán Aspuru-Guzik. What tigh-throughput virtual screening? A perspective from organic materials discovery. Annual Review of Materials Research, 2015.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VAE-2. arXiv preprint a9Xiv:1906. 00446, 2019.
Danilo Jimenez Rezende and Fabio Viola. Taming VAEs, arXiv preprint arXiv:1810.00597 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, 2014.
Salah Rifai, Paseal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto- encoders: Explicit invariance during feature extraction. In ICML, 2011.
Salah Rifai, Yoshua Bengio, Yann Dauphin, and Pascal Vincent. A generative process for sampling contractive auto-encoders. In JCML, 2012.
Mihaela Rosca, Balaji Lakshminarayanan, and Shakir Mohamed. Distribution matching in varia- tional inference. arXiv preprint arXiv:1802.0683 2018.

---

Published as a conference paper at ICLR 2020
Mehdi S. M. Sajjadi, Bernhard Schölkopf, and Michael Hirsch. Enhancenet: Single image super- resolution through automated texture synthesis. In ICCV, 2017.
Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In NeurIPS, 2018.
Aliaksei Severyn, Erhardt Barth, and Stanislau Semeniuta. hybrid convolutional variational au- toencoder for text generation. In Empirical Methods in Natural Language Processing, 2017.
Jocelyn Sietsma and Robert JF Dow. Creating artificial neural networks that generalize. In Neural networks. Elsevier, 1991.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In NeurIPS, 2015.
Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferenc Huszár. Amortised MAP Inference for Image Super-resolution. In ICLR, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-135 1958, 2014.
Lucas Theis, Aäron den Oord, and Matthias Bethge. A note the evaluation of generative models. In ICLR, 2016.
Andrey N Tikhonov and Vasilii lakkovlevich Arsenin. Solutions of ill-posed problems, volume 14. Winston, Washington, DC, 1977.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf. Wasserstein auto- encoders. In ICLR, 2017.
Jakub Tomezak and Max Welling. VAE with VampPrior. In AISTATS, 2018.
George Tucker, Andriy Mnih, Chris Maddison, John Lawson, and Jascha Sohl-Dickstein. REBAR: low-variance, unbiased gradient estimates for discrete latent variable models. In NeurIPS, 2017.
Aaron van den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurlPS, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. ICML, 2008
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In BMVC, 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Towards deeper understanding of variational autoencoding models. arXiv preprint arXiv:1702.0080 1702.08024 2017.
12

---

Published as a conference paper at ICLR 2020
APPENDIX
A RECONSTRUCTION AND REGULARIZATION TRADE-OFF
We train VAE on MNIST while monitoring the test set reconstruction quality by FID. Figure 3 (left) clearly shows the impact of more expensive k > Monte Carlo approximations of Eq. on sample quality during training. The commonly used 1-sample approximation is clear limitation for VAE training.
Figure 3 (right) depicts the inherent trade-off between reconstruction and random sample quality in VAEs. Enforcing structure and smoothness the latent space of a VAE affects random sample quality in negative way. In practice, compromise needs to be made, ultimately leading to subpar performance.
28
VAE K Sample Approddmatin
VAE Recorstruciinn KL, Looa Tridself
Rosansructioe
F1
20
100
16
10
Traning Batches
1a5
weight
Figure 3: (Left) Test reconstruction quality for a VAE trained on MNIST with different numbers of samples in the latent space as in Eq. 7 measured by FID (lower is better). Larger numbers of Monte-Carlo samples clearly improve training, however, the increased accuracy comes with larger requirements for memory and computation. In practice, the most common choice is therefore k = 1. (Right) Reconstruction and random sample quality (FID, y-axis, lower is better) of VAE on MNIST for different trade-off's between LrEC and LKL (x-axis, see Eq. 5). Higher weights for LKL improve random samples but hurt reconstruction. This is especially noticeable towards the optimality point (8 x 10'). This indicates that enforcing structure in the VAE latent space leads to penalty in quality.
B A PROBABILISTIC DERIVATION OF REGULARIZATION
In this section, we propose an alternative view on enforcing smoothness on the output Da by augmenting the ELBO optimization problem for VAEs with an explicit constraint. While we keep the Gaussianity assumptions over a stochastic Dg and p(z) for convenience, we however are not fixing parametric form for (z (z) x) yet. We discuss next how
parametric restrictions over (2 x) lead to variation of the RAE framework in Eq. 11, specifically the introduction of LGP as regularizer of deterministic version of the CV-VAE. To start, augment Eq. 5 as:
arg min (x- LREC + LKL
(12)
0.0
s.t. -
<E V2,, (z(z)x) Vx Pdata
where (z) μe E (x)) and the constraint on the decoder encodes that the output has to vary, in the sense of an L, norm, only by small amount for any two possible draws from the encoding of
Let (z) dim()) - dim( be given by set of dim(x) given by (D((z) : ddim(a) + R.).
Now we can upper bound the quantity [Do())) ) DDo(2))l by dim(()* sxpp
- (22)))
Using mean value theorem (21) (2 ((22) p S (, (1)-))) + 122
|1 z2llp- Hence
sup, (up,(I)D,G, (z))
(zz)) < (lup,[/V.DD. (1- t)zz + +l2))lp ||| z2 22 Now if we choose
the domain of (z x) to be isotopic the contribution of ||z - 2 k1 to the aforementioned quantity becomes constant factor. Loosely speaking it is the radius of the bounding ball of domain of
[2 x). Hence the above term simplifies to sup, [lpp: VI),,, t)z, + ttz))lp). Recognizing that here 2 and Z2 is arbitrary lets simplify this further
(Z))ll
From this form of the smoothness constraint, it is apparent why the choice a parametric form for (s(z] x) can be impactful during training. For compactly supported isotropic PDF qs(2lx), the
13

---

Published as a conference paper at ICLR 2020
extension of the support supfllzı zellp) would depend on its entropy Hg (z xx). through some functional т. For instance, a uniform posterior over a hypersphere in z would ascertain ((Hll(2s( x))) = Hllll xx)/nn where n is the dimensionality of the latent space.
Intuitively, one would look for parametric distributions that do not favor overfitting, e.g., degenerat- ing in Dirac-deltas (minimal entropy and support) along any dimensions. To this end, an isotropic nature of 4o (z(x) would favor such robustness against decoder over-fitting. We can now rewrite the constraint as
r(H(qs (z) x))) sup V (a(z)]lp) <E
(13)
The LKL term can be expressed in terms of Hg (z x)). ) by decomposing it as LKL = LCE LH. where Lh Hgga(2 (z x)) and LCE Hl(go x), p(z)) represents cross-entropy term. Therefore, the constrained problem in Eq. 12 can be written in a Lagrangian formulation by including Eq. 13:
arg min Ex-pan CREC + LcE LH + ALLANG
(14)
0,0
where LLANG (g x))) VDo(z)]l~ We argue that a reasonable simplifying assumption
for (2 x) is to fix H(g (z x)) to a single constant for all samples x. Intuitively, this can be understood as fixing the variance in g (z x) as we did for the CV-VAE in Section 2.2. With this
simplification, Eq. 14 further reduces to
arg min (x- PREC + LCE A/)VD((z)llz 0,0
(15)
We can see that (z)]l results be the gradient penalty LGP and LCE = ||z|| corresponds to KL thus recovering our RAE framework as presented in Eq. 11.
C NETWORK ARCHITECTURE. TRAINING DETAILS AND EVALUATION
We follow the models adopted by Tolstikhin al. (2017) with the difference that consistently ap- ply batch normalization (loffe & Szegedy, 2015). The latent space dimension is 16 for MNIST (Le- Cun et al., 1998), 128 for CIFAR-10 (Krizhevsky & Hinton, 2009) and 64 for CelebA (Liu al., 2015).
For all experiments, use the Adam optimizer with a starting learning rate of 10-3 which cut in half every time the validation loss plateaus. All models are trained for a maximum of 100 epochs on MNIST and CIFAR and 70 epochs on CelebA. We use mini-batch size of 100 and pad MNIST digits with zeros to make the size x2x
We use the official train, validation and test splits of CelebA. For MNIST and CIFAR, we set aside 10k train samples for validation. For random sample evaluation, we draw samples from (0, 7) for VAE and WAE-MMD and for all remaining models, samples are drawn from a multivariate Gaussian whose parameters (mean and covariance) are estimated using training set embeddings. For the GMM density estimation, we also utilize the training set embeddings for fitting and validation set embeddings to verify that GMM models not over fitting to training embeddings. However, due to the very low number of mixture components (10), we did not encounter overfitting at this step. The GMM parameters are estimated by running EM for at most 100 iterations.
MNIST
CI
CELEBA
ENCODER:
E R232332
2 E R32x32
27 E R51X66
CONV12S BN RELU
CoNV122 BN RELU
CONV128 + RELU
CoNV266 + BN + RELU
4 CONV266 + BN + RELU
CoNV256 - BBN + RELU
CONVs12 BN RELU
CONV512
RELU CONV512 +
CONV1624 + BN RALU
CONV1024 + BN RELU
CONV1624 + BN RELU
FLATTEN + FC16XM
FLATTEN FC128KM
FLATTEN + FC24X N
DECODER:
E R6 + FCsx9x1024
2E R129 + FCsx5x1024
2 E 4 FCsxex
RELU
+ RELU
BN + RALU
CONvTs12 BN RELU
CONVT BN RELU
CONVT BN RELU
CoNVT226 - BN + RELU
CONVT236 BN + RELU
CoNVT226 + BN - RELU
CONVT,
CONVT)
CoNVT12s BN RELU
CoNvT1
Conv,, represents a convolutional layer with n filters. All convolutions Conv, and transposed con- volutions ConyT, have a filter size of 4x4 4 for MNIST and CIFAR-10 and 5x5 for CELEBA. They
14

---

Published as a conference paper at ICLR 2020
all have stride of size 2 except for the last convolutional layer in the decoder. Finally, M for all models except for the VAE which has M = 2 as the encoder has to produce both mean and variance for each input.
D EVALUATION SETUP
We compute the FID of the reconstructions of random validation samples against the test set to evaluate reconstruction quality. For evaluating generative modeling capabilities, we compute the FID between the test data and randomly drawn samples from single Gaussian that is either the isotropic p(z) fixed for VAEs and WAEs, learned second stage VAE for 2s or a single also evaluate random samples
Gaussian fit (5(z) for CV-VAEs and RAEs. For all models, from 10-component Gaussian Mixture model (GMM) fit to (z(z). Using only 10 components prevents us from overfitting (which would indeed give good FIDs when compared with the test set)?.
For interpolations, we report the FID for the furthest interpolation points resulted by applying spher- ical interpolation to randomly selected validation reconstruction pairs.
We use 10k samples for all FID and PRD evaluations. Scores for random samples are evaluated against the test set. Reconstruction scores are computed from validation set reconstructions against the respective test set. Interpolation scores are computed by interpolating latent codes of a pair of randomly chosen validation embeddings test set samples. The visualized interpolation samples are interpolations between two randomly chosen test set images.
E EVALUATION BY PRECISION AND RECALL
MNIST
CIFAR-10
CELEBA
N
GMM
N
GMM
WV
GMM
VAE
0.96 0.96/
0.95 / 0.96
0.25 0..5
0.37 / 0.56
0.66
0.50 / 0.66
CV-VAE
0.73
0.96 / 0.89
0.31
0.42 0.66
0.25 0
0.32 / 0.55
WAE
0.88
0.98 0.95
0.38
0.51 0.81
0.59/
0.69 0.77
RAE-GP
0.88
0.97 / 0.98
0.36
0.46 0.77
0.38
0.44 0.67
RAE-L2
0.87
0.98 0.98
0..77
0.57 0.81
0.66
0.44 0.65
RAE-SN
0.99
0.98 0.99
0.52 0.76
0.54/
0.55 /.774
RAE 0.92
0.98 / 0.98
0.45 0.73
0.53 / 0.80
0 0.46
AE
/.90/0.90
0.98 / 0.97
0.37 0.73
0.50 / 0.80
0.45 0.66
0.77
Table 2: Evaluation of random sample quality by precision / recall (Sajjadi et al., 2018) (higher numbers are better, best value for each dataset bold). It notable that the proposed ex-post density estimation improves not only precision, but also recall throughout the experiment. For example, WAE seems to have a comparably low recall of only 0.88 on MNIST which raised considerably to 0.95 by fitting GMM. In all cases, GMM gives the best results. Another interesting point is the low precision but high recall of all models on CIFAR this also visible upon inspection of the samples in Fig. 9.
We note that fitting GMMs with up to 100 components only improved results marginally. Additionally, provide neirest-neighbours from the training set in Appendix G to show that models overfitting.
15

---

Published as a conference paper at ICLR 2020
PRD ALL RAFS
PRD ALL TRADITIONAL VAFS
WAF VS RAE-SN VS WME-GMM
1.0
1.0
1.0
0.8
0.8
0.8
Precision
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Recall
Recall
Recall
Figure 4: PRD curves of all RAE methods (left), reflects a similar story as FID scores do. RAE- SN seems to perform the best both precision and recall metric. PRD curves of all traditional VAE variants (middle). Similar to the conclusion predicted by FID scores there are no clear winner. PRD curves for the WAE (with isotropic Gaussian prior). WAE-GMM model with ex-post density estimation by 10-component GMM and RAE+SN-GMM (right). This finer grained view shows how the WAE-GMM scores higher recall but lower precision than a RAE+SN-GMI while scoring comparable FID scores. Note that ex-post density estimation greatly boosts the WAE model in both PRD and FID scores.
16

---

Published as a conference paper at ICLR 2020
MNIST
VAR
CV-VAR
WAF
Nimu. isigmal
0.8
0.8
0.8
Freasion
0.6
Prediaion
0.6
Precision
0.6
0.2
0.2
0.2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Recall
Recall
Recall
RAE-GP
RAE-L2
RAE-SN
1.0
1.0
NOC
MS5.
N20. 0
GNN
GVN 10
G900_10 10
0.6
Procision
0.6
Precktion
0.6
0.4
0.0
0.0
0.0
0.50
0.00
0.25
1.00
0.50 Recall
0.75
Recall
Recall
RAE
AB
1.0
1.0
NIC GMN_ 30
Nomu, tsigmal
GMM_10
Prectiion
0.4
0.4
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.75
1.00
Figure 5: PRD curves of all methods on image data experiments on MNIST, For each plot, we show the PRD curve when applying the fixed or the fitted one by ex ex-post density estimation (XPDE). XPDE greatly boosts both precision and recall for all models.
17

---

Published as a conference paper at ICLR 2020
CIFAR
VAR
CV-VAR
WAF
Ninu, isigmal
0.8
0.8
0.8
Fression
0.6
Preciaion
0.6
Precision
0.6
0.2
0.2
0.2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Recall
Recall
Recall
RAE-GP
RAE-L2
RAE-SN
1.0
1.0
NOC
NS5.
NC0. 1
GHN
GVM 10
G0
0.6
Procition
0.6
Precktion
0.6
0.4
0.0
0.0
0.0
0.50
0.50 Recall
0.75
Recall
Recall
RAE
AB
1.0
1.0
NOC.
Nomu, tsigmal
GMN_10
GMML1D
Precttion
0.4
0.4
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Figure 6: PRD curves of all methods on image data experiments on CIFARIO. For each plot, we show the PRD curve when applying the fixed or the fitted one by ex-post density estimation (XPDE). XPDE greatly boosts both precision and recall for all models.
18

---

Published as a conference paper at ICLR 2020
CELEBA
VAR
CV-VAR
WAF
isigmal
0.8
0.8
0.8
Pression
0.6
Prediaion
0.6
Precision
0.6
0.2
0.2
0.2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Recall
Recall
Recall
RAE-GP
RAE-L2
RAE-SN
1.0
1.0
NOC GHN J0
MS5.
N20. 0
GVM 10
GV
0.6
Procision
0.6
Precision
0.6
0.4
0.2
0.0
0.0
0.0
0.50
0.00
0.75
1.00
0.50
0.75
Recall
Recall
Recall
RAE
AB
1.0
1.0
NSC.
Nomu, tsigmal
GMN_10
GMML1D
Precttion
0.4
0.4
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Figure 7: PRD curves of all methods on image data experiments on CELEBA. For each plot, we show the PRD curve when applying the fixed or the fitted one by ex-post density estimation (XPDE). XPDE greatly boosts both precision and recall for all models.
19

---

Published as a conference paper at ICLR 2020
F
MORE QUALITATIVE RESULTS
RECONSTRUCTIONS
RANDOM SAMPLES
INTERPOLATIONS
GT
4
7
S
3
6
VAE
4
1
7
5
3
6
3
5
3
2
8
7
2
CV-VAR
4
1
7
5
3
6
3
7
8
3
3
2
2
6
6
WAE
4
I
7
5
3
6
0
6
5
f
3
2
2
6
VS
4
I
7
S
3
6
9
9
0
2
6
6
RAE-GP
4
I
7
5
3
6
3
6
3
3
0
0
6
RAE-L2
4
I
7
5
3
6
6
4
6
6
0
0
2
RAE-SN
4
1
7
5
3
6
/
8
0
/ 4 1
3
6
RAE
4
1
7
S
3
6
5
1
8
8
9
6
AE
4
/
7
S
3
6
2
0
7
2
7
2
6
GT
4
5
8
3
5
3
VAE
4 5
3 4 5 8 3
8
$
3
7
3
6
0
7
33399
9999999
CV-VAE
3
8
O
7
7 4 1 6
0
8
WAF
4 #5 8 3 $3
4
2
2 2 8
1 8 1 9 0
6 0
VS
4 5 8 3 $3
1
9
7
RAE-GP
4 5 8 3 $3
8
8
7
RAE-L2
5588 8
3 $3
3
553
3
4
3
7
RAE-SN
4 5 8
8
3
53
6
0
2
9
3
7
RAE
4 5 8
8
3
5
0
3
1
1
8
0
7
7
AE
5 8
3
5
3
3 9
8
0
/ 3
2
b
Figure 8: Qualitative evaluation for sample quality for VAEs, WAEs and RAEs MNIST. Left: reconstructed samples (top row is ground truth). Middle: randomly generated samples. Right: spherical interpolations between images (first and last column).
20

---

Published as a conference paper at ICLR 2020
RECONSTRUCTIONS
RANDOM SAMPLES
INTERPOLATIONS
GT
VAE
CV-VAE
WAF
2sVAE
RAE-GP
RAE-L2
RAE-SN
RAE
AE
GT
VAE
CV-VAE
WAE
2sVAE
RAE-GP
RAE-L2
RAE-SN
RAE
AE
Figure 9: Qualitative evaluation for sample quality for VAEs, WAEs and RAEs on CIFAR-10. Left: reconstructed samples (top is ground truth). Middle: randomly generated samples. Right: spherical interpolations between two images (first and last column).
21

---

Published as a conference paper at ICLR 2020
G
INVESTIGATING OVERFITTING
MNIST
CIFAR-10
CELERA
VAE
I
7
CV-VAR
9
8
9
4
9 4
WAE
2
2
2 2
O
O O
RAE-GP
5
5
5 5
S
C
RAE-L2
3
3 3 3
3
3 3
3
RAE-SN
7
6 6 6
9
9
RAE
0
9
AE
5 5 5
5
0
5 5 5
Figure 10: Nearest neighbors to generated samples (leftmost image, red box) from training set. It seems that the models have generalized well and fitting only 10 Gaussians to the latent space prevents overfitting.
22

---

Published as a conference paper at ICLR 2020
H VISUALIZING EX-POST DENSITY ESTIMATION
To visualize that ex-post density estimation does indeed help reduce the mismatch between the aggregated posterior and the prior we train VAE on the MNIST dataset whose latent space dimensional. The unique advantage of this setting is that one can simply visualize the density of test sample in the latent space by plotting them as scatterplot. As it can be seen from figure 11, an expressive density estimator effectively fixes the miss match and this as reported earlier results in better sample quality.
I)
Nía
GMM(K- 10)
-5
-5
110
Figure 11: Different density estimations of the 2-dimensional latent space of VAE learned on MNIST, The blue points are 2000 test set samples while the orange ones are drawn from the es- timator indicated in each column: isotropic Gaussian (left), multivariate Gaussian with mean and covariance estimated the training set (center) and 10-component GMM (right). This clearly shows the aggregated posterior mismatch w.r.t. to the isotropic Gaussian prior imposed by VAEs and how ex-post density estimation can help fix the estimate.
Here in figure 12 we perform the same visualization on with all the models trained on the MNIST dataset as employed on our large evaluation in Table 1. Clearly every model depicts rather large mis- match between aggregate posterior and prior. Once again the advantage of ex ex-post density estimate clearly visible.
23

---

Published as a conference paper at ICLR 2020
\00, 7)
N)
GMM(k 10)
VAE
CV-VAE
-10
WAE
RAE-GP
RAE-L22
23
RAB-SN
RAE
23
-30
109
AE
Figure 12: Different density estimations of the 16-dimensional latent spaces learned by all models on MNIST (see Table 1) here projected in 2d via T-SNE. The blue points are 2000 test set samples while the orange ones drawn from the estimator indicated in each column: isotropic Gaussian (left), multivariate Gaussian with mean and covariance estimated on the training set (center) and 10-component GMM (right). Ex-post density estimation greatly improves sampling the latent space.
24

---

Published as a conference paper at ICLR 2020
MNIST
CIFAR
CELEBÁ
SAMPLES
SAMPLES
SAMPLES
REC.
N
GMM
Interp.
REC.
N
GMM
Interp.
REC.
N
GMM
Interp.
11.54
15.32
76.33
64.08
RAE-L2
10.53
22.22
8.69
14.54
32.24
80.80
74.16
62.54
43.52
51.13
47.97
45.98
RAE-SN
15.65
19.67
11.74
15.15
27.61
84.25
75.30
63.62
36.01
44.74
40.95
39.53
RAE
11.67
23.92
9.81
14.67
29.05
83.87
76.28
63.27
40.18
48. 38.20
44.68
43.67
AE
12.95
58.73
10.66
17.12
30.52
84.74
76.47
61.57
40.79
127.85
45.10
50.94
AB-L2
11.19
315.15
9.36
17.15
34.35
247
75.40
61.09
44.72
346.25
48.42
16.16
RAE-GP-L2
9.70
72.64
9.07
16.07
33.25
187.07
79.03
62.48
47.06
72.09
51.55
50.28
RAE-L2-SN
10.67
50.63
9.42
15.73
24.17
240.27
74.10
61.71
39.90
180.39
44.39
42,97
RAE-SN GP
17.00
139.61 61
13.12
16.62
33.04
284.36
75.23
62.86
63.75
299.69
71.05
68.87
RAB-L2-SN-GP
16.75
144.51
13.93
16.75
29.96
290.34
74.22
61.93
68.86
318.67
75.04
74.29
Table 3: Comparing multiple regularization schemes RAE models. The improvement in recon- struction, random sample quality and interpolated test samples is generally comparable, but hardly much better. This can be explained with the fact that the additional regularization losses make tuning their hyperparameters more difficult, in practice.
I COMBINING MULTIPLE REGULARIZATION TERMS
The rather intriguing facts that without explicit decoder regularization performs reasonably well seen from table 1, indicates that convolutional neural networks when combined with gradient based optimizers inherit some implicit regularization. This motivates us to investigate few different combinations of regularizations e.g. regularize the decoder of an auto-encoder while drop the regularization in the 2 space. The results of this experiment reported in the row marked AE-L2 in table 3.
Further more recent GAN literature ? report that often a combination of regularizations boost performance of neural networks. Following this, combine multiple regularization techniques in out framework. However note that this rather drastically increases the hyper parameters and the models become harder to train and goes against the core theme of this work, which strives for simplicity. Hence we perform simplistic effort to tune all the hyper parameters to see if this can provide boost in the performance, which seem not to be the case. These experiments are summarized in the second half of the table 3
25


Reviews:
[{"review_id": "rkxPP-2itH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose an extension to deterministic autoencoders. Motivated from VAEs, the authors propose RAEs, which replace the noise injection in the encoders of VAEs with an explicit regularization term on the latent representations. As a result, the model becomes a deterministic autoencoder with a L_2 regularization on the latent representation z. To make the model generalize well, the authors also add a decoder regularization term L_REG. In addition, due to the encoder in RAE is deterministic, the authors propose several ex-post density estimation techniques for generating samples.\n\nThe idea of transferring the variational to deterministic autoencoders is interesting. Also, this paper is well-written and easy to understand. However, in my opinion, this paper needs to consider more cases for autoencoders and needs more rigorous empirical and theoretical study before it can be accepted. Details are as follow:\n\n1. The RAEs are motivated from VAEs, or actually CV-VAEs as in this paper. More precisely, the authors focus on VAEs with a constant covariance Gaussian distribution as the variational distribution and a Gaussian distribution with the identity matrix as the covariance matrix as the model likelihood. However, there might be many other settings for VAEs. For example, the model likelihood can be a Gaussian distribution with non-constant covariance, or even some other distributions (e.g. Multinomial, Bernoulli, etc). Similarly, the variational distribution can be a Gaussian distribution with non-constant covariance, or even some more complicated distributions that do not follow the mean-field assumption. Any of these more complex models may not be easily transferred to the RAE models that are mentioned in this paper. Perhaps it is better if the authors can consider RAEs for some more general VAE settings.\n\n2. Perhaps the authors needs more empirical study, especially on the gain of RAE over CV-VAE and AE. \na) As the motivated model (CV-VAE) and the most related model in the objective (AE), they are not appearing in the structured input experiment (Section 6.2). It will be great if they can be compared with in this experiment.\nb) The authors did not show us clearly whether the performance gain of RAE over VAE, AE and CV-VAE is due to the regularization on z (the term L_z^RAE) or the decoder regularization (the term L_REG) in the experiments. In table 1, the authors only compare the standard RAE with RAE without decoder regularization, but did not compare with RAE without the regularization on z (i.e. equivalent to AE + decoder regularization) and CV-VAE + decoder regularization. The authors would like to show that the explicit regularization on z is better than injecting the noise, hence the decoder regularization term should appear also in the baseline methods. It is totally possible that perhaps AE + decoder regularization or CV-VAE + decoder regularization perform better than RAE. \nc) The authors did not show how they tune the parameter \\sigma for CV-VAE. Since the parameter \\beta in the objective of RAE is tunable, for fair comparison, the authors needs to find the best \\sigma for CV-VAE in order to get the conclusion that explicit regularization is better than CV-VAE.\nd) Although the authors mention that the 3 regularization techniques perform similarly, from Table 1, it is still hard to decide which one should we use in practice in order to get a performance at least not too much worse compared to the baseline methods. RAE-GP and RAE-L2 perform not well on CelebA while RAE-SN perform not well on MNIST, compared to the baseline methods. We know that the best performance over the 3 methods is always comparable to or better than the baselines, but not none of the single methods do. It is better if the authors can provide more suggestions on the choice for decoder regularization for different datasets.\n\n3. The authors provided a theoretical derivation for the objective L_RAE (Equation 11), but this is only for the L_GP regularization. Besides, this derivation (in Appendix B) has multiple technique issues. For example, in the constraints in Equation 12, the authors wrote ||D_\\theta(z1) - D_\\theta(z2)|| < epsilon for all z1, z2 ~ q_\\phi(z | x), this is impossible for CV-VAE since this constraint requires D_theta() to be bounded while q_\\phi(z | x) in CV-VAE has an unbounded domain. Moreover, in the part  (||D_\\theta(z1) - D_\\theta(z2)||_p=\\nabla D_\\theta(\\tilde z)\\cdot ||z_1-z_2||_p) of Equation 13, \\nabla D_\\theta(\\tilde z) is a vector well the other two terms are scalars, which does not make sense. There are many other issues as well. Please go through the proof again and solve these issues.\n\n\nQuestions and additional feedback:\n\n1. Can the authors provide more intuitions why do you think the explicit regularization works better compared to the noise injection? Can you provide a theoretical analysis on that?\n\n2. Can the authors provide some additional experiments as mentioned above? Also, can the authors provide more details about how do they tune the parameters \\beta and \\lambda?\n\n========================================================================================================\n\nAfter the rebuttal:\n\nThanks the authors for the detailed response and the additional experiments. I agree that the additional experiment results help to support the claims from the authors, especially for the CV-VAE for the structured data experiments and the AE + L2 experiment. So I think now the authors have more facts to support that RAE is performing better compared to the baseline methods. \n\nTherefore, I agree that after the revision, the proposed method RAE is supported better empirically. So I am changing my score from \"weak reject\" to \"weak accept\". But I still think the baseline CV-VAE + regularization is important for Table 1 and the technical issues in the theoretical analysis needs to be solved. Hope the authors can edit them in the later version.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1lYVM27cr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies (the more conventional) deterministic auto-encoders, as they are easier to train than VAE. To then try to maintain the model's capability of approximating the data distribution and to draw/synthesize new unseen samples, the paper both looks at imposing additional regularization terms towards a smooth decoder and proposes to sample from a latent distribution that's induced from empirical embeddings (similar to an aggregate posterior in VAE). Experiments are mostly around contrasting VAEs with the proposed RAEs in terms of comparing the quality of the generated samples.\n\nThe paper is trying to answer an important and meaningful question, i.e. can a deterministic auto-encoder learn a meaningful latent space and approximates the data distribution just as well as the much more complicated VAEs. And they've made some interesting and reasonable trials. The methods developed in the paper are mostly easy to understand and also well motivated in general.\nHowever my main concern is on the empirical studies, as they don't seem particularly convincing to justify the claimed success of the RAE over VAE. That said, the evaluation of generative models is by itself still an active research topic in development, which certainly adds to the difficulty of coming up with a sensible and rigorous comparative study in this regard.\n\nDetailed comments and questions.\n1. On \"interpolation\": it might provide some insights on how well the decoder can \"generalize\" to unseen inputs (which is still important), but otherwise seems to provide very little insight on how well the actual \"distribution\" is being approximated, as it's much more about \"coverage\" of the support of the distribution than the actual probabilities themselves?\n2. I'm not so sure FID can be regarded as the golden standard when it comes to comparing data generation qualities, and especially for the given setting, as it might give all those methods employing the so-called \"ex-post density estimation\" an edge over VAEs due to the scraping of the gap between the aggregate posterior and the prior (by design).\n3. From Table 2, according to the \"Precision/Recall\" criteria, WAE clearly out-performs RAE on the CelebA dataset, contradicting the result with FID in Table 1. I think this might need a closer look (as to why this happened) and certainly should be worth some discussions in the paper.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgpIOihYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper analyzes Variational Autoencoders and formulates an alternative perspective on many common problems in the VAE framework like a mismatch between the aggregated posterior and the marginal distribution over latent variables. The authors perform an autopsy of the VAE objective and show how it could be formulated as a constrained optimization problem. Interestingly, the paper proposes to remove all stochasticity from an encoder and a decoder and use different regularizers instead. In order to be able to sample from the model, the marginal distribution over latents is trained after the encoder and decoder are learned.  In general, I find the paper very interesting and important for the VAE community.\n\n- (An open question) The marginal distribution p(z) allows to turn the AE into a generative model. However, it also serves as a code model in a compression framework (i.e., it is used in the adaptive entropy coding). It would be interesting to see how the proposed approach would compare to other compression methods.\n\n- It would be interesting to see how the post-training of p(z) performs. Maybe it would be a good idea to run some toyish problem (e.g., dim(z)=2 on MNIST) and see how the GMM fits samples from the aggregated posterior.\n\n- Did the authors try a mix of different regularizers they proposed to use? For instance, L2 + SN?\n \n- Could the authors comment on the choice of hyperparameters (weights in the objective)?\n\n======== AFTER REBUTTAL ========\nI would like to thank the authors for answering my questions. I really appreciate all effort to provide more empirical evidence. I still believe that this paper is extremely important for the AE/VAE community and it sheds some new light on representation learning. In my opinion the paper should be accepted. Therefore, I decided to rise my score from \"Weak Accept\" to \"Accept\".", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an extension to deterministic autoencoders, namely instead of noise injection in the encoders of VAEs to use deterministic autoencoders with an explicit regularization term on the latent representations. While the reviewers agree that the paper studies an important question for the generative modeling community, the paper has been limited in terms of theoretical analysis and experimental validation. The authors, however, provided further experimental results to support the claims empirically during the discussion period and the reviewers agree that the paper is now acceptable for publication in ICLR-2020. 
Decision: Accept (Poster)