Perceptual Generative Autoencoders
Authors: Zijun Zhang
Ruixiang Zhang
Zongpeng Li
Yoshua Bengio
Liam Paull
Venue:  (2020)

Abstract: Modern generative models are usually designed to match target distributions directly in the data space, where the intrinsic dimensionality of data can be much lower than the ambient dimensionality. We argue that this discrepancy may contribute to the difficulties in training generative models. We therefore propose to map both the generated and target distributions to the latent space using the encoder of a standard autoencoder, and train the generator (or decoder) to match the target distribution in the latent space. The resulting method, perceptual generative autoencoder (PGA), is then incorporated with a maximum likelihood or variational autoencoder (VAE) objective to train the generative model. With maximum likelihood, PGAs generalize the idea of reversible generative models to unrestricted neural network architectures and arbitrary latent dimensionalities. When combined with VAEs, PGAs can generate sharper samples than vanilla VAEs. Compared to other autoencoder-based generative models using simple priors, PGAs achieve state-of-the-art FID scores on CIFAR-10 and CelebA.

Under review as a conference paper at ICLR 2020
PERCEPTUAL GENERATIVE AUTOENCODERS
Anonymous authors
Paper under double-blind review
ABSTRACT
Modern generative models are usually designed to match target distributions di- rectly in the data space, where the intrinsic dimensionality of data can be much lower than the ambient dimensionality. We argue that this discrepancy may con- tribute to the difficulties in training generative models. We therefore propose map both the generated and target distributions to the latent space using the en- coder of standard autoencoder, and train the generator (or decoder) to match the target distribution in the latent space. The resulting method, perceptual generative autoencoder (PGA), is then incorporated with maximum likelihood or variational autoencoder (VAE) objective to train the generative model. With maximum like- lihood. PGAs generalize the idea of reversible generative models to unrestricted neural network architectures and arbitrary latent dimensionalities. When combined with VAEs, PGAs can generate sharper samples than vanilla VAEs. Compared to other autoencoder-bant based generative models using simple priors, PGAs achieve state-of- oftte-of-th-aate FID scores on CIFAR-10 and CelebA
INTRODUCTION
Recent years have witnessed great interest in generative models, mainly due to the success of generative adversarial networks (GANs) (Goodfellow et al. 2014 Radford et al. [016:] Karras et al.) 2018. Brock et al. 2019). Despite their prevalence, the adversarial nature of GANs can lead to number of challenges, such as unstable training dynamics and mode collapse. Since the advent of
GANs, substantial efforts have been devoted to addressing these challenges (Salimans et al. 2016]
Arjovsky et al. 2017: Gulrajani et al.) 2017 Miyato et al. 2018). while non-adversarial approaches that are free of these issues have also gained attention. Examples include variational autoencoders
(VAEs) (Kingma & Welling 2014). reversible generative models (Dinh al. 2014 2017 Kingma &
Dhariwal 2018), and Wasserstein autoencoders (WAEs) (Tolstikhin et al. 2018).
However, non-adversaiial approaches often have significant limitations. For instance, VAEs tend to generate blurry samples, while reversible generative models require restricted neural network architectures solving neural differential equations (Grathwohl et al., 2019). Furthermore, use the change of variable formula, the latent space of a reversible model must have the same dimensionality as the data space, which is unreasonable considering that real-world, high-dimensional data (e.g.. images) tends to lie on low-dimensional manifolds, and thus results in redundant latent dimensions and variability. Intriguingly, recent research (Arjovsky et al. 2017 Dai Wipf 2019) suggests that the discrepancy between the intrinsic and ambient dimensionalities of data also contributes to the difficulties in training GANs and VAËS.
In this work, we present a novel framework for training autoencoder based generative models, with non-adversarial losses and unrestricted neural network architectures. Given a standard autoeneoder and target data distribution, instead of matching the target distribution in the data space, we map both the generated and target distributions to the latent space using the encoder, and train the generator (or decoder) minimize the divergence between the mapped distributions. We prove, under mild assumptions, that by minimizing a form of latent reconstruction error, matching the target distribution in the latent space implies matching it in the data space. We call this framework percepiual generative autoencoder (PGA). We show that PGAs enable training generative autoencoders with maximum likelihood, without restrictions on architectures latent dimensionalities. In addition, when combined with VAEs, PGAs can generate sharper samples than vanilla VAEs
"oode available at httths/////ttttt 1/220kKRY1

---

Under review as a conference paper at ICLR 2020
We summarize main contributions follows:
training framework, PGA, for generative autoencoders is developed to match the target distribution in the latent space, which, we prove, ensures the matching in the data space.
We combine the PGA framework with maximum likelihood, and remove the restrictions of reversible generative models on neural network architectures and latent dimensionalities.
We combine the PGA framework with VAE, which solves the problem of blurry samples, without introducing any auxiliary models sophisticated model architectures.
2 RELATED WORK
Autoencoder-based generative models are trained by minimizing an data reconstruction loss with regularizations. As an early approach, denoising autoencoders (DAEs) (Vincent al. 2008) are trained to recover the original input from an intentionally corrupted input. Then a generative model can be obtained by sampling from a Markov chain (Bengio et al., 2013). To sample from a decoder directly, most recent approaches resort to mapping a simple prior distribution to a data distribution using the decoder. For instance, adversarial autoencoders (AAEs) (Makhzani et al.) 2016) and Wasserstein autoencoders (WAFs) (Tolstikhin et al. 2018) attempt to match the aggregated posterior and the prior, either by adversarial training or by minimizing their Wasserstein distance. However, due to the use of deterministic encoders, there can be "holes" in the latent space that are not covered by the aggregated posterior, which would result in poor sample quality (Rubenstein et al. 2018). By using stochastic encoders and variational inference, variational autoencoders (VAEs) suffer less from this problem, but are known to generate blurry samples (Rezende Viola 2018 Dai & Wipf, 2019). Nevertheless, as will show, the latter problem can be addressed by moving the VAE reconstruction loss from the data space to the latent space.
likely to
In different line of work, reversible generative models (Dinh et al., 2014 2017 Kingma & Dhariwal 2018) are developed to enable exact inference. Consequently, by the change of variables theorem, the likelihood of each data sample can be exactly computed and optimized. However, to avoid expensive Jacobian determinant computations, reversible models can only be composed of restricted transformations, rather than general neural network architectures. While this restriction can be relaxed by utilizing recently developed neural ordinary differential equations (Chen et al.) 2018 Grathwohl et al., 2019), they still rely on a shared dimensionality between latent and data spaces, which remains an unnatural restriction. In this work, we use the proposed training framework to trade exact inference for unrestricted neural network architectures and arbitrary latent dimensionalities, generalizing maximum likelihood training to secennoddrrbbsee models.
3 METHODS
3.1 PERCEPTUAL GENERATIVE MODEL.
Let fo : RD "f be the encoder parameterized by and go R + R. be the decoder parameterized by 6, Our goal is to obtain a decoder-based generative model, which maps a simple prior distribution to a target data distribution, D. A summary of notations provided in Appendix A Throughout this paper, we use N (O, as the prior distribution.
For z E IR. H the output of the decoder, go (z), lies in manifold that is most H-dimensional. Therefore, if we train the autoencoder to minimize
E2
| x
(I)
where & go (fe (x)), then & can be seen projection of the input data, onto the manifold of go (z). Let D denote the reconstructed data distribution, i.e., x ~ D. Given enough capacity of the encoder, D is the best approximation to D (in terms of lg-distance), that we can obtain from the decoder, and thus can serve as surrogate target distribution for training the recoder-based generative model.
Due the difficulty in directly matching the generated distribution with the data-space target distribution, D. reuse the encoder to map D to latent-space target distribution, H. We then
2

---

Under review as a conference paper at ICLR 2020
(.000.1)
x-
10
50
(a) PGA
2
A7
80
5V
f0
59
8-
(b) LPGA
VPGA
Figure 1: Illustraiion of the training process of PGAs. The overall loss function consists of (a) the basic PGA losses, and either (b) the LPGA-specific losses or (c) the VPGA losses. Circles indicate where the gradient is truncated, and dashed lines indicate where the gradient is ignored when updating parameters.
transform the problem of matching D in the data space into matching H. in the latent space. In other words, we aim to ensure that for ~ N (0, I), if fo (go (z)) ~ H. then go (z) D. In the following, we define fo go for notational convenience.
To this end, we minimize the following latent reconstruction loss w.r.t. Ø:
Lir 1 Bn-N((..)) [1 [z) 21/2
(2)
Let Z (x) be the set of all z's that are mapped to the same x by ge. we have the following theorem:
Theorem 1. Assuming E [z x] E Z (x) for all x generated by go. and sufficient capacity of fe: for ~ A I), if Eq. (2) is minimized and h (z) H. then go (z) D.
We defer the proof to Appendix B.1 Note that Theorem requuies that different x's generated by go
T
(from N (0, and H) mapped different z's by f6- In theory, minimizing Eq. (2) would suffice, since N (0, is supported on the whole F However, there be z's with low probabilities / (0, I), but with high probabilities in H that are not well covered by Eq. ( Therefore, is
sometimes helpful minimize another latent reconstruction loss H:
L = 2
BEn [IIA (z) z1(
(3)
In practice, we observe that L is often small without explicit minimization, which we attribute to its consistency with the minimization of Ly.
By Theorem 1 the problem of training the generative model reduces training ht to map / (0, H. which we refer to as the perceptual generative model. In the subsequent subsections, we present maximum likelihood approach, VAE-based approach, and unified approach to train the perceptual generative model. The basic loss function of PGAs is given by
Lpge = L, + at + 3
(4)
where and are hyperparameters be tuned. Eq.
(4) is also illustrated in Fig. la

---

Under review as a conference paper at ICLR 2020
3.2 A MAXIMUM LIKELIHOOD APPROACH
We first assume the invertibility of h. For & ~ D. let = La (x) H.. We can train à directly with maximum likelihood using the change of variables formula
E R logp p [2) = Ez logp (z) log det (ōh Oz
[hh (z)
(5)
Ideally, we would like maximize Eq. (5) only w.r.t. the parameters of the generative model (i.e., 0). However, directly optimizing the first term in Eq. (5) requires computing z = -h-1 (æ), which is usually unknown. Nevertheless, for 2 ~ H, we have minimize the following loss function w.r.t. Φ instead:
(2) fo (x) and x D, and thus we can
nll EE-H [logp (z))
E Ex ExND [lfe (xx
To avoid computing the Jacobian in the second term of Eq. which is slow for unrestricted architectures, we approximate the Jacobian determinant and derive loss function to be minimized w.r.t. 0:
H. Lan = 2 B
|// + +0) (z)
Ex log det
ahh (z) Oz
)
log
(7)
||
where S (E) can be either N (0, 21), or uniform distribution on small (Hf 1 1--shere of radius e centered at the origin. The latter choice is expected to introduce slightly less variance. We show below that the approximation gives upper bound when - O. Eqs. ( and (7) are illustrated in Fig. lb
Proposition 1. For -
Oh (z) Oz
H Sse)(e)
hh 1z + 8) (z)
log det
log
(8)
||812
The inequality is right if h is multiple of the identity function around z.
We defer the proof to Appendix B.2 We note that the above discussion relies on the invertibility of h. which, however, is not required by the resulting method. Indeed, when h is invertible at some point z, the latent reconstruction loss ensures that h is close to the identity function around z. and hence the tightness of the upper bound in Eq. (8). Otherwise, when h is not invertible at some the logarithm of the Jacobian determinant at z becomes infinite, in which case Eq. (5) cannot be optimized. Nevertheless, since |/4 (z + 6) - h (z) || is unlikely to be zero if the model is properly initialized, the approximation in Eq. remains finite, and thus be optimized regardless.
To summarize, we train the autoencoder to obtain a generative model by minimizing the following loss function:
Lipga Lpga + 7 (LAu + Emu)
We refer to this approach as maximum likelihood PGA (LPGA).
3.3 A VAE-BASED APPROACH
The original VAE is trained by maximizing the evidence lower bound on log (x) as
log p (x) log (x) KL(g (z' x) p1 x))
(10)
-Ee'ng(e' x) [log (x 22)] KL(g x) (z)),,
where p (x 2z) is modeled with the decoder, and q (z' x) is modeled with the encoder. Note that z' denotes the stochastic version of z. whereas remains deterministic for the basic PGA losses in Eqs. 2 and (3) In our case, we would like to modify Eq. [10) in a way that helps maximize log p (2). Therefore, replace p (x zz) on the r.h.s. of Eq. (10) with p (2 zz). and derive a lower bound on log p (2) as
log (2) > log (2) KLL(q (z' x) 1 (z') 2))
(11)
Eagina(z' x) [logpt (2 z')] (L (z' x) 11 (z')).
4

---

Under review as a conference paper at ICLR 2020
Similar to the original VAE, we make the assumption that q (z' x) and p (ž zz' are Gaussian; i.e. q (z' x) = (zzz! Me (x) diag (o3 (x))), and ( 2²) = N (2 /40. (z') , Here, Ho (+) fo (-), na (.) = h (-), and a > is tunable scalar. Note that if is fixed, the first term on the r.h.s. Eq. (TT) has a trivial maximum, where z, 2, and pa,o (z') are all close to zero. To circumvent this, set proportional to the f--norm of
The VAE variant is trained by minimizing
Looe Lur + L Ex-D [Ex'~g(x' x)) [log p (2 21)1 (L (z' x)
[z
(12)
where Tur and 1 are, respectively, the reconstruction and KI. divergence losses of VAF, as illustrated in Fig. 1c Accordingly, the overall loss function is given by
Lupga
Lpga + TLU
(13)
We refer this approach as variational PGA (VPGA).
3.4 A HIGH-LEVEL VIEW OF THE PGA FRAMEWORK
We summarize what each loss term achieves, and explain from a high-level how they work together.
Data reconstruction loss (Eq. TT] For Theorem I hold, need to the reconstructed data distribution (D), instead of the original data distribution (D), the target distribution. Therefore, minimizing the data reconstruction loss ensures that the target distribution is close to the data distribution.
Latent reconstruction loss (Eqs. 2 and 33: The encoder (f+) is reused to map data-space distri- butions to the latent space. As shown by Theorem minimizing the latent reconstruction loss (w.r.t. the parameters of the encoder) ensures that if the generated distribution and the target distribution can be mapped to the same distribution (H) in the latent space by the encoder, then the generated distribution and the target distribution are the same.
Maximum likelihood loss (Eqs. 66 and 7 or VAE loss (Eq. (22) The decoder (ga) and encoder (f)) together can be considered as a perceptual generative model (fo o ge), which is trained to map N (O. I) to the latent-space target distribution (H) by minimizing either the maximum likelihood loss or the VAE loss.
The first loss allows to use the reconstructed data distribution as the target distribution. The second loss transforms the problem of matching the target distribution in the data space into matching in the latent space. The latter problem is then solved by the third loss. Therefore, the three losses together ensure that the generated distribution close to the data distribution.
3.5 UNIFIED APPROACH
While the loss functions of maximum likelihood and VAE seem completely different in their original forms, they share remarkable similarities when considered in the PGA framework (see Figs. Tb and Icl. Intuitively, observe that
1 L + -BXD
L
Σ [o (x) log (o%. (x))] kE(M]]
(14)
which means both nit and L tend to attract the latent representations of data samples to the origin. In addition, L expands the volume occupied by each sample in the latent space, which can be also achieved by Ler with the second term Eq. [4].
More concretely, we observe that both Lhu and Lur are minimizing the difference between h (z) and (z + 8¹), where 5' is some additive zero-mean noise. However, they differ in that the variance of 8' is fixed for La but is trainable for L.r: and the distance between h (z) and (z + 5') are defined in two different ways. fact, Ler a squared fz-distance derived from the Gaussian assumption on whereas E. can be derived similarly by assuming that dH || h (z + 1) follows a reciprocal distribution
(d" ;a, = dH (log (b) log (a))
(15)
5

---

Under review as a conference paper at ICLR 2020
where S dd v and 0. The exact values of a and b are irrelevant, they only appear in an additive constant when we take the logarithm of p (d" a, b).
Since there is obvious reason for assuming Gaussian 2, we instead assume 2 to follow the distribution defined in Eq. (13), and multiply by a tunable scalar, y', similar to a. Furthermore, it is defined for VPGA with subtle difference that here o (x) is constrained be greater than e². As a result, LPGA and VPGA are unified into a single approach, which has combined loss function as
can replace d in Eq. with 8' (0, diag (o3 (x))
Llepge Lpga + > Lur + + + T
(16)
When >' y and rj O, Eq. 16 equivalent Eq. (9) considering that o (x) will be optimized to approach e². Similarly, when 7 = O, Eq. (16) is equivalent to Eq. (3). Interestingly, it also becomes possible to have mix of LPGA and VPGA by setting all three hyperparameters to positive values. refer to this approach as LVPGA.
EXPERIMENTS
In this section, we evaluate the performance of LPGA and VPGA on three image datasets, MNIST (LeCun et al. 1998), CIFAR-10 (Krizhevsky & Hinton 2009), and CelebA (Liu et al. 2013). For CelebA, we employ the discriminator and generator architecture of DCGAN (Radford al. 2016) for the encoder and decoder of PGA. We half the number of filters (i.e., 64 filters for the first convolutional layer) for faster experiments, while more filters are observed improve performance. Due to smaller input sizes, we reduce the number of convolutional layers accordingly for MNIST and CIFAR-10 10, and add a fully-connected layer of 1024 units for MNIST, as done in Chen et al. (2016). SGD with momentum of 0.9 is used to train all models. Other hyperparameters are tuned heuristically, and could be improved by a more extensive grid search. For fair comparison, tuned for both VAE and VPGA. All experiments are performed
single GPU.
Table 1: FID scores of autoencoder-basee generative models. The first block shows the results from Ghosh al. (2019), where CV-VAE stands for constant- variiacee VAE, and RAE stands for regularized autoencoder. The second block shows our results of LPGA, VPGA, LVPGA, and VAE.
Model
MNIST
CIFAR-10
CelebA
VAE
19.21
106.37
48.12
CV-VAE
33.79
94.75
48.87
WAE
20.42
117.44
53.67
RAE-L2
22.22
80.80
51.13
RAE-SN
19.67
84.25
44.74
VAE
15.55 0.18
115.74 0.63
43.60 0.33
LPGA
12.06 = 0.12
55.87 + 0.25
14.53 + 0.52
VPGA
11.67 0.21
51.51 1.16
24.73 1.25
LVPGA
11.45 + 0.25
52.94 + 0.89
13.80 + 0.20
As shown in Fig. 2 the visual quality of the PGA-generated samples is significantly improved that of VAES. In particular, PGAS generate much sharper samples on CIFAR-10 and CelebA compared vanilla VAES. The results of LVPGA much resemble that of either LPGA or VPGA, depending on the hyperparameter settings. In addition, we use the Fréchet Inception Distance (FID) (Heusel et al. 2017) to evaluate the proposed methods, as well as VAE. For each model and each dataset, we take 5,000 generated samples to compute the FID score. The results (with standard errors of or more runs) are summarized in Table. [1. Compared other autoencoder- based non-adversarial approaches (Tolstikhin et al. 2018 Kolouri et al. 2019 Ghosh et al. 2019), where similar but larger architectures are used, we obtain substantially better FID scores on CIFAR- 10 and CelebA. Note that the results from Ghosh et al. (2019) shown in Table. are obtained using slightly different architectures and evaluation protocols. Nevertheless, their results of VAE align well with ours, suggesting a good indeed combine the
unified approach, LVPGA best performances of LPGA and VPGA on different datasets. For CelebA, we further show results on 140x 140 crops in Fig. 5 and latent space interpolations in Fig. 6 (see Appendix CC.
comparability of the results. Interestingly,

---

Under review as a conference paper at ICLR 2020
0
7
3
9
8
5
3
0
/
9
6
4
6
1
3
3
0
4
8
6
6
1
2
1
3
9
7
6
5
6
/
/
3
5
8
O
2
3
9
7
6
3
4
9
2
1
2
4
&
4
2
4 9
1
4 8
/
8
7
O
37
9
7
6
1
2
3
3
6
4
3
7
8
0
3
5
9
0
3
7
9
9
6
9
9
[
6
/
9
3
9
4
8 3
8
4
3
3
0
7
8
0 4
9
9
7
0
0
5 9 0 6 /
9
9
1
9
/
0
2
1
5
9
5
3
9
2 9
9
9
4
9
9 3 9
9
2
7
4
3
7
2
5
0
1
4
7
4
0 7 2 9
7
/
S 1
4
9
9
6
0
8
7
6
3
6
2
9
9
4
9
8
8
4
7
6
7
0
2
7
1
O
O
6
8
7 9 0
3
1
6
6
9
4
7
0
0
7
7
6
8
7
0
6
5
6
8
4
7
0
4
3
6
2
4
7
2
9
O
2
9
3
2
7
2
9
4
5
3
0
4
4
(a)
MNIST by LPGA
(b) MNIST by VPGA
(c) MNIST by VAE
(d) CIFAR-IO by LPGA
(e) CIFAR-IO by VPGA
CIFAR-IO by VAE
(g) CelebA by LPGA
(h) CelebA by VPGA
(i) CelebA by VAE
Figure 2: Random samples generated by LPGA, VPGA, and VAF.
The training process of PGAs is stable in general, given the non-adversarial losses. As shown in Fig. 3a the total losses change little after the initial rapid drops. This is due to the fact that the encoder and decoder optimized towards different objectives,
can observed from Eqs. 4 (9) and (12) In contrast, the corresponding FIDs, shown in Fig. 3b tend to decrease monotonically during training. However, when trained on CelebA, there a significant performance gap between LPGA and VPGA, and the FID of the latter starts to increase after certain point of training. We suspect this phenomenon is related the limited expressiveness of the variational posterior, which not an issue for LPGA.
is worth noting that stability issues can occur when batch normalization (Toffe & Szegedy 2015) is introduced, since both the encoder and decoder are fed with multiple batches drawn from different distributions. At convergence, different input distributions to the decoder (e.g.. 7. and A (0, I)) expected to result in similar distributions of the internal representations, which, intriguingly, can be imposed to some degree by batch normalization. Therefore, it is observed that when batch normalization does not cause stability issues, it can substantially accelerate convergence and lead to
7

---

Under review as a conference paper at ICLR 2020
0.10
100
UNISTT
LPGA, LPGA_ MNIST
LPGA, GIFAR10
[PGA_CIFAR CIFAR- 10
0.08
LPGA, CHieAA
50
LPGA, CavbA
VPGA
CIFAR-10
T:dal Todal
VPGA, CalebA
VPCA_CaRbA CAIDA
0.04
0.02
20
Ta5
165
(a) Total loss
(b) FID
Figure 3: Training curves of LPGA and VPGA.
slightly better generative performance. Furthermore, observe that LPGA tends to be more stable than VPGA in the presence of batch normalization.
Finally, we conduct ablation study. While the loss functions of LPGA and VPGA both consist of multiple components, they are all theoretically motivated and indispensable. Specifically, the data reconstruction loss minimizes the discrepancy between the input data and its reconstruction. Since the reconstructed data distribution serves as the surrogate target distribution, removing the data reconstruction loss will result in a random target. Moreover, removing the maximum likelihood loss of LPGA or the VAE loss of VPGA will leave the perceptual generative model untrained. In both cases, no valid generative model can obtained. Nevertheless, it interesting to see how the latent reconstruction loss contributes to the generative performance. Therefore, we retrain the LPGAS without the latent reconstruction loss and report the results in Fig. 4 Compared to Fig. 2a 2d 2g and the results in Table the performance significantly degrades both visually and quantitatively. confirming the importance of the latent reconstruction loss.
4
9
4
8
5
4
5
9
9
4
4
4
7
9
1
4
2 2 2
7
A
3
6
0
8
3
6
7
2
0
4
8
I
9
5
7
O
4
6
6
5
f
7
Y
9
5
5
4
2
1
9
9
7 4
6 8 4
4
8
3
0
f
9
0
4
1
2
7
3
1 4 3
8
1
4
7
4
2
4
7
O
5
5
9
6
4
(a) MNIST, FID
18.13
(b) CIFAR-10, FID 114.19
(c) CelebA, FID = 48.02
Figure 4: Random samples generated by LPGA with a
CONCLUSION
We proposed a framework, PGA, for training nutoencoderr-asse generative models, with non- adversarial losses and unrestricted neural network architectures. By matching target distributions in the latent space, PGAs trained with maximum likelihood generalize the idea of reversible generative models to unrestricted neural network architectures and arbitrary latent dimensionalities. In addition, it improves the performance of VAE when combined together. Under the PGA framework, we further show that maximum likelihood and VAE can be unified into single approach.
In principle, the PGA framework can be combined with any method that can train the perceptual generative model. While we have only considered non-adversarial approaches, interesting future work would be to combine with an adversarial discriminator trained on latent representations. Moreover, the compatibility issue with batch normalization deserves further investigation.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, 2017.
Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto-encoders as generative models. In Advances in Neural Information Processing Systems, pp. 899-907, 2013.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference Learning Representations, 2019.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. In Advances in Neural Information Processing Systems, pp. 6571-6583, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Picter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. In International Conference on Learning Representationss, 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. In International Conference Learning Representations Workshop, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference on Learning Representations, 2017.
Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. arXiv preprint arXiv:/908 12436, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems, pp. 2672-2680, 2014.
Will Grathwohl, Ricky TQ Chen, Jesse Betterncourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. In International Confer- ence on Learning Representations, 2019.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, pp. 5767-5777, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-66337 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning, pp. 2015.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems, pp. (0236-10245. 2018.
Soheil Kolouri, Phillip E Pope, Charles E Martin, and Gustavo K Rohde. Sliced wasserstein auto- encoders. In International Conference Learning Representations, 2019.

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Yann LeCun, Corinna Cortes, and Chris C. Burges. The mnist handwritten digit database, 1998.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In International Conference Computer Vision, 2015.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and Ian Goodfellow. Adversarial autoencoders. In International Conference on Learning Representations Workshop, 2016.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations, 201 8.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Represen- tations, 2016.
Danilo Jimenez Rezende and Fabio Viola. Taming vaes. arXiv preprint arXiv:1810.0055: 2018.
Paul K Rubenstein, Bernhard Schoelkopf, and Ilya Tolstikhin. On the latent space of wasserstein auto-encoders. arXiv preprint
2018.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, pp. 2234-2242, 2016.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. In International Conference on Learning Representations, 2018.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning, pp. 1096-1 103. ACM, 2008
A NOTATIONS
Table Notations and definitions
Salgo
encoder/decoder of an autoencoder
h ===
o 9a
610
parameters the encoder/decoder
DIH
dimensionality of the data/latent space
D
distribution of data samples denoted by x
H
distribution of fø (x) for D
distribution of & = ge (fo (x)) for x ~
H
distribution of 2 = h (z) for z ~ H. standard reconstruction loss of the autoencoder
Lr
L
latent reconstruction loss of PGA for z ~ N (0, I), minimized w.r.t. e
L
latent reconstruction loss PGA for z H. minimized w.r.t.
Lg
part of the negative log- likelhoodd loss of LPGA, minimized w.r.t. Ø part of the negative log- lidel loss of LPGA, minimized w.r.t. 0
L
Lor
VAE reconstruction loss VPGA
Lokl
VAE KL-divergence loss of VPGA
byge
Lygae Ler Light VAE loss of VPGA
10

---

Under review as a conference paper at ICLR 2020
B PROOFS
B.1 THROREM
Proof sketch. We first show that any different x's generated by ge mapped to different z's by fø. Let X1 = ). X2 = 90 (Zz), and X1 # X2. Since So has sufficient capacity and Eq. (2) is minimized, we have fo (x1) Z1 X1] and fo (x2) E [z2 x2]. By assumption, fo (x1) Z (x1) and fo (x2) E Z (x2). Therefore, since Z (xi n Z (x2) = Ø, we have fo (x1) + fo (x2).
E
For .V (0, denote the distributions of go (z) and h (z), respectively, by D and H. We then consider the case where D and D are discrete distributions.
ge (z) D. then there exists an x that is generated by go. such that piz (fo (x)) = Pō (x) # Pb (x) = Pa (fo (x)), contradicting that (z) H. The result still holds when D and D approach continuous distributions, in which case D = D almost everywhere.
B.2 PROPOSITION II
Proof. Let J (z) = 0h (z) /0z. P= 101 62
81). and = J
[8]
82
8h].
where A = 06,,82,.... 51} is an orthogonal set of H H-dimensional vectors. Since det
p
del (J (z)) det (P), have
log |det (J (z)) log det (P) log |det (P)]
(17)
By the geometric interpretation of determinants, the volume of the parallelotope spanned by A is
Vol (A) = |det (P)
II |0:l12
(18)
(E(W]
where = = (1, H}. While Á = = 82.
8H is not necessarily orthogonal set, an
upper bound on Vol Á can be derived in a similar fashion. Let A.
82,, &x ). and as be
the included angle between 8, and the plane spanned by Å 1. We have
Vol (2)
81 |]
82 sin 02 and Vol (At) Vol (Ac-1) || sin ag-
(19)
Given fixed 8 2 vk [H]. Vol (A2) maximized when 02 T/2, i.e., ò1 and 82 are ( is maximized when Vol (Aa-1) maximized and O = x/2. By induction on k we can conclude that Vol (A) is maximized when À = Âh is an orthogonal set, and therefore
orthogonal: and Vol
Vol ( det (P)
II
8 [6111
(20)
<E[M]]
Combining Eq. (17 with Eqs. (18) and 220 we obtain
log /det (J (z))
<
log ||
6. log ll0.l12)
(21)
We proceed by randomizing A. Let As {81, 8,,,,,,....,54 We inductively construct an orthogonal set, A = A In step 1, 51 is sampled from S (c), uniform distribution on - 1)-sshrre of radius c, S (c). centered at the origin of an H-dimensional space. In step k. dg is sampled from S (e; Ax-1). uniform distribution
).
an (H-k)--phere, S (e; AR-1). )). in the orthogonal complement of the space spanned by A Step l is repeated until H mutually orthogonal vectors are obtained.
Obviously, when k H 1, for all j > k and j < H, (6, Ax) p p A
S (5, e; AH-) = S (6, l;; Ax). When 1 < k < H, assuming for all j A and j < H,
p (6, AR) = (6, E; Ax). we we get
/
p. (60 Ab-1)=
(6x 1 (5, Ax) dog,
(22)
((c,)-11(6,))
11

---

Under review as a conference paper at ICLR 2020
where S (e; 4k-1 U {8,1) is in the orthogonal complement of the space spanned by 1 (o))
Since p (6) A+-1) is a constant on (6x e; Ak-1), and S (e; Ax U (6;}) C (e;
(6x A+-1) is also constant on S (e; A U {,,)). In addition, dg E (cAA-1 (o,}) im-
plies that 03 E S (c; As), on which p (6j As) is also a constant. Then it follows from Eq. 22
that, for all 55 S (c; 1-1), p(; Ag-1) is constant. Therefore, for all j > k - 1 and j - H,
(Ō, Ax-1) S (0g E -b))) By backward induction on k we conclude that the marginal
probability density of òg, for all k E [L]] is p (6x) S (Ók E).
Since Eq. (21) holds for any randomly (as defined above) sampled A, we have
log det (I))) S EA
Σ
log
6
log
l(6.l22)
KEIH
(23)
[log
|| log [|0||]]
If h is multiple of the identity function around then J (z) = CI, where C' E R is constant. In
this case, Å becomes orthogonal set A, and therefore the inequalities in Eqs, 20 21). and 23 become tight. Furthermore, it is straightforward to extend the above result to the case 8 (0,
cT)
considering that W (0, I is mixture of S (e) with different e's.
The Taylor expansion around z gives
h (z (z 8) - h (z) (2) (2)
(24)
Therefore, for 5 + Or - 0, we have J J J h 6) (z). The result follows.
C MORE RESULTS ON CELEBA
In Fig. 5 we compare the generated samples and FID scores of LPGA and VAE on 140x140 crops. In this experiment, we use the full DCGAN architecture (i.e., 128 filters for the first convolutional layer) for both LPGA and VAE. Other hyperparameter settings remain the same as for 108x 108 crops. In Fig. we show latent space interpolations of CelebA samples.
(a) LPGA, FID 21.35
(b) VAE, FID 54.25
Figure 5: Random CelebA (140x140 crops) samples generated by LPGA and VAE,
12

---

Under review as a conference paper at ICLR 2020
(a) Interpolations generated by LPGA.
2
25
(b) Interpolations generated by VPGA.
(c) Interpolations generated by VAE.
Figure 6: Latent space interpolations on CelebA.
13


Reviews:
[{"review_id": "HkxXiM0zsB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main contribution of the paper is in the novel training procedure in which the target distribution and the generated distributions are mapped to a latent space where their divergence is minimised.  Training procedures for the same based on MLE and VAE are presented as well.\n\nThe idea is quite interesting but there are certain problems in clarity regarding the loss terms and the precise need for them. It appears that the work is proposing a novel framework for training VAEs where instead of comparing the target and generated distribution in the data space (the standard $L_r$), the loss is minimised by minimizing the divergence between these distributions after mapping them onto a latent space. To achieve this objective the author’s design two loss terms in addition to the standard reconstruction error. Authors provided two approaches for their training method based on MLE and VAE. Their framework is then experimentally evaluated on the standard dataset and demonstrate superior results; better-generated images and lower FID score) and provide ablation results.\n\n\nThe explanation of the Equation 2 is confusing, where the Figure-1a and the equation seem to agree but the subsequent description says that $h(.)$ should map ${N}(0, I)$ to $\\hat{H}$ but Equation-2 will bring $h(z)$ closer to ${N}(0, I)$. Is there something missing here? The notation and the derivation of the MLE approach in Section 3.3 are not clear. This section would need rewriting with clearer and compact notation for better readability. The main idea is quite clear but how this training procedure achieves that is not coming out clearly in this version. Thus I feel the paper, though has good content, is not publishable in the current format.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1e5UOxLqB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a set of losses to train auto-encoders using a stochastic latent code (PGA). The authors relate it to VAE and propose a variational variant of their framework (VPGA), but the initial intuition is distinct from VAEs.\n\nResults are presented on MNIST, CIFAR10 and CelebA and show both qualitative and quantitative improvements over VAEs.\n\nThe intuitions behind this framework seem sound and the authors added theoretical justifications when possible.\n\nThis paper seem to present a good idea that should be published, but is currently not clear enough. The writing of Section 3, especially section 3.1, need more work. The wording needs to be improved, maybe some notations are not needed, such as $\\hat{z} = h(z)$. There should be a clear description of what each loss is aiming to achieve. Currently, this is not clear. For instance, there are several mentions that h must map N(0, 1) to $\\hat{H}$, but loss (2) makes it map N(0, 1) to N(0, 1) (and I don't see other losses that would make it happen). It is likely I didn't understand it fully and a clearer section would help.\nAnother example of possible improvement would be calling the encoder and decoder with consistent names (for instance, Enc and Dec) could help, too. Currently, they are called encoder and decoder or generator in the text, f and g in the math, and theta and phi on figure 1.\n\nI am ready to change my rating after the rebuttal if the authors address clarity of section 3.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gdUc7CKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers generative models and proposes to change the VAE approach in the following way. While VAEs assume a generative model (prior on the latent space + stochastic generator) and aim at learning its parameters so as to maximize the likelihood of the (training) data distribution, the authors propose to derive the learning objective from a different view. In my understanding, they consider the two composed mappings generator(encoder) and encoder(generator) and require the first one to have the data distribution as a fixpoint and the second one to have the latent prior as a fixpoint. Starting from this idea they derive an objective function for the case that the mappings are deterministic and then further enrich the objective by either likelihood based terms or VAE based terms. The new approach is analysed experimentally on MNIST, CIFAR and CelebA datasets. The authors report quantitative improvements in terms of Frechet inception distance (FID) as well as sharper samples (when compared to standard VAEs).\n\n\nI find the main idea of the paper highly interesting and compelling. Nevertheless, I would not recommend to publish the paper in its present state . The technical part is in my view very hard to comprehend. This is partially due to the disadvantageous notation chosen by the authors. Furthermore, the derivation of the individual terms in the objective is hard to understand and the arguments given in the text are not convincing:\n- The two terms in the objective related to the fixpoint of the encoder(generator) mapping seem to enforce a fixpoint that is a mixture of the latent Gaussian prior and the encoder image of the data distribution. It remains unclear to me, why this is a good choice.\n- The derivation of the additional likelihood based terms and VAE based terms is in my view hard to understand.\nIt should be possible (and I believe, is possible) to derive a simpler objective ab initio, starting from the main idea of the authors. \n\nSide note: A close visual inspection of the presented generated samples seems to confirm the known doubts about the appropriateness of the FID measure for evaluation of generative models. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a new training procedure for generative models where the target and generated distributions are first mapped to a latent space and the divergence between then is minimised in this latent space. The authors achieve state of the art results on two datasets.

All reviewers agreed that the idea was vert interesting and has a lot of potential. Unfortunately, in the initial version of the paper the main section (section 3) was not very clear with confusing notation and statements. I thank the authors for taking this feedback positively and significantly revising the writeup. However, even after revising the writeup some of the ideas are still not clear. In particular, during discussions between the AC and reviewers it was pointed out that the training procedure is still not convincing. It was not clear whether the heuristic combination of the deterministic PGA parts of the objective (3) with the likelihood/VAE based terms (9) and (12,13), was conceptually very sound. Unfortunately, most of the initial discussions with the authors revolved around clarity and once we crossed the "clarity" barrier there wasn't enough time to discuss the other technical details of the paper. As a result, even though the paper seems interesting, the initial lack of clarity went against the paper. 

In summary, based on the reviewer comments, I recommend that the paper cannot be accepted. 

Decision: Reject