Imagining the Latent Space of a Variational Auto-Encoders
Authors: Zezhen Zeng
Jonathon Hare
Adam Prügel-Bennett
Venue:  (2020)
Keywords: VAE
GAN

Abstract:   Variational Auto-Encoders (VAEs) are designed to capture compressible information about a dataset.  As a consequence the information stored in the latent space is seldom sufficient to reconstruct a particular image.  To help understand the type of information stored in the latent space we train a GAN-style decoder constrained to produce images that the VAE encoder will map to the same region of latent space. This allows us to ''imagine'' the information captured in the latent space.  We argue that this is necessary to make a VAE into a truly generative model.  We use our GAN to visualise the latent space of a standard VAE and of a $\beta$-VAE.

Under review as a conference paper at ICLR 2020
IMAGINING THE LATENT SPACE OF A VARIATIONAL AUTO- ENCODESS
Anonymous authors Paper under double-blind review
ABSTRACT
Variational Auto-Encodesd (VAEs) are designed to capture compressible informa- tion about dataset. As consequence the information stored in the latent space is seldom sufficient to reconstruct particular image. To help understand the type of information stored in the latent space we train a GAN-style decoder constrained produce images that the VAE encoder will map to the same region of latent space. This allows us to "imagine" the information captured in the latent space. We argue that this is necessary to make VAE into truly generative model. We use our GAN to visualise the latent space of a standard VAE and of a 3-VAF.
INTRODUCTION
Variational auto-encoders (VAEs) have made a significant impact since their introduction by Kingma and Welling (2014). However, one of their perceived problems is their reconstruction performance. This has spawned wave of research into trying improve the reconstruction performance (Zhao ctt al. 2017 Dai and Wipf 2019 Larsen et al. 2016 Gao et al. 2017 Brock et al. 2017). We argue that such attempts are misguided. The whole point of VAEs to capture only compressible infor- mation and discard information specific to any particular image. This is a consequence of the well known evidence lower bound or ELBO objective function consisting of negative log-probability of generating the original image from the latent representation (this is often implemented as a mean squared error between the image and the reconstruction, although as we argue Appendix A this term should be proportional to the logarithm of the mean squared error) and a KL-divergence be- tween the probability distribution representing latent code and a 'prior distribution' (usually taken as a multivariate normal with mean zero and unit variance). These two terms have a nice inter- pretation in terms of the minimum description length (Rissanen, 1978)- this has been described elsewhere, for example, Chen et al. (2016). The KL-term can be viewed as measure of the amount of information in the latent code while the log-probability formation required to change the image produced by the decoder into the input image (see Section for details). That is, the latent space of a VAE can be viewed as a model of the dataset-capturing compressible information while not encoding any image specific information (which is cheaper to communicate using the reconstruction loss).
the image measures the amount of in-
The great strength of VAE is that it builds model of the dataset that does not over-fit (i.e. code for in-compressible features found in specific images). However, because of this it typically will not do good job of reconstructing images as the latent code does not contain enough information to do the reconstruction (for very restrictive dataset such as MNIST and Celeb-A a lot of information can be captured in the latent space, but for more complex datasets like ImageNet or CIFAR the reconstructions are poor). Of course, if you want good reconstructions on the training set then the simplest solution is to remove the Kl--divergence term and just use an autoencoder. However, having model that does not over-fit the dataset can be useful, but in this case the decoder of standard VAE should not be regarded as a generative model-that is not its purpose. we wish to generate realistic looking images we need to imagine the information discarded by the encoder. As rather simplified analogy, consider a verbal description of an image "a five year old girl in a blue dress standing on beach". If we asked different artists to depict such scene there is clearly not enough information to provide pixel-wise or feature- waie similarity between their interpretation although each artist could render convincing image that satisfies the description. In similar manner if

---

Under review as a conference paper at ICLR 2020
want VAE to act generative model need to build renderer that will imagine an image consistent with the latent variable representation.
A simple way to achieve this is using modified Generative Adversarial Network (GAN). We call such a model a latent space renderer-GAN (or LSR- -R-N. To generate an image we choose a latent vector from the prior distribution for the VAE. This is passed to generator network that generates an image, é, with the same dimensions as that of the dataset used to train the VAE. The generated image has both to convince a discriminator network that it is a real image-as usual for a GAN (Goodfellow et al. (014)) the same time the VAE encoder should map & close to z. To accomplish this we add an additional cost to the normal GAN loss function for the generator (LGEN)
LGEN Alog
where (=(+()) is the probability distribution generated by the VAE encoder given an image & and is the latent vector that was put into the GAN generator. Note that when training the LSR-G LSR-GAN we freeze the weights of the VAE encoder. The constant À is an adjustable hyperparameter providing a trade-off between how realistic the image should look and how closely it captures the information in the latent space. This modification of the objective function can clearly be applied to any GAN or used with any VAE. Although the idea is simple, it provides a powerful method for visualising (imagining) the information stored in a latent space. Interestingly, it also appears to provide a powerful regularisation mechanism to stabilize the training for GANs.
Combinations of VAEs and GANS are, of course, not new (Makhzani et al. 2016; Larsen et al.) 2016 Brock et al., 2017 Huang et al. 2018: Srivastava et al. 2017). In all cases GANs have been combined with VAES to "correct" for the poor reconstruction performance of the VAE (see Appendix for more detailed discussion of the literature VAE-GAN hybrids). As have argued (and expound on in more detail in Section 3), we believe that the decoder of a VAE does the job it is designed to do. They cannot reconstruct images accurately, because the latent space of VAE loses information about the image, by design. All we can do is imagine the type of image that a point in the latent space represents.
aware of
In the next section, we show examples of images generated by the LSR-GAN for both normal VAES and B- VAEE (we also spend time describing VAEs, 3-VAEs and the LSR-GAN in more detail). In addition, in this section we present a number of systematic experiments showing the performance of a VAE and LSR-GAN. why we believe a VAE is doomed to fail as generative model. We conclude in Section[] We cover more technical aspects in the appendices. In Appendix A we show that the correct loss function for a VAE requires minimising a term proportional to the logarithm of the mean squared error. In Appendix B we draw out the similarities and differences between our approach to hybridising VAEs with GANs and other work in this area. We present some additional experimental results in Appendix C A detailed description of the architecture of LSR-GAM is given in Appendix D We end the paper with Appendix [E] by showing some samples generated by randomly drawing latent variables and feeding them to the LSR-GAN.
Section 3 we revisit the minimum description length formalism to explain
IMAGINING LATENT SPACES
A natural question ask is what information about an image gets represented in the latent space of a VAE. To answer this we can use the VAE encoder to generate a distribution (a(2(az (()) representing that image in the latent space (see Sections 2.1 for details on VAEs). From this distribution we can sample points in the latent space and feed this to the -ANN generator. We show examples of this for both CIFAR- 10 and ImageNet (down-sampled to 64 x 64) in Figure [ In all cases in this paper the input images are taken from a test set that is independent of the training set. Note that both CIFAR-I0 10 and ImageNet are "hard" for VAES in the sense that they represent extremely diverse sets of images. As consequence, the VAE latent space will struggle to store detailed information about the images and the VAE reconstructions will be poor. We have repeated this for a 3-VAE (see section 2.3 for full description of 3-VAEs). We note that there is very little variation between the different samples drawn from g (z) ()) particularly for the standard VAE (5- = 1), showing that the latent space of the VAE is relatively smooth (there is more variation when B 20).
2

---

Under review as a conference paper at ICLR 2020
Input
Generated images based Latent representation
A=1
8=1
CIFAR-10
3=20
3=20
8=1
ImageNet
64x64
B=20
Figure 1: Images generated by our GAN from latent representations of an input image (test images) based on different value on CIFAR-10 10 and ImageNet, the first column is the original image.
To get a sense of the variation in the information stored in latent spaces we show in Figure 2 input- output pairs, where the left image is the input and right image is the output generated by the LSR- GAN generator seeded with latent vector encoding of the input image. The reconstructions capture the shape and background, but clearly loses lot of detail. In some cases appears that the type of object is being captured, although in the case of the boat with the 3-VAE (with 8 20) the wrong object is being rendered.
2.1 VARIATIONAL AUTOENCODERS
The structure of VAE is represented schematically below.
D
Encoder (parameless +)
(Me, oz))
Decoder (paraneeeess 0)
Do(z)
We sample input from some dataset, D. To be concrete will consider the where the inputs are images, although clearly a VAE can be used to represent many different types of data. For each input the encoder outputs mean vector, μt, and standard deviation vector, or, that describes an axis aligned normal distribution, 20 (2|=) = .V (a)) diaglo (z))). (2))( A latent
variable z sampled from this distribution and then fed to a decoder. For simple black and white datasets such MNIST the decoder outputs scalar each pixel location that can be interpreted as the probability that the pixel black. For more complex datasets the decoder ususal generates a "reconstruction" f Dz(z). The probability of generating pixel value is then usually taken as

---

Under review as a conference paper at ICLR 2020
CIFAR-10
ImageNet
64x64
Figure 2: Examples of input-output pairs of images. The left image is an image from the test dataset. The right image the image generated by the LSR-GAN seeded with a latent vector encoding of the input image (3 1)
4

---

Under review as a conference paper at ICLR 2020
Input
CIFAR-10
VAE
LSR-GAN
Figure The comparison between reconstruction images VAE and LSR-GAN on test dataset.
normal distribution with mean è; (i.e. (=(z;|z)= the expected size of the errors between the input images, æ, and the reconstructions,
(2 o²)) and variance o² that measures
The loss function for a VAB is equal to the negative evidence lower bound (ELBO)
L BgwD (z-oo(((a)) [log (pe [<]=)) + KL (g0(2)=))|
(2)
As explained in Appendix A log (=)()))) is chosen to be proportional to the logarithm of the reconstruction error between and the input image a x-in our experiments this produced better reconstructions than replacing log (po(z)l)) with the mean squared error.
2.2 LSR-GAN
LSR-GAN is novel hybridization of VAE and GAN model. The most distinet difference of LSR- GAN from previous work is that it is a two-stage model. In the first stage we train the VAE model. Having done this freeze the weights of the VAE and train the GAN. We train the discriminator, D, of LSR-ANN in the same way as a normal GAN. That is, we minimise a loss function
D = Ex log (D(=))] - Ez log (1 - D(G())))
(3)
where the generator or the decoder of LSR-GAN. The job of the discriminator, D is, to decide whether its import is a real image not. Thus, to optimise the loss function neet maximize the log- prg-obbabiliity of passing the real data, æ, while minimising the log-pr log-probability of accepting random sampling G(z) generated by generator G seeded with random latent vector The architecture of the generator the same as that of a normal GAN but the loss function is slightly different. We add additional term giving
fG (lgg (D(G())))) + log (log())(()((2)).
The parameters of the discriminator and generator trained in the usual tick-tock fashion using gradient descent. We built the VAE and the generator of GAN using ResNet (He et al.) 2016) as gave slightly better performance than using standard CNN. The architecture of the discriminator is the same as DCGAN (Radford et al. 2016). The architecture is described in Appendix D
To test the LSR-GAN we the VAE generate latent representation for image drawn from an independent test set. The latent vector is then used as a seed value for the generator in the LSR- GAN. The LSR-GAN can get sharper reconstruction images than VAE (see Figure 3). Although not visually so obvious, we have used a quantitative measure of sharpness computed as luminance- normalised Laplacian (San Pedro and Siersdorfer 2009 Section 3. 3.2) For the reconstructed images from the VAE we obtained a measure of 0.17 0.03 while for the LSR-GAN we obtain 0.28 + 0.08 (i.e. an improvement of factor of two). We have also computed the FID measure of image quality for CIFAR-10 (Heusel et al.) (017). For images seeded from testing example the VAE achieved score of 89.8 while LSR-GAN achieved a score of 44.1 1, while for images seeded with random latent variable (i.e. z (O, I)) the FID score for the VAE is 138.6 while for the LSR-GAN it is 47,4. This should not be surprising. The decoder of the VAE is training only where there are training images. Despite the fact that the KL-divergence tries ensure that much latent space possible used, the constraint of minimising the reconstruction loss means that most of the latent space is far from training example. Although the VAE does not do badly generating testing examples,
5

---

Under review as a conference paper at ICLR 2020
Input
CIFAR-10
8=0.1
3=20
Figure 4: Examples of input-output pairs of images for different values for 3-VAE The first row is from the test dataset. The second and third rows are Others reconstruction generated by LSR-GAN for VAEs trained with different / values.
these tend to be substantially closer in the latent space to the training examples than random samples. In contrast the LSR-GAN is trained on random samples that the generator will have to produce "realistic" images over the whole latent space. Of course, whether these generated images represents anything recognisable open to question. For diverse training sets such as CIFAR- 19 and ImageNet this may be very difficult. What image should expect from latent vector halfway between we show images generated by seeding LSR-GAN with random latent variables for CIFAR-10 10, ImageNet, MNIST and Celeb-A.
truck and bird? In Appendix E
2.3 BETA-VAB
A goal of generating a latent representation is for the representation to be disentangled. Intuitively disentanglement seems clear: We would want information that somehow causally independent to be encoded into orthogonal directions (or different variables) in our latent space (Bengio et al. 2013). Unfortunately, this is not only quite difficult to achieve practice (at least, in an unsupervised setting), but it is even difficult to formulate (see Locatell et al. (201 8)). Despite this difficulty, there have been many attempts to achieve disentanglement (Kim and Mnihj 2018 Chen al. [0188] Burgess et al. 2018). One of the most prominent has been the 3- 3-VAE introduced by Higgins et al. (2177), where the KL-divergence term in a normal VAE is weighted by parameter 3
L EEEDD [Ez=ga(=(a) [log (po (z=]))) B KL (==())=)])V((0
The argument is that by making 3 > 1 encourage disentanglement. Contrariwise, by making < 1 we make a VAE closer to an auto-encoder. This improves the reconstruction performance on the training examples, but at the cost of allowing the latent space to over-fit the training set.
In Figure 4 we show examples of input- pairs for different values of B. We observe that for large B3 the outputs are quite different from the input images in contrast to small / where many more details of the original input are captured.
Although the LSR-GAN model generates slightly clearer, less blurry, images, it has lower recon- struction error than the VAE decoder. We show the mean squared error measured on a testing set from CIFAR-I0 surprising, it uses the same information as the VAE (i.e. the information stored in the latent space). By producing sharper images it will pay the price of getting the boundary wrong. The blurry edges from the VAE is a way to hedge its bet and reduced the mean squared eтror. Interestingly, the mean squared error remains fairly constant as we increase from low value, until we reach which it rapidly increases. One interpretation of this fact is that the VAE with 3 = 1 is successfully encoding all the useful information (i.e. compressible information) so for reconstructing unseen im- ages it will perform well as auto-encoder. As we increase above 1, the reconstruction error increases rapidly.
function of in Figure 5 a). This poor performance of the LSR-GAN is un-
1 after
In Figure b)b) we show the classification performance measured by simple classifier trained on the CIFAR-10 training set. The classifier performance achieved an 84% correct classification on the raw images. We find little variation the classification accuracy falls off. Again we can attribute this to the latent space of the VAE (with 1) capturing most useful information. Interestingly the high-8 VAE fails capture
decrease B below 1. As we increase 3 above

---

Under review as a conference paper at ICLR 2020
Mean Square Error
Absolute Classification
LSR-GAM GRI-GAN
VAZ
250
Alanch Clamsiiccaton
130
100
LSR GAN
VAB
0-
-1.5
-1.5
-1.0
-0.5
1gg100)
Ing1000)
(a)
(b)
Figure 5: Performance the VAE (blue points) and LSR-GAN (red points) versus B. In (a) we show the means squared error, while in (b) we show the classification performance using a classifier taking the reconstructed images. The images are taken from CIFAR-10. 10. Error bars show #1 standard error.
"objectness"" well. This suggests that, at least for CIFAR-10, the type of object does not contain very much information about its appearance and rapidly discarded.
3 MINIMUM DESCRIPTION LENGTH
To understand what VAEs do it is useful to interpret them in the framework of the minimum de- scription length (MDL) formalism. In MDL we consider communication a dataset D through communication channel using as few bits as possible. We can do this using lossy compression, where we encode each input by a code z, which we communicate down our channel. The receiver decodes the message and produces an approximation of the input å. To communicate the original information we send the code z together with the error e = x- - between the input æ and the recon- struction æ. Because the distribution of errors, p(E), is more sharply concentrated than the original inputs, p(x). this method allows us to communicate the image more efficiently than transmitting the raw pixel values. The expected cost of transmitting input
L = ER-D [M (z) E(e)]
where Mz)) is the number of bits needed to communicate the code, 2, and E(e) is the number of bits required to communicate the error, e. the MDL. formalism we attempt to find code that minimises the description length C. To communicate the model and errors need to use an optimal coding strategy. Rather than specifier and actual code we can use the Shannon bound (i.e. the negative log- pee-babbbbily of the tokens transmit). For this to be meaningful, both the errors and code to a finite precision. The precision of the errors will determine the accuracy of the data we communicate. If the 7th component of the error is distributed according then the cost of communicating the error to a precision of A is approximately - log (p(e:) A)
need to specify
p(e:)
loo (p(+))) - log(A) A). The factor log( A) is common all coding schemes so irrelevant to choosing optimal codes z. In contrast the precision to which we transmit the model will directly determine the cost . (z). There a balance to struck: more precise model can potential lead to better reconstruction a reducing the reconstruction cost, F(()), but at the same time increasing the cost, (2), of communicating the code 2.
The KL-divergence term, KL (a(=)]lp(=) (also known the relative entropy) can be interrupted as the communication cost (in nats) of transmitting a random variable z with uncertainty given by q(z) assuming an underlying probability distribution of all random variables of p(z). Using this interpretation we see that the loss function of a VAE is equivalent to the expected message length (in nats) of communicating sample from the dataset D by using random variable with uncertainty (2)). By minimising the loss function we find coding scheme with the minimum description length (or, at least, an approximate local minimum). By encoding a message as a random variable drawn from a distribution Uo (z)z) (z(z) x)) the VAE is able to find optimal balance between accuracy to which it transmits the model (determined by the standard deviation vector, o, generated by the VAE encoder) and the need to reduce the reconstruction error. From MDL perspective the ELBO the correct objective function, and should not be regarded as a approximate lower bound to what really want to achieve. If there too many dimensions in the latent space then some of the
7

---

Under review as a conference paper at ICLR 2020
components of (channel in information theory terms) are such that 2, is approximated distributed by (2, 1) for all inputs æ. The channel is effectively "switched off" (and it will be ignored by the decoder as it is just a source of random noise). This is referred to as latent variable collapse and is sometimes viewed as problematic, however, from the MDL viewpoint it acts as an elegant automatic dimensionality selection technique.
The job of the decoder in a variational autoencoder is to reconstruct the image only using information that can be compressed. Image specific information is ignored. For example, information about the precise shape of an object is probably not compressible. As a result the decoder tends to hedge its bets and has blurry outline. Of course, some encoders and decoders will be better than others, but to date there is little evidence in the literature that the performances of VAEs are massively sub- optimal, at least, when working with images. With an extremely powerful encoder and decoder and a limited dataset it would be possible for the encoder to communicate an identifier of the input image and for the decoder to reproduce the image just from the identifier, thus avoiding communicating any information about the visual content of the image-this requires that the decoder memorises all the images. This would be an extreme case of what is sometimes called posterior collapse. There is some evidence that with very strong encoders and decoders that the amount of information stored in the latent space (as measured by the KL-divergence) decreases (Bowman et al.j [2015). This might point to weakness of the VAE set-up-the the MDL set-up really only makes sense when the dataset is arbitrarily large-, but this problem could be ameliorated by data augmentation. However, using standard CNN encoders and decoders we found no evidence for memorisation of the images (for example, the VAE would produce a similar level of reconstruction for images from a separate test set). For language modelling there seems be more evidence that VAEs often fail extract information in the latent space, but for images it seems likely that properly trained VAE will extract a good fraction of the compressible information. We believe that the failure of the VAE decoder to produce high quality reconstructions (except in the case very of simple datasets such as MNIST and possibly CELEB-A) is because to do so would require communicating information that non- compressible. As consequence we should not think of the decoder of VAE a generative model: will, by design, produce blurry and poor quality reconstructions. We want this to ensure that the latent space only captures information that is common across many images. We see the mapping from images to latent space as a many-to-one mapping. Thus, the mapping from the latent space to images will be ambiguous and the best variable: exactly what we have designed the LSR-GAN to do.
do is imagine an image compatible with the latent
4 CONCLUSION
VAEs are often taken to be pauper's GAN. That is, a method for generating samples that is easier to train than a GAN, but gives slightly worse results. If this is the only objective then it is clearly legitimate to modify the VAE in anyway that will improve its performance. However, we believe that this risks losing one of their most desirable propertics, namely their ability to learn features of the whole dataset while avoiding encoding information specific particular images. We have argued that because of this property, a VAE is not an ideal generative model. It will not be able to reconstruct data accurately and consequently will struggle even more with generating new samples. One of the weaknesses of the vast literature on VAEs is that it often attempts to improve them without regard to what makes VAES special.
As we have argued in this paper, a consistent way of using the latent space of a VAE is to use a GAN as data renderer, using the VAE encoder to ensure that the GAN is generating images that represent the information encoded in the VAE's latent space. This involves "imagining" the information that the VAE disregards. LSR-GAN can be partieularly useful in generating random samples, although, as shown in Appendix [ for very diverse datasets the samples are often not recognisable as real world objects. Although there are already many VAE-GAN hybrids, to the best of our knowledge, they are all designed to "fix" the VAE, In our view VAES are not broken and "fixing" them is actually likely to break them (i.e. by encoding image specific information in the latent space). Although, the main idea in this paper is relatively simple, we believe its main contribution is as a corrective to the swath of literature on VAEs that, in our view, often throws the baby out with the bath water in an attempt to fix VAEs despite the fact that perform in exactly the way they were designed to.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Confer- on Learning Representations, 2014.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. InfoVAE: Information Maximizing Variational Autoencoders. arXiv:1706.022 170.00226222 02262 [cs, stat], June 2017. URL http://arxiv.org//at
1706. 02262
arXiv: 1706.02262.
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In International Confer- Learning Representations, 2019. URL htttp:://operreee nww.neee/eeee forum?id-
Ble0X3C9tQ
Anders Boesen Lindbo Larsen, Sren Kaae Snderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using learned similarity metric. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1558-1566, New York, New York, USA, 20-22 Jun 2016. PMLR. URL 1arsen 6. html
Fei Gao, Yi Wang, Panpeng Li, Min Tan, Jun Yu, and Yani Zhu. Deepsim: Deep similarity for image quality assessment. Nearocomputing, 257:104-114, 2017.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. In International Conference on Learning Representations, 2017.
J. Rissanen. Modeling by shortest data description. Automatica, 14(5):465-471 Septem- ber 1978. ISSN 0005-1098. doi: 10.101660005-0000(3330000000 URL http: //www. sciencedirect http:////e.aeeeeeeerrr//////////
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational Lossy Autoencoder. arXiv: 1.27311111 [cs, stat], November 2016. Utt.htt:: http:////aaawwwoooo/////////////
ottp///////////1/./////// 02731 arXiv: 161 1.02731.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. 17Xiv:1406.260 [cs, stat], June 2014. URL
arXiv: 1406.2661.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. In International Conference on Learning Representations, 2016.
Huaibo Huang, Ran He, Zhenan Sun, Tieniu Tan, et al. IntroVAE: Introspective variational autoen- coders for photographic image synthesis. In Advances in Neural Information Processing Systems, pages 52-632 2018.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in GANs using implicit variational learning. In Advances in Neural Information Processing Systems, pages 3308-333 3318--318, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pages 770-778, 2016. URL https: : www. www..c--ounndtiio..oo
Residual openaccess/contentt _epretrtt___e_____eeeee ccvpr_2016///___eeeeeeeeeee 2016 ht11//tml/////mmmm htt//ttm//Heeeeeeee _ Learning_ 2V16 paperrhppppphhmmm
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Repre- sentations, 2016.
Jose San Pedro and Stefan Siersdorfer. Ranking and classifying attractiveness of photos in folk- sonomies. In WWW '09: Proceedings of the 18th international conference on World wide web, pages 771-780, New York, NY, USA, 2009. ACM. ISBN 978 978-1-60555-487-4 hddb..cc..ogy111.115555566000 cfm2id==1526799.11 122126699999.929.
doi: http:
URL bttp://poxtal. acw. http://ta.....c...ooct.....o....

---

Under review as a conference paper at ICLR 2020
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. CANs trained by a two time-scale update rule converge to local Nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6637, 2017.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on paltern analysis and machine intelligence, 35(8):1798-1828, 2013.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Berharrd Schlkopf, and Olivier Bachem. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. arXiv: 12359 [cs, stat], November 2018. URL http:
12359
arXiv: 1811.12359.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings Machine Learning Research, pages 2649-2658, Stockholmsmssan, Stock- www.eiiiinn....p.po..... mir pt..e///ssss........//.ss//
holm Sweden, 10-15 Jul 2018. PMLR. URL
kiml html
Tian Qi Chen, Xuechen Li, Roger Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. In Advances in Neural Information Processing Systems, pages 2610-2620, 2018.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in 3-VAE. arXiv preprint arXiv:1804.03599, 2018.
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. 8- B-V:E: Learning basic visual concepts with a constrained variational framework. ICLR, 2017.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from continuous space. arXiv preprint arXiv:1571.06344 2015.
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie L.. Mode regularized generative adversarial networks. In International Conference on Learning Representations, 2017.
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International Conference on Machine Learning, pages 1718-1727, 2015.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International
Conference on Learning Representations, 2015.
10

---

Under review as a conference paper at ICLR 2020
Supplementary Material
A ON THE ELBO
In the standard VAE maximise the prg-prbbability of generating the original image. In the origi- nal paper this was achieved by the decoder outputting a probability distribution akin to what happens in the latent space. More often it is assumed that the pixel errors normally distributed with some variance x². Thus the log-prote prgpobbbaylliyy of generating all the images is
N
(x. 2;)
N
(e -
Σ log (N(x)(,
log (2xx²)
2x²
sin D
where the sum is over all predicted pixels-is i.e. the number of pixels in an image times the number of colour channels times the number of examples (or, more usually, the mini-batch size). However,
0²
Σ
x 2;)²
N
(at least, if we make the natural assumption that the errors have mean zero). As a consequence
N N log (2TTa))
Σ (aggetta
2
so that we should minimise N log (o²) /2. In information theory terms this tells us that it cheaper to communicate the residues if they are more tightly concentrated. Note that since o² is proportional to the mean squared error, EMSE. suffices to minimise N log (EMSE) /2. We note that
θ N
313 24
log (2mT²)
ai,
which is precisely the gradient of
N
[ B.
2n²
1=1
we ignored the dependence of o² on f... In many publically available implementations of VAEs the algorithm minimises Σ (z. (2,,-21) 22 which arbitrarily assumes o² = 2 rather than its true value. This means that these implementations are effectively rumning a B-VAE with some unknown 8 (in our experience with B > 1). This makes comparing results from different VAE implementations difficult. For example, rescaling outputs to lie in the range [-1,1] rather than [0, 1] would change the effective 3-value.
B VAE-GAN HYBRIDS
The hybridisation of VAE (or autoencoder) and GAN models have been developed for several years. There are many attempts this area and compare LSR-GAN the most related work in this section.
The adversarial types autoencoder is the most intuitive and simplest way combine a VAE an au- toenedoer and a GAN models, Most of these models introduce discriminator into the autoencoder training. AAE (Makhzani et al., 2016) applies discriminator to distinguish the output of encoder and the random sample from the prior distribution. It uses this discriminator to replace the KI. term in VAE. VAE/GAN (Larsen et al., 2016) is the first model that applied feature-wise errors and the input of its generator contains three different types images: the reconstruction images, the generated images and the real images. The same as our model, it collapse the decoder and the generator into one. MDGAN (Che al. 2017) is another AE-GAN hybrid which is close VAE/GAN, they try to match the manifold of GAN to real data by adding a geometric metrics regulariser and mode regulariser. None of these methods feed the output of generator back into the encoder train their
11

---

Under review as a conference paper at ICLR 2020
network in two-stages, which is the biggest difference between these methods and ours. Also, many of these hybrid models adopt an autoencoder instead of VAE while the VAE in our model cannot be replaced by autoencoder.
There are not many models that use the output of decoder to feed the encoder. The Introspective Adversarial Network (IAN) (Brock et al. 2017) is unified model which means the discriminator is not separate. IAN only encodes the feature that extracted by discriminator rather than the raw images. The discriminator of IAN extracts features from both raw images and synthetic images. The generator accept both random sample and the output of the discriminator as inputs at the same time. In contrast, our models only accept one imput. Another model that adopts the introspective method is Intro (Huang et al. 2018), it constructs the inference model E and generator model G in a circulation loop. Intro VAE has the ability to generate high-resolution images. But it does not contain any discriminator network.
The most closely work to our -SR-AAN is VEEGAN (Srivastava et al. 2017). It introduces a second network Fg to the GAN. The task of Fe is to map both the real images and synthetic images to a Gaussian distribution which what we ask the encoder to do. When the input of Fa is the output of generator, the objective function minimise the distance between the input of generator and the output of Fg. If the input of Fg is real data, the objective function minimise the cross entropy between Gaussian prior and the output of Fg. Another related model is the Generative moment matching networks (GMMN) (Li et al. 2015). In this model the autoencoder is frozen and they then minimize the maximum mean discrepancy (MMD) between the generated representation and data representation, and they use an uniform prior to generate the representations. In LSR-GAN, match two Gaussian distributions in maximizing the probability distance. None of these related works are two-stages models except GMMN. Also, to the best of our knowledge, LSR-GAN is the first WAE-GAN hybrid model that applies the probability distance in the loss function.
C ADDITIONAL EXPERIMENTS
We briefly present some additional experimental data.
C.I DEPENDENCE OF LSR-GAN ON
In Table we present measurements of the performance of outputs from both VAEs and LSR-GAN for different values of 3. Some of this data is also presented graphically Figure 3 but we have included additional measurements.
Table 1: The measurement for different 3 values. Variance is the variance among images generated by same latent representations. Absolute accuracy is the rate that classifier classifies reconstruction images right. Relative accuracy is the rate that classifier classifies reconstruction images the same raw images.
ALDit
A.M
ALDS
ALs
Ruld
A:15
0.20
0.86
Cailague
136-7
0.3
91:00:0:113
0.34
Varazce Varaee11.SB-GAAN (LS884IAA)
Modii Litlape
0M1 426 -31
021:2
1.7
438.33 3.00
40
4820.33 47
0204 18.23
Aleeate Clasifiation (VAE)
41,0 = 0.20%
45050015
4500 00.05
25.1740.185
21.06500.003
15.29 = 1818
Abestaee Eluchhatvvn (I SIL-GAN
Modal Lillagu
11.39 1138
13/05 199611.200 00205
47.09 1..18
10.33 1
322661.0.111
37:43 1 1121%
25.15 0.17%
Relaire Clasifiction (WAE)
45,00 0.21%
44.1 0200
43.38
43.00 0:3355
25.84:201008
21:00:00:210
18.00 s
10,50 0.215
Rehaive Classficaion GA58))AN)
Mosic Collagic
45.22 1.18%
48 0:3305
43.54
07.10:20.117
33:08 11220%
$1.91
C.2 DEPENDENCE OF LSR-GAN ON A
The performance of the LSR-GAN depends on the hyper-parameter X. This balances the need to produce convincing images (from the discriminator's point of view) with the requirement that the latent space of the GAN should be close to that for the VAE. These two objectives are not necessarily contradictory, although we will see that changing À has benefits and drawbacks.
In Figure 6 we show the effect of changing À over approximately three orders of magnitude on (a) the absolute classification accuracy (b) the classification accuracy compared the class labels
12

---

Under review as a conference paper at ICLR 2020
The measurement for different À values. Variance is the variance among the images gen- erated by same latent representations. Absolute accuracy is the rate that classifier classifies recon- struction images right. Relative accuracy is the rate that classifier classifies reconstruction images the same as raw images.
Table 2:
J-0.01
Aw0.1
AwI15
Aw1
Aed
Am10
A-15
A=20
Mean Square Eince
255.90 =: 1.30
251.34 4: 1.14
150.72 0.50
163.06 Je 0.73
104.15 22 0.46
90,06 11.43
86.81 = 0.42
54.09 0,40
0.25
Absolutc Classiliaation
55.52 + 0.2356
45.2 00.21%
46,96 + 0.285
47.19 0.18%
67.21 + 0.2055
47.45 0.18%
47.89 0.21%
46.72 0.235
Relative Clavilication
36.37 : 0.185
$6.40 0.20%
48.05 0.2156
48.41 0.2000%
68.60 0.1595
48.72 0.30%
48.79 = 0.28%
50.22 0.215
predicted by the classifier the raw images (c) the squared reconstruction error and (d) the variance in the predictions when choosing different samples from gop (z). We see that increasing improves the classification performance (both relative and absolute). However, and perhaps surpris- ingly, increasing À produces a significant reduction in the reconstruction error. More intuitively also causes reduction in the variance between images sampled independently from (o (z)az). That is, using the encoder in the LSR-GAN acts regulariser ensuring close by points in latent space map similar images. More details are given in Table 2
Absolute Classificatio
Relative Classification
Absonbe Clanofication Accuncy
Clasuitcatton
10-
-20
-15
-1.0
-0.5
05
100
-2.0
-1.
-1.0
-0.5
00
log100A)
kg2000)
(a)
(b)
Square
Variance Images from Representation
250
60
200
Vean
20
30
10-
-20
-15
-1.0
-0.5
00
05
10
-2.
-3.5
-1.0
-0.5
00
05
10
log10K)
kgg00))
(c)
(d)
Figure 6: Graphs showing the classification performance of images generated by our GAN with different values (on test dataset). The left-bottom graph shows the mean square error between reconstruction images and imagination images. The right-bottom graph shows the variance between images generated by our GAN from latent representations of an input image. The x-axis the log value of different A, the errors are too small which make error bars look like lines.
13

---

Under review as a conference paper at ICLR 2020
D ARCHITECTURE OF THE LSR-GAN
In this appendix we describe the detailed architecture of the VAE and LSR-GAN we used. Table describes the structure of the VAE's encoder and decoder and the GAN's generator and discriminator networks. The encoder and decoder/generator are based on a ResNet. The ResNet block structure
shown in Figure 7 Both networks are optimized using Adam (Kingma and Ba 2013) with
learning rate of 2 x 10 4 and 31 = 0.5. The code we used to implement the models is available at
https://gicub..comiic wotb.ccccc/cc/22000000 zzz/LSR LAR-
Table 3: Architectures for the three networks LSR-GAN, BN denotes batch normalization
Encoder
Decoder(Generator)
Discriminator
ResBlock down 64
512 FC Network, BN, ReLU
x 4, stride=2, padding=1,
32 CNN, LeakyReLU
ResBlock down 128
ResBlock up 128
x 4, stride=2, padding=1,
64 CNN, BN, LeakyReLU
ResBlock down 32
ResBlock up 64
x 4, stride=2, padding=1,
128 CNN, BN, LeakyReLU
128 FC Network, BN, ReLU
ResBlock up 32
x 4, stride=2, padding=l, CNN, Sigmoid
128 FC Network
1, CNN, Tanh
Conv
ReLU
Conv
ReLU
Avgpool
Figure 7: ResBlock architecture.
14

---

Under review as a conference paper at ICLR 2020
E SAMPLING
In this appendix we show sample images generated by LSR-GAN starting with random seed (O, I). These shown in Figure 9 for LSR-GAN trained CIFAR-10 and Ima- geNet. Although the images superficially look reasonable on close inspection it is clear that most samples for the LSR-GAN trained on CIFAR- 10 and ImageNet are not real world objects. This flects the fact that the images for these two dataset are very variable leaving most of the latent space representing rather surreal objects.
Figure 8: Random samples generated by LSR-GAN trained on CIFARIO and ImageNet with 3 =1
and A=
15

---

Under review as a conference paper at ICLR 2020
We have also trained LSR-GAN GAN MNIST and Celeb-A with samples shown in Figure Perhaps unsurprisingly, most samples are identifiable.
395
8
5
2
7
7 4
7
4
497380
3
0
0 5
3 8 8
0 8 9
S 2
3 3877
9
7
4
3
6
8
2
4
Figure 9: Random samples generated by LSR-GAN trained MNIST and Celeb-A with 8 =1 and
d=1.
16


Reviews:
[{"review_id": "Syg8gRa6tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new method for improving generative properties of VAE model. The idea is to train VAE in two stages: at first, train the vanilla VAE, then at the second stage freeze the encoder part and train the decoder part as a GAN generator with an additional regularizer which encourages cycle consistency in the latent space. Also the authors claim that other VAE-GAN hybrids which try to improve VAE model are “misguided” and poor samples and reconstructions of VAE are the consequence of minimum description length problem. \n\nConcerns:\n1) The main concern about this paper is the inaccuracy and very general statements without theoretical or empirical justification. For example, the authors claim that “the whole point of VAEs is to capture only compressible information and discard information specific to any particular image”. What is the definition of “only compressible information” or “information specific to any particular image”? Is there an experiment which can support this statement? Other examples of such general statements: “strength of a VAE is that it builds a model of the dataset that does not over-fit”, “the latent code does not contain enough information to do the reconstruction”, “VAEs are not broken and “fixing” them is actually likely to break them”. \n2) Poor experiment comparisons with other baselines. The authors compare their method only with the vanilla VAE which is clearly insufficient. The authors claim that other VAE-GAN hybrids break the “strength of the VAE”. Could you please provide examples of problems and provide experiments where VAE-GAN baselines will be worse than VAE or the proposed method?\n3) The paper structure is very confusing. The main part and experiments part are mixed. Therefore, it is hard to follow the text. The experiment setup is not clearly stated. For example, it is unclear for which dataset luminance-normalized Laplacian was computed. \n\nOverall, the paper proposes a new method and  gives an alternative view on the VAE model. However, statements from the paper are very pretentious and are not rigorously proven. Also there are not empirical comparisons with other VAE-GAN hybrids. Therefore, I would suggest rejecting the current version.\n\n--------------------------------------------------------\nUpdate after author rebuttal\n\nThank you for your thoughtful response. However, I still think that the paper does not give new insights about VAE model and has poor experiment justifications of their statements. Considering MDL interpretation of VAE it is not new (see [1]). Therefore, the contribution of this paper is very limited. \n\nAfter reviewing the other reviews and the author rebuttal, I do not change my original score.\n\n[1] Xi Chen et al.,  Variational  Lossy  Autoencoder, 2016\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Skg_RY4tdB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to augment the VAE objective with two additional terms: (1) a GAN-like objective that ensures that the generated samples are not distinguishable from real samples, and (2) an additional regularizer to make sure that the latent variable that generated the image can be reconstructed from the VAE encoder. \n\nOverall the paper is written very well and was a pleasure to read. In particular I appreciated the differences between the present work and the many VAE/GAN hybrids in section B of the Appendix. However, after reading the paper a couple of times, it was still not clear to me what the \"main point\" of the paper is. To be more specific:\n\n- From the perspective of obtaining good unconditional samples from a VAE-like model, the authors do not compare their approach against like methods like IntroVAE (https://arxiv.org/pdf/1807.06358.pdf) which also adds an adversarial objective to the VAE. Qualitatively at least, it seems apparent that the generated samples here are worse in quality that the IntroVAE work.\n\n- From the perspective of learning more faithful reconstructions, it seems clear from Figure 5(a) that a regular beta-VAE does better. And to me it is not clear why we would want faithful reconstructions?\n\n- The main novelty of the proposed method is in adding an extra term to the generator loss controlled by lambda (equations 1 and 4). In the appendix the authors experiment with varying the lambda parameter and find that generally having lambda > 0 produces \"better\" metrics, though the choice of these metrics are somewhat questionable. It would be great to see the generated samples as \\lambda is varied (to me this is more interesting than seeing the generations as \\beta is varied).\n\nFurther, I take several issues with the authors' point of view regarding the current state-of-affairs in VAEs:\n\n1. \"However, one of their perceived problems is their reconstruction performance.\" \n\nI somewhat disagree with this characterization. Sure, there has been much work on modifying the VAE objective such that the latent variable is not ignored (i.e. posterior collapse), but the point of these works is not to get \"better reconstruction performance\". \n\n2. \"However, having a model that does not over-fit the dataset can be useful, but in this case the decoder of a standard\nVAE should not be regarded as a generative model—that is not its purpose. If we wish to generate\nrealistic looking images we need to imagine the information discarded by the encoder\"\n\nI am not sure I understand this characterization. The decoder is by definition a generative model. Depending on the decoder/encoder capacities (e.g. PixelCNN decoder vs DeConvNet decoder), different types of information will be encoded in the latent space.\n\n3. \"The job of the decoder in a variational autoencoder is to reconstruct the image only using information\nthat can be compressed. Image specific information is ignored. For example, information about the\nprecise shape of an object is probably not compressible. As a result the decoder tends to hedge its\nbets and has a blurry outline.\"\n\nAgain, all of this is dependent on how the encoder/decoder is parameterized. I do not agree that \"information about the precise shape of an object is probably not compressible\". For example see https://hal.archives-ouvertes.fr/hal-01676326/document\n\n4. \"VAEs are often taken to be a pauper’s GAN. That is, a method for generating samples that is easier\nto train than a GAN, but gives slightly worse results.\"\n\nMy view on VAEs is that they are a way of training latent variable models with likelihood training. One potential application of this is to generate samples, but that is not the only (nor the primary) application. \n\n\nFinally, I hope I am not coming across as nitpicking or overly combative, but I am genuinely confused as to the problem that this paper is addressing. I look forward to discussing further among other reviews and the authors during the rebuttal period.\n\n[Response to author rebuttal]\n\nThank you very much for your thoughtful response. However, I must say that I just fundamentally disagree with motivations and some of the statements made in the paper. In particular, there seems to be some conflation (I could be misunderstanding) of \"good reconstruction\" vs \"good generation\". For example:\n\n- \"Our point is that it does not generate a realistic image by design (in contrast to a GAN which is designed to generate a realistic image).\" \n\nI am not sure I understand. Would you say an autoregressive model (e.g. PixelCNN) does not generate realistic images by design? (Clearly, they do!). Would you say that autoregressive language models (e.g. GPT2) do not generative realistic language by design? The argument seems to be that likelihood-based training of generative models does not, by design, encourage realistic-looking images (indeed it is true that good likelihood does not *necessarily* imply good generative models). However, there is massive empirical evidence that likelihood training does result in good generation.\n\nAfter reviewing the other reviews and the author rebuttals, I am maintaining my original score.\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkxzxw96Kr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a hybrid VAE-GAN model, called the latent space renderer-GAN (LSR-GAN), with the goal to “imagine” the latent space of a VAE, and to improve the decoding and sampling quality of a VAE. First, a VAE-like model is trained, after which the encoder weights are frozen, and the decoder is trained as the generator of a GAN (together with an auxiliary discriminator). The generator loss also contains a reconstruction-like term in the latent space, described by the negative log density of the encoding distribution of the original latent conditioned on the output of the generator: -log q(z|g(z)). \n\nDecision: reject\nThis paper contains incorrect claims. Leaving those mistakes aside, the experiments don’t lead to new insights and no comparison against other VAE-GAN hybrids is made. \n\nSupporting arguments for decision:\nIn the introduction the authors state “This is a consequence of the well known evidence lower bound or ELBO objective function consisting of a negative log-probability of generating the original image from the latent representation (this is often implemented as a mean squared error between the image and the reconstruction, although as we argue in Appendix A this term should be proportional to the logarithm of the mean squared error)...”. This statement is surprising and I don’t see how it can be correct. If -log p(x|z) is something like the log of the mean squared error, then the density p(x|z) should be a squared error function, which is not even a valid distribution.\n\nTo be more precise, according to the authors, if one takes a Gaussian p(x|z), the reconstruction error should be modeled with  \n\n-log p(x|z) = log 1/N sum (x_i - \\hat x_i)^2  + const \t\t\t\t\t(1)\n\nwhere \\hat x_i is a function of z. Note the logarithm on the right-hand side here. In appendix A, the authors observe that most other implementations instead optimize \n\n-log p(x|z) = sum_{i=1}^N (x_i -\\hat x_i)^2/(2\\sigma^2) + N/2 log(2 pi sigma^2) \t\t(2)\n\nWhere often sigma is set to ½ (so that the last term on the lhs of (2) drops out when gradients are taken). The authors claim the latter is incorrect because sigma should be equal to the (biased) empirical variance. They then insert sigma=empirical variance into (2), take derivatives while ignoring the dependence of sigma on \\hat x_i, and then arrive at something like eq. 1. They also argue that those implementations that use sigma=½ are actually optimizing a beta-VAE because of this “incorrect” prefactor of the reconstruction error. \nI’m confident that the above claims are not correct. \nAs an example, using a Gaussian decoder distribution requires parameterizing the mean and variance of that Gaussian distribution as a function of z. Here, setting sigma = ½ (as is commonly done in other literature) is valid, contrary to the above claim. One is always allowed to just set the variance to a constant and ignore its dependence on z. This just leaves you with a less flexible distribution to model p(x|z).\n\nThe authors appear to use (1) as the reconstruction error term in the ELBO during VAE optimization, and then play with different prefactors of the KL term similar to beta-VAEs. The combination of the reconstruction error (1) and the prefactor no longer makes this a VAE, but more like a regularized auto-encoder with an unconventional reconstruction error that should not be interpreted as coming from the negative of a log density. Due to the logarithm in front of the mean squared error, the loss function is less sensitive to large errors in reconstruction. It is surprising that the reconstruction error does not cause overflow as log (0) --> - infinity. \n\nComments on experiments and related work: \n- Experiments only show images of reconstructions of a VAE and LSR-GAN, a mean squared error plot of reconstruction errors and an accuracy plot of a classifier evaluated on the reconstructions produced by both models. In terms of MSE the LSR-GAN actually performs worse than their VAE baseline. The authors also train a classifier on cifar-10 and measure its accuracy using the ground truth labels and the reconstructed images of a VAE and LSR-GAN. Here the LSR-GAN performs marginally better than the VAE, but the overall accuracy is very poor.\n- No experimental comparison is made against other VAE-GAN hybrids. Related work on hybrid VAE-GANS is discussed in the appendix, not in the main paper.\n\nOther sections:\n- Section 3 contains a very long interpretation of the minimum description length (MDL). It is unclear what the goal of this section is, as it does not lead to any results, nor does it help bring the point across as to why the proposed model is good at imagining the latent space of a VAE.\n\n\nAdditional feedback to improve the paper (not part of decision assessment):\nThe additional loss term of the generator is very similar to what is used in reweighted wake sleep, although this is not mentioned. It could be worth making a connection here. \nIn the conclusion the authors state “VAEs are often taken to be a pauper’s GAN’. This is not a very scientific statement and can be perceived as insulting for various reasons. Please rephrase this.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new method for improving generative properties of VAE model.  The reviewers unanimously agree that this paper is not ready to be published, particularly being concerned about the unclear objective and potentially misleading claims of the paper. Multiple reviewers pointed out about incorrect claims and statements without theoretical or empirical justification. The reviewers also mention that the paper does not provide new insights about VAE model as MDL interpretation of VAE it is not new.
Decision: Reject