Title: EMPIRICAL OBSERVATIONS PERTAINING TO LEARNED PRIORS FOR DEEP LATENT VARIABLE MODELS

Abstract: There exist many forms of deep latent variable models, such as the variational autoencoder and adversarial autoencoder. Regardless of the specific class of model, there exists an implicit consensus that the latent distribution should be regularized towards the prior, even in the case where the prior distribution is learned. Upon investigating the effect of latent regularization on image generation our results indicate that in the case when a sufficiently expressive prior is learned, latent regularization is not necessary and may in fact be harmful insofar as image quality is concerned. We additionally investigate the benefit of learned priors on two common problems in computer vision: latent variable disentanglement, and diversity in image-to-image translation.

1. INTRODUCTION:

In the machine learning subfield of deep latent variable models such as variational autoencoders (VAEs) Kingma & Welling (2014) and adversarial autoencoders (AAEs) Makhzani et al. (2016) have attracted a significant amount of research interest Bowman et al. (2016) ; Kingma et al. (2016) ; Huang et al. (2017) ; Rezende & Viola (2018) ; Dai & Wipf (2019) ; Tolstikhin et al. (2018) . Despite this, in their standard form they are still largely outperformed in terms of synthesized image quality by other deep generative models such as generative adversarial networks (GANs) Goodfellow et al. (2014) ; Karras et al. (2018a; b) , autoregressive models Van den Oord et al. (2016) and flow-based models Dinh et al. (2015; 2017) ; Kingma & Dhariwal (2018) . Even so, latent variable models maintain a number of properties that make them an attractive alternative, such as stable and efficient training as well as efficient synthesis.

In virtually all research done using such models, some form of regularization is imposed on the encoder distribution in order to push it towards the prior distribution. For VAEs, this regularization exists in the form of a KL divergence between approximate posterior and prior, while AAEs force the marginal posterior to match the prior using an adversarial loss. This is of course necessary if the prior is fixed as is often the case, however we argue that when the prior is learnable it is possible to achieve a tight fit between marginal posterior and prior without regularization, and that regularization in this case may actually have a negative impact on sample quality. Experimental results demonstrating this are given in section 4. We also discuss how better disentanglement of the latent variable is achieved as a consequence of removing latent regularization, with experimental results provided in section 5. Finally, we show that removing latent regularization can improve sample diversity in image-to-image translation tasks; results are provided in section 6.

2. BACKGROUND AND PRELIMINARIES:

2.1 DEEP LATENT VARIABLE MODELS Let x be a random vector residing in space X with x ∼ p(x). Additionally, let X = {x (1) , ..., x (n) } be a set of i.i.d. observations drawn from the data distribution p(x). When training likelihood-based generative models, we are interested in maximizing the log-likelihood log p(X) = n i=1 log p(x (i) )

(1)

Latent variable models introduce a latent vector z residing in latent space Z which together with x forms the joint distribution p(x, z). The likelihood of an individual datapoint is then given by

p(x (i) ) = Z p(x (i) |z)p(z)dz (2)

In the case when the parameterization of p(x (i) |z) is complex, such as when it is parameterized by a neural network, Eq. 2 becomes intractable.

Variational Autoencoders:

Variational autoencoders circumvent the issue of intractability by noting that the log-likelihood of an individual datapoint can be expressed as:

EQUATION

where the right hand side of Eq. 3 is commonly referred to as the Variational Lower Bound, which provides a tractable lower bound for the data likelihood that can be optimized directly.

However, variational autoencoders suffer from a number of problems that degrade the quality of samples. For example, when the posteriors of a set of data points overlap at some point in Z, then the optimal decoding at that point will be a weighted mean of the corresponding data points Rezende & Viola (2018) . This contributes to the well known problem of blurry samples. The standard method of encoding the posterior distribution as a diagonal Gaussian is commonly blamed for exarcerbating this issue; the true posterior p(z|x (i) ) will almost certainly never be a diagonal Gaussian, therefore the model is in a sense forced to fit square pegs into round holes, resulting in either posterior overlap or "holes" that have high probability under the prior but are never seen by the decoder Zhao et al. (2017) . Works such as IAF Kingma et al. (2016) propose to increase the flexibility of the approximate posterior by turning it into a normalizing flow (discussed in a later section), allowing a tighter fit between true and approximate posteriors. In Chen et al. (2017) , the authors propose modelling the prior of a variational autoencoder using an autoregressive flow, and demonstrate that it is equivalent to modelling the posterior using IAF along the encoder path, while being deeper along the decoder path.

It is also a well known issue that VAEs tend to not make full use of the latent code, as the objective becomes trapped in a local minima in which the posterior is close to the prior. Such a state occurs early on when the signal from the latent code is weak, resulting in a weak reconstruction term log p(x (i) |z) that is easily outweighed by the KL divergence term in the objective. This exacerbates the problems caused by overlapping posteriors, resulting in even blurrier samples, and additionally results in lower sample diversity. This issue is particularly pernicious in the conditional setting if care is not taken, as it is easily possible for the model to entirely ignore the latent code when it is conditioned on a relevant context, resulting in a deterministic mapping.

Many methods for encouraging use of the latent code have been proposed. For instance, annealing the KL divergence term from 0 to full strength Bowman et al. ( 2016); Huang et al. (2018b) allows the model to largely ignore the KL divergence term at the beginning of training, such that the autoencoder is able to learn a good reconstruction without being constrained by the prior. By the time the KL divergence term is annealed to full strength, the autoencoder is expected to have made good use of the latents, preventing the model from regressing to a state in which the posterior is close the prior. "Free bits", introduced in Kingma et al. (2016), places a limit on the information in nats per latent subset that can contribute to the KL divergence term, ensuring that each subset can contribute at least λ nats of information without penalty. In the context of conditional variational autoencoders, Zhu et al. (2017) proposed to add a latent reconstruction term to the objective to encourage the model to make full use of the latent code.

At their core, all of these techniques are intended as a means of alleviating over-regularization imposed by the KL divergence term in the objective.

This term is of course necessary if the prior takes on a fixed form (e.g. a standard normal distribution). Indeed, all other issues aside, forcing the aggregate posterior distribution towards a pre-determined prior may in and of itself over-regularize the model. In the case that the prior is learned, however, it is possible to eliminate regularization entirely by removing the KL divergence term from the objective, hence obviating the need for any aforementioned techniques.

Generative Moment Matching Networks Li et al. (2015) Generative Moment Matching Networks (GMMNs) Li et al. (2015) are a very relevant work in which the authors propose using an unregularized autoencoder as a generative model. This is the only work to our knowledge that proposes a latent variable model without any latent regularization.

In their proposal they use maximum mean discrepancy (MMD) in order to match the statistics of the latent distribution of the autoencoder and a random variable z T = h(z 0 ), where h is parameterized by a neural network and z 0 ∼ U(-1, 1). In their implementation of MMD they use the Gaussian

EQUATION

2 ) with bandwidth parameter σ 2 which, when the kernel trick is applied, allows matching between all possible moments of the two distributions. The most obvious drawback with this method is that it requires careful tuning of the bandwidth parameter σ 2 , as it plays a crucial role in determining the accuracy of the model. Additionally, training with large batch sizes can be problematic as the complexity of the objective calculation scales quadratically. At the same time, accuracy of training with MMD has been shown to degrade rapidly as the batch size becomes smaller Li et al. (2017) .

Adversarial Autoencoders Makhzani et al. (2016) Another model, closely related to GMMN+AE, is the Adversarial Autoencoder (AAE) Makhzani et al. (2016) ; Tolstikhin et al. (2018) . Rather than matching distributions using MMD, the authors propose utilizing an adversarial objective to regularize the distribution of the autoencoder latent space towards N (0, I). Regularization of the autoencoder in this way may introduce excess noise, the reasoning being that GAN discriminators are known to constantly shift their probability mass around during training in response to the generator, and so the decoder of an AAE is forced to deal with noisy latent codes. Additionally, if the data distribution lies on a manifold that is of lower dimensionality than the latent space and if deterministic encoders are used the model will necessarily produce a degenerate latent distribution, in which case the "holes" problem discussed earlier manifests.

Therefore we propose an alternative: train an unregularized autoencoder and separately learn a generator which attempts to generate samples from the same distribution as the autoencoder's latent space, i.e. a GAN is used to learn the prior. We refer to this modification of AAE as Prior-AAE. Both the AAE and Prior-AAE models inherit the drawbacks of adversarial training in general, e.g. mode collapse, unstable training, and lack of convergence Salimans et al. (2016) , although many techniques have emerged in recent years to mitigate these issues Metz et al. (2017) ; Arjovsky et al. (2017); Mescheder (2018) .

2.2. NORMALIZING FLOWS:

Normalizing flows are a class of model that allows tractable and exact latent variable inference and log-likelihood evaluation by taking advantage of the change of variables formula for an invertible function f : X → Z

p x (x) = p z (f (x)) det( ∂f (x) ∂x T ) (4)

Choosing a simple distribution for p z (z) makes evaluation of p z (f (x)) tractable, and also makes sampling straightforward. In general, computing det( ∂f (x) ∂x T ) takes O(n 3 ) time, where n is the dimensionality of x. We can improve on this however by carefully choosing the form of the bijection f (x). Autoregressive flows Kingma et al. (2016) ; Papamakarios et al. (2017) ; Huang et al. (2018a) constrain f (x) to be autoregressive, such that the Jacobian ∂f (x) ∂x T is triangular. Since the determinant of a triangular matrix is the product of its diagonal entries, computation of the Jacobian determinant becomes O(n). For autoregressive flows with affine transformations, z is given by z i = x i s(x 1:i-1 ) + t(x 1:i-1 ) where s and t are neural networks. Greater model capacity can be achieved by composing multiple autoregressive bijections and permuting the order of latent dimensions in-between each, giving

f = f 1 • w 1 • f 2 • w 2 • ... • f T -1 • w T -1 • f T where w t (x) = W t x

and W t is some permutation matrix. Following from this we define the set of random variables

{z t ∈ Z t } T t=0 where z t-1 = f t (z t ), z T = x and z 0 = f (x).

It is worth noting that, as autoregressive flows are universal approximators Huang et al. (2017) , they can be used to model arbitrarily complex distributions.

By transforming the posterior samples of a VAE based on an autoregressive flow, the capacity of the posterior can be increased in a way that admits tractable posterior density evaluation, and in doing so it is easy to see that the issue of posterior overlap can be mitigated. However, due to the highly non-convex nature of the objective, it is still easily possible for the model to get trapped in local minima in which the posterior is close to the prior, preventing the model from making full use of the latent code.

3. REGULARIZATION EXPERIMENT APPROACH:

In the description of our approach we use the following notation: The encoder is denoted by E φ (x), the decoder is denoted by D ψ (z), and the latent code is denoted by z ∼ q φ (z).

In order to empirically investigate the effect of latent regularization on image quality, we consider two kinds of models. For the first model, we consider VAEs with Gaussian decoder and prior learned via normalizing flow as proposed in Chen et al. (2017) , with the KL divergence term weighted by a hyperparameter β. Rather than the autoregressive flow used in Chen et al. (2017) , we use a flow that splits the vector in half at each step as in Dinh et al. (2017) , as this allows efficient computation of the bijection in both directions, facilitating both efficient inference and efficient sampling. We note again that this gives rise to a set of latents {z t ∈ Z t } T t=0 , where z 0 ∼ N (0, I) and z T should have distribution as close as possible to q φ (z). The bijection is given by f

θ : Z T → Z 0 with z 0 = f θ (z T ).

To evaluate the effect of β we train the model in the following way. Firstly we assume that the encoder E φ encodes the parameters of the posterior q φ (z|x), and that the decoder D ψ encodes the parameters of p ψ (x|z). We then split the objective into two separate functions:

EQUATION

The encoder and decoder terms can be dropped from L θ since they are independent of θ. Using the gradients ∇ φ,ψ L φ,ψ (φ, ψ; x (i) ) and ∇ θ L θ (θ; x (i) ) concurrently in a gradient ascent step is then equivalent to performing gradient ascent on the original objective. Introducing β into the model, we can rewrite L φ,ψ as

EQUATION

For VAEs with a fixed prior (e.g. a standard normal distribution), as β becomes larger we achieve a more structured latent space at the expense of reconstruction quality Matthey et al. (2017) , and vice versa as β becomes smaller. When the parameters of the prior p θ (z) are learnable, however, decreasing β does not necessarily sacrifice latent structure, as any additional incurred divergence between the aggregate posterior q φ (z) and the prior p θ (z) can be mitigated by adjusting the parameters of p θ (z). When β = 0 we are left with

EQUATION

which is the objective of an unregularized autoencoder. It is worth pointing out that at this point, if L2 loss is used for the decoder, we are in fact minimizing the 2-Wasserstein distance between the model and data distributions. This comes from the proof in Tolstikhin et al. (2018) showing the equivalence between the optimal transport objective and

E x∼p(x) [E z∼q φ (z|x) [c(x, z)]] under the constraint that q φ (z) = p θ (z)

, where c is a cost function. In their case the constraint is relaxed and replaced by a regularizer that pushes q φ (z) towards p θ (z). In our case, rather than regularizing q φ (z), we are instead learning p θ (z). It can be seen that while the autoencoder learns a complex latent distribution with classes well separated, the normalizing flow is able to learn a close match.

In Dai & Wipf (2019) the authors propose learning the variance of the decoder distribution, i.e. the decoder loss becomes N (x (i) |D ψ (z), γI) where γ is a learnable parameter. The authors prove that it is always possible to achieve a better VAE cost by lowering the value of γ. We therefore also consider such a decoder in our experiments. Although we lose the property of 2-Wasserstein distance minimization in the case β = 0, in practice performance should not differ greatly. In the case where β > 0, however, the decoder loss will grow to quickly dominate the KL divergence term, and so it is expected that behaviour will be more akin to the case where β = 0.

The second model that we consider is an AAE with learnable prior. This is equivalent to an AAE with fixed prior with the addition that the adversarial loss that regularizes q φ (z) is additionally used to learn the prior parameters. We can apply a hyperparameter β to control the strength of the adversarial loss applied to q φ (z) in the same way as we did for the first model. As above, if L2 loss is used for the decoder this model minimizes the 2-Wasserstein distance. A disadvantage of this model is that it has no way to infer the latent variable z 0 from an arbitrary data sample; its generative components can freely generate samples of z T but the process is not invertible. The ability to infer z 0 can be beneficial for certain tasks. For example, if the dataset is known to have been generated by a set of independent factors of variation, then it is possible that these factors will be recovered along the axes of Z 0 in an unsupervised manner, an outcome generally referred to as disentanglement. Additionally, interpolation in Z 0 is likely to yield more probable images than interpolation in Z T as the distribution of z T is likely to be highly chaotic and multi-modal.

4. EXPERIMENTS ON VALUES OF β:

We first train our proposed model on MNIST with a 2-dimensional latent code in order to demonstrate visually the learned latent distributions. This is shown in Figure 1 ; while the autoencoder has learned a latent distribution that is complex and multi-modal, the samples from the learned prior are a close match.

Our hypothesis is that lowering β towards zero will enable more accurate reconstruction, while learning the parameters of the prior will be sufficient to mitigate the divergence between q φ (z) and p θ (z). If the hypothesis is correct, decreasing β can only ever be beneficial for sample quality, therefore we experiment with various values of β between 0 and 1 to confirm that this is the case. We carried out experiments on the MNIST, Fashion-MNIST and CIFAR-10 datasets. We chose to adopt the Fréchet Inception Distance (FID) Heusel et al. (2017) to measure image quality, a common measure used in GAN evaluation, for our quantitative comparisons. FID scores are given by the Fréchet distance between layer activations of the Inception v3 network Szegedy et al. (2015) , with lower scores indicating greater similarity between two image sets.

Consistent with the hypothesis, FID scores typically decrease as β is decreased for all models as can be seen in Figure 2 . We report the average across 5 runs, and random seeds were kept fixed between runs such that the only changing hyperparameter is β. Interpolations in latent space are shown in Figure 3 in order to demonstrate that the model has learned a smooth manifold and is not memorizing training images. We use spherical linear interpolation as suggested by White (2016) .

We note that we intentionally used priors that were of sufficient expressiveness to learn the latent distribution. For normalizing flows especially, this can result in a significant number of parameters being added to the model. We acknowledge that in cases where a lightweight model is desired or required such that a high capacity prior is not feasible, our results are not applicable, as latent regularization may still be required in order to achieve a good fit between encoder and prior distributions.

For more details regarding the model architecture and training details, please see the appendix.

5. EXPERIMENTAL RESULTS ON CELEBA:



5.1. DISENTANGLEMENT:

The term "disentanglement" can cover a broad range of definitions, but a generalized high-level notion is that the model should capture individual factors of variation within linear subspaces of Z.

We hypothesize that when the underlying factors of variation of the dataset are not independent, our model will learn a more disentangled representation in the distribution of z T than in the distribution of z 0 . As posited by Karras et al. (2018b) , the decoder is likely to pressure z T to take on a disentangled form, since intuitively this should make accurate reconstruction easier as opposed to trying to unwarp a highly entangled representation. In contrast, z 0 is forced to take on a fixed distribution that is highly unlikely to align linearly with the factors of variation. This line of reasoning is equally applicable to standard VAEs that impose a fixed form prior.

The CelebA dataset contains 40 binary attributes that we can consider as such factors of variation, and thus we can use these attributes to calculate an approximate measure of disentanglement. If the model is successful at disentanglement, then for each attribute it should be easy to find a linear hyperplane that partitions the latent encodings into two sets, with each side of the hyperplane corresponding to one of the two possible values of the given attribute. We therefore consider the linear separability score proposed in Karras et al. (2018b) . In their work they first train a deep network classifier on the training images that predicts image attributes, and then train a linear SVM classifier that predicts the classifier network's output given the latent variable. After this they calculate the conditional entropy H(Y|X) where Y represents the labels predicted by the deep network classifier, and X represents the labels predicted by the SVM. It can be seen that lower conditional entropy will correspond to better linear separation, since the SVM will have higher prediction accuracy and thus observing Y will give less information. By following their procedure exactly, we can quantitatively measure disentanglement purely as a function of the generative process of the model.

We evaluated disentanglement of z 0 and z T on three separate models: one with a learned flow prior, a standard VAE, and β-VAE with β set to 25. Results are reported in Table 1 . As expected, z T has a more disentangled representation than z 0 and, interestingly, also achieves a more disentangled representation than either a standard VAE or β-VAE. While β-VAE achieves a slightly better linear disentanglement than a standard VAE, it takes a heavy hit in FID score. In contrast, the model with a flow prior achieves both improved disentanglement and improved reconstruction quality.

5.2. INTERPOLATION:

For the qualitative results in this section, we use a model trained using VGG19 perceptual features. As pointed out in Hou et al. (2017) , we can use the VGG19 network Simonyan & Zisserman (2015) Figure 4 : PCA of E φ (x) for each x in the CelebA training set. The black line shows the path taken when interpolating between two points inZ T , while the red line shows the path taken when interpolating between the same two points in Z 0 . The black line takes the most direct route, while the red line follows a path of higher average density. to improve the appearance of reconstructions, making them sharper and more realistic. We therefore apply VGG19 perceptual loss Johnson et al. (2016) using the relu 1 1, relu 2 1, relu 3 1 and relu 4 1 layers of the VGG19 network.

When interpolating between points in latent space the motivation is often to achieve some semantic mixture between two images, or to change some semantic feature of an image such as putting glasses on a person's face. Here we discuss potential differences between interpolating in Z 0 or in Z T . As a point of clarification, when we say we are interpolating between two images x (a) and x (b) in Z T we are calculating D ψ (lerp(E φ (x (a) ), E φ (x (b) ); t)), and when we say we are interpolating in Z 0 we are calculating

D ψ (f -1 θ (slerp(f θ (E φ (x (a) )), f θ (E φ (x (b) )); t)))

, where t varies between 0 and 1.

As demonstrated in the previous section, the warping of z T via f θ causes the representation to become highly entangled. Therefore interpolation in Z 0 is not appropriate when attempting to change semantic features, as moving along the path of interpolation is likely to change many different semantics with varying rates of change at each point on the path. Even if interpolation manages to coincide with the direction of change of a single semantic feature, the rate of change along the path will vary considerably if there exists class imbalance of the feature; this is due to the fixed distribution of z 0 . As an example, consider the case of a uniform distribution: the relative amount of space occupied by points with a particular semantic feature would be proportional to the class probability of the feature. What this means in practice is that if we were to generate a sequence of images by interpolating in Z 0 along the direction of a semantic feature with heavy class imbalance, there would be very little change for most of the sequence followed by an abrupt change at the end. We posit that interpolating in Z T is more likely to achieve the ideal scenario of having a constant rate of change, as it is not constrained by a fixed distribution. To demonstrate this, we first calculate the mean of E φ (x) for all data points labelled as having glasses, and the mean for all data points without, and substract the latter from the former. We can then add this vector to the encoding of an image without glasses, and interpolate between the original and new encodings. Results are shown in Figure 5 .

Interpolating between two different images, on the other hand, is best done in Z 0 , as the interpolation is more likely to occur along a path of higher average density, resulting in more probable images. A visualization of the difference in paths taken can be seen in Figure 4 . If the two images are very different semantically, then interpolation in Z T can generate highly improbable images, see Figure 6 .

6. DIVERSITY IN IMAGE-TO-IMAGE TRANSLATION:

Achieving high sample diversity in multi-modal image-to-image translation tasks is often an explicit goal Zhu et al. (2017) ; Huang et al. (2018c) . When using conditional VAEs for image-to-image translation tasks, the decoder is often able to learn a fairly accurate reconstruction based on the conditioned image alone, and so may ignore the latent code entirely if the KL divergence weight is too strong. Even when methods such as annealing are used, the model may still only make limited use of the latent code, severely affecting the diversity of generated samples. In order to quantitatively test whether our proposed method is able improve diversity, we experiment with the Variational Unet model proposed in Esser et al. (2018) . In their work, they attempt to learn the distribution over images of people conditioned on their pose. We modified their implementation such that the prior distribution is learned via normalizing flow, and dropped the KL divergence term in the objective. Their model conditions the prior distribution on the given pose such that their objective becomes log p(x (i) |y (i) , z) -D KL [q(z|x (i) , y (i) )||p(z|y

EQUATION

)

where y is the pose and x is the real image. For compatibility we therefore learn the mean of z 0 conditioned on the pose and additionally condition each flow transformation f t on the pose. To measure diversity, we compute the LPIPS distance Zhang et al. (2018) between randomly sampled pairs which were generated by conditioning on the images in the test set. We additionally calculated the Inception score Salimans et al. (2016) of the samples to ensure image quality was not affected. Results comparing our modification with the original implementation after training on the DeepFashion dataset are reported in Table 2 . We also show samples in Figure 7 .

7. CONCLUSION:

We have proposed removing latent regularization from the objective in a latent variable models and demonstrated that this is justified in the case where the prior is sufficiently expressive. We have also demonstrated empirically that a learned prior results in better sample quality as measured by FID 

:



Figures and Tables:

Figure 1: Distribution of the latents of a flow prior with β = 0 after training on MNIST with dim(Z) = 2. (a) Distribution of f θ (E φ (x)) for each x in the test set. Datapoints are colored according to class. (b) Distribution of E φ (x) for each x in the test set. (c) Same as (b) but without class coloring. (d) Distribution of f -1θ (z 0 ) where z 0 ∼ N (0, I). It can be seen that while the autoencoder learns a complex latent distribution with classes well separated, the normalizing flow is able to learn a close match.

Figure 2: Experimental results using different values of β for different models. X-axis represents β, not to scale. Y-axis represents FID score.

Figure 3: Interpolation between samples from the CIFAR-10 test set for a flow prior with fixed γ and β = 0. Leftmost and rightmost columns contain real images from the test set before encoding, middle columns contain interpolations between them.

Figure 5: Adding glasses to a face. Top row of each set: interpolation in Z 0 . Bottom row of each set: interpolation in Z T . Leftmost column: the original image before encoding. The top row shows a more abrupt change towards the end, while the bottom row is closer to a constant rate of change.

Figure 6: Interpolation between two random samples from the training set. Top row of each set: interpolation in Z 0 . Bottom row of each set: interpolation in Z T . Leftmost and rightmost columns: the original images before encoding. The top row shows more realistic images as a result of the interpolation path having higher average density.

Figure 7: Conditional samples using Variational U-net. Top row: original. Bottom row: proposed. Leftmost column contains the original image and the pose being conditioned on.

Linear separability and FID scores after training on CelebA.

Table content: <table><tr><td/><td colspan="2">Separability FID</td></tr><tr><td>VAE</td><td>2.14</td><td>41.4</td></tr><tr><td>β-VAE</td><td>2.09</td><td>82.69</td></tr><tr><td colspan="2">Flow prior (z 0 ) 2.82 Flow prior (z T ) 1.67</td><td>33.1</td></tr></table>

Inception and LPIPS scores (higher is better) on the DeepFashion dataset after training Variational U-net. scores, better disentanglement as measured by linear separability, and greater conditional sample diversity as measured by LPIPS. Our results indicate that fixed-form priors should be eschewed in favour of learned priors with little to no latent regularization.

Table content: <table><tr><td/><td>IS</td><td>LPIPS</td></tr><tr><td>VUNET+VAE</td><td colspan="2">2.63 0.184</td></tr><tr><td colspan="3">VUNET+Proposed 2.70 0.236</td></tr></table>

Peer review (iclr2020):
In this paper, the authors study VAEs which are trained with a loss that does not contain the typical KL divergence term. Additionally, the authors focus on the case where the prior (latent) distribution is learned via a normalizing flow. The authors empirically show that higher quality samples can be generated by VAEs without the KL divergence (than in comparison to those with the term) in this context; that the linear separability of the latent representations increases without the KL term, and that greater diversity can be obtained by conditional sampling. The observation that the authors report is intriguing and definitely worth further inquiry, but I feel that the paper in its current form does not present a coherent picture that other researchers can easily grasp and build upon. The quality of the writing is a significant barrier toward understanding the deeper meaning of the paper. As a result, I recommend declining this paper for publication in ICLR. (1) The paper needs a much clearer focus. For example, the title of *empirical observations pertaining to ...* should be replaced by something related to the actual claim of the paper (2) The abstract is too vague for experts or non-experts to get a clear indication of what is in the paper (3) Where does this work leave the relative performance levels of VAEs versus other generative models? (4) How broadly should the results be interpreted for VAEs? Does it only apply to flow-based priors? Or is it making a claim for a broader set of learnable priors? This will also affect what is in the title and abstract. Is it only being claimed for L2 loss (which then admits a 2-Wasserstein distance problem) or is it being claimed about a broader category of losses? If so, what is the principle under which removing the KL terms is reasonable when it doesn*t lead to a 2-Wasserstein problem? (5) Individual sentences of the paper are adequately written, but the writing of the paper currently feels like a long list of facts and details, which makes extracting the deep takeaway much harder. It is unclear after reading this paper what I would tell another researcher or a student, though I feel that with a substantial revision there could be some very interesting things that could be told based a continuation by the authors of this work.