Title: VARIATIONAL LOSSY AUTOENCODER

Abstract: Representation learning seeks to expose certain aspects of observed data in a learned representation that's amenable to downstream tasks like classification. For instance, a good representation for 2D images might be one that describes only global structure and discards information about detailed texture. In this paper, we present a simple but principled method to learn such global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN. Our proposed VAE model allows us to have control over what the global latent code can learn and by designing the architecture accordingly, we can force the global latent code to discard irrelevant information such as texture in 2D images, and hence the VAE only "autoencodes" data in a lossy fashion. In addition, by leveraging autoregressive models as both prior distribution p(z) and decoding distribution p(x|z), we can greatly improve generative modeling performance of VAEs, achieving new state-of-the-art results on MNIST, OMNIGLOT and Caltech-101 Silhouettes density estimation tasks as well as competitive results on CIFAR10.

1. INTRODUCTION:

A key goal of representation learning is to identify and disentangle the underlying causal factors of the data, so that it becomes easier to understand the data, to classify it, or to perform other tasks (Bengio et al., 2013) . For image data this often means that we are interested in uncovering the "global structure" that captures the content of an image (for example, the identity of objects present in the image) and its "style", but that we are typically less interested in the local and high frequency sources of variation such as the specific textures or white noise patterns.

A popular approach for learning representations is to fit a probabilistic latent variable model, an approach also known as analysis-by-synthesis (Yuille & Kersten, 2006; Nair et al., 2008) . By learning a generative model of the data with the appropriate hierarchical structure of latent variables, it is hoped that the model will somehow uncover and untangle those causal sources of variations that we happen to be interested in. However, without further assumptions, representation learning via generative modeling is ill-posed: there are many different possible generative models with different (or no) kinds of latent variables that all encode the same probability density function on our observed data. Thus, the results we empirically get using this approach are highly dependent on the specific architectural and modeling choices that are made. Moreover, the objective that we optimize is often completely disconnected from the goal of learning a good representation: An autoregressive model of the data may achieve the same log-likelihood as a variational autoencoder (VAE) (Kingma & Welling, 2013) , but the structure learned by the two models is completely different: the latter typically has a clear hierarchy of latent variables, while the autoregressive model has no stochastic latent variables at all (although it is conceivable that the deterministic hidden units of the autoregressive models will have meaningful and useful representations). For this reason, autoregressive models have thus far not been popular for the purpose of learning representations, even though they are extremely powerful as generative models (see e.g. van den Oord et al., 2016a) .

A natural question becomes: is it possible to have a model that is a powerful density estimator and at the same time has the right hierarchical structure for representation learning? A potential solution would be to use a hybrid model that has both the latent variable structure of a VAE, as well as the powerful recurrence of an autoregressive model. However, earlier attempts at combining these two kinds of models have run into the problem that the autoregressive part of the model ends up explaining all structure in the data, while the latent variables are not used (Fabius & van Amersfoort, 2014; Chung et al., 2015; Bowman et al., 2015; Serban et al., 2016; Fraccaro et al., 2016; Xu & Sun, 2016) . Bowman et al. (2015) noted that weakening the autoregressive part of the model by, for example, dropout can encourage the latent variables to be used. We analyze why weakening is necessary, and we propose a principled solution that takes advantage of this property to control what kind of information goes into latent variables. The model we propose performs well as a density estimator, as evidenced by state-of-the-art log-likelihood results on MNIST, OMNIGLOT and Caltech-101, and also has a structure that is uniquely suited for learning interesting global representations of data.

2. VAES DO NOT AUTOENCODE IN GENERAL:

A VAE is frequently interpreted as a regularized autoencoder (Kingma & Welling, 2013; Zhang et al., 2016) , but the conditions under which it is guaranteed to autoencode (reconstruction being close to original datapoint) are not discussed. In this section, we discuss the often-neglected fact that VAEs do not always autoencode and give explicit reasons why previous attempts to apply VAE in sequence modeling found that the latent code is generally not used unless the decoder is weakened (Bowman et al., 2015; Serban et al., 2016; Fraccaro et al., 2016) . The understanding of when VAE does autoencode will be an essential building piece for VLAE.

2.1. TECHNICAL BACKGROUND:

Let x be observed variables, z latent variables and let p(x, z) be the parametric model of their joint distribution, called the generative model defined over the variables. Given a dataset X = {x 1 , ..., x N } we wish to perform maximum likelihood learning of its parameters:

EQUATION

but in general this marginal likelihood is intractable to compute or differentiate directly for flexible generative models that have high-dimensional latent variables and flexible priors and likelihoods. A solution is to introduce q(z|x), a parametric inference model defined over the latent variables, and optimize the variational lower bound on the marginal log-likelihood of each observation x:

EQUATION

where θ indicates the parameters of p and q models.

There are various ways to optimize the lower bound L(x; θ); for continuous z it can be done efficiently through a re-parameterization of q(z|x) (Kingma & Welling, 2013; Rezende et al., 2014) .

This way of optimizing the variational lower bound with a parametric inference network and reparameterization of continuous latent variables is usually called VAE. The "autoencoding" terminology comes from the fact that the lower bound L(x; θ) can be re-arranged:

EQUATION

where the first term can be seen as the expectation of negative reconstruction error and the KL divergence term can be seen as a regularizer, which as a whole could be seen as a regularized autoencoder loss with q(z|x) being the encoder and p(x|z) being the decoder. In the context of 2D images modeling, the decoding distribution p(x|z) is usually chosen to be a simple factorized distribution, i.e. p(x|z) = i p(x i |z), and this setup often yields a sharp decoding distribution p(x|z) that tends to reconstruct original datapoint x exactly.

2.2. BITS-BACK CODING AND INFORMATION PREFERENCE:

It's straightforward to see that having a more powerful p(x|z) will make VAE's marginal generative distribution p(x) = z p(z)p(x|z)dz more expressive. This idea has been explored extensively in previous work applying VAE to sequence modeling (Fabius & van Amersfoort, 2014; Chung et al., 2015; Bowman et al., 2015; Serban et al., 2016; Fraccaro et al., 2016; Xu & Sun, 2016) , where the decoding distribution is a powerful RNN with autoregressive dependency, i.e., p(x|z) = i p(x i |z, x <i ). Since RNNs are universal function approximators and any joint distribution over x admits an autoregressive factorization, the RNN autoregressive decoding distribution can in theory represent any probability distribution even without dependence on z.

However, previous attempts have found it hard to benefit from VAE when using an expressive decoding distribution p(x|z). Indeed it's documented in detail by Bowman et al. (2015) that in most cases when an RNN autoregressive decoding distribution is used, the latent code z is completely ignored and the model regresses to be a standard unconditional RNN autoregressive distribution that doesn't depend on the latent code. This phenomenon is commonly attributed to "optimization challenges" of VAE in the literature (Bowman et al., 2015; Serban et al., 2016; Kaae Sønderby et al., 2016) because early in the training the approximate posterior q(z|x) carries little information about datapoint x and hence it's easy for the model to just set the approximate posterior to be the prior to avoid paying any regularization cost D KL (q(z|x)||p(z)).

Here we present a simple but often-neglected observation that this phenomenon arises not just due to optimization challenges and instead even if we can solve the optimization problems exactly, the latent code should still be ignored at optimum for most practical instances of VAE that have intractable true posterior distributions and sufficiently powerful decoders. It is easiest to understand this observation from a Bits-Back Coding perspective of VAE.

It is well-known that Bits-Back Coding is an information-theoretic view of Variational Inference (Hinton & Van Camp, 1993; Honkela & Valpola, 2004 ) and specific links have been established between Bits-Back Coding and the Helmholtz Machine/VAE (Hinton & Zemel, 1994; Gregor et al., 2013) . Here we briefly relate VAE to Bits-Back Coding for self-containedness:

First recall that the goal of designing an efficient coding protocol is to minimize the expected code length of communicating x. To explain Bits-Back Coding, let's first consider a more naive coding scheme. VAE can be seen as a way to encode data in a two-part code: p(z) and p(x|z), where z can be seen as the essence/structure of a datum and is encoded first and then the modeling error (deviation from z's structure) is encoded next. The expected code length under this naive coding scheme for a given data distribution is hence:

EQUATION

This coding scheme is, however, inefficient. Bits-Back Coding improves on it by noticing that the encoder distribution q(z|x) can be used to transmit additional information, up to H(q(z|x)) expected nats, as long as the receiver also has access to q(z|x). The decoding scheme works as follows: a receiver first decodes z from p(z), then decodes x from p(x|z) and, by running the same approximate posterior that the sender is using, decodes a secondary message from q(z|x).

Hence, to properly measure the code length of VAE's two-part code, we need to subtract the extra information from q(z|x). Using Bit-Back Coding, the expected code length equates to the negative variational lower bound or the so-called Helmholtz variational free energy, which means minimizing code length is equivalent to maximizing the variational lower bound:

EQUATION

Casting the problem of optimizing VAE into designing an efficient coding scheme easily allows us to reason when the latent code z will be used: the latent code z will be used when the two-part code is an efficient code. Recalling that the lower-bound of expected code length for data is given by the Shannon entropy of data generation distribution:

H(data) = E x∼data [-log p data (x)],

we can analyze VAE's coding efficiency:

EQUATION

Since Kullback Leibler divergence is always non-negative, we know that using the two-part code derived from VAE suffers at least an extra code length of D KL (q(z|x)||p(z|x)) nats for using a posterior that's not precise. Many previous works in Variational Inference have designed flexible approximate posteriors to better approximate true posterior (Salimans et al., 2014; Rezende & Mohamed, 2015; Tran et al., 2015; Kingma et al., 2016) . Improved posterior approximations have shown to be effective in improving variational inference but none of the existing methods are able to completely close the gap between approximate posterior and true posterior. This leads us to believe that for most practical models, at least in the near future, the extra coding cost D KL (q(z|x)||p(z|x)) will exist and will not be negligible.

Once we understand the inefficiency of the Bits-Back Coding mechanism, it's simple to realize why sometimes the latent code z is not used: if the p(x|z) could model p data (x) without using information from z, then it will not use z, in which case the true posterior p(z|x) is simply the prior p(z) and it's usually easy to set q(z|x) to be p(z) to avoid incurring an extra cost D KL (q(z|x)||p(z|x)). And it's exactly the case when a powerful decoding distribution is used like an RNN autoregressive distribution, which given enough capacity is able to model arbitrarily complex distributions. Hence there exists a preference of information when a VAE is optimized: information that can be modeled locally by decoding distribution p(x|z) without access to z will be encoded locally and only the remainder will be encoded in z.

We note that one common way to encourage putting information into the code is to use a factorized decoder p(x|z) = i p(x i |z) but so long as there is one dimension x j that's independent of all other dimensions for true data distribution, p data (x) = p data (x j )p data (x =j ), then the latent code doesn't contain all the information about x since at least x j will be modeled locally by factorized p(x|z). This kind of independence structure rarely exists in images so common VAEs that have factorized decoder autoencode almost exactly. Other techniques to encourage the usage of the latent code include annealing the relative weight of of D KL (q(z|x)||p(z)) in the variational lower bound (Bowman et al., 2015; Kaae Sønderby et al., 2016) or the use of free bits (Kingma et al., 2016) , which can serve the dual purpose of smoothing the optimization landscape and canceling out part of the Bits-Back Code inefficiency D KL (q(z|x)||p(z|x)).

3. VARIATIONAL LOSSY AUTOENCODER:

The discussion in Section 2.2 suggests that autoregressive models cannot be combined with VAE since information will be preferred to be modeled by autoregressive models. Nevertheless, in this section, we present two complementary classes of improvements to VAE that utilize autoregressive models fruitfully to explicitly control representation learning and improve density estimation.

3.1. LOSSY CODE VIA EXPLICIT INFORMATION PLACEMENT:

Even though the information preference property of VAE might suggest that one should always use the full autoregressive models to achieve a better code length/log-likelihood, especially when slow data generation is not a concern, we argue that this information preference property can be exploited to turn the VAE into a powerful representation learning method that gives us fine-grained control over the kind of information that gets included in the learned representation.

When we try to learn a lossy compression/representation of data, we can simply construct a decoding distribution that's capable of modeling the part of information that we don't want the lossy representation to capture, but, critically, that's incapable of modelling the information that we do want the lossy representation to capture.

For instance, if we are interested in learning a global representation for 2D images that doesn't encode information about detailed texture, we can construct a specific factorization of the autoregressive distribution such that it has a small local receptive field as decoding distribution, e.g.,

p local (x|z) = i p(x i |z, x WindowAround(i) ).

Notice that, as long as x WindowAround(i) is smaller than x <i , p local (x|z) won't be able to represent arbitrarily complex distribution over x without dependence on z since the receptive field is limited such that not all distributions over x admit such factorizations. In particular, the receptive field window can be a small rectangle adjacent to a pixel x i and in this case long-range dependency will be encoded in the latent code z. On the other hand, if the true data distribution admits such factorization for a given datum x and dimension i, i.e.

p data (x i |x WindowAround(i) ) = p data (x i |x <i ), then the information preference property discussed in Section 2.2 will apply here, which means that all the information will be encoded in local autoregressive distribution for x i . Local statistics of 2D images like texture will likely be modeled completely by a small local window, whereas global structural information of an images like shapes of objects is long-range dependency that can only be communicated through latent code z. Therefore we have given an example VAE that will produce a lossy compression of 2D images carrying exclusively global information that can't be modeled locally.

Notice that a global representation is only one of many possible lossy representations that we can construct using this information preference property. For instance, the conditional of an autoregressive distribution might depend on a heavily down-sampled receptive field so that it can only model long-range pattern whereas local high-frequency statistics need to be encoded into the latent code.

Hence we have demonstrated that we can achieve explicit placement of information by constraining the receptive field/factorization of an autoregressive distribution that's used as decoding distribution.

We want to additionally emphasize the information preference property is an asymptotic view in a sense that it only holds when the variational lowerbound can be optimized well. Thus, we are not proposing an alternative to techniques like free bits Kingma et al. (2016) or KL annealing, and indeed they are still useful methods to smooth the optimization problem and used in this paper's experiments.

3.2. LEARNED PRIOR WITH AUTOREGRESSIVE FLOW:

Inefficiency in Bits-Back Coding, i.e., the mismatch between approximate posterior and true posterior, can be exploited to construct a lossy code but it's still important to minimize such inefficiency to improve overall modeling performance/coding efficiency. We propose to parametrize the prior distribution p(z; θ) with an autoregressive model and show that a type of autoregressive latent code can in theory reduce inefficiency in Bits-Back coding.

It is well-known that limited approximate posteriors impede learning and therefore various expressive posterior approximations have been proposed to improve VAE's density estimation performance (Turner et al., 2008; Mnih & Gregor, 2014; Salimans et al., 2014; Rezende & Mohamed, 2015; Kingma et al., 2016) . One such class of approximate posteriors that has been shown to attain good empirical performance is based on the idea of Normalizing Flow, which is to apply an invertible mapping to a simple random variable, for example a factorized Gaussian as commonly used for q(z|x), in order to obtain a complicated random variable. For an invertible transformation between a simple distribution y and a more flexible z, we know from the change-of-variable technique that log q(z|x) = log q(y|x) -log det dz dy and using q(z|x) as approximate posterior will decrease the coding efficiency gap D KL (q(z|x)||p(z|x)) provided the transformation is sufficiently expressive. Kingma et al. (2016) introduced Inverse Autoregressive Flow, which is a powerful class of such invertible mappings that have simple determinant: z i = yi-µi(y1:i-1) σi(y1:i-1) , where µ i (.) ∈ R, σ i (.) ∈ R + are general functions that can be parametrized by expressive neural networks, such as MADE and PixelCNN variants (Germain et al., 2015; van den Oord et al., 2016a) . Inverse autoregressive flow is the inverse/whitening of autoregressive flow: y i = z i σ i (y 1:i-1 ) + µ i (y 1:i-1 ). We refer interested readers to (Rezende & Mohamed, 2015; Kingma et al., 2016) for in-depth discussions on related topics.

In this paper, we propose to parametrize our learnable prior as an autoregressive flow from some simple noise source like spherical Gaussian. Next, we show that using latent code transformed by autoregressive flow (AF) is equivalent to using inverse autoregressive flow (IAF) approximate posterior, which explains why it can similarly improve Bits-Back Coding efficiency. Moreover, compared with an IAF posterior, an AF prior has a more expressive generative model that essentially "comes for free".

For an autoregressive flow f , some continuous noise source is transformed into latent code z: z = f ( ). Assuming the density function for noise source is u( ), we similarly know that log p(z) = log u( ) + log det d dz .

Simply re-arranging the variational lowerbound for using AF prior reveals that having an AF latent code z is equivalent to using an IAF posterior for that we can interpret as the new latent code:

EQUATION

)

AF prior is the same as IAF posterior along the encoder path, f -1 (q(z|x)), but differs along the decoder/generator path: IAF posterior has a shorter decoder path p(x|z) whereas AF prior has a deeper decoder path p(x|f ( )). The crucial observation is that AF prior and IAF posterior have the same computation cost under the expectation of z ∼ q(z|x), so using AF prior makes the model more expressive at no training time cost.

4. EXPERIMENTS:

In (Salimans et al., 2014; Kingma et al., 2016) . For the decoder, we use a variant of PixelCNN that has 6 layers of masked convolution with filter size 3, which means the window of dependency, x WindowAround(i) , is limited to a small local patch. During training, "free bits" (Kingma et al., 2016) is used improve optimization stability. Experimental setup and hyperparameters are detailed in the appendix. Reported marginal NLL is estimated using Importance Sampling with 4096 samples.

We designed experiments to answer the following questions:

• Can VLAE learn lossy codes that encode global statistics?

• Does using AF priors improves upon using IAF posteriors as predicted by theory?

• Does using autoregressive decoding distributions improve density estimation performance? x decompressed from VLAE: z ∼ q(z|x data ), x decompressed ∼ p(x|z). We observe that none of the However, we remark that the lossy code z doesn't always capture the kind of global information that we care about and it's dependent on the type of constraint we put on the decoder. For instance, in Fig 4b , we show decompressions for OMNIGLOT dataset, which has more meaningful variations in small patches than MNIST, and we can observe that semantics are not preserved in some cases. This highlights the need to specify the type of statistics we care about in a representation, which will be different across tasks and datasets, and design decoding distribution accordingly. To verify whether AF prior is able to improve upon IAF posterior alone, it's desirable to test this model without using autoregressive decoder but instead using the conventional independent Bernoulli distribution for p(x|z). Hence we use the best performing model from Kingma et al. (Rezende & Mohamed, 2015) 85.10 DRAW (Gregor et al., 2015) < 80.97 Discrete VAE (Rolfe, 2016) 81.01 PixelRNN (van den Oord et al., 2016a) 79.20 IAF VAE (Kingma et al., 2016) 79 In addition, we hypothesize that the separation of different types of information, the modeling global structure in latent code and local statistics in PixelCNN, likely has some form of good inductive biases for 2D images. In order to evaluate if VLAE is an expressive density estimator with good inductive biases, we will test a single VLAE model, with the same network architecture, on all binary datasets. We choose hyperparameters manually on statically binarized MNIST and use the same hyperparameters to evaluate on dynamically binarized MNIST, OMNIGLOT and Caltech-101 Silhouettes. We also note that better performance can be obtained if we individually tune hyperparameters for each dataset. As a concrete demonstration, we report the performance of a fine-tuned VLAE on OMNIGLOT dataset in Table 3 . (Salimans et al., 2014) 81.94 DLGM 2hl + IWAE (Burda et al., 2015a) 82.90 Discrete VAE (Rolfe, 2016) 80.04 LVAE (Kaae Sønderby et al., 2016) 81.74 DRAW + VGP (Tran et al., 2015) < 79.88 IAF VAE (Kingma et al., 2016) 79 As seen in Table 2 ,3,4, with the same set of hyperparameters tuned on statically binarized MNIST, VLAE is able to perform well on the rest of datasets, significantly exceeding previous state-ofthe-art results on dynamically binarized MNIST and Caltech-101 Silhouettes and tying statistically with best previous result on OMNIGLOT. In order to isolate the effect of expressive PixelCNN as decoder, we also report performance of the same PixelCNN trained without VAE part under the name "Unconditional Decoder".

4.3. NATURAL IMAGES: CIFAR10:

In addition to binary image datasets, we have applied VLAE to the CIFAR10 dataset of natural images. Density estimation of CIFAR10 images has been a challenging benchmark problem used by many recent generative models and hence is great task to position VLAE among existing methods.

We investigated using ResNet (He et al., 2016) and DenseNet (Huang et al., 2016) as building blocks for VAE networks and observed that DenseNet reduces overfitting. We also propose a new optimization technique that blends the advantages of KL annealing (Serban et al., 2016) and "free bits" (Kingma et al., 2016) to stabilize learning on this challenging dataset. Detailed experimental setup is described in Appendix.

VLAE is compared to other methods on CIFAR10 in We also investigate learning lossy codes on CIFAR10 images. To illustrate how does the receptive field size of PixelCNN decoder influence properties of learned latent codes, we show visualizations of similar VLAE models with receptive fields of different sizes. Specifically we say a receptive field,

x WindowAround(i) , has size AxB when a pixel x i can depend on the rectangle block of size AxB immediately on top of x i as well as the A-1 2 pixels immediately to the left of x i . We use this notation to refer to different types of PixelCNN decoders in Figure 3 .

From (a)-(c) in Figure 3 , we can see that larger receptive fields progressively make autoregressive decoders capture more structural information. In (a), a smaller receptive field tends to preserve rather detailed shape information in the lossy code whereas the latent code only retains rough shape in (c) with a larger receptive field. It's interesting to also note that in (a)-(c), oftentimes color information is partially omitted from latent codes and one explanation can be that color is very predictable locally. However, color information can be important to preserve if our task is, for example, object classification. To demonstrate how we can encode color information in the lossy code, we can choose to make PixelCNN decoder depend only on images' grayscale versions. In other words, instead of choosing the decoder to be p local (x|z) = i p(x i |z, x WindowAround(i) ), we use a decoder of the form p local (x|z) = i p(x i |z, Grayscale(x WindowAround(i) )). In (d) of Figure 3 , we visualize lossy codes for a VLAE that has the same receptive field size as (c) but uses a "grayscale receptive field". We note that the lossy codes in (d) encode roughly the same structural information as those in (c) but generally generate objects that are more recognizable due to the preservation of color information. This serves as one example of how we can design the lossy latent code carefully to encode what's important and what's not.

5. RELATED WORK:

We investigate a fusion between variational autoencoders with continuous latent variables (Kingma & Welling, 2013; Rezende et al., 2014) and neural autoregressive models. For autoregression, we specifically apply a novel type of architecture where autoregression is realised through a carefully constructed deep convolutional network, introduced in the PixelCNN model for images (van den Oord et al., 2016a,b) . These family of convolutional autoregressive models was further explored, and extended, for audio in WaveNet (Oord et al., 2016) , video in Video Pixel Networks (Kalchbrenner et al., 2016b) and language in ByteNet (Kalchbrenner et al., 2016a) .

The combination of latent variables with expressive decoder was previously explored using recurrent networks mainly in the context of language modeling (Chung et al., 2015; Bowman et al., 2015; Serban et al., 2016; Fraccaro et al., 2016; Xu & Sun, 2016) . Bowman et al. (2015) has also proposed to weaken an otherwise too expressive decoder by dropout to force some information into latent codes.

Concurrent with our work, PixelVAE (Gulrajani et al., 2016) also explored using conditional Pixel-CNN as a VAE's decoder and has obtained impressive density modeling results through the use of multiple levels of stochastic units.

Using autoregressive model on latent code was explored in the context of discrete latent variables in DARN (Gregor et al., 2013) . Kingma et al. (2016) , Kaae Sønderby et al. ( 2016), Gregor et al. (2016) and Salimans (2016) explored VAE architecture with an explicitly deep autoregressive prior for continuous latent variables, but the autoregressive data likelihood is intractable in those architectures and needs to inferred variationally. In contrast, we use multiple steps of autoregressive flows that has exact likelihood and analyze the effect of using expressive latent code.

Optimization challenges for using (all levels of) continuous latent code were discussed before and practical solutions were proposed (Bowman et al., 2015; Kaae Sønderby et al., 2016; Kingma et al., 2016) . In this paper, we present a complementary perspective on when/how should the latent code be used by appealing to a Bits-Back interpretation of VAE.

Learning a lossy compressor with latent variable model has been investigated with Con-vDRAW (Gregor et al., 2016) . It learns a hierarchy of latent variables and just using high-level latent variables will result in a lossy compression that performs similarly to JPEG. Our model similarly learns a lossy compressor but it uses an autoregressive model to explicitly control what kind of information should be lost in compression.

6. CONCLUSION:

In this paper, we analyze the condition under which the latent code in VAE should be used, i.e. when does VAE autoencode, and use this observation to design a VAE model that's a lossy compressor of observed data. At modeling level, we propose two complementary improvements to VAE that are shown to have good empirical performance.

VLAE has the appealing properties of controllable representation learning and improved density estimation performance but these properties come at a cost: compared with VAE models that have simple prior and decoder, VLAE is slower at generation due to the sequential nature of autoregressive model.

Moving forward, we believe it's exciting to extend this principle of learning lossy codes to other forms of data, in particular those that have a temporal aspect like audio and video. Another promising direction is to design representations that contain only information for downstream tasks and utilize those representations to improve semi-supervised learning.

images into gray-scale images via this specific transformation: (0.299 * R) + (0.587G) + (0.114B).

Best density estimation result is obtained with 7x4 receptive field experiments.

C SOFT FREE BITS "Free bits" was a technique proposed in (Kingma et al., 2016) where K groups of stochastic units are encouraged to be used through the following surrogate objective:

L λ = E x∼M E q(z|x) [log p(x|z)] - K j=1 maximum(λ, E x∼M [D KL (q(z j |x)||p(z j ))])

This technique is easy to use since it's usually easy to determine the minimum number of bits/nats, λ, stochastic units need to encode. Choosing λ is hence easier than setting a fixed KL annealing schedule (Serban et al., 2016) .

On the other hand, Kl annealing has the benefit of the surrogate objective will smoothly become the true objective, the variational lower bound where as "free bits" has a sharp transition at the boundary. Therefore, we propose to still use λ as hyperparameter to specify at least λ nats should be used but try to change the optimization objective as slowly as possible:

L SoftFreeBits (x; θ) =E q(z|x) [log p(x|z)] -γD KL (q(z|x)||p(z))

where 0 < γ ≤ 1.

And we make the optimization smoother by changing γ slowly online to make sure at least λ nats are used: when Kl is too much higher than λ (we experimented wide range of thresholds from 3% to 30%, all of which yield improved results, and we tend to use 5% us a threshold), γ is increased, and when Kl lower than λ, γ is decreased to encourage information flow.

We found it sufficient to increase/decrease at 10% increment and didn't further tune this parameter.

D AUTOREGRESSIVE DECODER WITHOUT AUTOREGRESSIVE PRIOR:

In this section, we investigate the scenario of just using an autoregressive decoder without using an autoregressive prior. We compare the exact same model in three configurations: 1) using smallreceptive-field PixelCNN as an unconditional density estimator; 2) using small-receptive-field as a decoder in a VAE with Gaussian latent variables; 3) replacing Gaussian latent variables with autoregressive flow latent variables in 2). In Table 1 , we can observe that each step of modification improves density estimation performance. In addition, using an autoregressive latent code makes the latent code transmit more information as shown in the difference of E[D KL (q(z|x)||p(z))].

E CIFAR10 GENERATED SAMPLES:



:

We use the version provided by Hugo Larochelle.

Figures and Tables:

Figure 1: Statically Binarized MNIST

Figure 2: OMNIGLOT

Figure 3: CIFAR10: Original test-set images (left) and "decompressioned" versions from VLAE's lossy code (right) with different types of receptive fields

Statically Binarized MNIST

Table content: <table><tr><td>Model</td><td>NLL Test</td></tr><tr><td>Normalizing flows</td><td/></tr></table>

on statically binarized MNIST and make the single modification of replacing the original IAF posterior with an equivalent AF prior, removing the context. As seen in Table1, VAE with AF prior is outperforming VAE with an equivalent IAF posterior, indicating that the deeper generative model from AF prior is beneficial. A similar gain carries over when an autoregressive decoder is used: on statically binarized MNIST, using AF prior instead of IAF posterior reduces train NLL by 0.8 nat and test NLL by 0.6 nat.Next we evaluate whether using autoregressive decoding distribution can improve performance and we show in Table1that a VLAE model, with AF prior and PixelCNN conditional, is able to outperform a VAE with just AF prior and achieves new state-of-the-art results on statically binarized MNIST.

Table content: <table><tr><td/><td>.88</td></tr><tr><td>AF VAE</td><td>79.30</td></tr><tr><td>VLAE</td><td>79.03</td></tr><tr><td>(2016)</td><td/></tr></table>

Dynamically binarized MNIST

Table content: <table><tr><td>Model</td><td>NLL Test</td></tr><tr><td>Convolutional VAE + HVI</td><td/></tr></table>

We show that VLAE models attain new state-of-the-art performance among other variationally trained latent-variable models. DenseNet VLAE model also outperforms most other tractable likelihood models including Gated PixelCNN and PixelRNN and has results only slightly worse than currently unarchived state-of-the-art Pixel-CNN++.

Table content: <table/>

CIFAR10. Likelihood for VLAE is approximated with 512 importance samples.

Table content: <table><tr><td>[1]</td></tr></table>

Ablation on Dynamically binarized MNIST

Table content: <table><tr><td>Model</td><td>NLL Test</td><td>KL</td></tr><tr><td>Unconditional PixelCNN</td><td>87.55</td><td>0</td></tr><tr><td>PixelCNN Decoder + Gaussian Prior</td><td>79.48</td><td>10.60</td></tr><tr><td>PixelCNN Decoder + AF Prior</td><td>78.94</td><td>11.73</td></tr></table>

Peer review (iclr2017):
The AR prior and its equivalent - the inverse AR posterior - is one of the more elegant ways to improve the unfortunately poor generative qualities of VAE-s. It is only an incremental but important step. Incremental, because, judging by the lack of, say, CIFAR10 pictures of the VLAE in its *creative* regime ( i.e., when sampling from prior), it will not answer many of the questions hanging over. We hope to see the paper accepted: in relative terms, the paper shines in the landscape of the other papers which are rich on engineering hacks but lacking on theoretical insights. Some disagreements with the theoretical suppositions in the paper: i) The VAE-s posterior converges to the prior faster than we would like because the gradients of the *generative* error (the KL divergence of prior and posterior) w.r.t. mu & sigma are simple, inf differentiable functions and their magnitude far exceeds the magnitude of the resp. gradients of the reconstruction error. Especially when more *hairy* decoders like pixelCNN are used. We always considered this obvious and certainly not worthy of one page of CS mumbo-jumbo to explain. Dumbing-down the decoder via variations of dropout or *complexifying* the sampler as in here, or slapping the generative error with a *DeepMind* constant (beta_VAE), are the natural band-aids, but seem to fail in the *creative* regime, for real-life sets like CIFAR10 or more complex ones. Other conceptual solutions are needed, some are discussed in [2]. ii) The claim near the end of section 2.2 that *the extra coding cost a.k.a. variational error will exist and will not be negligible* is a speculation, which, in our empirical experience at least, is incorrect. The variational error is quantifiable for the Gibbs/exponential family of priors/posteriors, as described in [1], section 3.8, and as Tim Salimans knows from his own previous work. In the case of CIFAR10 for example, the variational error is negligible, even for simple sampling families like Gaussian, Laplacian, etc. Moreover, in hindsight, using the closed-form for generative error (the KL divergence of prior and posterior) in the pioneering VAE papers, was likely a mistake inherited by the unnecessary Bayseanism which inspired them (beautiful but a mistake nonetheless): The combo of generative and variational error should together be approximated by the same naive Monte Carlo used for the reconstruction error (easily follows from equation (3.13) in [1]) i.e. arithmetic average over observations. On the lighter side, guys, please do not recycle ridiculous terms like *optimizationally challenged*, as in section 2.2! The English language has recently acquired *mentally-challenged*, *emotionally-challenged*, etc, and now political correctness has sadly found its way to machines? [1] https://arxiv.org/pdf/1508.06585v5.pdf [2] https://arxiv.org/pdf/1511.02841v3.pdf