Title: AUXILIARY GUIDED AUTOREGRESSIVE VARIATIONAL AUTOENCODERS

Abstract: Generative modeling of high-dimensional data is a key problem in machine learning. Successful approaches include latent variable models and autoregressive models. The complementary strengths of these approaches, to model global and local image statistics respectively, suggest hybrid models combining the strengths of both models. Our contribution is to train such hybrid models using an auxiliary loss function that controls which information is captured by the latent variables and what is left to the autoregressive decoder. In contrast, prior work on such hybrid models needed to limit the capacity of the autoregressive decoder to prevent degenerate models that ignore the latent variables and only rely on autoregressive modeling. Our approach results in models with meaningful latent variable representations, and which rely on powerful autoregressive decoders to model image details. Our model generates qualitatively convincing samples, and yields stateof-the-art quantitative results.

1. INTRODUCTION:

Unsupervised modeling of complex distributions with unknown structure is a landmark challenge in machine learning. The problem is often studied in the context of learning generative models of the complex high-dimensional distributions of natural image collections. Latent variable approaches can learn disentangled and concise representations of the data (Bengio et al., 2013) , which are useful for compression (Gregor et al., 2016) and semi-supervised learning (Kingma et al., 2014; Rasmus et al., 2015) . When conditioned on prior information, generative models can be used for a variety of tasks, such as attribute or class-conditional image generation, text and pose-based image generation, image colorization, etc. (Yan et al., 2016; van den Oord et al., 2016; Reed et al., 2017; Deshpande et al., 2017) . Recently significant advances in generative (image) modeling have been made along several lines, including adversarial networks (Goodfellow et al., 2014; Arjovsky et al., 2017) , variational autoencoders (Kingma & Welling, 2014; Rezende et al., 2014) , autoregressive models (Oord et al., 2016; Reed et al., 2017) , and non-volume preserving variable transformations (Dinh et al., 2017) .

In our work we seek to combine the merits of two of these lines of work. Variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) can learn latent variable representations that abstract away from low-level details, but model pixels as conditionally independent given the latent variables. This renders the generative model computationally efficient, but the lack of lowlevel structure modeling leads to overly smooth and blurry samples. Autoregressive models, such as pixelCNNs (Oord et al., 2016) , on the other hand, estimate complex translation invariant conditional distributions among pixels. They are effective to model low-level image statistics, and yield stateof-the-art likelihoods on test data (Salimans et al., 2017) . This is in line with the observations of Kolesnikov & Lampert (2017) that low-level image details account for a large part of the likelihood. These autoregressive models, however, do not learn a latent variable representations to support, e.g., semi-supervised learning. See Figure 1 for representative samples of VAE and pixelCNN models.

The complementary strengths of VAEs and pixelCNNs, modeling global and local image statistics respectively, suggest hybrid approaches combining the strengths of both. Prior work on such hybrid models needed to limit the capacity of the autoregressive decoder to prevent degenerate models that completely ignore the latent variables and rely on autoregressive modeling only (Gulrajani et al., 2017; Chen et al., 2017) . In this paper we describe Auxiliary Guided Autoregressive Variational autoEncoders (AGAVE), an approach to train such hybrid models using an auxiliary loss function that controls which information is captured by the latent variables and what is left to the AR decoder, rather than limiting the capacity of the latter. Using high-capacity VAE and autoregressive components allows our models to obtain quantitative results on held-out data that are on par with the state of the art, and to generate samples with both global coherence and low-level details, see Figure 1 .

2. RELATED WORK:

Generative image modeling has recently taken significant strides forward, leveraging deep neural networks to learn complex density models using a variety of approaches. These include the variational autoencoders and autoregressive models that form the basis of our work, but also generative adversarial networks (GANs) (Goodfellow et al., 2014; Arjovsky et al., 2017) and variable transformation with invertible functions (Dinh et al., 2017) . While GANs produce visually appealing samples, they suffer from mode dropping and their likelihood-free nature prevents measuring how well they model held-out test data. In particular, GANs can only generate samples on a non-linear manifold in the data space with dimension equal to the number of latent variables. In contrast, probabilistic models such as VAEs and autoregressive models generalize to the entire data space, and likelihoods of held-out data can be used for compression, and to quantitatively compare different models. The non-volume preserving (NVP) transformation approach of Dinh et al. (2017) chains together invertible transformations to map a basic (e.g. unit Gaussian) prior on the latent space to a complex distribution on the data space. This method offers tractable likelihood evaluation and exact inference, but obtains likelihoods on held-out data below the values reported using state-of-the-art VAE and autoregressive models. Moreover, it is restricted to use latent representations with the same dimensionality as the input data, and is thus difficult to scale to model high-resolution images.

Autoregressive density estimation models, such as pixelCNNs (Oord et al., 2016) , admit tractable likelihood evaluation, while for variational autoencoders (Kingma & Welling, 2014; Rezende et al., 2014) accurate approximations can be obtained using importance sampling (Burda et al., 2016) . Naively combining powerful pixelCNN decoders in a VAE framework results in a degenerate model which ignores the VAE latent variable structure, as explained through the lens of bits-back coding by Chen et al. (2017) . To address this issue, the capacity of the the autoregressive component can be restricted. This can, for example, be achieved by reducing its depth and/or field of view, or by giving the pixelCNN only access to grayscale values, i.e. modeling p(x i |x <i , z) = p(x i |gray(x <i ), z) (Chen et al., 2017; Gulrajani et al., 2017) . This forces the model to leverage the latent variables z to model part of the dependencies among the pixels. This approach, however, has two drawbacks. 

3. AUXILIARY GUIDED AUTOREGRESSIVE VARIATIONAL AUTOENCODERS:

We give a brief overview of variational autoencoders and their limitations in Section 3.1, before we present our approach to learn variational autoencoders with autoregressive decoders in Section 3.2.

3.1. VARIATIONAL AUTOENCODERS:

Variational autoencoders (Kingma & Welling, 2014; Rezende et al., 2014) learn deep generative latent variable models using two neural networks. The "decoder" network implements a conditional distribution p θ (x|z) over observations x given a latent variable z, with parameters θ. Together with a basic prior on the latent variable z, e.g. a unit Gaussian, the generative model on x is obtained by marginalizing out the latent variable:

EQUATION

The marginal likelihood can, however, not be optimized directly since the non-linear dependencies in p θ (x|z) render the integral intractable. To overcome this problem, an "encoder" network is used to compute an approximate posterior distribution q φ (z|x), with parameters φ. The approximate posterior is used to define a variational bound on the data log-likelihood, by subtracting the Kullback-Leibler divergence between the true and approximate posterior:

ln p θ (x) ≥ L(θ, φ; x) = ln(p θ (x)) -D KL (q φ (z|x)||p θ (z|x)) (2) = IE q φ [ln(p θ (x|z)] Reconstruction -D KL (q φ (z|x)||p(z)) Regularization .

(3)

The decomposition in (3) interprets the bound as the sum of a reconstruction term and a regularization term. The first aims to maximize the expected data log-likelihood p θ (x|z) given the posterior estimate q φ (z|x). The second term prevents q φ (z|x) from collapsing to a single point, which would be optimal for the first term.

Variational autoencoders typically model the dimensions of x as conditionally independent,

EQUATION

for instance using a factored Gaussian or Bernoulli model, see e.g. Kingma & Welling (2014) ; Kingma et al. (2016); Yan et al. (2016) . The conditional independence assumption makes sampling from the VAE efficient: since the decoder network is evaluated only once for a sample z ∼ p(z) to compute all the conditional distributions p θ (x i |z), the x i can then be sampled in parallel.

A result of relying on the latent variables to account for all pixel dependencies, however, is that all low-level variability must also be modeled by the latent variables. Consider, for instance, a picture of a dog, and variants of that image shifted by one or a few pixels, or in a slightly different pose, with a slightly lighter background, or with less saturated colors, etc. If these factors of variability are modeled using latent variables, then these low-level aspects are confounded with latent variables relating to the high-level image content. If the corresponding image variability is not modeled using latent variables, it will be modeled as independent pixel noise. In the latter case, using the mean of p θ (x|z) as the synthetic image for a given z results in blurry samples, since the mean is averaged over the low-level variants of the image. Sampling from p θ (x|z) to obtain synthetic images, on the other hand, results in images with unrealistic independent pixel noise.

3.2. AUTOREGRESSIVE DECODERS IN VARIATIONAL AUTOENCODERS:

Autoregressive density models, see e.g. (Larochelle & Murray, 2011; Germain et al., 2015) , rely on the basic factorization of multi-variate distributions,

EQUATION

with x <i = x 1 , . . . , x i-1 , and model the conditional distributions using a (deep) neural network. For image data, PixelCNNs (Oord et al., 2016; van den Oord et al., 2016) use a scanline pixel ordering, and model the conditional distributions using a convolution neural network. The convolutional filters are masked so as to ensure that the receptive fields only extend to pixels x <i when computing the conditional distribution of x i .

PixelCNNs can be used as a decoder in a VAE by conditioning on the latent variable z in addition to the preceding pixels, leading to a variational bound with a modified reconstruction term:

EQUATION

The regularization term can be interpreted as a "cost" of using the latent variables. To effectively use the latent variables, the approximate posterior q φ (z|x) must differ from the prior p(z), which increases the KL divergence. Chen et al. (2017) showed that for loss in (6) and a decoder with enough capacity, it is optimal to encode no information about x in z by setting q(z|x) = p(z). To ensure meaningful latent representation learning Chen et al. (2017) and Gulrajani et al. (2017) restrict the capacity of the pixelCNN decoder. In our approach, in contrast, it is always optimal for the autoregressive decoder, regardless of its capacity, to exploit the information on x carried by z. We rely on two decoders in parallel: the first one reconstructs an auxiliary image y from an intermediate representation f θ (z) in a non-autoregressive manner. The auxiliary image can be either simply taken to be the original image (y = x), or a compressed version of it, e.g. with lower resolution or with a coarser color quantization. The second decoder is a conditional autoregressive model that predicts x conditioned onf θ (z). Modeling y in a non-autoregressive manner ensures a meaningful representation z and renders x and z dependent, inducing a certain non-zero KL "cost" in (6). The uncertainty on x is thus reduced when conditioning on z, and there is no longer an advantage in ignoring the latent variable for the autoregressive decoder. We provide a more detailed explanation of why our auxiliary loss ensures a meaningful use of latent variables in powerful decoders in Appendix A. To train the model we combine both decoders in a single objective function with a shared encoder network:

EQUATION

Treating x and y as two variables that are conditionally independent given a shared underlying latent vairable z leads to λ = 1. Summing the lower bounds in Eq. (3) and Eq. ( 6) of the marginal loglikelihoods of y and x, and sharing the encoder network, leads to λ = 2. Larger values of λ result in valid but less tight lower bounds of the log-likelihoods. Encouraging the variational posterior to be closer to the prior, this leads to less informative latent variable representations. Sharing the encoder across the two decoders is the key of our approach. The factored auxiliary VAE decoder can only model pixel dependencies by means of the latent variables, which ensures that a meaningful representation is learned. Now, given that the VAE encoder output is informative on the image content, there is no incentive for the autoregressive decoder to ignore the intermediate representation f (z) on which it is conditioned. The choice of the regularization parameter λ and auxiliary image y provide two levers to control how much and what type of information should be encoded in the latent variables. See Figure 2 for a schematic illustration of our approach.

4. EXPERIMENTAL EVALUATION:

In this section we describe our experimental setup, and present results on CIFAR10.

4.1. DATASET AND IMPLEMENTATION:

The CIFAR10 dataset (Krizhevsky, 2009) contains 6,000 images of 32×32 pixels for each of the 10 object categories airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. The images are split into 50,000 training images and 10,000 test images. We train all our models in a completely unsupervised manner, ignoring the class information.

We implemented our model based on existing architectures. In particular we use the VAE architecture of Kingma et al. (2016) , and use logistic distributions over the RGB color values. We let the intermediate representation f (z) output by the VAE decoder be the per-pixel and per-channel mean values of the logistics, and learn per-channel scale parameters that are used across all pixels. The cumulative density function (CDF), given by the sigmoid function, is used to compute probabilities across the 256 discrete color levels, or fewer if a lower quantization level is chosen in y. Using RGB values y i ∈ [0, 255], we let b denote the number of discrete color levels and define c = 256/b. The probabilities over the b discrete color levels are computed from the logistic mean and variance µ i and s i as

EQUATION

)

For the pixelCNN we use the architecture of Salimans et al. ( 2017), and modify it to be conditioned on the VAE decoder output f (z), or possibly an upsampled version if y has a lower resolution than x. In particular, we apply standard non-masked convolutional layers to the VAE output, as many as there are pixelCNN layers. We allow each layer of the pixel-CNN to take additional input using non-masked convolutions from the feature stream based on the VAE output. This ensures that the conditional pixelCNN remains autoregressive.

To speed up training, we independently pretrain the VAE and pixelCNN in parallel, and then continue training the full model with both decoders. We use the Adamax optimizer (Kingma & Ba, 2015) with a learning rate of 0.002 without learning rate decay. We will release our TensorFlowbased code to replicate our experiments upon publication. 

4.2. EXPERIMENTAL RESULTS:

Quantitative performance evaluation. Following previous work, we evaluate models on the test images using the bits-per-dimension (BPD) metric: the negative log-likelihood divided by the number of pixels values (3×32×32). It can be interpreted as the average number of bits per RGB value in a lossless compression scheme derived from the model. 1 shows that our model performs on par with the state-of-the-art results of the pixelCNN++ model (Salimans et al., 2017) . Here we used the importance sampling-based bound of Burda et al. (2016) with 150 samples to compute the BPD metric for our model. 1 We refer to Figure 1 for qualitative comparison of samples from our model and pixelCNN++, the latter generated using the publicly available code.

The comparison in Table:

Effect of KL regularization strength. In Figure 3 we show reconstructions of test images and samples generated by the VAE decoder, together with their corresponding conditional pixelCNN samples for different values of λ. As expected, the VAE reconstructions become less accurate for larger values of λ, mainly by lacking details while preserving the global shape of the input. At the same time, the samples become more appealing for larger λ, suppressing the unrealistic highfrequency detail in the VAE samples obtained at lower values of λ. Note that the VAE samples and For both reconstructions and samples, the pixelCNN clearly takes into account the output of the VAE decoder, demonstrating the effectiveness of our auxiliary loss to condition high-capacity pixelCNN decoders on latent variable representations. Samples from the pixelCNN faithfully reproduce the global structure of the VAE output, leading to more realistic samples, in particular for higher values of λ. In Appendix B we provide more samples of the autoregressive component conditioned on the same output of the VAE decoder. These confirm the dependence on the intermediate representation f (z). Moreover, the samples show that their variability is structured, and could not be modeled by a simple factored decoder.

For λ = 2 the VAE reconstructions are near perfect during training, and the pixelCNN decoder does not significantly modify the appearance of the VAE output. For larger values of λ, the pixelCNN clearly adds significant detail to the VAE outputs.

Figure 4 traces the BPD metrics of both the VAE and pixelCNN decoder as a function of λ. We also show the decomposition in regularization and reconstruction terms. By increasing λ, the KL divergence can be pushed closer to zero. As the KL divergence term drops, the reconstruction term for the VAE rapidly increases and the VAE model obtains worse BPD values, stemming from the inability of the VAE to model pixel dependencies other than via the latent variables. The reconstruction term of the pixelCNN decoder also increases with λ, as the amount of information it receives drops. However, in terms of BPD which sums KL divergence and pixelCNN reconstruction, a substantial gain of 0.2 is observed increasing λ from 1 to 2, after which smaller but consistent gains are observed.

In Appendix C we present the results of a control experiment where during training we first optimize our objective function in Eq. ( 7), i.e. including the auxiliary reconstruction term, and then switch to optimize the standard objective function of Eq. ( 6) without the auxiliary term. This leads to models that completely ignore the latent variable structure, as predicted by the analysis of Chen et al. (2017) .

Effect of different auxiliary images. We assess the effect of using coarser RGB quantizations and lower spatial resolutions in the auxiliary image. Both make the VAE reconstruction task easier, and transfer modeling of color nuances and/or spatial detail to the pixelCNN.

The VAE reconstructions in Figure 5 (a) obtained using coarser color quantization carry less detail than reconstructions based on the original images using 256 color values, as expected. To understand the relatively small impact of the quantization level on the reconstruction, recall that the VAE decoder outputs the continuous means of the logistic distributions regardless of the quantization level. Only the reconstruction loss is impacted by the quantization level via the computation of the probabilities over the discrete color levels in Eq. ( 8). In Figure 5 gains in the BPD metric as the number of color bins is reduced, showing that it is more effective to model color nuances using the pixelCNN, rather than the latent variables. We trained models with auxiliary images down-sampled to 16×16 and 8×8 pixels, which yield 2.94 and 2.93 BPD, respectively. Which is comparable to the 2.92 BPD obtained using our best model at scale 32×32. In Figure 6 (a) and (b) we show samples obtained using models trained with 256 and 32 color levels in the auxiliary image, and in Figure 6 (c) and (d) with auxiliary images of size 16×16 and 8×8. The samples are qualitatively comparable, showing that in all cases the pixelCNN is able to compensate the less detailed outputs of the VAE decoder.

5. CONCLUSION:

We presented a new approach to training generative image models that combine a latent variable structure with an autoregressive model component. Unlike prior approaches, it does not require careful architecture design to trade-off how much is modeled by latent variables and the autoregressive decoder. Instead, this trade-off can be controlled using a regularization parameter and choice of auxiliary target images. We obtain quantitative performance on par with the state of the art on CIFAR10, and samples from our model exhibit globally coherent structure as well as fine details. 

A ON THE INFORMATION PREFERENCE PROPERTY:

In this appendix we begin by presenting the information preference property from a bits-back coding perspective. We then show that in our setting, unlike in the standard one, it is always optimal for the autoregressive decoder to use the latent variables it is conditioned on.

Combining a VAE with a flexible decoder (for instance an autoregressive one) leads to the latent code being ignored. That is commonly attributed to optimization challenges: at the start of training q(z|x) carries little information about x, the KL term pushes the model to set it to the prior to avoid any penalty, and training never recovers from falling in that local minimum. Chen et al. (2017) have proposed extensive explanations showing that if a sufficiently expressive decoder is used, ignoring the latents actually is the optimal behavior. Because of that, it is necessary to restrict the expressivity of the decoder so that there can be benefits from using the latents. Our main contribution is to propose a method with which it is always optimal for the autoregressive decoder to use the latents it is conditioned on regardless of the expressivity of the decoder. For self-containedness, we begin by presenting the information preference property, using the bits-back coding perspective given in Chen et al. (2017) . We then show that with our formulation, using the latent variables is always the optimal behavior. This is true no matter how expressive the decoder might be, and so our solution can use autoregressive models to the full extent of their potential.

A.1 THE INFORMATION PREFERENCE PROPERTY EXPLAINED WITH BITS-BACK CODING:

Given a random variable X and a a density model p(X) a message x can be encoded with an average code length of -log p(x). Given an encoder q(z|x), a decoder p(x|z) and a prior p(z), the following lossless, two-part coding scheme can be used: i) Sample z ∼ q(z|x) and encode it in a lossless manner2 using p(z). ii) Encode x losslessly using p(x|z). iii) The receiver can decode the message provided he has access to p(z) and p(x|z). The expected length of the message is

C N aive = E x∼D,z∼q(.|x) [-log(p(z)) -log(p(x|z))].

This coding scheme is inefficient because the message is encoded using p(z) while it is sampled from q(z|x)3 . It is necessary because the receiver does not know x, so does not have access to q(z|x). However once the receiver has decoded x, q(z|x) becomes available and a secondary message can be decoded from it. This yields and average code length of:

C BitsBack = E x∼D,z∼q(.|x) [log(q(z|x)) -log(p(z)) -log(p(x|z))].

Notice that C BitsBack corresponds to the standard VAE objective. Let us examine the efficiency of this coding scheme. The lower-bound on the expected code length for the data being encoded is given by the Shannon entropy:

H(D) = E x∼D [-log p D (x)].

The following derivation shows that the bits-back coding scheme cannot reach that lower-bound if q(z|x) is imperfect:

C BitsBack = E x∼D,z∼q(.|x) [log(q(z|x)) -log(p(z)) -log(p(x|z))] = E x∼D [-log(p(x)) + D KL (q(z|x)||p(z|x))] ≥ E x∼D [-log(p D (x)) + D KL (q(z|x)||p(z|x))] = H(D) + E x∼D [D KL (q(z|x)||p(z|x))].

If p(.|x j<i ) is expressive enough, or if q(.|x) is poor enough, the following inequality can be verified:

H(D) ≤ E x∼D [-log p(x|x j<i )] < H(D) + E x∼D [D KL (q(z|x)||p(z|x))]

In that case, any use of the latents that p make, would make the KL divergence worse. The optimal behavior is to set q(z|x) = p(z) to avoid the extra KL cost. Then z becomes independent from x and no information about x is encoded in z. Let us denote p * (x|x j<i ) the best autoregressive model that could be trained without using the latents. If the approximate posterior q(.|x) is precise enough, or if p * (x|x j<i ) is poor enough, then the following inequation can be verified:

H(D) ≤ H(D) + E x∼D [D KL (q(z|x)||p(z|x))] ≤ E x∼D [-log p * (x|x j<i )]

In that case it is possible to achieve better performance by using the latent variables. This situation can be achieved by improving q or by restraining the capacity of the autoregressive model p * . The conclusion is that given an encoder, the latent variables will only be used if the capacity of the autoregressive decoder is sufficiently restricted. This is the approach taken by Chen et al. (2017) and Gulrajani et al. (2017) . This approach works: it has obtained competitive quantitative and qualitative performance. However, it is not satisfactory in the sense that autoregressive models cannot be used to the full extent of their potential, while learning a meaningful latent variable representation.

A.2 OUR MODEL CIRCUMVENTS THE INFORMATION PREFERENCE PROPERTY.:

We now show that in our setup, it is optimal to use the latent representations no matter how expressive the autoregressive model may be. This justifies that in theory our model should always learn meaningful latent structure.

Staying with the message coding point of view, with an AGAVE model both (Y, X) have to be sent to and decoded by the receiver. If one were to only send the auxiliary representation, the setup would be that of a standard VAE: p(y|z) is factorial, and dependencies among the variables can only be modeled through the latent variables. If z were ignored, this would be detrimental to the reconstruction performance, and in practice VAEs with factorial decoder never ignore their latent code. Let us denote C V AE the expected code length required to send the auxiliary message: E x∼D,z∼q(.|x) [log(q(z|x)) -log(p(z)) -log(p(y|z))]

C V AE =

We now show that using the latents is optimal no matter the choice of autoregressive decoder in our setup. Once y has been sent, sending x costs E z∼q(z|x) [i log(p(x i |z, x j<i ))], so the full cost is:

C AGAV E = C V AE + E z∼q(z|x) [- i log(p(x i |z, x j<i ))].

Suppose that the autoregressive decoder ignores Z. Using the fact that the Shannon entropy is the optimal expected code length, we obtain the following lower-bound, denoted L x :

C AGAV E ≥ C V AE + H(X) = L x

If the autoregressive decoder is allowed to use z a new lower-bound L x,z , on C AGAV E is obtained (attained if our decoder and coding schemes are perfect): 

C AGAV E ≥ C V AE + H(X|Z) = L x,z

:

The graphs in Figure4and Figure5are based on the bound in Eq. (7) to reduce the computational effort.

Note that z can be encoded without loss using any model p(z) but the average code length -log(p(z)) is minimized if p(z) matches the frequency of messages q(z|x).

The coding could be optimal in the degenerate case where p(z) = q(z|x), but then z is independent from x ie no information about x is encoded

Figures and Tables:

Figure 1: Randomly selected samples from unsupervised models trained on 32×32 CIFAR10 images: (a) IAF-VAE Kingma et al. (2016), (b) pixelCNN++ Salimans et al. (2017), and (c) our hybrid AGAVE model. For our model, we show the intermediate high-level representation based on latent variables (left), that conditions the final sample based on the pixelCNN decoder (right).

Figure 2: Schematic illustration of our auxiliary guided autoregressive variational autoencoder (AGAVE). The objective function has three components: KL divergence regularization, per-pixel reconstruction with the VAE decoder, and autoregressive reconstruction with the pixelCNN decoder.

Figure 3: Effect of the regularization parameter λ. Reconstructions (a) and samples (b) of the VAE decoder (VR and VS, respectively) and corresponding conditional samples from the pixelCNN (PS).

Figure 4: Bits per dimension of the VAE decoder and pixelCNN decoder, as well as decomposition in KL regularization and reconstruction terms.

Figure 5: Impact of the color quantization in the auxiliary image. (a) Reconstructions of the VAE decoder for different quantization levels (λ = 8). (b) BPD as a function of the quantization level.

Figure 8: Auxiliary reconstructions obtained after dropping the auxilliary loss. (GT) denotes ground truth images unseen during training, f (z) is the corresponding intermediate reconstruction, (PS) denotes pixelCNN samples, conditionned on f (z).

(i) Curbing the capacity of the model is undesirable in unsupervised settings where training data is abundant and overfitting unlikely. (ii) Balancing what is modeled by the VAE and the pixelCNN by means of architectural design choices requires careful hand-design and tuning of the architectures. To overcome these drawbacks, we propose to instead control what is modeled by the VAE and pixelCNN with an auxiliary loss on the VAE decoder output before it is used to condition the autoregressive decoder. This allows us to "plug in" powerful high-capacity VAE and pixelCNN architectures, and balance what is modeled by each component by means of the auxiliary loss.

Table content: <table><tr><td>In a similar vein, Kolesnikov &amp; Lampert (2017) force pixelCNN models to capture more high-level</td></tr><tr><td>image aspects using an auxiliary representation y of the original image x, e.g. a low-resolution</td></tr><tr><td>version of the original. They learn a pixelCNN for y, and a conditional pixelCNN to predict x</td></tr><tr><td>from y, possibly using several intermediate representations. This approach forces modeling of more</td></tr><tr><td>high-level aspects in the intermediate representations, and yields visually more compelling samples.</td></tr><tr><td>Reed et al. (2017) similarly learn a series of conditional autoregressive models to upsample coarser</td></tr><tr><td>intermediate latent images. By introducing partial conditional independencies in the model they</td></tr><tr><td>scale the model to efficiently sample high-resolution images of up to 512×512 pixels. Gregor et al.</td></tr><tr><td>(2016) use a recurrent VAE model to produces a sequence of RGB images with increasing detail</td></tr><tr><td>derived from latent variables associated with each iteration. Like our work, all these models work</td></tr><tr><td>with intermediate representations in RGB space to learn accurate generative image models.</td></tr></table>

Bits per dimension (lower is better) of models on the CIFAR10 test data.

Table content: <table><tr><td>Model</td><td/><td>BPD .|z .|xj&lt;i</td></tr><tr><td>NICE</td><td>(Dinh et al., 2015)</td><td>4.48</td></tr><tr><td>Conv. DRAW</td><td colspan="2">(Gregor et al., 2016) ≤ 3.58</td></tr><tr><td>Real NVP</td><td>(Dinh et al., 2017)</td><td>3.49</td></tr><tr><td>MatNet</td><td colspan="2">(Bachman, 2016) ≤ 3.24</td></tr><tr><td>PixelCNN</td><td>(Oord et al., 2016)</td><td>3.14</td></tr><tr><td>VAE-IAF</td><td colspan="2">(Kingma et al., 2016) ≤ 3.11</td></tr><tr><td>Gated pixelCNN</td><td>(van den Oord et al., 2016)</td><td>3.03</td></tr><tr><td>Pixel-RNN</td><td>(Oord et al., 2016)</td><td>3.00</td></tr><tr><td>Aux. pixelCNN</td><td>(Kolesnikov &amp; Lampert, 2017)</td><td>2.98</td></tr><tr><td>Lossy VAE</td><td colspan="2">(Chen et al., 2017) ≤ 2.95</td></tr><tr><td>AGAVE, λ = 12</td><td colspan="2">(this paper) ≤ 2.92</td></tr><tr><td>pixCNN++</td><td>(Salimans et al., 2017)</td><td>2.92</td></tr></table>

(b)  we observe small but consistent

Table content: <table><tr><td>Test images</td><td>2.990</td><td/><td/><td/></tr><tr><td/><td>2.985</td><td/><td/><td/></tr><tr><td/><td>2.980</td><td/><td/><td/></tr><tr><td>Reconstructions</td><td>2.970 2.975</td><td/><td/><td/></tr><tr><td/><td>2.965</td><td/><td/><td/></tr><tr><td>32</td><td>2.960</td><td/><td/><td/></tr><tr><td/><td>2.955</td><td/><td/><td/></tr><tr><td>64</td><td>2.945 2.950</td><td/><td/><td/></tr><tr><td>256</td><td>16 32 2.940</td><td>64</td><td>128 Number of color levels</td><td>256</td></tr><tr><td>(a)</td><td/><td/><td/><td/></tr></table>

S. Reed, A. van den Oord, N. Kalchbrenner, S. Gómez Colmenarejo, Z. Wang, D. Belov, and N. de Freitas. Parallel multiscale autoregressive density estimation. In ICML, 2017. D. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, 2014.

Table content: <table><tr><td>T. Salimans, A. Karpathy, X. Chen, and D. Kingma. Pixelcnn++: Improving the pixelcnn with</td></tr><tr><td>discretized logistic mixture likelihood and other modifications. In ICLR, 2017.</td></tr><tr><td>A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukcuoglu. Con-</td></tr><tr><td>ditional image generation with PixelCNN decoders. In NIPS, 2016.</td></tr><tr><td>X. Yan, J. Yang, K. Sohn, and H. Lee. Attribute2image: Conditional image generation from visual</td></tr><tr><td>attributes. In ECCV, 2016.</td></tr></table>

Peer review (iclr2018):
 Summary: This paper attempts to solve the problem of meaningfully combining variational autoencoders (VAEs) and PixelCNNs. It proposes to do this by simultaneously optimizing a VAE with PixelCNN++ decoder, and a VAE with factorial decoder. The model is evaluated in terms of log-likelihood (with no improvement over a PixelCNN++) and the visual appearance of samples and reconstructions. Review: Combining density networks (like VAEs) and autoregressive models is an unsolved problem and potentially very useful. To me, the most interesting bit of information in this paper was the realization that you can weight the reconstruction and KL terms of a VAE and interpret it as variational inference in a generative model with multiple copies of pixels (below Equation 7). Unfortunately the authors were unable to make any good use of this insight, and I will explain below why I don’t see any evidence of an improved generative model in this paper. As the paper is written now, it is not clear what the goal of the authors is. Is it density estimation? Then the addition of the VAE had no measurable effect on the PixelCNN++’s performance, i.e., it seems like a bad idea due to the added complexity and loss of tractability. Is it representation learning? Then the paper is missing experiments to support the idea that the learned representations are in any way an improvement. Is it image synthesis (not a real application by itself), then the paper should have demonstrated the usefulness of the model on a real task and probably involve human subjects in a quantitative evaluation. Much of the authors’ analysis is based on a qualitative evaluation of samples. However, samples can be very misleading. A lookup table storing the training data generates samples containing objects and perfect details, but obviously has not learned anything about either objects or the low-level statistics of natural images. In contrast to the authors, I fail to see a meaningful difference between the groups of samples in Figure 1. The VAE samples in Figure 3b) look quite smooth. Was independent Gaussian noise added to the VAE samples or are those (as is sometimes done) sampled means? If the former, what was sigma and how was it chosen? On page 7, the authors conclude that “the pixelCNN clearly takes into account the output of the VAE decoder” based on the samples. Being a mixture model, a PixelCNN++ could easily represent the following mixture: p(x | z) = 0.01 .prod_i p(x_i | x_{<i}) + 0.99 .prod_i p(x_i | z) The first term is just like a regular PixelCNN++, ignoring the latent variables. The second term is just like a variational autoencoder with factorial decoder. The samples in this case would be dominated by the VAE, which depends on the latent state. The log-likelihood would be dominated by the first term and would be minimally effected (see Theis et al., 2016). Note that I am not saying that this is exactly what the model has learned. I am merely providing a possible counter example to the notion that the PixelCNN++ has learned to use of the latent representation in a meaningful way. What happens if the KL term is simply downweighted but the factorial decoder is not included? This seems like it would be a useful control to include. The paper is well written and clear.