Title: FEATURE MAP VARIATIONAL AUTO-ENCODERS

Abstract: There have been multiple attempts with variational auto-encoders (VAE) to learn powerful global representations of complex data using a combination of latent stochastic variables and an autoregressive model over the dimensions of the data. However, for the most challenging natural image tasks the purely autoregressive model with stochastic variables still outperform the combined stochasticautoregressive models. In this paper, we present simple additions to the VAE framework that generalize to natural images by embedding spatial information in the stochastic layers. We significantly improve the state-of-the-art results on MNIST, OMNIGLOT, CIFAR10 and ImageNet when the feature map parameterization of the stochastic variables are combined with the autoregressive PixelCNN approach. Interestingly, we also observe close to state-of-the-art results without the autoregressive part. This opens the possibility for high quality image generation with only one forward-pass.

1. INTRODUCTION:

In representation learning the goal is to learn a posterior latent distribution that explains the observed data well (Bengio et al., 2013) . Learning good representations from data can be used for various tasks such as generative modelling and semi-supervised learning (Kingma, 2013; Rezende et al., 2014; Kingma et al., 2014; Rasmus et al., 2015; Maaløe et al., 2016) . The decomposition of variational auto-encoders (VAE) (Kingma, 2013; Rezende et al., 2014) provides the potential to disentangle the internal representation of the input data from local to global features through a hierarchy of stochastic latent variables. This makes the VAE an obvious candidate for learning good representations. However, in order to make inference tractable VAEs contain simplifying assumptions. This limits their ability to learn a good posterior latent representation.

In complex data distributions with temporal dependencies (e.g. text, images and audio), the VAE assumption on conditional independence in the input distribution limits the ability to learn local structures. This has a significant impact on its generative performance, and thereby also the learned representations. Additionally, the one-layered VAE model with a N (0, I) latent prior poses serious constraints on the posterior complexity that the model is able to learn. A deep hierarchy of stochastic latent variables should endow the model with more expressiveness, but the VAE has a tendency to skip the learning of the higher representations since they pose a direct cost in its optimization term.

There have been several attempts to eliminate the limitations of the VAE. Some concern formulating a more expressive variational distribution (Burda et al., 2015b; Rezende & Mohamed, 2015; Tran et al., 2016; Maaløe et al., 2016) where other concerns learning a deeper hierarchy of latent variables (Sønderby et al., 2016) . These contributions have resulted in better performance, but are still limited when modelling complex data distributions where a conditional independence does not apply. When parameterizing the VAE decoder with recurrent neural networks (Krishnan et al., 2015; Bowman et al., 2015; Fraccaro et al., 2016) , the decoding architecture gets too powerful which results in unused latent stochastic variables (Chen et al., 2017) .

The limitations of the VAE have spawned interest towards other generative models such as Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) and the autoregressive Pixel-CNN/PixelRNN models (van den Oord et al., 2016b) . These methods have proven powerful in learning good generative models, but the lack of stochastic latent variables makes them less suitable for representation learning purposes (Chen et al., 2017) . Lately, we have seen several successful attempts to combine VAEs with PixelCNNs (Gulrajani et al., 2016; Chen et al., 2017) . This results in a model where the global structure of the data is learned in the stochastic latent variables of the VAE and the local structure is learned in the PixelCNN. However, despite the additional complexity and potential extra expressiveness, these models do not outperform a simple autoregressive model (van den Oord et al., 2016a; Salimans et al., 2017) .

In this paper we present the Feature Map Variational Auto-Encoder (FAME) that combines the top-down variational approximation presented in the Ladder Variational Auto-Encoder (LVAE) (Sønderby et al., 2016) with a spatial (feature map) representation of the stochastic latent variables and an autoregressive decoder. We show that (i) FAME outperforms previously state-of-the-art loglikelihood on MNIST, OMNIGLOT, CIFAR10 and ImageNet, (ii) FAME learns a deep hierarchy of stochastic latent variables without inactivated latent units, (iii) by removing the autoregressive decoder FAME performs close to previous state-of-the-art log-likelihood suggesting that it is possible to get good quality generation with just one forward pass.

2. FEATURE MAP VARIATIONAL AUTO-ENCODER:

The VAE (Rezende et al., 2014; Kingma, 2013 ) is a generative model with a hierarchy of stochastic latent variables:

EQUATION

where z = z 1 , ..., z L , θ denotes the parameters, and L denotes the number of stochastic latent variable layers. The stochastic latent variables are usually modelled as conditionally independent Gaussian distributions with a diagonal covariance:

EQUATION

Since the posterior p(z|x) often is intractable we introduce a variational approximation q φ (z|x) with parameters φ. In the original VAE formulation q φ (z|x) is decomposed as a bottom-up inference path through the hierarchy of the stochastic layers:

EQUATION

EQUATION

EQUATION

We optimize an evidence lower-bound (ELBO) to the log-likelihood log p θ (x) = log z p θ (x, z)dz. Burda et al. (2015a) introduced the importance weighted bound:

EQUATION

and proved that L K (θ, φ; x) ≥ L L (θ, φ; x) for K > L. For K = 1 the bound co-incides with the standard ELBO: L(θ, φ; x) = L 1 (θ, φ; x). The hierarchical structure of both the variational approximation and generative model give the VAE the expressiveness to learn different representations of the data throughout its stochastic variables, going from local (e.g. edges in images) to global features (e.g. class specific information). However, we can apply as recursive argument Maaløe et al. (2017) to show that when optimizing with respect to the parameters θ and φ the VAE is regularized towards

q φ (z L |z L-1 ) = p θ (z L ) = N (z L ; 0, I)

. This is evident if we rewrite Equation 6for K = 1:

L(θ, φ; x) = E q φ (z 1:L-1 |x) p θ (x, z 1:L-1 |z L ) q φ (z 1:L-1 |x) -E q φ (z 1:L-1 |x) KL q φ (z L |z L-1 )||p θ (z L ) . KL q φ (z L |z L-1 )||p θ (z L )

= 0 is a local maxima and learning a useful representation in z L can therefore be disregarded throughout the remainder of the training. The same argumentation can be used for all subsequent layers z 2:L , hence the VAE has a tendency to collapse towards not using the full hierarchy of latent variables. There are different ways to get around this tendency, where the simplest is to down-weight the KL-divergence with a temperature term (Bowman et al., 2015; Sønderby et al., 2016) . This term is applied during the initial phase of optimization and thereby downscales the regularizing effect. However, this only works for a limited number of hierarchically stacked latent variables (Sønderby et al., 2016) .

Formulating a deep hierarchical VAE is not the only cause of inactive latent variables, it also occurs when the parameterization of the decoder gets too powerful (Krishnan et al., 2015; Fraccaro et al., 2016; Chen et al., 2017) . This can be caused by using autoregressive models such as p(x, z) = j p(x j |x <j , z)p(z). Chen et al. ( 2017) circumvent this by introducing the Variational Lossy Auto-Encoder (VLAE) where they define the architecture for the VAE and autoregressive model such that they capture global and local structures. They also utilize the power of more expressive posterior approximations using inverse autoregressive flows (Rezende & Mohamed, 2015; Kingma et al., 2016) . In the PixelVAE, Gulrajani et al. (2016) takes a similar approach to defining the generative model but makes a simpler factorizing decomposition in the variational approximation q φ (z|x) = L i q φ (z i |x), where the terms have some degree of parameter sharing. This formulation results in a less flexible model.

In Kingma et al. (2016) ; Gulrajani et al. (2016) ; Chen et al. (2017) we have seen that VAEs with simple decompositions of the stochastic latent variables and a powerful autoregressive decoder can result in good generative performance and representation learning. However, despite the additional cost of learning a VAE we only see improvement in the log-likelihood over the PixelCNN for small gray-scale image datasets (Salimans et al., 2017) . We propose FAME that extends the VAE with a top-down variational approximation similar to the LVAE (Sønderby et al., 2016) combined with spatial stochastic latent layers and an autoregressive decoder, so that we ensure expressive latent stochastic variables learned in a deep hierarchy (cf. Figure 1 ).

2.1. TOP-DOWN VARIATIONAL APPROXIMATION:

The LVAE (Sønderby et al., 2016) does not change the generative model but changes the variational distribution to be top-down like the generative model. Furthermore the variational distribution shares parameters with the generative model which can be viewed as a precision-weighted (inverse variance) combination of information from the prior and data distribution. The variational approximation is defined as:

EQUATION

The stochastic latent variables are all fully factorized Gaussian distributions and are therefore modelled by q φ (z i |z i+1 , x) = N (z i |µ i , diag(σ 2 i )) for layers i = 1, ..., L. Instead of letting q and p have separate parameters (as in the VAE), the LVAE let the mean and variance be defined in terms of a function of x (the bottom-up data part) and the generative model (the top-down prior):

EQUATION

EQUATION

where µ φ,i = µ φ,i (x) and µ θ,i = µ θ,i (z i+1 ) and like-wise for the variance functions. This precision weighted parameterization has previously yielded excellent results for densely connected networks (Sønderby et al., 2016) .

2.2. CONVOLUTIONAL STOCHASTIC LAYERS:

We have seen multiple contributions (e.g. Gulrajani et al. (2016) ) where VAEs (and similar models) have been parameterized with convolutions in the deterministic layers h i j , for j = 1, ..., M , and M is the number of layers connecting the stochastic latent variables z i . The size of the spatial feature maps decreases towards higher latent representations and transposed convolutions are used in the generative model. In FAME we propose to extend this notion, so that each of the stochastic latent layers z i , ..., z L-1 are also convolutional. This gives the model more expressiveness in the latent layers, since it will keep track of the spatial composition of the data (and thereby learn better representations). The top stochastic layer z L in FAME is a fully-connected dense layer, which makes it simpler to condition on a non-informative N (0, I) prior and sample from a learned generative model p θ (x, z). For the i = 1, ..., L -1 stochastic latent variables, the architecture is as follows:

h M,i = CNN(h <M,i ) µ φ∨θ,i = Linear(CONV(h M,i )) σ φ∨θ,i = Softplus(CONV(h M,i )) ,

where CNN and CONV denote a convolutional neural network and convolutional layer respectively. The top-most latent stochastic layer z L is computed by:

h M,L = Flatten(CNN(h <M,L )) µ φ∨θ,L = Linear(Dense(h M,L )) σ φ∨θ,L = Softplus(Dense(h M,L )) .

This new feature map parameterization of the stochastic layers should be viewed as a step towards a better variational model where the test ELBO and the amount of activated stochastic units are direct meaures hereof.

2.3. AUTOREGRESSIVE DECODING:

From van den Oord et al. (2016b;a); Salimans et al. (2017) we have seen that the PixelCNN architecture is very powerful in modelling a conditional distribution between pixels. In FAME we introduce a PixelCNN in the input dimension of the generative model p θ (x|z) (cf. Figure 1 ). During training we concatenate the input with the reconstruction data in the channel dimension and propagate it through the PixelCNN, similarly to what is done in Gulrajani et al. (2016) . When generating samples we fix a sample from the stochastic latent variables and generate the image pixel by pixel autoregressively.

3. EXPERIMENTS:

We test FAME on images from which we can compare with a wide range of generative models. First we evaluate on gray-scaled image datasets: statically and dynamically binarized MNIST (LeCun et al., 1998) and OMNIGLOT (Lake et al., 2013) . The OMNIGLOT dataset is of particular interest due to the large variance amongst samples. Secondly we evaluate our models on natural image datasets: CIFAR10 (Krizhevsky, 2009) and 32x32 ImageNet1 (van den Oord et al., 2016b) . When modelling the gray-scaled images we assume a Bernoulli B distribution using a Sigmoid activation function as the output and for the natural images we assume a Categorical distribution π by applying the 256-way Softmax approach introduced in van den Oord et al. (2016b) . We evaluate the grayscaled images with L 5000 (cf. Equation 6) and due to runtime and space complexity we evaluate the natural images with L 1000 .

We use a hierarchy of 5 stochastic latent variables. In case of gray-scaled images the stochastic latent layers are dense with sizes 64, 32, 16, 8, 4 (equivalent to Sønderby et al. ( 2016)) and for the natural images they are spatial (cf. Table 1 ). There was no significant difference when using feature maps (as compared to dense layers) for modelling gray-scaled images. We apply batchnormalization (Ioffe & Szegedy, 2015) and ReLU activation functions as the non-linearity between all hidden layers h i,j and use a simple PixelCNN as in van den Oord et al. ( 2016b) with 4 residual blocks.

Because of the concatenation in the autoregressive decoder (cf. Figure 1 ), generation is a cumbersome process that scales linearly with the amount of pixels in the input image. Therefore we have defined a slightly changed parameterization denoted FAME No Concatenation, where the concatenation with the input is omitted. The generation has no dependency on the input data distribution and can therefore be performed in one forward-pass through the generative model.

For optimization we apply the Adam optimizer (Kingma & Ba, 2014) with a constant learning rate of 0.0003. We use 1 importance weighted sample and temperature (Sønderby et al., 2016) scaling from .3 to 1. during the initial 200 epochs for gray-scaled images and .01 to 1. during the first 400 epochs for natural images. All models are trained using the same optimization scheme.

3.1. GENERATIVE PERFORMANCE ON GRAY-SCALED IMAGES:

The MNIST dataset serves as a good sanity check and has a myriad of previously published generative modelling benchmarks. We experienced much faster convergence rate on FAME compared to training a regular LVAE. On the dynamically binarized MNIST dataset we see a sig-

GRAY-SCALED IMAGES 28X28 NATURAL IMAGES 32X32 h:,1 1 X CONV F=5X5, K=32, S=2 2 X CONV F=3X3, K=96, S=1 1 X CONV F=3X3, K=64, S=1 1 X CONV F=3X3, K=96, S=2 z1 1 X DENSE D=64 1 X CONV F=3X3, K=8, S=1 64 FEATURE VECTOR 16 X 16 X 8 FEATURE MAPS h:,2 1 X CONV F=3X3, K=64, S=2 2 X CONV F=3X3, K=192, S=1 1 X CONV F=3X3, K=64, S=1 1 X CONV F=3X3, K=192, S=2 z2 1 X DENSE D=32 1 X CONV F=3X3, K=16, S=1 32 FEATURE VECTOR 8 X 8 X 16 FEATURE MAPS h:,3 1 X CONV F=3X3, K=64, S=2 2 X CONV F=3X3, K=192, S=1 1 X CONV F=3X3, K=64, S=1 1 X CONV F=3X3, K=192, S=2 z3 1 X DENSE D=16 1 X CONV F=3X3, K=16, S=1 16 FEATURE VECTOR 4 X 4 X 16 FEATURE MAPS h:,4 1 X CONV F=3X3, K=64, S=2 2 X CONV F=3X3, K=192, S=1 1 X CONV F=3X3, K=64, S=1 1 X CONV F=3X3, K=192, S=2 z4 1 X DENSE D=8 1 X CONV F=3X3, K=16, S=1 8 FEATURE VECTOR 2 X 2 X 16 FEATURE MAPS h:,5 1 X CONV F=3X3, K=64, S=2 2 X CONV F=3X3, K=192, S=1 1 X CONV F=3X3, K=64, S=1 1 X CONV F=3X3, K=192, S=2 z5 1 X DENSE D=4 1 X DENSE D=64 4 FEATURE VECTOR 64 FEATURE VECTOR

Table 1 : The convolutional layer (Conv), filter size (F), depth (K), stride (S), dense layer (Dense) and dimensionality (D) used in defining FAME for gray-scaled and natural images. The architecture is defined such that we ensure dimensionality reduction throughout the hierarchical stochastic layers.

The autoregressive decoder is a PixelCNN (van den Oord et al., 2016b) with a mask A convolution F=7x7, K=64, S=1 followed by 4 residual blocks of convolutions with mask B, F=3x3, K=64, S=1.

Finally there are three non-residual layers of convolutions with mask B where the last is the output layer with a Sigmoid activation for gray-scaled images and a 256-way Softmax for natural images.

nificant improvement (cf. Table 2 ). However, on the statically binarized MNIST, the parameterization and current optimization strategy was unsuccessful in achieving state-of-the-art results (cf. Table 1 ). In Figure 4a we see random samples drawn from a N (0, I) distribution and propagated through the decoder parameters θ. We also trained the FAME No Concatenation which performs nearly on par with the previously state-of-the-art VLAE model (Chen et al., 2017) that in comparison utilizes a skip-connection from the input distribution to the generative decoder:

p local (x|z) = i p(x i |z, x WindowAround(i)

). This proves that a better parameterization of the VAE improves the performance without the need of tedious autoregressive generation. There was no significant difference in the KL q(z|x) OMNIGLOT consists of 50 alphabets of handwritten characters, where each character has a limited amount of samples. Each character has high variance which makes it harder to fit a good generative model compared to MNIST. Table 3 presents the negative log-likelihood of FAME for OMNIGLOT and demonstrates significant improvement over previously published state-of-the-art. Figure 4b shows generated samples from the learned θ parameter space.

From Sønderby et al. (2016) we have seen that the LVAE is able to learn a much tighter L 1 ELBO compared to the VAE. For the MNIST experiments, the L 1 ELBO is at 80.11 nats compared to the L 5000 77.82 nats. Similarly the OMNIGLOT L 1 ELBO is 86.62 nats compared to 82.54 nats. This shows significant improvements when using importance weighted samples and indicates that the parameterization of the FAME can be done in a way so that the bound is even tighter. We also find that the top-most latent stochastic layer is not collapsing into its prior, since the KL q(z 5 |x)||p(z 5 ) is 5.04 nats for MNIST and 3.67 nats for OMNIGLOT. In order to analyze the contribution from the autoregressive decoder we experimented on masking the contribution from either the concatenated image or the output of the FAME decoder before feeding it into the PixelCNN layers (cf. Figure 1 ). In Figure 5a we see the results of reconstructing MNIST images when masking out the contribution from the stochastic variables and in Figure 5b we mask out the contribution from the concatenated input image.

3.2. GENERATIVE PERFORMANCE ON NATURAL IMAGES:

We investigate the performance of FAME on two natural image datasets: CIFAR10 and ImageNet. Learning a generative model on natural images is more challenging, which is also why there are many tricks that can be done in regards to the autoregressive decoding (van den Oord et al., 2016a; Salimans et al., 2017; Chen et al., 2017) . However, since we are interested in the additional expressiveness of a LVAE parameterization with convolutional stochastic latent variables, we have chosen a suboptimal architecture for the autoregressive decoding (cf. Table 1 ) (van den Oord et al., 2016b ). An obvious improvement to the decoder would be to incorporate the PixelCNN++ (Salimans et al., 2017) , but by using the simpler architecture we ensure that the improvements in log-likelihood is not a result of a strong autoregressive model.

From 4 ), whereas the previously published 64x64 ImageNet models consistently outperform their 32x32 counterpart. In Figure 2 we show samples from FAME on the CIFAR10 dataset. Similarly to previously published results it is difficult to analyze the performance from the samples. However, we can conclude that FAME is able to capture spatial correlations in the images for generating sharp samples. It is also interesting to see how it captures the contours of objects in the images.

4. CONCLUSION:

We have presented FAME, an extension to the VAE that significantly improve state-of-the-art performance on standard benchmark datasets. By introducing feature map representations in the latent stochastic variables in addition to top-down inference we have shown that the model is able to capture representations of complex image distributions while utilizing a powerful autoregressive architecture as a decoder.

In order to analyze the contribution from the VAE as opposed to the autoregressive model, we have presented results without concatenating the input image when reconstructing and generating. This parameterization shows on par results with the previously state-of-the-art results without depending on the time consuming autoregressive generation.

Further directions for FAME is to (i) test it on larger image datasets with images of a higher resolution, (ii) expand the model to capture other data modalities such as audio and text, (iii) combine the model in a semi-supervised framework.

:

http://image-net.org/small/download.php

Figures and Tables:

Figure 1: A visualization of FAME where the solid lines denote the variational approximation (inference/encoder/recognition) network and dashed lines denote the generative model (decoder) network for training. When performing reconstructions during training, the input image is concatenated with the output of the generative model (blue) and when generating the model follows a normal autoregressive sampling flow (red) while also using the stochastic latent variables z = z 1 , ..., z L . Both the variational approximation and the generative model follow a top-down hierarchical structure which enables precision weighted stochastic variables in the variational approximation.

Figure 2: 10 randomly picked CIFAR10 images (left) and 200 random samples drawn from a N (0, I) distribution and propagated through the generative model (right).

Figure 3: Negative log-likelihood performance on OMNIGLOT in nats. The evidence lower-bound is computed with 5000 importance weighted samples L 5000 (θ, φ; x).

Figure 4: Random samples drawn from a N (0, I) distribution and propagated through the generative model of FAME for the dynamically binarized MNIST (a) and OMNIGLOT (b) dataset.

Figure 5: MNIST reconstructions when masking the output from the FAME stochastic variables (a) and the concatenated input image (b) prior to feeding them to the autoregressive PixelCNN. It is interesting to see how the edge information comes from the autoregressive dependency on the input image.



Table content: <table/>

||p(z) between FAME and FAME No Concatenation. FAME use 10.85 nats in average to encode images, whereas FAME No Concatenation use 12.29 nats.

Table content: <table><tr><td/><td>NLL</td></tr><tr><td>IWAE (BURDA ET AL., 2015A)</td><td>103.38</td></tr><tr><td>LVAE (SØNDERBY ET AL., 2016)</td><td>102.11</td></tr><tr><td>RBM (BURDA ET AL., 2015B)</td><td>100.46</td></tr><tr><td>DVAE (ROLFE, 2017)</td><td>97.43</td></tr><tr><td>DRAW (GREGOR ET AL., 2015)</td><td>96.50</td></tr><tr><td colspan="2">CONV DRAW (GREGOR ET AL., 2016) 91.00</td></tr><tr><td>VLAE CHEN ET AL. (2017)</td><td>89.83</td></tr><tr><td>FAME</td><td>82.54</td></tr></table>

Table 3 we see the performance from FAME and FAME No Concatenation on the CIFAR10 dataset. Similarly to the gray-scaled images, FAME outperforms current state-of-the-art results sig-

Table content: <table><tr><td>Under review as a conference paper at ICLR 2018</td><td/></tr><tr><td/><td>BITS/DIM</td></tr><tr><td>UNIFORM DISTRIBUTION (VAN DEN OORD ET AL., 2016B)</td><td>8.00</td></tr><tr><td>DEEP DIFFUSION (SOHL-DICKSTEIN ET AL., 2015)</td><td>5.40</td></tr><tr><td colspan="2">MULTIVARIATE GAUSSIAN (VAN DEN OORD ET AL., 2016B) 4.70</td></tr><tr><td>NICE (DINH ET AL., 2014)</td><td>4.48</td></tr><tr><td>DEEP GMMS (VAN DEN OORD &amp; SCHRAUWEN, 2014)</td><td>4.00</td></tr><tr><td>CONV DRAW (GREGOR ET AL., 2016)</td><td>3.58</td></tr><tr><td>REAL NVP (DINH ET AL., 2016)</td><td>3.49</td></tr><tr><td>PIXELCNN (VAN DEN OORD ET AL., 2016B)</td><td>3.14</td></tr><tr><td>IAF VAE KINGMA ET AL. (2016)</td><td>3.11</td></tr><tr><td>GATED PIXELCNN (VAN DEN OORD ET AL., 2016A)</td><td>3.03</td></tr><tr><td>PIXELRNN (VAN DEN OORD ET AL., 2016B)</td><td>3.00</td></tr><tr><td>VLAE (CHEN ET AL., 2017)</td><td/></tr><tr><td>PIXELCNN++ (SALIMANS ET AL., 2017)</td><td>2.92</td></tr><tr><td>FAME NO CONCATENATION</td><td>2.98</td></tr><tr><td>FAME</td><td>2.75</td></tr></table>

Negative log-likelihood performance on CIFAR10 in bits/dim. The evidence lower-bound is computed with 1000 importance weighted samples L 1000 (θ, φ; x). VAN DEN OORD ET AL., 2016A) 3.83 GATED PIXELCNN 64X64 (VAN DEN OORD ET AL., 2016A) 3.57

Table content: <table><tr><td>BITS/DIM</td></tr></table>

Negative log-likelihood performance on ImageNet in bits/dim. The evidence lower-bound is computed with 1000 importance weighted samples L 1000 (θ, φ; x). nificantly. It is also interesting to see how FAME No Concatenation performs close to the previously published state-of-the-art results. Especially in the image space, this could prove interesting, since the FAME No Concatenation has no additional autoregressive runtime complexity. We only investigated the 32x32 ImageNet dataset, since the training time is significant and it outperformed the 64x64 models (cf. Table

Table content: <table/>

Peer review (iclr2018):
 Update: In light of Yoon Kim*s retraction of replication, I*ve downgraded my score until the authors provide further validation (i.e. CIFAR and ImageNet samples). Summary This paper proposes VAE modifications that allow for the use multiple layers of latent variables. The modifications are: (1) a shared en/decoder parametrization as used in the Ladder VAE [1], (2) the latent variable parameters are functions of a CNN, and (3) use of a PixelCNN decoder [2] that is fed both the last layer of stochastic variables and the input image, as done in [3]. Negative log likelihood (NLL) results on CIFAR 10, binarized MNIST (dynamic and static), OMNIGLOT, and ImageNet (32x32) are reported. Samples are shown for CIFAR 10, MNIST, and OMNIGLOT. Evaluation Pros: The paper’s primary contribution is experimental: SOTA results are achieved for nearly every benchmark image dataset (the exception being statically binarized MNIST, which is only .28 nats off). This experimental feat is quite impressive, and moreover, in the comments on OpenReview, Yoon Kim claims to have replicated the CIFAR result. I commend the authors for making their code available already via DropBox. Lastly, I like how the authors isolated the effect of the concatenation via the ‘FAME No Concatenation’ results. Cons: The paper provides little novelty in terms of model or algorithmic design, as using a CNN to parametrize the latent variables is the only model detail unique to this paper. In terms of experiments, the CIFAR samples look a bit blurry for the reported NLL (as others have mentioned in the OpenReview comments). I find the authors’ claim that FAME is performing superior global modeling interesting. Is there a way to support this experimentally? Also, I would have liked to see results w/o the CNN parametrization; how important was this choice? Conclusion While the paper*s conceptual novelty is low, the engineering and experimental work required (to combine the three ideas discussed in the summary and evaluate the model on every benchmark image dataset) is commendable. I recommend the paper’s acceptance for this reason. [1] C. Sonderby et al., “Ladder Variational Autoencoders.” NIPS 2016. [2] A. van den Oord et al., “Conditional Image Generation with PixelCNN Decoders.” ArXiv 2016. [3] I. Gulrajani et al., “PixelVAE: A Latent Variable Model for Natural Images.” ICLR 2017.