Title: HIERARCHICAL BAYES AUTOENCODERS

Abstract: Autoencoders are powerful generative models for complex data, such as images. However, standard models like the variational autoencoder (VAE) typically have unimodal Gaussian decoders, which cannot effectively represent the possible semantic variations in the space of images. To address this problem, we present a new probabilistic generative model called the Hierarchical Bayes Autoencoder (HBAE). The HBAE contains a multimodal decoder in the form of an energybased model (EBM), instead of the commonly adopted unimodal Gaussian distribution. The HBAE can be trained using variational inference, similar to a VAE, to recover latent codes conditioned on inputs. For the decoder, we use an adversarial approximation where a conditional generator is trained to match the EBM distribution. During inference time, the HBAE consists of two sampling steps: first a latent code for the input is sampled, and then this code is passed to the conditional generator to output a stochastic reconstruction. The HBAE is also capable of modeling sets, by inferring a latent code for a set of examples, and sampling set members through the multimodal decoder. In both single image and set cases, the decoder generates plausible variations consistent with the input data, and generates realistic unconditional samples. To the best our knowledge, Set-HBAE is the first model that is able to generate complex image sets.

1. INTRODUCTION:

Autoencoders are a crucial generative modeling architecture for many applications involving complex data, including image (e.g., Razavi et al., 2019) and scene synthesis (Eslami et al., 2018) , image compression (Minnen et al., 2018) , video prediction (Lee et al., 2018) , and model-based reinforcement learning (Ha & Schmidhuber, 2018) . The variational approach exemplified by the VAE (Kingma & Welling, 2013) has the benefit of tractable inference and uncertainty estimation while also allowing reasonable control over bottleneck representation and reconstruction quality. While a number of variants have been proposed to incorporate expressive priors for the VAE latent distribution, such as mixtures of Gaussians (Dilokthanakul et al., 2016) and discrete formulations (Rolfe, 2016) , the VAE decoder is very limited given that data dimensions are decoded as conditionally independent Gaussian distributions. VAE samples therefore tend to be blurry and exhibit little meaningful variation around the predicted mean.

On the other hand, complex multimodal decoder distributions can be learned by alternative generative models, including GANs (Goodfellow et al., 2014) and autoregressive models (e.g., pixelCNN) (van den Oord et al., 2016) . Conditional formulations of these models (Mirza & Osindero, 2014; Van den Oord et al., 2016) allow controlling the sampling behavior to be class or embedding specific. Typically a known class label or metric embedding is given when training these conditional models. Conditioning on a class label or embedding allows, for instance in the case of face images, to generate a range of different images of a given person identity varying in pose, lighting, expression, etc., by conditioning on the latent embedding learned by a face recognition network (e.g., Van den Oord et al., 2016) . These models can be difficult to train to ensure that the decoder pays attention to the conditioning variable.

Encoder-decoder architectures such as VAE- GAN (Larsen et al., 2015) and BiGAN (Donahue et al., 2016) improve sample and reconstruction sharpness through their use of adversarial losses. However, none of these models incorporate multimodal decoders into their formulations, and therefore do not lead to diverse sampling behavior.

In this work we introduce the Hierarchical Bayes Autoencoder (HBAE). The model incorporates a multimodal decoder, in the form of an EBM, which we show under mild assumptions can be trained similarly to a conditional GAN, where the conditioning is the latent code of a VAE. We show that the model is capable of learning to generate crisp and varied samples. Empirically, the stochastic reconstructions produced by the decoder retain semantic features of the respective input examples as verified by distance in Inception feature space, while unconditionally generated samples are diverse and cover the data manifold, as verified with FID scores (Heusel et al., 2017) .

We also introduce a simple extension called the Set-HBAE where each data point is a collection of examples (e.g., set of images). In this scenario, the traditional VAE is no longer applicable, because of the inherent one to many mapping between the latent code and the set elements. We show that Set-HBAE is able to be successfully applied to such a setting, and learns to reconstruct input sets in a semantically meaningful way, as well as generate new sets unconditionally by sampling from the latent code prior. Furthermore, we show that the Set-HBAE works well as a method for few-shot classification on ShapeNet (Chang et al., 2015) relative to a supervised baseline, especially in the small data regime.

To summarize, the contributions of this work are:

1. A novel probabilistic generative model trained using variational inference, with a flexible learned multimodal decoder in the form of an EBM.

2.. Empirical results:

showing that the multimodal decoder is able to produce stochastic reconstructions varying around the local data manifold of examples, and diverse unconditional samples.

3. An extension of the HBAE formulation to model sets of inputs, allowing a finer degree of semantic control over sampling behavior, and a new method for few-shot classification of sets.

Figure 1 : Left: Graphical model for a VAE for sampling N data points, where each sample x is conditioned on a latent code Z sampled from a prior, and data dimensions x 1 , x 2 , x 3 are conditionally independent given z. Middle: In the HBAE, the decoder distribution p θ (x|z) is replaced by a deep EBM, where data dimensions can interact via undirected connections. Right: In the Set-HBAE, we first sample N set codes z, and we then sample M set elements by repeatedly generating from the EBM conditioned on the set latent code.

Figure 2 : Architectures for different models. For each architecture, we use same colors to group together modules that contribute cooperating terms to the adversarial loss formulation. The HBAE can be interpreted as a cGAN with a VAE encoder where all modules are learned jointly.

2. METHODOLOGY:

2.1 VAES VAEs (Kingma & Welling, 2013) represent a family of deep directed graphical models. Given an input x ∈ X , a VAE formulates a joint distribution of x and a latent variable z ∈ R l as

EQUATION

where θ denotes the set of trainable parameters; p θ (z) and p θ (x|z) denote the prior and decoder distribution, respectively. As a common design choice, p θ (z) is usually set as a simple low dimensional distribution (e.g., an Isotropic Gaussian) which can be easily sampled from; p θ (x|z), referred to as the decoder, is parameterized as a Gaussian distribution with a learned mean µ(z; θ) and an identity covariance matrix. This popular instantiation of VAEs offers a simple objective function which corresponds to a regularized autoencoder where reconstructions are penalized with 2 loss.

We argue that such a unimodal decoder is not optimal for at least two scenarios. Firstly, in the case where the dimensionality of z is much lower than the inherent dimensionality of the input x, a unimodal decoder becomes the bottleneck of learning. In an extreme case, if the latent space is limited to contain only k bits, e.g., a categorical VAE as in Jang et al. (2016) with 2 k latent units, with k being a small quantity, the model is forced to assign multiple inputs to any given latent code z. This makes having a multimodal decoder essential to effectively fit the data distribution. Secondly, for settings where the input does not have a vector representation, such as set inputs (Zaheer et al., 2017) , it is not clear how the unimodal decoder can be applied.

2.2. HIERARCHICAL BAYES AUTOENCODER:

We now present the Hierarchical Bayes Autoencoder (HBAE), which addresses the drawbacks of the VAE as discussed previously. The key insight is to design a multimodal decoder that can flexibily model the implicit (and explicit in the set case) one-to-many mapping from a latent code to target samples. To do so, we replace the decoder distribution of a VAE with an EBM-based decoder LeCun et al. (2006) .

Formally, we start by using the joint distribution in Equation 1. We then define p θ (x|z) =

e -E(x|z;θ)

x e -E(x|z;θ) dx , with E(x|z; θ) being the energy function. From the graphical model perspective, this formulation removes the conditional independence assumption between the dimensions of x given z, and allows for modeling rich interactions between them instead. This enables the decoder to assign high probability to multiple possible outputs, if needed (refer to Figure 2 for an illustration). Note that the EBM formulation of the decoder is a strict generalization of Gaussian and other exponential family energy-based models; see LeCun et al. (2006) for more explanations.

During training, we similarly apply a variational distribution q φ (z|x) as in a VAE, which leads to the following variational objective:

EQUATION

where the first row is the standard variational lower bound of a VAE, and the following two rows are the result of plugging in the EBM defination of the decoder.

Directly minimizing Equation 2 as an objective function is difficult as it involves the logarithm of a partition function. As a solution, we turn to recent connections between deep EBMs and GANs (Zhao et al., 2016; LeCun et al., 2006; Wang & Liu, 2016; Zhai et al., 2016) . As a brief recap, following Zhai et al. (2016) , for an EBM with energy E(x; θ), we can write its negative log likelihood (NLL) as: E(x;θ) dx], which can be further developed as:

E x∼p data (x) [E(x; θ)] + log[ x e -

E x∼p data (x) [E(x; θ)] + log x q ψ (x) e -E(x;θ) q ψ (x) dx = E x∼p data (x) [E(x; θ)] + log E x∼q ψ (x) [ e -E(x;θ) q ψ (x) ] ≥E x∼p data (x) [E(x; θ)] + E x∼q ψ (x) [log e -E(x;θ) q ψ (x) ] = E x∼p data (x) [E(x; θ)] -E x∼q ψ (x) [E(x; θ)] + H(q ψ ),

(3) where q ψ (x) is an auxiliary distribution with parameters ψ, with H(q ψ ) denoting its entropy. We then let D(x; θ) = -E(x; θ) and q ψ (x) = z p ψ (z )δ(G(z ; ψ))dz (i.e., an implicit distribution defined by a generator G and a known prior p ψ (z )), which leads to an optimization procedure as follows:

EQUATION

In practice, we rely on implicit entropy regularizers such as batch normalization (Ioffe & Szegedy, 2015) in place of the entropy term H(q ψ ), which gives us a learning objective:

EQUATION

which can be essentially implemented as a GAN, in particular a WGAN (Arjovsky et al., 2017) , with D being the discriminator and G being the generator.

We then accordingly apply the same derivation and change of variables to Equation 2, and obtain a practical objective for HBAE:

min θ,φ max ψ E x∼p data (x) [E z∼q φ (z|x) [-D(x|z; θ)+E z ∼p ψ (z |z) D(G(z |z; ψ)|z; θ)]+KL(q φ (z|x)||p θ (z))].

(6) Here both the discriminator D and generator G are conditional on z, which is a sample from the encoder q φ . The additional noise distribution p ψ (z |z) can also be conditional, but we set it to a parameter free Gaussian distribution in practice. Furthermore, we can also extend the objective above in a similar way to other VAE variants such as Higgins et al. (2017) ; Zhao et al. (2017) ; Tolstikhin et al. (2017) where the KL divergence term is generalized to a weighted distance measure between the aggregated posterior distribution q φ (z) = E x∼p data (x) q φ (z|x) and the prior p θ (z), which gives rise to our final objective:

min θ,φ max ψ E x∼p data (x) [E z∼q φ (z|x) [-D(x|z; θ)+E z ∼p ψ (z |z) D(G(z |z; ψ)|z; θ)]]+βD(q φ (z)||p θ (z)), (7)

where D denotes a distance between distributions, which we choose the MMD distance as in Tolstikhin et al. (2017) , β is a regularization factor.

The training procedure of HBAE is similar to that of a conditional GAN, except that the conditioning variable is learned jointly. To be concrete, given a fixed encoder q φ and discriminator D, the conditional generator G is updated in the inner loop to increase the conditional discriminator's score on G's output. In the outer loop, the encoder q φ and discriminator D are jointly updated adversarially against G. During inference, we can sample from the generator as with other generative models. However, in a HBAE, sampling follows a two step hierarchy. One first sample a latent variable z, either from the encoder distribution q φ (z|x), or from the prior p θ (z). One then samples from a secondary z ∼ p ψ (z |z) to generate an example. This sampling procedure resembles the inference process of a Hierarchical Bayes model, and hence we name it Hierarchical Bayes Autoencoder.

2.3. SET-HBAE:

Given the HBAE formulation, it is also easy to extend the same generative modeling setting to cases where inputs are sets of images (e.g., different views of the same object). We refer to this variant as the Set-HBAE. In order to do so, we explicitly express the input in the set format x = {x 1 , x 2 , ..., x k } where k denotes the size of the set. By construction, we let the elements within a set x to be conditionally independent given the set latent code z (see Figure 1 ). We then accordingly modify the HBAE objective Equation 7 as:

EQUATION

where |x| denotes the size of the set. The encoder q φ in this case takes form of a set encoder. Here we adopt the DeepSet (Zaheer et al., 2017) mechanism to average pool the codes of elements within the set; the form of the generator remains unchanged. During inference time, the hierarchical sampling procedure naturally corresponds to sampling a set code z first, followed by sampling an element code z . In this way, the Set-HBAE is able to perform both stochastic set reconstruction, and well as unconditional set generation.

3. RELATED WORK:

Autoencoding generative models. The HBAE is similar to a handful of existing works which involve an encoder, decoder and a discriminator. Notably, BiGAN (Donahue et al., 2016) trains a two channel discriminator that tries to separate the two joint distributions p(z)p(x|z) and p(x)p(z|x), which involves an encoder and decoder, respectively. BiGAN differs from the HBAE in that the mapping between z and x is unimodal. Also, in a BiGAN, the encoder and decoder work jointly against the discriminator, whereas in the HBAE, the encoder and the discriminator work jointly against the generator. The VAE-GAN (Larsen et al., 2015) is a VAE/GAN hybrid model that consists of the three modules. In addition to the deocder being unimodal, a VAE-GAN deploys an unconditional discriminator, which contrasts with the conditional discriminator used in HBAE.

Conditional GANs. A conditional GAN (cGAN) (Mirza & Osindero, 2014; Miyato & Koyama, 2018) modifies the generator and discriminator of a GAN to be conditional on class labels. The HBAE can be considered a generalization of cGANs, as it learns the conditioning representation jointly with the generator and discriminator. To this end, it is possible to come up with a semisupervised learning version of HBAE where a small portion of input data have observed labels, which are jointly trained with unlabelled data. Refer to Figure 2 for a comparison of the HBAE with the cGAN, BiGAN and VAE-GAN architectures.

PixelCNN-based multimodal decoders. Another line of work that employs a multimodal encoder is represented by PixelVAE (Gulrajani et al., 2016) , which uses a PixelCNN-based decoder ( Van den Oord et al., 2016) . While similar in spirit to our work, PixelCNN suffers from capturing long term dependency of images, and is also expensive during inference time. We see HBAE as providing an alternative approach to this line of work, which leverages the power of state of the art GANs for decoding.

VAEs with large latent codes. Another design philosophy of VAEs is represented by VQ-VAE2 (Razavi et al., 2019) , where a large latent code is used for each example, with preserved spatial dimensions. VQ-VAE2 then learns a powerful prior in the form of a PixelCNN, which is able to generate good quality samples at inference time. We note that in this scenario, the use of a unimodal decoder is well justified where the latent space is large enough to allow learning a oneto-one mapping between a latent code and an input. The HBAE on the other hand, works best in the setup where a compact latent code is used, or in applications where explicit multimodality is desired, such as with sets as inputs.

Generative Modeling of sets. Generative modeling of sets is a challenging task. Hierachical Bayes models have previously been widely applied to document modeling, which can be considered as a set of words, represented by the LDA model (Blei et al., 2003) . However, due to the discrete nature of words, LDA only needs to rely on a hierarchy of simple distributions such as the Dirichlet and multinomial distributions. Recently, there has been a a body of work dedicated to point cloud generative modeling, where sets are composed of 3D points in Euclidean space (Li et al., 2018; Achlioptas et al., 2017) . However, sets considered in our work contain much richer structure than point clouds, which contain very little structure per element. It is thus unclear how methods developed from the point cloud or document modeling lines of work generalize to more complex data.

4. EXPERIMENTS:



4.1. ARCHITECTURE:

The HBAE defined in Equation 7has three trainable modules: an encoder which corresponds to q φ (z|x)1 , a conditional discriminator D(x|z; θ), and a conditional generator G(z |z; ψ). Our architecture design largely mimics that of a conditional GAN, in particular the projection-based discriminator proposed in Miyato & Koyama (2018) . Here the latent code, z, takes place of the label, y, in a conditional GAN, and is jointly learned with other modules instead of being given. We follow a simple design choice for the architectures, where both the encoder and discriminator consist of stacks of residual blocks with Spectral Normalization (Miyato et al., 2018) , and the generator consists of stacks of residual blocks with Batch Normalization (Ioffe & Szegedy, 2015) . Refer to Figure 3 for more details. 

4.2. TRAINING PROTOCOL:

In all the experiments, we follow the training protocols of standard GANs as in Donahue et al. (2016) , where the encoder and discriminator are trained jointly against the generator, as shown in Equation 7. We use Adam with a learning rate 2e-4 for the generator , and learning rate 4e-4 for the encoder and discriminator. The moment terms are set as {} for both optimizers. In practice, one failure mode we have found by directly following Equation 7 is that the encoder can collapse to a constant mapping which makes HBVE regress to an unconditional GAN. As a remedy, we put an additional reconstruction regularization term on the encoder's parameters

R(φ) = λE x∼p data (x) E z∼q φ (z|x)

x -G(0|z) 1 (for the Set-HBAE, this corresponds to mapping z to all elements within the set:

λE x∼p data (x) E z∼q φ (z|x) 1 |x| x i ∈x x i -G(0|z) 1 ).

Here λ is the regularization strength which we set as 1. We note that this regularization term is not essential for HBAE to train, but in most cases helps with the stability.

4.3. SINGLE IMAGE INPUTS:

In the first set of experiments, we test the HBAE in the standard setting where inputs are single images. We use the CelebA Dataset (Liu et al., 2015) with resolution 64 × 64. In Figure 4 , we show some of the qualitative results. We see that the HBAE is able to reconstruct input examples in a semantically meaningful way by preserving certain aspects of the input face, while outputting variations that exhibit rich and changing details, such as adding or removing glasses, varying expressions subtly, or changing hairstyles. When sampled unconditionally, the HBAE is also able to produce sharp examples. As control experiments, we have accordingly trained a GAN and VAE using the same architectures (encoder and generator), and reported the FID score of the unconditional generations, and reconstructions (when applicable), in Table 1 . We have also included the reconstruction MSE of the VAE and HBAE evaluated in the pixel space and Inception V3's feature space. Refer to the caption for detailed discussions.

Additionally, we show that the HBAE can be used to sample images in a Markov chain, exploring the data manifold around an example as a random walk. The Markov chain starts from a real example, and recursively passes it through the encoder and the generator. We show two such chains in Figure 5 , starting from the same test examples, both of which exhibit smooth transitions from one sample to the next, without burning into a single mode. Interestingly, the two chains also lead to different sampling paths, due to the multimodality of the generator. This behavior is not possible with regular VAE or GAN formulations.

4.4. SET OF IMAGES AS INPUT:

In the second set of experiments, we change the inputs to be a set of images. The first dataset we use is the ShapeNet dataset from Chang et al. (2015) , which consists of 3d objects with projected 2d views. We use a subset which contains 13 popular categories, namely airplane, bench, cabinet, car, chair, monitor, lamp, speaker, gun, couch, table, phone, ship. The training set consists of 32,837 Table 1 : FID scores together with reconstruction MSE evaluated in pixel space and Inception's feature space. In a control experiment setting, HBAE achieves significantly better FID scores than the VAE counterpart, while being more comparable to unconditionally trained GANs. As a reference, the FID score for ground truth test data is 4.08. Next, reconstruction MSE for the HBAE is significantly larger than the VAE counterpart evaluated in pixel space, as expected given that the decoder is multimodal and stochastic. However, when evaluated in Inception's feature space, HBAE and VAE metrics are comparable. As a reference, the Inception MSE for a random pair of test images is 0.126. unique objects and 788,088 images in total. The second dataset we use is the VGGFace2 dataset from Cao et al. (2018) , which is a face dataset containing 9,131 subjects and 3.31M images.

Model:

For both datasets, we randomly construct fixed-size sets of 8 elements. For ShapeNet, this is done by randomly selecting an object and 8 of its views; for VGGFace2, we similarly select one identity and 8 of her/his faces. A mini-batch is constructed by selecting N (which we set to 32) objects/subjects, which amounts to N x8 images. Example set inputs together with their stochastic reconstructions are shown in Figure 6 . The qualitative results indicate that the samples are diverse, yet are consistent with each other and tend to resemble the semantic traits of their respective input sets (i.e., same object in different views for the ShapeNet case and same identity with varying attributes for the VGGFace2 case). As a further quantitative measure, we also conducted few shot classification experiments on ShapeNet with the learned features. We compare the features learned from our Set-HBAE with a control VAE, as well as with randomly initialized features, and with supervised training results, all with the same encoder architecture. For all settings, we trained a linear classifier with crossentropy loss, without any explicit data augmentation or regularization (except for the supervised learning setting where the encoder is jointly trained with the linear classifier). We vary the number of training images from 64 to 2048, and report the results in Table 2 . We see that in the few shot classification domain, the learned features derived from the Set-HBAE largely outperform the supervised learning approach as well as the VAE results, all of which far exceed random features.

Finally, we show the Set-HBAE's ability to generate unconditional samples. The generated results are shown in Figure 7 for both ShapeNet and VGGFace2. Notably, the Set-HBAE is able to generate diverse sets, where elements within the generated sets are also diverse but consistent in ways that preserves object shape or face attributes. To the best of our knowledge, this is the first generative model that is able to generate complex sets with richly varying yet coherent structure.

5. CONCLUSION:

In this paper, we presented the HBAE, a novel probabilistic generative model with an EBM-based multimodal decoder. In our implementation, we jointly trained the encoder with variational inference and the EBM decoder with a powerful conditional GAN. We demonstrated that the decoder is capable of generating diverse conditional and unconditional samples. Then we introduced the Set-HBAE, a novel generative model for sets of examples, generalizing the HBAE framework. The Set-HBAE takes sets as input and can sample plausible variations that are consistent with the conditioning set, but generalize beyond the specific input instances. In addition, we show that the Set-HBAE performs well on a ShapeNet Chang et al. (2015) few-shot classification task, showing the advantage of leveraging generative models in the small data regime. The ability to sample plausible variations conditioned on an input example or set of examples is a key building block for enabling predictive models, for which we see future applications in areas such as video (Lee et al., 2018) , 3D scene generation (Eslami et al., 2018) , and model-based reinforcement learning (Ha & Schmidhuber, 2018) . 

:

In our implementations, we adopt a deterministic encoder which corresponds to setting q φ as a δ distribution, which gives the most stable results.

Figures and Tables:

Figure 3: HBAE implementation. Left: Encoder. Middle: Conditional generator. Right: Conditional discriminator, which takes as input ground truth x or sample x. The discriminator score is a dot product of the intermediate representation and the latent code D(x|z) = h(x) T z.

Figure 4: Reconstructions and samples from HBAE. Left: input images from the test set (first row) and four stochastic reconstructions with different z (second through last rows). Right: unconditional samples (first row) and four stochastic samples with shared z and different z .

Figure 5: Markov chains starting from real examples from the test set (top row) and 6 recursive consecutive stochastic samples (second through last rows). We run two consecutive Markov chains staring from the same examples, corresponding to columns in the left and right plots.

Figure 6: Reconstructions of the ShapeNet (left figure) and VGGFace2 (right figure) daatasets with a Set-HBAE. For each block, the first row corresponds to an input set from the test split, and the second row shows one (stochastic) reconstruction of the set from the Set-HBAE.

Figure 7: Uncurated unconditional generations from a Set-HBAE trained on ShapeNet (top) and VGGFace2 (bottom) with set size fixed to 8. Each column corresponds to a sampled set. The Set-HBAE generates sets that are coherent, and often with good visual quality.

Figure 8: VAE results on the Celeba Dataset. Top left: random samples of Celeba image; top right: reconstructions of a VAE; bottom: unconditional samples from a VAE.

Few shot classification results on ShapeNet(Chang et al., 2015). We show the test accuracy of each features while varying the number of training examples. Here random denotes uninitialized features with the same architecture, and supervised denotes the supervised training setting with the same encoder architecture.

Table content: <table><tr><td/><td colspan="5">FID recon FID gen MSE Inception MSE pixel</td></tr><tr><td>HBAE (ours)</td><td>12.08</td><td/><td>16.72</td><td>0.076</td><td>0.105</td></tr><tr><td>VAE</td><td>44.57</td><td/><td>65.97</td><td>0.071</td><td>0.019</td></tr><tr><td>GAN</td><td>-</td><td/><td>9.13</td><td>-</td><td>-</td></tr><tr><td colspan="2"># Training example</td><td>64</td><td colspan="3">128 256 512 1024 2048</td></tr><tr><td colspan="2">Set-HBAE (ours)</td><td colspan="4">50.5 56.0 65.1 70.8 74.0 76.2</td></tr><tr><td>VAE</td><td/><td colspan="4">39.2 49.9 56.3 60.0 60.7 62.9</td></tr><tr><td>Random</td><td/><td colspan="4">19.2 19.2 19.2 19.2 19.2 19.2</td></tr><tr><td>Supervised</td><td/><td colspan="4">44.7 52.2 60.5 64.4 70.2 76.3</td></tr></table>

Peer review (iclr2020):
The article proposes to replace the conditional distribution p(x|z) in an VAE by a general graphical model parameterized by a deep neural network. Using a series of approximations it arrives at a triplet of models, where the graphical model is split into a model representing the likelihood and a model that generates samples from the distribution. the third model is the usual autoencoder. The core idea is that the latent space of the VAE serves as bottleneck so that the graphical model develops a multi-modal distribution. At the same time, the use of a neural network allows for a much better modeling than using the typical local Gaussian distribution as is used in most VAE-approaches. Experimental results show that reconstructions and markov-chains starting from reconstructions produce decent results. An extension to sets is proposed as well as a task showing performance for semi-supervised learning. ------------------------------------------- The text above serves mainly as a summary of what the paper sets out to do. However, we have to check whether the paper is actually achieving this goal. In my view, it does not, and it suffers from severe problems on the theoretical side. Furthermore, important evaluations that are standard within the VAE framework are missing: samples from the true underlying distribution. In the remainder of this review, I will try to substantiate the claims regarding theory which lead me to an overall decision to reject the article in its current form. Let us start from a birds-eye perspective: the model consists of three parts: the Encoder E, which i will refer to as q(z|x) the Energy-function, also called E(x|z) and with -E referred to as D later on. a generator function G(z,z*) that is supposed to sample from the distribution p(x|z) = 1/Z(z) exp(-E(x|z)) using entropy generated by z*~N(0,I). I will refer to this distribution as r(x|z) This split is smart: using a general graphical model to model p(x|z) makes it very hard to sample from the distribution and it is difficult to estimate the normalization constant Z. The paper refers to Zhai et.al (2016) to find eq(3). I really like (3) because the inequality is tight when G is producing samples from p(x|z). To be more exact, we have that the estimator for the normalization constant is log(Z) - KL(r(x|z) || p(x|z) ). This reveals the first issue: the generator must have an r(x|z) > 0 for all x except a countable infinite subset. This is difficult to achieve with dimensionality of z and z* smaller than dimensionality of x and G being deterministic. Therefore, KL(r(x|z) || p(x|z) ) is already infinite. This is a technicality that i will ignore, since we can always add a minimal amount of noise on the output of G, even though this paper does not. The real problem starts with eq(4). Since r(x|z) is implicit, there is no way to compute its entropy, which is then just discarded. The paper argues that this is no problem, because they intend to somehow bound the entropy. We will look at this point, later. The issue with bounding is that for this to be meaningful, we also need a bound on the entropy of p(x|z), because otherwise there is no way for r(x|z) to achieve a small KL-divergence. If we now look at (4), we can compute the optimal r without bounded entropy. The optimal r is the distribution that by (5) maximizes -E(x|z) for a given z. this result is not a multi-modal distribution but a delta-peak on the optimum and certainly not p(x|z). so the bound is super important and technically we need a bound on log|det(d/dz* G(z,z*))|. This is difficult from a theoretical view-point as well as from a modeling view-point because we know that we have to have nearly discontinuous transformations to create sharp edges. Further, to transform z*~N(0,I), into a multi-modal distribution, there must be a function g(z*)->w so that w is multi-modal. This is difficult to achieve with bounded determinant. Looking at the experiments, let*s see whether the determinant is actually bounded. The paper claims that bounding can be achieved using batch-normalization. First this requires a reference on why this should be the case and show that this holds as implemented for the resnet block and second, this requires an estimate for how large the entropy can still be. if the entropy can be very large, we can not claim to train a VAE, because the error going from (3) to (4) is large. Secondly, we have to ensure that both E and G have bounded entropy, because otherwise KL(r(x|z) || p(x|z) ) will be large and there is no way to that the approximation in (3) is good, at which point E can not be claimed as doing maximum-likelihood on the data. As shown in Figure 3, the Generator has a Conv-Block and the initial layer is a linear layer. Similarly the discriminator (or E) has a linear output layer. Therefore, the entropy is not bounded as the paper does not claim to have any entropy normalization here. The paper does not say what the activation functions are, but if the final layer of E is linear or Relu, we can add yet another problem to the list (next to the unbounded Jacobian): the VAE framework stops making sense. If we look at eq (6) or (7), the final divergence term (multiplied by beta) hinges on the fact that it provides a suitable error contribution. However, if the final layer of D is linear, we can just scale that layer arbitrarily large until the error contribution of the divergence vanishes. Lastly, the Energy-function does not even necessarily model a probability function. Remember, we need that Z(z)=int exp(-E(x|z)) dx <infty. However, the definition by Figure (3) is E(x|z)=-h(x)^T(z) and the output layer of h is linear. so we can find weights such that h(x)=c forall x. In which case Z(z)=int exp(-c^Tz) dx = exp(-c^Tz) int 1 dx which is only bounded if x is bounded, which I can not see since the generator is not bounded as well. This is only one of many counter-examples. A few other small things: - For z from a normal distribution the bottleneck discussion in 2.1 last paragraph does not apply, because for a dataset with N samples it is always possible to partition the real-space of Z into N partitions with probability 1/N each. This even works with 1-dimensional normal distributions, independently of the dimensionality of x. So a multi-modal distribution for p(x|z) is really not necessary if encoder and decoder are strong enough to memorize the dataset. - below (5) it is claimed that this is similar to a WGAN, but WGAN have a difficult constraint to be 1-Lipschitz to make it work (which proves to be extremely difficult in practice as it translates to a similar inequality constraint on the jacobian) - In Fig(5) it is unclear to me how a markov-chain is defined using the model. I assume this is done by taking samples from z*, generating a new x and encoding this to obtain a new z. If z* models different features from z (assuming z* is multi-modal) we would assume that the z would remain the same in each point, so no exploration of p(z) takes place. However, we can already see in the lat rows that the pictures look more strange, so some mixing is taking place but it is not clear how that relates to p(z). Therefore i would really like to see images generated by sampling z~N(0,I) and z*~N(0,I) to see whether the generated images look reasonable. - Moreover, to show that the Generator actually produces images that are likely under p(x|z), we would also need samples generated from the true underlying distribution p(x,z), e.g. using hamilton monte-carlo. - Finally, i have to notice that the model, while called hierarchival VAE, does not provide a way to compute z* from the input. is that than really an autoencoder?