Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: I don't think this paper should be accepted.  In my opinion, the mix of EBM and VAE is not really compelling;  and it is not clear at all to me that one gets much from the "V" in this setting.  Furthermore,  the experimental results are not great either qualitatively (by the standards of generative-models-of-images in 2019) or quantitatively (even by the standards of GAN papers).  Finally, the author's claims about sets seems tacked on, and unrelated to the rest of the paper.  Modern neural networks (attention/transformers/graph-nn, etc...) handle sets naturally, and could be used with any other conditional generative model.  To my eye, the results in e.g. figure 6 do not really seem like the model is matching the set, and the authors make no attempt to formalize or quantify how well their model generates things "in the set".  

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: In this work a novel probabilistic generative model is introduced mixing several existing frameworks: The Hierarchical Bayes Autoencoder (HBAE) can be interpreted as a cGAN with a VAE encoder. They also incorporate a flexible learned multimodal decoder in the form of an EBM. The authors claim to produce stochastic reconstructions varying around the local data manifold of examples, and diverse unconditional samples. In addition, they present an extension of their HBAE formulation in order to model sets of inputs. This is one of the main contributions of this work since generative modeling of sets is a challenging and unsolved task. 

Although there is a clear explanation of the contributions of this paper, the motivation of this study is not precisely described. The introduction provides a good understanding of the topic; however, the authors may wish to provide several examples on the interest of the study. They provide a good description of the existing generative modeling architectures and the applications involved but not mention the importance of their contributions to the real world. 

The proposed formulation seems encouraging thanks to the incorporation of multimodal decoders. The derivation of HBAE is theoretically well justified, as a result, one could replicate or further work in this paper. The authors explain in detail the procedure they took in order to arrive to the final HBAE formulation giving a clear understanding of the topic they present.

By describing the related works, they give an understandable perspective about the different drawbacks and differences of the existing methods. One could deduce the motivations of this work thanks to this section, however they should have been state clearer at the beginning. 

The authors explain in detail all the different parts of the architecture, presenting precise information about the different layers. At first sight, it seems like one could reproduce the methodology used. It is clearly explained and discussed.

I think the experiments conducted to evaluate this work are not enough. The authors performed several experiments in order to prove the efficiency of their methodology by showing the generated images and investigate the qualitative results. If we look at the results in figure 4, we can observe some artifacts and blur regions in the image. Moreover, the variations on the faces sometimes seem more deformations than a different human feature. It would have been interesting to evaluate the identity preservation to further investigate the quality of the images. Although the methodology, conceptually, suggested an interesting approach, the results are not so encouraging given the quality of the images achieved by other methodologies.

Apart from comparing quantitatively the capacity of the generated images with other related works, it would have been interesting to show that qualitatively by showing images from the mention methods. In general, there is some information missing in order to replicate the results and more details would have been appreciated. 

Regarding the experiments about the set of images as input, the quality of the images is similar to the simple HBAE. Since this is a more challenging approach and one could not compare with so many existing methodologies, it seems like they are going to the right direction to achieve the desired images. However, looking at the exposed results, the quality of the images is not realistic and has a lot of artifacts. Some other experiments could have been performed in order to show more interesting results.

Questions:

-	Could you better explain the meaning of figure 1? I think it was a good idea to exemplify the differences but is not clear enough.
-	Did you perform any face alignment when dealing with the different faces databases?
-	Did you think about the possibility of giving a concrete condition for varying the image?
-	Which could be some concrete applications for your work?

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The article proposes to replace the conditional distribution p(x|z) in an VAE by a general graphical model parameterized by a deep neural network. Using a series of approximations it arrives at a triplet of models, where the graphical model is split into a model representing the likelihood and a model that generates samples from the distribution. the third model is the usual autoencoder. The core idea is that the latent space of the VAE serves as bottleneck so that the graphical model develops a multi-modal distribution. At the same time, the use of a neural network allows for a much better modeling than using the typical local Gaussian distribution as is used in most VAE-approaches. Experimental results show that reconstructions and markov-chains starting from reconstructions produce decent results. An extension to sets is proposed as well as a task showing performance for semi-supervised learning.

-------------------------------------------
The text above serves mainly as a  summary of what the paper sets out to do. However, we have to check whether the paper is actually achieving this goal. In my view, it does not, and it suffers from severe problems on the theoretical side. Furthermore, important evaluations that are standard within the VAE framework are missing: samples from the true underlying distribution. In the remainder of this review, I will try to substantiate the claims regarding theory which lead me to an overall decision to reject the article in its current form.

Let us start from a birds-eye perspective: the model consists of three parts:
the Encoder E, which i will refer to as q(z|x)
the Energy-function, also called E(x|z) and with -E referred to as D later on.
a generator function G(z,z') that is supposed to sample from the distribution p(x|z) = 1/Z(z) exp(-E(x|z)) using entropy generated by z'~N(0,I). I will refer to this distribution as r(x|z)

This split is smart: using a general graphical model to model p(x|z) makes it very hard to sample from the distribution and it is difficult to estimate the normalization constant Z. The paper refers to Zhai et.al (2016) to find eq(3). I really like (3) because the inequality is tight when G is producing samples from p(x|z). To be more exact, we have that the estimator for the normalization constant is log(Z) - KL(r(x|z) || p(x|z) ). This reveals the first issue: the generator must have an r(x|z) > 0 for all x except a countable infinite subset. This is difficult to achieve with dimensionality of z and z' smaller than dimensionality of x and G being deterministic. Therefore, KL(r(x|z) || p(x|z) ) is already infinite. This is a technicality that i will ignore, since we can always add a minimal amount of noise on the output of G, even though this paper does not.

The real problem starts with eq(4). Since r(x|z) is implicit, there is no way to compute its entropy, which is then just discarded. The paper argues that this is no problem, because they intend to somehow bound the entropy. We will look at this point, later. 

The issue with bounding is that for this to be meaningful, we also need a bound on the entropy of p(x|z), because otherwise there is no way for r(x|z) to achieve a small KL-divergence. If we now look at (4), we can compute the optimal r without bounded entropy. The optimal r is the distribution that by (5) maximizes -E(x|z) for a given z. this result is not a multi-modal distribution but a delta-peak on the optimum and certainly not p(x|z). so the bound is super important and technically we need a bound on log|det(d/dz' G(z,z'))|. 

This is difficult from a theoretical view-point as well as from a modeling view-point because we know that we have to have nearly discontinuous transformations to create sharp edges. Further, to transform z'~N(0,I), into a multi-modal distribution, there must be a function g(z')->w so that w is multi-modal. This is difficult to achieve with bounded determinant.

Looking at the experiments, let's see whether the determinant is actually bounded. The paper claims that bounding can be achieved using batch-normalization. First this requires a reference on why this should be the case and show that this holds as implemented for the resnet block and second, this requires an estimate for how large the entropy can still be. if the entropy can be very large, we can not claim to train a VAE, because the error going from (3) to (4) is large. Secondly, we have to ensure that both E and G have bounded entropy, because otherwise KL(r(x|z) || p(x|z) ) will be large and there is no way to that the approximation in (3) is good, at which point E can not be claimed as doing maximum-likelihood on the data.

As shown in Figure 3, the Generator has a Conv-Block and the initial layer is a linear layer. Similarly the discriminator (or E) has a linear output layer. Therefore, the entropy is not bounded as the paper does not claim to have any entropy normalization here. The paper does not say what the activation functions are, but if the final layer of E is linear or Relu, we can add yet another problem to the list (next to the unbounded Jacobian): the VAE framework stops making sense. If we look at eq (6) or (7), the final divergence term (multiplied by beta) hinges on the fact that it provides a suitable error contribution. However, if the final layer of D is linear, we can just scale that layer arbitrarily large until the error contribution of the divergence vanishes. 

Lastly, the Energy-function does not even necessarily model a probability function. Remember, we need that Z(z)=\int exp(-E(x|z)) dx <\infty. However, the definition by Figure (3) is E(x|z)=-h(x)^T(z) and the output layer of h is linear. so we can find weights such that h(x)=c \forall x. In which case 
Z(z)=\int exp(-c^Tz) dx = exp(-c^Tz) \int 1 dx 
which is only bounded if x is bounded, which I can not see since the generator is not bounded as well. This is only one of many counter-examples.


A few other small things:
- For z from a normal distribution the bottleneck discussion in 2.1 last paragraph does not apply, because for a dataset with N samples it is always possible to partition the real-space of Z into N partitions with probability 1/N each. This even works with 1-dimensional normal distributions, independently of the dimensionality of x. So a multi-modal distribution for p(x|z) is really not necessary if encoder and decoder are strong enough to memorize the dataset.
- below (5) it is claimed that this is similar to a WGAN, but WGAN have a difficult constraint to be 1-Lipschitz to make it work (which proves to be extremely difficult in practice as it translates to a similar inequality constraint on the jacobian)
- In Fig(5) it is unclear to me how a markov-chain is defined using the model. I assume this is done by taking samples from z', generating a new x and encoding this to obtain a new z. If z' models different features from z (assuming z' is multi-modal) we would assume that the z would remain the same in each point, so no exploration of p(z) takes place. However, we can already see in the lat rows that the pictures look more strange, so some mixing is taking place but it is not clear how that relates to p(z). Therefore i would really like to see images generated by sampling z~N(0,I) and z'~N(0,I) to see whether the generated images look reasonable. 
- Moreover, to show that the Generator actually produces images that are likely under p(x|z), we would also need samples generated from the true underlying distribution p(x,z), e.g. using hamilton monte-carlo.
- Finally, i have to notice that the model, while called hierarchival VAE, does not provide a way to compute z' from the input. is that than really an autoencoder?
review_title: Official Blind Review #3

Meta-review: This paper introduces a probabilistic generative model which mixes a variational autoencoder (VAE) with an energy based model (EBM). As mentioned by all reviewers (i) the motivation of the model is not well justified (ii) experimental results are not convincing enough. In addition (iii) handling sets is not specific to the proposed approach, and thus claims regarding sets should be revised.

Decision: Reject