Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The main contribution of the paper is in the novel training procedure in which the target distribution and the generated distributions are mapped to a latent space where their divergence is minimised.  Training procedures for the same based on MLE and VAE are presented as well.

The idea is quite interesting but there are certain problems in clarity regarding the loss terms and the precise need for them. It appears that the work is proposing a novel framework for training VAEs where instead of comparing the target and generated distribution in the data space (the standard $L_r$), the loss is minimised by minimizing the divergence between these distributions after mapping them onto a latent space. To achieve this objective the author’s design two loss terms in addition to the standard reconstruction error. Authors provided two approaches for their training method based on MLE and VAE. Their framework is then experimentally evaluated on the standard dataset and demonstrate superior results; better-generated images and lower FID score) and provide ablation results.


The explanation of the Equation 2 is confusing, where the Figure-1a and the equation seem to agree but the subsequent description says that $h(.)$ should map ${N}(0, I)$ to $\hat{H}$ but Equation-2 will bring $h(z)$ closer to ${N}(0, I)$. Is there something missing here? The notation and the derivation of the MLE approach in Section 3.3 are not clear. This section would need rewriting with clearer and compact notation for better readability. The main idea is quite clear but how this training procedure achieves that is not coming out clearly in this version. Thus I feel the paper, though has good content, is not publishable in the current format.

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper presents a set of losses to train auto-encoders using a stochastic latent code (PGA). The authors relate it to VAE and propose a variational variant of their framework (VPGA), but the initial intuition is distinct from VAEs.

Results are presented on MNIST, CIFAR10 and CelebA and show both qualitative and quantitative improvements over VAEs.

The intuitions behind this framework seem sound and the authors added theoretical justifications when possible.

This paper seem to present a good idea that should be published, but is currently not clear enough. The writing of Section 3, especially section 3.1, need more work. The wording needs to be improved, maybe some notations are not needed, such as $\hat{z} = h(z)$. There should be a clear description of what each loss is aiming to achieve. Currently, this is not clear. For instance, there are several mentions that h must map N(0, 1) to $\hat{H}$, but loss (2) makes it map N(0, 1) to N(0, 1) (and I don't see other losses that would make it happen). It is likely I didn't understand it fully and a clearer section would help.
Another example of possible improvement would be calling the encoder and decoder with consistent names (for instance, Enc and Dec) could help, too. Currently, they are called encoder and decoder or generator in the text, f and g in the math, and theta and phi on figure 1.

I am ready to change my rating after the rebuttal if the authors address clarity of section 3.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper considers generative models and proposes to change the VAE approach in the following way. While VAEs assume a generative model (prior on the latent space + stochastic generator) and aim at learning its parameters so as to maximize the likelihood of the (training) data distribution, the authors propose to derive the learning objective from a different view. In my understanding, they consider the two composed mappings generator(encoder) and encoder(generator) and require the first one to have the data distribution as a fixpoint and the second one to have the latent prior as a fixpoint. Starting from this idea they derive an objective function for the case that the mappings are deterministic and then further enrich the objective by either likelihood based terms or VAE based terms. The new approach is analysed experimentally on MNIST, CIFAR and CelebA datasets. The authors report quantitative improvements in terms of Frechet inception distance (FID) as well as sharper samples (when compared to standard VAEs).


I find the main idea of the paper highly interesting and compelling. Nevertheless, I would not recommend to publish the paper in its present state . The technical part is in my view very hard to comprehend. This is partially due to the disadvantageous notation chosen by the authors. Furthermore, the derivation of the individual terms in the objective is hard to understand and the arguments given in the text are not convincing:
- The two terms in the objective related to the fixpoint of the encoder(generator) mapping seem to enforce a fixpoint that is a mixture of the latent Gaussian prior and the encoder image of the data distribution. It remains unclear to me, why this is a good choice.
- The derivation of the additional likelihood based terms and VAE based terms is in my view hard to understand.
It should be possible (and I believe, is possible) to derive a simpler objective ab initio, starting from the main idea of the authors. 

Side note: A close visual inspection of the presented generated samples seems to confirm the known doubts about the appropriateness of the FID measure for evaluation of generative models. 
review_title: Official Blind Review #2

Meta-review: The authors present a new training procedure for generative models where the target and generated distributions are first mapped to a latent space and the divergence between then is minimised in this latent space. The authors achieve state of the art results on two datasets.

All reviewers agreed that the idea was vert interesting and has a lot of potential. Unfortunately, in the initial version of the paper the main section (section 3) was not very clear with confusing notation and statements. I thank the authors for taking this feedback positively and significantly revising the writeup. However, even after revising the writeup some of the ideas are still not clear. In particular, during discussions between the AC and reviewers it was pointed out that the training procedure is still not convincing. It was not clear whether the heuristic combination of the deterministic PGA parts of the objective (3) with the likelihood/VAE based terms (9) and (12,13), was conceptually very sound. Unfortunately, most of the initial discussions with the authors revolved around clarity and once we crossed the "clarity" barrier there wasn't enough time to discuss the other technical details of the paper. As a result, even though the paper seems interesting, the initial lack of clarity went against the paper. 

In summary, based on the reviewer comments, I recommend that the paper cannot be accepted. 

Decision: Reject