Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new training objective for generative models that combines the objectives of VAEs and GANs. The objective is equivalent to minimizing the Jeffreys divergence (a type of f-divergence) between the true probability of the data and its probability under the model.  Furthermore, the objective comes with a knob to tradeoff the relative importance of each of the two terms.  In addition, the authors develop a implicit likelihood formulation which they claim and show empirically to outperform typical explicit formulations typically used in VAEs. 

Overall, it is an interesting paper that reuses a few good ideas to develop a novel training objective. The results show that using an implicit likelihood helps (Figure 2) and that it does relatively better than either GAN or VAE approaches. I have detailed comments below about the organization of the paper, some of the experimental claims as well as a few other works which may be good to cite. 


- Paper organization: I would suggest moving the related work to after the background.

- GANs and VAEs are not models per se but rather training frameworks for generative models.

- While VAEs and GANs can work on many types of data (at the very least continuous), your model seems to be developed for images. Could you make it clear what changes would be needed to apply it to non-image data?

- There are many minor grammatical errors throughout the text.

- It would be useful to provide the full algorithm somewhere (e.g., using an algorithm "box")

- Possible related work. It may be worth citing these two paper:
  - f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization, NIPS'16
  - Deep Generative Learning via Variational Gradient Flow, ICML'19
 
- It would be useful to mention early that for IS higher is better and LPIPS lower is better.

- Even though Figures 2 and 3 (to a certain extent) seem to show that results are somewhat robust to the exact value \lambda how would you propose to set it in practice?

- Figure 3 Left (CIFAR 10), it's not absolutely clear to me that alpha-GAN and perhaps AGE isn't at least as good as your approach. The meaning of the units of the axes is a bit unclear. Do you have a particular reason to prefer your method over these in this case?

  Related: in Table 1, why are there no bolded results for CIFAR + Reconstruction?

- In Figures 5 and 7 the reconstruction of IJAE sometimes seems to be pretty far from the original image (i.e., it's not that it's blurry as for VAEs, it's that the model seems to be reconstructing a completely different image). How do you explain these results?
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper introduces a model named lambda-IJAE, which combines the VAE and GAN training schemes to train a generative model achieving competitive performance. The combination of VAE and GAN is justified by its theoretical interpretation as a an optimization of the lambda-Jeffreys divergence between the real data distribution and the generation distribution. This work also introduces a reformulation of the reconstruction term of the VAE loss, allowing it to be estimated implicitly using an adversarial mechanism. Finally, the latent space of the VAE is also modelled implicitly using an adversarial mechanism, following (Mescheder et al. 2017).

I am ambivalent about this paper. The proposed implicit likelihood mechanism is very interesting, but the paper contains several weaknesses that together make me unwilling to accept it.

First of all, the paper presents itself as centered on the notion of optimizing the lambda-Jeyffreys distribution, while the main contribution is actually clearly the formulation of the implicit likelihood. The use of a weighted sum of the forward & KL divergences to train a generative model is hardly new, and has already been presented a few times (Larsen et al. 2015, Dosovitskiy & Brox 2016).

In this context the paper does not present the impact of its main contribution alone. How would behave a VAE trained solely with this implicit likelihood, but a regular Gaussian latent space and without the GAN loss? This ought to be part of the ablation study in my opinion.

Secondly, the paper discusses the issue of VAE generating unrealistic samples. This is indeed a very real issue of the VAE linked to it being trained by maximum-likelihood. However illustrating it by "blurry images" (like is done several times in the paper) is a common misconception, as while this is a very classical issue with VAEs, it is mostly unrelated to the MLE estimation.

It is rather a simple consequence of the fact that using an unweighted squared error loss to model the reconstruction of the VAE is almost always a poor model. It is equivalent to modelling the observation with a Gaussian noise of variance 1/2, which is a huge noise when considering data normalized in [0;1] or [-1;1] like is traditional to do with images. Reducing this variance to a more sensible value (like a std of 0.1 for example) or allowing the model to learn it reveals the real failure mode of the VAE generating unrealistic images, which can hardly be described as "blurry".

Similarly, the ablation study evaluates the use of L1 or L2 noise instead of the cyclic shift likelihood, but does not say what variance has been used for these, which would (as explained above) be an important parameter to take into account. If a variance of 1 was used, then the results of figure 2 are unsurprising and not insightful, as the discriminator would have merely learned to differentiate between images containing a visible Gaussian noise from images that do not.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes to replace the KL-divergence in VAE training with the lambda-jeffreys divergence of which the symmetric KL-divergence is a special case. The paper proposes a pure implicit likelihood approach that uses three discriminator models to estimate the KL-divergences. Experiments are conducted on CIFAR-10 and TinyImageNet and several scores are reported to show that the proposed method performs as good if not better than current approaches.
---------
I think the paper tries to achieve too much in too little space and foregoes scientific exactness for the sake of claiming SOTA. Since there is a difference between claiming SOTA on a task and validating a new method, the small amount of space makes it difficult to substantiate both claims at the same time. In the rest of the review i will try to substantiate the claim:

1. The paper claims on page 2: "These models do not have a sound theoretical justification about what distance [...] they optimize". While the paper tries to substantiate its claims by showing theoretically that it does the right thing using the optimal discriminator, it leaves the question open what happens with any other discriminator. The theory does not justify non-optimal solutions. It is argued on page 6 that non-optimality of the discriminator serves as some form of regularization, but  this requires some justification.
Moreover, the paper uses LPIPS to measure reconstruction quality - but this measure is a deep neural network. So if those measures are good enough to compare solutions with and the theoretical justification of the proposed method is shaky in practice - why not use LPIPS for training?

2. The paper proposes the discriminator in order to allow for an implicit likelihood. However, the r-function used in the experiments does not fulfill the property of a well defined likelihood, and Theorem 1 does not hold, since technically the KL-divergence is infinity. If we ignore this by adding a small amount of Gaussian noise around the sampled cyclical shifts - like the r' used in the experiments, we can easily write down the explicit likelihood function since:

r(y|x)=\sum_i w_i N(y|Shift_i(x), \sigma)

where Shift_i is the i-th shift in the set described in the paper and w_i its probability  p(y|q). So the explicit solution of theorem 1 can be written down and another ablation study would be training the method with the explicit formulation for this KL-term(i.e. only training two discriminator models). If the results are not equivalent, this implies that the discriminator does not reach the optimum. The implications of that should be discussed regarding 1. 

3.  Existing ablation studies are a bit of a straw-man: the paper compares changing r(y|x) by standard Gaussian or Laplace. However, we know that a large variance does not make any sense and almost all papers use tiny variances (e.g. in beta-VAE the beta-values tend to be very small, which is equivalent to small variances here). 


---------------------------
Smaller things
- Are the experimental results all with the same architecture for encoder/generator for all results you compared to? if not, the effect of that should also be tested.

- my personal biased view on the generated images is: it looks worse than alpha-GAN. Every reconstructed image has a grey tone and the generated images also offer a strong grey palette. The details don't look better as well.

- typo inroduce->introduce

review_title: Official Blind Review #3

Meta-review: The paper received Weak Reject scores from all three reviewers. The AC has read the reviews and lengthy discussions and examined the paper. AC feels that there is a consensus that the paper does not quite meet the acceptance threshold and thus cannot be accepted. Hopefully the authors can use the feedback to improve their paper and resubmit to another venue.
Decision: Reject