Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Originality:
The paper presents an interesting generative model termed generalized energy-based model (GEBM), which is essentially an exponential tilting of a plain generator model P_G by an energy based model exp(-E).
The appearance of such a model in generative modeling looks to be new. But the structure of the model is not surprisingly new.

Significance:

The pros:
1. The purpose of introducing such a model, as stated by the authors, is to make use of the fact that target distribution may concentrate in a low dimensional manifold in the target domain, which can be captured by a low-dimensional generator model P_G. Once tilted by exp(-E), the overall model can generate sharper samples than the plain energy based model.
2. The paper also presents tractable methods to train and sample from the proposed GEBM.
3. The authors show that GEBM performs better than GANs on certain image generation experiments.

The cons:

While it may be easier to understand why GEBM can be better than EBM on certain generative modeling tasks, the authors did not provide much explanation or justification on why GEBM can outperform GANs. As mentioned by the authors, the energy based models and functional generators are quite different generative models. By combining them, how to guarante that their pros rather than the cons will be strengthened? In what kind of tasks GEBM can perform better than GANs? Hopefully the authors can give more theoretical comparisons or intuitions to address these questions.


Quality:

Pros:
The overall quality of the paper is good. Nearly all results regarding the properties of GEBM come with theoretical justifications.
The proposed alternative training method and sampling methods look reasonable. 

Cons:
It would be better if there is some analytical comparison between GEBM and GAN.
Also, in the experiment section, the authors should reveal more detail on how the energy function (class) is chosen.
Another concern is that, for Proposition 1 to hold, does it require the generator G to be invertible?






review_title: Good paper, interesting but not suprising model, wish to see more comparisons with GAN

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: In this work, a generalized energy-based model (GEBM) is proposed. During the generation, the base distribution and the energy cooperate to combine the strengths of both the energy-based model and the implicit generative model.
 
+ves: 
 
1. This paper has proposed a framework so that the energy function can be used to refine the probability mass on the learned base distribution. The framework is trained by alternating between learning the energy and the base. Empirically, the framework outperforms GAN with the same complexity. 
 
2. KL Approximate Lower-bound Estimate (KALE) is used for energy training. There is a lot of detail on this derivation. 

Concerns:
1. In this work, a new framework is proposed for training. The whole process seems complicated. Is there is a simple way to refine the probability mass on the learned base distribution? How about  considering exp(-E(x))G(x). Here G(x) is learned base distribution. A simple way to refine the probability mass is by sampling several generated output near the point and weight the outputs with exp(-E(x))G(x)? This seems easier. Could you explain the more possible benefits of your method?

2. During the generation process, two algorithms are proposed: ULA and KLA.  It seems a large variance. It is unclear which one is better.



Questions during the rebuttal period: 
Please address and clarify the cons above:
Missed related work.
 
For the training of energy based models, there are several related work [2][3], etc. 
In the work [1], the residual energy is used to better generation. MCMC is also used for generations. 

[1] Residual Energy-Based Models for Text Generation. ICLR 2020

[2] Structured Prediction Energy Networks, ICML 2016

[3] Learning Approximate Inference Networks for Structured Prediction. ICLR 2018
review_title: reviewer 4

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a framework called GEBM that combine an implicit generator and an EBM to define a probabilistic model on low dimensional manifold. Specifically, the implicit generator defines the base distribution, and the EBM refines the base. Finally, this method is equivalent to define an EBM on the latent space of the implicit generator together with a mapping from the latent space to the data space. The authors propose to use the KALE to train the generator, and provide a theoretical guarantee about the validness of the KALE.

I personally enjoy reading this paper. It provides a good method to solve the problem where the EBM cannot model the distribution whose support is a low-dimensional  manifold in the data space. However, my main concern is about the contribution of this paper, which can be summarized as follows:
1. The novelty. This paper mainly has two contributions: the generalized ebm framework and the KALE objective function. For the first contribution, the difference between the related work is discussed. But what’s the advantages? When the base distribution is an implicit model, GEBM is still a GAN-like model: cannot estimate density. When the base distribution is a flow model, GEBM is still an EBM: cannot provide tractable partition function.
2. The validness of the proposed method. If the base distribution is an implicit model, which means that the support is only a subset of the data space. However, most implicit models, especially GANs, suffer from the mode collapse problem. In this case, the distribution defined by GEBM cannot recover the data distribution. Is there any explanation about the mode collapse problem in GEBM?


Some minor concerns about this paper are provided as follows:
1. The experimental results are chaotic. First, the neural architectures are not introduced, which is the key factor in the performance of GAN-like models. Second, Figure 1 needs more explanation. What is the architectures of the models? Which GAN variant is used? Is EBM trained by MLE using MCMC or score matching?
2. What’s the main purpose of using the flow model as the density estimator? As mentioned in Proposition 2, GEBM is equivalent an EBM. Besides, in the experiments, the energy model is also an NVP. More explanation should be added to improve the clarity. Further, how NVP is trained with CD?
review_title: Lack of Novelty

Meta-review: This paper introduces a generative model termed generalized energy-based model (GEBM).

The goal is modelling complex distributions supported on low-dimensional manifolds, while offering more flexibility in refining the distribution of mass on those manifolds. The key idea is presented as parametrizing the base measure (called a generator in the paper) and the density with respect to this base measure separately. Figure 1 of the paper sketches the idea on a very clear toy example.

The pros:
* Flexibility: Decomposing the full problem as learning the support and learning the density on this support 
* Theoretical justification
* Introducing the KALE objective
* Comparative empirical results with GANs show the additional benefits. Empirically, the framework outperforms GAN with the same complexity.
* Clear written paper

The lack of a comparison with GANs has been raised as a concern.  The authors have satisfactorily answered key questions and
others raised during rebuttal and added several new references. They have also improved the narrative and included an additional experiment to contrast GEBM and GANs in response to AnonReviewer2, also provided more detail 
on how the energy function (class) is chosen. 

Decision: Accept (Poster)