Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a bidirectional bound on the EBM log-likelihood, such that it can maximize a lower bound and minimize an upper bound when solving the minimax game of EBM. The paper demonstrates the effectiveness of the proposed method on density estimation and sample generation. The major contribution of the current paper is to use the bidirectional bounds to train the EBMs.
main_review: The paper addresses an important problem of training EBMs by using the bidirectional bounds to train the EBMs. The paper is easy to follow. And the results look promising.  The major contribution of the current paper is the usage of the bidirectional bounds.

However, the connection of the paper to the existing prior works and pioneering works is pretty bad. A lot of related works are missing and a necessary comparison with some baselines methods is also missing.  Related works are pointed out below.
(1)	About deep generative EBM (from the second paragraph in the related work). There has been a long history for generative EBM for image modeling and generation. For example, [1] is the first EBM with MCMC, i.e., Gibbs sampling, for texture images. [2] is the first EBM using Langevin dynamics for texture images. [3] and [4] are EBMs using Gibbs and HMC for object images. The EBMs proposed in [1][2][3][4] are not deep model but can be still considered a two-layer CNN (because convolution or filtering is used). Of course, it is OK for the current paper to not connect them because they are not DEEP generative EBM.  However, the pioneering work that uses a modern deep neural network to parameterize the EBM and uses Langevin dynamics-based MLE to train the EBM is [5], and [6][7] have even generalized to video and voxel domains. The earliest deep generative EBM references cited in the current paper are from 2019, e.g., Du 2019 and Grathwohl 2021. It is not OK to skip the first EBM with deep CNN.   

[1] Filters, random fields and maximum entropy (FRAME): Towards a unified theory for texture modeling. IJCV, 1998.

[2] Grade: Gibbs reaction and diffusion equations. ICCV 1998.

[3] Inducing Wavelets into Random Fields via Generative Boosting. Journal of Applied and Computational Harmonic Analysis (ACHA) 2015

[4] Learning Sparse FRAME Models for Natural Image Patterns. International Journal of Computer Vision (IJCV) 2014

[5] A Theory of Generative ConvNet (ICML 2016)

[6] Synthesizing Dynamic Pattern by Spatial-Temporal Generative ConvNet (CVPR 2017)

[7] Learning Descriptor Networks for 3D Shape Synthesis and Analysis. (CVPR 2018)
     
(2)	About adversarial interpretation of EBM learning. The paper mentioned the minimax game of EBM or adversarial interpretation of EBM learning, but failed to cite the right paper or the first paper that pointing out this. [6] in 2017 is the first one to point out that. [7] and [8] also mentioned the adversarial game of EBM.  

[8] Sparse and Deep Generalizations of the FRAME Model. Annals of Mathematical Sciences and Applications 2018

(3)	About learning a generator as an approximate sampler for EBMs (from the third paragraph in the related work). The current paper missed to discuss cooperative learning or CoopNet [9][10][11][12], which is about the joint training of EBM and top-down generator. In the cooperative training, the generator serves as an effective proposal distribution to efficiently evaluate the EBM. The current paper is related to the cooperative learning idea but didn’t cite or discuss it. 

[9] Cooperative learning of energy-based model and latent variable model via MCMC teaching. AAAI 2018

[10] Cooperative learning of descriptor and generator networks. PAMI 2018

[11] Learning Energy-Based Model with Variational Auto-Encoder as Amortized Sampler. AAAI 2021

[12] Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning. (arXiv 2019 or PAMI 2021)

(4)	About baselines. Since the proposed model is close to cooperative learning. The cooperative learning [9] that trains an EBM with a generator and the EBM-VAE [11] that trains an EBM with a VAE sampler should be compared in your experiments.   


After Rebuttal:

In the rebuttal stage, authors made efforts to fill the gap between the current paper and the prior works. I updated my score from 3 to 5.  I urge the authors to substantially revise their paper by including the discussion and comparison in their reply. To be specific, (1) in your introduction, build a connection with pioneering works about joint training of EBM and generator. (2) complete the related work parts for both single EBM and joint training of EBM and generator. (3) include the quantitative comparison with the baselines, such as CoopNets, etc in experiments.   


limitations_and_societal_impact: Given the fact that the authors are not aware of the related literature and lack to compare their models to those closely related works. This might lead to a potential negative ethical impact.  I recommend a resubmission with a rejection.

To better understand the development and the advance of the EBMs, I suggest the authors review a new tutorial on EBMs in CVPR 2021 for a complete list of references, since the references I pointed out here are still not complete. This will be helpful to improve the current manuscript.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a method to train EBMs with lower and upper variational bounds. Previous approaches used only one side of the bound. This causes significant training instabilities as the training objective involves alternately maximizing and minimizing the same quantity.

The lower bound is estimated by lower-bounding the log determinant of the jacobian multiplied with itself with the smallest singular value of the jacobian. 

The upper bound is inspired by tricks used to make WGAN and related EBM training approaches stable by regularizing the gradient. 

The minimum singular value, needed for computing the lower bound, is computed using an iterative algorithm run for a small number of steps. Automatic differentiation is used cleverly to ensure full Jacobians don’t need to be instantiated.

The upper bound is loosened, and Hutchinson’s trace estimator is employed to efficiently compute. 

main_review: Describe the strengths of the work:

The authors develop a fast and stable approach to optimize EBMs like a GAN. They compare to strong baselines from both the EBM and GAN literature. They compare models on large image datasets. The authors perform an important ablation where they remove their upper bound and use the simpler gradient penalty method.

The authors develop a theoretical justification for a previous heuristic that was found empirically to significantly stabilize training. 

The authors investigate the utility of their approach on a number of tasks including toy data, where density modelling can be compared exactly, mode coverage on MNIST, and sample quality and OOD on image data. 

The authors give an explanation of some future work and open questions about their method.

Explain the limitations of this work:

The authors claim that their approach is better because it doesn’t require additional networks or rely on hard-to-tune parameters. It might be interesting to scale their approach up to larger images, for example 128x128 or even 256x256 images, but I understand this is difficult especially given the compute used as stated in the paper. 

Correctness:

The authors carefully empirically verify a number of their claims. In Figure 3, they show the quality of their entropy estimator on a tractable problem.

Clarity:
The writing is clear.

Relation to prior work: 

This work clearly describes how it builds on existing work by training models with a bidirectional bound, while previous approaches used only a bound in one direction, and a gradient regularization.

Reproducibility: Are there enough details to reproduce major results of this work

Details of the architecture, optimization, and other hyperparameters are provided in the appendix.

Additional feedback, comments, suggestions:

Please include details of data processing the appendix. For example, is the data normalized to be in [0, 1] or [-1, 1]? Was data dequantization used? What augmentations were used? What were the train-test splits used for optimizing models?

Why does the rank-order of timings in Table 5 change between the two experimental setups?

limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose to sandwich the EBM log-likelihood with newly introduced upper and lower bounds, with which a minimax objective is formed to train EBMs. 
Specifically, based on the variational lower bound from Grathwohl et al. (2021), the authors approximate the entropy term therein to form the presented lower bound, and then adding a gradient-penalty term mimicking WGAN to form the new upper bound. 
The lower and upper bounds are optimized wrt different groups of parameters, as, intuitively, both minimizing and maximizing a lower bound wrt the same group of parameters is potentially unstable (this is the main motivation for this submission).
main_review: Originality. Several techniques used to develop the upper/lower bounds are believed sound and novel; they might be useful to audiences from different domains. As for related works, important discussions associated with popular methods for training EBMs (e.g., [1,2]) are believed missing, as well as the empirical comparisons.

Quality. Minor flaws may exist in some presented techniques: (1) Eq. (8) may not hold for general GAN generators; a GAN generator is not an injective function in general, which means the matrix J_z^T J_z may not be invertible. Please elaborate on that. (2) How to guarantee that the condition in Line 86 is satisfied in practice? Please add the related analysis. (3) Is Eq (15) guaranteed to be an upper bound of the log-likelihood? 

Clarity. The submission is well-written and easy to follow.

Significance. The training of EBMs is a popular research direction that is of great interest to the ML community. However, I am not fully convinced by the presented method, because of the missing comparisons with [1,2] and the minor flaws mentioned above.

Other comments:

In Line 49-50, why "both minimizing and maximizing a lower bound is potentially unstable"? Also, what's the ''conceptual issue''? These are not clear in the current submission.

Is Eq. (10) "the new and efﬁcient estimator of the Jacobi-determinant of the EBM generator," mentioned in the Abstract? That contribution is not clearly stated. 

Since the lower bound in Eq. (11) is not tight, discussions on the gap are necessary. Also, how expensive is the evaluation of Eq. (11)?

I am concerned about the training stability. How stable is the training empirically? 

It seems Eqs. (20) and (15) show different upper bounds respectively. Which one is actually used in the experiment? Why?

In the experiments, does the proposed method perform better than the popular methods for training EBMs [1,2]?

Minors:
1. In Line 12, the abbreviation ''EMBs" is a typo.
2. In Line 53, the statement "the missing entropy term being the only difference" may not be right, since the function E_{\theta}(x) in Eq. (6) must be a 1-Lipschitz function.

[1] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11895–11907, 2019.
[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33, 2020.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes a new method for density estimation using Energy-based model. Classical variational bounds replace the negative log-likelihood by a computable lower bound, and phrase the maximum likelihood problem as a minimax game on this lower bound. A possible algorithm to solve the problem is then to alternate between minimization and maximization of this lower bound (with respect to different variables). Although this approach has shown good performance, the authors claim that the concept of minimizing such a lower bound can lead to instabilities. In particular, they claim that the WGAN objective tends to be unstable to optimize, and that while clipping the gradient in this case solves the instability issue, the origin of the instability lies in minimizing an upper bound. To circumvent this issue, the current work constructs both an upper and a lower bound to the negative log-likelihood, and alternate between minimizing the upper bound and maximizing the lower bound. The upper bound is inspired by what is done in WGAN, i.e., by adding a gradient penalty. The algorithm is tested on several benchmarks, and is shown to be rather competitive compared to existing algorithms.
main_review: The authors claim that the reason for adding gradient penalty in WGAN is to stabilize the training. However, the reason for this penalty comes from the problem formulation of learning a distribution by minimizing the Wasserstein distance. By Kantorovich duality, this leads to a minimax game with Lipschitz constraint, which is enforced by using either gradient clipping or gradient penalty. The use of gradient penalty in WGAN is hence very well motivated. In contrast, the use of gradient penalty in the current framework seems much more abstract than in WGAN.

The proposed algorithm bears strong similarities with WGAN with gradient penalty. The empirical evaluation actually shows quite similar performance, although the proposed method takes about twice more time to train (Table 5).
Hence, I don’t see the benefit of this approach compared to WGAN, both in theory and in practice.


Typos:

l.54 tends

Thm 1 there exist

limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact
review_title: null

Meta-review: The main contribution of the paper is the development of lower and upper bounds for training EBMs in a stable manner, by drawing inspiration from the WGAN objective. The methodology presented in the paper can be useful for the research community. The reviewers raised some concerns about the paper that were adequately addressed during the rebuttal phase. I therefore recommend the paper for acceptance, but strongly encourage the authors to address the following points for the camera-ready version:

+ Rephrase the points about the instability & grad clipping of WGAN, as discussed with reviewer rhEU.

+ Add the experiment details to the appendix, as suggested by reviewer y6Eh.

+ Add some discussion and references in the related work section. In particular, it'd be useful to include as much of the similarities and differences with previous methods (from the response to reviewer vzR3) as possible.

+ Add some clarification about the invertibility of the Jacobian determinant and the change-of-variables formula, as discussed with reviewer xLGS and myself. In particular, the assumptions under which Eq. (8) holds should be clearly stated in the paper - these are pointed out by the authors in their rebuttal, but some additional questions are: Does this limit the architecture of the generative NN? For example, does each hidden layer need to have a number of hidden units between $d$ and $D$, with non-decreasing sizes along the layers? Is any non-linearity valid? (e.g., ReLu's map many input values to 0, so it shouldn't be used). If any assumption is made, it should be clearly stated in the final version. In addition, the text should clarify that $p(G(z))$ in Eq. (8) is *not* the Lebesgue measure on $\mathbb{R}^D$ but it is restricted to some low-dim manifold.

+ Incorporate the experiments that were ran during the rebuttal period.

Besides the items above, I also have two technical questions that should be clarified in the paper:

+ Below Eq. (20), the paper says that $\zeta=1$ allows for the simplified bound from Eq. (15). Shouldn't it be $\zeta=M$ instead, which corresponds to $m=1$ in Eqs. (13)-(15)?

+ Theorem 2 guarantees the existence of constants ($m$, $M$, and $p\geq 1$) such that the bound in Eq. (13) holds, but unless I'm missing something, it doesn't say what values of the constants guarantee the bound in Eq. (13). But, according to the rebuttal, Eq. (15) is a valid bound for any $p\geq 1$ (as long as the "expression$\geq 1$" assumption is met). So, even though it was obtained from Eq. (13), Eq. (15) makes an explicit statement about what values of the constants constitute a valid bound. How is such discrepancy possible? Is the "expression$\geq 1$" assumption the explanation for that? Does Theorem 2 actually say something about the values of the constants that guarantee the correctness of the bound?

Decision: Accept (Poster)