Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes the Hat EBM, which is a method to incorporate a sampler (such as a pre-trained generator) into an energy-based model (EBM). The basic idea is to define a joint probability distribution over both the latent space of the pre-trained generator and a “residual space” – the Hat EBM takes as input a synthesized image from the generator combined with a “residual image,” and uses the output of the Hat EBM in an MCMC sampler. The distribution over these residual images is also learned jointly via MCMC. This approach allows for unconditional image generation, image refinement from a pretrained generator such as a GAN, defining implicit samplers for deterministic decoders, and out-of-distribution (OOD) detection.  Hat EBM achieves strong empirical results on complex datasets such as ImageNet 128x128.

strengths_and_weaknesses: Strengths:
- The paper’s empirical results are strong across a wide range of tasks.
- The presentation is fairly clear and easy to follow. They also provided good background material on concepts that were important for understanding the paper, such as cooperative learning.

Weaknesses:
- The proposed approach hinges on some theoretical assumptions and appears to be quite compute-intensive. However, the authors included a thorough description of the computational limitations of their method in Appendix B and D. Appendix E also included a description of the theoretical assumptions required for their method (e.g. correct model specification and convergence of MCMC), as well as precautions necessary for incorporating the residual vector Y during training in Appendix H. I appreciated the care they took here.


questions: Questions:
- Clarification: what does it mean on l.107 when the text says that the “majority of the appearance of G(Z) + Y” comes from the generator network? Is that just saying that the generator is responsible for most of the content/variation in the image?

Very minor miscellaneous things:
- “We propose [to] initialize Langevin sampling in l. 167

limitations: The authors included a description of the potential negative societal impacts in Appendix B, though it could have been a bit more thorough.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes “Hat EBM”, an energy-based generative model that incorporates a generator network that maps points from a latent space to the data space into the EBM. The resulting EBM energy is a function of the image generated by the generator and an image residual. The authors propose methods to: train the EBM on top of a fixed generator and train both the generator and the EBM jointly. The authors show the applicability of the proposed method to sample refinement, synthesis and out-of-distribution detection tasks.
strengths_and_weaknesses: [Strengths]

- The paper presents an interesting and novel formulation of an energy-based model.
- The proposed method is technically sound.
- Experiments have been presented on three different tasks of refinement, generation, and OOD detection. The results show a significant improvement over the baseline EBM models for the image synthesis task.

[Weaknesses]

- While the the key idea of the paper is presented clearly, parts of the paper are difficult to parse. For example, the paragraph beginning at line 180 lacks clarity. Following are some other comments and suggestions:
    - It appears from the text that training algorithm is not straightforward. Thus, the main text would benefit significantly from a separate algorithm/pseudocode of the proposed method.
    - Section 2: Instead of saying “This work [D]…” consider using “XYZ et al. [D]”.
    - Line 110: What is \sigma?
    - Line 112: log determinant of **the Jacobian** of G.
    - Line 139: \phi has not been defined yet.
    - Line 158: What does the subscript t indicate in X_t? Is this a typo?
    - Line 167: propose **to** initialize…
- From the results in Table 1, it appears that sample refinement is not a strong suit of hat EBM. The FID scores are barely an improvement. This is particularly important because the baseline refinement methods (e.g., DDLS) do not require training of another network unlike hat EBM. Furthermore, the current manuscript doesn’t discuss/compare against other sample refinement baselines, e.g., DOT [1] and DGflow [2]. In particular, DGflow also studies a method to refine samples from all kinds of probabilistic generators.
- A discussion of and comparison against the very relevant GEBM [3] is missing. GEBM also uses a base generator and an energy function on top of the generator to draw/refine samples. Furthermore, EBM baselines considered in Table 2 have not been properly discussed in the paper. This discussion is important to correctly contextualize this work.
- While the synthesis results are impressive, the overall significance is marred by a lack of discussion. Why does hat EBM perform so well against these baseline EBMs? Is the comparison fair (e.g., in terms of networks)?

[1] Tanaka, Akinori. "Discriminator optimal transport." *Advances in Neural Information Processing Systems*
 32 (2019).

[2] Ansari, Abdul Fatir, Ming Liang Ang, and Harold Soh. "Refining deep generative models via discriminator gradient flow." arXiv preprint arXiv:2012.00780 (2020).

[3] Arbel, Michael, Liang Zhou, and Arthur Gretton. "Generalized energy based models." arXiv preprint arXiv:2003.05033 (2020).
questions: - What was the original motivation for the image residual formulation of the EBM? It appears that the original motivation was the problem of refinement; however, the proposed method doesn’t refine samples better than the baseline (which does not require additional training).
- Line 232: How does the incorporation of the residual Y help prevent the assignment of “increasingly high energy to generator samples”?
- Do the authors have any comments on why hat EBM performs so well against baseline EBMs on image synthesis?
limitations: The authors have not discussed limitations of their work. I suggest that they add a few lines to discuss the shortcomings of the proposed work and potential future directions to fix these limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper shows how one can use existing generator networks within an energy based model. The authors introduce a *hat network* that takes as input the output of the generator plus a residual image, and returns a scalar representing the energy value. The parameters of the hat network (and potentially of the generator) can be trained with Maximum Likelihood learning where negative samples can be obtained with MCMC steps (Gibbs sampler with Langevin updates) 

 

The authors explore different learning scenarios and show how this framework can be used to 1) Refine samples of an existing pretrained generator; 2) Generate images by learning the whole EBM model (generator and hat network) from scratch; 3) Allowing sampling over deterministic generators; 4) Use the EBM for anomaly detection. 
strengths_and_weaknesses: STRENGTHS 

1. There has been an increased focus lately on using EBM for generative learning. This work presents a novel idea that I believe can inspire further research in the neurips community. 

2. This is a quite general formulation that can be used for any type of deep generative model 

3. Learning does not require posterior inference over the latent states, making the model simpler and more scalable.  

4. The authors show empirically that the hat EMB outperforms competing methods in a number of different tasks. It also generates compelling unconditional Imagenet samples.  

5. The paper is well written and clear 

 

WEAKNESSES 

The authors should provide more discussions/experimental details, particularly to make this framework more accessible to people not familiar with MCMC (and its pitfalls): 

1. In your experiment you chose 50/100 MCMC steps. What is a good strategy to choose this hyperparameter and ensure that the chains have converged well enough for learning to be efficient? 

2. In the paper you do not mention the accept-reject step needed when doing the Langevin updates to define a valid Metropolis-Hastings sampler. Do you monitor the acceptance rate during training? 

3. What would it take to use Hamiltonian Monte Carlo instead of the Langevin updates to improve the convergence of the chains? 

4. How robust is the algorithm to the choice of the size of bank memory? 

 

 
questions: See questions in the weaknesses section above.
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes a method for using a generator network as the foundation for an Energy-Based Model (EBM). In essence, the paper considers that the output of the generator network is superimposed with a latent variable defined in the image space; this produces the final image. The generator network is presented with a low-dimensional latent variable input. The low dimensionality of this input space renders the modeling problem less multimodal, thus more tractable. 
The described formulation of the model renders it amenable to MCMC-type sampling (Langevin gradients) for performing training, in a contrastive divergence type of maximum likelihood. 
strengths_and_weaknesses: Strengths: The model is solid and the training algorithm is straightforward and correct. The authors provide a variety of experiments that deal with different application types. In all cases, they use standard benchmark data to experiment with. They compare to related EBM-type approaches, as well as some state of the art alternatives based on the GAN principle.

Weaknesses: The authors do not explain how the method compares to the alternatives in terms of computational costs, both for training and at test time. In addition, the results are not fully convincing of the usefulness of the method. Specifically, while the method improves over previous EBMs, it seems of lackluster performance compared to GAN-type methods. How do the authors explain this shortcoming? Why would someone prefer their method over the GAN-type alternatives?
questions: How the method compares to the alternatives in terms of computational costs, both for training and at test time?
How do the authors explain the better performance of GAN-type alternatives?
What is the main merit of the proposed approach that adds important value to the state of the art?
limitations: n/a
review_title: null

Meta-review: metareview: **Summary**: This paper proposes an approach for using existing generator networks in an energy based model. The authors introduce a "hat" network that accepts output of the generator and a residual image, and returns a scalar representing the energy value. The parameters of the hat network (and potentially of the generator) can be trained using persistent contrastive divergence, where negative samples are generated by using a Gibbs sampler with Langevin updates to generate both the latent variable for the generator and the residual. The authors demonstrate how this approach can be used to refine samples from an existing generator, generate images by jointly training the generator and the energy function, allow sampling over deterministic generators, and use the EBM for anomaly detection. 

**Strengths**: Reviewers [9epW] and [45fe] appreciate the interesting and novel formulation of an energy-based model, which can inspire further work on EBMs for generative learning. Moreover, reviewers agree [9epW,45fe,oJLY] that the proposes formulation is very general and is technically sound. [N2ii] and [45fe] note the fairly clear presentation, with good background on topics such as cooperative learning. All reviewers [9epW, N2ii, 45fe, oJLY] agree that experiments on refinement, generation and OOD detection show improvements relative to baselines. 

**Weaknesses:** While reviewers were overall appreciative of this submission, they also note weaknesses with respect to clarity, discussion, and experimental results. One issue that reviewers feel is overlooked is the additional computation cost associated with the approach [9epW,N2ii]. Reviewer [9epW] states that while main idea is presented clearly, parts of the paper are difficult to parse (multiple examples given). Reviewer [45fe] similarly finds discussion lacking and that experimental details are missing (4 examples given).

Reviewer [9epW] notes that experimental results are also not strong across the board. Sample refine does not appear to a strong suit of hat EBM. FID scores are barely an improvement. This is particularly relevant becase refinement baselines do not require training, whereas paper does not compare to refinement baselines like DOT and DGflow. Moreover, there are some problems with baselines. Discussion and comparison against GEBM is missing, which is a model that also combines a base generator and an energy function to draw/refine samples. EBM baselines in Table 2 are not properly discussed in the paper. While synthesis results are impressive, discussion of why the proposed approach works well are missing. 

**Author Reviewer Discussion**: Authors provided clarifications to all reviewers, and amended the paper to reference related work on GEBM and DGflow. The authors indicate that they will also attempt to compare against these baselines. 

**Reviewer AC Discussion**: The AC indicated that this appears to be highly borderline paper with a significant variance in reviewer scores. Unfortunately, only reviewer [9epW] engaged with the AC during the discussion phase. This reviewer indicated that while the proposed is interesting, they remain of the opinion that the related work section and baselines need work, and that the paper would benefit from a clearer motivation for the proposed approach. On balance they remain of the opinion that this paper is on the borderline. 

**Overall Recommendation:** Based on the numerical average of the scores, this submission would be just about above the threshold for acceptance. At the same time, the AC is concerned with the comments about clarity and in particular with the comments about the lack of discussion of related work. In this context, the sparse discussion and the fact that no reviewers have championed the paper is somewhat worrying. On balance, the AC recommendation is to consider this paper as borderline. It is narrowly above the bar for acceptance, but may have to be cut in favor of other papers.
confidence: Less certain
Decision: Accept