Title: Learning Energy-based Model via Dual-MCMC Teaching
Venue: NeurIPS 2023
Accepted: yes
Keywords: Energy-based model

MCMC

Joint-training

Generator model
Abstract: This paper studies the fundamental learning problem of the energy-based model (EBM). Learning the EBM can be achieved using the maximum likelihood estimation (MLE), which typically involves the Markov Chain Monte Carlo (MCMC) sampling, such as the Langevin dynamics. However, the noise-initialized Langevin dynamics can be challenging in practice and hard to mix. This motivates the exploration of joint training with the generator model where the generator model serves as a complementary model to bypass MCMC sampling. However, such a method can be less accurate than the MCMC and result in biased EBM learning. While the generator can also serve as an initializer model for better MCMC sampling, its learning can be biased since it only matches the EBM and has no access to empirical training examples. Such biased generator learning may limit the potential of learning the EBM. To address this issue, we present a joint learning framework that interweaves the maximum likelihood learning algorithm for both the EBM and the complementary generator model. In particular, the generator model is learned by MLE to match both the EBM and the empirical data distribution, making it a more informative initializer for MCMC sampling of EBM. Learning generator with observed examples typically requires inference of the generator posterior. To ensure accurate and efficient inference, we adopt the MCMC posterior sampling and introduce a complementary inference model to initialize such latent MCMC sampling. We show that three separate models can be seamlessly integrated into our joint framework through two (dual-) MCMC teaching, enabling effective and efficient EBM learning.

Learning Energy-based Model via Dual-MCMC Teaching

Abstract

This paper studies the fundamental learning problem of the energy-based model (EBM). Learning the EBM can be achieved using the maximum likelihood estimation (MLE), which typically involves the Markov Chain Monte Carlo (MCMC) sampling, such as the Langevin dynamics. However, the noise-initialized Langevin dynamics can be challenging in practice and hard to mix. This motivates the exploration of joint training with the generator model where the generator model serves as a complementary model to bypass MCMC sampling. However, such a method can be less accurate than the MCMC and result in biased EBM learning. While the generator can also serve as an initializer model for better MCMC sampling, its learning can be biased since it only matches the EBM and has no access to empirical training examples. Such biased generator learning may limit the potential of learning the EBM. To address this issue, we present a joint learning framework that interweaves the maximum likelihood learning algorithm for both the EBM and the complementary generator model. In particular, the generator model is learned by MLE to match both the EBM and the empirical data distribution, making it a more informative initializer for MCMC sampling of EBM. Learning generator with observed examples typically requires inference of the generator posterior. To ensure accurate and efficient inference, we adopt the MCMC posterior sampling and introduce a complementary inference model to initialize such latent MCMC sampling. We show that three separate models can be seamlessly integrated into our joint framework through two (dual-) MCMC teaching, enabling effective and efficient EBM learning.

1 Introduction

Deep generative models have made significant progress in learning complex data distributions [35, 18, 37, 34, 19, 22, 5] and have found successful applications in a wide range of real-world scenarios [26, 6, 11, 15]. Among these, the energy-based model (EBM) [5, 6, 29, 8, 39, 2, 3] has gained particular interest as a flexible and expressive generative model with an energy function parameterized by a neural network. Learning the EBM can be accomplished via the maximum likelihood estimation (MLE), which involves the Markov Chain Monte Carlo (MCMC) sampling in high-dimensional data space. However, such MCMC sampling has shown to be challenging [30, 7, 33, 13], as it may take a long time to mix between different local modes with a non-informative noise initialization [43, 15].

To address this challenge, recent advances have explored employing complementary models to substitute for MCMC sampling [15, 16, 12, 17, 24]. One notable example is the generator model. The generator model incorporates a top-down generation network that is capable of mapping lowdimensional latent space to high-dimensional data space and admits efficient sample generation. The generator is learned to match the EBM so that MCMC sampling can be replaced by generator ancestral

sampling. However, such direct generator sampling has shown to be less accurate and suboptimal [43]. To alleviate this issue, [42, 43] introduced cooperative learning, where samples generated by the generator model serve as initial points, and then followed by a finite-step MCMC revision process. While this gradient-based MCMC revision process can be more accurate, the generator model learned relies solely on the EBM and has no access to the observed empirical observations. As a result, this learning scheme may render biased generator learning, which in turn caps the potential of learning a strong EBM. An effective joint learning scheme for the EBM and its complementary generator model is needed, yet still in its infancy.

In this paper, we present a novel learning scheme that can seamlessly integrate the EBM and complementary models into a joint probabilistic framework. Specifically, both the EBM and complementary generator model are learned to match the empirical data distribution, while the generator model, at the same time, is also learned to match the EBM. Learning the generator model with empirical training examples can be achieved with MLE, which typically requires access to the generator posterior as an inference process. To ensure an effective and efficient inference, we employ the MCMC posterior sampling with the complementary inference model learned as an initializer. Together with MCMC sampling of EBM being initialized by the generator model, such two MCMC samplings can be further used as two MCMC revision processes that teach the generator and inference model to absorb MCMC-revised samples, thus we term our framework dual-MCMC teaching . We show that our joint framework is capable of teaching the complementary models and thus learning a strong EBM.

Our contributions can be summarized as follows:

We introduce a novel method that integrates the EBM and its complementary models into a joint learning scheme.

We propose the use of dual-MCMC teaching for generator and inference models to facilitate efficient yet accurate sampling and inference, which in turn leads to effective EBM learning.

We conduct extensive experiments to demonstrate the superior performance of our EBM.

2 Preliminary

2.1 Energy-based Model

Let x ∈ R D be the high-dimensional observed examples. The energy-based model (EBM) [41, 6, 5, 29] represents data uncertainty with an undirected probability density defined as

where -f α ( x ) is the energy function parameterized with parameters α , and Z( α ) ( = ∫ x exp[ f α ( x )] d x ) is the partition function or normalizing constant.

Maximum likelihood estimation. The maximum likelihood estimation (MLE) is known for being an asymptotically optimal estimator and can be used for training the EBM. In particular, with observed examples, { x ( i ) , i = 1 , 2 , ..., n } , the MLE learning of EBM maximizes the log-likelihood L π ( α ) = 1 n ∑ n i =1 log π α ( x ( i ) ) . If the sample size n is large enough, the maximum likelihood estimator minimizes the KL( p d ( x ) ∥ π α ( x )) which is the Kullback-Leibler (KL) divergence between the empirical data distribution p d ( x ) and the EBM distribution π α ( x ) . The gradient ∂ ∂α L π ( α ) is computed as

Given such a gradient, the EBM can be learned via stochastic gradient ascent.

Sampling from EBM. The Eqn.2 requires sampling from the EBM π α ( x ) , which can be achieved via Markov Chain Monte Carlo (MCMC) sampling, such as the Langevin dynamics [28]. Specifically, to sample from the EBM, the Langevin dynamics iteratively updates as

where τ indexes the time step, s is the step size and U τ ∼ N (0 , I D ) .

As s → 0 , and τ → ∞ , the distribution of x τ will converge to the target distribution π α ( x τ ) regardless of the initial distribution of x 0 [28]. The existing practice [29, 30, 5] adopts non-informative distribution for x 0 , such as unit Gaussian or uniform, to initialize the Langevin transition, but it can be extremely inefficient and ineffective as they usually take a long time to converge between different modes and are also non-stable in practice [43]. The ability to generate efficient and effective samples from the model distribution becomes the key step toward training successful EBMs. In this paper, we study the complementary model, i.e., generator model , as an informative initializer for effective yet efficient MCMC exploration toward better EBM training.

2.2 Generator Model

Various works [15, 12, 17, 24] have explored the use of the generator as an amortized sampler to replace the costly noise-initialized MCMC sampling for EBM training. Such a learning approach relies on samples directly drawn from complementary models, which can be less accurate than iterative MCMC sampling as it lacks a fine-grained exploration of the energy landscape. [42, 43] propose the cooperative scheme in which MCMC sampling of EBM is initialized by the generated samples from the generator model. However, the generator model has no access to the observed training examples, and such a biased generator learning makes the EBM sampling ineffective and renders limited model training.

In this paper, the generator model is learned by MLE to match both the empirical data distribution and the EBM distribution. Such training ensures a stronger generator model which will further facilitate a more effective EBM sampling and learning. We present the background of the generator model and its MLE learning algorithm below, which shall serve as the foundation of our proposed method.

Generator model. Let z ∈ R d ( d < D ) be the low-dimensional latent variables. The generator model [14, 10, 21] seeks to explain the observation signal x by a latent vector z and can be specified as

where p ( z ) is a known prior distribution such as unit Gaussian, e.g., p ( z ) ∼ N (0 , I d ) , and p θ ( x | z ) ∼ N ( g θ ( z ) , σ 2 I D ) is the generation model that is specified by neural network g θ ( . ) that maps from latent space to data space.

Maximum likelihood estimation. The MLE learning of the generator model computes log-likelihood over the observed examples as L p ( θ ) = 1 n ∑ n i =1 log p θ ( x ( i ) ) , where p θ ( x )(= ∫ z p θ ( x , z ) d z ) is the marginal distribution. If the sample size n is large, it is equivalent to minimizing the KL divergence KL( p d ( x ) ∥ p θ ( x )) . The gradient of the likelihood L p ( θ ) can be obtained via:

With such a gradient, the generator model can be learned via stochastic gradient ascent.

Sampling from generator posterior. The Eqn.5 requires the sampling from the generator posterior p θ ( z | x ) . One can use MCMC sampling such as Langevin dynamics [28] that iterates

where U τ ∼ N (0 , I d ) . Such a Langevin process is an explaining-away inference where the latent factors compete with each other to explain each training example. As s → 0 , and τ → ∞ , the distribution of z τ will converge to the posterior p θ ( z | x ) regardless of the initial distribution of z 0 [28]. However, noise-initialized Langevin [14, 31] can be ineffective in traversing the latent space and hard to mix. In this paper, we introduce a complementary model, i.e., inference model , as an informative initializer for effective yet efficient latent space MCMC exploration for better generator and EBM training.

2.3 Inference model

The inference model q ϕ ( z | x ) is adopted in VAEs [21, 32] as an amortized sampler to bypass the costly noise-initialized latent space MCMC sampling. In VAEs, q ϕ ( z | x ) is Gaussian parameterized, i.e., N ( µ ϕ ( x ) , V ϕ ( x )) , where µ ϕ ( x ) is the mean d -dimensional mean vector and V ϕ ( x ) is the d -dimensional diagonal covariance matrix. Such a Gaussian parameterized inference model is a tractable approximation to the true generator posterior p θ ( z | x ) , but can be limited to approximate the multimodal posterior. We adopt the same Gaussian parametrization of q ϕ ( z | x ) in this paper, but unlike the VAEs, our inference model serves as an initializer network that jump-starts the latent MCMC sampling from an informative initialization. The marginal distribution obtained after Langevin can be more general and multi-modal than the Gaussian distribution.

3 Methodology

To effectively learn the EBM, we propose a joint learning framework that interweaves maximum likelihood learning algorithms for both the EBM and its complementary models. For the MLE learning of the EBM, MCMC sampling can be initialized through the complementary generator model, while for the MLE learning of the generator model, the latent MCMC sampling can be initialized by the complementary inference model. Three models are seamlessly integrated into our joint framework and are learned through dual-MCMC teaching .

3.1 Dual-MCMC Sampling

The EBM π α ( x ) , generator p θ ( x ) and the inference model q ϕ ( z | x ) defined in Sec.2 naturally specify the three densities on joint space ( x , z ) , i.e.,

The generator density P θ ( x , z ) specifies the joint density through ancestral generator sampling from prior latent vectors. Both the joint EBM density Π α,ϕ ( x , z ) and data density Q ϕ ( x , z ) include inference model q ϕ ( z | x ) to bridge the marginal distribution to joint ( x , z ) space. However, q ϕ ( z | x ) is modeled and learned from two different perspectives, one on empirical observed data distribution p d ( x ) for real data inference , and one on EBM density π α ( x ) for generated sample inference .

The joint learning schemes [15, 12, 17, 24] based on these joint distributions can be limited, because 1) the generator samples from P θ ( x , z ) is conditionally Gaussian distributed (Sec.2.2) which can be ineffective to capture the high-dimensional multi-modal empirical data distribution, and 2) the inference model q ϕ ( z | x ) on observed training examples is assumed to be conditionally Gaussian distributed (Sec.2.3) that is incapable of explaining-away inference [14].

To address the above limitations, we introduce two joint distributions that incorporate MCMC sampling as revision processes,

where T z θ ( · ) denotes the Markov transition kernel of finite step Langevin dynamics that samples z from p θ ( z | x ) (see Eqn.6), and T x α ( · ) denotes the transition kernel that samples x from π α ( x ) as shown in Eqn.3. Therefore, T x α p θ ( x )(= ∫ x ′ ∫ z T x α ( x ′ ) p θ ( x ′ , z ) d z d x ′ ) indicates the marginal distribution of x obtained by running MCMC transition T x α ( · ) that is initialized from p θ ( x ) . Similarly, T z θ q ϕ ( z | x ) represents the marginal distribution of z obtained by running T z θ ( · ) that is initialized from q ϕ ( z | x ) given observation x (i.e., T z θ q ϕ ( z | x ) = ∫ z ′ T z θ ( z ′ ) q ϕ ( z ′ | x ) d z ′ ).

The ˜ P θ,α ( x , z ) , as a revised generator density , is more expressive on x -space than P θ ( x , z ) as the generated samples from p θ ( x ) are refined via the EBM-guided MCMC sampling. ˜ Q ϕ,θ ( x , z ) , as a revised data density , can be more expressive on z -space than Q ϕ ( x , z ) , as the latent samples from q ϕ ( z | x ) are revised via the generator-guided explaining-away MCMC inference. These MCMCrevised joint densities will be used for better EBM training, while at the same time, they will guide and teach the generator and inference model to better initialize and facilitate MCMC samplings.

We jointly train three models within a probabilistic framework based on KL divergence between joint densities. We present below our learning algorithm in an alternative and iterative manner where the new model parameters are updated based on the current model parameters. We present the learning algorithm in Appendix.

3.2 Learning Energy-based Model

Learning the EBM is based on the minimization of KL divergences as

where α t , θ t , ϕ t denote fixed copies of EBM, generator, and inference model at the t -th step in an iterative algorithm. The joint densities ˜ Q ϕ t ,θ t and ˜ P θ t ,α t are based on this current iteration.

Comparing Eqn.7 to Eqn.2, we compute sampling from EBM through Langevin transition with current p θ t ( x ) as an initializer, i.e., T x α t p θ t ( x ) . Such a generator initialized MCMC is more effective and efficient compared to the noise-initialized transition, T x α t ( ϵ x ) where ϵ x ∼ N (0 , I D ) , that is used in recent literature [29, 5, 6].

MLEperturbation. The above joint space KL divergences are equivalent to the marginal version,

Additionally, T x α t p θ t ( x ) → π α t ( x ) if s → 0 and τ → ∞ (see Eqn.3), thus Eqn.8 amounts to the approximation of MLE objective function with a KL perturbation term, i.e.,

Such surrogate form is more tractable than the MLE objective function, since the log Z( α ) term is canceled out. The π α ( x ) seeks to approach the data distribution p d ( x ) while escapes from its current version π α t ( x ) , thus can be treated as its own critic. The learning of EBM can then be interpreted as a self-adversarial learning [15, 40].

Connection to variational learning. It is also tempting to learn the EBM without MCMC sampling via gradient E p d ( x ) [ ∂ ∂α f α ( x )] -E p θ t ( x ) [ ∂ ∂α f α ( x )] (i.e., min α KL( Q ϕ t ( x , z ) ∥ Π α,ϕ ( x , z )) -KL( P θ t ( x , z ) ∥ Π α,ϕ ( x , z )) ), which underlies the variational joint learning [15, 4, 12, 24]. Compared to Eqn.7, their generator serves as a direct sampler for EBM, while we perform the EBM self-guided MCMCrevision for more accurate samples.

3.3 Learning Generator Model via Dual-MCMC Teaching

As a complementary model for learning the EBM, the generator model becomes a key ingredient toward success. The generator model is learned through the minimization of KL divergences as

where both the ˜ Q ϕ t ,θ t and ˜ P θ t ,α t are based on the current iteration. The revised data density ˜ Q ϕ t ,θ t teaches the generator to better match with empirical data observations through the first KL term, and the revised generator density ˜ P θ t ,α t teaches the generator to better match with generated samples through the second KL term. As we describe below, such a joint minimization scheme provides a tractable approximation of the generator learning with marginal distribution, i.e.,

where generator model p θ ( x ) learns to match the p d ( x ) on empirical data observations and catch up with the current EBM density π α t ( x ) through guidance of its generated samples.

MLE perturbation on p d . Our generator model matches empirical data distribution p d ( x ) through KL( ˜ Q ϕ t ,θ t ( x , z ) ∥ P θ ( x , z )) and is equivalent to the marginal version that follows,

Given s → 0 and τ →∞ , T z θ t q ϕ t ( z | x ) → p θ t ( z | x ) (see Eqn.6), the first KL term (in Eqn.10) thus approximates the true MLE objective function with additional KL perturbation term, i.e.,

Such surrogate form in joint density upper-bounds (i.e., majorizes) the true MLE objective KL( p d ( x ) ∥ p θ ( x )) and can be more tractable as it involves the complete-data model with latent vector z has been inferred in the current learning step. Minimizing the surrogate form in the iterative algorithm makes the generator p θ ( x ) to be closer to the empirical p d ( x ) due to its majorization property [15].

MLE perturbation on π α t . Our generator model is learned to catch up with the EBM model π α ( x ) through the second term KL( ˜ P θ t ,α t ( x , z ) ∥ P θ ( x , z )) . It is equivalent to the marginal version as

With s → 0 and τ →∞ , T x α t p θ t ( x ) → π α t ( x ) (see Eqn.3), our second KL term approximates (in Eqn.10) the MLE objective on π α t for generator,

Such surrogate in joint density again upper-bounds (i.e., majorizes) the true MLE objective on generated samples, i.e., KL( π α t ( x ) ∥ p θ ( x )) , and thus the generator p θ ( x ) updates to be closer to the EBM π α ( x ) at the current iteration.

Connection to variational learning. Without MCMC inference, the generator model can be learned with inference model to match the empirical data distribution, i.e., min θ KL( Q ϕ t ( x , z ) ∥ P θ ( x , z )) , which underlies VAEs [21, 32, 26, 37]. Compared to Eqn.11, VAEs seek to minimize KL( p d ( x ) ∥ p θ ( x )) + E p d ( x ) [KL( q ϕ t ( z | x ) ∥ p θ ( z | x ))] where q ϕ ( z | x ) is assumed to be Gaussian distributed which has limited capacity for generator learning.

Connection to cooperative learning. Cooperative learning schemes [42, 43] share similar EBM training procedures but can be fundamentally different in generator learning. The generators are learned through min θ KL(Π α t ,ϕ t ( x , z ) ∥ P θ ( x , z )) [43] or min θ KL( ˜ P θ t ,α t ( x , z ) ∥ P θ ( x , z )) [42], however, generators have no access to the empirical observations which lead to biased and sub-optimal generator models.

3.4 Learning Inference Model via Dual-MCMC Teaching

The inference model q ϕ ( z | x ) serves as a key component for generator learning which will in turn facilitate the EBM training. In this paper, the inference model is learned by minimizing the KL divergences D q ( ϕ ) as

The revised data density ˜ Q ϕ t ,θ t teaches the inference model on empirical data observations for better real data inference through the first KL term, and the revised generator density ˜ P θ t ,α t teaches the inference model for better generated sample inference through the second KL term.

Real data inference. Optimizing the first term KL( ˜ Q ϕ t ,θ t ( x , z ) ∥ Q ϕ ( x , z )) in Eqn.15 is equivalent to min ϕ E p d ( x ) [KL( T z θ t q ϕ t ( z | x ) ∥ q ϕ ( z | x ))] . Given the long-run optimality condition of the MCMC transition T z θ t q ϕ t ( z | x ) → p θ t ( z | x ) , our first term KL( ˜ Q ϕ t ,θ t ( x , z ) ∥ Q ϕ ( x , z )) tends to learn q ϕ ( z | x ) by minimizing the E p d ( x ) [KL( p θ t ( z | x ) ∥ q ϕ ( z | x ))] . The inference model is learned to match the true generator posterior p θ ( z | x ) on real observations in the current learning step. Specifically, latent samples are initialized from current q ϕ t ( z | x ) , and the generator-guided MCMC revision T z θ t q ϕ t ( z | x ) is then performed to obtain the revised latent samples. The inference model is updated to amortize the MCMC and to absorb such sample revision. The MCMC revision not only drives the evolution of the latent samples, but also drives the evolution of the inference model.

Generated sample inference. Optimizing the second term KL( ˜ P θ t ,α t ( x , z ) ∥ Π α,ϕ ( x , z )) in Eqn.15 is equivalent to min ϕ E T x αt p θ t ( x ) [KL( p θ t ( z | x ) ∥ q ϕ ( z | x ))] which tends to minimizing E π αt ( x ) [KL( p θ t ( z | x ) ∥ q ϕ ( z | x ))] given long-run optimality condition (i.e., T x α t p θ t ( x ) → π α t ( x ) ). The inference model is learned to match the true generator posterior p θ ( z | x ) on generated samples from EBM in the current learning step. Noted that both the generated sample and its latent factor can be readily available where the latent factor z is drawn from prior distribution p ( z ) , which is assumed to be unit Gaussian, and the generated sample is obtained directly from generator.

4 Related Work

Energy-based model. The EBM is flexible and theoretically appealing with various approaches to learning, such as the noise-contrastive estimation (NCE) [1, 38] and the diffusion approach [8]. Most existing works learn the EBM via MLE [29, 30, 5, 6, 39], which typically involves MCMC sampling, while some advance [15, 16, 12, 17, 24] propose to amortize MCMC sampling with the generator model and learn EBM in a close-formed, variational learning scheme. Instead, [42, 43] recruit ancestral Langevin dynamics with the generator model being the initializer model. In this paper, we propose a joint framework where the generator model matches both the EBM and empirical data distribution through dual-MCMC teaching to better benefit the EBM sampling and learning.

Generator model. In recent years, the success of generator models has given rise to various ways of learning methods. Generative adversarial network (GAN) [10, 19, 27, 20] jointly trains the generator model with a discriminator, while V AE [21, 32, 36, 9] is trained with an inference model (or encoder) approximating the generator posterior. Without the inference model, [14, 31] instead utilize MCMC sampling to sample from the generator posterior. Our work differs from theirs by employing the MCMCinference based on informative initialization from the inference model, and we aim to learn the generator model to facilitate effective learning of the EBM.

5 Experiment

In this section, we address the following questions: (1) Can our method learn an EBM with highquality synthesis? (2) Can both the complementary generator and inference model successfully match their MCMC-revised samples? and (3) What is the influence of the inference model and generator model? We refer to implementation details and additional experiments in Appendix.

5.1 Image Modelling

We first evaluate the EBM in image data modelling. Both the generator model and EBM are learned to match empirical data distribution, and if the generator model is well-trained, it can serve as an informative initializer model, making the EBM sampling easier. As a result, the EBM should be capable of generating realistic image synthesis. For evaluation, we generate images from the EBM by obtaining x 0 from the generator and running Langevin dynamics with x 0 being the initial points.

We benchmark our method on standard datasets such as CIFAR-10 [23] and CelebA-64 [25], as well as challenging high-resolution CelebA-HQ-256 [19] and large-scale LSUN-Church-64 [44]. We consider the baseline models, including Divergence Triangle [15], No MCMC EBM [12], Cooperative EBM [42], and Amortized EBM [43], as well as modern advanced generative models, including other EBMs [29, 6, 8, 39, 1], GANs [27, 20] and score-based models [34, 18]. We recruit Fr ´ e chet Inception Distance (FID) and Inception Score (IS) metrics to evaluate the quality of image synthesis. Results are reported in Tab.1 and Tab.2 where our EBM shows the capability of generating realistic image synthesis and renders competitive performance even compared to GANs and score-based models.

5.2 MCMCRevision

The complementary generator and inference model are learned to match their MCMC-revised samples and thus can serve as informative initializers. We demonstrate that both the generator and inference model can successfully catch up with the MCMC revision. We train our model on CelebA-64 using Langevin steps k x = 30 for the MCMC revision on x and k z = 10 for the MCMC revision on z .

Generator model. If the generator model captures different modes of the EBM, the MCMC revision on x should only need to search around the local mode and correct pixel-level details. To examine the generator model, we visualize the Langevin transition by drawing x i for every three steps from generated samples x 0 to MCMC-revised samples x k . As shown in Fig.2, only minor changes can be observed during the transition, suggesting that the generator has matched the EBM-guided MCMC revision. By measuring FID of x 0 and x k , it still improves from 5.94 to 5.15, which indicates pixel-level refinements.

Inference model. We then show the Langevin transition on z . For visualization, latent codes are mapped to data space via the generation network. We draw z i for each step and show corresponding images in Fig.2, where the inference model also catches up with the generator-guided explaining-away MCMCinference, leading to faithful reconstruction as a result.

Energy Profile

MSE

0.20

0.011

0.15

0.010

0.009

0.10

0.008

0.05

0.007

0.00

0.006

10

15

20

25

30

Q

2

5.3 Analysis of Inference Model

The inference model serves as an initializer model for generator learning which in turn facilitates the EBM sampling and learning. To demonstrate the benefit of the inference model, we adopt noise-initialized Langevin dynamics for generator posterior sampling and compare with the Langevin dynamics initialized by the inference model.

Inf + L=10

L=10

L=30

Inf + L=10

L=10

L=30

MSE

SSIM

PSNR

CIFAR-10

CelebA-64

Specifically, we conduct noise-initialized Langevin dynamics with increasing steps from k z = 10 to k z = 30 , and compare with the Langevin dynamics using only k z = 10 steps but is initialized by the inference model. We recruit MSE, Peak Signal-to-Noise Ratio (PSNR), and Structural SIMilarity (SSIM) to measure the inference accuracy of reconstruction and present the results in Fig.3. As the Langevin steps increase, the inference becomes more accurate (lower MSE, higher PSNR, and SSIM), however, it is still less accurate than the proposed method (L = 30 vs. Inf+L = 10). This result highlights the inference model in our framework. We then compare with other models that also characterize an inferential mechanism, such as VAE [21], Wasserstein auto-encoders (WAE) [36], RAE [9], Alternating Back-propagation (ABP) [14], and Short-run ABP (SR-ABP) [31]. As shown in Tab.3, our model can render superior performance with faithful reconstruction.

5.4 Analysis of Generator Model

With the generator model being the initializer for EBM sampling, exploring the energy landscape should become easier by first traversing the low- dimensional latent space. We intend to examine if our generator model can deliver smooth interpolation on the latent space, thus making a smooth transition in the data space. We employ linear interpolation among latent space, i.e., ˜ z = (1 -α ) · z 1 + α · z 2 , and consider two scenarios, such as the image synthesis and image reconstruction. As shown in Fig.4, our generator model is capable of smooth interpolation for both scenarios, which suggests its effectiveness in exploring the energy landscape.

6 Ablation Studies

MCMCsteps of T z θ . We analyze the impact of the inference accuracy in our framework by increasing the Langevin steps of T z θ q ϕ ( z | x ) . With an inference model initializing the MCMC posterior sampling, further increasing the MCMC steps should deliver more accurate inference and thus benefit the generator and EBM for better performance. Thus, we compute the FID, MSE, and wall-clock training time (seconds / per iteration) in Tab.4. It can be seen that increasing MCMC steps from 10 to 30 indeed slightly improves the generation quality and inference accuracy but requires more training time. We thus report the result of Inf+L=10 in Tab.1 and Tab.3.

MCMCstepsof T x α . Then, we discuss the impact of the Langevin steps of T x α . Increasing the MCMC steps of T x should explore the energy landscape more effectively and render better performance in the generation. In Tab.5, starting MCMC steps from 10 to 30, our model exhibits largely improved performance in generation quality but only minor improvement even when we use L = 50 steps. Thus, we report L = 30 steps in Tab.1.

7 Conclusion

We present a joint learning scheme that can effectively learn the EBM by interweaving the maximum likelihood learning of the EBM, generator, and inference model through dual-MCMC teaching. The generator and inference model are learned to initialize MCMC sampling of EBM and generator posterior, respectively, while these EBM-guided MCMC sampling and generator-guided MCMC inference, in turn, serve as two MCMC revision processes that are capable of teaching the generator and inference model. This work may share the limitation with other MCMC-based methods in terms of the computational cost, but we expect to impact the active research of learning the EBMs.

L=10

FID

Time (s)

14.78

0.861

11.51

1.241

L=30

9.26

1.594

L=50

9.07

2.454

References

Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. Advances in neural information processing systems , 34:480-493, 2021. 7

Jiali Cui, Ying Nian Wu, and Tian Han. Learning joint latent space ebm prior model for multi-layer generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3603-3612, June 2023. 1

Jiali Cui, Ying Nian Wu, and Tian Han. Learning hierarchical features with joint latent space energy-based prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 2218-2227, October 2023. 1

Zihang Dai, Amjad Almahairi, Philip Bachman, Eduard H. Hovy, and Aaron C. Courville. Calibrating energy-based generative adversarial networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id=SyxeqhP9ll . 5

Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.08689 , 2019. 1, 2, 3, 5, 7

Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.01316 , 2020. 1, 2, 5, 7

Marylou Gabrié, Grant M Rotskoff, and Eric Vanden-Eijnden. Adaptive monte carlo augmented with normalizing flows. Proceedings of the National Academy of Sciences , 119(10):e2109420119, 2022. 1

R Gao, Y Song, B Poole, YN Wu, and DP Kingma. Learning energy-based models by diffusion recovery likelihood. In International Conference on Learning Representations (ICLR 2021) , 2021. 1, 7

Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. arXiv preprint arXiv:1903.12436 , 2019. 7, 8, 9

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014. 3, 7

Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv:1912.03263 , 2019. 1

Will Sussman Grathwohl, Jacob Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud. No {mcmc} for me: Amortized sampling for fast and stable training of energy-based models. In International Conference on Learning Representations , 2021. URL https://openreview. net/forum?id=ixpSxO9flk3 . 1, 3, 4, 5, 7

Louis Grenioux, Éric Moulines, and Marylou Gabrié. Balanced training of energy-based models with adaptive flow sampling. arXiv preprint arXiv:2306.00684 , 2023. 1

Tian Han, Yang Lu, Song-Chun Zhu, and Ying Nian Wu. Alternating back-propagation for generator network. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, 2017. 3, 4, 7, 8, 9

Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Divergence triangle for joint training of generator model, energy-based model, and inferential model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8670-8679, 2019. 1, 3, 4, 5, 6, 7, 8

Tian Han, Erik Nijkamp, Linqi Zhou, Bo Pang, Song-Chun Zhu, and Ying Nian Wu. Joint training of variational auto-encoder and latent energy-based model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020. 1, 7

Mitch Hill, Erik Nijkamp, Jonathan Craig Mitchell, Bo Pang, and Song-Chun Zhu. Learning probabilistic models from generator latent spaces with hat EBM. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https: //openreview.net/forum?id=AluQNIIb_Zy . 1, 3, 4, 7

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020. 1, 7

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 , 2017. 1, 7

Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems , 33:12104-12114, 2020. 7

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013. 3, 4, 6, 7, 8, 9

Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems , 31, 2018. 1, 7

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 7

Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint arXiv:1901.08508 , 2019. 1, 3, 4, 5, 7

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. CoRR , abs/1411.7766, 2014. URL http://arxiv.org/abs/1411.7766 . 7

Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. Advances in neural information processing systems , 32, 2019. 1, 6

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 , 2018. 7

Radford M Neal et al. Mcmc using hamiltonian dynamics. Handbook of markov chain monte carlo , 2(11): 2, 2011. 2, 3

Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run mcmc toward energy-based model. Advances in Neural Information Processing Systems , 32, 2019. 1, 2, 3, 5, 7

Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of mcmc-based maximum likelihood learning of energy-based models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 5272-5280, 2020. 1, 3, 7

Erik Nijkamp, Bo Pang, Tian Han, Linqi Zhou, Song-Chun Zhu, and Ying Nian Wu. Learning multilayer latent variable model via variational optimization of short run mcmc for approximate inference. In European Conference on Computer Vision , pages 361-378. Springer, 2020. 3, 7, 8, 9

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning , pages 1278-1286. PMLR, 2014. 4, 6, 7

Sergey Samsonov, Evgeny Lagutin, Marylou Gabrié, Alain Durmus, Alexey Naumov, and Eric Moulines. Local-global mcmc kernels: the best of both worlds. Advances in Neural Information Processing Systems , 35:5178-5193, 2022. 1

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems , 32, 2019. 1, 7

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020. 1

Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. arXiv preprint arXiv:1711.01558 , 2017. 7, 8, 9

Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems , 33:19667-19679, 2020. 1, 6

Zhisheng Xiao and Tian Han. Adaptive multi-stage density ratio estimation for learning latent space energy-based model. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/paper/2022/ hash/874a4d89f2d04b4bcf9a2c19545cf040-Abstract-Conference.html . 7

Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy-based models. In International Conference on Learning Representations , 2020. 1, 7

Zhisheng Xiao, Qing Yan, and Yali Amit. Ebms trained with maximum likelihood are generator models trained with a self-adverserial loss. arXiv preprint arXiv:2102.11757 , 2021. 5

Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning , pages 2635-2644. PMLR, 2016. 2

Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. IEEE transactions on pattern analysis and machine intelligence , 42(1):27-45, 2018. 2, 3, 6, 7, 8

Jianwen Xie, Zilong Zheng, and Ping Li. Learning energy-based model with variational auto-encoder as amortized sampler. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10441-10451, 2021. 1, 2, 3, 6, 7

Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015. 7

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method for training and inference of energy-based model (EBM). The author claim that although the use of generator as an initializer model may improve MCMC sampling, training unbiased generator is an open problem . Thus they propose a joint learning framework in which generator is learned by MLE via posterior sampling. While EBMs have a number of appealing properties they are difficult to inference. Therefore learning an unbiased generator for faster MCMC sampling from the EBM distribution is a hot topic.\n\nSTRENGTHS:\nThe paper is clearly written and well presented. The potential is impact of the proposed dual-MCMC teaching is considerable. The problem arised in the paper is actual and mostly uncovered by the previous research. The paper provides a number of novel ideas which could encorage the future research in this direction. The experiments are very clear and solid including fair comparison with concurent works. Remarkably, the numerical results for the proposed method are spectacular.\n\nWEAKNESSES:\n1. The section 2 is clear-written and self-contained but it is large and has little connection to the methodology proposed in the paper. As the authors reforlulate each component described in the section 2 it might be helpful to better explain the benefits of the choice for proposed method. \n\n2. The objective proposed for learning EBM includes maximization of Kulback-Leibler divergence term (equation 7) which is stated to perform as a self-critic for EBM. The grounding beneath this “​​self-adversarial learning” is not very confident. Either literature referenced by the authors lack the verification or some formal reasoning behind this objective. Compared to the objectives for generator and inference model this one doesn’t pose an upper bound for MLE objective.\n\n3. In the Figure 2 in the main text you provide an energy profile after performing $30$ steps (as during training) of MCMC revision on $x$. FID improves from $5.94$ on $x_0$ to $5.15$ on $x_{30}$, still the energy profile is not convergent. It would be more convincing if you provide a result for long-run MCMC (until energy profile convergence) revision for learned EBM to verify your EBM. \n\n4. Please include the training details, including optimization hyperparameters, step sizes for Langevin dynamics, etc.\n\nTypos:\n1. In the line 156 there is a typo for the marginal distribution of x, but inside the brackets you use a proper notation. \n2. In the line 214 “second” is used instead of “first”.\n\nQUESTIONS:\n1. It was noticed [1] that for EBM distribution with density $\\pi(x)$ and generator $p_{\\theta} = p_{\\theta}(x|z)p(z)$ sampling latent variable from $p(z)\\pi(g_{\\theta}(z))$ and pushing it by $g_{\\theta}$ to the ambient space is equivalent to sampling from $\\pi(x)$. Why do you propose to perform EBM inference in the ambient space instead of latent space? As traversing high-dimensional space with MCMC sampling is a hard problem. Please elaborate on why don’t you use an advantage of sampling from EBM in the latent space of the generator model.\n\n2. The paper massively adopts the method proposed in [2], nevertheless considering results on CIFAR-10 dataset FID drops from $30.10$ for [2] to $9.26$ for the current paper. What is a major reason for such a dramatic improvement? In the Appendix you provide a quality for a \"simple network\" which seems to share architecture with paper [2]. It would be helpful for understanding if you provide a comparison of model complexity and/or time, GPU consumption in addition to image quality comparison.\n\n[1] Che, Tong, et al. \"Your gan is secretly an energy-based model and you should use discriminator driven latent sampling.\" Advances in Neural Information Processing Systems 33 (2020): 12275-12287.\n\n[2] Han, Tian, et al. \"Divergence triangle for joint training of generator model, energy-based model, and inferential model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\nLIMITATIONS:\nThe limitations of the method are obscured. The authors only mention that “this work may share the limitation with other MCMC-based methods in terms of the computational cost” which is a bit vague.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the problem of Maximum Likelihood (ML) estimation for Energy-Based Models (EBM). Building on previous research, the authors propose a novel approach that involves learning a surrogate generative model to initialize the costly Langevin steps. The authors chose a latent-based generative model thus requiring additional Markov Chain Monte Carlo (MCMC) sampling for estimation. This new MCMC procedure is amortized using a simpler third generative model, referred to as the inference model. This work provides a procedure to learn the three models jointly. Through extensive experimental evaluations, the authors demonstrate the effectiveness of their proposed framework. They showcase the benefits of incorporating the additional models and highlight the superiority of their combined algorithm over other existing methods. The results substantiate the value of joint learning, showcasing improved accuracy.\n\nSTRENGTHS:\n- Experiments are well detailled.\n- Experiments on Out-of-Distribution Detection (OOD) in App 1.2 allow to evaluate the EBM beyond generating samples.\n\nWEAKNESSES:\n- The motivation to learn the generator with observed samples is unclear.\n- The improved accuracy of this method is not balanced with the computational and memory costs generated.\n- The quality of the density estimation task is barely considered.\n\nQUESTIONS:\n- One of the main justifications of your learning procedure is to learn the generator using training data. Can you explain why not doing this (like in cooperative learning [1,2]) would introduce a bias ? This claim was made multiple times in the main paper.\n- As a justification of your revisited densities (or in L129), you say that Langevin steps could transform something unimodal into something multimodal which is not the case (see for instance [3]) as shown in Fig 2 of Sec 5.2. Is there another justification for those Langevin steps ?\n- The ablation studies (Sec 5.2 and 5.3) highlight the usefulness of the inference model with surprisingly low computational overhead. What about the memory overhead ?\n\n[1] Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. IEEE transactions on pattern analysis and machine intelligence, 42(1):27–45, 2018.\n\n[2] Jianwen Xie, Zilong Zheng, and Ping Li. Learning energy-based model with variational auto-encoder as amortized sampler. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10441–10451, 2021\n\n[3] Véronique Gayrard, Anton Bovier, Markus Klein, Metastability in reversible diffusion processes II: precise asymptotics for small eigenvalues. J. Eur. Math. Soc. 7 (2005), no. 1, pp. 69–99\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a new approach for training energy-based models (EBMs). These are commonly trained by approximating the maximum-likelihood estimate of the model parameters. This is done by minimising a certain Kullback--Leibler divergence via stochastic-gradient descent. To estimate the necessarily gradient, one must approximate an expectation w.r.t. the law of the data under the energy-based model. This is typically done via an MCMC algorithm.\n\nThe present work attempts to speed up this procedure by training an additional surrogate model (the \"generator\" model) from which the MCMC chain can be initialised (so that fewer MCMC iterations are needed). This generator model and the EBM are trained as part of a larger scheme which additionally trains an inference model (i.e. the posterior distribution of the latent variables given the observed data).\n\n-----------------------\nEDIT (2023-09-04): I have now read all the reviews and rebuttals. All my questions have been addressed. I continue to be happy for this paper to be published.\n\nSTRENGTHS:\n1. The method appears to be sound and novel.\n2. The paper is mostly well written.\n\nWEAKNESSES:\nMain comments:\n\nMy expertise with training EBMs in this fashion is too limited to offer deeper insights. However:\n\n1. Some of the objectives used for training the three models appear a little ad-hoc and their motivation/justification sometimes leans on convergence of the MCMC chains (which seems unlikely to occur in practice given the small number of MCMC iterations used). At the very least, some added justification/derivation of these objectives would help the reader gain more intuition for the method.\n2. I'd ask the authors to confirm that the numerical comparisons are fair in the sense that all algorithms have roughly the same computational cost (see question below).\n3. I think it would help to have some (brief/high-level) pseudo code in the main paper.\n\n\nMinor comments/typos:\n\nL114: grammar in \"such Langevin process\"\nL180: are the KL divergences equivalent to the marginal version or is the /minimisation/ of these KL divergences equivalent to minimising the marginal divergences?\nL152--160: I understand what the authors mean with the \"transition-kernel\" notation. But I think the notation/explanation in this paragraph needs to be improved. For example, in Line 156, the stated expression is referred to as \"marginal distribution on $x$\" but it clearly represents distribution on joint the space of $(x, z)$. I would state the formal definitions (i.e. those found in Lines 158 and 160) right from the start to enhance clarity. It might also be worth reminding the reader that $p_\\theta(x)$ is the marginal of $p(z)p_\\theta(x|z)$.\nL256--257: the MCMC sampling -> MCMC sampling\n\n- there are sometimes missing spaces between \"Eq\" and the equation number and between \"Sec.\" and the section number.\n- punctuation is needed even if a sentence ends in a (displayed) equation\n- the font size in most of the figures is too small (especially in Figure 3) and axis labels are sometimes missing\n- it may just be my lack of familiarity with the literature in this area but is it clear to the reader what \"explaining away inference\" means in this specific context? If not, maybe it'd be worth adding a brief explanation.\n- table captions normally go /above/ tables (unless the NeurIPS style guide says otherwise)\n\nQUESTIONS:\nIn Table 1 (and also in the numerical results from Sections 5.2 and 5.3), is the computational cost of all algorithms comparable?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates learning EBMs for image data using auxiliary generator and inference models. The model generates samples by drawing a latent normal vector, passing it through the generator, and refining the generator sample using MCMC with the EBM. An inference network, which predicts latent vectors from images, is used to assist with training the EBM and generator. The key innovation is to update both generator samples using image space MCMC and inferred latent vectors for data using latent space MCMC. This allows the generator to be efficiently trained using both revised MCMC samples, as in Cooperative Learning, and revised latent codes for data, which is unique to this work. Training the generator to match both EBM and data samples is crucial for improving performance beyond the results achieved by related methods. Training algorithms for the EBM, generator, and inference network are presented using several KL terms between different joint distributions of images and latent codes. Experiments show strong EBM synthesis results for CIFAR-10, CelebA, CelebA HQ, and LSUN-Church-64.\n\nSTRENGTHS:\n* The work identifies and addresses a key limitation of the Cooperative Learning framework, which is the fact that the generator is trained to reconstruct only refined EBM samples. This can limit the potential of the generator since EBM samples are less realistic than true data. While other works such as Divergence Triangle train the generator to reconstruct data images using inferred codes for observed data, these works lack an MCMC refinement stage which is crucial for improving sample quality. This work bridges the gap between Cooperative Learning style methods and Divergence Triangle style methods to incorporate both MCMC sampling and generator data reconstruction into a single learning framework.\n* Experimental results show significant benefits from the proposed method compared to a variety of other EBM learning methods.\n\nWEAKNESSES:\n* The loss functions appear to be significantly inspired by the Divergence Triangle, but the paper lacks a thorough comparison between the proposed method and Divergence Triangle. Such a discussion would help elucidate the differences between the proposed method and prior work. To my understanding, the primary differences are 1) replace all LHS of KL terms with distributions defined by short-run MCMC distributions with frozen model parameters, 2) add KL($\\tilde{P} || P$) and KL($\\tilde{Q} || Q$) terms to the generator and inference network updates respectively.\n* The motivation of each loss term is not entirely clear, which makes the proposed loss gradients appear somewhat ad-hoc. An intuitive explanation of the role of each KL term would be very helpful for the reader. A clearer discussion of connections to the Divergence Triangle could also help with understanding the KL terms.\n* The proposed method is somewhat complex, and the loss functions are defined sequentially given the current fixed model weights. Such an approach is common to Cooperative Learning and related methods, while amortized methods like Divergence Triangle have a loss function which can be written in closed form without reference to fixed model weights. While this is not a major issue in my view, the reliance on loss defined by a sequence of fixed model weights as opposed to a single loss function should be discussed as a limitation of the proposed method.\n\nMinor Problems\n* The related work [12] uses MCMC sampling initialized from the generator to update the EBM similar to [36, 37], instead direct generator samples like [11, 9, 19].\n\nQUESTIONS:\nCould the authors provide a detailed comparison between their proposed method and the Divergence Triangle?\n\nLIMITATIONS:\nA broader impact section was not included. Computational limitations are very briefly discussed, but more thorough details about runtime and compute costs compared to related methods would strengthen the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}