SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiali Cui, Tian Han
Department of Computer Science, Stevens Institute of Technology
{jcui7,than6}@stevens.edu
This paper studies the fundamental learning problem of the energy-based model (EBM). Learning the EBM can be achieved using the maximum likelihood estimation (MLE), which typically involves the Markov Chain Monte Carlo (MCMC) sampling, such as the Langevin dynamics. However, the noise-initialized Langevin dynamics can be challenging in practice and hard to mix. This motivates the exploration of joint training with the generator model where the generator model serves as a complementary model to bypass MCMC sampling. However, such a method can be less accurate than the MCMC and result in biased EBM learning. While the generator can also serve as an initializer model for better MCMC sampling, its learning can be biased since it only matches the EBM and has no access to empirical training examples. Such biased generator learning may limit the potential of learning the EBM. To address this issue, we present a joint learning framework that interweaves the maximum likelihood learning algorithm for both the EBM and the complementary generator model. In particular, the generator model is learned by MLE to match both the EBM and the empirical data distribution, making it a more informative initializer for MCMC sampling of EBM. Learning generator with observed examples typically requires inference of the generator posterior. To ensure accurate and efficient inference, we adopt the MCMC posterior sampling and introduce a complementary inference model to initialize such latent MCMC sampling. We show that three separate models can be seamlessly integrated into our joint framework through two (dual-) MCMC teaching, enabling effective and efficient EBM learning.
Deep generative models have made significant progress in learning complex data distributions [35, 18, 37, 34, 19, 22, 5] and have found successful applications in a wide range of real-world scenarios [26, 6, 11, 15]. Among these, the energy-based model (EBM) [5, 6, 29, 8, 39, 2, 3] has gained particular interest as a flexible and expressive generative model with an energy function parameterized by a neural network. Learning the EBM can be accomplished via the maximum likelihood estimation (MLE), which involves the Markov Chain Monte Carlo (MCMC) sampling in high-dimensional data space. However, such MCMC sampling has shown to be challenging [30, 7, 33, 13], as it may take a long time to mix between different local modes with a non-informative noise initialization [43, 15].
To address this challenge, recent advances have explored employing complementary models to substitute for MCMC sampling [15, 16, 12, 17, 24]. One notable example is the generator model. The generator model incorporates a top-down generation network that is capable of mapping low-dimensional latent space to high-dimensional data space and admits efficient sample generation. The generator is learned to match the EBM so that MCMC sampling can be replaced by generator ancestral
sampling. However, such direct generator sampling has shown to be less accurate and suboptimal [43]. To alleviate this issue, [42, 43] introduced cooperative learning, where samples generated by the generator model serve as initial points, and then followed by a finite-step MCMC revision process. While this gradient-based MCMC revision process can be more accurate, the generator model learned relies solely on the EBM and has no access to the observed empirical observations. As a result, this learning scheme may render biased generator learning, which in turn caps the potential of learning a strong EBM. An effective joint learning scheme for the EBM and its complementary generator model is needed, yet still in its infancy.
In this paper, we present a novel learning scheme that can seamlessly integrate the EBM and complementary models into a joint probabilistic framework. Specifically, both the EBM and complementary generator model are learned to match the empirical data distribution, while the generator model, at the same time, is also learned to match the EBM. Learning the generator model with empirical training examples can be achieved with MLE, which typically requires access to the generator posterior as an inference process. To ensure an effective and efficient inference, we employ the MCMC posterior sampling with the complementary inference model learned as an initializer. Together with MCMC sampling of EBM being initialized by the generator model, such two MCMC samplings can be further used as two MCMC revision processes that teach the generator and inference model to absorb MCMC-revised samples, thus we term our framework dual-MCMC teaching. We show that our joint framework is capable of teaching the complementary models and thus learning a strong EBM.
- We introduce a novel method that integrates the EBM and its complementary models into a joint learning scheme.
- We propose the use of dual-MCMC teaching for generator and inference models to facilitate efficient yet accurate sampling and inference, which in turn leads to effective EBM learning.
- We conduct extensive experiments to demonstrate the superior performance of our EBM.
2.1 Energy-based Model
Let  $\mathbf{x} \in R^{D}$  be the high-dimensional observed examples. The energy-based model (EBM) [41, 6, 5, 29] represents data uncertainty with an undirected probability density defined as
$$
\pi_ {\alpha} (\mathbf {x}) = \frac {1}{Z (\alpha)} \exp [ f _ {\alpha} (\mathbf {x}) ], \tag {1}
$$
where  $-f_{\alpha}(\mathbf{x})$  is the energy function parameterized with parameters  $\alpha$ , and  $Z(\alpha) (= \int_{\mathbf{x}} \exp[f_{\alpha}(\mathbf{x})] d\mathbf{x})$  is the partition function or normalizing constant.
Maximum likelihood estimation. The maximum likelihood estimation (MLE) is known for being an asymptotically optimal estimator and can be used for training the EBM. In particular, with observed examples,  $\{\mathbf{x}^{(i)}, i = 1,2,\dots,n\}$ , the MLE learning of EBM maximizes the log-likelihood  $L_{\pi}(\alpha) = \frac{1}{n}\sum_{i=1}^{n}\log \pi_{\alpha}(\mathbf{x}^{(i)})$ . If the sample size  $n$  is large enough, the maximum likelihood estimator minimizes the  $\mathrm{KL}(p_{\mathrm{d}}(\mathbf{x})\|\pi_{\alpha}(\mathbf{x}))$  which is the Kullback-Leibler (KL) divergence between the empirical data distribution  $p_{\mathrm{d}}(\mathbf{x})$  and the EBM distribution  $\pi_{\alpha}(\mathbf{x})$ . The gradient  $\frac{\partial}{\partial\alpha} L_{\pi}(\alpha)$  is computed as
$\max_{\alpha}L_{\pi}(\alpha) = \min_{\alpha}\mathrm{KL}(p_{\mathrm{d}}(\mathbf{x})\| \pi_{\alpha}(\mathbf{x}))$  where
$$
\frac {\partial}{\partial \alpha} L _ {\pi} (\alpha) = \mathbb {E} _ {p _ {\mathrm {d}} (\mathbf {x})} \left[ \frac {\partial}{\partial \alpha} f _ {\alpha} (\mathbf {x}) \right] - \mathbb {E} _ {\pi_ {\alpha} (\mathbf {x})} \left[ \frac {\partial}{\partial \alpha} f _ {\alpha} (\mathbf {x}) \right] \tag {2}
$$
Given such a gradient, the EBM can be learned via stochastic gradient ascent.
Sampling from EBM. The Eqn.2 requires sampling from the EBM  $\pi_{\alpha}(\mathbf{x})$ , which can be achieved via Markov Chain Monte Carlo (MCMC) sampling, such as the Langevin dynamics [28]. Specifically, to sample from the EBM, the Langevin dynamics iteratively updates as
$$
\mathbf {x} _ {\tau + 1} = \mathbf {x} _ {\tau} + s \frac {\partial}{\partial \mathbf {x} _ {\tau}} \log \pi_ {\alpha} (\mathbf {x} _ {\tau}) + \sqrt {2 s} U _ {\tau} \tag {3}
$$
where  $\tau$  indexes the time step,  $s$  is the step size and  $U_{\tau} \sim \mathcal{N}(0, I_D)$ .
As  $s \to 0$ , and  $\tau \to \infty$ , the distribution of  $\mathbf{x}_{\tau}$  will converge to the target distribution  $\pi_{\alpha}(\mathbf{x}_{\tau})$  regardless of the initial distribution of  $\mathbf{x}_0$  [28]. The existing practice [29, 30, 5] adopts non-informative distribution for  $\mathbf{x}_0$ , such as unit Gaussian or uniform, to initialize the Langevin transition, but it can be extremely inefficient and ineffective as they usually take a long time to converge between different modes and are also non-stable in practice [43]. The ability to generate efficient and effective samples from the model distribution becomes the key step toward training successful EBMs. In this paper, we study the complementary model, i.e., generator model, as an informative initializer for effective yet efficient MCMC exploration toward better EBM training.
2.2 Generator Model
Various works [15, 12, 17, 24] have explored the use of the generator as an amortized sampler to replace the costly noise-initialized MCMC sampling for EBM training. Such a learning approach relies on samples directly drawn from complementary models, which can be less accurate than iterative MCMC sampling as it lacks a fine-grained exploration of the energy landscape. [42, 43] propose the cooperative scheme in which MCMC sampling of EBM is initialized by the generated samples from the generator model. However, the generator model has no access to the observed training examples, and such a biased generator learning makes the EBM sampling ineffective and renders limited model training.
In this paper, the generator model is learned by MLE to match both the empirical data distribution and the EBM distribution. Such training ensures a stronger generator model which will further facilitate a more effective EBM sampling and learning. We present the background of the generator model and its MLE learning algorithm below, which shall serve as the foundation of our proposed method.
Generator model. Let  $\mathbf{z} \in R^{d}$  ( $d < D$ ) be the low-dimensional latent variables. The generator model [14, 10, 21] seeks to explain the observation signal  $\mathbf{x}$  by a latent vector  $\mathbf{z}$  and can be specified as
$$
p _ {\theta} (\mathbf {x}, \mathbf {z}) = p (\mathbf {z}) p _ {\theta} (\mathbf {x} | \mathbf {z}) \tag {4}
$$
where  $p(\mathbf{z})$  is a known prior distribution such as unit Gaussian, e.g.,  $p(\mathbf{z}) \sim \mathcal{N}(0, I_d)$ , and  $p_{\theta}(\mathbf{x}|\mathbf{z}) \sim \mathcal{N}(g_{\theta}(\mathbf{z}), \sigma^2 I_D)$  is the generation model that is specified by neural network  $g_{\theta}(.)$  that maps from latent space to data space.
Maximum likelihood estimation. The MLE learning of the generator model computes log-likelihood over the observed examples as  $L_{p}(\theta) = \frac{1}{n}\sum_{i=1}^{n}\log p_{\theta}(\mathbf{x}^{(i)})$ , where  $p_{\theta}(\mathbf{x}) (= \int_{\mathbf{z}} p_{\theta}(\mathbf{x}, \mathbf{z}) d\mathbf{z})$  is the marginal distribution. If the sample size  $n$  is large, it is equivalent to minimizing the KL divergence  $\mathrm{KL}(p_{\mathrm{d}}(\mathbf{x}) \| p_{\theta}(\mathbf{x}))$ . The gradient of the likelihood  $L_{p}(\theta)$  can be obtained via:
$$
\max  _ {\theta} L _ {p} (\theta) = \min  _ {\theta} \operatorname {K L} \left(p _ {\mathrm {d}} (\mathbf {x}) \| p _ {\theta} (\mathbf {x})\right), \quad \text {w h e r e}
$$
$$
\frac {\partial}{\partial \theta} L _ {p} (\theta) = \mathbb {E} _ {p _ {\mathrm {d}} (\mathbf {x}) p _ {\theta} (\mathbf {z} | \mathbf {x})} \left[ \frac {\partial}{\partial \theta} \log p _ {\theta} (\mathbf {x}, \mathbf {z}) \right] \tag {5}
$$
With such a gradient, the generator model can be learned via stochastic gradient ascent.
Sampling from generator posterior. The Eqn.5 requires the sampling from the generator posterior  $p_{\theta}(\mathbf{z}|\mathbf{x})$ . One can use MCMC sampling such as Langevin dynamics [28] that iterates
$$
\mathbf {z} _ {\tau + 1} = \mathbf {z} _ {\tau} + s \frac {\partial}{\partial \mathbf {z} _ {\tau}} \log p _ {\theta} (\mathbf {z} _ {\tau} | \mathbf {x}) + \sqrt {2 s} U _ {\tau} \tag {6}
$$
where  $U_{\tau} \sim \mathcal{N}(0, I_d)$ . Such a Langevin process is an explaining-away inference where the latent factors compete with each other to explain each training example. As  $s \to 0$ , and  $\tau \to \infty$ , the distribution of  $\mathbf{z}_{\tau}$  will converge to the posterior  $p_{\theta}(\mathbf{z}|\mathbf{x})$  regardless of the initial distribution of  $\mathbf{z}_0$  [28]. However, noise-initialized Langevin [14, 31] can be ineffective in traversing the latent space and hard to mix. In this paper, we introduce a complementary model, i.e., inference model, as an informative initializer for effective yet efficient latent space MCMC exploration for better generator and EBM training.
2.3 Inference model
The inference model  $q_{\phi}(\mathbf{z}|\mathbf{x})$  is adopted in VAEs [21, 32] as an amortized sampler to bypass the costly noise-initialized latent space MCMC sampling. In VAEs,  $q_{\phi}(\mathbf{z}|\mathbf{x})$  is Gaussian parameterized, i.e.,  $\mathcal{N}(\mu_{\phi}(\mathbf{x}),V_{\phi}(\mathbf{x}))$ , where  $\mu_{\phi}(\mathbf{x})$  is the mean  $d$ -dimensional mean vector and  $V_{\phi}(\mathbf{x})$  is the  $d$ -dimensional diagonal covariance matrix. Such a Gaussian parameterized inference model is a tractable approximation to the true generator posterior  $p_{\theta}(\mathbf{z}|\mathbf{x})$ , but can be limited to approximate the multimodal posterior. We adopt the same Gaussian parametrization of  $q_{\phi}(z|x)$  in this paper, but unlike the VAEs, our inference model serves as an initializer network that jump-starts the latent MCMC sampling from an informative initialization. The marginal distribution obtained after Langevin can be more general and multi-modal than the Gaussian distribution.
To effectively learn the EBM, we propose a joint learning framework that interweaves maximum likelihood learning algorithms for both the EBM and its complementary models. For the MLE learning of the EBM, MCMC sampling can be initialized through the complementary generator model, while for the MLE learning of the generator model, the latent MCMC sampling can be initialized by the complementary inference model. Three models are seamlessly integrated into our joint framework and are learned through dual-MCMC teaching.
3.1 Dual-MCMC Sampling
The EBM  $\pi_{\alpha}(\mathbf{x})$ , generator  $p_{\theta}(\mathbf{x})$  and the inference model  $q_{\phi}(\mathbf{z}|\mathbf{x})$  defined in Sec.2 naturally specify the three densities on joint space  $(\mathbf{x},\mathbf{z})$ , i.e.,
$$
P _ {\theta} (\mathbf {x}, \mathbf {z}) = p _ {\theta} (\mathbf {x} | \mathbf {z}) p (\mathbf {z}), \quad \Pi_ {\alpha , \phi} (\mathbf {x}, \mathbf {z}) = \pi_ {\alpha} (\mathbf {x}) q _ {\phi} (\mathbf {z} | \mathbf {x}), \quad Q _ {\phi} (\mathbf {x}, \mathbf {z}) = p _ {\mathrm {d}} (\mathbf {x}) q _ {\phi} (\mathbf {z} | \mathbf {x})
$$
The generator density  $P_{\theta}(\mathbf{x},\mathbf{z})$  specifies the joint density through ancestral generator sampling from prior latent vectors. Both the joint EBM density  $\Pi_{\alpha,\phi}(\mathbf{x},\mathbf{z})$  and data density  $Q_{\phi}(\mathbf{x},\mathbf{z})$  include inference model  $q_{\phi}(\mathbf{z}|\mathbf{x})$  to bridge the marginal distribution to joint  $(\mathbf{x},\mathbf{z})$  space. However,  $q_{\phi}(\mathbf{z}|\mathbf{x})$  is modeled and learned from two different perspectives, one on empirical observed data distribution  $p_{\mathrm{d}}(\mathbf{x})$  for real data inference, and one on EBM density  $\pi_{\alpha}(\mathbf{x})$  for generated sample inference.
The joint learning schemes [15, 12, 17, 24] based on these joint distributions can be limited, because 1) the generator samples from  $P_{\theta}(\mathbf{x}, \mathbf{z})$  is conditionally Gaussian distributed (Sec.2.2) which can be ineffective to capture the high-dimensional multi-modal empirical data distribution, and 2) the inference model  $q_{\phi}(\mathbf{z}|\mathbf{x})$  on observed training examples is assumed to be conditionally Gaussian distributed (Sec.2.3) that is incapable of explaining-away inference [14].
To address the above limitations, we introduce two joint distributions that incorporate MCMC sampling as revision processes,
$$
\tilde {P} _ {\theta , \alpha} (\mathbf {x}, \mathbf {z}) = \mathcal {T} _ {\alpha} ^ {\mathbf {x}} p _ {\theta} (\mathbf {x} | \mathbf {z}) p (\mathbf {z}) \quad \tilde {Q} _ {\phi , \theta} (\mathbf {x}, \mathbf {z}) = p _ {\mathrm {d}} (\mathbf {x}) \mathcal {T} _ {\theta} ^ {\mathbf {z}} q _ {\phi} (\mathbf {z} | \mathbf {x})
$$
where  $\mathcal{T}_{\theta}^{\mathbf{z}}(\cdot)$  denotes the Markov transition kernel of finite step Langevin dynamics that samples  $\mathbf{z}$  from  $p_{\theta}(\mathbf{z}|\mathbf{x})$  (see Eqn.6), and  $\mathcal{T}_{\alpha}^{\mathbf{x}}(\cdot)$  denotes the transition kernel that samples  $\mathbf{x}$  from  $\pi_{\alpha}(\mathbf{x})$  as shown in Eqn.3. Therefore,  $\mathcal{T}_{\alpha}^{\mathbf{x}}p_{\theta}(\mathbf{x}) (= \int_{\mathbf{x}'}\int_{\mathbf{z}}\mathcal{T}_{\alpha}^{\mathbf{x}}(\mathbf{x}')p_{\theta}(\mathbf{x}',\mathbf{z})d\mathbf{z}d\mathbf{x}')$  indicates the marginal distribution of  $\mathbf{x}$  obtained by running MCMC transition  $\mathcal{T}_{\alpha}^{\mathbf{x}}(\cdot)$  that is initialized from  $p_{\theta}(\mathbf{x})$ . Similarly,  $\mathcal{T}_{\theta}^{\mathbf{z}}q_{\phi}(\mathbf{z}|\mathbf{x})$  represents the marginal distribution of  $\mathbf{z}$  obtained by running  $\mathcal{T}_{\theta}^{\mathbf{z}}(\cdot)$  that is initialized from  $q_{\phi}(\mathbf{z}|\mathbf{x})$  given observation  $\mathbf{x}$  (i.e.,  $\mathcal{T}_{\theta}^{\mathbf{z}}q_{\phi}(\mathbf{z}|\mathbf{x}) = \int_{\mathbf{z}'}\mathcal{T}_{\theta}^{\mathbf{z}}(\mathbf{z}')q_{\phi}(\mathbf{z}'|\mathbf{x})d\mathbf{z}'$ ).
The  $\tilde{P}_{\theta,\alpha}(\mathbf{x},\mathbf{z})$ , as a revised generator density, is more expressive on  $\mathbf{x}$ -space than  $P_{\theta}(\mathbf{x},\mathbf{z})$  as the generated samples from  $p_{\theta}(\mathbf{x})$  are refined via the EBM-guided MCMC sampling.  $\tilde{Q}_{\phi,\theta}(\mathbf{x},\mathbf{z})$ , as a revised data density, can be more expressive on  $\mathbf{z}$ -space than  $Q_{\phi}(\mathbf{x},\mathbf{z})$ , as the latent samples from  $q_{\phi}(\mathbf{z}|\mathbf{x})$  are revised via the generator-guided explaining-away MCMC inference. These MCMC-revised joint densities will be used for better EBM training, while at the same time, they will guide and teach the generator and inference model to better initialize and facilitate MCMC samplings.
We jointly train three models within a probabilistic framework based on KL divergence between joint densities. We present below our learning algorithm in an alternative and iterative manner where the new model parameters are updated based on the current model parameters. We present the learning algorithm in Appendix.
3.2 Learning Energy-based Model
Learning the EBM is based on the minimization of KL divergences as
$$
\min _ {\alpha} D _ {\pi} (\alpha) = \min _ {\alpha} \mathrm {K L} \left(\tilde {Q} _ {\phi_ {t}, \theta_ {t}} (\mathbf {x}, \mathbf {z}) \| \Pi_ {\alpha , \phi} (\mathbf {x}, \mathbf {z})\right) - \mathrm {K L} \left(\tilde {P} _ {\theta_ {t}, \alpha_ {t}} (\mathbf {x}, \mathbf {z}) \| \Pi_ {\alpha , \phi} (\mathbf {x}, \mathbf {z})\right)
$$
$$
\text {w h e r e} - \frac {\partial}{\partial \alpha} D _ {\pi} (\alpha) = \mathbb {E} _ {p _ {\mathrm {d}} (\mathbf {x})} \left[ \frac {\partial}{\partial \alpha} f _ {\alpha} (\mathbf {x}) \right] - \mathbb {E} _ {\mathcal {T} _ {\alpha_ {t}} ^ {\mathbf {x}} p _ {\theta_ {t}} (\mathbf {x})} \left[ \frac {\partial}{\partial \alpha} f _ {\alpha} (\mathbf {x}) \right] \tag {7}
$$
where  $\alpha_{t},\theta_{t},\phi_{t}$  denote fixed copies of EBM, generator, and inference model at the  $t$  -th step in an iterative algorithm. The joint densities  $\tilde{Q}_{\phi_t,\theta_t}$  and  $\tilde{P}_{\theta_t,\alpha_t}$  are based on this current iteration.
Comparing Eqn.7 to Eqn.2, we compute sampling from EBM through Langevin transition with current  $p_{\theta_t}(\mathbf{x})$  as an initializer, i.e.,  $\mathcal{T}_{\alpha_t}^{\mathbf{x}}p_{\theta_t}(\mathbf{x})$ . Such a generator initialized MCMC is more effective and efficient compared to the noise-initialized transition,  $\mathcal{T}_{\alpha_t}^{\mathbf{x}}(\epsilon_{\mathbf{x}})$  where  $\epsilon_{\mathbf{x}} \sim \mathcal{N}(0,I_D)$ , that is used in recent literature [29, 5, 6].
MLE perturbation. The above joint space KL divergences are equivalent to the marginal version,
$$
\min  _ {\alpha} D _ {\pi} (\alpha) = \min  _ {\alpha} \mathrm {K L} \left(p _ {\mathrm {d}} (\mathbf {x}) \| \pi_ {\alpha} (\mathbf {x})\right) - \mathrm {K L} \left(\mathcal {T} _ {\alpha_ {t}} ^ {\mathbf {x}} p _ {\theta_ {t}} (\mathbf {x}) \| \pi_ {\alpha} (\mathbf {x})\right) \tag {8}
$$
Additionally,  $\mathcal{T}_{\alpha_t}^{\mathbf{x}}p_{\theta_t}(\mathbf{x})\to \pi_{\alpha_t}(\mathbf{x})$  if  $s\rightarrow 0$  and  $\tau \rightarrow \infty$  (see Eqn.3), thus Eqn.8 amounts to the approximation of MLE objective function with a KL perturbation term, i.e.,
$$
\operatorname {K L} \left(p _ {\mathrm {d}} (\mathbf {x}) \| \pi_ {\alpha} (\mathbf {x})\right) - \operatorname {K L} \left(\pi_ {\alpha_ {t}} (\mathbf {x}) \| \pi_ {\alpha} (\mathbf {x})\right) \tag {9}
$$
Such surrogate form is more tractable than the MLE objective function, since the  $\log Z(\alpha)$  term is canceled out. The  $\pi_{\alpha}(\mathbf{x})$  seeks to approach the data distribution  $p_{\mathrm{d}}(\mathbf{x})$  while escapes from its current version  $\pi_{\alpha_t}(\mathbf{x})$ , thus can be treated as its own critic. The learning of EBM can then be interpreted as a self-adversarial learning [15, 40].
Connection to variational learning. It is also tempting to learn the EBM without MCMC sampling via gradient  $\mathbb{E}_{p_{\mathrm{d}}(\mathbf{x})}[\frac{\partial}{\partial\alpha} f_{\alpha}(\mathbf{x})] - \mathbb{E}_{p_{\theta_t}(\mathbf{x})}[\frac{\partial}{\partial\alpha} f_{\alpha}(\mathbf{x})]$  (i.e.,  $\min_{\alpha}\mathrm{KL}(Q_{\phi_t}(\mathbf{x},\mathbf{z})\| \Pi_{\alpha ,\phi}(\mathbf{x},\mathbf{z}))-$ $\mathrm{KL}(P_{\theta_t}(\mathbf{x},\mathbf{z})\| \Pi_{\alpha ,\phi}(\mathbf{x},\mathbf{z}))$  ), which underlies the variational joint learning [15, 4, 12, 24]. Compared to Eqn.7, their generator serves as a direct sampler for EBM, while we perform the EBM self-guided MCMC revision for more accurate samples.
3.3 Learning Generator Model via Dual-MCMC Teaching
As a complementary model for learning the EBM, the generator model becomes a key ingredient toward success. The generator model is learned through the minimization of KL divergences as
$$
\min _ {\theta} D _ {p} (\theta) = \min _ {\theta} \mathrm {K L} \left(\tilde {Q} _ {\phi_ {t}, \theta_ {t}} (\mathbf {x}, \mathbf {z}) \| P _ {\theta} (\mathbf {x}, \mathbf {z})\right) + \mathrm {K L} \left(\tilde {P} _ {\theta_ {t}, \alpha_ {t}} (\mathbf {x}, \mathbf {z}) \| P _ {\theta} (\mathbf {x}, \mathbf {z})\right)
$$
$$
\text {w h e r e} - \frac {\partial}{\partial \theta} D _ {p} (\theta) = \mathbb {E} _ {p _ {\mathrm {d}} (\mathbf {x}) \mathcal {T} _ {\boldsymbol {t}} ^ {\mathbf {z}}} q _ {\phi_ {t}} (\mathbf {z} | \mathbf {x}) [ \frac {\partial}{\partial \theta} \log p _ {\theta} (\mathbf {x}, \mathbf {z}) ] + \mathbb {E} _ {\mathcal {T} _ {\alpha_ {t}} ^ {\mathbf {x}} p _ {\theta_ {t}} (\mathbf {x} | \mathbf {z}) p (\mathbf {z})} [ \frac {\partial}{\partial \theta} \log p _ {\theta} (\mathbf {x}, \mathbf {z}) ] \tag {10}
$$
where both the  $\tilde{Q}_{\phi_t,\theta_t}$  and  $\tilde{P}_{\theta_t,\alpha_t}$  are based on the current iteration. The revised data density  $\tilde{Q}_{\phi_t,\theta_t}$  teaches the generator to better match with empirical data observations through the first KL term, and the revised generator density  $\tilde{P}_{\theta_t,\alpha_t}$  teaches the generator to better match with generated samples through the second KL term. As we describe below, such a joint minimization scheme provides a tractable approximation of the generator learning with marginal distribution, i.e.,
$$
\min _ {\theta} \mathrm {K L} (p _ {\mathrm {d}} (\mathbf {x}) \| p _ {\theta} (\mathbf {x})) + \mathrm {K L} (\pi_ {\alpha_ {t}} (\mathbf {x}) \| p _ {\theta} (\mathbf {x}))
$$
where generator model  $p_{\theta}(\mathbf{x})$  learns to match the  $p_{\mathrm{d}}(\mathbf{x})$  on empirical data observations and catch up with the current EBM density  $\pi_{\alpha_t}(\mathbf{x})$  through guidance of its generated samples.