SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MLE perturbation on  $p_{\mathrm{d}}$ . Our generator model matches empirical data distribution  $p_{\mathrm{d}}(\mathbf{x})$  through  $\mathrm{KL}(\tilde{Q}_{\phi_t,\theta_t}(\mathbf{x},\mathbf{z})\| P_\theta (\mathbf{x},\mathbf{z}))$  and is equivalent to the marginal version that follows,
$$
\mathrm {K L} \left(\tilde {Q} _ {\phi_ {t}, \theta_ {t}} (\mathbf {x}, \mathbf {z}) \| P _ {\theta} (\mathbf {x}, \mathbf {z})\right) = \mathrm {K L} \left(p _ {\mathrm {d}} (\mathbf {x}) \| p _ {\theta} (\mathbf {x})\right) + \mathbb {E} _ {p _ {\mathrm {d}} (\mathbf {x})} \left[ \mathrm {K L} \left(\mathcal {T} _ {\theta_ {t}} ^ {\mathbf {z}} q _ {\phi_ {t}} (\mathbf {z} | \mathbf {x}) \| p _ {\theta} (\mathbf {z} | \mathbf {x})) \right] \right. \tag {11}
$$
Given  $s\to 0$  and  $\tau \rightarrow \infty$ ,  $\mathcal{T}_{\theta_t}^{\mathbf{z}}q_{\phi_t}(\mathbf{z}|\mathbf{x})\to p_{\theta_t}(\mathbf{z}|\mathbf{x})$  (see Eqn.6), the first KL term (in Eqn.10) thus approximates the true MLE objective function with additional KL perturbation term, i.e.,
$$
\operatorname {K L} \left(p _ {\mathrm {d}} (\mathbf {x}) \| p _ {\theta} (\mathbf {x})\right) + \mathbb {E} _ {p _ {\mathrm {d}} (\mathbf {x})} \left[ \operatorname {K L} \left(p _ {\theta_ {t}} (\mathbf {z} | \mathbf {x}) \| p _ {\theta} (\mathbf {z} | \mathbf {x})\right) \right] \tag {12}
$$
Such surrogate form in joint density upper-bounds (i.e., majorizes) the true MLE objective  $\mathrm{KL}(p_{\mathrm{d}}(\mathbf{x})\| p_{\theta}(\mathbf{x}))$  and can be more tractable as it involves the complete-data model with latent vector  $\mathbf{z}$  has been inferred in the current learning step. Minimizing the surrogate form in the iterative algorithm makes the generator  $p_{\theta}(\mathbf{x})$  to be closer to the empirical  $p_{\mathrm{d}}(\mathbf{x})$  due to its majorization property [15].
MLE perturbation on  $\pi_{\alpha_t}$ . Our generator model is learned to catch up with the EBM model  $\pi_{\alpha}(\mathbf{x})$  through the second term  $\mathrm{KL}(\tilde{P}_{\theta_t,\alpha_t}(\mathbf{x},\mathbf{z})\| P_\theta (\mathbf{x},\mathbf{z}))$ . It is equivalent to the marginal version as
$$
\operatorname {K L} \left(\tilde {P} _ {\theta_ {t}, \alpha_ {t}} (\mathbf {x}, \mathbf {z}) \| P _ {\theta} (\mathbf {x}, \mathbf {z})\right) = \operatorname {K L} \left(\mathcal {T} _ {\alpha_ {t}} ^ {\mathbf {x}} p _ {\theta_ {t}} (\mathbf {x}) \| p _ {\theta} (\mathbf {x})\right) + \mathbb {E} _ {\mathcal {T} _ {\alpha_ {t}} ^ {\mathbf {x}} p _ {\theta_ {t}} (\mathbf {x})} \left[ \operatorname {K L} \left(p _ {\theta_ {t}} (\mathbf {z} | \mathbf {x}) \| p _ {\theta} (\mathbf {z} | \mathbf {x})\right) \right] \tag {13}
$$
With  $s\to 0$  and  $\tau \rightarrow \infty$ ,  $\mathcal{T}_{\alpha_t}^{\mathbf{x}}p_{\theta_t}(\mathbf{x})\rightarrow \pi_{\alpha_t}(\mathbf{x})$  (see Eqn.3), our second KL term approximates (in Eqn.10) the MLE objective on  $\pi_{\alpha_t}$  for generator,
$$
\mathrm {K L} \left(\pi_ {\alpha_ {t}} (\mathbf {x}) \| p _ {\theta} (\mathbf {x})\right) + \mathbb {E} _ {\pi_ {\alpha_ {t}} (\mathbf {x})} \left[ \mathrm {K L} \left(p _ {\theta_ {t}} (\mathbf {z} | \mathbf {x}) \| p _ {\theta} (\mathbf {z} | \mathbf {x})\right) \right] \tag {14}
$$
Such surrogate in joint density again upper-bounds (i.e., majorizes) the true MLE objective on generated samples, i.e.,  $\mathrm{KL}(\pi_{\alpha_t}(\mathbf{x})\| p_\theta (\mathbf{x}))$  , and thus the generator  $p_{\theta}(\mathbf{x})$  updates to be closer to the EBM  $\pi_{\alpha}(\mathbf{x})$  at the current iteration.
Connection to variational learning. Without MCMC inference, the generator model can be learned with inference model to match the empirical data distribution, i.e.,  $\min_{\theta}\mathrm{KL}(Q_{\phi_t}(\mathbf{x},\mathbf{z})\| P_\theta (\mathbf{x},\mathbf{z}))$  which underlies VAEs [21, 32, 26, 37]. Compared to Eqn.11, VAEs seek to minimize  $\mathrm{KL}(p_{\mathrm{d}}(\mathbf{x})\| p_{\theta}(\mathbf{x})) + \mathbb{E}_{p_{\mathrm{d}}(\mathbf{x})}[\mathrm{KL}(q_{\phi_t}(\mathbf{z}|\mathbf{x})\| p_{\theta}(\mathbf{z}|\mathbf{x}))]$  where  $q_{\phi}(\mathbf{z}|\mathbf{x})$  is assumed to be Gaussian distributed which has limited capacity for generator learning.
Connection to cooperative learning. Cooperative learning schemes [42, 43] share similar EBM training procedures but can be fundamentally different in generator learning. The generators are learned through  $\min_{\theta} \mathrm{KL}(\Pi_{\alpha_t, \phi_t}(\mathbf{x}, \mathbf{z}) \| P_\theta(\mathbf{x}, \mathbf{z}))$  [43] or  $\min_{\theta} \mathrm{KL}(\tilde{P}_{\theta_t, \alpha_t}(\mathbf{x}, \mathbf{z}) \| P_\theta(\mathbf{x}, \mathbf{z}))$  [42], however, generators have no access to the empirical observations which lead to biased and sub-optimal generator models.
3.4 Learning Inference Model via Dual-MCMC Teaching
The inference model  $q_{\phi}(\mathbf{z}|\mathbf{x})$  serves as a key component for generator learning which will in turn facilitate the EBM training. In this paper, the inference model is learned by minimizing the KL divergences  $D_{q}(\phi)$  as
$$
\min  _ {\phi} D _ {q} (\phi) = \min  _ {\phi} \mathrm {K L} \left(\tilde {Q} _ {\phi_ {t}, \theta_ {t}} (\mathbf {x}, \mathbf {z}) \| Q _ {\phi} (\mathbf {x}, \mathbf {z})\right) + \mathrm {K L} \left(\tilde {P} _ {\theta_ {t}, \alpha_ {t}} (\mathbf {x}, \mathbf {z}) \| \Pi_ {\alpha , \phi} (\mathbf {x}, \mathbf {z})\right)
$$
$$
\text {w h e r e} - \frac {\partial}{\partial \phi} D _ {q} (\phi) = \mathbb {E} _ {p _ {\mathrm {d}} (\mathbf {x}) \mathcal {T} _ {\theta_ {t}} ^ {\mathbf {z}}} q _ {\phi_ {t}} (\mathbf {z} | \mathbf {x}) [ \frac {\partial}{\partial \phi} \log q _ {\phi} (\mathbf {z} | \mathbf {x}) ] + \mathbb {E} _ {\mathcal {T} _ {\alpha_ {t}} ^ {\mathbf {x}}} p _ {\theta_ {t}} (\mathbf {x}, \mathbf {z}) [ \frac {\partial}{\partial \phi} \log q _ {\phi} (\mathbf {z} | \mathbf {x}) ] \tag {15}
$$
The revised data density  $\tilde{Q}_{\phi_t,\theta_t}$  teaches the inference model on empirical data observations for better real data inference through the first KL term, and the revised generator density  $\tilde{P}_{\theta_t,\alpha_t}$  teaches the inference model for better generated sample inference through the second KL term.
Real data inference. Optimizing the first term  $\mathrm{KL}(\tilde{Q}_{\phi_t,\theta_t}(\mathbf{x},\mathbf{z})\| Q_\phi (\mathbf{x},\mathbf{z}))$  in Eqn.15 is equivalent to  $\min_{\phi}\mathbb{E}_{p_{\mathrm{d}}(\mathbf{x})}[\mathrm{KL}(\mathcal{T}_{\theta_t}^{\mathbf{z}}q_{\phi_t}(\mathbf{z}|\mathbf{x})\| q_\phi (\mathbf{z}|\mathbf{x}))]$ . Given the long-run optimality condition of the MCMC transition  $\mathcal{T}_{\theta_t}^{\mathbf{z}}q_{\phi_t}(\mathbf{z}|\mathbf{x})\to p_{\theta_t}(\mathbf{z}|\mathbf{x})$ , our first term  $\mathrm{KL}(\tilde{Q}_{\phi_t,\theta_t}(\mathbf{x},\mathbf{z})\| Q_\phi (\mathbf{x},\mathbf{z}))$  tends to learn  $q_{\phi}(\mathbf{z}|\mathbf{x})$  by minimizing the  $\mathbb{E}_{p_{\mathrm{d}}(\mathbf{x})}[\mathrm{KL}(p_{\theta_t}(\mathbf{z}|\mathbf{x})\| q_\phi (\mathbf{z}|\mathbf{x}))]$ . The inference model is learned to match the true generator posterior  $p_{\theta}(\mathbf{z}|\mathbf{x})$  on real observations in the current learning step. Specifically, latent samples are initialized from current  $q_{\phi_t}(\mathbf{z}|\mathbf{x})$ , and the generator-guided MCMC revision  $\mathcal{T}_{\theta_t}^{\mathbf{z}}q_{\phi_t}(\mathbf{z}|\mathbf{x})$  is then performed to obtain the revised latent samples. The inference model is updated to amortize the MCMC and to absorb such sample revision. The MCMC revision not only drives the evolution of the latent samples, but also drives the evolution of the inference model.
Generated sample inference. Optimizing the second term  $\mathrm{KL}(\tilde{P}_{\theta_t,\alpha_t}(\mathbf{x},\mathbf{z})\| \Pi_{\alpha ,\phi}(\mathbf{x},\mathbf{z}))$  in Eqn.15 is equivalent to  $\min_{\phi}\mathbb{E}_{\mathcal{T}_{\alpha_t}^{\times}p_{\theta_t}(\mathbf{x})}[\mathrm{KL}(p_{\theta_t}(\mathbf{z}|\mathbf{x})\| q_\phi (\mathbf{z}|\mathbf{x}))]$  which tends to minimizing  $\mathbb{E}_{\pi_{\alpha_t}(\mathbf{x})}[\mathrm{KL}(p_{\theta_t}(\mathbf{z}|\mathbf{x})\| q_\phi (\mathbf{z}|\mathbf{x}))]$  given long-run optimality condition (i.e.,  $\mathcal{T}_{\alpha_t}^{\mathbf{x}}p_{\theta_t}(\mathbf{x})\to \pi_{\alpha_t}(\mathbf{x}))$  The inference model is learned to match the true generator posterior  $p_{\theta}(\mathbf{z}|\mathbf{x})$  on generated samples from EBM in the current learning step. Noted that both the generated sample and its latent factor can be readily available where the latent factor  $\mathbf{z}$  is drawn from prior distribution  $p(\mathbf{z})$  , which is assumed to be unit Gaussian, and the generated sample is obtained directly from generator.
Energy-based model. The EBM is flexible and theoretically appealing with various approaches to learning, such as the noise-contrastive estimation (NCE) [1, 38] and the diffusion approach [8]. Most existing works learn the EBM via MLE [29, 30, 5, 6, 39], which typically involves MCMC sampling, while some advance [15, 16, 12, 17, 24] propose to amortize MCMC sampling with the generator model and learn EBM in a close-formed, variational learning scheme. Instead, [42, 43] recruit ancestral Langevin dynamics with the generator model being the initializer model. In this paper, we propose a joint framework where the generator model matches both the EBM and empirical data distribution through dual-MCMC teaching to better benefit the EBM sampling and learning.
Generator model. In recent years, the success of generator models has given rise to various ways of learning methods. Generative adversarial network (GAN) [10, 19, 27, 20] jointly trains the generator model with a discriminator, while VAE [21, 32, 36, 9] is trained with an inference model (or encoder) approximating the generator posterior. Without the inference model, [14, 31] instead utilize MCMC sampling to sample from the generator posterior. Our work differs from theirs by employing the MCMC inference based on informative initialization from the inference model, and we aim to learn the generator model to facilitate effective learning of the EBM.
In this section, we address the following questions: (1) Can our method learn an EBM with high-quality synthesis? (2) Can both the complementary generator and inference model successfully match their MCMC-revised samples? and (3) What is the influence of the inference model and generator model? We refer to implementation details and additional experiments in Appendix.
5.1 Image Modelling
We first evaluate the EBM in image data modelling. Both the generator model and EBM are learned to match empirical data distribution, and if the generator model is well-trained, it can serve as an informative initializer model, making the EBM sampling easier. As a result, the EBM should be capable of generating realistic image synthesis. For evaluation, we generate images from the EBM by obtaining  $\mathbf{x}_0$  from the generator and running Langevin dynamics with  $\mathbf{x}_0$  being the initial points.
Table 1: FID and IS on CIFAR-10 and CelebA-64.
Table 2: FID on CelebA-HQ-256 and LSUNChurch-64.
Methods	CIFAR-10	CelebA-64
IS (↑)	FID (↓)	FID (↓)
Ours	8.55	9.26	5.15
Cooperative EBM [42]	6.55	33.61	16.65
Amortized EBM [43]	6.65	-	-
Divergence Triangle [15]	7.23	30.10	18.21
No MCMC EBM [12]	-	27.5	-
Short-run EBM [29]	6.21	-	23.02
IGEBM [5]	6.78	38.2	-
ImprovedCD EBM [6]	7.85	25.1	-
Diffusion EBM [8]	8.30	9.58	5.98
VAEBM [39]	8.43	12.19	5.31
NCP-VAE[1]	-	24.08	5.25
SNGAN [27]	8.22	21.7	6.1
StyleGANv2 w/o ADA[20]	8.99	9.9	2.32
NCSN[34]	8.87	25.32	25.30
DDPM[18]	9.46	3.17	3.93
Methods	CelebA-HQ-256	LSUN-Church-64
Ours	15.89	4.56
Diffusion EBM [8]	-	7.02
VAEBM [39]	20.38	13.51
NCP-VAE[1]	27.79	-
GLOW[22]	68.93	59.35
PGGAN [19]	21.7	6.1
Figure 1: Image synthesis on CIFAR-10.
We benchmark our method on standard datasets such as CIFAR-10 [23] and CelebA-64 [25], as well as challenging high-resolution CelebA-HQ-256 [19] and large-scale LSUN-Church-64 [44]. We consider the baseline models, including Divergence Triangle [15], No MCMC EBM [12], Cooperative EBM [42], and Amortized EBM [43], as well as modern advanced generative models, including other EBMs [29, 6, 8, 39, 1], GANs [27, 20] and score-based models [34, 18]. We recruit Fréchet Inception Distance (FID) and Inception Score (IS) metrics to evaluate the quality of image synthesis. Results are reported in Tab.1 and Tab.2 where our EBM shows the capability of generating realistic image synthesis and renders competitive performance even compared to GANs and score-based models.
5.2 MCMC Revision
The complementary generator and inference model are learned to match their MCMC-revised samples and thus can serve as informative initializers. We demonstrate that both the generator and inference model can successfully catch up with the MCMC revision. We train our model on CelebA-64 using Langevin steps  $k_{\mathbf{x}} = 30$  for the MCMC revision on  $\mathbf{x}$  and  $k_{\mathbf{z}} = 10$  for the MCMC revision on  $\mathbf{z}$ .
Generator model. If the generator model captures different modes of the EBM, the MCMC revision on  $\mathbf{x}$  should only need to search around the local mode and correct pixel-level details. To examine the generator model, we visualize the Langevin transition by drawing  $\mathbf{x}_i$  for every three steps from generated samples  $\mathbf{x}_0$  to MCMC-revised samples  $\mathbf{x}_k$ . As shown in Fig.2, only minor changes can be observed during the transition, suggesting that the generator has matched the EBM-guided MCMC revision. By measuring FID of  $\mathbf{x}_0$  and  $\mathbf{x}_k$ , it still improves from 5.94 to 5.15, which indicates pixel-level refinements.
Inference model. We then show the Langevin transition on  $\mathbf{z}$ . For visualization, latent codes are mapped to data space via the generation network. We draw  $\mathbf{z}_i$  for each step and show corresponding images in Fig.2, where the inference model also catches up with the generator-guided explaining-away MCMC inference, leading to faithful reconstruction as a result.
Figure 2: Left top: MCMC revision on x. The leftmost images are sampled from the generator model, and the rightmost images are at the final step of the EBM-guided MCMC sampling. Left bottom: Energy profile over steps. Right top: MCMC revision on z. The leftmost images are reconstructed by latent codes inferred from the inference model, and the rightmost images are reconstructed by latent codes at the final step of the generator-guided MCMC inference. Right bottom: Mean Squared Error (MSE) over steps.
5.3 Analysis of Inference Model
The inference model serves as an initializer model for generator learning which in turn facilitates the EBM sampling and learning. To demonstrate the benefit of the inference model, we adopt noise-initialized Langevin dynamics for generator posterior sampling and compare with the Langevin dynamics initialized by the inference model.
Figure 3:  $\mathrm{MSE}(\downarrow)$ ,  $\mathrm{SSIM}(\uparrow)$  and PSNR  $(\uparrow)$ .
Table 3: Comparison of MSE.  $\mathbf{Inf} + \mathbf{L} = \mathbf{10}$  denotes using Langevin dynamics initialized by inference model for  $k_{\mathbf{z}} = 10$  steps.
Methods	CIFAR-10	CelebA-64
VAE[21]	0.0341	0.0438
WAE[36]	0.0291	0.0237
RAE[9]	0.0231	0.0246
ABP[14]	0.0183	0.0277
SR-ABP[31]	0.0262	0.0330
Cooperative EBM[42]	0.0271	0.0387
Divergence Triangle[15]	0.0237	0.0281
Ours (Inf)	0.0214	0.0227
Ours (Inf+L=10)	0.0072	0.0164
Specifically, we conduct noise-initialized Langevin dynamics with increasing steps from  $k_{\mathrm{z}} = 10$  to  $k_{\mathrm{z}} = 30$ , and compare with the Langevin dynamics using only  $k_{\mathrm{z}} = 10$  steps but is initialized by the inference model. We recruit MSE, Peak Signal-to-Noise Ratio (PSNR), and Structural SIMilarity (SSIM) to measure the inference accuracy of reconstruction and present the results in Fig.3. As the Langevin steps increase, the inference becomes more accurate (lower MSE, higher PSNR, and SSIM), however, it is still less accurate than the proposed method (L=30 vs. Inf+L=10). This result highlights the inference model in our framework. We then compare with other models that also characterize an inferential mechanism, such as VAE [21], Wasserstein auto-encoders (WAE) [36], RAE [9], Alternating Back-propagation (ABP) [14], and Short-run ABP (SR-ABP) [31]. As shown in Tab.3, our model can render superior performance with faithful reconstruction.
5.4 Analysis of Generator Model
With the generator model being the initializer for EBM sampling, exploring the energy landscape
should become easier by first traversing the low-dimensional latent space. We intend to examine if our generator model can deliver smooth interpolation on the latent space, thus making a smooth transition in the data space. We employ linear interpolation among latent space, i.e.,  $\tilde{\mathbf{z}} = (1 - \alpha)\cdot \mathbf{z}_1 + \alpha \cdot \mathbf{z}_2$ , and consider two scenarios, such as the image synthesis and image reconstruction. As shown in Fig.4, our generator model is capable of smooth interpolation for both scenarios, which suggests its effectiveness in exploring the energy landscape.
Figure 4: Linear interpolation on latent space. The top and bottom three rows indicate image generation and reconstruction, respectively.
6 Ablation Studies
MCMC steps of  $\mathcal{T}_{\theta}^{\mathbf{z}}$ . We analyze the impact of the inference accuracy in our framework by increasing the Langevin steps of  $\mathcal{T}_{\theta}^{\mathbf{z}}q_{\phi}(\mathbf{z}|\mathbf{x})$ . With an inference model initializing the MCMC posterior sampling, further increasing the MCMC steps should deliver more accurate inference and thus benefit the generator and EBM for better performance. Thus, we compute the FID, MSE, and wall-clock training time (seconds / per iteration) in Tab.4. It can be seen that increasing MCMC steps from 10 to 30 indeed slightly improves the generation quality and inference accuracy but requires more training time. We thus report the result of  $Inf + L = 10$  in Tab.1 and Tab.3.
MCMC steps of  $\mathcal{T}_{\alpha}^{\mathbf{x}}$ . Then, we discuss the impact of the Langevin steps of  $\mathcal{T}_{\alpha}^{\mathbf{x}}$ . Increasing the MCMC steps of  $\mathcal{T}_{\mathbf{x}}$  should explore the energy landscape more effectively and render better performance in the generation. In Tab.5, starting MCMC steps from 10 to 30, our model exhibits largely improved performance in generation quality but only minor improvement even when we use  $L = 50$  steps. Thus, we report  $L = 30$  steps in Tab.1.
Table 4: Increasing MCMC steps of  ${\mathcal{T}}_{\theta }^{\mathbf{z}}$  .
L=10	L=30	Inf+L=10	Inf+L=30
FID	17.32	14.51	9.26	9.18
MSE	0.0214	0.0164	0.0072	0.0068
Time (s)	1.576	2.034	1.594	2.112
Table 5: Increasing MCMC steps of  ${\mathcal{T}}_{\alpha }^{\mathrm{x}}$  .
L=10	L=20	L=30	L=50
FID	14.78	11.51	9.26	9.07
Time (s)	0.861	1.241	1.594	2.454
We present a joint learning scheme that can effectively learn the EBM by interweaving the maximum likelihood learning of the EBM, generator, and inference model through dual-MCMC teaching. The generator and inference model are learned to initialize MCMC sampling of EBM and generator posterior, respectively, while these EBM-guided MCMC sampling and generator-guided MCMC inference, in turn, serve as two MCMC revision processes that are capable of teaching the generator and inference model. This work may share the limitation with other MCMC-based methods in terms of the computational cost, but we expect to impact the active research of learning the EBMs.
[40] Zhisheng Xiao, Qing Yan, and Yali Amit. Ebms trained with maximum likelihood are generator models trained with a self-adversarial loss. arXiv preprint arXiv:2102.11757, 2021. 5
[41] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pages 2635-2644. PMLR, 2016. 2
[42] Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. IEEE transactions on pattern analysis and machine intelligence, 42(1):27-45, 2018. 2, 3, 6, 7, 8
[43] Jianwen Xie, Zilong Zheng, and Ping Li. Learning energy-based model with variational auto-encoder as amortized sampler. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10441–10451, 2021. 1, 2, 3, 6, 7
[44] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015. 7