SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[2] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of International Conference on Machine Learning (ICML), 2014.
[3] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
[4] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR, 2015.
[5] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of International Conference on Machine Learning (ICML), 2015.
[6] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 2019.
[7] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[8] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[9] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2020.
[10] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[11] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761, 2020.
[12] Peiyu Yu, Sirui Xie, Xiaojian Ma, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Unsupervised foreground extraction via deep region competition. Advances in Neural Information Processing Systems, 2021.
[13] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[14] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[15] Peiyu Yu, Sirui Xie, Xiaojian Ma, Baoxiong Jia, Bo Pang, Ruigi Gao, Yixin Zhu, Song-Chun Zhu, and Ying Nian Wu. Latent diffusion energy-based model for interpretable text modeling. arXiv preprint arXiv:2206.05895, 2022.
[16] Jun-Yan Zhu, Philipp Krahenbuhl, Eli Shechtman, and Alexei A Efros. Generative visual manipulation on the natural image manifold. In Proceedings of European Conference on Computer Vision (ECCV), 2016.
[73] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[74] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[75] Alexander Li and Deepak Pathak. Functional regularization for reinforcement learning via learned fourier features. Advances in Neural Information Processing Systems, 2021.
[76] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.
[77] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[78] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[79] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[80] Shohei Taniguchi, Yusuke Iwasawa, Wataru Kumagai, and Yutaka Matsuo. Langevin autoencoders for learning deep latent variable models. Advances in Neural Information Processing Systems, 35:13277-13289, 2022.
A Theoretical Discussion
A.1 Monotonically Decreasing KLD
We state in the main text that  $\mathcal{D}[q_{\phi_k}||\pi ]\leqslant \mathcal{D}[q_{\phi_{k - 1}}||\pi ]$  , where  $\pi$  is the stationary distribution. To show this, we first provide a proof of  $\mathcal{D}[\pi_{t + T}||\pi ]\leqslant \mathcal{D}[\pi_t||\pi ]$  , where  $\pi_t$  and  $\pi_{t + T}$  are the distributions of  $z$  at  $t$  -th and  $(t + T)$  -th iteration, respectively. This is a known result from [39], and we include it here for completeness.
$$
\begin{array}{l} \mathcal {D} [ \pi_ {t} | | \pi ] = \mathbb {E} _ {\pi_ {t} (\boldsymbol {z} _ {t})} \left[ \log \frac {\pi_ {t} (\boldsymbol {z} _ {t})}{\pi (\boldsymbol {z} _ {t})} \right] = \mathbb {E} _ {\pi_ {t} (\boldsymbol {z} _ {t}), \mathcal {K} (\boldsymbol {z} _ {t + 1} | \boldsymbol {z} _ {t})} \left[ \log \frac {\pi_ {t} (\boldsymbol {z} _ {t}) \mathcal {K} (\boldsymbol {z} _ {t + 1} | \boldsymbol {z} _ {t})}{\pi (\boldsymbol {z} _ {t}) \mathcal {K} (\boldsymbol {z} _ {t + 1} | \boldsymbol {z} _ {t})} \right] \\ \stackrel {(i)} {=} \mathbb {E} _ {\pi_ {t + 1} (\boldsymbol {z} _ {t + 1}), \mathcal {K} ^ {\prime} _ {\pi_ {t + 1}} (\boldsymbol {z} _ {t} | z _ {t + 1})} \left[ \log \frac {\mathcal {K} ^ {\prime} _ {\pi_ {t + 1}} (\boldsymbol {z} _ {t} | z _ {t + 1}) \pi_ {t + 1} (\boldsymbol {z} _ {t + 1})}{\mathcal {K} ^ {\prime} _ {\pi} (\boldsymbol {z} _ {t} | z _ {t + 1}) \pi (\boldsymbol {z} _ {t + 1})} \right] \tag {10} \\ = \mathcal {D} \left[ \pi_ {t + 1} | | \pi \right] + \mathbb {E} _ {\pi_ {t + 1} \left(\boldsymbol {z} _ {t + 1}\right)} \mathcal {D} \left[ \mathcal {K} _ {\pi_ {t + 1}} ^ {\prime} \left(\boldsymbol {z} _ {t} \mid \boldsymbol {z} _ {t + 1}\right) \right| | \mathcal {K} _ {\pi} ^ {\prime} \left(\boldsymbol {z} _ {t} \mid \boldsymbol {z} _ {t + 1}\right) ] \\ \stackrel {(i i)} {\geqslant} \mathcal {D} \left[ \pi_ {t + 1} | | \pi \right], \\ \end{array}
$$
where we denote the forward-time transition kernel as  $\mathcal{K}$  and the reverse-time kernel as  $\mathcal{K}'$ . (i) holds because we are just re-factorizing the joint density of  $[z_t, z_{t+1}]: \pi_t(z_t) \mathcal{K}(z_{t+1}|z_t) = \mathcal{K}'_{\pi_{t+1}}(z_t|z_{t+1}) \pi_{t+1}(z_{t+1})$  and  $\pi(z_t) \mathcal{K}(z_{t+1}|z_t) = \mathcal{K}'_{\pi}(z_t|z_{t+1}) \pi(z_{t+1})$ . (ii) holds because the KLD is non-negative. We can see that  $\mathcal{D}[\pi_{t+T}||\pi] \leqslant \mathcal{D}[\pi_t||\pi]$  is a direct result from Eq. (10), and that  $\pi_t \to \pi$  as  $t \to \infty$  under proper conditions [32].
In the main text, we describe the update rule of the sampler  $q_{\phi}$  as follows:
$$
q _ {\phi_ {k}} \leftarrow \underset {q _ {\phi} \in \mathcal {Q}} {\arg \min } \mathcal {D} \left[ q _ {\phi_ {k - 1}, T} \| q _ {\phi} \right], q _ {\phi_ {k - 1}, T} := \mathcal {K} _ {T} q _ {\phi_ {k - 1}}, q _ {\phi_ {0}} \approx \pi_ {0}, k = 0, \dots , K - 1. \tag {11}
$$
In the ideal case, we can assume that the objective in Eq. (6) is properly optimized and that  $\{q_{\phi}\}$  is expressive enough to parameterize each  $q_{\phi_{k - 1},T}$ . With  $q_{\phi_0} \approx \pi_0$  and  $q_{\phi_k} \approx \mathcal{K}_T q_{\phi_{k - 1}}$ , we can conclude that  $\mathcal{D}[q_{\phi_k}||\pi ] \leqslant \mathcal{D}[q_{\phi_{k - 1}}||\pi ]$  for each  $k = 1,\dots,K$  according to Eq. (10). We will discuss in the following section the scenario where we apply gradient-based methods to minimize  $\mathcal{D}[q_{\phi_{k - 1},T}||q_{\phi}]$  and approximate Eq. (6) for the update from  $q_{\phi_{k - 1}}$  to  $q_{\phi_k}$ .
A.2 Discussion about Diffusion-Based Amortization
We can see that the statement in Appendix A.1 holds when  $q_{\phi_k}$  is a close approximation of  $q_T \coloneqq \mathcal{K}_T q_{\phi_{k-1}}$ . This motivates our choice of employing DDPM to amortize the LD transition, considering its capability of close approximation to the given distribution  $\{q_T\}$ . Based on the derivation of DDPM learning objective in [1], we know that
$$
\begin{array}{l} \underset {q _ {\phi}} {\arg \min } \mathcal {D} [ q _ {T} | | q _ {\phi} ] = \underset {q _ {\phi}} {\arg \min } - \mathcal {H} (q _ {T}) + \mathcal {H} (q _ {T}, q _ {\phi}) = \underset {q _ {\phi}} {\arg \min } - \mathbb {E} _ {q _ {T}} \left[ \log q _ {\phi} \right] \\ \leqslant \underset {q _ {\phi}} {\arg \min } \mathbb {E} _ {\boldsymbol {\epsilon}, \lambda} \left[ \| \boldsymbol {\epsilon} _ {\phi} (\boldsymbol {z} _ {\lambda}) - \boldsymbol {\epsilon} \| _ {2} ^ {2} \right] \approx \underset {q _ {\phi}} {\arg \min } \frac {1}{N} \sum_ {j = 1} ^ {N} \left[ \| \boldsymbol {\epsilon} _ {\phi} (\boldsymbol {z} _ {j, \lambda_ {j}}) - \boldsymbol {\epsilon} _ {j} \| _ {2} ^ {2} \right], \tag {12} \\ \end{array}
$$
where  $z_{\lambda}$  is drawn from  $q(z_{\lambda}|z_0) = \mathcal{N}(z_{\lambda};\beta_{\lambda}z_0,\sigma_{\lambda}^2\mathbf{I}_d)$ .  $z_0 \sim q_T$ .  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I}_d)$  and  $\lambda$  is drawn from a distribution of log noise-to-signal ratio  $p(\lambda)$ . In practice, we use Monte-Carlo average to approximate the objective and employ a gradient-based update rule for  $q_{\phi}$ :
$$
\phi_ {k - 1} ^ {(i + 1)} \leftarrow \phi_ {k - 1} ^ {(i)} - \eta \nabla_ {\phi} \frac {1}{N} \sum_ {j = 1} ^ {N} \left[ \| \epsilon_ {\phi} \left(z _ {j, \lambda_ {j}}\right) \epsilon_ {j} \| _ {2} ^ {2} \right], \phi_ {k} ^ {(0)} \leftarrow \phi_ {k - 1} ^ {(M)}, i = 0, 1, \dots , M - 1. \tag {13}
$$
This can be viewed as a M-estimation of  $\phi$ . Recall that  $q_{T} = \mathcal{K}_{T}q_{\phi_{k - 1}}$ , we can construct  $\phi_{k} = \left(\phi_{k - 1},\tilde{\phi}\right)$  to minimize the KLD, where  $\tilde{\phi}$  models the transition kernel  $\mathcal{K}_T$ . Therefore, initializing  $q_{\phi}$  to be optimized with  $q_{\phi_{k - 1}}$ , we are effectively maximizing  $\mathcal{L}(\tilde{\phi}) = \frac{1}{N}\sum_{j = 1}^{N}\log p_{\tilde{\phi}}(\hat{z}_j|\boldsymbol {z}_j)$ ,
where  $\{z_{j}\}$  are from  $q_{\phi_{k - 1}}$  and  $\{\hat{z}_j\}$  are from  $\mathcal{K}_T q_{\phi_{k - 1}}$ . Let  $\tilde{\phi}$  be the result of M-estimation and  $\tilde{\phi}^{*}$  be the true target parameter. Then based on the derivation in [25], asymptotically we have
$$
\sqrt {N} \left(\tilde {\phi} - \tilde {\phi} ^ {*}\right)\rightarrow \mathcal {N} \left(0, \mathcal {I} \left(\tilde {\phi} ^ {*}\right) ^ {- 1}\right), \tag {14}
$$
where  $\mathcal{I}\left(\tilde{\phi}^{*}\right) = \mathbb{E}_{\hat{z},z}\left[-\nabla^{2}\log p_{\tilde{\phi}}(\hat{z} |z)\right]$  is the Fisher information matrix. This interpretation tells us that i) when the sample size  $N$  is large, the estimation  $\tilde{\phi}$  is asymptotically unbiased, and ii) if we want to obtain the estimation  $\tilde{\phi}$  with a few gradient-based updates, then the eigenvalues of the Fisher information matrix would be relatively small but non-zero. ii) suggests that  $\mathcal{K}_Tq_{\phi_{k - 1}}$  should be significantly different from  $q_{\phi_{k - 1}}$ , which is confirmed by [38] and our preliminary experiments, but it should not be too far away because that would require more gradient-based updates. We find that setting  $T = 30$  and  $M = 6$  works well in the experiments.
A.3 Further Discussion about the Learning Algorithm
For completeness, we first derive the learning gradients for updating  $\theta$ .
$$
\begin{array}{l} \nabla_ {\boldsymbol {\theta}} \log p _ {\boldsymbol {\theta}} (\boldsymbol {x}) = \frac {1}{p _ {\boldsymbol {\theta}} (\boldsymbol {x})} \nabla_ {\boldsymbol {\theta}} \int_ {\boldsymbol {z}} p _ {\boldsymbol {\theta}} (\boldsymbol {x}, \boldsymbol {z}) d \boldsymbol {z} = \int_ {\boldsymbol {z}} \frac {p _ {\boldsymbol {\theta}} (\boldsymbol {x} , \boldsymbol {z})}{p _ {\boldsymbol {\theta}} (\boldsymbol {x})} \nabla_ {\boldsymbol {\theta}} \log p _ {\boldsymbol {\theta}} (\boldsymbol {x}, \boldsymbol {z}) d \boldsymbol {z} \\ = \mathbb {E} _ {p _ {\boldsymbol {\theta}} (\boldsymbol {z} | \boldsymbol {x})} \left[ \nabla_ {\boldsymbol {\theta}} \log p _ {\boldsymbol {\theta}} (\boldsymbol {z}, \boldsymbol {x}) \right] \\ = \left(\mathbb {E} _ {p _ {\theta} (\boldsymbol {z} | \boldsymbol {x})} \left[ \nabla_ {\boldsymbol {\alpha}} \log p _ {\boldsymbol {\alpha}} (\boldsymbol {z}) \right], \mathbb {E} _ {p _ {\theta} (\boldsymbol {z} | \boldsymbol {x})} \left[ \nabla_ {\boldsymbol {\beta}} \log p _ {\boldsymbol {\beta}} (\boldsymbol {x} | \boldsymbol {z}) \right]\right) \tag {15} \\ = (\underbrace {\mathbb {E} _ {p _ {\boldsymbol {\theta}} (\boldsymbol {z} | \boldsymbol {x})} \left[ \nabla_ {\boldsymbol {\alpha}} f _ {\boldsymbol {\alpha}} (\boldsymbol {z}) \right]} _ {\delta_ {\boldsymbol {\alpha}} (\boldsymbol {x})} - \underbrace {\mathbb {E} _ {p _ {\boldsymbol {\alpha}} (\boldsymbol {z})} \left[ \nabla_ {\boldsymbol {\alpha}} f _ {\boldsymbol {\alpha}} (\boldsymbol {z}) \right]} _ {\delta_ {\boldsymbol {\beta}} (\boldsymbol {x})}, \underbrace {\mathbb {E} _ {p _ {\boldsymbol {\theta}} (\boldsymbol {z} | \boldsymbol {x})} \left[ \nabla_ {\boldsymbol {\beta}} \log p _ {\boldsymbol {\beta}} (\boldsymbol {x} | \boldsymbol {z}) \right]} _ {\delta_ {\boldsymbol {\beta}} (\boldsymbol {x})}). \\ \end{array}
$$
We can see that  $\theta$  is estimated in a MLE-by-EM style. The learning gradient is the same as that of directly maximizing the observed data likelihood, while we need to approximate the expectations in Eq. (15). Estimating the expectations is like the E-step, and update  $\theta$  with Eq. (15) is like the M-step in the EM algorithm. The proposed diffusion-based amortization brings better estimation of the expectations in the E-step, and incorporate the feedback from the M-step by running prior and posterior sampling LD as follows
$$
\begin{array}{l} \boldsymbol {z} _ {t + 1} = \boldsymbol {z} _ {t} + \frac {s ^ {2}}{2} \nabla_ {\boldsymbol {z} _ {t}} \underbrace {\left(f _ {\boldsymbol {\alpha}} (\boldsymbol {z} _ {t}) - \frac {1}{2} \| \boldsymbol {z} _ {t} \| _ {2} ^ {2}\right)} _ {\log p _ {\boldsymbol {\alpha}} (\boldsymbol {z} _ {t})} + s \boldsymbol {w} _ {t}, \\ \boldsymbol {z} _ {t + 1} = \boldsymbol {z} _ {t} + \frac {s ^ {2}}{2} \nabla_ {\boldsymbol {z} _ {t}} \underbrace {\left(- \frac {\| \boldsymbol {x} - g _ {\beta} (\boldsymbol {z} _ {t}) \| _ {2} ^ {2}}{2 \sigma^ {2}} + f _ {\boldsymbol {\alpha}} (\boldsymbol {z} _ {t}) - \frac {1}{2} \| \boldsymbol {z} _ {t} \| _ {2} ^ {2}\right)} _ {\log p _ {\theta} (\boldsymbol {z} | \boldsymbol {x}) = \log p _ {\theta} (\boldsymbol {x}, \boldsymbol {z}) + C} + s \boldsymbol {w} _ {t}, \tag {16} \\ \end{array}
$$
to obtain training data. Here  $t = 0,1,\dots,T$ .  $z_0\sim q_\phi (z|x)$  for posterior sampling and  $z_{0}\sim q_{\phi}(z)$  for prior sampling.  $\pmb{w}_t\sim \mathcal{N}(\mathbf{0},\mathbf{I}_d)$ . Note that we plug-in  $p_{\theta}(\pmb {x},\pmb {z})$  for the target distribution of posterior sampling LD. This is because given the observed data  $\pmb{x}$ , by Bayes' rule we know that  $p_{\theta}(z|x)\propto p_{\theta}(x,z) = p_{\beta}(x|z)p_{\alpha}(z)$ . The whole learning iteration can be viewed as a variant of the variational EM algorithm [71].
B Network Architecture and Training Details
Architecture The energy score network  $f_{\alpha}$  uses a simple fully-connected structure throughout the experiments. We describe the architecture in details in Table S1. The generator network has a simple deconvolution structure similar to DCGAN [72] shown in Table S2. The denoising diffusion model is implemented by a light-weight MLP-based U-Net [73] structure (Table S3). The encoder network to embed the observed images has a fully convolutional structure [74], as shown in Table S4.
Table S1: Network structures of the energy score network. LReLU denotes the Leaky ReLU activation function. The slope in Leaky ReLU is set to 0.2. For SVHN and CelebA datasets, we use nz=100. For the CIFAR-10 and CelebA-HQ datasets, we use nz=128. We use nz=8 for anomaly detection on the MNIST dataset, and nz=7168 for GAN inversion. We use ndf=512 for GAN inversion and ndf=200 for the rest experiments.
Layers	Out Size
Input: z	nz ∈ {8, 100, 128, 7168}
Linear, LReLU	ndf ∈ {200, 512}
Linear, LReLU	ndf ∈ {200, 512}
Linear	1
Table S2: Network structures of the generator networks used for the SVHN, CelebA, CIFAR-10, CelebAHQ and MNIST (from top to bottom) datasets. For GAN inversion, we use the StyleGAN [10] structure as our generator network.  $\mathrm{ConvT}(n)$  indicates a transposed convolutional operation with  $n$  output channels. We use ngf=64 for the SVHN dataset and ngf=128 for the rest. LReLU indicates the Leaky-ReLU activation function. The slope in Leaky ReLU is set to be 0.2.
Layers	Out Size	Stride
Input: z	1x1x100	-
4x4 ConvT(ngf x 8), LReLU	4x4x(ngf x 8)	1
4x4 ConvT(ngf x 4), LReLU	8x8x(ngf x 4)	2
4x4 ConvT(ngf x 2), LReLU	16x16x(ngf x 2)	2
4x4 ConvT(3), Tanh	32x32x3	2
Layers	Out Size	Stride
Input: z	1x1x100	-
4x4 ConvT(ngf x 8), LReLU	4x4x(ngf x 8)	1
4x4 ConvT(ngf x 4), LReLU	8x8x(ngf x 4)	2
4x4 ConvT(ngf x 2) , LReLU	16x16x(ngf x 2)	2
4x4 ConvT(ngf x 1), LReLU	32x32x(ngf x 1)	2
4x4 ConvT(3), Tanh	64x64x3	2
Layers	Out Size	Stride
Input: z	1x1x128	-
8x8 ConvT(ngf x 8), LReLU	8x8x(ngf x 8)	1
4x4 ConvT(ngf x 4), LReLU	16x16x(ngf x 4)	2
4x4 ConvT(ngf x 2), LReLU	32x32x(ngf x 2)	2
3x3 ConvT(3), Tanh	32x32x3	1
Layers	Out Size	Stride
Input: z	1x1x128	-
4x4 ConvT(ngf x 16), LReLU	4x4x(ngf x 16)	1
4x4 ConvT(ngf x 8), LReLU	8x8x(ngf x 8)	2
4x4 ConvT(ngf x 4), LReLU	16x16x(ngf x 4)	2
4x4 ConvT(ngf x 4), LReLU	32x32x(ngf x 4)	2
4x4 ConvT(ngf x 2), LReLU	64x64x(ngf x 2)	2
4x4 ConvT(ngf x 1), LReLU	128x128x(ngfx1)	2
4x4 ConvT(3), Tanh	256x256x3	2
Layers	Out Size	Stride
Input: z	1x1x8	-
7x7 ConvT(ngf x 8), LReLU	7x7x(ngf x 8)	1
4x4 ConvT(ngf x 4), LReLU	14x14x(ngf x 4)	2
4x4 ConvT(ngf x 2), LReLU	28x28x(ngf x 2)	2
3x3 ConvT(1), Tanh	28x28x1	1
Table S3: Network structure of the denoising diffusion network. a) We use the sinusoidal embedding to embed the time index as in [25, 8]. b) We use the learned fourier feature module [75] to embed the input  $\mathbf{z}$ . c) The merged time embedding and context embedding is used to produce a pair of bias and scale terms to shift and scale the embedding of input  $\mathbf{z}$ . nz is the input dimension, as in Table S1. nemb is the dimension of image embedding as in Table S4.
Layers	Out size	Note
Time Embedding
Input: t	1	time index
Sin. emb.a	128
Linear, SiLU	128
Linear	128
Input Embedding
Input: z	nz
Fr. emb.b	2x(nz)	Fourier feature
Basic Block
Input: z, zctx, zt	nzf, nemb, 128	z, ctx. and t emb.
Cat, SiLU	nemb + 128	merge ctx. & t emb.
Linear, SiLU	nout
Linear	nout	Input emb.
Scale-shiftc	nout	scale-shift z emb.
Add z	nout	skip connection from z
Denoising Diffusion Network
Input: z, zctx, t	nz, nemb, 128	Input
Embedding	2x(nz), 128	Input & t emb.
Basic Block	128	Encoding
Basic Block	256
Basic Block	256
Basic Block	256	Intermediate
Basic Block	256	Cat & Decoding
Basic Block	128
Basic Block	nz	Output
Hyperparameters and training details As mentioned in the main text, for the posterior and prior DAMC samplers, we set the number of diffusion steps to 100. The number of iterations in Eq. (8) is set to  $M = 6$  for the experiments. The LD runs  $T = 30$  and  $T = 60$  iterations for posterior and prior updates during training with a step size of  $s = 0.1$ . For test time sampling from  $\mathcal{K}_{T,\boldsymbol{z}_i|\boldsymbol{x}_i}q_{\phi_k}(\boldsymbol{z}_i|\boldsymbol{x}_i)$ , we set  $T = 10$  for the additional LD. For test time prior sampling of LEBM with LD, we follow [22, 41] and set  $T = 100$ . To further stabilize the training procedure, we i) perform gradient clipping by setting the maximal gradient norm as 100, ii) use a separate target diffusion network which is the EMA of the current diffusion network to initialize the prior and posterior updates and iii) add noise-initialized prior samples for the prior updates. These set-ups are identical across different datasets.
The parameters of all the networks are initialized with the default pytorch methods [77]. We use the Adam optimizer [78] with  $\beta_{1} = 0.5$  and  $\beta_{2} = 0.999$  to train the generator network and the energy score network. We use the AdamW optimizer [79] with  $\beta_{1} = 0.5$ ,  $\beta_{2} = 0.999$  and weight Decay=1e-4 to train the diffusion network. The initial learning rates of the generator and diffusion networks are 2e-4, and 1e-4 for the energy score network. The learning rates are decayed with a factor of 0.99 every 1K training iterations, with a minimum learning rate of 1e-5. We run the experiments on a A6000 GPU with the batch size of 128. For GAN inversion, we reduce the batch size to 64. Training typically converges within 200K iterations on all the datasets.
Table S4: Network structures of the encoder networks used for the SVHN, CelebA, CIFAR-10, CelebA-HQ and MNIST (from top to bottom) datasets. For GAN inversion, the encoder network structure is the same as in [18]. Conv(n)Norm indicates a convolutional operation with  $n$  output channels followed by the Instance Normalization [76]. We use nif=64 and nemb=1024 for all the datasets. LReLU indicates the Leaky-ReLU activation function. The slope in Leaky ReLU is set to be 0.2.
Layers	Out Size	Stride
Input: x	32x32x3	-
3x3 Conv(nif x 1)Norm, LReLU	32x32x(nif x 1)	1
4x4 Conv(nif x 2)Norm, LReLU	16x16x(nif x 2)	2
4x4 Conv(nif x 4)Norm, LReLU	8x8x(nif x 4)	2
4x4 Conv(nif x 8)Norm, LReLU	4x4x(nif x 8)	2
4x4 Conv(nemb)Norm, LReLU	1x1x(nemb)	1
Layers	Out Size	Stride
Input: x	64x64x3	-
3x3 Conv(nif x 1)Norm, LReLU	64x64x(nif x 1)	1
4x4 Conv(nif x 2)Norm, LReLU	32x32x(nif x 2)	2
4x4 Conv(nif x 4)Norm, LReLU	16x16x(nif x 4)	2
4x4 Conv(nif x 8)Norm, LReLU	8x8x(nif x 8)	2
4x4 Conv(nif x 8)Norm, LReLU	4x4x(nif x 8)	2
4x4 Conv(nemb)Norm, LReLU	1x1x(nemb)	1
Layers	Out Size	Stride
Input: x	32x32x3	-
3x3 Conv(nif x 1)Norm, LReLU	32x32x(nif x 1)	1
4x4 Conv(nif x 2)Norm, LReLU	16x16x(nif x 2)	2
4x4 Conv(nif x 4), Norm, LReLU	8x8x(nif x 4)	2
4x4 Conv(nif x 8)Norm, LReLU	4x4x(nif x 8)	2
4x4 Conv(nemb)Norm, LReLU	1x1x(nemb)	1
Layers	Out Size	Stride
Input: x	256x256x3	-
3x3 Conv(nif x 1)Norm, LReLU	256x256x(nif x 1)	1
4x4 Conv(nif x 2)Norm, LReLU	128x128x(nif x 2)	2
4x4 Conv(nif x 4)Norm, LReLU	64x64x(nif x 4)	2
4x4 Conv(nif x 4)Norm, LReLU	32x32x(nif x 4)	2
4x4 Conv(nif x 8)Norm, LReLU	16x16x(nif x 8)	2
4x4 Conv(nif x 8)Norm, LReLU	8x8x(nif x 8)	2
4x4 Conv(nif x 8)Norm, LReLU	4x4x(nif x 8)	2
4x4 Conv(nemb)Norm, LReLU	1x1x(nemb)	1
Layers	Out Size	Stride
Input: x	28x28x3	-
3x3 Conv(nif x 1)Norm, LReLU	28x28x(nif x 1)	1
4x4 Conv(nif x 2)Norm, LReLU	14x14x(nif x 2)	2
4x4 Conv(nif x 4)Norm, LReLU	7x7x(nif x 4)	2
4x4 Conv(nif x 8)Norm, LReLU	3x3x(nif x 8)	2
3x3 Conv(nemb)Norm, LReLU	1x1x(nemb)	1
C Pytorch-style Pseudocode
We provide pytorch-style pseudocode to help understand the proposed method. We denote the generator network as G, the energy score network as E and the diffusion network as Q. The first page sketches the prior and posterior sampling process. The second page outlines the learning procedure.
Listing 1: Prior and posterior LD sampling.