SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets and model architectures. We pre-train a deep (multi-scale) EBM [57] denoted as  $E_{d}(.)$  with 12 residual layers [65] as is used in DeepEBM<sup>8</sup>. We set the starting noise level and ending noise level to be  $\sigma_{min} = 0.3$  and  $\sigma_{max} = 3.0$ . The energy for different noises is parametrized with  $E_{\sigma}(\boldsymbol{x}) = f(\boldsymbol{x}) / \sigma$ . The training learning rate is set to 0.001. We pre-train the EBM for 200k iterations. Samples from deep EBM are obtained by simulating annealed Langevin dynamics algorithm that is the same as the one used in Li et al. [57]. Our implicit sampler is also a neural network. The network consists of four inverse convolutional layers which hidden dimensions to be 1024, 512, 256, and 3. For each inverse convolutional layer, we use a 2D BatchNormalization and LeakyReLU non-linearity with a leak hyper-parameter to be 0.2. The prior distribution of the implicit generator is the standard Multi-variate Gaussian distribution. The score network follows a
$^{8}$ https://github.com/zengyi-li/MDSM
UNet architecture that is adapted from the codebase  $^9$ . In order to match the design of the multi-scale EBM, we parametrize the score function in the same way as  $S_{\sigma}(\pmb{x}) \coloneqq S(\pmb{x}) / \sigma$
**Training details.** We pre-train deep EBM on the MNIST dataset in the house. Then the pre-trained EBM is viewed as a multi-scale un-normalized target distribution. We then use the KL training method to update the score network and the generator in order to let the generator match the target distribution. We initialize the generator and the score network randomly. For each training iteration, we randomly select a noise  $\sigma \in [\sigma_{min}, \sigma_{max}]$ . We then generate a sample from the generator and add a Gaussian noise with variance  $\sigma^2$ . Then we update the score network with generated samples according to denoising score matching. Then we generate a batch of samples and add Gaussian noise with the same variance from the generator and use the gradient estimation in Algorithm 1 to update the generator's parameter. We use the Adam optimizer with a learning rate of 0.0001 for both the score network and the generator. For the score network, we set the hyper-parameter of the optimizer to be  $\beta_0 = 0.9$  and  $\beta_1 = 0.99$ . For the generator, we set  $\beta_0 = 0$  and  $\beta_1 = 0.99$ .
Evaluation metrics. To qualitatively evaluate the quality of generated samples, we adapt the definition of the Frechet Inception Distance (FID) [58]. We pre-train an MNIST image classifier with architecture the WideResNet [66] model with a depth of 16 and widen factor of 8. We then calculate the Wasserstein distance in the feature space of the pre-trained classifier with the same method as the FID.
Additional discussion. In this experiment, we use the KL training method to train a generator to sample from pre-trained multi-scale EBM. We also try the Fisher training method. But the Fisher training fails to train implicit samplers to sample from EBMs. One of the reasons is that the Fisher training requires calculating the trace of the Jacobian matrix of the score network. For high-dimensional data such as image data, such trace term is very expensive. We also try the stochastic trace estimation method such as the Skilling-Hutchison trace estimation technique [67]. But for the Fisher training, we find that stochastic trace estimation does not make the Fisher training work for high-dimensional samplers.
<sup>9</sup>https://github.com/huggingface/notebooks/blob/main/examples/annotated_diffusion.ipynb