SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the sequence  $\beta_0 < \beta_1 < \ldots < \beta_T \to \infty$  converges to a large enough value as  $T$  increases. Below, we present the problem formulation and the energy functions used for MaxCut and MIS problems. We present several results of the samplers solution for these combinatorial optimization problems as illustrative examples in the main text and report the remaining results in the Appendix A.2.
MaxCut The objective of MaxCut problem is to find a cut on a graph  $G = (V, E)$  that partitions the graph nodes into two complementary sets  $V = V_1 \cup V_2$ , such that the number of edges in  $E$  between  $V_1$  and  $V_2$  is as large as possible. MaxCut is an unconstrained problem, which makes its formulation relatively simple. We can set  $\mathcal{C} = \{0, 1\}$  such that  $x_i = 0$  represents  $i \in V_1$  and  $x_i = 1$  means  $x_i \in V_2$ . Then we can write  $a(x) = -x^\top A x, b(x) \equiv 0$ , where  $A$  is the adjacency matrix of the graph  $G$ .
We apply the same simulated annealing temperature scheduling set up for all the samplers and compare the samplers performances against each other. In the MaxCut problem, we compute the ratio of our solution against the optimal solution found by Gurobi, running for one hour (Dai et al., 2020a). The results presented at Figure 4 show the cut ratio throughout the chain generation over the number of M-H steps and the running time (s). The legends are sorted according to the most optimal solution each sampler finds. One can see that the PAS leads the results. Also, locally balanced samplers significantly outperform the traditional samplers, especially when the graph size increases.
ER-[256-300]
ER-[512-600]
ER-[1024-1100]
Figure 4: Results for MaxCut on ER graphs. The ratio is computed by dividing the optimal cut size obtained from running Gurobi for 1 hour. (top) ratio with respect to the number of M-H steps, (bottom) ratio with respect to running time.
MIS On a graph  $G = (V, E)$ , an independent set  $S \subset V$  means that for any  $i, j \in S$ ,  $(i, j) \notin E$ . We can set  $\mathcal{C} = \{0, 1\}$  such that  $x_i = 0$  means  $i \notin S$  and  $x_i = 1$  means  $i \in S$ . Then we can write  $a(x) = -\sum_{i \in V} x_i$  and  $b(x) = \sum_{(i, j) \in E} x_i x_j$ . For the penalty coefficient  $\lambda$ , we follow Sun et al. (2022c) to select  $\lambda = 1.0001$  being a value slightly larger than 1. We run all samplers on five groups of small ER graphs with 700 to 800 nodes, each group has 128 graphs with densities varying 0.05, 0.10, 0.15, 0.20, and 0.25. We also run all samplers on 16 large ER graphs with 9000 to 11000 nodes. For each configuration, we run 32 chains with the same running time and report the average of the best results found by each chain in Table 1. One can easily see that PAS obtains the best result.
Table 1: Results for MIS on ER graphs. The set found by the sampling algorithm is not necessarily an independent set, we report a lower bound: set size - # pair of adjacent nodes in the set.
Sampler	ER[700-800]	ER[9000-11000]
0.05	0.10	0.15	0.20	0.25	0.15
HB-10-1	100.374	58.750	41.812	32.344	26.469	277.149
BG-2	102.468	60.000	42.820	32.250	27.312	316.170
RWM	97.186	56.249	40.429	31.219	25.594	-555.674
GWG-nA	104.812	62.125	44.383	34.812	28.187	367.310
DMALA	104.750	62.031	44.195	34.375	28.031	357.058
PAS	105.062	62.250	44.570	34.719	28.500	377.123
DLMCf	104.450	62.219	44.078	34.469	28.125	354.121
DLMC	104.844	62.187	44.273	34.500	28.281	355.058
4.4 Sampling from Energy Based Generative Models
The discrete samplers can also play the role of decoder in generative models. In particular, given a dataset  $\mathcal{D} = \{X_{i}\}_{i = 1}^{N}$  sampled from the target distribution  $\pi$ , one can train an energy function  $f_{\theta}(\cdot)$  such that the energy based model  $\pi_{\theta}(\cdot) \propto \exp(-f_{\theta}(\cdot))$  fits the dataset  $\mathcal{D}$ . DISCS provides multiple checkpoints for the energy function trained on real-world image or language datasets. Researchers can easily evaluate their samplers after loading the learned energy function. We provide further experimental details and mathematical formulations at Appendix A.3.
For the models that are relatively simple, for example, Restricted Boltzmann Machine (RBM) trained on MNIST (LeCun, 1998) and fashion-MNIST (Xiao et al., 2017b), one can continue using ESS as the metric. In Figure 5, we evaluate the samplers on RBMs trained on MNIST with 25 and 200 hidden variables. One can see that DLMC has the best performance. We further report the results of samplers on categorical RBM trained on fashion-MNIST dataset at Appendix 16. For more complicated deep energy based models, a sampler may fail to mix within reasonable steps. In this case, ESS is not a good metric. To address this problem, DISCS provides multiple alternative measurements, including snapshots and domain specific scores.
Snapshots After loading the checkpoint of energy based generative models, DISCS can generate snapshots of the sampling chains. For example, in Figure 6, we display the snapshots of sampling on a deep residual network trained on MNIST data (Sun et al., 2021) and on pretrained language model BERT  $^{3}$ . One can see that locally balanced samplers generate samples with higher qualities, and can typically visit multiple modalities in the distribution. We report further results on deep residual network trained on Omniglot and Caltech dataset at 17.
Domain Specific Scores In many deep generative tasks, the goal is to efficiently sample high-quality samples, instead of mixing in the learned energy based models. In this scenario, domain specific scores that directly evaluate the sample qualities are better choices. For example, DISCS provides text filling task based on pre-trained language models like BERT (Wang & Cho, 2019; Devlin et al., 2018). Following the settings in prior work (Zhang et al., 2022), we randomly samples 20 sentences from TBC (Zhu et al., 2015) and WiKiText-103 (Merit et al., 2016) and mask four words in each sentence (Donahue et al., 2020) resulting in the dataset provided at DISCS DATA. For each masked sentence, we sample 25 sentences by generating 25 chains with length of 50, following the target probability distribution provided by BERT, and then selecting the last sample of the chain. As a common practice
3 loading the checkpoint from https://huggingface.co/bert-base-uncased.
Figure 5: Results on RBMs trained on MNIST dataset. (top) RBM with 25 binary hidden variables, (bottom) RBM with 200 binary hidden variables.
in non-auto-regressive text generation, we select the top-5 sentences with the highest likelihood out of 25 sentences to avoid low-quality generation (Gu et al., 2017; Zhou et al., 2019).
We evaluate the generated samples in terms of diversity and quality. For diversity, we use self-BLEU (Zhu et al., 2018) and the number of unique n-grams (Wang & Cho, 2019) to measure the difference between the generated sentences. For quality, we measure the BLEU score (Papineni et al., 2002) between the generated texts and the original dataset, which is the combination of TBC and WikiText-103. We report the quantitative results in Table 2. We do not have the results for HB and BG as they are computationally infeasible for this task with  $30\mathrm{k + }$  tokens. In this task, the locally balanced sampler still outperforms RWM. Also, one can notice that the weight function  $\frac{t}{t + 1}$  significantly outperforms  $\sqrt{t}$  on diversity metrics and reaches comparable results on the quality. The reason is that the overparameterized neural network is a low temperature system with sharp landscape. This phenomenon is consistent with the results in Figure 2. We provide further results for the non-adaptive cases with binary search fine tuning in Appendix A.3.3.
MNIST
Figure 6: Snapshots of energy based generative models: (left) snapshots for every 1k steps on MNIST ResNet, (right) snapshots for text filling task on BERT in Table 2
knowing even [MASK] details [MASK] [MASK] to understand [MASK] people involved and how they relate to one another .
RMW	knowing even the details gives lemmon to understand the people involved and how they relate to one another. knowing even the details gives one to understand the people involved and how they relate to one another.
GWG	knowing even the details was necessary to understand the people involved and how they relate to one another. knowing even simple details is important to understand the people involved and how they relate to one another.
DMALA-nA	knowing even conversation details is good to understand the people involved and how they relate to one another. knowing even no details accountable, to understand the people involved and how they relate to one another.
DMALA	knowing even the details also easy to understand the people involved and how they relate to one another. knowing even the details is primitive to understand the people involved and how they relate to one another.
PAS	knowing even the details is important to understand the people involved and how they relate to one another. knowing even the details always helps to understand the people involved and how they relate to one another.
DLMcf	knowing even the details was easy to understand the people involved and how they relate to one another. knowing even the details also easy to understand the people involved and how they relate to one another.
DLMC-nA	knowing even complex details is important to understand the people involved and how they relate to one another. knowing even different details is good to understand the people involved and how they relate to one another.
DLMC	knowing even the details was good to understand the people involved and how they relate to one another. knowing even complex details allows you to understand the people involved and how they relate to one another.
Table 2: Quantative results on text infilling. The reference text for computing the Corpus BLEU is the combination of WT103 and TBC.
Methods	Self-BLEU (↓)	Unique n-grams (%) (↑)	Corpus BLEU (↑)
Self	WT103	TBC
n = 2	n = 3	n = 2	n = 3	n = 2	n = 3
RWM	92.41	6.26	9.10	18.97	26.73	19.33	26.67	16.24
GWG√t	85.93	11.22	17.14	23.16	35.56	23.58	35.56	16.75
DMALA√t	85.88	11.58	17.14	22.07	34.08	23.22	34.15	17.06
PAS√t	85.39	11.37	17.60	22.61	35.53	23.65	35.47	16.57
DLMCf√t	88.39	9.53	14.06	21.00	31.85	22.27	31.98	16.70
DLMC√t	85.28	12.05	17.65	24.03	36.34	24.51	36.27	16.45
GWG t/t+1	81.15	15.47	22.70	25.62	38.91	25.62	38.58	16.68
DMALA t/t+1	80.21	16.36	23.71	25.60	39.39	26.75	39.72	16.53
PAS t/t+1	81.02	15.62	22.65	25.59	39.28	26.08	39.48	16.69
DLMCf t/t+1	80.12	16.25	23.76	25.41	39.31	26.86	39.57	16.73
DLMC t/t+1	84.55	12.62	18.47	24.27	37.28	24.94	37.14	16.69
DISCS is a tailored benchmark for discrete sampling. It implements a range of discrete sampling tasks and state-of-the-art discrete samplers and enables a fair comparison. From the results, we know that DLMC leads in sampling from classical graphical models, PAS leads in solving combinatorial optimization problems, DLMCf and DMALA have the best performance on language models. We believe more efficient discrete samplers can be obtained by designing better discretization of DLD (Sun et al., 2022a). DISCS is a convenient tool during this process. The researcher can freely set the configurations for tasks and samplers and DISCS will automatically compile the program and run the processes in parallel. Besides, we observe that the choice of the locally balanced weight function should depend on the critical temperature of the target distribution. We believe this observation is insightful and will lead to a deeper understanding of locally balanced samplers.
Of course, DISCS does not include all existing tasks or samplers in discrete sampling, for example, the zero order (Xiang et al., 2023) and second order (Sun et al., 2023a) approximation methods. We will keep iterating DISCS and more features will be added in the future. We wrap DISCS to a JAX library. Researchers can conveniently implement customer tasks or samplers to accelerate their study and, in the meanwhile, contribute the code to DISCS for further improvement. We believe DISCS will be a powerful tool for researchers and facilitate future research in discrete sampling.
The authors would like to thank Bo Dai, Bethany Wang, Emily Xue for providing helpful discussions, and Pengcheng Ying, Sherry Yang and anonymous reviewers for the helpful comments on the paper.
Baumgartner, A., Burkitt, A., Ceperley, D., De Raedt, H., Ferrenberg, A., Heermann, D., Herrmann, H., Landau, D., Levesque, D., von der Linden, W., et al. The Monte Carlo method in condensed matter physics, volume 71. Springer Science & Business Media, 2012.
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., and Riddell, A. Stan: A probabilistic programming language. Journal of statistical software, 76(1), 2017.
Dai, H., Chen, X., Li, Y., Gao, X., and Song, L. A framework for differentiable discovery of graph algorithms. 2020a.
Dai, H., Singh, R., Dai, B., Sutton, C., and Schuurmans, D. Learning discrete energy-based models via auxiliary-variable local exploration. arXiv preprint arXiv:2011.05363, 2020b.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Donahue, C., Lee, M., and Liang, P. Enabling language models to fill in the blanks. arXiv preprint arXiv:2005.05339, 2020.
Du, Y., Li, S., Tenenbaum, J., and Mordatch, I. Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.01316, 2020.
Edwards, S. F. and Anderson, P. W. Theory of spin glasses. Journal of Physics F: Metal Physics, 5 (5):965, 1975.
Ghahramani, Z. and Jordan, M. Factorial hidden markov models. Advances in Neural Information Processing Systems, 8, 1995.
Grathwohl, W., Swersky, K., Hashemi, M., Duvenaud, D., and Maddison, C. J. Oops I took a gradient: Scalable sampling for discrete distributions. arXiv preprint arXiv:2102.04509, 2021.
Gu, J., Bradbury, J., Xiong, C., Li, V. O., and Socher, R. Non-autoregressive neural machine translation. arXiv preprint arXiv:1711.02281, 2017.
Hamze, F. and de Freitas, N. From fields to trees. arXiv preprint arXiv:1207.4149, 2012.
Hastings, W. K. Monte Carlo sampling methods using Markov chains and their applications. 1970.
Hinton, G. E. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
Hubbard, J. Calculation of partition functions. Physical Review Letters, 3(2):77, 1959.
Hukushima, K. and Nemoto, K. Exchange monte carlo method and application to spin glass simulations. Journal of the Physical Society of Japan, 65(6):1604-1608, 1996.
Ising, E. Beitrag zur theorie des ferro-und paramagnetismus. PhD thesis, Grefe & Tiedemann, 1924.
Katzgraber, H. G., Palassini, M., and Young, A. Monte carlo simulations of spin glasses at low temperatures. Physical Review B, 63(18):184422, 2001.
Kirkpatrick, S., Gelatt Jr, C. D., and Vecchi, M. P. Optimization by simulated annealing. science, 220 (4598):671-680, 1983.
LeCun, Y. The mnist database of handwritten digits. http://yann.lecun.com/exdb/mnist/, 1998.
LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., and Huang, F. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Merit, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., and Teller, E. Equation of state calculations by fast computing machines. The journal of chemical physics, 21(6):1087-1092, 1953.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Potts, R. B. Some generalized order-disorder transformations. In Mathematical proceedings of the Cambridge philosophical society, volume 48, pp. 106-109. Cambridge University Press, 1952.
Rhodes, B. and Gutmann, M. Enhanced gradient-based mcmc in discrete spaces. arXiv preprint arXiv:2208.00040, 2022.
Robert, C. and Casella, G. Monte Carlo statistical methods. Springer Science & Business Media, 2013.
Sansone, E. Lsb: Local self-balancing mcmc in discrete spaces. In International Conference on Machine Learning, pp. 19205-19220. PMLR, 2022.
A Experiments Details
A.1 Classical Graphical Models
For all the experiments of classical graphical models, we run 100 chains. The chains run in parallel on 4 V100 GPUs, with each GPU managing a mini-batch of 25 chains. We report the performance of all the samplers on Bernoulli Model 7, Categorical Model 8, Ising Model 9, Potts Model 10 and FHMM 11. We study the effect of number of discrete variables (sample dimension), the number of categories for each discrete variable, weight function for locally balanced samplers, and the smoothness/sharpness of different models. Note that the result for BG-2 on Potts 10 and Categorical 8 model with 256 categories are omitted as it takes over 100 hours. The chain length is set as 1 million steps when studying the effect of the number of categories and sample dimension. In the other cases, we use 100k steps as the chain length. For each experiment, as we sample the chains, each sample from each chain at each step is mapped to a lower dimension of 1. The samples are mapped on the same arbitrary sample which we randomly initialize in the beginning of the experiment. We calculate ESS on the mapped samples after the burn-in phase i.e. after the generation of half of the chain. We explain more details on ESS computation in section A.6.1. The average and standard deviation of ESS is computed over all the chains and is reported over the running time and number of energy evaluation of each sampler.
In the following sections, we provide the energy function we used for each of the classical graphical models.
A.1.1 Factorized Models
Factorized models are the simplest distributions in a discrete space, where each site is independent from the others. Consider the category set of one hot vectors  $\mathcal{C} = \{e_1,\dots,e_C\}$  and the state space  $\mathcal{X} = \mathcal{C}^N$ . We have  $|\mathcal{C}| = C$  is the number of category and  $N$  is the number of variables. The energy function of a factorized model is:
$$
f (x) = \sum_ {n = 1} ^ {N} \left\langle x _ {n}, \theta^ {n} \right\rangle \tag {9}
$$
where  $\theta^d \in \mathbb{R}^C$ . We denote the target distribution as Bernoulli model when  $C = 2$  and Categorical model when  $C > 2$ . We report the results on Bernoulli models and Categorical models in Figure 7 and 8, respectively.
A.1.2 Ising Models
The Ising model (Ising, 1924) is a mathematical model of ferromagnetism in statistical mechanics. It consists of binary random variables arranged in a lattice graph  $G = (V, E)$  and allows a node to interact with its neighbors. The Potts model (Potts, 1952) is a generalization of the Ising model where the random variables are categorical. The energy function for Ising model and Potts model can be described as:
$$
f (x) = - \sum_ {n = 1} ^ {N} \left\langle x _ {n}, \theta_ {n} \right\rangle - \sum_ {(i, j) \in E} J _ {i j} \left(x _ {i}, x _ {j}\right) \tag {10}
$$
where we set  $\theta^d \in \mathbb{R}^n$ , and  $J_{ij}(x_i, y_j) = 1_{\{x_i = y_j\}}$ . For Ising model, we use  $\theta^n \sim \mathrm{Uniform}(-2, 1)$  for the outer part of the lattice graph, and  $\theta^n \sim \mathrm{Uniform}(-1, 2)$  for the inner part of the lattice graph. We report the results on Ising model and Potts model in Figure 9, 10.
A.1.3 Factorial Hidden Markov Model (FHMM)
FHMM (Ghahramani & Jordan, 1995) uses latent variables to characterize time series data. In particular, it assumes the continuous data  $y \in \mathbb{R}^{L}$  is generated by hidden state  $x \in \mathcal{C}^{L \times K}$ . The probability function is:
$$
p (x) = p \left(x _ {1}\right) \prod_ {l = 2} ^ {L} p \left(x ^ {t} \mid x ^ {t - 1}\right), \quad p (y | x) = \prod_ {l = 1} ^ {L} \mathcal {N} \left(y _ {t}; \sum_ {k = 1} ^ {K} \langle W _ {k}, x _ {l, k} \rangle + b; \sigma^ {2}\right) \tag {11}
$$
In particular, for binary model, we consider  $\mathbb{P}(x_1 = 0) = 0.9$ ,  $\mathbb{P}(x^t = x^{t - 1}|x^{t - 1}) = 0.8$ ,  $\sigma = 2.0$ . We use  $L = 200$ ,  $K = 50$  for high temperature setting and  $L = 1000$ ,  $K = 10$  in low temperature setting. For categorical model, we use  $p(x_{1}|x_{1}\neq 0)$  and  $p(x^{t}|x^{t - 1},x^{t}\neq x^{t - 1})$  as uniform distribution and we use  $L = 200$ ,  $K = 10$  with category number  $C = 4,8$ . We report the results in Figure 11.
A.2 Combinatorial Optimization
Table 3: Synthetic data statistics.
Name	MIS	Max Clique	Maxcut
ER-[700-800]	ER-[9000-11000]	RB	ER	BA
Max # nodes	800	10,915	475	1,100	1,100
Max # edges	47,885	1,190,799	90,585	91,239	4,384
# Test instances	128	16	500	1,000	1,000
Table 4: Real-world data statistics.
Name	MIS SATLIB	Max Clique Twitter	Maxcut Optsicom	MNIST	VGG	Balanced Graph Partition
ALEXNET	RESNET	INCEPTION
Max # nodes	1,347	247	125	414	1,325	798	20,586	27,114
Max # edges	5,978	12,174	375	623	2,036	1,198	32,298	40,875
# Test instances	500	196	10	1	1	1	1	1
Here we first provide the experimental details for the combinatorial optimization problems, Maximum Independent Set (MIS), Max Clique, MaxCut and, Balanced Graph Partition. The statistics of the synthetic datasets, including the maximum number of nodes/edges in a graph, and the number of test instances are reported in 3. Additionally the statistics of real-world graphs are in 4. For MaxCut-ba, all Balanced Graph Partition and MIS graphs, we use 32 as the number of chains and for MaxCut-opticscom, MaxCut-er, and all Max Clique graphs we use 16. The data used for these experiments could be found at DISCS DATA.
We run all the experiments on 8 V100 GPUs in parallel. For only MaxCut Optsicom graph, we use 2 V100 GPUs. The test instances are divided evenly between the GPUs and are run in parallel. For each experiment, we report the average and standard deviation of the best solution found over the number of test instances along with the end-to-end run time (in seconds) of each in tables. We report the results for all the samplers and plot their solution as the chain is being generated over M-H step and the running time. Note that for reporting the standard deviation for the plots, for better visualization, we clip the found solutions by a minimum of zero (removing the penalty effect of the optimization objective). However, the tables include the accurate standard deviation of the best solution found over the instances.
In the following sections, we provide the energy function we use for each of the combinatorial optimization problems. For a graph  $G = (V,E)$  we label the nodes in  $V$  from 1 to  $d$ . The adjacency matrix is represented as  $A$ . For a weighted graph, we simply let  $A_{ij}$  denote the edge weight between node  $i$  and  $j$ . For constraint problems, we follow Sun et al. (2022c) to select penalty coefficient  $\lambda$  as the minimum value of  $\lambda$  such that  $x^{*} \coloneqq \arg \min f(x)$  is achieved at  $x^{*}$  satisfying the original constraints. Such a choice of the coefficient guarantees the target distribution converges to the optimal solution of the original CO problems while keeping the target distribution as smooth as possible.
A.2.1 Maximum Independent Set (MIS)
The MIS has the integer programming formulation as
$$
\min  _ {x \in \{0, 1 \} ^ {d}} - \sum_ {i = 1} ^ {d} c _ {i} x _ {i}, \quad \text {s . t .} x _ {i} x _ {j} = 0, \forall (i, j) \in E \tag {12}
$$
We use the corresponding energy function in the following quadratic form:
$$
f (x) := - c ^ {T} x + \lambda \frac {x ^ {T} A x}{2} \tag {13}
$$