SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In our experiments  $c$  equals to 1 and we use  $\lambda = 1.0001$ . In post processing, we iteratively go through all nodes  $x_{i}$  for  $i = 1, \dots, d$ . If there exists  $x_{j} = 1$  for  $(x_{i}, x_{j}) \in E$ , we flip its value  $x_{j} = 0$ . After post processing, the state  $x$  is guaranteed to be feasible in the original MIS problem. We provide the average and standard deviation of the best solutions over all the number of instances along with their corresponding running time (s) at 5. The plots of the experiments could be found at 12.
Table 5: MIS.
Sampler	Graphs Density	ER[700-800]	ER[9000-11000]	SATLIB
0.05	0.10	0.15	0.20	0.25	0.15
HB-10-1	Size	100.374 ± 2.073	58.750 ± 1.172	41.812 ± 0.864	32.344 ± 0.643	26.469 ± 0.749	277.149 ± 3.206	422.427 ± 14.403
Time(s)	426.185	390.810	684.590	414.067	429.879	15139.425	5381.857
BG-2	Size	102.468 ± 2.015	60.000 ± 0.707	42.820 ± 0.765	32.250 ± 0.500	27.312 ± 0.634	316.170 ± 3.187	422.200 ± 14.390
Time(s)	291.427	290.042	562.986	295.024	288.109	13079.125	3027.204
RWM	Size	97.186 ± 1.943	56.249 ± 0.968	40.429 ± 0.777	31.219 ± 0.599	25.594 ± 0.605	-555.674 ± 359.008	420.284 ± 14.263
Time(s)	284.092	293.517	499.577	297.140	281.772	12401.737	2955.729
GWG-nA	Size	104.812 ± 1.590	62.125 ± 0.739	44.383 ± 0.830	34.812 ± 0.583	28.187 ± 0.463	367.310 ± 4.383	422.971 ± 14.407
Time(s)	278.885	308.873	737.671	303.435	310.551	24698.296	3540.670
DMALA	Size	104.750 ± 1.803	62.031 ± 0.684	44.195 ± 0.781	34.375 ± 0.599	28.031 ± 0.467	357.058 ± 9.622	423.641 ± 14.506
Time(s)	291.271	292.131	714.614	297.848	298.732	24769.380	3465.343
PAS	Size	105.062 ± 1.560	62.250 ± 0.790	44.570 ± 0.669	34.719 ± 0.6242	28.500 ± 0.500	377.123 ± 4.498	424.143 ± 14.531
Time(s)	299.004	310.765	759.372	299.569	308.475	25242.166	4826.039
DLMCF	Size	104.450 ± 1.561	62.219 ± 0.926	44.078 ± 0.746	34.469 ± 0.558	28.125 ± 0.414	354.121 ± 10.683	423.387 ± 14.441
Time(s)	291.366	301.554	726.287	302.667	300.413	24892.216	3679.425
DLMC	Size	104.844 ± 1.769	62.187 ± 0.882	44.273 ± 0.788	34.500 ± 0.707	28.281 ± 0.450	355.058 ± 10.128	423.479 ± 14.483
Time(s)	293.235	294.975	725.326	294.688	299.884	24976.312	3523.320
We also conduct experiments to justify the results are robust regarding the choice of the penalty coefficient. In Figure 13, we use penalty coefficient  $\lambda \in \{1.001, 1.01, 1.1, 2\}$  on ER-[700-800] graphs with density  $\{0.05, 0.10, 0.15, 0.20, 0.25\}$ . We also use a dashed line to represent the optimal value obtained by running Gurobi-10 for 1 hour. From the results, we can observe that 1) PAS consistently obtains the best results, 2) locally balanced samplers have results consistently better than traditional sampler and Gurobi.
A.2.2 Max Clique
The max clique problem is equivalent to MIS on the dual graph. In our experiments  $c$  equals to 1.
$$
\min  _ {x \in \{0, 1 \} ^ {d}} - \sum_ {i = 1} ^ {d} c _ {i} x _ {i}, \quad \text {s . t .} x _ {i} x _ {j} = 0, \forall (i, j) \notin E \tag {14}
$$
The energy function is
$$
f (x) := - c ^ {T} x + \frac {\lambda}{2} \left(\mathbf {1} ^ {\top} x \cdot \left(\mathbf {1} ^ {\top} x - 1\right) - x ^ {T} A x\right) \tag {15}
$$
In our experiments  $c$  equals to 1 and we use  $\lambda = 1.0001$ . In post processing, we iteratively go through all nodes  $x_{i}$  for  $i = 1, \dots, d$ . If there exists  $x_{j} = 1$  for  $(x_{i}, x_{j}) \notin E$ , we flip its value  $x_{j} = 0$ . After post processing, the state  $x$  is guaranteed to be feasible in the original Max Clique problem. We provide the average and the standard deviation of the best solutions over all number of instances along with their corresponding running time at 6. The plots of the experiments could be found at 14.
A.2.3 MaxCut
We optimize the following problem:
$$
\min  _ {x \in \{- 1, 1 \} ^ {d}} - \sum_ {(i, j) \in E} A _ {i, j} \left(\frac {1 - x _ {i} x _ {j}}{2}\right) \tag {16}
$$
Note that for simplicity each dimension of  $x$  is selected from  $\{-1, 1\}$ . To represent the corresponding energy function for  $x \in \{0, 1\}^d$ , we have
$$
f (x) := - \sum_ {(i, j) \in E} A _ {i, j} \left(\frac {1 - \left(2 x _ {i} - 1\right) \left(2 x _ {j} - 1\right)}{2}\right) \tag {17}
$$
In our experiments  $A_{ij}$  equals to 1. Since the problem is always feasible, the post processing is an identity map. We provide the average and standard deviation of the best solutions over all number of instances along with their corresponding running time at 7. The plots of the experiments could be found at 15.
Table 6: Max Clique.
Sampler	Results	RB	TWITTER
HB-10-1	Ratio α	0.850 ± 0.0620	0.966 ± 0.056
Time(s)	1724.893	6.817
BG-2	Ratio α	0.859 ± 0.061	0.995 ± 0.030
Time(s)	1592.808	6.327
RWM	Ratio α	0.841 ± 0.0633	0.584 ± 0.484
Time(s)	1683.397	5.664
GWG-nA	Ratio α	0.878 ± 0.062	0.999 ± 0.010
Time(s)	2525.801	6.032
DMALA	Ratio α	0.876 ± 0.0620	0.999 ± 0.004
Time(s)	2561.617	6.190
PAS	Ratio α	0.878 ± 0.0618	0.999 ± 0.011
Time(s)	2542.538	6.160
DLMCF	Ratio α	0.871 ± 0.061	0.999 ± 0.011
Time(s)	2532.835	5.988
DLMC	Ratio α	0.875 ± 0.062	0.999 ± 0.009
Time(s)	2639.588	6.124
Table 7: Maxcut.
Sampler	Results	BA	ER
16-20	32-10	64-75	128-150	256-300	512-600	1024-1100	256-300	512-600	1024-1100
HB-10-1	Ratio α	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.001	1.008±0.005	1.014±0.004	1.020±0.017	1.000±0.001	0.998±0.001
Time(s)	742.568	754.613	749.626	783.278	792.338	1143.302	1890.534	331.019	416.002	1488.382
BG-2	Ratio α	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.009±0.005	1.014±0.004	1.021±0.018	1.001±0.001	0.999±0.001
Time(s)	517.183	538.258	550.082	553.863	531.720	578.991	1157.571	269.116	337.014	1295.219
RWM	Ratio α	0.998±0.000	1.000±0.000	1.000±0.000	1.000±0.000	0.999±0.001	1.005±0.005	1.007±0.004	1.019±0.017	0.997±0.002	0.996±0.001
Time(s)	534.215	534.615	528.641	558.608	541.302	574.778	1065.852	267.071	333.402	1266.630
GWG-nA	Ratio α	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.010±0.005	1.017±0.004	1.021±0.017	1.002±0.001	1.001±0.001
Time(s)	522.094	531.425	578.917	551.923	545.634	724.721	1427.577	264.202	466.199	1666.021
DMALA	Ratio α	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.010±0.005	1.018±0.004	1.021±0.017	1.002±0.001	1.002±0.001
Time(s)	531.433	538.938	568.224	549.026	544.568	750.909	1490.872	277.855	461.179	1643.135
PAS	Ratio α	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.010±0.005	1.018±0.004	1.021±0.017	1.002±0.001
Time(s)	519.842	538.814	550.035	550.578	580.051	940.408	1917.954	278.005	543.607	1689.071
DLMCF	Ratio α	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.010±0.005	1.018±0.004	1.021±0.017	1.002±0.001
Time(s)	521.592	526.289	545.877	557.564	533.119	765.719	1510.380	272.841	452.252	1639.539
DLMC	Ratio α	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.010±0.005	1.018±0.004	1.021±0.017	1.002±0.001
Time(s)	531.003	550.118	543.287	544.611	542.677	765.104	1564.198	271.262	451.080	1642.223
A.2.4 Balanced Graph Partition
We find the following objective for balanced graph partition gives the best result:
$$
f (x) := \sum_ {s = 1} ^ {k} \sum_ {(i, j) \in E} \mathbb {I} \left(x _ {i} \neq x _ {j} \& \& \left(x _ {i} = s \mid \mid x _ {j} = s\right)\right) + \sum_ {s = 1} ^ {k} \left(d / k - \sum_ {i = 1} ^ {d} \mathbb {I} \left(x _ {i} = s\right)\right) ^ {2} \tag {18}
$$
where  $k$  is the number of partitions. Since the problem is always feasible, the post processing is identity map. We provide the edge cut ratio and balanceness of the best samples over all the chains at 8. Further details on the calculated metrics could be found at A.6.2.
A.3 Energy Based Generative Models
A.3.1 Restricted Boltzmann Machine
The RBM is an unnormalized latent variable model, with a visible random variable  $v \in \mathcal{C}^N$  and a hidden random variable  $h \in \{0,1\}^M$ . When  $v$  is binary, we call it a binary RBM (binRBM) and when  $v$  is categorical, we call it a categorical RBM (catRBM). The energy function of both binRBM and catRBM (Tran et al., 2011) can be written as:
$$
f (v) = \sum_ {h} \left[ - \sum_ {n = 1} ^ {N} \left\langle v _ {n}, \theta_ {n} \right\rangle - \sum_ {m = 1} ^ {M} \beta_ {m} h _ {m} - \sum_ {d, m} \left\langle h _ {m} \theta_ {m, d}, v _ {n} \right\rangle \right] \tag {19}
$$
Unlike the previous three models, where the parameters are hand designed, we train binary RBM on MNIST (LeCun, 1998) and categorical RBM on Fashion-MNIST (Xiao et al., 2017a) using contrastive divergence Hinton (2002). Across all settings, we have  $D = 784$ . For binary models, we use  $M = 25$  for high temperature setting and  $M = 200$  for low temperature setting. For categorical models, we use  $M = 100$ . We report the results in Figure 16. The experimental setup is similar to classical graphical models.
Table 8: Balanced graph partition.
Metric	Samplers	VGG	MNIST-conv	ResNet	AlexNet	Inception-v3
Edge cut ratio ↓	HB-10-1	0.050	0.046	0.050	0.037	0.065
BG-2	0.048	0.045	0.050	0.038	0.069
RWM	0.054	0.046	0.092	0.052	0.117
GWG	0.102	0.046	0.159	0.063	0.164
DMALA	0.084	0.058	0.178	0.063	0.176
DMALA-nA	0.059	0.045	0.048	0.039	0.054
PAS	0.053	0.045	0.047	0.037	0.052
PAS-nA	0.084	0.050	0.138	0.053	0.144
DLMCF	0.086	0.063	0.178	0.053	0.176
DLMCF-nA	0.092	0.069	0.048	0.085	0.052
DLMC	0.105	0.056	0.183	0.097	0.182
DLMC-nA	0.113	0.048	0.082	0.091	0.086
Balanceness ↑	HB-10-1	0.999	0.999	0.999	0.999	0.999
BG-2	0.999	0.997	0.999	0.999	0.999
RWM	0.999	0.998	0.999	0.999	0.999
GWG	0.999	0.997	0.999	0.999	0.999
DMALA	0.999	0.998	0.999	0.999	0.999
DMALA-nA	0.999	0.997	0.999	0.999	0.999
PAS	0.999	0.997	0.999	1.000	0.999
PAS-nA	0.999	0.998	0.999	0.999	0.999
DLMCF	0.999	0.997	0.999	0.999	0.999
DLMCF-nA	0.999	0.995	0.999	0.999	0.999
DLMC	0.999	0.994	0.999	0.999	0.999
DLMC-nA	0.999	0.993	0.999	0.999	0.999
A.3.2 Deep residual network
In this experiment, we train a deep residual network on MNIST, Omniglot and Caltech dataset. The models checkpoints could be found at DISCS DATA. We use all the samplers to sample from the trained energy based generative models. We use the chain length of 10k and number of chains of 100. We randomly selected one chain from the 100 chains and save its sample after each 1k steps, giving us 10 images per each chain for each sampler 17. We can see that locally balanced samplers are able to generate higher quality images faster and visit more diverse modalities.
A.3.3 Text Infilling
The experimental set up is explained in detail at 4.4. Here we additionally provide the performance of the locally balanced samplers in their non adaptive condition observed at 9. The data used for this experiment could be found at DISCS DATA.
A.4 DISCS Code
The source code is open source at DISCS with extensive documentation.
A.5 Data-Set
The data used in this paper is available at DISCS DATA. Under the data set folder, you can find:
- Restricted Boltzmann Machine (RBM) checkpoints, more specifically two binary RBM checkpoints trained on MNIST dataset, one with 25 hidden dimensions and other with 200. Two categorical RBM checkpoints, trained on Fashion-MNIST data set with 4 and 8 categories. 16
- Deep residual network checkpoints, more specifically three checkpoints trained on MNIST, Omniglot and Caltech dataset.
Graphs data used for combinatorial optimization experiments found at sco directory.
- Text infilling data used for text infilling task experimentation.
Table 9: Quantative results on text infilling. The reference text for computing the Corpus BLEU is the combination of WT103 and TBC.
Methods	Self-BLEU (↓)	Unique n-grams (%) (↑)	Corpus BLEU (↑)
Self	WT103	TBC
n = 2	n = 3	n = 2	n = 3	n = 2	n = 3
RWM	92.41	6.26	9.10	18.97	26.73	19.33	26.67	16.24
GWG√t	85.93	11.22	17.14	23.16	35.56	23.58	35.56	16.75
GWG t+1	81.15	15.47	22.70	25.62	38.91	25.62	38.58	16.68
DMALA-nA√t	83.99	13.26	19.52	24.33	36.40	25.30	36.40	16.37
DMALA-nA t+1	80.44	15.86	23.58	25.79	39.88	26.57	40.20	16.64
DMALA√t	85.88	11.58	17.14	22.07	34.08	23.22	34.15	17.06
DMALA t+1	80.21	16.36	23.71	25.60	39.39	26.75	39.72	16.53
PAS√t	85.39	11.37	17.60	22.61	35.53	23.65	35.47	16.57
PAS t+1	81.02	15.62	22.65	25.59	39.28	26.08	39.48	16.69
DLMCf-nA√t	91.57	7.25	10.42	19.53	28.31	20.13	28.18	16.56
DLMCf-nA t+1	81.66	15.31	21.78	26.39	39.56	27.60	39.69	16.31
DLMCf√t	88.39	9.53	14.06	21.00	31.85	22.27	31.98	16.70
DLMCf t+1	80.12	16.25	23.76	25.41	39.31	26.86	39.57	16.73
DLMC-nA√t	83.74	12.74	19.64	24.27	37.27	24.94	37.34	16.73
DLMC-nA t+1	82.26	14.18	21.41	25.51	39.10	26.18	39.29	16.55
DLMC√t	85.28	12.05	17.65	24.03	36.34	24.51	36.27	16.45
DLMC t+1	84.55	12.62	18.47	24.27	37.28	24.94	37.14	16.69
A.6 Metrics and Evaluation Methods
Depending on the task we are solving, we use different metrics and evaluation methods to measure the quality of the generated samples. Below we explain in detail the process of evaluation for different tasks in the benchmark. Additionally, referring to an example provided in 4.4, DISCS is structured in a way that researchers can easily plug in new evaluation methods and define their own domain specific scores. For further code implementation details, you can refer to DISCS-Evaluator
A.6.1 Classical Graphical Models
For each of the experiments of classical graphical models, we run multiple chains in parallel. We rely on Effective Sample Size (ESS) to measure the efficiency of different samplers. We compute the ESS using tfp.substrates.jax.mcmc.effective_sample_size from Python TensorFlow library on the second half of the chain after the burn-in phase. The average and the standard deviation of the ESS is computed and reported over the chains.
A.6.2 Combinatorial Optimization
For the combinatorial optimization problems, for each of the test instances, we run multiple chains in parallel. Throughout the chain generation, we keep track of the best solution over all the chains for each of the instances. We report the average and standard deviation of the best solutions found over the test instances for each problem.
For balanced graph partition problem, we additionally store the sample resulting in the best solution of the optimization problem and do further post processing on it. Let  $n$  be the number of nodes and the  $g$  number of disjoint sets. For each of the instances, we load the corresponding graph and the best solution and compute the edge cut ratio as the ratio of the cut to the total number of edges and the balanceness as one minus the MSE of a number of nodes in every partition and balances partition  $n / g$ .
A.6.3 Energy Based Generative Models
For the text infilling task, we evaluate the generated text from two perspectives, diversity, and quality. For diversity, we use self-BLEU (Zhu et al., 2018) and the number of unique n-grams (Wang & Cho, 2019) to measure the difference between the generated sentences. For quality, we measure the BLEU score (Papineni et al., 2002) between the generated texts and the original dataset, which is
the combination of TBC and WikiText-103. We rely on Natural Language Toolkit (NLTK) Python library, more specifically nltktranslate.bleu_score and nltk.util.grams to compute the metrics above.
To gain insights into the performance and the quality of the generated samples we also rely on visual representation 17. As the chains are generated, the samples are saved as images at different time intervals. The images could provide insights on the performance of the sampler in terms of the quality and diversity of generated samples.
Figure 7: Results of Bernoulli Models
Figure 8: Results of Categorical model
Figure 9: Results of Ising model
Figure 10: Results of Potts model
Figure 11: Results of FHMMs
ER[800-800-0.05]
ER[800-800-0.10]
ER[800-800-0.20]
ER[800-800-0.25]
SATLIB
Figure 12: Solving progress on MIS
Figure 13: Reuslts on MIS: effect of penalty coefficient. The dashed line represents the best result obtained by running Gurobi for 1 hour.
Figure 14: Solving progress on Max Clique
aannnnnne aannnnnne
1- 100
oosssss
Figure 15: Solving progress on MaxCut
Figure 16: Results of RBMs
Omniglot
MNIST
Caltech
Figure 17: Resnet EBM trained on different data set with snapshots for every 1k sampling steps