LSB: Local Self-Balancing MCMC in Discrete Spaces

Markov Chain Monte Carlo ( MCMC ) methods are promising solutions to sample1 from target distributions in high dimensions . While MCMC methods enjoy nice2 theoretical properties , like guaranteed convergence and mixing to the true target , in3 practice their sampling efficiency depends on the choice of the proposal distribution4 and the target at hand . This work considers using machine learning to adapt the5 proposal distribution to the target , in order to improve the sampling efficiency in6 the purely discrete domain . Specifically , ( i ) it proposes a new parametrization for a7 family of proposal distributions , called locally balanced proposals , ( ii ) it defines8 an objective function based on mutual information and ( iii ) it devises a learning9 procedure to adapt the parameters of the proposal to the target , thus achieving fast10 convergence and fast mixing . We call the resulting sampler as the Locally Self-11 Balancing Sampler ( LSB ) . We show through experimental analysis on the Ising12 model and Bayesian networks that LSB is indeed able to improve the efficiency13 over a state-of-the-art sampler based on locally balanced proposals , thus reducing14 the number of iterations required to converge , while achieving comparable mixing15 performance.16 1 Introduction17 Sampling from complex and intractable probability distributions is of fundamental importance for18 learning and inference [ 16 ] . MCMC algorithms are promising solutions to handle the intractability19 of sampling in high dimensions and they have found numerous applications , in Bayesian statistics20 and statistical physics [ 17 , 23 ] , bioinformatics and computational biology [ 3 , 2 ] as well as machine21 learning [ 4 , 14 , 18 ] .22 Although , MCMC can be applied to sample from any target distribution , in practice its efficiency23 strongly depends on the choice of the proposal . Indeed , common phenomena , like slow convergence24 and slow mixing , are typically the result of wrong choices of the proposal distribution . Therefore , it ’ s25 extremely important to devise strategies enabling the tuning of the proposal to target distributions [ 5,26 13 ] . While there has been a lot of work focusing on designing machine learning-based strategies to27 improve the efficiency of MCMC in the continuous domain [ 29 , 20 , 1 , 6 , 19 ] , less effort has been28 devoted to the discrete counterpart . Most common solutions consider continuous relaxations of29 the problem by using embeddings and then leverage existing sampling strategies designed for the30 continuous case . These strategies are suboptimal , either because they consider limited settings , where31 the target distribution has specific analytic forms [ 29 ] , or because they make strong assumptions on32 the properties of the embeddings , thus not having guarantees of preserving the topological properties33 of the original discrete domain [ 20 , 1 , 6 , 19 ] .34 This work focuses on MCMC strategies for the purely discrete domain . Specifically , ( i ) we introduce35 a new parametrization for a family of proposal distributions , called locally balanced proposals , which36 have been recently studied in [ 28 ] , ( ii ) we define an objective function based on mutual information,37 which reduces the distance between the proposal and the target distribution and also reduces the38 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . statistical dependence between consecutive samples , and ( iii ) we devise a learning procedure to adapt39 the parameters of the proposal to the target distribution using our objective . The resulting procedure,40 called the Local Self-Balancing sampler ( LSB ) , automatically discovers an optimal locally balanced41 proposal , with the advantage of reducing the amount of user intervention and of improving the overall42 sampling efficiency , both in terms of convergence speed and mixing time.43 We provide some empirical analysis of sampling from the 2D Ising model and from Bayesian44 networks and show that in some cases LSB is able to halve the number of iterations required to45 converge , while achieving similar mixing performance to [ 28 ] .46 We start by providing some background on locally balanced proposal distributions ( Section 2 ) ,47 we introduce LSB by describing the parametrizations , the objective and the learning procedure48 ( Section 3 ) , we discuss the related work ( Section 4 ) and the experiments ( Section 5 ) , and finally we49 conclude by highlighting the main limitations of LSB and possible future directions ( Section 6 ) .50 2 Background51 We consider the problem of sampling from a distribution p with a support defined over a large finite52 discrete sample space X , i.e . p ( x ) = p̃ ( x ) / ∑ x′′∈X p̃ ( x ′′ ) , where the normalization term can not be53 tractably computed and only p̃ can be evaluated . One solution to the problem consists of sampling54 using MCMC [ 17 ] . The main idea of MCMC is to sequentially sample from a tractable surrogate55 distribution , alternatively called proposal , and to use an acceptance criterion to ensure that generated56 samples are distributed according to the original distribution . More formally , MCMC is a Markov57 chain with a transition probability of the form:58 T ( x′|x ) = A ( x′ , x ) Q ( x′|x ) + 1 [ x′ = x ] ∑ x′′∈X ( 1−A ( x′′ , x ) ) Q ( x′′|x ) ) ( 1 ) where Q ( x′|x ) is the probability of sampling x′ given a previously sampled x , namely the proposal59 distribution , 1 [ · ] is the Kronecker delta function and A ( x′ , x ) is the probability of accepting sample60 x′ given x , e.g . A ( x′ , x ) = min { 1 , p̃ ( x ′ ) Q ( x|x′ ) p̃ ( x ) Q ( x′|x ) } .1 In this work , we consider the family of locally61 informed proposals [ 28 ] , which are characterized by the following expression:62 Q ( x′|x ) = g ( p̃ ( x′ ) p̃ ( x ) ) 1 [ x′ ∈ N ( x ) ] Z ( x ) ( 2 ) where N ( x ) is the neighborhood of x based on the Hamming metric.263 Note that the choice of g has a dramatic impact on the performance of the Markov chain , as64 investigated in [ 28 ] . In fact , there is a family of functions called balancing functions , satisfying the65 relation g ( t ) = tg ( 1/t ) ( for all t > 0 ) , which have extremely desirable properties , briefly recalled66 hereunder.67 Acceptance rate . The balancing property allows to rewrite the acceptance function in a simpler68 form , namely A ( x′ , x ) = min { 1 , Z ( x ) Z ( x′ ) } . Therefore , a proper choice of g can increase the ratio69 between the normalization constants Z ( x ) and Z ( x′ ) with consequent increase of the acceptance70 rate even in high dimensional spaces.71 Detailed balance . Note that for all x′ = x , detailed balance trivially holds , viz . p ( x ) T ( x′|x ) =72 p ( x′ ) T ( x|x′ ) . In all other cases , detailed balance can be proved , by exploiting the fact that T ( x′|x ) =73 A ( x′ , x ) Q ( x′|x ) and by using the balancing property ( see the Supplementary material for more74 details ) . Detailed balance is a sufficient condition for invariance . Consequently , the target p is a fixed75 point of the Markov chain.76 Ergodicity . Under mild assumptions , we have also ergodicity ( we leave more detailed discussion77 to the Supplementary material ) . In other words , the Markov chain converges to the fixed point p78 independently from its initialization.79 1Other choices are available [ 17 ] as well . 2In other words , we consider all points having Hamming distance equal to 1 from x . Efficiency . The efficiency of MCMC is generally measured in terms of the resulting asymptotic80 variance for sample mean estimators . This is indeed a proxy to quantify the level of correlation81 between samples generated through MCMC . Higher levels of asymptotic variance correspond to82 higher levels of correlation , meaning that the Markov chain produces more dependent samples and83 it is therefore less efficient . Balancing functions are asymptotically optimal according to Peskun84 ordering [ 28 ] .85 The work in [ 28 ] proposes a pool of balancing functions with closed-form expression together with86 some general guidelines to choose one . However , this pool is only a subset of the whole family of87 balancing functions and several cases do not even have an analytical expression . Consequently , it is88 not clear which function to use in order to sample efficiently from the target distribution . Indeed , we89 will see in the experimental section that ( i ) the optimality of the balancing function depends on the90 target distribution and that ( ii ) in some cases the optimal balancing function may be different from the91 ones proposed in [ 28 ] . In the next sections , we propose a strategy to automatically learn the balancing92 function from the target distribution , thus achieving fast convergence ( burn-in ) and fast mixing.93 3 LSB : Local Self-Balancing Strategy94 We start by introducing two different parametrizations for the family of balancing functions in95 increasing order of functional expressiveness . Then , we propose an objective criterion based on96 mutual information , that allows us to learn the parametrization with fast convergence and fast mixing97 on the target distribution.98 3.1 Parametrizations99 We state the following proposition and then use it to devise the first parametrization.100 Proposition 1 . Given n balancing functions g ( t ) = [ g1 ( t ) , . . . , gn ( t ) ] T and a vector of scalar101 positive weights w = [ w1 , . . . , xn ] T , the linear combination g ( t ) . = wTg ( t ) satisfies the balancing102 property.103 Proof . g ( t ) = wTg ( t ) = ∑n i=1 wigi ( t ) = t ∑n i=1 wigi ( 1/t ) = tw Tg ( 1/t ) = tg ( 1/t ) 104 Despite its simplicity , the proposition has important implications . First of all , it allows to convert the105 problem of choosing the optimal balancing function into a learning problem . Secondly , the linear106 combination introduces only few parameters ( in the experiments we consider n = 4 ) and therefore107 the learning problem can be solved in an efficient way . The requirement about positive weights is108 necessary to guarantee ergodicity ( see Supplementary material on ergodicity for further details ) .109 The first parametrization ( LSB 1 ) consists of the relations wi = eθi/ ∑n j=1 e θj for all i = 1 , . . . , n,110 where θ = [ θ1 , . . . , θn ] ∈ Rn . Note that the softmax is used to smoothly select one among the n111 balancing functions . Therefore , we refer to this parametrization as learning to select among existing112 balancing functions.113 The second parametrization ( LSB 2 ) is obtained from the following proposition.114 Proposition 2 . Given gθ ( t ) = min { hθ ( t ) , thθ ( 1/t ) } , where hθ is a universal real valued function115 approximator parameterized by vector θ ∈ Rk ( e.g . a neural network ) , and any balancing function ` ,116 there always exists θ̃ ∈ Rk such that gθ̃ ( t ) = ` ( t ) for all t > 0.117 Proof . Given any balancing function ` , we can always find a θ̃ such that hθ̃ ( t ) = ` ( t ) for all t > 0118 ( because hθ is a universal function approximator ) . This implies that hθ̃ satisfies the balancing property,119 i.e . hθ̃ ( t ) = thθ̃ ( 1/t ) for all t > 0 . Consequently , by definition of gθ , we have that gθ ( t ) = hθ̃ ( t ) .120 And finally we can conclude that gθ ( t ) = ` ( t ) for all t > 0.121 In theory , LSB 2 parameterizes the whole family of balancing functions and it allows to find the122 optimal one from the whole set.3 In practice , it is better to restrict the analysis only to monotonic123 increasing functions , as we prefer to choose a proposal distribution sampling from regions of higher124 3Note also that gθ ( t ) is a balancing function for any parameter θ. probability mass . In the following proposition , we provide sufficient conditions to ensure the125 monotonicity of LSB 2.126 Proposition 3 . Define gθ as in LSB 2 . Assume that hθ ( t ) is a differentiable and monotonic increasing127 function with respect to variable t , satisfying the relation hθ ( 1/t ) ≥ 1t dhθ ( 1/t ) dt for all t > 0 . Then,128 gθ is a monotonic increasing function in t.129 Proof . We have that hθ ( 1/t ) − 1t dhθ ( 1/t ) dt ≥ 0 for all t > 0 . Also , note that dthθ ( 1/t ) dt =130 hθ ( 1/t ) − 1t dhθ ( 1/t ) dt > 0 . Therefore , thθ ( 1/t ) is a monotonic increasing function in t for131 all t > 0 . Now , consider any t1 , t2 with t1 ≥ t2 > 0 , gθ ( t1 ) = min { hθ ( t1 ) , t1hθ ( 1/t1 ) } 132 and gθ ( t2 ) = min { hθ ( t2 ) , t2hθ ( 1/t2 ) } . By monotonicity of hθ ( t ) and thθ ( 1/t ) , we have that133 hθ ( t1 ) ≥ hθ ( t2 ) and t1hθ ( 1/t1 ) ≥ t2hθ ( 1/t2 ) . Therefore , we can conclude that gθ ( t1 ) ≥ gθ ( t2 ) for134 all t1 , t2 with t1 ≥ t2 > 0.135 Therefore , to build gθ , we need a monotonic function hθ ( t ) . Specifically , we can choose a monotonic136 network [ 24 ] and constrain hθ to satisfy the condition hθ ( 1/t ) ≥ 1t dhθ ( 1/t ) dt for all t > 0 ( see the137 Supplementary material for further information on how to impose the condition on hθ ) . In the138 next paragraphs , we propose an objective and a learning strategy to train the parameters of the two139 parametrizations.140 3.2 Objective and Learning Algorithm141 The goal here is to devise a criterion to find the balancing function with the fastest speed of conver-142 gence/mixing on the target distribution p. Note that the ideal case would be to sample from p in an143 independent fashion . We have already mentioned that this operation is computationally expensive144 due to the intractability of computing the normalizing constant . In our case , we have to consider145 the agnostic case , because the proposal distribution is a tractable surrogate for p. In this regard , we146 define a criterion taking into account the distance from this ideal case . Specifically , we measure the147 distance of the transition probability of the Markov chain in Eq . 1 from the target p and the amount148 of dependence between consecutive samples generated through it . In other words , we introduce the149 following criterion , which is indeed a form of mutual information objective:150 Iθ = KL { p ( x ) T̃θ ( x ′|x ) ‖p ( x ) p ( x′ ) } = Ep { KL { T̃θ ( x′|x ) ‖p ( x′ ) } } ( 3 ) where KL is the Kullback Leibler divergence and Ep is the expected value of random vector x151 distributed according to p and152 T̃θ ( x ′|x ) .= { Tθ ( x ′|x ) ZT ∀x′ 6= x 0 otherwise ( 4 ) is a conditional distribution defined over the transition probability Tθ ( x′|x ) in Eq . 1 , where we have153 explicited the dependence on θ and we have introduced the normalizing constant ZT , to ensure154 that T̃θ ( x′|x ) is a proper probability distribution . Note also that T̃θ ( x′|x ) discards all pair of equal155 samples , i.e . x′ = x , as they are samples rejected by the Markov chain.156 Minimizing Eq . 3 allows us to find the configuration of parameters bringing us `` closer '' in terms157 of Kullback Leibler to the ideal case , namely Tθ ( x′|x ) = p ( x′ ) for all x′ 6= x . The expectation in158 Eq . 3 requires access to samples from p and therefore can not be computed . Nevertheless , note that159 the KL term in Eq . 3 can be rewritten in an equivalent form ( see the Supplementary material for the160 derivation ) :161 KL { T̃θ ( x′|x ) ‖p ( x′ ) } ∝ J ( θ , x ) . = EQθ0 { ωθ , θ0Aθ ( x ′ , x ) log Aθ ( x ′ , x ) Qθ ( x ′|x ) p̃ ( x′ ) } ( 5 ) where ωθ , θ0 = Qθ ( x ′|x ) Qθ0 ( x ′|x ) and θ0 is the reference parameter vector for the proposal distribution.162 Alternatively to Eq . 3 , we can minimize the following quantity:4163 J ( θ ) = EQinit { J ( θ , x ) } + EQθ0 { J ( θ , x ) } ( 6 ) where Qinit is the distribution used at initialization , typically uniform on the support X . Note that164 the first and the second terms in Eq . 6 encourage fast burn-in and fast mixing , respectively . Therefore,165 θ can be learnt using the procedure described in Algorithm 1.166 4See the Supplementary material for the modification of the objective for parametrization LSB 2 . Algorithm 1 : Local Self-Balancing Training Procedure . Learning rate η = 1e− 4 , initial parameter θ0 , burn-in iterations K and batch of samples N . { x ( i ) 0 } Ni=1 ∼ Qinit ; while k=1 : K do { x ( i ) init } Ni=1 ∼ Qinit ; while i=1 : N do Ĵ ( i ) ′ ← Estimate of J ( θ , x ( i ) init ) using one sample from Qθ0 ( x|x ( i ) init ) ; Ĵ ( i ) ← Estimate of J ( θ , x ( i ) 0 ) using one sample from Qθ0 ( x|x ( i ) 0 ) ; end Ĵ ( θ ) = 1N ∑N i=1 Ĵ ( i ) ′ + 1 N ∑N i=1 Ĵ ( i ) ; θ ← θ − ηN∇θ Ĵ ( θ ) ; Update { x ( i ) 0 } Ni=1 with accepted samples ; θ0 ← θ ; end 4 Related Work167 It ’ s important to devise strategies , which enable the automatic adaption of proposals to target168 distributions , not only to reduce user intervention , but also to increase the efficiency of MCMC169 samplers [ 5 , 13 ] . Recently , there has been a surge of interest in using machine learning and in170 particular deep learning to learn proposals directly from data , especially in the continuous domain.171 Here , we provide a brief overview of recent integrations of machine learning and MCMC samplers172 according to different parametrizations and training objectives.173 Parametrizations and objectives in the continuous domain . The work in [ 27 ] proposes a strategy174 based on block Gibbs sampling , where blocks are large motifs of the underlying probabilistic graphical175 structure . It parameterizes the conditional distributions of each block using mixture density networks176 and trains them using meta-learning on a log-likelihood-based objective . The work in [ 25 ] considers177 a global sampling strategy , where the proposal is parameterized by a deep generative model . The178 model is learnt through adversarial training , where a neural discriminator is used to detect whether or179 not generated samples are distributed according to the target distribution . Authors in [ 10 ] propose a180 global sampling strategy based on MCMC with auxiliary variables [ 12 ] . The proposals are modelled181 as Gaussian distributions parameterized by neural networks and are trained on a variational bound182 of a log-likelihood-based objective . The works in [ 15 , 8 ] propose a gradient-based MCMC [ 7 , 9 ] ,183 where neural models are used to learn the hyperparameters of the equations governing the dynamics184 of the sampler . Different objectives are used during training . In particular , the work in [ 8 ] uses185 a log-likelihood based objective , whereas the work in [ 15 ] considers the expected squared jump186 distance , namely a tractable proxy for the lag-1 autocorrelation function [ 21 ] . The work in [ 30 ] 187 proposes a global two-stage strategy , which consists of ( i ) sampling according to a Gaussian proposal188 and ( ii ) updating its parameters using the first- and second-order statistics computed from a properly189 maintained pool of samples . The parameter update can be equivalently seen as finding the solution190 maximizing a log-likelihood function defined over the pool of samples . Finally , the work in [ 22 ] 191 extends this last strategy to the case of Gaussian mixture proposals . All these works differ from the192 current one in at least two aspects . Firstly , it is not clear how these parametrizations can be applied193 to sampling in the discrete domain . Secondly , the proposed objectives compute either a distance194 between the proposal distribution and the target one , namely using an adversarial objective or a195 variational bound on the log-likelihood , or a proxy on the correlation between consecutive generated196 samples , namely the expected squared jump distance . Instead , our proposed objective is more general197 in the sense that it allows to ( i ) reduce the distance between the proposal and the target distribution as198 well as to ( ii ) reduce the statistical dependence between consecutive samples , as being closely related199 to mutual information.200 Sampling in the discrete domain . Less efforts have been devoted to devise sampling strategies201 for a purely discrete domain . Most of the works consider problem relaxations by embedding202 the discrete domain into a continuous one , applying existing strategies like Hamiltonian Monte203 Carlo [ 29 , 20 , 1 , 6 , 19 ] on it and then moving back to the original domain . These strategies are204 suboptimal , either because they consider limited settings , where the target distribution has specific205 analytic forms [ 29 ] , or because they make strong assumptions on the properties of the embeddings,206 thus not preserving the topological properties of the discrete domain [ 20 , 1 , 6 , 19 ] .5 . The work207 in [ 28 ] provides an extensive experimental comparison between several discrete sampling strategies,208 including the ones based on embeddings , based on stochastic local search [ 11 ] and the Hamming ball209 sampler [ 26 ] , which can be regarded as a more efficient version of block Gibbs sampling . Notably,210 the sampling strategy based on locally informed proposals and balancing functions proposed in [ 28 ] 211 can be considered as the current state of the art for discrete MCMC . Our work builds and extends212 upon this sampler by integrating it with a machine learning strategy . To the best of our knowledge,213 this is the first attempt to consider the integration of machine learning and MCMC in the discrete214 setting.215 5 Experiments216 Firstly , we analyze samplers ’ performance on the 2D Ising model . Then , we perform experiments217 on additional UAI benchmarks . Code to replicate the experiments in this section is available in the218 Supplementary material . All experiments are performed on a laptop provided with 4 Intel i5 cores ( 2219 GHz ) and 16 GB of RAM memory.220 5.1 2D Ising Model221 The Ising model has been introduced in statistical mechanics in 1920 and it has been applied in222 several domains since then . In this section , we consider an application to image analysis , where the223 goal is to segment an image to identify an object from its background . Consider a binary state space224 X = { −1 , 1 } V , where ( V , E ) defines a square lattice graph of the same size of the analyzed image,225 namely n× n. For each state configuration x = ( xi ) i∈V ∈ X , define a prior distribution226 pprior ( x ) ∝ exp { λ ∑ ( i , j ) ∈E xixj } where λ is a non-negative scalar used to weight the dependence among neighboring variables in the227 lattice . Then , consider that each pixel yi is influenced only by the corresponding hidden variable228 xi and generated according to a Gaussian density with mean µxi and variance σ2 . Note that each229 variable in the lattice tells whether the corresponding pixel belongs to the object or to the background230 ( 1 or -1 , respectively ) . The corresponding posterior distribution of a hidden state x given an observed231 image is defined as follows:232 p ( x ) = 1 Z exp { ∑ i∈V αixi + λ ∑ ( i , j ) ∈E xixj } ( 7 ) where αi = yiµ/σ2 is a coefficient biasing xi towards either 1 or −1 . Therefore , α = ( αi ) i∈V233 contains information about the observed image . Figure 1 shows two synthetically generated examples234 of α . Our goal is to analyze the sampling performance on the distribution defined in Eq . 7.235 Universally optimal balancing function . We start by comparing the balancing functions proposed236 in [ 28 ] , namely g ( t ) = t/ ( 1+t ) ( a.k.a Barker function ) , √ t , min { 1 , t } and max { 1 , t } , in order to test237 5For example by considering transformations that are bijective and/or by proposing transformations which allow to tractably compute the marginal distribution on the continuous domain . whether there is a universally optimal balancing function among this subset . In particular , we run the238 samplers over two instances of the Ising model , viz . a setting with independent ( λ , µ , σ ) = ( 0 , 1 , 3 ) 239 and another one with dependent ( λ , µ , σ ) = ( 1 , 1 , 3 ) variables . We evaluate the performance over 30240 repeated trials by analyzing the convergence speed during the burn-in phase ( using traceplots ) and241 the mixing time ( computing the autocorrelation function and the effective sample size ) . We visualize242 the corresponding results in Figure 2 and report the quantitative performance in Table 1 . Further243 details about the simulations are available in the Supplementary Material.244 By comparing the performance of convergence speed and mixing time in Figure 2 , we observe that245 unbounded functions , like max { 1 , t } and √ t , tend to converge faster while having slower mixing246 compared to the other two functions , thus being in line with the empirical findings of [ 28 ] . This is due247 to the fact that unbounded functions have an intrinsic preference for visiting more likely regions at the248 cost of a reduced amount of exploration . Moving a step further , we compare Figure 2a and Figure 2b249 and observe that the optimal function is different for the two cases ( i.e . max { 1 , t } in the independent250 case and √ t in the dependent one ) . These results suggest that optimality not only depends on the251 performance of burn-in and mixing but also on the distribution we are sampling from . This allows252 us to reject the hypothesis about the existence of a universal optimum among the pool of balancing253 functions proposed in [ 28 ] and to motivate our next set of experiments , where the aim is to learn to254 adapt the balancing function to the target distribution.255 Learning the balancing function . We compare the four balancing functions used in the previous256 set of experiments with our two parametrizations on four different settings of the Ising model , namely257 independent and noisy ( λ , µ , σ ) = ( 0 , 1 , 3 ) , independent and clean ( λ , µ , σ ) = ( 0 , 3 , 3 ) , dependent258 and noisy ( λ , µ , σ ) = ( 1 , 1 , 3 ) and dependent and clean ( λ , µ , σ ) = ( 1 , 3 , 3 ) cases and show the259 corresponding performance in Figure 3 and Table 2 . We leave additional details and results to the260 Supplementary Material.261 From Figure 3 , we can see that our first parametrization LSB 1 is able to always `` select '' an unbounded262 balancing function during burn-in , while when approaching convergence it is able to adapt to preserve263 fast mixing , as measured by the effective sample size in Table 2 . It ’ s interesting to mention also that264 the softmax nonlinearity used in LSB 1 can sometimes slow down the adaptation due to vanishing265 gradients . This can be observed by looking at the case 4 of Figure 3 , where for a large part of the266 burn-in period the strategy prefers max { 1 , t } over √ t. Nevertheless , it is still able to recover a267 solution different from max { 1 , t } at the end of burn-in , as confirmed by the larger effective sample268 size in Table 2 compared to the one achieved by max { 1 , t } .269 Furthermore , we observe that our second parametrization LSB 2 , which is functionally more expres-270 sive compared to LSB 1 , allows to outperform all previous cases in terms of convergence speed , while271 preserving optimal mixing , as shown in Figure 3 and Table 2 . This provides further evidence that272 the optimality of the balancing function is influenced by the target distribution and that exploiting273 such information can dramatically boost the sampling performance ( e.g . in case 3 of Figure 3 , LSB 2274 converges twice time faster as the best balancing function √ t ) . Figure 4 provides some realizations275 obtained by the samplers for the cases with dependent variables λ = 1 . We clearly see from these276 pictures that convergence for LSB 2 occurs at an earlier stage than the other balancing functions and277 therefore the latent variables in the Ising model converge faster to their ground truth configuration.278 5.2 Bayesian Networks : UAI data279 We evaluate how our strategy generalizes to different graph topologies compared to the one of the280 Ising model . In particular , we consider two Bayesian networks , with 100 discrete variables each281 and near-deterministic dependencies , from the 2006 UAI competition.6 Similarly to the previous282 experiments for the Ising model , we measure the performance over 5 repeated trials by analyzing283 the convergence speed during the burn-in phase ( using traceplots ) and the mixing time ( computing284 the autocorrelation function and the effective sample size ) . Further details about the simulations are285 available in the Supplementary Material.286 We observe that the proposed strategy is able to adapt to the target distribution , thus achieving fast287 convergence ( Figure 5 ) while preserving fast mixing ( both Figure 5 and Table 3 ) compared to existing288 balancing functions.289 6 Conclusion290 We have presented a strategy to learn locally informed proposals for MCMC in discrete spaces . The291 strategy consists of ( i ) a new parametrization of balancing functions and ( ii ) a learning procedure292 adapting the proposal to the target distribution , in order to improve the sampling performance , both293 in terms of convergence speed and mixing.294 Note that the LSB sampler belongs to the family of local sampling strategies , thus inheriting their295 limitations . The locality assumption can be quite restrictive , for example when sampling from discrete296 distributions with deterministic dependencies among variables . In such situations , local sampling297 might fail to correctly sample from the target in a finite amount of time , as being required to cross298 regions with zero probability mass . This remains an open challenge to be investigated in future work.299 It ’ s important to mention that this work is foundational and general . The proposed strategy reduces300 the amount of user intervention and improves the sampling efficiency . However , these results could301 have potential impacts on society . At first glance , one could argue that the automation of the sampling302 procedure could have a negative impact on society as reducing the amount of human labour required303 to run the sampling strategy . However , we think that the benefits are of a far greater number compared304 to the negative aspects . In fact , the procedure reduces the costs of domain knowledge , with the305 advantage of democratizing the sampling strategy and reducing suboptimal configurations of the306 algorithm resulting from possibly wrong human decisions . Furthermore , the proposed strategy307 introduces a small amount of additional computation , which is used to reduce the amount of queries308 to the target distribution , thus improving the query and the sampling efficiency . We think that this last309 aspect could contribute positively towards devising more energy-efficient algorithms and therefore310 being more environment friendly.311 References312 [ 1 ] H. Mohasel Afshar , J. Domke , et al . Reflection , Refraction , and Hamiltonian Monte Carlo . In313 NeurIPS , pages 3007–3015 , 2015.314 [ 2 ] N. Alexeev , J. Isomurodov , V. Sukhov , G. Korotkevich , and A. Sergushichev . Markov Chain315 Monte Carlo for Active Module Identification Problem . BMC Bioinformatics , 21 ( 6 ) :1–20 , 2020.316 [ 3 ] G. Altekar , S. Dwarkadas , J. P. Huelsenbeck , and F. Ronquist . Parallel Metropolis Coupled317 Markov Chain Monte Carlo for Bayesian Phylogenetic Inference . Bioinformatics , 20 ( 3 ) :407–318 415 , 2004.319 6http : //sli.ics.uci.edu/~ihler/uai-data/ [ 4 ] C. Andrieu , N. De Freitas , A. Doucet , and M. I. Jordan . An Introduction to MCMC for Machine320 Learning . Machine learning , 50 ( 1 ) :5–43 , 2003.321 [ 5 ] C. Andrieu and J. Thoms . A Tutorial on Adaptive MCMC . Statistics and Computing , 18 ( 4 ) :343–322 373 , 2008.323 [ 6 ] V. Dinh , A. Bilge , C. Zhang , and Frederick A. Matsen I. V. Probabilistic Path Hamiltonian324 Monte Carlo . In ICML , pages 1009–1018 , 2017.325 [ 7 ] S. Duane , A. D. Kennedy , B. J. Pendleton , and D. Roweth . Hybrid Monte Carlo . Physics letters326 B , 195 ( 2 ) :216–222 , 1987.327 [ 8 ] W. Gong , Y. Li , and J. M. Hernández-Lobato . Meta-Learning for Stochastic Gradient MCMC.328 In ICLR , 2019.329 [ 9 ] U. Grenander and M. I. Miller . Representations of Knowledge in Complex Systems . Journal of330 the Royal Statistical Society : Series B ( Methodological ) , 56 ( 4 ) :549–581 , 1994.331 [ 10 ] R. Habib and D. Barber . Auxiliary Variational MCMC . In ICLR , 2018.332 [ 11 ] C. Hans , A. Dobra , and M. West . Shotgun Stochastic Search for “ Large P ” Regression . Journal333 of the American Statistical Association , 102 ( 478 ) :507–516 , 2007.334 [ 12 ] D. M. Higdon . Auxiliary Variable Methods for Markov Chain Monte Carlo with Applications.335 Journal of the American statistical Association , 93 ( 442 ) :585–595 , 1998.336 [ 13 ] M. D. Hoffman and A. Gelman . The No-U-Turn Sampler : Adaptively Setting Path Lengths in337 Hamiltonian Monte Carlo . J. Mach . Learn . Res. , 15 ( 1 ) :1593–1623 , 2014.338 [ 14 ] D. Koller and N. Friedman . Probabilistic Graphical Models : Principles and Techniques . MIT339 press , 2009.340 [ 15 ] D. Levy , Matt D. H. , and J. Sohl-Dickstein . Generalizing Hamiltonian Monte Carlo with Neural341 Networks . In ICLR , 2018.342 [ 16 ] D. J. C. MacKay . Information Theory , Inference and Learning Algorithms . Cambridge university343 press , 2003.344 [ 17 ] R. Neal . Probabilistic Inference Using Markov Chain Monte Carlo Methods . Department of345 Computer Science , University of Toronto Toronto , Ontario , Canada , 1993.346 [ 18 ] E. Nijkamp , M. Hill , T. Han , S. C. Zhu , and Y. N. Wu . On the Anatomy of MCMC-Based347 Maximum Likelihood Learning of Energy-Based Models . In AAAI , pages 5272–5280 , 2020.348 [ 19 ] A. Nishimura , D. B. Dunson , and J. Lu . Discontinuous Hamiltonian Monte Carlo for Discrete349 Parameters and Discontinuous Likelihoods . Biometrika , 107 ( 2 ) :365–380 , 2020.350 [ 20 ] A. Pakman and L. Paninski . Auxiliary-Variable Exact Hamiltonian Monte Carlo Samplers for351 Binary Distributions . In NeurIPS , 2013.352 [ 21 ] C. Pasarica and A. Gelman . Adaptively Scaling the Metropolis Algorithm Using Expected353 Squared Jumped Distance . Statistica Sinica , pages 343–364 , 2010.354 [ 22 ] Pompe , E. and Holmes , C. and Łatuszyński , K. and others . A Framework for Adaptive MCMC355 Targeting Multimodal Distributions . Annals of Statistics , 48 ( 5 ) :2930–2952 , 2020.356 [ 23 ] C. Robert and G. Casella . Monte Carlo Statistical Methods . Springer Science & Business357 Media , 2013.358 [ 24 ] J. Sill . Monotonic Networks . In NeurIPS , pages 661–667 , 1997.359 [ 25 ] J . Song , S. Zhao , and S. Ermon . A-NICE-MC : Adversarial Training for MCMC . In NeurIPS,360 2017.361 [ 26 ] M. K. Titsias and C. Yau . The Hamming Ball Sampler . Journal of the American Statistical362 Association , 112 ( 520 ) :1598–1611 , 2017.363 [ 27 ] T. Wang , Y. Wu , D. A. Moore , and S. J. Russell . Meta-Learning MCMC Proposals . In NeurIPS,364 pages 4150–4160 , 2018.365 [ 28 ] G. Zanella . Informed Proposals for Local MCMC in Discrete Spaces . Journal of the American366 Statistical Association , 115 ( 530 ) :852–865 , 2020.367 [ 29 ] Y. Zhang , Z. Ghahramani , A. J. Storkey , and C. Sutton . Continuous Relaxations for Discrete368 Hamiltonian Monte Carlo . NeurIPS , 25:3194–3202 , 2012.369 [ 30 ] M. Zhu . Sample Adaptive MCMC . In NeurIPS , volume 32 , 2019.370 Checklist371 1 . For all authors ... 372 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s373 contributions and scope ? [ Yes ] 374 ( b ) Did you describe the limitations of your work ? [ Yes ] We included a paragraph in the375 conclusions ( Section 6 ) discussing the main limitations of our strategy.376 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] We377 included a paragraph in the conclusions ( Section 6 ) discussing both pros and cons of378 the proposed work from the societal perspective.379 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to380 them ? [ Yes ] 381 2 . If you are including theoretical results ... 382 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] Please check383 the three propositions in Section 3.384 ( b ) Did you include complete proofs of all theoretical results ? [ Yes ] Proofs are shown in385 the main paper in Section 3.386 3 . If you ran experiments ... 387 ( a ) Did you include the code , data , and instructions needed to reproduce the main exper-388 imental results ( either in the supplemental material or as a URL ) ? [ Yes ] Code and389 instructions to replicate the experiments in the paper are available in the Supplementary390 Material.391 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they were392 chosen ) ? [ Yes ] The Supplementary Material provide all details about the experiments.393 Furthermore , we are sharing the scripts to run all experiments.394 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-395 ments multiple times ) ? [ Yes ] 396 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type397 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] At the beginning of Section 5 , we398 have described the resources used to run the experiments.399 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 400 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] The only external401 asset used in this work is the UAI benchmark data , which is publicly available at402 http : //sli.ics.uci.edu/~ihler/uai-data/ . We have explicitly mentioned the403 source of the data in the main paper.404 ( b ) Did you mention the license of the assets ? [ Yes ] UAI data is freely accessible online405 without any explicit license , that means that copyright law applies in this case . We will406 not share the data , but only the source code and include a README file with a pointer407 to the main website and credit the main author.408 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 409 We have included the URL of the UAI dataset in the main paper.410 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re411 using/curating ? [ Yes ] See answer ( b ) .412 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable413 information or offensive content ? [ N/A ] This doesn ’ t apply to UAI data.414 5 . If you used crowdsourcing or conducted research with human subjects ... 415 ( a ) Did you include the full text of instructions given to participants and screenshots , if416 applicable ? [ N/A ] 417 ( b ) Did you describe any potential participant risks , with links to Institutional Review418 Board ( IRB ) approvals , if applicable ? [ N/A ] 419 ( c ) Did you include the estimated hourly wage paid to participants and the total amount420 spent on participant compensation ? [ N/A ] 421

Summary: This paper improves upon a recently proposed MCMC method ([Zanella. JASA 2020], cited as [28] in the paper) and considers optimizing the choice of balancing function in that method. A collection of rigorous techniques are proposed so that the balancing function can be represented by a neural network, and the network is trained by minimizing a heuristic objective (two possibilities are provided), which seems to be an approximation of the KL divergence from a Metropolis-Hasting proposal to the target distribution. Limited empirical evidence of effectiveness is provided.