A Neural Network MCMC sampler that maximizes Proposal Entropy
Authors: ZENGYI LI
Yubei Chen
Friedrich Sommer
Venue:  (2021)
Keywords: MCMC
Adaptive MCMC
Neural MCMC
Normalizing Flow
Entropy based speed Measure
HMC
Energy-based Model
Sampling

Abstract: Markov Chain Monte Carlo (MCMC) methods sample from unnormalized probability distributions and offer guarantees of exact sampling. However, in the continuous case, unfavorable geometry of the target distribution can greatly limit the efficiency of MCMC methods. Augmenting samplers with neural networks can potentially improve their efficiency. Previous neural network based samplers were trained with objectives that either did not explicitly encourage exploration, or used a L2 jump objective which could only be applied to well structured distributions. Thus it seems promising to instead maximize the proposal entropy for adapting the proposal to distributions of any shape. To allow direct optimization of the proposal entropy, we propose a neural network MCMC sampler that has a flexible and tractable proposal distribution. Specifically, our network architecture utilizes the gradient of the target distribution for generating proposals. Our model achieves significantly higher efficiency than previous neural network MCMC techniques in a variety of sampling tasks. Further, the sampler is applied on training of a convergent energy-based model of natural images. The learned sampler achieves significantly higher proposal entropy and sample quality compared to Langevin dynamics sampler.

Under review as a conference paper at ICLR 2021
A NEURAL NETWORK MCMC SAMPLER THAT MAXI- MIZES PROPOSAL ENTROPY
Anonymous authors Paper under double-blind review
ABSTRACT
Markov Chain Monte Carlo (MCMC) methods sample from unnormalized prob- ability distributions and offer guarantees of exact sampling. However, in the con- tinuous case, unfavorable geometry of the target distribution can greatly limit the efficiency of MCMC methods. Augmenting samplers with neural networks can potentially improve their efficiency. Previous neural network based samplers were trained with objectives that either did not explicitly encourage exploration, used a 1.2 jump objective which could only be applied to well structured distributions. Thus seems promising instead maximize the proposal entropy for adapting the proposal to distributions of any shape. To allow direct optimization of the pro- posal entropy, we propose neural network MCMC sampler that has flexible and tractable proposal distribution. Specifically, our network architecture utilizes the gradient of the target distribution for generating proposals. Our model achieves significantly higher efficiency than previous neural network MCMC techniques in variety of sampling tasks. Further, the sampler is applied on training of a convergent energy-based model of natural images. The learned sampler achieves significantly higher proposal entropy and sample quality compared to Langevin dynamics sampler.
INTRODUCTION
Sampling from unnormalized distributions is important for many applications, including statisties, simulations physical systems, and machine learning. However, the inefficiency of state-of-the-ar sampling methods remains main bottleneck for many challenging applications, such as protein folding (Noé et al. 2019), energy-based model training (Nijkamp et al. 2019), etc.
A prominent strategy for sampling is the Markov Chain Monte Carlo (MCMC) method (Neal) 1993).
In MCMC, one chooses transition kernel that leaves the target distribution invariant and constructs Markov Chain by applying the kernel repeatedly. The MCMC method relies only on the ergodic- ity assumption, other than that it is general. If enough computation is performed, the Markov chain generates correct samples from any target distribution, no matter how complex the distribution is. However, the performance of MCMC depends critically on how well the chosen transition kernel ex- plores the state space of the problem. exploration ineffective, samples will be highly correlated and of very limited use for downstream applications. Despite some favorable theoretical argument on the effectiveness of some MCMC algorithms, practical implementation of them may still suffer from inefficiencies.
Take, for example, the Hamiltonian Monte Carlo (HMa) et al. 2011) algorithm, a type of MCMC technique. HMC is regarded state-of-the-art for sampling in continuous spaces Radivojevié & Akhmatskaya (2020). It uses a set of auxiliary momentum variables and generates new samples by simulating Hamiltonian dynamics starting from the previous sample. This allows the sam- ple to travel in state space much further than possible with other techniques, most of whom have more pronounced random walk behavior. Theoretical analysis shows that the cost of traversing d-dimensional state space and generating an uncorrelated proposal is O(dt) for HMC, which is lower than Odd)) for Langevine Monte Carlo, and (dd) for random walk. However, unfavorable geometry of target distribution may still cause HMC to be ineffective because the Hamiltonian dynamics has to be simulated numerically. Numerical errors in the simulation are commonly cor- rected by Metropolis- (MH) accept-reject step for proposed sample. If the the target

---

Under review as a conference paper at ICLR 2021
Sampler stay close identity function if training objective does not encourage exploration
Proposal learned by Entropy- exploration speed objective covers target distribution well.
A less desirable proposal distribu- tion with higher L2 expectedjjupp
Figure 1: Illustration of learning explore state space. Larger yellow dot in top left is the ini-
tial point x, blue and black dots are accepted and rejected samples from the proposal distribution q(x1lz). Solution obtained from optimizing entropy objective is close the target distribution p(x)). However, we can easily construct a less desirable solution with higher L2 jump.
distribution has unfavorable geometric properties, for example, very different variances along dif- ferent directions, the numerical integrator in HMC will have high error, leading to a very low accept probability (Betancourt et al. 2017). For simple distributions this inefficiency can be mitigated by an adaptive re-scaling matrix (Neal et al., (01I)) For analytically tractable distributions, one can also use the Riemann manifold HMC method (Girolami Calderhead 2011). But in most other cases, the Hessian required in Riemann manifold HMC algorithm is often intractable expensive to compute, preventing application.
Recently, approaches have been proposed that possess the exact sampling property of the MCMC method, while potentially mitigating the described issues with unfavorable geometry. Such ap- proaches include MCMC samplers augmented with neural networks (Song et al., 2017 Levy et al.) 2018 Gu et al.) 2019), and neural transport MCMC techniques (Hoffman et al. 2019 Nijkamp al. 2020). A disadvantage of these recent techniques is that their objectives optimize the quality of proposed samples, but do not explicitly encourage exploration speed of the sampler. One notable exception is 1.2HMC (Levy al., 2018). a method whose objective includes the size of the expected 1.2 jump. thereby encouraging exploration. But the L.2 expected jump objective is not very general, only works for simple distributions (see Figure [ and below).
Another recent work (Titsias & Dellaportas) 2019) proposed a quite general objective to encourage exploration speed by maximizing the entropy of the proposal distribution. In continuous space, the entropy of distribution is essentially the logarithm of its volume in state space. Thus, the entropy objective naturally encourages the proposal distribution to "fill up" the target state space as well as possible, independent of the geometry of the target distribution. The authors demonstrated the effectiveness of this objective on samplers with simple linear adaptive parameters.
Here we employ the entropy-based objective in a neural network MCMC sampler for optimizing exploration speed. To build the model, we design flexible proposal distribution for which the optimization of the entropy objective is tractable. Inspired by the HMC algorithm, the proposed sampler uses special architecture that utilizes the gradient of the target distribution to aid sampling.
For 2-D distribution the behavior of the proposed model is illustrated in Figure [ The sampler, trained with the entropy-based objective, generates samples that explore the target distribution quite well, while it is simple construct a proposal with higher I.2 expected jump (right panel). Later show the newly proposed method achieves significant improvement in sampling efficiency compared to previous techniques, then apply the method to the training of an energy-basee image model.
2 PRELIMINARY: MCMC METHODS, FROM VANILLA TO LEARNED
Consider the problem of sampling from target distribution p(x) = (U() /Z defined by the energy function U(x) ) in a continuous state space. MCMC methods solve the problem by constructing and running Markov Chain, with transition probability p(x') p()'(x),, that leaves p(x) invariant. The most general invariance condition is: p(x') = ] pp=x(==)p(x)) for all x', which is typically enforced by the simpler but more stringent condition ofdetailed balance: p(x)p((') p(æ (()))((=(x))).
2

---

Under review as a conference paper at ICLR 2021
For a general distribution p(œ) it is difficult to directly construct p(z'(z) that satisfies detailed balance, but one can easily make any transition probability satisfy it by including an additional Metropolis-Hare Hastings accept-reject step (Hastings) 1970). When we sample x' at step t from an arbitrary proposal distribution g(x' (x²), the M-H accept-reject process accepts the new sample at-1 x' with probability A(a'; x) = min (1, p(="ig(=']a') If x' rejected, the new sample p(a">yla'">"" >y(a">=*))
is set to the previous state 2²+1 = x². This transition kernel p(z'[z) constructed from q(>' (z) and A(z', .x) leaves any target distribution p(z) invariant.
The most popular MCMC techniques use the described M-H accept-reject step to enforce detailed balance, for example, Random Walk Metropolis (RWM), Metropolis-Adjuste Langevin Algorithm (MALA) and Hamiltonian Monte Carlo (HMC). For brevity, we will focus on MCMC methods that use the M-H step, although some alternatives do exist (Sohl-Dickstein et al., 2014). All these methods share the requirement that the accept probability in the M-H step must be tractable to compute. For two of the mentioned MCMC methods this is indeed the case. In the Gaussian vandom-walk sampler, the proposal distribution is a Gaussian around the current position: x' = x + *N,( I), which has the form x' x x z. Thus, forward and reverse proposal probabilities are given by g( (zr) = PN [< - =x/{] and g(<](=²) > PAr [-(x' - r)/c). where pN denote the density function of Gaussian. The probability ratio g(2"=²)
g(=" "(="m=²) ) used in the M-H step is therefore equal to 1. In MALA the proposal distribution is single step of Langevin dynamics with step size e: x' = x +
with 2 (x) + (0, We then have (<<'lx) = PM [(<' - z)/E+ + [0) and + (x')) Both, the forward and reverse proposal probability are tractable since they are the density of Gaussians evaluated at a known location.
((>|=') = PN (-'=--
Next we show how the HMC sampler can also be formulated as a M-H sampler. Basic HMC in- volves a Gaussian auxiliary variable v of the same dimension as which plays the role of the momentum in Physics. HMC sampling consists of two steps: 1. The momentum is sampled from normal distribution ((;; O, I). 2. The Hamiltonian dynamics is simulated for certain duration with initial condition x and v, typically by running a few steps of the leapfrog integrator. Then, a
M-H accept-reject process with accept probability A(x', v', 2), min (I, p(x' w "'"oouwoll'"
pla,))a(2" ""(,,,
min (1, p(z')nx(())) is performed to correct for error in the integration process. We have p(a))a()))
since the Hamiltonian transition is volume-preserving over (z) Both HMC steps leave the joint distribution p(z, v) invariant, therefore HMC samples from the correct dis- tribution p(x) after marginalizing over v. To express basic HMC in the standard M-H scheme, step I and 2 can be aggregated into a single proposal distribution on x with the proposal probability: q<<'lx)) (x) (v) and q(x|x²) (t"). Note, although the probability q(>" (x) can be calculated after the Hamiltonian dynamics is simulated, this term is intractable for general z and x'. The rea- son is that it is difficult solve for the e at z to make the transition to x' using the Hamiltonian dynamics. This issue is absent in RWM and MALA, where g( ((= is tractable for any and x'.
q("", (((,e)
Previous work on augmenting MCMC sampler with neural networks also relied on the M-H proce- dure to ensure asymptotic correctness of the sampling process, for example (Song et al. (017) and (Levy et al. 2018). They used HMC style accept-reject probabilities that lead to intractable g a (zr). Here, we strive for a flexible sampler for which g(a'|xr) is tractable. This maintains the tractable M-H step while allowing us to train this sampler to explores the state space by directly optimizing the proposal entropy objective, which is function of g(x'|1x) (z).
GRADIENT BASED SAMPLER WITH TRACTABLE PROPOSAL PROBABILITY
We "abusee the power of neural networks to design a sampler that is flexible and has tractable proposal probability ((<'l<r) between any two points. However, without some extra help of the gradient of the target distribution, the sampler would be modeling a conditional distribution g( [x) with brute force, which might be possible but requires large model capacity. Thus, our method uses the gradient of the target distribution. We use an architecture similar to L2HMC (Levy et al.) 2018.. which itself was inspired by the HMC algorithm and Real et al. (0016). To quantify
Up to ergodic and aperiodic assumptions
3

---

Under review as a conference paper at ICLR 2021
the benefit of using the target distribution gradient, we provide ablation studies of our model in the Appendix A.I
3.1 MODEL SETUP AND HOW TO USE GRADIENT INFORMATION
We restrict our sampler to the simple general form x' = x + 2. As discussed in Section 2 the sampler will have tractable proposal probability if one can calculate the probability of any given 2. To fulfill this requirement, we model vector by a flow mode == f(z0; x, U), with inverse 2o = f-; æ, U). Here Zy is sampled from a fixed Gaussian base distribution. The flow model f is a flexible and trainable invertible function of conditioned on æ, U, and it has tractable Jacobian determinant w.r.t. 2. The flow model f can be viewed as a change of variable from the Gaussian base distribution =0 to The proposed sampler then has tractable forward and reverse proposal probability: q(x" (r) = pz (a' x: 2). q(x]x²) ) = Pz - x'), where Pz (2(2;x) x) = par(2)) azg is the density defined by the flow model f. Note, this sampler is ergodic and aperiodic, since q(x'|<>) * 0 for and and z', which follows from the invertibility of f. Thus, combined with the M-H step, the sampling process will be asymptotically correct. The sampling process first consists of drawing from (2u and then evaluating
0s
f x, U) and g<<'(=²). Next, the reverse 2/1 (-2(-2; 2; + z, U) is evaluated at x' = x + z to obtain the reverse proposal probability g(<lx²). Finally, the sample is accepted with the standard M-H rule.
For the flow model /. we use an architecture similar to a non-volume preserving coupling-based flow RealNVP (Dihh et al. 2016). In the coupling flow, half of the components of the state vector are kept fixed and are used to update the other half through an affine transform parameterized by neural network. The gradient of the target distribution enters our model in those affine transformations. To motivate the particular form we choose, we take a closer look at the HMC algorithm. Basic HMC starts with drawing a random initial momentum nº, followed by several steps of leapfrog integration. Let x" be the momentum variable after updates by v" and position. In leapfrog step, followed by the integrator first updates e with half step of the gradient: v" = -nー- -000-="00(<""-
a full step of x update: x" = x==1 =="-++ev" tev"', and another half step of o update: y = y" v" -0,0 -,,,0 (x").
After several steps, the overall update of x can be written as: x = x² + Ei= v", which has the form x' = x + 2 with = Σ. nd = nu - ne [a_U(=9)] < (n ((r) The equation for generating 2 through affine transformations, describes how the gradient of the target distribution, evaluated at some intermediate point of x, should be included.
2
3.2 MODEL FORMULATION
To formulate our model (Equation
we use a mask m and its complement m to update half of z's dimensions at a time. As discussed above, we include the gradient term with negative sign in the shift term. We also use an element-wise scaling on the gradient term as in (Levy et al.) 2018). However, two issues remain. First, as required by the coupling-
architecture, the gradient term can only depend on the masked version of vector Second, it is unclear where the gradient should be evaluated to sample effectively. As discussed above, the sampler should evaluate the gradient at points far away from x, similar as in HMC, to travel long distances in the state space. To handle these issues, we use another neural network R which depends takes x and the masked & as input, and evaluate gradient based on the masked z.
R. During training, R learns where the gradient should be evaluated
We denote the input network R by 5t. (Ir, m @ 2²) and the input the other networks by en = (I, m @ 2", OU + R((R))), where 3 is the Hadamard product (element wise multiply). Further, we denote the neural network outputs that parameterize the affine transform by S(em) ) Q(Em) and T(CM). For notational clarity we omit dependencies of the mask m and all neural network terms on the step number n.
Additionally, we introduce a scale parameter e, which modifies the x update to = x + = z. We also define e' = </(2N,, with N the total number of = update steps. This parameterization makes our sampler equivalent to the MALA algorithm with step size at initialization, where the neural
Forr details on flow models, Kobyzev et al. 2019 Papamakarios
2019
4

---

Under review as a conference paper at ICLR 2021
network outputs are zero. The resulting update rule is:
Td m ④ z"" m ④ (z"-le c exp(S(Em= [0U[z+ R(Ch explQ(<m +7((m=)))) (1)
m ④ zd + + @ exp ((()))) - {oU [x+ R(S @ exp(Q((m)) ))
(2)
The log determinant of N steps of transformation is:
Oz
log
= =el+1+
Σ 1 m SHOSSC S(m e S((m))
(3)
0=0
where 1 is the vector of 1-entries with the same dimension as 2.
3.3 OPTIMIZING THE PROPOSAL ENTROPY OBJECTIVE
The proposal entropy can be expressed as:
H (X'IX = =)== dx' (x' (x) log (-/[==]]= faee (=") [og (pN (z"))-log
22N
(4)
020
For each 2, aim to optimize S(x) exp [3 z)] o(z), where a(x)
A(x', x ())(<"(<x)d is the average accept probability of the proposal distribution at æ. Following Titsias & Dellaportas, 2019), transform this objective into log space and use Jensen's inequality
to obtain a lower bound:
log S(x) log ((z(,))g(<')))))) + (X x))
f log [A(x (a(aa2)))ggle (x)ll'' ( = x)) L(x)
The distribution q(a' (x)) is reparameterizable, therefore the expectation over (<<l(>>) can ex- pressed as expectation over (zo) Expanding the lower bound [(x) and ignoring the entropy of the base distribution Px)(z)), we arrive at:
L(x) = dz px(2)) min 0, log p(x') + log ((1122) p(x) q(<'|z)
Oz N
3 log
(5)
θz"
During training we maximize (xx) with z sampled from the target distribution p(x) if is available, or with r obtained from the bootstrapping process (Song et al., (017)) which maintains a buffer of samples and updates them continuously. Typically, only one sample of 20 is used for each x.
A curious feature of our model is that during training one has to back- propagate over the gradient of the target distribution multiple times to optimize R. In (Titsias & Dellaportas) (019) the authors avoid multiple back-propagation by stopping the derivative calculation the density gradient term. In our experiment we do not use this trick and perform full pook-propagation without encountering any issue. We found that stopping the derivative computation instead harms performance.
The entropy-based exploration objective contains a parameter 3 that controls the balance between acceptance rate and proposal entropy. As in (Titsias & Dellaportas 2019), We use simple adaptive scheme to adjust 3 to maintain a constant accept rate close to a target accept rate. The target accept rate is chosen empirically. As expected, find that the target accept rate needs to be lower for more complicated distributions.
4 RELATED WORKS: NEURAL NETWORK MCMC SAMPLERS INSPIRED BY HMC
Here discuss other neural network MCMC samplers and how they differ from method. Meth- ods we compare ours to in the Results are marked with bold font.
A-NICE-MC (Song al. 2017). which was generalized in (Spanbauer al. 2020 used the same
accept probability as HMC, but replaced the Hamiltonian dynamics by a flexible volume-preserviee flow (Dinh al. 2014). A-NICE-MC matches samples from g(x'|1>) directly samples from p(r)),
5

---

Under review as a conference paper at ICLR 2021
using adversarial loss. This permits training the sampler on empirical distributions, i..., in cases where only samples but not the density function is available. The problem with this method is that samples from the resulting sampler can be highly correlated because the adversarial objective only optimizes for the quality of the proposed sample. If the sampler produces a high quality sample J, the learning objective does not encourage the next sample x' to be substantially different from x. The authors used a pairwise discriminator that empirically mitigated this issue but the benefit in exploration speed is limited.
Another related sampling approach is Neural Transport MCMC (Marzouk et al. 2016 Hoffman al. 2019; Nijkamp et al. 22200) which fits distribution defined by a flow model Pg (z) to the target distribution using KL Py (x)x)]lp(2)) Sampling is then performed with HMC in the latent space of the flow model. Due to the invariance of the KL.-divergence with respect to a change of variables, the "transported distribution" in Z space (z) will be fitted to resemble the Gaussian prior (z). Samples of x can then be obtained by passing z through the transport map. Neural transport MCMC improves sampling efficiency compared to sampling in the original space because a distribution closer to a Gaussian is easier to sample. However, the sampling cost is not a monotonic function of the KL-divergence used to optimize the transport map. (Langmore et al. 2019).
Another line of work connects the MCMC method to Variational Inference (Salimans et al., 2015 Zhang al. 2018). Simply put, they improve the variational approximation by running several steps of MCMC transitions initialized from a variational distribution. The MCMC steps are optimized by minimizing the KL-divergence between the resulting distribution and the true posterior. This amounts to optimizing a "burn in" process in MCMC. In our setup however, the exact sampling guaranteed by the M-H process, thus the KL divergence loss is no longer applicable. Like in variational inference, the Normalizing flow Langevin MC (NFLMC) (Gu et al. (019) also used a KL divergence loss. Strictly speaking, this model is a normalizing flow but not MCMC method. We compare our method to it, because the model architecture, like ours, uses the gradient of the target distribution.
Another related technique is (Neklyudov et al. 2018), where the authors trained an independent M- sampler by minimizing KL [(())(< (>) Ip(=' "q(2) [[(])))). This objective can be viewed as lower bound of the M-H accept rate. However, discussed in (Titsias & Dellaportas) 2019), this type of objective is not applicable for samplers that condition on the previous state.
All the mentioned techniques have in common that their objective does not encourage exploration speed. In contrast, L2HMC (Levy et al. 20188] Thin et al. (2000) does encourage fast exploration of the state space by employing variant of the expected square jump objective (Pasarica & Gelman 2010 L(x)= = ʃ dx' g( A(x', when is drawn from the exact target distribution p(æ:). L2HMC generalized the Hamiltonian dynamics with flexible non-volume- -on--olll---eeerviing transformation (Dinh et al.) (2016). The architecture of L2HMC is very flexible and uses gradient of target distribution. However, the L2 expected jump objective in L2HMC improves exploration speed only in well-structured distributions (see Figure [].
, |aa z²| 2. This objective provides a learning signal even
The shortcomings of the discussed methods led us to consider the use of an entropy-based objec- tive. However, L2HMC does not have tractable proposal probability p(æ'Ian [r). preventing the direct application of the entropy-basee -asee objective. In principle, the proposal entropy objective could be optimized for the L2HMC sampler with variational inference (Poole et al.) 20199 Song Ermon 2019). but our preliminary experiments using this idea were not promising. Therefore, we designed our sampler that possess tractable proposal probability and investigated tractable optimization of the proposal entropy objective.
EXPERIMENTAL RESULT
5.1 SYNTHETIC DATASET AND BAYESIAN LOGISTIC REGRESSION
First we demonstrate that our technique accelerates sampling of the funnel distribution, a particularly pathological example from (Neal 2003). We then compare our model with A-NICE-MC (Song et al. 2017), L2HMC (Levy et al. 2018), Normalizing flow Langevin MC (NFLMC) (Gu et al.
2019 as well NeuTra (Hoffman et al. 2019) on several other synthetic datasets and Bayesian

---

Under review as a conference paper at ICLR 2021
Dutaset (measure)
L2HMC
Ours
50d ICG (ESS/HH)
0.783
0.86
24 SCG (ESS/MH)
0.497
0.89
ICG (FSS/grd)
7.83
10
24 SCG (ESS/gmd)
2.32 10-2
2.2 10
Datasct (mensure)
Neutra
Oues
Funnel-1 I zo (ESSigrad)
8.9 10
3.7 10
Funnel-1 (SS/grrd))
10
Dutaset (measure)
gradMALA
A-NICE-MC NFLMC
Ours
German (ESS:55)
702.05 871.5
926.49
1176.8
3150
Australion (FSS/5K)
1015.75
1586.4
2950
Heart (ESS/SK)
1251,16
Figure 2: Comparison with HMC on the 20d
Table I: Performance Comparisons. SCG:
Funnel-3 distribution. a) Chain and samples of To (from neck to base direction) for HMC. b) Same as a) but for our learned sampler. Note, samples in a) look significantly more correlated than those in b), although they are plotted over
strongly correlated Gaussian, ICG: Ill- conditioned Gaussian. German, Autralian, Heart: Datasets for Bayesian Logistic regres- sion. ESS: Effective Sample Size (a correlation measure)
longer time scale.
logistic regression task. We additionally compare to gradMALA Titsias & Dellaportas 2019) to show the benefit of using neural network over linear adaptive sampler. For all experiments, we report Effectiie Sample Size (Hoffman & Gelman 2014) per M-H step (ESS/MH) and/or ESS per target density gradient evaluation (ESS/grad). All results are given in minimum ESS over all dimensions unless otherwise noted.
Here a brief description of the datasets used in our experiments:
III Conditioned Gaussian: 50d ill-conditioned Gaussian task described in (Lev et al.) 2018 2018),
Gaussian with diagonal covariance matrix with log-linearly distributed entries between [10 2 10
Strongly correlated Gaussian: 2d Gaussian with variance [10? 10=11 rotated by z. same in (Levy et al. 2018).
The density function is Pfunner . M(ro; 0, o² (Z O, Txp (xpp-10 (-2xr) This is a challenging distribution because the spatial scale of Z1:n varies drastically depending on the value of To. This geometry causes problems to adaptation algorithms that rely spatial scale. An important detail is that earlier work, such as (Betancourt) 2013) used a = 3, while some recent works used 1. We run experiments with 1 for comparison with recent techniques and also demonstrate our method on a 20 dimensional funnel distribution with a 3. We denote the two variants by Funnel-1 versus Funnel-3.
Funnel distribution:
Bayesian Logistic regression: We follow the setup in (Hoffman & Gelman 2014) and used Ger- man, Heart and Australian datasets from the UCI data registry.
In Figure we compare our method with HMC on the 20d Funnel-3 distribution. As discussed in Betancourt 20113), the stepsize of HMC needs to be manually tuned down allow traveling into the neck of the funnel, otherwise the sampling process will be biased. We thus tune the stepsize of HMC to be the largest that still allows traveling into the neck. Each HMC proposal set to use the same number of gradient steps as each proposal of the trained sampler. As can be seen, the samples proposed by our method travel significantly further than the HMC samples. Our method achieves 0.256 (ESS/MH), compared to 0.0079 (ESS/MH) with HMC.
As demonstration we provide visualization of the resulting chain of samples in Figure and the learned proposal distributions in Appendix A.2 The energy value for the neck of the funnel can be very different than for the base, which makes it hard for methods such as HMC to mix between them (Betancourt) 2013). In contrast, model can produce very asymmetric q(= [2) and q(<[>"), making mixing between different energy levels possible.
Performances other synthetic datasets and the Bayesian Logistic Regression shown in Table In all these datasets our method outperformed previous neural network based MCMC approaches by significant margin. Our model also outperform gradMALA (Titsias & Dellaportas 2019). which

---

Under review as a conference paper at ICLR 2021
a)
b)
Figure 3: Training of convergent EBM with pixel space sampling. a) Samples from replay buffer after training. b) Proposal entropy of trained sampler vs MALA early during training, learned sam- pler has significantly higher entropy, and achieves better FID at convergence. e) Samples from 100k sampling steps by the learned sampler, initialized at samples from replay buffer. Large transitions like the one in the first row is rare, here its selected for display.
use the same objective but only use linear adaptive parameters. The experiments used various pa- rameter settings, as detailed Appendix [A.2 Results of other models are adopted or converted from numbers reported in the original papers. The Appendix provides further experimental results, ablation studies, visualizations and details on the implementation the model.
5.2 TRAINING A CONVERGENT DEEP ENERGY-BASER MODEL
very challenging application of the MCMC method is training a deep energy-based model of images (Xie et al. 2016; Nijkamp et al. 2019 Du & Mordatch, 2019). We demonstrate stable training of a convergent EBM, and that the learned sampler achieves better proposal entropy early during training, as well better sample quality at convergence, compared to the MALA algorithm. An added benefit that, like in adaptive MALA, tuning the Langevin dynamics step size is no longer needed, instead, one only need to specify target accept rate. This contrast with unadjusted Langevin dynamics used in previous works, where step size need to be carefully (uned(Nijkamy et al. 2019).
Similar to (Nijkamp et al., 2019), we use the Oxford flowers dataset of 8189 28+28 colored images. We dequantize the images 5bits by adding uniform noise and use logit transform (Dinh et al.) 2016). Sampling is performed in the logit space with variant 2 of the sampler that does not have R network (See Appendix A.II. During training, we use Persistent Contrastive Divergence (PCD) Tieleman] 2008) with replay buffer size of 10000. We alternate between training the sampler and updating samples for the EBM training. Each EBM training step uses 40 sampling steps, with target accept rate of 0.6.
Figure 3 depicts samples from the trained EBM replay buffer, as well as samples from a 100k step sampling process -for demonstrating stability of the attractor basins. We also show that the proposal entropy of the learned sampler is higher early during training than that of an adaptive MALA algorithm with the same accept rate target. Later during training, the proposal entropy is not significant different (See Figure A.3 A.3Ja)). a)). This is likely because the explorable volume around samples becomes too small for the learned sampler to make difference. Additionally, we show the model trained with the learned sampler achieves better sample quality by calculating the FID (Heusel al. 2017) between the replay buffer for late checkpoint and ground truth data. Model trained with learned sampler achieves 38.1 FID, while model trained with MALA achieves 43.0 FID (lower is better). We provide a plot that tracks the FID during training in Appendix Figure A.3
8

---

Under review as a conference paper at ICLR 2021
6 DISCUSSION
In this paper we propose gradient based neural network MCMC sampler with tractable proposal probability. The training is based on the entropy-based exploration speed objective. Thanks to an objective that explicitly encouraging exploration, our method achieves better performance than previous neural network based MCMC samplers on a variety of tasks. Compared to the manifold HMC (Betancourt) 2013) methods, our model provides a more scalable alternative for mitigating unfavorable geometry in the target distribution.
There are many potential applications of our method beyond what was demonstrated in this paper. For example, training latint-variable models (Hoffmann 2017), latent sampling in GANs (Che al.) 2020), and others applications outside machine learning, such molecular dynamics simulation (Noé et al. 2019). In the future, architectural improvement would also be interesting, use auto- regressive architecture different masking strategy may improve the expressiveness of model. will also be interesting to combine our technique with neural transport MCMC.
REFERENCES
Michael Betancourt. general metric for riemannian manifold hamiltonian monte carlo. In Inter- national Conference on Geometric Science of Information, pp. p 327-334. Springer, 2013.
Michael Betancourt, Simon Byrne, Sam Livingstone, Mark Girolami, al. The geometric founda- tions of hamiltonian monte carlo. Bernoulli, 23(4A):2257-2298, 2017.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your gan is secretly an energy-based model and you should use discriminator driven latent sampling. arXiv preprint arXiv:2003.0mm 06060, 2020.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv:/410.85/6 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 000.00000000,,3,, 2016.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Mark Girolami and Ben Calderhead. Riemann manifold langevin and hamiltonian monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73(2): 123-214, 2011.
Will Grathwohl, Kuan-Chieh Wang, Joern-Henrie Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations, 2019.
Minghao Gu, Shiliang Sun, and Yan Liu. Dynamical sampling with langevin normalization flows. Entropy, 21(11): 2019.
W.K. Hastings. Monte carlo sampling methods using markov chains and their applications. Biometrika, 57(1)997-109 1970.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Matthew Hoffman, Pavel Sountsov, Joshua V Dillon, Ian Langmore, Dustin Tran, and Srinivas Vasudevan. Neutra-lizing bad geometry in hamiltonian monte carlo using neural transport. arXiv preprint arXiv: 1XX3:1903.03704 2019.
Matthew D Hoffman. Learning deep latent gaussian models with markov chain monte carlo. In International conference machine learning, pp. 1510-1519, 2017.

---

Under review as a conference paper at ICLR 2021
Matthew D Hoffman and Andrew Gelman. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15(1):1593-166 1623, 2014.
Ivan Kobyzev, Simon Prince, and Marcus Brubaker. Normalizing flows: Introduction and ideas. arXiv preprini arXiv: 190000999..097 2019.
Ian Langmore, Michael Dikovsky, Scott Geraedts, Peter Norgaard, and Rob Von Behren. A condi- tion number for hamiltonian monte carlo. arXiv preprint PXXiv:190500981 2019.
Daniel Levy, Matt D Hoffman, and Jascha Sohl-Dickstein. Generalizing hamiltonian monte carlo
with neural networks. In International Conference on Learning Representations, 2018.
Youssef Marzouk, Tarek Moselhy, Matthew Parno, and Alessio Spantini. An introduction to sam- pling via measure transport. arXiv preprint arXiv:1602.0502: 2016.
Radford M Neal. Probabilistic inference using Markov chain Monte Carlo methods. Department of Computer Science, University of Toronto Toronto, Ontario, Canada, 1993.
Radford M Neal. Slice sampling. Annals of statistics, pp. 705-741, 2003.
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
Kirill Neklyudov, Evgenii Egorov, Pavel Shvechikov, and Dmitry Vetrov. Metropolis-hastings view on variational inference and adversarial training. arXiv preprint arXiv: 2018.
Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of meme- based maximum likelihood learning of energy-based models. In Proceedings of the Conference on Artificial Intelligence (AAAI), 2019.
Erik Nijkamp, Ruiqi Gao, Pavel Sountsov, Srinivas Vasudevan, Bo Pang, Song- Chug-Chun& Zhu, and Ying Nian Wu. Learning energy-based model with flow-based backbone by neural transport meme. arXiv preprint arXiv:2006.0m 066977 2020.
Frank Noé, Simon Olsson, Jonas Köhler, and Hao Wu. Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning. Science, 363(6457) 1177, 2019.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lak- shminarayanan. Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.02262,, 2019.
Cristian Pasarica and Andrew Gelman, Adaptively scaling the metropolis algorithm using expected squared jumped distance. Statistica Sinica, pp. 343-364, 2010.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander Alemi, and George Tucker. On varia- tional bounds of mutual information. arXiv preprint arXiv:1905.06922 2019.
Tijana Radivojević and Elena Akhmatskaya. Modified hamiltonian monte carlo for bayesian infer- ence. Statistics and Computing, 30(2))377 2020.
Tim Salimans, Diederik Kingma, and Max Welling. Markov chain monte carlo and variational inference: Bridging the gap. In International Conference Machine Learning, pp. 1218-1226 2015.
Jascha Sohl-Dickstein, Mayur Mudigonda, and Michael R DeWeese. Hamiltonian monte carlo without detailed balance. arXiv preprint 1rXiv:1409.570 5191, 2014.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. arXiv preprint arXiv: arXiv:1910.06222. 2019.
Jiaming Song, Shengjia Zhao, and Stefano Ermon. A-nice-me: Adversarial training for meme. In Advances in Neural Information Processing Systems, pp. 5140-5150, 2017.
Span Spanbauer, Cameron Freer, and Vikash Mansinghka. Deep involutive generative models for neural meme. arXiv preprint arXiv:2006. 15/67, 2020.
10

---

Under review as a conference paper at ICLR 2021
Achille Thin, Nikita Kotelevskii, Alain Durmus, Maxim Panov, and Eric Moulines. Metropolized flow: from invertible flow to meme. ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2020.
Tijmen Tieleman. Training restricted boltzmann machines using approximations the likelihood gradient. In Proceedings of the 25th international conference on Machine learning, pp. 1064- 1071, 2008.
Michalis Titsias and Petros Dellaportas. Gradient-based adaptive markov chain monte carlo. In Advances in Neural Information Processing Systems, pp. 15730-15739, 2019.
Jianwen Xie, Yang Lu, Song-Chun Long-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pp. 2635-2644 2016.
Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon. Training deep energy-based models with E-divergence minimization. In International Conference on Machine Learning, 2020.
Yichuan Zhang, José Miguel Hernández-Lobato, and Zoubin Ghahramani. Ergodic measure pre-
serving flows. arXiv preprint arXiv: 1805. /0377, 3(4), 2018


Reviews:
[{"review_id": "WIgnue3JOQb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe author proposes a novel MCMC sampler parametrized by the neural networks. In particular, the neural network is chosen to be a flow-based model that allows the exact evaluation of the proposal probability. \n\nThe update equations of MCMC mimic the dynamics of HMC by incorporating the gradient of the energy function into the flow. To ensure the invertibility, each update equation only depends on the other part of the variables. In addition, the author also designs another network $R$ to propose an evaluation location for the target gradient. This ensures the similarity between the proposed method and HMC. \n\nAs for the training objective, the author proposes to maximize the proposal entropy and acceptance rate, controlled by coefficient $\\beta$. Due to the tractability of the proposal density, this objective can be analytically computed. Stop gradient trick is used to stabilize the training and reduce the cost of back-propagation. \n\nEmpirically, the author evaluates the proposed sampler in some toy datasets, logistic regression, and deep energy-based models. The proposed sampler achieves higher ESS compared to baselines.\n\n---------\nReview:\nClarity: The paper is clearly written and easy to follow. The author also addresses the related work and mention the difference compared to previous baselines. \n\nTechnical soundness:\nI have a quick look at the details of the proposed method. It seems the derivation is correct. \n\nNovelty: \nAlthough the structure of the sampler is inspired by L2HMC, there are still some differences. L2HMC generalizes the HMC update equation by partitioning $\\pmb{x}$ into two parts, each part is parameterized by neural networks. The proposed sampler instead partitions the state $\\pmb{z}$ instead of parameter $\\pmb{x}$ with additional network $R$. The idea of using entropy as an objective to encourage exploration is not new.  However, the novelty lies in the usage of the flow model to allows tractability of proposal density. Overall, the proposed method is novel to some extent.  \n\nSignificance of the work:\nThe proposed method can be regarded as a variant of L2HMC with slightly different NN parameterization and training objectives. The advantage of the proposed method is the higher ESS compared to previous samplers. This may be helpful to some audiences.\n\nWeakness:\n1. Although the author demonstrates the better ESS can be obtained using the proposed method, I still prefer more analysis to understand the properties of this sampler. For example, I am curious to know how important the network $R$ is? If $R$ is removed, and the leapfrog integrator is used (i.e. update $z^{n'}$ followed by updating $x^{n}$ and finally $z^{n}$ to mimic HMC update rule). Then the gradient evaluation is also evaluated at different locations. What are the differences in terms of performances?\n2. The reason that the proposed method can have higher ESS is due to the maximization of the proposal entropy. Although the training objective has an acceptance rate term, I am curious to know: does the entropy term hurts the sample quality? For example, in logistic regression, the author only reports the ESS. What about the convergence speed of the sampler compared to others that use sample quality as the training objective? What about their performances in logistic regression?\n3. For training deep EBM, apart from ESS. I also want to know the convergence speed of the EBM training and the quality of the generated images compared to MALA. These are the standard evaluation metric for EBM.\n4. For EBM, why only compared to MALA. Any reasons why excludes other baselines?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IKM9Q_wQWKU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper argues that a better objective to train neural MCMC kernels is to maximize the proposal entropy (Titsias & Dellaportas, 2019) and demonstrate a method on doing so. The method shows improved sampling efficiency compared to previous method, especially one that optimize the alternative L2 expected jump. The novelty is not of the training objective but a neural instantiation with improved sampling efficiency.\n\n## Pros\n\n1. The method is well-motivated in Section 1 and clearly demonstrated by Figure 1.\n2. The neural instantiation consists a few clever tricks to make the network tractable and explore the space well.\n3. The method demonstrates improved sampling efficiency, measured by ESS.\n\n## Cons\n\n1. The paper may need more work on presentation.\n- Section 3 is hard to follow. It's very long and maybe adding some subsection would help.\n- I feel Section 4 should go before Section 3, given how it is currently presented.\n\n2. There is a false statements in the paper: \"one can easily make any transition probability satisfy it by including an additional Metropolis-Hastings accept-reject step (Hastings, 1970)\"\n- It's not true. One need to additionally ensure that the kernel is irreducible and aperiodic.\n- For the reason above, it's worth the mention that the proposed neural approach is irreducible and aperiodic, and give some justificaiton.\n\n3. No experiments for correctness check. I understand that showing better ESS is good, but even some biased sampler can lead to much improved ESS. I think there should be some results on comparing the proposed method against well-established HMC on a few Bayesian inference problem in terms of the posterior. Using some hypothesis testing methods to check if the posterior actually matches would give me more confidence that the method is sampling from the target; or at least, comparing a few moments.\n\n## Questions\n\n1. \"In (Titsias & Dellaportas, 2019), the authors avoid multiple back-propagation by stopping the derivative calculation at the density gradient term. In our experiment, we find it is necessary for good performance.\": does it mean you effectively using a biased gradient and it turns out to be better than an unbiased one? It looks quite weird to me as it means you are in fact optimizing some other objective which turns out to be better.\n\n2. I'm generally unsatisfied with the setup of Section 5.2.\n- \"which removes the need to choose the step size. The only tunable parameter is a target accept rate.\": I think vanilla HMC or MALA can do so by some online otpimization for step size as well (e.g. dual averaging as it's done in NUTS).\n- The unstable training of EBMs via contrastive divergence comes from the fact that the gradient estimated by a short-run MCMC is **unbiased**. The proposed method itself doesn't deal with the biasness directly, so the only plausible reason to explain why it improves the training is that the mixing is so good such that with the short chain, the bias of the gradient is so small. Is this what happening?\n- I wonder whether or not the simultaneous/interweaved training of samplers and EBMs would cause any issue. In particular, will it encourage that EBM only putting energy around data points and the sampler only effectively draw samples from some noisy distribution nearby? Are samples from EBMs too similar to data points.\n- For Figure 3.b, what's the convergence behaviour, i.e. do learnable sampler and MALA converge to a similar entropy or not?\n- For Figure 3.c, does long-run MALA produce sensible samples? If yes, it indicates that this way of training EBMs avoid some pathology which CD with short-run MCMC has; if no, then it may indicate that somehow the learnable sampler co-adapt with the EBM somehow to avoid the pathology. Is the latter we wanted?", "extra_scores": {}, "extra_text": {}}, {"review_id": "E1RuBmmtJHc", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors describe an approach for adaptive MCMC which uses a proposal distribution parameterized by a neural network and which optimizes the entropy of the resulting proposal. Overall I found it to be an interesting algorithm, with fairly good results as compared to alternative methods, however the description of the algorithm itself was somewhat confusing and/or convoluted.\n\nThe introduction (including section 2) was great, and provided a concise but very readable introduction to recent approaches for adaptive MCMC. The beginning of section 3 was also quite well presented. However, when the authors turn to the parts of their approach inspired by HMC it gets a bit murkier. Part of the difficulty here is that at this point (the first paragraph of p4) the authors are describing HMC rather than their method, which is not entirely clear. Perhaps this could have been simplified by moving the \"related work\" section earlier in the paper and allowing for the description of HMC before diving into their own approach.\n\nThe relation between HMC and their approach which makes use of intermediate steps x+R could also have been more thoroughly explained and given more intuition. The link is there, but it doesn't seem to be as close to the leapfrog step as x+R doesn't correspond to an intermediate step (which would be x+z^n). Not that I'm saying in any way that this makes the algorithm incorrect, just that the connection isn't quite as clear cut.\n\nSimilarly I would also have liked to see a more clear description of the Q, T, and S matrices.\n\nThroughout this work the authors also refer to \"during training\". I assume they perform their adaptation steps during the sample process as is the case of the Titsias and Dellaportas work, however this could use some clarification. Overall, I think that this and other confusions cited above could have been done away with by including a clearer outline/overview of the algorithm as a whole.\n\nFinally, overall the results seem to be quite a bit better than competing methods (although I'm not an expert in this area). However, although the authors hasted to add that they do not compare \"exact computation time or ESS/second\" it would have been nice to see a more thorough discussion of the relative computational complexity of the alternatives. While ESS/grad does in some sense get close to this, the computation necessary for networks (depending on their size) could play a factor here. Similarly, I would like to see more discussion with regards to the choice of architecture for these networks---ie were they a simple MLP? It's possible I missed this, but do not think it was discussed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1wc8pLPBa4z", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new MCMC transition kernel. This kernel is parameterized by neural networks and is optimized through an objective maximizing the proposal entropy. Specifically, the authors use a combination of a flow model and non-volume preserving flow in [Dinh et al., 2016] as the neural network parameterized kernel. Then they use the objective in [Titsias & Dellaportas, 2019] which maximizes the proposal entropy to optimize the kernel. The proposed method is tested on synthetic datasets, Bayesian logistic regression and a deep energy-based model.\n\nThe problem of improving the exploration efficiency of MCMC kernel is important. The proposed method is well-motivated. As far as I understand, the proposed method appears to be technically sound. \n\nHowever, I have the following concerns about the paper.\n\n-\tThe connection and the difference to previous work are not very clear. If I understand it correctly, the proposed method seems a combination of L2HMC and [Titsias & Dellaportas, 2019] with some slight modification (a flow model) since the naïve combination did not work well (as stated in Section 4). I think it would improve the clarity a lot if the authors explain more clearly how the proposed method differs from previous work. \n-\tSince the use of a flow model is the main difference compared to the naïve combination of two previous methods, the authors should explain more about this choice. Currently, it is not clear why this helps and there is no explanation on why the naïve combination fails. \n-\tThe proposed method seems to use more neural networks (e.g. an additional network R to handle gradient) than previous neural network MCMC. I wonder how the method performs if considering the cost. For example, the authors may instead show ESS per second on the experiments in Section 5.1. Though the authors mentioned the difficulty of computing ESS per second in the paper, I’m not entirely convinced. As the experiments in Section 5.1 are all very small-scale, is it really necessary to use GPUs? \n\n-\tThe baselines vary from experiments to experiments for no reason. For example, the authors compare their method to L2HMC on ill-conditioned Gaussian and strongly correlated Gaussian, to Neutra on Funnel distribution, and to MALA on EBM. I think this experiment design needs explanation. Also, there is no empirical comparison to [Titsias & Dellaportas, 2019] which is closely related to the proposed method.\n\nSome minor comments:\n\n-\tA.1 intends to show the benefit of using gradient information. But Variant 2 also uses gradient. What is the point of showing it? \n-\tIt is not clear to me how to interpret the empirical results in Section 5.2. For example, how does Figure 3 show that the proposed method needs less sampling steps? \n-\tThe color of points in figure 1 is hard to read.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed an MCMC sampler that combines HMC and neural network based proposal distribution. It is an improvement over L2HMC and [Titsias & Dellaportas, 2019], with the major innovation being that, the proposed normalizing flow-based proposal is engineered such that the density of the proposal $q(x'|x)$ is tractable. Experiments are conducted on synthetic distributions, Bayesian logistic regression and deep energy-based model training.

While reviewers are overall happy about the novelty of the approach, some clarity issues have been raised in some of the reviewers' initial reviews. Also concerns on the evaluation settings, including the missing evaluation metric such as ESS/second, are also raised by the reviewers. The revision addressed some of the clarity issues, but some experimental evaluation issues still exist (e.g. comparing with L2HMC in terms of ESS/second), and the replaced MALA baseline results make the improvement of the proposed approach less clear.

I personally find the proposed approach as a very interesting concept. However I also agree with the reviewers that more experimental studies need to be done in order to understand the real gain of the approach. 
Decision: Reject