Ergodic Inference: Accelerate Convergence by Optimisation
Authors: Yichuan Zhang
José Miguel Hernández-Lobato
Venue:  (2020)
Keywords: MCMC
variational inference
statistical inference

Abstract: Statistical inference methods are fundamentally important in machine learning. Most state-of-the-art inference algorithms are 
variants of Markov chain Monte Carlo (MCMC) or variational inference (VI). However, both methods struggle with limitations in practice: MCMC methods can be computationally demanding; VI methods may have large bias. 
In this work, we aim to improve upon MCMC and VI by a novel hybrid method based on the idea of reducing simulation bias of finite-length MCMC chains using gradient-based optimisation. The proposed method can generate low-biased samples by increasing the length of MCMC simulation and optimising the MCMC hyper-parameters, which offers attractive balance between approximation bias and computational efficiency. We show that our method produces promising results on popular benchmarks when compared to recent hybrid methods of MCMC and VI.

Under review as a conference paper at ICLR 2020
ERGODIC INFERENCE:
ACCELERATE CONVERGENCE BY OPTIMISATION
Anonymous authors Paper under double-blind review
ABSTRACT
Statistical inference methods are fundamentally important in machine learning. Most state-of-the-art inference algorithms are variants of Markov chain Monte Carlo (MCMC) or variational inference (VI). However, both methods struggle with limitations in practice: MCMC methods can be computationally demanding: VI methods may have large bias. In this work, we aim to improve upon MCMC and VI by a novel hybrid method based on the idea of reducing simulation bias of finite-length MCMC chains using gradient-based optimisation. The proposed method can generate low-biased samples by increasing the length of MCMC simulation and optimising the MCMC hyper- spper-parameters, which offers attractive balance between approximation bias and computational efficiency. We show that our method produces promising results on popular benchmarks when compared to recent hybrid methods of MCMC and VI.
INTRODUCTION
Statistical inference methods in machine learning are dominated by two approaches: simulation and optimisation. Markov chain Monte Carlo (MCMC) well-known simulation-based method, which promises asymptotically unbiased samples from arbitrary distributions at the cost of expensive Markov simulations. Variational inference (VI) well-known method using optimisation, which fits a parametric approximation to the target distribution. VI is biased but offers a computationally efficient generation of approximate samples.
There is a recent trend of hybrid methods of MCMC and VI to achieve better balance between computational efficiency and bias. Hybrid methods often use MCMC or VI as an algorithmic component of the other. In particular, Salimans et al.] (2015) proposed promising modified VI method that reduces approximation bias by using MCMC transition kernels. Another technique reduces the computational complexity of MCMC by initialising the Markov simulation from a pre- trained variational approximation (Hoffman) 2017 Han et al. (017). Levy et al. (2018) proposed to improve MCMC using flexible non- lineiincar transformations given by neural networks and gradient- based auto-tuning strategies.
In this work, propose novel hybrid method, called ergodic inference (El). EI improves flexible finite-step MCMC chain so that
both MCMC and VI by tuning the hyper- papee--rraseters its last state sampling distribution converges fast to target distribution. El optimises tractable objective function which only requires to evaluate the logarithm of the unnormalized target density. Furthermore, unlike traditional MCMC methods, the samples generated by El from the last state of the MCMC chain are independent and have no correlations, EI offers an appealing option to balance computational complexity vs. bias on popular benchmarks machine learning. Compared with previous hy brid methods, EI has following advantages:
El's hyperparameter tuning produces sampling distributions with lower approximation bias.
The bias is guarantced to decrease as the length of the MCMC chain increases.
By stopping gradient computations, El has less computational cost than related baselines.
We also state some disadvantages of our method:
The initial state distribution in EI's MCMC chain has to have higher entropy than the target.

---

Under review as a conference paper at ICLR 2020
The computational complexity per simulated sample of EI is in general higher than in VI.
2 BACKGROUND
2.1 MONTE CARLO STATISTICAL INFERENCE
Monte Carlo (MC) statistical inference approximates expectations under a given distribution using simulated samples. Given target distribution MC estimations of an expectation E,, (x)) are defined as empirical average of the evaluation of f on samples from æ. To generate samples from t, we assume that the unnormalized density function æ² (x(x) can be easily computed. In a Bayesian setting we typically work with x²" (xly) given by the product of the prior p(x) and the likelihood p(y x), where y denotes observed variables and x denotes the model parameters specifying p(y) x).
2.2 MARKOV CHAIN MONTE CARLO
Markov chain Monte Carlo (MCMC) casts inference as simulation of ergodic Markov chains that converge to the target æ. The MCMC kernel M(x (x) is characterised by the detailed balance (DB)
property: πt (x)) (x' x) TT (T(x) ) (x|x'). Given unnormalised target density n², MCMC kernel can be constructed in three steps: first, sample an auxiliary random variable r from an auxiliary distribution 9á, with parameters $1 second, create new candidate sample as (x', .r) foz (x,-1, where foz is a deterministic function with parameters Φ2; finally, accept the proposal as x, = x' with probability рMH min (0, x= (x') ]oo, (r')/(T" [xx 1)46. (r)]), otherwise duplicate the previous sample as X, X,-1- The last step is well known in the literature as the Mersooolis-Hasinngs (M-H) correction step (Robert & Casella 2005) and it results in MCMC kemnlls that satisfy the DB condition.
In the following, we denote the joint MCMC parameters (ø1, Φ2) by Φ. If før does not preserve volume, then it requires Jacobian correction factor in the ratio in HM
Hamiltonian Monte Carlo (HMC) is a successful MCMC method which has drawn great attention. A few recent works based on this method are Salimans et al. (215)) Hoffmann| (017)) Levy et al. (2018).
In HMC, the auxiliary distribution 9d, is often chosen to be Gaussian with zero mean and a constant diagonal covariance matrix specified by d1- The most common føz in HMC a numeric integrator called the leapfrog algorichm, which simulates Hamiltonian dynamics defined by log n (Neal) 2010). The leapfrog integrator requires the gradient of log π and a step size parameter given by Φ2.
Given any initial state Xo. MCMC can generate asymptotically unbiased samples Ximn- For this, MCMC iteratively simulates the next sample x, through the application of the MCMC transition kemnel to the previous sample X,-1- It is well known in the literature that MCMC is computationally demanding (MacKay) 2002, Bishop 2006). In particular, is often necessary to run sufficiently long burn-in MCMC simulations to reduce simulation bias. Another drawback of MCMC is sample correlation, which increases the variance of the MC estimator (Neal 2010). To avoid strong sample correlation, the common practice in MCMC tune typee--psameeters manually using sample quality metrics like effective sample size, (Hoffmaan) [0177; Robert & Casella 2005). which has been developed into automated gradient-based tuning strategies in recent work (Levy et al. 2018).
2.3 MC ESTIMATION USING VARIATIONAL INFERENCE
Variational inference (VT) is popular alternative to MCMC for generating approximate samples from т. Unlike MCMC reducing sample bias by long burn-in simulation, VI casts the sample bias reduction an optimisation problem, where parametric approximate sampling distribution P is fit to the target म. In particular, VI optimises the evidence lower bound (ELBO) given by
(I) where H (P) = EE, log p(x)] also known as the entropy, must be tractable to compute. (EL T²) is lower bound on the log normalising constant log Z log ʃ H²" (x) dx. This bound is tight when P = π. therefore, the approximation bias in VI can be defined as the gap between LP x²") and log Z, that is,
(P (P || x)) loo "T" (x)) + H(P),
Anins Arins()) (P)
log Z - LEIBO T) DEr. [1]
(2)
where DxI(P (P] π) denotes the Kullback-Leible (KI.) divergence.
2

---

Under review as a conference paper at ICLR 2020
Variational approximations often belong to simple parametric families like the multivariate Gaussian distribution with diagonal covariance matrix. This results in computationally efficient algorithms for bias reduction and sample generation, but may also produce highly biased samples in cases of over-simplified approximation that ignores correlation. Designing variational approximation to achieve low bias under the constraint of tractable entropy and efficient sampling procedures is possible using flexible distributions parameterised by neural networks (NNs) (Rezende & Mohamed] 2015 Kingma et al. 2016). However, how design such NNs for is still a research challenge.
2.4 HYBRID METHODS AND VARIANTS OF MCMC AND VI
The balance between computational efficiency and bias is a challenge at the heart of all inference methods. MCMC represents family of simulation-based methods that guarantee low-bias samples at cost of expensive simulations; VI represents a family of optimisation-bas high-bias samples
methods that generate
low computational cost.
Many recent works seek a better balance between efficiency and bias by combining MCMC and VI. Salimans et al. (2015) proposed to reduce variational bias by optimising ELBO specified in terms of the tractable joint density of short MCMC chains. The idea seems initially promising, but the proposed BLBO becomes looser and looser the chain grows longer. Caterini et al. (2018) construct an alternative BLBO for HMC that still has problems since the auxiliary momentum variables are sampled only once the beginning of the chain, which reduces the empirical performance of HMC. Inspired by contrastive divergence. Ruiz & Titsias (2019) proposed novel variational objective function to optimise variational parameters by adding additional term that minimise the KL between a MCMC distribution and variational approximation to reduce variational bias.
Hoffmman 2017) and Han et al. (2017) proposed to replace expensive burn-in simulations in MCMC with samples from pre-trained variational approximations, This approach is effective at finding good initial proposal distributions. However, it does not offer solution for tuning HMC parameters (Hoffman 2017), which are critical for good empirical performance.
Another line of research has focused on improving inference using flexible distributions, which are transformed from simple parametric distributions by non-linear non-volume preserving (NVP) functions. [evy et al. (2018) proposed to tune NVP parameterised MCMC w.r.t. a variant of the expected squared jumped distance (ESJD) loss proposed by Pasarica & Gelman (2010). [Song et al.) (2017) proposed a similar auto-tuning for NVP parameterised MCMC using an adversarial loss.
3 ERGODIC INFERENCE
Ergodic inference (EI) is motivated by the well-known convergence of MCMC chains (Robert & Casella 2005): MCMC chains converge in terms of the total variation (TV) distance between the marginal distribution of the MCMC chain and the target π. Inspired by the convergence property of MCMC chains, we define an ergodic approximation Pr to æ with I' MCMC steps as following. Given a parametric distribution Po with tractable density Pu Φo) parameterlized by Φu and an MCMC kernel (x'' xx; Φ) constructed using the unnormalised target density л and with MCMC hyperparameter db an ergodic approximation of is the marginal distribution of the final state of an T-step MCMC chain initialized from Po:
/ II ())x Po) do)
T
(x Pa. Φ)=
We call d and & the ergodic parameters of Pr. Well known in MCMC literature like (Robert & Casella 2005; Murray & &alakhutdinov 2008), the ergodic approximation pr converges to after every MCMC transition and with sufficiently long chain pr is guarantced to be arbitrarily close to m with arbitrary & and Φo-
is important to clarify that ergodic approximation is different from the modified variational methods like (Ruiz& & Titsias 2019) which only optimise the variational parameters 6o. but the optimisation objective functions involve MCMC similation. In the following section, we show how EI can tune the ergodic parameters to minimise the bias of Pr as approximation to the target with finite T.
3

---

Under review as a conference paper at ICLR 2020
3.1 ERGODIC APPROXIMATION OBJECTIVE
To reduce the KL divergence DRI.(PH) 7). one could tune the burn-in parameter d and the MCMC
parameter by minimizing equation 2 However, this is infeasible because we cannot analytically evaluate pr in equation 3 Instead, we exploit the convergence of ergodic Markov chains and propose to optimise an alternative objective as the following constrained optimisation problem:
max Epr. flog xt (x)] + (PL || x²) = L(Φo. Φ T²"
(4)
db
subject (P)) > h,
(5)
where h is a hyperparameter that should be close to the entropy of the target, that is, / R (T). We call the objective in equation 4 the ergodic modified lower bound (EMLBO), denoted by L(do, Φ T²)) Note that the EMLBO is similar to LELBO(P), (Pr || x²) with the intractable entropy H (Pr) replaced by the tractable LELBO (Po || x²")
We now give some motivation for this constrained objective.
First, we explain the inclusion of the term (PL || π²) in equation[4anne and its connection to H (.r). If we maximised only the first term result would be a point probability mass at the mode of the target π. This degenerate solution is avoided in VI by optimising the sum of Ep flog ग (x)] and the entropy term H HP), which enforces P to move away from a point probability mass. However, H (Pr) is intractable in ergodic approximation. Fotunately, we notice that maximising the term LELBO (Po T*) = Epo llg π' (x)] + H1 (Pb) has similar effect of maximising (Pr) for preventing Po from collapsing to the mode of т. It is easy to show that Pr cannot be a delta unless (Pr) -oo, which also implies LELBO (rr || HT) does
with respect to a fully flexible distribution P, the
not exist. Since the KL divergence DRI.(Pt (P: I) never increases after each MCMC transition step
(Murray Salakhutdinoy 2008), LEI.BO Pt || m²) 2 LALBO || T²) 2 LELRO(PO Po || x by maximising (PL TT) T"), also maximise (EL T*). which implies LELBO (rr x*) must exist.
The constraint in equation 5 is necessary to eliminate the following pathology. If Po does not satisfy (Po) > (TT,, then Epg log π (x)] could be higher than E,, [log π (x)). When this happens, if we maximise Ep,, [log x "xx), we will favor Py to stay close to Po instead of making it converge to faster. This is illustrated by the plot in the right part of Figure 2 To avoid this pathological case, note Epp [lg ग (x)) > H(P) ((T). Therefore, when (Po) > H(T) is satisfied, we have that pe llog म (x)] = E,, [log x (x)] >
that ( || x²) L (Po x²) leads to E,, [lo n" (x)
Ep, llog π (x)) and maximising Epy [lg ' (x)) in equation 4 is expected accelerate convergence.
It is interesting to compare the EMLBO with the objective function optimised by Salimans et al. (015)), that is, the ELBO given by
LE (Pr || x²) DKI. (1(xx.-11xxx)) |(x))--(xx))
(6)
where p(Xo.T-1 -1(T)) denotes the conditional density of the first 7 states of the MCMC chain given the last one XT and r(Xo:T- |xx) ) is an auxiliary variational distribution that approximates p(Xo:T-1 xT). Note that the negative KL term in equation 6 will increase as T increases. This makes the ELBO in equation 6 become looser and looser the chain length increases. In this case, the optimisation of equation results in an MCMC sampler that fits well the biased inverse model r(xo:r-1|x/ but whose marginal distribution for XT does not approximate x well. This limits the effectiveness of this method in chains with multiple MCMC transitions. By contrast, the EMLBO does not have this problem and its optimisation will produce a more and more accurate PT T increases.
EI combines the benefits of MCMC and VI and avoids their drawbacks, as shown in Table [ In particular, the bias in El is reduced by using longer chains, as in MCMC, and El generates independent samples, as in VI. Futhermore, EI optimises an objective that directly quantifies the bias of the generated samples, as in VI. Methods for tuning MCMC do not satisfy the latter and optimise instead indirect proxies for mixing speed, e.g. expected squared jumped distance (Levy et al. 2018). Importantly, EI can use gradients to tune different MCMC parameters at each step of the chain, as suggested by (Salimans et al. (2015). This gives El an extra flexibility which existing MCMC methods do not have. Finally, El is different from parallel-chain MCMC: while El generates independent samples, parallel-chain MCMC draws correlated samples from several chains running in parallel.
3.2 STOCHASTIC GRADIENT APTIMISATION FOR THE ERGODIC OBJECTIVE
We now show how to maximise the ergodic objective using gradient- based optimisation. The gradient
Deo. db L(vo &, x) is equal to the sum of two gradient terms. The first one 0g. (Po || (²) is
4

---

Under review as a conference paper at ICLR 2020
Optimises objective
Asymptotically unbiased
Samples
Method
quantifying bias of samples
generated
MCMC
No
Yes
Correlated
VI
Yes
No
Independent
EI
Yes
Yes
Independent
Table 1: Ergodic inference combines with pros of MCMC and and avoids their cons.
affected by the constraint H(P)) > h, while the second term 2g [log "" (xr) is not. If we ignore the constraint, the first gradient term can be estimated by Monte Carlo using the reparameterization trick proposed in (D.P. Kingma 2014, Rezende & Mohamed 2015):
O00 (Po )
Se log (Foo (E:)) + SΦ (Po),,
where f (.) is a deterministic function that maps the random variable 6, sampled from a simple distribution, e.g. a factorized standard Gaussian, into the random variable xi sampled from Po(:: Φo). To guarantee that initialize Φo so that H (Po) > h and, afterwards, we force the gradient descent optimiser to leave Do unchanged if H (P6) is to get lower than h during the optimisation process.
gradient-based optimiser yields solution satisfying the constraint, first
The Monte Carlo estimation of 2 Epr [log π: (x)] can also be computed using the reparameterization trick. For this, the Metropolis- Hesttpolis-Hastinngs (M-H) correction step the MCMC transitions, as described in Section 2.2 can be reformulated as applying the following transformation to X;-1:
X Iel Xx Fri u.) = x (рMи: uz) + X [1 (PMH); uz)]
(
where (x', = (x as described Section 2.2 Fr ~ q(r). u, ~ Unif(O, 1). PMH min {1, n² (x')q(r")) (xx (x,-1)q(r,))
and (pMi; u) indicator function that takes value if РMн > u and zero otherwise. In Hamiltonian Monte Carlo (HMC), fa is the leapfrog integrator of Hamiltonian dynamics with the leapfrog step size &. We define the T-times composition of Gø. given in equation 8 as the transformation xr 56 (Xo, F1:7; (Ir)). Then, the second gradient term be estimated by Monte Carlo as follows:
Bg llog x" (XT))
Σ
Os log xr" 10 (x): (errulr)))
N
where xf, rí.r and wi.r are sampled independently from po Xo) II q(r,) ((ir:))ni((u;;:: 0, 1). Note that the gradient term equation 9 is correct under the assumption Sa is volume-preserving in the joint space of (x;-1, otherwise additional gradient term of the Jacobian of fo w.r.t. & is required. However, it is not a concern for many popular MCMC kernels. For example, the leapfrog integrator in HMC fo guarantees the preservation of volume as shown (Neal 2010). is worth mention that the indicator function in equation iss not continuous but differentiable almost every everywhere. Therefore, the gradient in equation 9 can be computed conveniently using standard autodifferentiation tools.
The gradient in equation requires computing Opgz (Xo, F1:7; uI:7), which can be done easily by using auto diffrretifiirnnniaiiion and gradient backpropagation through the transfromations (-, rri uz) with t = T...., However, backpropagation deep compositions can be computationally demanding. We discovered trick to accelerate the gradient computation by stopping the backpropagation of the gradient at the input x;-1 of gø (x4-11 rı; ), for t = 1,... T. Empirically this trick has almost no impact the convergence speed of the ergodic approximation, as shown in Figure]
3.3 THE ENTROPY CONSTRAINT AND HYPERPARAMETER TUNING
As mentioned previously, ignoring the constraint H(P)) > H may lead to pathological results when optimising the ergodic objective. To illustrate this, we consider fitting an ergodic approximation given by Hamilton Monte Carlo (HMC) transition kernel with T 9. Py denotes the initial ergodic approximation before traing and Pj denotes the same approximation after training. The target distribution correlated bivariate Gaussian given by (o, (2.0, 1.5; 1.5; 1.6)). Samples
5

---

Under review as a conference paper at ICLR 2020
from this distribution shown in plot (a) in Figure We optimise different separate HMC parameter Φ1, as described Section 2.2 for each HMC step t. We consider two initial distributions. The first one is Po (O, 31) which Satisfies the assumption (Pa) > (T). The second one is Po ( I) with the entropy (P)) < (T)), which violates the assumption. In this latter case, (c) in Figure[]shov samples from Pg and P3 for the valid Po. In this first example, maximising the ergodic objective under equation Significantly accelerates the chain convergence
perform the unconstrained optimisation of equation 4 Plots (b) and
further shown by the left plot in Figure 2 Plots (d) and (e) in Figure show samples from Ps and P for the invalid initial distribution Pg In the second example. Ep [log (x) is higher than E, [log x² (x) and, consequently, maximising the unconstrained ergodic objective actually deteriorates the quality of the resulting approximation. This further illustrated by the right plot in Figure 2 which shows how the convergence of PE (log π (x)]
to E,, [log x= (x) is significantly slowed down by the optimisation under the invalid Pó-
Fortunately, it is straightforward to prevent this type of failure cases by appropriately tuning the scalar hyperparameter h in equation 5 A value of h that is too low may result in higher bias of PT after optimisation as illustrated by the convergence of Ep, log π (x)] in the blue and orange curves in Plot (b) in Figure Furthermore, in many cases, estimating example, Bayesian inference, the entropy of the prior distribution p(x) is often higher than the entropy of the posterior p(xly). Therefore, the prior entropy can be used reference for tuning h.
upper bound (TT) is feasible. For
(a) π
(b) Po
(c) P7 with Po
(d) Pg with P
(c) Pg with P/
Figure 1: Histograms of samples from ergodic inference using HMC transition kernlss. Pg denotes the ergodic approximation before traing; Pÿ denotes the ergodic approximation after training.
Expectation of log (())
Expectation of log mf
-1.8
-4
-6
-2.0
training
Fl after training, stopping gradients
-2.2
-10
efter training, na stopping gradients
sampling from
-244
maxllog (x:)
-14
-2.6
-16
7
8
Chain length T
Chain length T
Figure 2: The plot of Epr [og π (x)] as a function of the length of the chain using 10000 samples: Left: with the valid Po H (Pp) (m)); Right: with invalid P H(P)) H(x). SG training
means the stop gradient is applied to the x from previous HMC step in equation 9
4 EXPERIMENTS
We first describe the general configuration of the ergodic inference method used in our experiments. Our ergodic approximation constructed using HMC, one of the most successful MCMC methods in machine learning literature. We use T HMC transitions, each one involving 5 steps of the vanilla leapfrog integrator which was implemented following Neal (2010). The leapfrog pseudocode can be found in the appendix. In each HMC transition, the auxiliary variables are sampled from zero-mean Gaussian distribution with diagonal covariance matrix. We tune the following HMC parameters: the variance of the auxiliary variables and the leapfrog step size, mentioned in Section 2.2 We use and optimise a different value the HMC parameters for each the T HMC transitions considered. We call our ergodic inference method Hamiltonian ergodic inference (HEI). The burn-in model Po is

---

Under review as a conference paper at ICLR 2020
factorized Gaussian. The initial entropy of Po is chosen to be the same the entropy of the prior.
The stocastic optimisation algorithm is Adam (Kingma & Ba 2013) with TensorFlow implemtation
Abadi et al. (2015) and the optimiser hyperparameter setting [ 0.9, $2 0.999, e = 100-). ))
The initial HMC leapfrog step sizes are sampled uniformly between 0.01 and 0.025. Additional experiment Bayesian neural networks is included in Appendix 6.3
4.1 DEMONSTRATIONS
We first compare Hamiltonian ergodic inference (HEI) with previous related methods on 6 synthetic bivariate benchmark distributions. Histograms of ground truth samples from each target distribution using rejection sampling are shown in Figure 3 The baselines considered include: 1) Hamiltonian
variational inference (HVI) (Salimans et al. 2015); ) 2) generalized Hamiltonian Monte Carlo (GHMC) using an NVP parameterized HMC kernel and gradient-based auto-tuning of MCMC parameters w.r.t. sample correlation loss (Levy et al. 2018); ; 3) Hamiltonian annealed importance sampling (HAIS) "Sohl-Dickstein & Culpepper| 2012). It is worth to mention that we do not consider other hybrid inference methods like (Ruiz& & Titsias 2019 Hoffman (017) in experiment, because these methods only combines MCMC simulation with VI but not optimise the parameters of MCMC kernel using the gradient- based approach like EI.
HVI is the most similar method to HEI among all three baselines, because both HEI and HVI methods generate samples from the last state of MCMC chains and use gradient-based MCMC hyperparameter tuning to reduce bias. For a fair comparison between HVI and HEI, we consider the HMC chains with exactly the same setting in both methods: the initial state follows standard Gaussian distribution and the length of HMC chain is T = 10. The key difference between HVI and HEI is the hyperparameter tuning objective, as mentioned in Section 3.1 We trained HVI for 1000 iterations and verified the ELBO converges to a (local) minimum (plots of the training ELBO values are included in Appendix
6.2 We trained HEI for 50 iterations. Following the setting of HAIS by |Wu al. (2017). we used
1.000 intermediate distributions with 5 leapfrog steps per HMC transition and manually tuned the
was run using 100 parallel chains with 5 leapfrog steps per GHMC transition, 100 burn-in steps and 1000 auto-tuned training iterations
HMC parameters to have acceptance rate around 70%. GHMC _
Levy et al. 2018). The verification of the convergence of Ep, (log T (x)) to Eg [log ' (x)] for HEI shown plot (a) of Figure 5
We generate 100,000 samples with each method and evaluate sample quality using two metrics: 1)
the histogram of simulated samples for visual inspection; 2) the MC estimation of [lo x" (x)]
Effective sample size (ESS) is a popular sample correlation based evaluation metric in recent MCMC
literature (Levy al. 2018). However, we do not consider ESS in this experiment, because GHMC is the only method among all methods generating correlated samples. Therefore, the ESS of GHMC guaranteed to be lower than HVI and HEI. To generate ground truth samples from benchmark distributions, we use The resulting sample histograms of the ground truth using rejection sampling are shown in figures 3 and considered approx sampling methods are shown in Table 2 shows the resulting estimates of - (log x (x) together with the wall-clock simulation time for generating 100,000 samples. The left part of Table 3 shows the training time of the MCMC parameter optimisation for all methods except HAIS, which does not support gradient-based HMC hyperparameter tuning. HEI is faster than HVI and GHMC. Note, however, that the acceleration of HEI over HSVI is due to the stopping gradient trick described in Section 3.2 The histograms and the estimates of - E, [log T (x)) generated by HEI are consistent with the results of the more expensive unbiased samplers GHMC and HAIS, which are close the ground truth. By contrast, HVI exhibits a clear bias in all benchmarks. Regarding the sampling time, HVI and HEI simulate HMC chains with the same length and, consequently, perform similarly in this case while sample simulation from HAIS and GHMC is much more expensive.
'The code used was obtained from http://gghhha...om//naacceeee///
7

---

Under review as a conference paper at ICLR 2020
(a)
(b)
(c)
(d)
(e)
(f)
Figure 3: Histograms of samples generated by rejection sampling on each benchmark problem.
HVI
HEI
HAIS
GHMC
HVI
HEI
HAIS
GHMC
Figure 4: Histograms of 100,000 samples generated by each method after parameter optimisation.
Training hours
Training Epochs
Test
Encoders
log(x)
ESS
Conv VA Salimans el al. [2015
88.20
HVI(7=1, 16LF, n,, =800, ConvNet encoder) Salimans et al. 2015
-81.94
300, ConvNet encoder) Conv VAE(n,,=500) (Baseline)
HVAE(T==1 1. 20LF.
Caterm et al. 2018
-84.78
6.00
3000
-83.57
50
HVI(T=1, 16LF, =800 ConvNet encoder)
6.00
360
-83.68
48
HVAE(T=1, 16LF, =500 ConvNet encoder)
6.00
360
-84,22
HE1 51.F, n0,5000,, no neural net encoder)
1.65
-83.17
48
HEI(T=30, 5LF ny 5500 no neural net encoder) HEI(T=30, 5LF, m5=500, no neural net encoder) HEI(T=30, 5LF, n,,500, no neural net encoder) HEI(T=15, 5LF, m,,5000, no neural net encoder)
3.00
100
-82.76
46
6.00
200
-82.65
45
12.00
400
-81.43
38
8.00
540
-83.30
48
Table Comparisons in terms of compuational efficiency and log-likelihood in the training of deep generative models on the MNIST dataset. We implemented the deconvolutional decoder network in Salimans al. (2015) to test HVI. In Salimans al. (2015), the test likelihood estimated using importence-weightee samples from the encoder network. In our experiment, we use Hamiltonian annealled importance sampling and report the effective sample size (ESS).
4.2 TRAINING DEEP GENERATIVE MODELS
We now evaluate HEI in the task of training deep generative models. MNIST is standard benchmark problem in this case with 60,000 grey level 28 x 28 images of handwritten digits. For fair comparison with previous works, use the 10,000 prebinarised MNIST test images") used by Burda et al. (2015). The architecture of the generative model considered follows the deconvolutional network from Salimans al. (215)). In particular, the unnormalised target po (x, y) consists of 32 dimensional latent variables x with Gaussian prior p(x) = (0, I) and a deconvolutional network ( x) from top to bottom including a single fully-connected layer with 500 RELU hidden units, then three deconvolutional layers with x 5 filters, (16, 32, 32) feature maps, RELU activations and logistic output layer. We consider baseline given by a standard VAE with factorised Gaussian approximate
Phaps//ggthuh..om/yyuraaias
8

---

Under review as a conference paper at ICLR 2020
- F.g [gg]log (x)] estimate/ time
c
HVI Salimaas et al. 2015
4.94/0.20
1.22/0.41
5.22/0.92
1.70/0.25
1.37/0.87
1.64/0.54
HEI
3.49/0.25
0.79/0.44
4.78/1.58
0.99/00
0.59/0.58
0.57/0..5
GHMC Neal (001
3.43/35.7
0.80/64,28
4.74/41.23
1.03/48.54
0.63/85.30
0.59/81.0
HAIS Levy el al. (2018
3.38/16.62
0.78/26.94
4.70/ 125.32
1.00/22.61
0.60/33.07
/3.9/33.69
Ground Truth
3.31/-
0.76/-
4.66/-
0.98/-
0.58/-
0.49/-
Table 2: Estimation of -E, log n (x)] and the sampling time on CPU: Each score (a/b) above refers to: a) flog n- (x)] estimated by 100k samples; b) time in seconds to generate 100,000 samples.
Training Time on Synthetic Problems
Training Time on DGM (sec epoch)
Method
sec / 100 iters
Method
T=15
T=30
HVT (Salimans et al.
2015
2.367
HEI
257.8
551.2
HEI (stop gradient)
1.620
HEI (stop gradient)
56.3
114.0
GHMC (Neal 2001
7.100
Table Left. The training time of MCMC parameter optimisation seconds for 100 iterations for all candidate methods to produce the results in Figure 4 The training time of HEI is lower than HVI because of the stop gradient trick mentioned in Section 3.2 We do not report the training time for HAIS, because HAIS requires manual tuning of MCMC hyperparameters which is not directly comparable to the gradient- based autotuning used by the other methods. Right. The training time in seconds per epoch for the experiments with deep generative models (DGM).
posterior generated by an encoder network g(xly) which mirrors the architecture of the decoder (Salimans et al.] 2015).
The code for HVI Salimans et al. (2015) their convolutional VAE and were able to reproduce the marginal likelihood reported by Salimans al. (2015). as shown in Table 4 This verifies that our implementation of the generation network is correct. We implemented HVI in (Salimans et al., 2015) using an auxiliary reverse model in the ELBO parameterized by a single hidden layer network with 640 hidden units and RELU activations. We also implemented the Hamiltonian variational encoder (HVAH) method (Caterini et al. 2018), which similar to HVI but without the reverse model. Unlike the original HVAE, our implementation does not use tempering but still produces results similar to those from Caterini et al. (2018).
publicly available. Nevertheless, reimplemented
For the HEI encoder, we use = 30 HMC steps, each with 5 leapfrog steps. The initial approximation Po is kept fixed to be the prior p(x). We optimise the decoder and the HEI encoder jointly using Adam. Table shoows the marginal test log- likelihoodd for HEI and the other methods, as estimated with 1,000 HAIS samples (Sohi-Dickstein & Culpepper 2012). Following [Li al. (2017), also include the effective sample size (ESS) of HAIS samples for the purpose of verifying the reliability of the reported test iogeliosisoo.. Overall, HEI outperforms HVI, HVAE and the standard VAE in test log-likelihood when the training time of all methods is fixed to be 6 hours. HEI still produces significant gains when the training time is extended 12 hours and, with only 1.6 hours of training, HEI can already outperform the convolutional VAE of Salimans et al. (2013) with 6 hours of training.
To verify the convergence of HEI, we show in plot (b) of Figure estimates of Epy [log x** (x)] E, [log n² (x)) for T' 1,.... 10 on five randomly chosen test images, where the ground truth E, [og n" [x) estimated by HAIS, after HMC hyper-paramett tuning in HEI (blue) and without hyper-parameter tuning in HEI (green), i.e. just using the initial hyper-parametee values. Plot (c) in
Figure 5 shows similar results, but using the maximum mean discrepancy (MMD) score (Gretton al.] 2012) to quantify the similarity of samples from pr samples from æ, where the latter ground truth samples are generated by HAIS. These plots suggests that shortening the HEI chain to T = 10 HMC steps will have negligible effect on final simulation accuracy. Finally, the right part of Table B shows the training time of HEI with and without the stopping gradient trick. These resuls show that the former method is up 5 times faster.

---

org/ Software available from Under review as a conference paper at ICLR 2020
(b)
600
Refore traiting Aftor training
Bcfore traniing
E, Bollogaa Eyg [loga
500
50
After trainin
400
MMD Py
300
20
100
15 Chain length T
Chain length? 7
Chain length T
Figure 5: The verification of the convergence of Epr [log x (x)] to Ex [log ' (xx)]: a: the targets are 2D benchmarks with the ground truth of E, [log π [x)]; b: the target π is the VAE posterior p(xly) each curve represents one random chosen test MNIST image y with the ground truth of E, [log π (x)] estimated by HATS using 100 samples; MMD score between HEI samples and HAIS samples.
5 SUMMARY
We have proposed Ergodic Inference (EI), a novel hybrid inference method that bridges MCMC and VI. El a) reduces the approximation bias by increasing the number of MCMC steps, b) generates independent samples and c) tunes MCMC hyperparameters by optimising an objective function that directly quantifies the bias of the resulting samples. The effectiveness of El was verified synthetic examples and on popular benchmarks for deep generative models. We have shown that we can generate samples much closer gold standard sampling method than similar hybrid inference methods and at a low computational cost. However, one disadvantage of EI is that it requires the entropy the first MCMC step to be larger than the entropy of the target distribution.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals. Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems. 2015. URL http: tensorflow.org.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov, Importance weighted autoencoders. arXiv
preprint arXiv:1509.00:: 00519, 2015.
Anthony L Caterini, Arnad Doucet, and Dino Sejdinovic. Hamiltonian variational auto-encoder. In Advances in Neural Information Processing Systems, pp. 8167-8177, 2018
M. Welling D.P. Kingma. Auto-encoding variational bayes. In The International Conference on Learning Representations (ICLR), 2014.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13:723-773, March 2012. ISSN 1532-4435.
Tian Han, Yang Lu, Song-Chun Zhu, and Ying Nian Wu. Alternating back-propag for generator network. In AAAI, volume 3, pp. 13, 2017.
Matthew D. Hoffman. Learning deep latent Gaussian models with Markov chain Monte Carlo. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1510-1519, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
10

---

Under review as a conference paper at ICLR 2020
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Nya Sutskever, and Max Welling. Improved Variational Inference with Inverse Autoregressive Flow. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and Garnett (eds.). Advances Neural Information Processing Systems 29, pp. 4743 4751. Curran Associates, Inc., 2016.
Daniel Levy, Matt D. Hoffmann, and Jascha Sohl-Dickstein. Generalizing Hamiltonian Monte Carlo with neural networks. In International Conference on Learning Representations, 2018.
Yingzhen Li, Richard E. Turner, and Qiang Liu. Approximate inference with amortised memc. arXiv preprint arXiv: orXiv:1702.08343, 2017.
David J. C. MacKay. Information Theory, Inference & Learning Algorithms. Cambridge University Press, New York, NY, USA, 2002. ISBN 0521642981.
lain Murray and Ruslan Salakhutdinov. Notes the KL-divergence between a Markov chain and its equilibrium distribution. preprint, 2008.
Radford M Neal. Annealed importance sampling. Statistics and computing, 11(2):125-139, 2001.
Radford M. Neal. MCMC using Hamiltonian dynamics. 2010.
Cristian Pasarica and Andrew Gelman. Adaptively scaling the Metropolis algorithm using expected squared jumped distance. Statistica Sinica, pp. 343-364, 2010.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning Volume 37, ICML'15, pp. 1530-1538. JML...rg. 2015.
Christian P. Robert and George Casella. Monte Carlo Statistical Methods (Springer Texts in Statistics). Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2005. ISBN 0387212396.
Francisco JR Ruiz and Michalis K Titsias. A contrastive divergence for combining variational inference and meme. arXiv preprint arXiv:1905.044 2019.
Tim Salimans, Diederik Kingma, and Max Welling. Markov chain Monte Carlo and variational inference: Bridging the gap. In International Conference on Machine Learning, pp. 1226-1226, 2015.
Jascha Sohl-Dickstein and Benjamin J Culpepper. Hamiltonian annealed importance sampling for partition function estimation. arXiv preprint 1205.v /925, 2012.
Jaaming Song, Shengjia Zhao, and Stefano Ermon. A-NICE-MC: Adversarial training for MCMC. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5140-5150. Curran Associates, Inc., 2017.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian Optimization with Robust Bayesian Neural Networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4132-4142. Curran Associates, Inc., 2016.
Yuhuai Wu, Yuri Burda, Ruslan Salakhutdinov, and Grosse Roger. On the Quantitative Analysis of Deep Belief Networksnalysis of Decoder-based Generative Models. In /CLR. 2017.
11

---

Under review as a conference paper at ICLR 2020
6 APPENDIX
6.1 LEAPFROG ALGORITHM
Here is the code for the vanilla leapfrog algorithm we used in HVI HEI and HAIS.
Algorithm 1: Leapfrog
Input: x: state, r: momenta, Φ1 r variance, Φ2 step size, m: number of steps
Result: x': new state, r': new momentum
X ==x; x:
T= r;
for t f- 1 to do
F = F - -0.5020&U(x):
X x 02/011;;
F = F OU
end
x' =x
y'= r;
return x' and r';
6.2 RESULTS OF HVI ON SYNTHETIC BENCHMARKS
The plots in Figure 6 show training loss (negative ELBO) of HVI and the training expected log likeli- hood Epr: [lgg nr (x) with T = 10 HMC steps with Adam with hyperparameter setting described in Section 4 It is clear that HVI is well trained but the approximation is biased, because Epy [log TT (x)] does not converge to the true loss (the red line on the right plots). In comparison, in Figure 6(Left) in our paper, Epy [log n= (x)] of HEI converges to the ground true by optimising our ergodic loss.
6.3 BAYESIAN INFERENCE WITH NEURAL NETWORKS
In this additional experiment we approximate the posterior distribution of Bayesian neural networks with standard Gaussian priors. We consider four UCI datasets and compare HEI with the stochastic gradient Hamilton Monte Carlo (SGHMC) method from SSpringenbrg et al.) (2016). The networks used in this experiment have 50 hidden layers and I real valued output unit, as stated in [Springenberg eet al. (2016). The HEI chain contains 50 HMC transformation with Leapfrog steps each. The initial proposal distribution Po a factorised Gaussian distribution with mean values obtained by running standard mean-field VI using Adam for 200 iterations. We do not use in Po the variance values returned by VI because these are unlikely to result in higher entropy than the exact posterior since VI tends to understimate uncertainty. Instead, we choose the marginal variances to be n -0.5 where n is the number of inputs to the neural network layer for the weight. To reduce computational cost, we use in this case stochastic gradients in the leapfrog integrator. For this, split the training data into 19 mini- batches and only use one random sampled mini- nini-batch for computing the gradient in each leapfrog iteration. We train HEI for 10 epochs and the stationary distribution is chosen as approximate posterior on a random sampled mini-batch. The resulting test log- ligllikess are shown in Table 5 Overall, HEI produce significantly better results than SGHMC. We also show in the right plot of Figure estimates esstimates of Ep, ((x,((x,3) y)] for t = I,..., 50 after HMC hyper-parameter tuning and without hyper-parameter tuning.
Method/Dataset
Boston
Yacht
Concrete
Wine
SGHMC (best average) Springenberg 2016
33.7710.31
-13.5810.98
4.8710.05
-1.82.10755
SGHMC (tuned per dataset) Spruneeberg
2016
-2.4910.15
-1.75.10.19
4.16=0.72
-1.2911028
SGHMC (scale-adapted) Springenberg et al.
2016
-2.54+0.04
-1.11+0.088
-1.04+0.17
HEL
2.17
40.4710.06
-2.71 0.03 40.7110.03
Table 5: The log likelihood on UCI datasets averaged over 20 splits.
12

---

Under review as a conference paper at ICLR 2020
neguthe E100
Target Fotendsl
regative CUBO
Peterta Poteetial
1.0
traniing dortoon
Degathi
Targer
Tegative TUEO
350
600
EC0
1000
200
buikingg Borazera
builiigg Horalctt
braniing Bondboro
t-ariln Accutorn
14.5
1.25
4.0
1.0
3.75
0.
0.0
-0%
200
250
BCD
1000
420
600
2013
5020
200
200
500
1000
<ic
1000
Figure 6: histograms of samples from benchmarks by rejection sampling
4.5
4.0
3.5
logn
3.0
2.5
Bostons nt
Poston baaredd
2.0
Secni Init
Yranittraaees hed
1.5
Concrute
Concrite traneed
1.0
10
20
30
40
Chain length
Figure 7: The convergence of sample likelihood from ergodic approximation on Bayesian NN
13


Reviews:
[{"review_id": "HJlkXwkAtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a new hybrid method to unify MCMC and VI. The key idea is to interpret a ﬁnite-length MCMC/HMC chain as a parametric procedure, whose parameters can be optimized via a VI-motivated objective. Specifically, the authors propose to modify the well-known ELBO (which is now non-trivial due to the intractable entropy) to form a new constrained and tractable objective. The presented techniques are tested on synthetic datasets and with the experiments of a VAE on MNIST. \n\nThe presented technique is interesting. However, there are several concerns of mine that should be addressed, as detailed below.\n\nThe notations of \\pi and \\pi^* are very confusing. I guess \\pi represents the marginal distribution of the last state of the MCMC chain, while \\pi^* is the target distribution. Is that right? Please clarify their meanings. \n\nThere are related works that combine MCMC and VI, such as [1]. What are the advantages of the proposed method compared to that method? \n[1] Francisco J. R. Ruiz and Michalis K. Titsias. A Contrastive Divergence for Combining Variational Inference and MCMC. International Conference on Machine Learning (ICML). 2019.\n\nIn equation 4, given fixed P_0 and a long enough MCMC chain, P_T will decorrelate with P_0. How to prevent P_T from collapsing to a delta function? Also intuitively, there should be a weight balancing the two terms of the loss; why a weight of 1 is used?\n\nIn equation 8, the function g_{phi} is not continuous because of the indicator function 1(). How do you back-propagate through that function? In the paragraph before Section 3.3, how would you defend the adopted stop-gradient trick?\n\nIn the paragraph before Figure 1, how to choose the hyperparameter h? It might not be suitable to set h as the entropy of the prior, as in practice prior and posterior might be different dramatically.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkgsjnv_Yr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The presented method is very useful to deep learning in the era of uncertainty modelling, which requires the use of Bayesian inference arguments. It's a valuable improvement upon variational inference, it's novel, and the derivations are correct. The presentation is elaborate and covers all expected aspects. The literature review is up to date. \nThe experimental results are diverse enough and convincing. The authors have considered both proof of concept experiments and deep learning architectures. The comparisons are valid. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJxNl7DdOH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new combination of Markov chain Monte Carlo (MCMC) and variational inference (VI) for improving approximate inference. The main contribution is the optimization objective that allows improving the quality of samples obtained from the combination of VI and MCMC. Specifically, the authors minimize the \"approximate\" version of the Kullback-Leibler (KL) divergence between the distribution of MCMC + VI and the true distribution. The authors validate the effectiveness of their formulation through experiments on 6 synthetic benchmarks and generative modeling of MNIST (experiments on Bayesian neural networks are also provided in the appendix). \n\nOverall, I think the paper provides a solid contribution towards combining MCMC and VI by proposing a way to optimize the MCMC part. The experiments validate the method by showing consistent improvement over existing methods. However, I believe the justification behind the proposed formulation, i.e., Equation (4) and (5), needs to be improved before being published at the conference.\n\nFirst, for Equation (4), the explanation behind \"replacing\" H(P_{T}) with ELBO w.r.t. P_{0} is confusing. Specifically, it is reasoned that ELBO w.r.t. P_{t} only increase after MCMC steps. This statement is misleading since the replacement was done for H(P_{T}), not the ELBO w.r.t. P_{T}. \n\nI also think the Equation (5) is not properly justified. it is stated that the constraint is needed for preventing P_{T} to be closer to P_{0}. However, nothing is stated about the reason on why P_{T} gets closer to \\pi when Equation (5) is satisfied.  Note that even if the expected log-likelihood of the distribution is high, it does not necessarily mean that the distribution is more similar. \n\nMinor comments:\n- I was unable to understand why the algorithm is named \"ergodic\" inference. Both HVI and the proposed EI rely on the ergodic property of Markov chain for improving the variational distribution. I hope the authors could better illustrate on this point. I also think the term \"ergodic approximation\" in page 3. is hard to understand.\n- I (weakly) suggest changing y-axis of Figure 5. to log-scale for better readability. It almost seems that the brown plot does not converge in Fig 5-(a).\n- The paper could have been strengthened by performing experiments on more challenging datasets, e.g., CIFAR-10 or CIFAR-100. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a way of adapting an HMC-based posterior inference algorithm. It's based on two approximations: replacing the entropy of the final state with the entropy of the initial state, and differentiating through the MH acceptance step. Experiments show it is able to sample from some toy distributions and achieves slightly higher log-likelihood on binarized MNIST than competing approaches.

The paper is well-written, and the experiments seem pretty reasonable.

I don't find the motivations for the aforementioned approximations very convincing. It's claimed that encouraging entropy of P_0 has a similar effect to encouraging entropy of P_T, but it seems easy to come up with situations where the algorithm could "cheat" by finding a high-entropy P_0 which leads straight downhill to an atypically high-density region. Similarly, there was some reviewer discussion about whether it's OK to differentiate through the indicator function; while we differentiate through nondifferentiable functions all the time, it makes no sense to differentiate through a discontinuous function. (This is a big part of why adaptive HMC is hard.)

This paper has some promising ideas, but overall the reviewers and I don't think this is quite ready.

Decision: Reject