Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposed to use the uncorrected HMC kernel in an AIS-type variational scheme. This enables optimizing the ELBO via reparameterization gradients for many algorithm tuning parameters, including step size, momentum covariance, and annealing schedule. The method is shown to outperform competitors in most cases. 
main_review: The proposed method appears to be sound and potentially quite useful. A few suggestions:

1. The proof of Thm 2 lacks some rigor – it would be better to argue measure theoretically.

2. Many of the papers cited as being on arXiv are published either in workshops or conferences. The author(s) should check all arXiv citations carefully. One tricky case is [17], where there's a related published paper https://papers.nips.cc/paper/2016/file/0e9fa1f3e9e66792401a6972d477dcc3-Paper.pdf

3. There are a few missing pieces and areas for improvement in the experiments:

- Please include training runtimes for the different methods
- I would have liked to see a "toy" example with known log Z to demonstrate the tightness of the lower bounds produced by the proposed method
- Given that the proposed method produces a posterior approximation, some comparison of the approximation accuracy compared to, say, Stan's dynamic HMC implementation would have been very interesting. I realize posterior approximation per se is not the focus of the paper, but I wonder how competitive the proposed method would be with dynamic HMC when measured in terms of number of gradient evaluations. Even if it were not quite as good, the fact that is also produces an estimate of the marginal likelihood could make the method a valuable alternative when the user wants to do model selection 
limitations_and_societal_impact: There was some discussion of limitations
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes UHA, an original and clever modification of Hamiltonian AIS. The idea is to change the AIS ratio such that it becomes tunable, but can still be computed in closed form (ratios of momentum distributions). This is achieved by dropping the accept-reject step in the HMC update such that a single UHA step only involves (re)sampling momenta and leapfrog integration. This modification allows the authors to tune parameters such as the proposal distribution, the step size and damping coefficients (used in momentum resampling). Many interesting ideas for future extensions are proposed. Experiments on various models demonstrate that UHA obtains a tighter ELBO than other combinations of VI and HMC. 

main_review: # Contributions

* Variational inference with Uncorrected Hamiltonian Annealing (UHA)

* Achieves tighter ELBO and allows optimization of parameters 

# Originality

The paper proposes UHA, an original and clever modification of Hamiltonian AIS. The idea is to change the AIS ratio such that it becomes tunable, but can still be computed in closed form (ratios of momentum distributions). This is achieved by dropping the accept-reject step in the HMC update such that a single UHA step only involves (re)sampling momenta and leapfrog integration. This modification allows the authors to tune parameters such as the proposal distribution, the step size and damping coefficients (used in momentum resampling). Many interesting ideas for future extensions are proposed. 

# Quality

The paper is technically sound and the experiments are instructive. 

# Clarity

In general, the paper is well written. 

Although the notation is generally consistent, there are a few ambiguities:

- Page 2, Lines 47/48: Here you denote the number of bridging distributions by $K$ although in the remainder $M$ is the number of bridging distributions (and $K$ the number of likelihood evaluations)

- Page 5, Line 172: How are the $\beta_m$ chosen? When tuning also the $\beta_m$ (Figure 2), it would be interesting to see the tuned annealing schedule. 

- Page 6, Lines 210-213: What is $d$? I guess the dimension of $z$ space? 

# Significance

The experiments show that UHA outperforms other variants of Hamiltonian sampling and variational inference. 

However, to better judge the improvements achieved with UHA it would be useful to also show the correct log evidence (or rather an accurate estimate obtained, for example, with standard AIS using many bridging distributions and multiple parallel trajectories). It would be helpful to show these estimates in Figure 1-3. 

# Comments

* Line 204: You perform only a single leapfrog step. Have you run tested with more leapfrog steps?

* In standard AIS, multiple annealing sequences ("trajectories") are simulated independently. Have you considered doing something similar with UHA (resulting in yet another augmentation of the sample space)?

limitations_and_societal_impact: -
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper propose to use uncorrected HMC kernels (i.e. HMC kernels without accept-reject steps) in AIS to make the estimate of log normalisation constant differentiable w.r.t. to various parameters (e.g. step-size for numerical integration).
The author(s) name(s) the method UHA.
The promise is that one can then using reparameterisation gradient to tune these parameters.
Through experiments on inference tasks,
the benefit of tuning parameters in UHA is demonstrated against other methods that use HMC in AIS.
For experiments on VAE training,
UHA with tuned parameters has shown better test ELBO than IW.
main_review: Originality: 
UHA is a novel method derived by a neat modification to the common way of using HMC in AIS.
It solves an important limitation of previous methods: they are sensitive to hyper-parameters but there is no easy way to tune them.
Differences against previous methods are made clear through the motivation as well as the dedicated related work section.

Quality: 
The method is technically sound and the benefit of being able to tuning all parameters is clearly demonstrated by experiments in section 5.1 and 5.2.

Clarity: 
The paper is very well-written and easy to follow.
Technical details are also enough for reproducing the results.

Significance: 
The fact that the proposed method leads to a differentiable estimate is a solid contribution to the community.
One of the biggest challenges to use HMC in AIS is about hyper-parameter tuning and UHA and I can see how UHA would popularise the use of AIS as now the hyper-parameter tuning is made easy.

Misc:
- An interesting baseline that is not included is tuning parameters using the REINFORCE estimator (either in UHA or HAIS), although I could expect it would be worse than the reparameterisation estimator.
- The grid search for HAIS is insufficient. One interesting thing to check from a more fine-grained search of HAIS is that we could see how much we lose by getting rid of the accept-reject step. Another interesting thing to check is how different the optimal parameters are for HAIS and UHA.
- For the last sentence from L291, how do we know the true $\log Z$ to compute the variational gap?
limitations_and_societal_impact: Limitations are discussed in section 6.
There is no societal impact discussed but I don't think it's needed for this technical paper.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper studies the annealed important sampling (AIS) using the Hamiltonian dynamics kernels in each interpolant. To circumvent the problem that the Hamiltonian dynamics involves the non-differentiable operations, this paper proposes to discard the accept-reject operations in the kernel, and derives the resulting density ratios in a simple form related to the momentum variables. The resulting approach, named Uncorrected Hamiltonian Annealing (UHA), is a fully differentiable AIS method. This paper validates the effectiveness of UHA through extensive experiments and highlights the importance of differentiability. 
main_review: ## Novelty and Significance
Variational inference and MCMC are mainstream approaches for posterior inference, with both merits and shortcomings. Using the MCMC kernels in the variational posterior is an appealing direction to combine both merits together. And this paper makes an important step. By discarding the accept-reject step in the Hamiltonian kernel, UHA enables full differentiability. Compared to the standard AIS where the density ratios can be computed by the interpolated distributions, this paper presents an expression that can be simply obtained from the momentum variables in the Hamiltonian dynamics. Therefore, I think the proposed UHA is novel and  important in the field.

## The tightness of the bound
By using more samples, IWAE [1] provably tighten the lower bound compared to the standard ELBO, which enables better model learning. In contrast, the UHA introduces an augmented space for the inference. Is there any guarantees on the tightness of the bound ?

## Test-Time Encoder
Does the encoder at test-time also involve the UHA procedure ? If so, how large is the computational overhead compared to a standard NN encoder ? 


## Clarity
This paper is well written. This paper clearly introduces the background knowledge and presents the improvement of the proposed approach. Detailed algorithm details are provided.

## Experiment
I do not have the expertise to fully evaluate the experiments. But I particularly enjoy seeing the improvements by turning more parameters in Figure 2 and Figure 3. I think they are strong evidences on the benefits of the proposed approach.
limitations_and_societal_impact: I do not see a big limitation of the approach.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors introduce a variational inference method that leverages Annealed Importance Sampling (AIS) and Langevin/HMC MCMC. In particular HMC dynamics are used to target a sequence of intermediate bridging densities in order to generate approximate samples from the target posterior distribution. Importantly, since the method is uncorrected (i.e. does without a accept/reject step) the resulting lower bound is fully differentiable. This makes various hard-to-set parameters (e.g. temperatures for bridging densities) learnable. In experiments the authors demonstrate that their method, Uncorrected Hamiltonian Annealing (UHA), leads to tight lower bounds on the log evidence. 
main_review: This work presents an elegant approach for combining AIS and HMC within the confines of variational inference. While various authors have proposed methods that marry MCMC with variational inference over the years, some of the resulting constructions are somewhat clunky, requiring e.g. learning reverse models. In contrast the proposed method, UHA, is comparatively simple and does not require learning complex neural networks or the like. What is particularly attractive about this approach is that it effectively defines an implicit/semi-parametric variational distribution that makes extensive use of the target's gradient information. For this reason one would expect UHA to be an effective inference algorithm in a wide variety of scenarios without the need for model specific modifications (at least if the method is reasonably numerical stable). So while this work is in some sense incremental in that it combines existing algorithms from the literature, it does so in a principled way that cleverly addresses an important class of problems. The experiments are reasonably comprehensive and convincing.

Detailed questions and comments:

- The related work section is rather cursory. I think it would be very valuable if the authors extended the discussion there to give a more comprehensive comparison to existing methods that combine MCMC with variational inference. Even if there is not sufficient room in the main text, this discussion could be included in the supplemental materials. Apart from this shortcoming, however, the paper is generally clearly written.

- "In all cases, for UHA and Hamiltonian AIS we use under-damped Langevin dynamics, that is, we perform just one leapfrog step per transition and partially resample momentum." Can you comment on the choice of one leapfrog step? For a fixed amount of computation cost is the regime with more leapfrog steps and fewer bridging densities less effective? Or is this choice driven by numerical stabilities or some other concern?

- Can you comment on using different momentum distributions for each bridging density? It seems like this could be important. Is there a simple parametric scaling that suggests itself?

- Can you comment on numerical stability of the method? It seems you initialize with a vanilla mean field distribution obtained from ELBO maximization. Does everything break down without this initialization scheme, e.g. if you just initialize q(z) to be the unit normal distribution?

- How well can UHA deal with multi-modality? It would be informative to e.g. consider a 2-dimensional multi-modal example and depict the learned variational distribution for increasing values of K.

- "However, for smaller K, IW sometimes had better log-likelihoods than UHA (despite worse ELBOs)" Any hypothesis as to why this is?

- There is no discussion of runtime or computational complexity. It would be useful to get an idea of how expensive UHA becomes for large K.

- "For K ≥ 64 we tune the UHA’s parameters using K = 64 and extrapolate them as explained in Appendix D." Is this entirely because of computational cost or are there other factors in play here as well (e.g. stability)?

## After author response

I find that the authors have answered questions raised by the reviewers adequately and continue to believe that this paper should be accepted for publication. The elegance and (relative) simplicity of UHA make it a valuable contribution to the literature, and I imagine it is quite likely that follow-up work will exploit the ideas introduced here in other fruitful contexts.
limitations_and_societal_impact: The authors should more explicitly address some of the (potential) limitations of their work. For example: computational complexity, performance on multi-modal problems, non-applicability to discrete latent variables, etc

The authors do not address potential negative societal impact of their work.
review_title: null

Meta-review: The reviewers all came to a clear consensus that this paper should be accepted. The authors addressed all points raised by reviewers thoroughly. In the camera ready, please make sure to follow through on the edits discussed during the review process. In particular, multiple reviewers pointed out that the experiments should include runtimes, and that there should also be an experiment that provides a comparison to the true log marginal -- please ensure these are included in the camera-ready. Please also improve the rigour of the proof of Theorem 2 using measure-theoretic arguments.

Decision: Accept (Poster)