Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper approaches the problem of learning the optimal parameters of the MCMC chain. Namely, it tries to design an objective function that allows for the learning of a perfect proposal distribution in the Metropolis-Hastings algorithm. The main contribution of the current paper is based on the developments from [Titsias, 2019]. [Titsias, 2019] proposes the following objective for the optimization of the proposal
$$\text{GSM} = \text{Acceptance Rate} + \beta \cdot \text{Entropy of the proposal}.$$
Maximization of this objective results in better mixing properties of the chain. Indeed, the next sample comes from a distribution with high entropy and should be accepted with a high probability, and the proposal should converge to the independent sampler in the perfect scenario. 

However, the choice of $\beta$ remains to be a question since it defines the tradeoff between the acceptance rate and the entropy, which is known to be different for different proposals and different targets. The original paper [Titsias, 2019] solves this issue by adaptively setting the acceptance rate to some desired value.

The current paper then can be summarized as follows. Instead of setting $\beta$ adaptively based on the acceptance rate, the authors propose to set it constant. The constant is then found by "manual trial and error" in such a way that the acceptance rate of Random-Walk Metropolis-Hastings is 0.234 for high-dimensional standard Gaussian, which is known to be optimal for this task. Then this constant is claimed to be universal for all targets and proposals. To verify this empirically, the authors first optimize the parameters of MALA and RW Metropolis-Hastings for several synthetic targets (Uniform, Laplace, Cauchy, Gaussian) and demonstrate that the procedure yields the right parameters.

Finally, the authors perform an empirical study of the proposed procedure. They use the proposed objective to learn a proposal parameterized by the normalizing flow and compare it to the optimization of the original objective from [Titsias, 2019] targeting different acceptance rates. The target distributions for the empirical study are a mixture of four Gaussians (2-D), Bayesian logistic regression (21-D, 14-D).

[Titsias, 2019]: Titsias, Michalis, and Petros Dellaportas. "Gradient-based adaptive markov chain monte carlo." Advances in Neural Information Processing Systems 32 (2019): 15730-15739.
main_review: Although the paper approaches an important problem in the MCMC field, its contribution is incremental. One may consider it as a special case of the objective from [Titsias, 2019]. The proposed approach has several major issues:
- The conjecture of the existence of universal $\beta$ is not well-supported. The authors motivate the choice of GSM objective by verifying several properties. However, further developments are based on the conjecture that there should be some optimal tradeoff between the acceptance rate and the proposal entropy for all target distribution and all proposals. This conjecture is validated neither theoretically nor empirically.
- The way of finding $\beta$ by "manual trial and error" is not acceptable. Although the authors mention that it is possible to estimate the constant using some grounded techniques, they describe their procedure as "trial and error". This fact cannot persuade practitioners to use the proposed objective. The constant should be estimated at least by using grid search demonstrating the performance of the algorithm for different values. Furthermore, this empirical study has to be extended to different proposals and different targets demonstrating the universality of the constant. Finally, the variance of the estimation should be reported.
- The evaluation of the proposed approach is not practical. To be more precise, the proposed objective requires samples from the target distribution. Throughout the experiments, the authors collect these samples separately running a long HMC chain. However, obtaining a large set of uncorrelated samples from the target using HMC could be impossible in practice. Not to mention that this is the final goal of any MCCM algorithm. Note that in [Titsias, 2019], this issue is bypassed by collecting the samples online and treating them as samples from the target distribution. This setting is very important for practical applications and should be explored empirically.

Other comments:
- Section 7.2 is not motivated. Indeed, some MCMC algorithms use augmented space for sampling. However, the reason for using it here is unclear. This section could give more insights to the reader if two algorithms have been compared (with augmented space and without).
- Section 7.3 is unclear. The authors conclude that the proposed objective yields a robust measure of MCMC efficiency and can be used as a metric for comparison. However, this conclusion is not argued. In all previous examples, the authors rely on ESS to measure the efficiency of chains and motivate the effectiveness of their objective also by ESS. Thus, we have a question why would one use the proposed objective instead of ESS to measure the efficiency?

summary_of_the_review: The paper proposes and analyses a special case of the objective from [Titsias, 2019]. Unfortunately, the empirical study is not sufficient to make positive conclusions about the proposed technique.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors design an objective function from first principles that can be used to optimize proposal distributions for MCMC. They empirically verify that their objective function with a fixed hyperparameter--tuned a priori on another task--can reproduce theoretical results over a variety of target distributions.
main_review: # Strengths

I think the problem was amazingly motivated. The description of the objective function was very clear: both in terms of its description as well as its need. I am also fascinated that for a fixed $A$, the proposed approach is able to reproduce classic theory regarding the optimal acceptance probability for random walk metropolis and metropolis adjust langevin diffusion. 

# Weaknesses

I first want to note that the length of the weakness section relative to the strength section does not reflect my opinion of the paper. While I'm a big fan of the approach I'm somewhat confused about the actual proposed method. First, the proposed objective function is an expectation with respect to the target density, $\pi(x)$. While I understand that theoretically, this is important, practically this is *very* restrictive as one either needs to have access to the target density--which if this is the case, why do MCMC--or one needs to run access to approximate samples, but if you have access to these samples why run this method. 

The second thing that isn't clear is the actual implementation: is the proposal optimized before running the MCMC sampler or is it optimized while the MCMC algorithm is running? Also, in Appendix D it says "For optimization, 20000 gradient steps are taken, with each training batch taken from a single starting point sampled i.i.d. from the target distribution from which 50 proposals are generated to calculate expected loss". I'm confused about what is meant by 50 proposals? Is the gradient estimated using 50 samples? Is the sampler run for 50 time steps, and those samples are used to approximate expectations? This same question is also relevant to the experiment section as well. The confusion around proposals makes it much harder to understand what is actually going on. 

Lastly, I have some comments about the experiments. I think for all quantities, it would be useful to state whether higher/lower is better.
In experiment 7.1, the authors state "Position dependence within the proposal distribution lead MSJD and L2HMC’s modification to optimize towards arbitrarily non-ergodic proposal distributions, sampling only either the horizontal or vertical components of the target distribution.". From this description, I would expect the MJSD and L2HMC to not mix well but according to the results in Table 3, they both have very high ESS (also I thought ESS was upper bounded by 1?) and MSJD. I noticed a similar discrepancy in 7.3 where the authors state "The resampling normalizing flow did not produce a single accepted proposal within these evaluation Markov Chains, hence we are unable to estimate ESS and our estimates of acceptance rates and MSJD for this proposal distribution should be viewed as highly uncertain." but the resampling normalizing flow has the highest Ab Initio! Also, I'm confused about what resampling normalizing flow is (I also don't think it was described anywhere in the paper).

# Comments
1. There should be a section on how the ESS was computed.
2. It would be beneficial to the reader if MSJD was defined.
summary_of_the_review: While I like the motivation of the paper, I think the actual algorithm and the experimental details are somewhat opaque. This can be easily fixed by fixing the manuscript and making these things very clear. I'm also somewhat concerned about the practicality of the algorithm and I think this needs to be addressed by the authors in the manuscript. For these reasons, I lean towards a weak reject though I am more than happy to increase my scores if the authors address my concerns.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes four desirable properties that should be satisfied by objective functions used to tune MCMC proposals. These are based on first principles, such as that the objective must have a unique global minimum when the proposal is equal to the target, must be representation invariant, and must recover known optimal results in well-studied settings (e.g. optimal acceptance rate for Langevin dynamics with a Gaussian target). Using these principles it proposes a new objective based on GSM (Titsias and Dellaportas, 2019) that satisfies all proposed principles.
main_review: Strengths:
- The paper tackles the important problem of developing objectives to automatically tune MCMC proposals. This is necessary to tune complex and flexible proposals parameterized by neural networks.
- The first principles provide a sensible guide to design of objectives to tune MCMC proposals. So far different objectives have been proposed without a clear underlying framework. This paper takes a first step in that direction.

Weaknesses:
- While the formulation with the guiding principles is quite general, the only concrete objective proposed is a modification of GSM. Also, even in this case, some parameters are tuned using some ad-hoc methods. For instance, the value of the constant A is set by targeting some optimally known acceptance ratio for a specific setting, and a factor ``d'' is added to correct for the magnitude miss-match between different terms in the objective. Thus, despite the generic presentation, it is not clear to me whether this framework actually simplifies the task of creating new objectives to tune MCMC proposals.
- I'm not fully convinced by the empirical evaluation. The models considered, while illustrative, are quite simple (e.g. Gaussians, low dimensional logistic regression). In addition, the method is not tested for online tuning of MCMC kernel parameters, which is how it would be used in practice. All the evaluations are using true samples from the target, or samples obtained running a long HMC chain. While the authors add this as future work, I think including it in this work would be quite good.
summary_of_the_review: Overall I think the whole idea of providing a set of guiding principles for the design of objectives to tune MCMC proposals is interesting and relevant. However, I feel the presentation in this paper may be a bit too generic, and the derivation of the one concrete objective requires a decent amount of manual tuning. Thus, it is not extremely clear to me whether this framework, as presented, simplifies the development of new objectives significantly or not. In addition, I think the concrete objective proposed could be tested more thoroughly, specifically in some settings where samples from the target are not available (i.e. tuning its parameters online as most adaptive MCMC methods do).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed an objective function for training neural MCMC proposal, called Ab Initio objective. The new objective seems to improve upon existing alternatives by maintaining property and representation invariance, and reducing the requirement of prior knowledges (e.g., optimal acceptance rate). The author demonstrated the effectiveness of Ab Initio objective on several benchmark models.
main_review: Finding an effective objective function for training neural MCMC proposals is of great importance for MCMC methods. However, the effectiveness of the proposed Ab Initio objective requires further justifications. Below are my major concerns.

1. Although the motivation for the new objective is quite general, the author ends up with a rather heuristic formulation of the Ab Initio objective matching the optimal acceptance rate of RWM for multivariate Gaussian targets. The general applicability hence is quite doubtful given the relatively simple target distributions used in the demonstrations. The author may try out more complicated models. Also, no significant advantages over MSJD and L2HMC had been found in the experiments. 

2. The computation of the Ab Initio objective is not clearly stated. Given that \pi is unknown, the expectation in the first term of L[g;\pi] seems to be intractable in practice.

3. The author claimed (3) can approximate HMC proposals. This should be verified by comparing with the hand-tuned HMC methods.
summary_of_the_review: The proposed Ab Initio objective seems to be an interesting addition to the current candidates of training objectives for MCMC proposals. However, given the heuristic formulation and inadequate empirical evidence, I am cautious about its general applicability.
review_title: null

Meta-review: This paper proposes guiding principles with which to design objective functions for proposal distributions for MCMC. They design one such objective based on GSM (Titsias and Dellaportas, 2019). The two concerns raised by reviewers that resonated the most with me were:

- it was not clear that the actual proposed objective was the best way to implement these guiding principles
- a weak empirical evaluation that did not consider online tuning and high-dim, highly non-Gaussian targets. 

After rebuttal, revision, and discussion, reviewers felt that the authors did a reasonable job of addressing the issue of online tuning, but very highly non-Gaussian targets were not addressed. There was still a sense that the ultimate instantiation of the design principles was a somewhat adhoc loss. Ultimately, I think that this work is just below the bar for acceptance and it can be improved by clarifying the choices made in implementing the objective and some more ambitious experiments.
Decision: Reject