Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper proposes a new MCMC transition kernel. This kernel is parameterized by neural networks and is optimized through an objective maximizing the proposal entropy. Specifically, the authors use a combination of a flow model and non-volume preserving flow in [Dinh et al., 2016] as the neural network parameterized kernel. Then they use the objective in [Titsias & Dellaportas, 2019] which maximizes the proposal entropy to optimize the kernel. The proposed method is tested on synthetic datasets, Bayesian logistic regression and a deep energy-based model.

The problem of improving the exploration efficiency of MCMC kernel is important. The proposed method is well-motivated. As far as I understand, the proposed method appears to be technically sound. 

However, I have the following concerns about the paper.

-	The connection and the difference to previous work are not very clear. If I understand it correctly, the proposed method seems a combination of L2HMC and [Titsias & Dellaportas, 2019] with some slight modification (a flow model) since the naïve combination did not work well (as stated in Section 4). I think it would improve the clarity a lot if the authors explain more clearly how the proposed method differs from previous work. 
-	Since the use of a flow model is the main difference compared to the naïve combination of two previous methods, the authors should explain more about this choice. Currently, it is not clear why this helps and there is no explanation on why the naïve combination fails. 
-	The proposed method seems to use more neural networks (e.g. an additional network R to handle gradient) than previous neural network MCMC. I wonder how the method performs if considering the cost. For example, the authors may instead show ESS per second on the experiments in Section 5.1. Though the authors mentioned the difficulty of computing ESS per second in the paper, I’m not entirely convinced. As the experiments in Section 5.1 are all very small-scale, is it really necessary to use GPUs? 

-	The baselines vary from experiments to experiments for no reason. For example, the authors compare their method to L2HMC on ill-conditioned Gaussian and strongly correlated Gaussian, to Neutra on Funnel distribution, and to MALA on EBM. I think this experiment design needs explanation. Also, there is no empirical comparison to [Titsias & Dellaportas, 2019] which is closely related to the proposed method.

Some minor comments:

-	A.1 intends to show the benefit of using gradient information. But Variant 2 also uses gradient. What is the point of showing it? 
-	It is not clear to me how to interpret the empirical results in Section 5.2. For example, how does Figure 3 show that the proposed method needs less sampling steps? 
-	The color of points in figure 1 is hard to read.

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors describe an approach for adaptive MCMC which uses a proposal distribution parameterized by a neural network and which optimizes the entropy of the resulting proposal. Overall I found it to be an interesting algorithm, with fairly good results as compared to alternative methods, however the description of the algorithm itself was somewhat confusing and/or convoluted.

The introduction (including section 2) was great, and provided a concise but very readable introduction to recent approaches for adaptive MCMC. The beginning of section 3 was also quite well presented. However, when the authors turn to the parts of their approach inspired by HMC it gets a bit murkier. Part of the difficulty here is that at this point (the first paragraph of p4) the authors are describing HMC rather than their method, which is not entirely clear. Perhaps this could have been simplified by moving the "related work" section earlier in the paper and allowing for the description of HMC before diving into their own approach.

The relation between HMC and their approach which makes use of intermediate steps x+R could also have been more thoroughly explained and given more intuition. The link is there, but it doesn't seem to be as close to the leapfrog step as x+R doesn't correspond to an intermediate step (which would be x+z^n). Not that I'm saying in any way that this makes the algorithm incorrect, just that the connection isn't quite as clear cut.

Similarly I would also have liked to see a more clear description of the Q, T, and S matrices.

Throughout this work the authors also refer to "during training". I assume they perform their adaptation steps during the sample process as is the case of the Titsias and Dellaportas work, however this could use some clarification. Overall, I think that this and other confusions cited above could have been done away with by including a clearer outline/overview of the algorithm as a whole.

Finally, overall the results seem to be quite a bit better than competing methods (although I'm not an expert in this area). However, although the authors hasted to add that they do not compare "exact computation time or ESS/second" it would have been nice to see a more thorough discussion of the relative computational complexity of the alternatives. While ESS/grad does in some sense get close to this, the computation necessary for networks (depending on their size) could play a factor here. Similarly, I would like to see more discussion with regards to the choice of architecture for these networks---ie were they a simple MLP? It's possible I missed this, but do not think it was discussed.
review_title: Interesting paper, but somewhat convoluted description of the algorithm

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The author proposes a novel MCMC sampler parametrized by the neural networks. In particular, the neural network is chosen to be a flow-based model that allows the exact evaluation of the proposal probability. 

The update equations of MCMC mimic the dynamics of HMC by incorporating the gradient of the energy function into the flow. To ensure the invertibility, each update equation only depends on the other part of the variables. In addition, the author also designs another network $R$ to propose an evaluation location for the target gradient. This ensures the similarity between the proposed method and HMC. 

As for the training objective, the author proposes to maximize the proposal entropy and acceptance rate, controlled by coefficient $\beta$. Due to the tractability of the proposal density, this objective can be analytically computed. Stop gradient trick is used to stabilize the training and reduce the cost of back-propagation. 

Empirically, the author evaluates the proposed sampler in some toy datasets, logistic regression, and deep energy-based models. The proposed sampler achieves higher ESS compared to baselines.

---------
Review:
Clarity: The paper is clearly written and easy to follow. The author also addresses the related work and mention the difference compared to previous baselines. 

Technical soundness:
I have a quick look at the details of the proposed method. It seems the derivation is correct. 

Novelty: 
Although the structure of the sampler is inspired by L2HMC, there are still some differences. L2HMC generalizes the HMC update equation by partitioning $\pmb{x}$ into two parts, each part is parameterized by neural networks. The proposed sampler instead partitions the state $\pmb{z}$ instead of parameter $\pmb{x}$ with additional network $R$. The idea of using entropy as an objective to encourage exploration is not new.  However, the novelty lies in the usage of the flow model to allows tractability of proposal density. Overall, the proposed method is novel to some extent.  

Significance of the work:
The proposed method can be regarded as a variant of L2HMC with slightly different NN parameterization and training objectives. The advantage of the proposed method is the higher ESS compared to previous samplers. This may be helpful to some audiences.

Weakness:
1. Although the author demonstrates the better ESS can be obtained using the proposed method, I still prefer more analysis to understand the properties of this sampler. For example, I am curious to know how important the network $R$ is? If $R$ is removed, and the leapfrog integrator is used (i.e. update $z^{n'}$ followed by updating $x^{n}$ and finally $z^{n}$ to mimic HMC update rule). Then the gradient evaluation is also evaluated at different locations. What are the differences in terms of performances?
2. The reason that the proposed method can have higher ESS is due to the maximization of the proposal entropy. Although the training objective has an acceptance rate term, I am curious to know: does the entropy term hurts the sample quality? For example, in logistic regression, the author only reports the ESS. What about the convergence speed of the sampler compared to others that use sample quality as the training objective? What about their performances in logistic regression?
3. For training deep EBM, apart from ESS. I also want to know the convergence speed of the EBM training and the quality of the generated images compared to MALA. These are the standard evaluation metric for EBM.
4. For EBM, why only compared to MALA. Any reasons why excludes other baselines?


review_title: Interesting MCMC samplers but need some improvements in empirical analysis.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper argues that a better objective to train neural MCMC kernels is to maximize the proposal entropy (Titsias & Dellaportas, 2019) and demonstrate a method on doing so. The method shows improved sampling efficiency compared to previous method, especially one that optimize the alternative L2 expected jump. The novelty is not of the training objective but a neural instantiation with improved sampling efficiency.

## Pros

1. The method is well-motivated in Section 1 and clearly demonstrated by Figure 1.
2. The neural instantiation consists a few clever tricks to make the network tractable and explore the space well.
3. The method demonstrates improved sampling efficiency, measured by ESS.

## Cons

1. The paper may need more work on presentation.
- Section 3 is hard to follow. It's very long and maybe adding some subsection would help.
- I feel Section 4 should go before Section 3, given how it is currently presented.

2. There is a false statements in the paper: "one can easily make any transition probability satisfy it by including an additional Metropolis-Hastings accept-reject step (Hastings, 1970)"
- It's not true. One need to additionally ensure that the kernel is irreducible and aperiodic.
- For the reason above, it's worth the mention that the proposed neural approach is irreducible and aperiodic, and give some justificaiton.

3. No experiments for correctness check. I understand that showing better ESS is good, but even some biased sampler can lead to much improved ESS. I think there should be some results on comparing the proposed method against well-established HMC on a few Bayesian inference problem in terms of the posterior. Using some hypothesis testing methods to check if the posterior actually matches would give me more confidence that the method is sampling from the target; or at least, comparing a few moments.

## Questions

1. "In (Titsias & Dellaportas, 2019), the authors avoid multiple back-propagation by stopping the derivative calculation at the density gradient term. In our experiment, we find it is necessary for good performance.": does it mean you effectively using a biased gradient and it turns out to be better than an unbiased one? It looks quite weird to me as it means you are in fact optimizing some other objective which turns out to be better.

2. I'm generally unsatisfied with the setup of Section 5.2.
- "which removes the need to choose the step size. The only tunable parameter is a target accept rate.": I think vanilla HMC or MALA can do so by some online otpimization for step size as well (e.g. dual averaging as it's done in NUTS).
- The unstable training of EBMs via contrastive divergence comes from the fact that the gradient estimated by a short-run MCMC is **unbiased**. The proposed method itself doesn't deal with the biasness directly, so the only plausible reason to explain why it improves the training is that the mixing is so good such that with the short chain, the bias of the gradient is so small. Is this what happening?
- I wonder whether or not the simultaneous/interweaved training of samplers and EBMs would cause any issue. In particular, will it encourage that EBM only putting energy around data points and the sampler only effectively draw samples from some noisy distribution nearby? Are samples from EBMs too similar to data points.
- For Figure 3.b, what's the convergence behaviour, i.e. do learnable sampler and MALA converge to a similar entropy or not?
- For Figure 3.c, does long-run MALA produce sensible samples? If yes, it indicates that this way of training EBMs avoid some pathology which CD with short-run MCMC has; if no, then it may indicate that somehow the learnable sampler co-adapt with the EBM somehow to avoid the pathology. Is the latter we wanted?
review_title: Interesting paper

Meta-review: This paper proposed an MCMC sampler that combines HMC and neural network based proposal distribution. It is an improvement over L2HMC and [Titsias & Dellaportas, 2019], with the major innovation being that, the proposed normalizing flow-based proposal is engineered such that the density of the proposal $q(x'|x)$ is tractable. Experiments are conducted on synthetic distributions, Bayesian logistic regression and deep energy-based model training.

While reviewers are overall happy about the novelty of the approach, some clarity issues have been raised in some of the reviewers' initial reviews. Also concerns on the evaluation settings, including the missing evaluation metric such as ESS/second, are also raised by the reviewers. The revision addressed some of the clarity issues, but some experimental evaluation issues still exist (e.g. comparing with L2HMC in terms of ESS/second), and the replaced MALA baseline results make the improvement of the proposed approach less clear.

I personally find the proposed approach as a very interesting concept. However I also agree with the reviewers that more experimental studies need to be done in order to understand the real gain of the approach. 
Decision: Reject