Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper deals with performing Langevin dynamics as a form of sampling, only using an amortisation mechanism. This reduces the computational overhead of performing the sampling. The method is then verified on synthetic and generated data sets

strengths_and_weaknesses: The paper is very clearly written and cleanly presented. It is worth mentioning that the authors also did a very good job introducing other related works (albeit a bit late in the text at section 4).
More so I think The main theorem is an important one. The idea of unfing the generated Markov chains with respect to the data set is novel.
In top of that using amortisation is a fantastic idea for a Markovian process since the lack of time correlation can make this efficient much like Mori Zwanzig operators untangle non markovian processes 
That being said, there are a few things I am a bit worried about. It seems like to sample the ELBO there is a need to resample all the chains ?

questions: You have used EM numerical solver, which has a lof of well defined numerical stability criterion, for example : Higham, D.J., 2001. An algorithmic introduction to numerical simulation of stochastic differential equations. SIAM review, 43(3), pp.525-546. Did you check any of them ?

What is really fascinating about Langevin dynamics is that for a certain class you derive a shadow lemma which means you are tractable towards your “real” underlying distribution for example : https://arxiv.org/abs/1107.2967
Can this analysis somehow improve the theorem on G

limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces amorized Langevin dynamics (ALD), a method for initializing Langevin dynamics in the function parameter space that makes it efficient to perform posterior sampling. Based on this method, the paper presents the Langevin autoencoder (LAE), a deep latent variable model that is easy to implement and has competitive performance compared to similar existing methods.
strengths_and_weaknesses: Strengths:
- The preliminaries and the core method are explained clearly. The drawbacks of existing MCMC and variational inference methods motivate the main ideas very well.
- The algorithm of LAE is presented very clearly (Algorithm 2).
- The idea of performing Langevin sampling in encoder parameter space seems quite novel and interesting. Proof is given on why this MCMC in the encoder parameter space will converge to the true data posterior.

Weaknesses:
- Although it doesn't get in the way of understanding the key ideas, the writing can certainly be improved at a couple of places. Examples include "a more straightforward and sophisticated framework" (line 48), "replacing MCMC on the latent space into the encoder's parameter space" (line 52) and "we substitue it for its evidence lower bound" (line 277, I believe it should be the other way around). Should be easy to fix.
- I find the claim that ALD "completely removes datapoint-wise iterations" slightly misleading, since it does so trivially by turning datapoint-wise iterations into parameter space iterations. In essence it still relies on the encoder (up to the second to last layer) to initialize the MCMC in a high-density region, which is still pretty similar to Hoffman (2017). Maybe it can be shown that this method can have the same or better performance with less MCMC iterations than Hoffman, at which point it would be interesting to discuss why moving to parameter space gives such an advantage.
- Without providing details on hyperparameter search (which can influence results greatly), the experimental results might not be sufficient for concluding that the LAE "outperforms" the existing methods. As mentioned in the last point, I think it would be really beneficial to show how the performance of the Hoffman (2017) model is influenced by the number of MCMC steps used.
questions: - In Hoffman (2017) the encoder parameters are updated via gradient descent on the ELBO. However in this work the gradients are taken w.r.t. $V$ instead. I did not find derivations for this gradient update. How is this gradient update justified? What is the objective function in this case?
- What do you think could be the reason behind the efficiency of performing MCMC in the parameter space instead of the data space?
limitations: The main limitation of the method seems to be the requirement that the size of the last linear encoder layer needs to be larger than the batch size. This is addressed in the experimental section 5.1.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents Amortized Langevin Dynamics (ALD), a scalable MCMC algorithm that can be used to sample from high dimensional deep latent variable models. 

Unlike competing MCMC algorithms that need to rely on per-datapoint iterations, ALD can sample from the correct posterior distribution by performing updates to an encoder. This gives rise to Langevin Autoencoders, a family of deep LVMs that relies on ALD. 

Experiments on a number of benchmark datasets show that Langevin Autoencoders outperform competing models while maintaining scalability. 

 

 


strengths_and_weaknesses: STRENGTHS 

Most applications of deep LVMs use variational methods to approximate the intractable posterior distribution.
Variational methods are however inherently biased, since they try to approximate a complex posterior distribution with distributions from a typically simple variational family (e.g. Gaussian). 

This is not an issue in sample-based MCMC methods, but they do not scale for high dimensional deep LVMs trained on large datasets, since they require expensive per-datapoint iterations. 

This paper provides a novel solution to this, that relies on applying amortization ideas to MCMC methods. 
The idea of obtaining samples from the posterior distribution by updating an encoder that data points to latent variables is very interesting to me, as it preserves the scalability of variational methods while not making any strong assumption on the approximate posterior distribution.
Also, theoretical and empirical results show that ALD matches the correct stationary posterior distribution even for the complex posterior distributions that are typical of DLVMs.

I believe this paper can have an impact in the neurips community, since it can inspire a new research direction on enoder-based MCMC methods that use amortization and avoid datapoint-wise iterations.


WEAKNESSES 

There are some experimental improvements that I believe could further increase the impact of this paper.

1. You state that the goal of the experiments is to show that ALD works, and not that of achieving SOTA results, which is ok. However, for this type of paper that presents a method that can be readily applied to any deep LVM, trying it out on a more SOTA architecture would have made the results much more compelling. The fact that you have not done it makes me wonder if there are some complications to applying it to more complex architectures.

2. How can one monitor the convergence of the MCMC chain? Which convergence diagnostics have you used? 

3. Did you ever get in situations where the MH acceptance rate was too low/high? 

4. What are the training times of the different models considered in the experiments?


 Minor comments:

* typo in title: variabel -> variable 

* line 29. The issue is not that the distribution is tractable as it is implied in the sentence. The issue is that the posterior for these types of models is complex, so the tractable distributions we use are not a good approximation. 

* line 41: "in test time" -> "at test time" 



questions: See comments in the "weaknesses" section.
limitations: Yes
review_title: null

Meta-review: metareview: The problem addressed in this paper is the one of inference in deep generative latent variable model. The paper proposes a novel approach with an amortized approximation to the joint distribution of data and latent. 

All three reviewers liked the paper with one reviewer being a bit concerned about the soundness of the MCMC convergence proof.  In this meta-reviewers' opinion the algorithm (Alg 2) is sound although there might be subtleties here because it mixes posterior parameter updates with generative model parameter updates and latent samples.

The paper is original, clear and numerical evaluations sufficient so acceptance is recommended.
confidence: Certain
Decision: Accept