Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces amortized Langevin dynamics for latent variables and extends it to unconditional distributions. A Langevin autoencoder is further developed by using amortized Langevin dynamics for prior and posterior sampling. The proposed method has been tested on synthetic distributions and image generation.
main_review: It is not clear to me how to get the function g in the inference network. Is the function g obtained through some preprocessing? How does the quality of it affect the posterior approximation result? How do the authors get it in the experiment section? It will be better to provide more explanation and details about it, since it seems to play an important role in approximating the posterior as shown in Theorem 1.

The second condition for ALD to converge to the true distribution does not seem mild. Since the dataset size is typically very large, it is impractical to have such high dimension for the linear layer. Therefor the claim seems a bit misleading that “ALD has the target posterior as a stationary distribution with a mild assumption”. 

How does the storage cost of the proposed method compared to LD without amortization and AVI? By looking at Algorithm 1, the storage cost of the proposed method seems much higher, since it has to save Z^1 to Z^n and each Z^i contains several samples.

For the experiments, it will be better to add the results of SGALD in Figures 1 and 4 to check the posterior estimation when using minibatch of data. Since SGALD is the one being used in LAE in practice. 

On the synthetic distributions, the dimension of the linear layer is much larger than the dataset size (128 vs 3). However in practice the dataset size will be much larger than the dimension of the linear layer (as what the authors did for the experiment on image generation). It will be much convincing to simulate this scenario on synthetic data and verify that with small dimension ALD/SGALD can still approximate posterior well. It will be very helpful to further show how the posterior estimation changes with respect to the dimension of the linear layer.


summary_of_the_review: In summary, I think the idea is reasonable and developing a cheap MCMC-based autoencoder will be of interest to the community. However, I have some concerns with respect to the methodology and the experiments mentioned above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an amortized Langevin dynamics for sampling from a posterior distribution of a top-down generative model or from an energy-based model. The key is to recruit a sampler function that generates many samples directly in parallel, and run a single Langevin dynamics on the parameters of this sampler function. The method is illustrated by image generation. 
main_review: Strengths: 

(1) The idea of the paper is novel and interesting. Theoretical analysis is provided. 

(2) The proposed method can be applied to sampling both the posterior distribution and the energy-based model. 

(3) The proposed method is different from variational approximation. 

Weaknesses: 

(1) One thing I am concerned with is, if all the randomness is accounted for by the Langevin sampling of the parameters, will the generated samples be correlated? 

(2) Can you always find a good sampler function to approximate complex target distribution? 
summary_of_the_review: The paper proposes a new idea on sampling from unnormalized densities. It can be useful for learning deep generative models.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use amortized MCMC to learn latent space energy-based model (EBM). Specifically, instead of running Langevin dynamics on (latent) data space, such sampling is performed on the parameter space of inference model and is shown to converge to the true posterior under several mild conditions. The experiments demonstrate the generation quality of the proposed model.  
main_review: (+) The paper considers an important and challenging problem. The idea discussed is quite interesting and somewhat new. 

However, there are some downsides of the current submission: 

(-) The presentation of the paper needs to be improved. Quite a few places are unclear which need to be better motivated or to be further described. For example, what does $u$ represents in Eqn. 14? Any motivations/insights on why consider Langevin process on the parameter space (the parameter space usually has much higher dimensions compared to the latent codes itself)? How to choose $g(x)$ as in theorem 1? If only the last layer is trainable, does the inference model parametrized in this way has very limited expressive power? Also, in algorithm 2, all the update is on the parameter space including generator as well as encoder. The paper should also need to be well motivated on why performing Langevin on their parameter spaces (as in Eqn 10, 11) as well. 

(-) The experiments need to be improved. The sample quality as well as score is relatively weak. Why not use reported FID of the baselines in table 1? For example, the LEBM achieves lower FID on cifar10 on their paper, why not choose their model structure and retrain the proposed model for the comparison?  
summary_of_the_review: The paper study an important problem, but the current presentation makes it hard to follow. Also the extra experiments and analysis need to be added to motivate and back-up their claims.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Typically, approximating the (analytically intractable) posterior of deep latent variable models can be done in two ways, variational inference methods learn an (analytically tractable) distribution to approximate the true posterior. For large datasets where each dataitem has its own posterior distribution, the parameters of the optimal approximate distribution may be predicted as a function of the dataitem, the comon practice of “amortization”. Thus the approximate posterior is quick to infer at test time however it is also limited by the flexibility of the familty of distributino used for the analytic approximation.

Meanwhile, Markov Chain Monte-Carlo methods generate a collection of samples to approximate true posterior. As each dataitem in a large dataset has a unique posterior, each would require its own Markov chain of samples which is a more accurate approximation, however is time consuming.

The author propose a mixture of these approaches. At a very high level, the authors propose to combine a single “communal” Markov chain, $\Phi$, together with a function that maps dataitems into a weight vector, $g(x)$, that linearly maps the communal markov chain onto a markov chain that approximates the posterior for the given dataitem p(z|x)~$\Phi g(x)$. Replace one chain per item (traditional MCMC) with a communal chain and one linear mapping per dataitem, where the mapping is amortized.

The authors describe application of this new trick to latent space of an autoencoder latent variable model and the sampling from an unconditioned density such as an energy based model. The MCMC algorithm of choice is the well known Langevin Dynamics solved by the Euler-Maruyama method. The authors describe training in minibatches simulataneously regenerating the MCMC chains which reduces to standard SGD with noise. Toy experiments show proof of concept and image generation tasks show positive results as measured by Frechet Inception distance.

main_review: I enjoyed the overall paper. I felt the presentation was perhaps a little unecessarily complex. I have a few technical concerns

# Good points
- I felt the writing and presentation of existing literature was generally very good, clear and concise.

- the theoretical concept "one chain for all datapoints" (and linear projections) also feels like a good idea to me. The theorem requires that all samples for all posteriors $z^{(i)}$ must be varying linear projections ($g(x)$) of a single communal chain $\Phi$. Or alternatively, all data item specific posterior distributions are linear projections of a single higher dimensional distribution. This sounds really cool!

- I enjoyed the theorem and the discussion of the rank of the G matrix.


# Concerns

- In algorithm 2, the generative model weights, $\Theta$, are also sampled using the Euler-Maruyama method, however this does not correspond to the theorem 2.1 which requires that the generative model $\Theta$ is fixed (otherwise there is no true posterior to approximate). (Upon further inspection, it looks like the given loss function (14) is the evidence lower bound of the data and hence sampling model weights is similar to Bayesian neural network sampling of weights with the ELBO and not model evidence).

- P4, line 3: Test time sampling: why not pregenerate a markov chain during/after training that can be reused again and again at test time? Surely the whole point of amortization is the “one-chain fits-all” hypothesis? If the authors propose resampling at test time then the proposed method is not strictly amortized and there is no actual speedup?

- In the experiments, would it be possible to also report estimated marginal dataset likelihood $\log p(x)$? There are many generative modelling works that use this as a metric (e.g. PixelCNN).

- I am surprised by the *extremely* high variance of of the Frechet Inception distance among baselines, I struggle to believe that all the standard methods are *that* bad, presumably either the error bars are not accurate or the implementation of baselines is not ideal. For me, unfortunately, this make the paper hard to accept. (I realise this may contradict other venues/reviewers that emphasise empirical results).

- in the experiments, I think it is valuable to know why LD and LD+EMB are worse than LAE? Surely LD + EMB is the gold standard (one MCMC chain per dataitem) that the LAE strives to achieve, while this may be computationally expensive, I believe that such a baseline should at least be implemented to outperform (or match) LAE, (i.e. agreeing with the theory) this could also show the MCMC amortization gap that this new amortization method introduces (again agreeing with the theory). Currently these are unanswered question that feels rather central to the contribution of the paper.

- regarding Theorem 2.1, and the implication that all posteriors are linear projections of a bigger common distribution, this feels very sensitive to the big distribution dimension, currently this is not properly investigated. If the dimension is large, then the communal MCMC chain is no cheaper than parallelized low dim chains across dataitems, if the dimension is too small, this imposes too much similarity across all the posteriors and the method suffers, this trade off is not clear.

# Minor Points

- there appears almost no mention of Bayesian neural networks. MCMC sampling of network weights has been extensively studied, the proposed method is also MCMC sampling network weights while just using a different  distribution to sample from. Presumably, any Bayesian neural network approach for the encoder may be used? 

- If I understand correctly, training is using the Eurler Maruyama method and so the algorithm simplifies to SGD with Gaussian noise, presumably this makes convergence much slower than a VAE or VAE-flow which may use RMSprop or Adam or any method with momentum across minibatches?

- (very minor) regarding the disadvantages of VAEs, the ELBO training objective is the KL divergence of true and approximate posterior, i.e. it makes a inference network Gaussian that looks like the posterior but it also, *theoretically*, it makes a posterior that looks Gaussian,  hence the claims that VAEs are existentially flawed feel slightly overstated to me, though I realise this is not a common opinion and in practice this is indeed very often not the case.

- the presentation of the proposed method seems a little overly complex or difficult to follow, as I have described, I personally found it much easier to think in terms of a communal chain and linear projections, I am sure there must be many other intepretations that are simpler and easier to follow than the paper in its current form.


I enjoyed the paper and I am less well read in the area hence absolute novelty I cannot confidently judge (I wouldn't be surprised if this idea is well established in the more pure "non-deep" statistics community) though I do believe the idea has merit and is worth accepting. However the rather chaotic numerical results are very disturbing.

summary_of_the_review: I like the paper, I feel that the writing could be simplified somewhat, the method treats latent posteriors over data points as unique linear projections of a global distribution over network final layer weights.

- to me, the proposed method is an instance Bayesian neural networks with a different distribution (the ELBO) over weights (eg https://arxiv.org/abs/1902.02476, http://proceedings.mlr.press/v48/gal16.pdf)

- the experimental results have very high variance amongst baselines, this is concerning, particularly, the "truth" that the proposed method aims to approximate performs *worse* that the approximation.

- treating all dataitem posteriors as a linear projections of a single communal distribution seems risky, further comment would be appreciated.
review_title: null

Meta-review: This paper proposes an amortization strategy for MC sampling from a single chain rather than per-datapoint chains, and uses this strategy to define a new Bayesian autoencoder based on Langevin dynamics.

The reviewers find the line of thought very promising, and a potentially interesting addition to the latent variable literature, while also raising some concerns. The dimension of the single chain must match the dataset size, which limits the computational benefits coming from amortization, and in fact this restriction seems hard, as empirical results (added in the discussion period) are qualitatively worse in the `d<n` case. This could be emphasized much more strongly in the current version, and seems worth deeper investigation. In the discussion, the authors agreed that in the case when the feature matrix G is the identity matrix then there can be no amortization improvement, but for other choices of (fixed) features, amortization *can* yield improvements; this is quite unclear. In addition, in response to the reviewers' observation, the authors improved in the discussion period the implementation of the EBM baseline, leading to much less clear cut differences on metrics. To improve the work further, the authors should clarify the source of amortization improvement, and discuss more the relationship to Bayesian Neural Networks (perhaps by evaluating against Bayesian / hyper-net / hyper-GAN generative models.)
Decision: Reject