Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors proposed a method for amortizing latent variable sampling that is applicable to a variety of problems. The demonstrated examples show that generative models using their proposed method do better than ones of existing benchmark models. The paper is relatively easy to follow.

Some comments follow.

1. The notation for the $z^{(i)}$ in the description of Algorithm 1 seems to be inconsistent (need to have bold $z^{(i)}$?). 

2. A key idea in this paper is to find parameters of a parametrized mapping between data $x$ and latent variables $z$ using Langevin-like steps. The updates to the parameters and to the latent variables are done alternately. After convergence of the parameters, you can map the observed data into latent variables using the learned mapping. 

3. At this point, I don't understand or it is unclear to me how this helps to draw posterior samples of the latent variables themselves. Perhaps after learning an inference model on one data set you can use same parameters $\phi$ on another data set, initialize a MCMC in the latent space at the output of $f_{z|x}$, then you don't need to run the chain long as the learned function allows to achieve a "warm start" of the MCMC run?

4. What if you don't use noisy Langevin training for determining $\phi$ of the amortization model as you have, and just perform standard optimization on $\phi$? What is it about Gaussian noise in training the amortization model that makes the learned parameters better?

I think this is an interesting direction to explore, particularly if you can somehow make a connection with an exact method for sampling the latent variables $z$ (as you mention in the conclusion). It isn't clear how one would go about this. 

I would certainly be willing revise my review if the above points are satisfactorily clarified.
review_title: Interesting paper, proposed method appears to work well

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the author presented an advanced autoencoder framework LAE. Instead of element-wise MCMC, LAE collected samples from the posterior using the amortized Langevin dynamics of a potential energy distribution. In CLAE, an extended version of LAE, the author used an intractable energy function as the prior, and collected samples using its Langevin function. The author claims that LAE and CLAE are more efficient in large scale data and have better performance compared with traditional autoencoders and variational autoencoders.

[Strengths]
1. The proposed model replaces the posterior in VAE with a potential energy distribution, enabling fast sampling with its Langevin function.
2. The model is flexible with intractable prior distributions with unnormalized energy function, which can enhance the approximation power of the model.
3. The extended generative model CLAE with intractable prior indeed achieves has higher performance.

[Critiques and weaknesses]
1. Figures are poorly organized in the manuscript and captions in some of the figures are missing, making the figures difficult to understand without reading the context.
2. Although the author claimed that LAE is able to collect samples efficiently from the posterior, the runtime comparison in large scale data was not shown in the result section.

[Typos]
1. In the caption of figure 1:  f_{x|z} should be f_{z|x}
2. In the last paragraph of section 3, the first ‘figure 3’ should be ‘figure 2’

review_title: an incremental paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: # Summarize what the paper claims to contribute
The papers introduce an amortisation for inference by Langevin dynamics (LD). Rather than making each particle track the posterior for a given data point as in normal LD, this new method couples the posterior samples of multiple data points by a dynamic recognition model; the parameters of the model are updated following the Langevin dynamics for a collection of data points. Authors also extend its use to two related models of data: variational autoencoder (LAE) with normalised and unnormalised prior p(z) (CLAE), producing increasingly better fits to data distributions.

# Strong points:

+ The idea to amortise inference for a collection of posterior distributions is not new, but I have not seen before its extension to posteriors induced by sampling dynamics, so this paper makes an interesting proposal. In particular, introducing dynamics at the recognition parameter level is very interesting. 

+ The authors did not stop at applying to the common Gaussian generative models as in VAE, but extended it to more complex energy-based prior distributions.

+ The *method itself* is introduced clearly with well-written descriptions. The toy experimental results are helpful and demonstrate the power of the amortised Langevin.

# Weak points:

- My main concern is the experiment which is somewhat lacking on both the design and the quality of the results (detailed below)

- The example for dequantization inference is confusing, and the problem might be technically ill-defined.

- The description of CLAE is ok, but the authors try to link it to adversarial training which is a very different training objective. I also do not find the discussions clear enough to make it the contrast worthwhile

# Recommendation: 
I am slightly tending to reject as it stands because of the experiments and some unclear discussions, but I am happy to increase the score if the authors provide a better argument for their design and clarify on their discussions.

# Comments on experiments.
- Evaluations based on reconstruction error is always prone to the trivial solution of learning an identity mapping. Could the authors try to perform denoising instead? 

- VAEs with continuous observations, especially when trained on MNIST, is known to be hard. How about comparing it to a binary/Bernoulli likelihood evaluated on binarised MNIST? If it outperforms VAE, then this provides much stronger validation. I do not see from the 10-13 that the samples from LAE is much better than traditional VAE. 

- The comparison between CLAE and other models is unfair: CLAE has an energy-based prior which may be more flexible (is this the case? Can the authors clarify on the choice of prior energy function? Sorry if I have missed the description.)

- There are only FID and for CIFAR-10 and CelebA, but not for MNIST or SVHN. Can the authors report this figure using features extracted by a relevant neural network, e.g. one trained on MNIST classification for FID on MNIST?

- The learning curve in Figure 5 is the unnormalised likelihood: it's unclear how its stability or convergence implies about learning speed of the normalised likelihood objective. 

- Also, how does CLAE look on Figure 5? 

- A critical question is whether the leaned dynamics is indeed more efficient than normal Langevin. The authors claim this but did not show results on the comparison. The effective sample in Table 1 provides some clue, but a convergence plot would be much stronger to support this claim. 

# Questions:

? I find that updating all the recognition model parameters for each new data point a bit counter-intuitive. What if the initial samples for a set of x are very far away from the true distribution? The authors compare this method with the "amortised" MCMC in which the initial proposal is drawn from a recognition model, could this be combined with the dynamic-parameter recognition model to yield better results. Basically, if all parameters are dynamic, I do not see how the initial proposals can adapt to the generative model and characteristics of the data distribution. 

? Figure 4: are the samples from the conditional or unconditional Langevin? 

? What's the space of fixed random positions u? Also, could authors clarify the dimensionality of variables of z and u in Table 4?

? I haven't been able to understand this claim:
"In VAEs, noise is used to sample from the variational distribution in the calculation of potential U, i.e., in forward calculation. However, in LAEs, noise is used for calculating gradient ∇φU, i.e., in backward calculation."
The reparametrised samples in VAE are indeed used for the backward calculation. The forward pass simply evaluates the objective and retains dependence on recognition parameters φ.

? Figure 7: The After dequantization figure is better plotted after passing through a sigmoid? the huge difference in support range makes it hard to compare. 

? I am very puzzled with the content in appendix C and would like to authors to help with understanding. In particular, the sentence preceding (25) doesn't seem logical. I see that any \hat{x} from (25) is mapped to the x, meaning that "the likelihood is a constant", which is OK. However, the first term on the RHS of (26) is, in fact, a (log) conditional distribution of \hat{x} given x, but there is no distribution over the data x itself. Does this mean that the model is just a conditional distribution, and does not learn the data distribution p(x) at all? Also, what is the distribution for this first term? A delta or Gaussian?

? Could the authors clarify the following sentence in appendix D:
"In other words, the latent variable is identical to the observation (i.e., p (x | z) = 1_{x=z}) in GANs"
I do not see the connection of this method to GAN, I'm happy to start again by better understanding this part. 
More generally, I do not believe making the comparison is worthwhile if at all correct, because the different training objectives differ a lot. Also, the solution of GAN is obtained at an equilibrium established by the two players, but the authors do not show such results for this new method. Could something similar be established, i.e. the optimal recognition and data model is established at the minimax solution? 

# Detailed comments and suggestions (these points are here to help, and not necessarily part of your decision assessment)

The discussion on EBM in the appendix seems more relevant to the GAN discussion. If space permits, it should be moved in to the main text. Also, CLAE applies to a model with energy-based prior. How hard is it to extend to a fully latent variable EBM? 

Some typos:
* Eqns (12) and (13), commas (,) before \theta should be semicolons (;)
* Second lines below (24) "Altough"-> Although
* Line above (28) "rewrited"-> rewritten
*  Third line below (28) "enough continous" -> continuous enough
* Third line from bottom of page 15. "caan" -> can

===== update =====

I am very grateful for the patient and detailed response. Due to limited time, I wasn't able to quickly follow up on the discussion.  I think the current quality of the paper is improved, so I increase the score slightly. However, I still struggle to follow some of the statements even after reading the response, it could be my comprehension or something to do with style/writing.

review_title: An interesting idea applied to several generative models, but experiments are not rigorous enough.

Meta-review: The paper had three borderline reviews. While the idea of posterior sampling of a neural network is potentially useful and Langevin dynamics are a way to attempt to address that, the reviewers did not appear convinced by the experiments and what the MCMC sampling was doing wasn't really front and center there.
Decision: Reject