Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: I enjoyed reading this paper. One point of confusion I had was about the way in which the step-number dependence was incorporated into p^(t)( z^(t) | Z^(t-1) ). In theory, you could learn a different conditional distribution for each t. In the first paragraph in Section 4, it seems like you are actually using the same function for every value of t though. Then, at the end of that paragraph, you note that you have a hard-coded t dependence in the variance, but you suggest that this is only for reasons of stability and ease of training.

Could you clarify what the t dependence of p^(t)( z^(t) | Z^(t-1) ) is in your experiments? If you believe that p^(t)( z^(t) | Z^(t-1) ) should be the same distribution at every time step, could you talk a little bit about why?

Thanks!
review_title: time dependence of infusion chain

final_score: rating: null
review_content: The reviewer apologizes for not entering a pre-review question in time. I was under the impression that this is an optional step for clearing up initially discovered flaws. Please see my review for my main questions/points of criticism. I will gladly respond to any answers during the review period.
review_title: -

final_score: rating: null
review_content: The authors state the goal of training as “samples from [the training set] are likely of being generated under the model sampling chain.” Unfortunately none of the experiments performed is appropriate to verify if this goal has been achieved. Have you tried using importance sampling to evaluate log-likelihoods?
review_title: Using IS to evaluate log-likelihood

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper trains a generative model which transforms noise into model samples by a gradual denoising process. It is similar to a generative model based on diffusion. Unlike the diffusion approach:
- It uses only a small number of denoising steps, and is thus far more computationally efficient.
- Rather than consisting of a reverse trajectory, the conditional chain for the approximate posterior jumps to q(z(0) | x), and then runs in the same direction as the generative model. This allows the inference chain to behave like a perturbation around the generative model, that pulls it towards the data. (This also seems somewhat related to ladder networks.)
- There is no tractable variational bound on the log likelihood.

I liked the idea, and found the visual sample quality given a short chain impressive. The inpainting results were particularly nice, since one shot inpainting is not possible under most generative modeling frameworks. It would be much more convincing to have a log likelihood comparison that doesn't depend on Parzen likelihoods.

Detailed comments follow:

Sec. 2:
"theta(0) the" -> "theta(0) be the"
"theta(t) the" -> "theta(t) be the"
"what we will be using" -> "which we will be doing"
I like that you infer q(z^0|x), and then run inference in the same order as the generative chain. This reminds me slightly of ladder networks.
"q*. Having learned" -> "q*. [paragraph break] Having learned"
Sec 3.3:
"learn to inverse" -> "learn to reverse"
Sec. 4:
"For each experiments" -> "For each experiment"
How sensitive are your results to infusion rate?
Sec. 5: "appears to provide more accurate models" I don't think you showed this -- there's no direct comparison to the Sohl-Dickstein paper.
Fig 4. -- neat!

review_title: interesting idea

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper introduces a heuristic approach for training a deep directed generative model, where similar to the transition operator of a Markov chain each layer samples from the same conditional distribution. Similar to optimizing a variational lower bound, the approach is to approximate the gradient by replacing the posterior over latents with an alternative distribution. However, the approximating distribution is not updated to improve the lower bound but heuristically constructed in each step. A further difference to variational optimization is that the conditional distributions are optimized greedily rather than following the gradient of the joint log-likelihood.

Review:
The proposed approach is interesting and to me seems worth exploring more. Given that there are approaches for training the same class of models which are 1) theoretically more sound, 2) of similar computational complexity, and 3) work well in practice (e.g. Rezende & Mohamed, 2015), I am nevertheless not sure of its potential to generate impact. My bigger concern, however, is that the empirical evaluation is still quite limited.

I appreciate the authors included proper estimates of the log-likelihood. This will enable and encourage future comparisons with this method on continuous MNIST. However, the authors should point out that the numbers taken from Wu et al. (2016) are not representative of the performance of a VAE. (From the paper: “Therefore, the log-likelihood values we report should not be compared directly against networks which have a more flexible observation model.” “Such observation models can easily achieve much higher log-likelihood scores, […].”)

Comparisons with inpainting results using other methods would have been nice. How practical is the proposed approach compared to other approaches? Similar to the diffusion approach by Sohl-Dickstein et al. (2015), the proposed approach seems to be both efficient and effective for inpainting. Not making this a bigger point and performing the proper evaluations seems like a missed opportunity.

Minor:
– I am missing citations for “ordered visible dimension sampling”
– Typos and frequent incorrect use of \citet and \citep
review_title: Interesting idea with lacking theoretical motivation and limited empirical evaluation

final_score: rating: 7: Good paper, accept
review_content: The paper presents a method for training a generative model via an iterative denoising procedure. The denoising process is initialized with a random sample from a crude approximation to the data distribution and produces a high quality sample via multiple denoising steps. Training is performed by setting-up a Markov chain that slowly blends propositions from the current denoising model with a real example from the data distribution; using this chain the current denoising model is updated towards reproducing the changed, "better", samples from the blending process.

This is a clearly written paper that considers an interesting approach for training generative models. I was intrigued by the simplicity of the presented approach and really enjoyed reading the paper.
The proposed method is novel although it has clear ties to other recent work aiming to use denoising models for sampling from distributions such as the work by Sohl-Dickstein and the recent work on using DAEs as generative models.
I think this general direction of research is important. The proposed procedure takes inspiration from the perspective of generating samples by minimizing an energy function via transitions along a Markov chain and, if successful, it can potentially sidestep many problems of current procedures for training directed generative models such as:
- convergence and mode coverage problems as in generative adversarial networks
- problems with modeling multi-modal distributions which can arise when a too restrictive approximate inference model is paired with a powerful generative model

That being said, another method that seems promising for addressing these issues that also has some superficially similarity to the presented work is the idea of combining Hamiltonian Monte Carlo inference with variational inference as in [1]. As such I am not entirely convinced that the method presented here will be able to perform better than the mentioned paper; although it might be simpler to train. Similarly, although I agree that using a MCMC chain to generate samples via a MC-EM like procedure is likely very costly I am not convinced such a procedure won't at least also work reasonably well for the simple MNIST example. In general a more direct comparison between different inference methods using an MCMC chain like procedure would be nice to have but I understand that this is perhaps out of the scope of this paper. One thing that I would have expected, however, is a direct comparison to the procedure from Sohl-Dickstein in terms of sampling steps and generation quality as it is so directly related.

Other major points (good and bad):
- Although in general the method is explained well some training details are missing. Most importantly it is never mentioned how alpha or omega are set (I am assuming omega is 0.01 as that is the increase mentioned in the experimental setup). It is also unclear how alpha affects the capabilities of the generator. While it intuitively seems reasonable to use a small alpha over many steps to ensure slow blending of the two distributions it is not clear how necessary this is or at what point the procedure would break (I assume alpha = 1 won't work as the generator then would have to magically denoise a sample from the relatively uninformative draw from p0 ?). The authors do mention in one of the figure captions that the denoising model does not produce good samples in only 1-2 steps but that might also be an artifact of training the model with small alpha (at least I see no a priori reason for this). More experiments should be carried out here.
- No infusion chains or generating chains are shown for any of the more complicated data distributions, this is unfortunate as I feel these would be interesting to look at.
- The paper does a good job at evaluating the model with respect to several different metrics. The bound on the log-likelihood is nice to have as well!
- Unfortunately the current approach does not come with any theoretical guarantees. It is unclear for what choices of alpha the procedure will work and whether there is some deeper connection to MCMC sampling or energy based models. In my eyes this does not subtract from the value of the paper but would perhaps be worth a short sentence in the conclusion.

Minor points:
- The second reference seems broken
- Figure 3 starts at 100 epochs and, as a result, contains little information. Perhaps it would be more useful to show the complete training procedure and put the x-axis on a log-scale ?
- The explanation regarding the convolutional networks you use makes no sense to me. You write that you use the same structure as in the "Improved GANs" paper which, unlike your model, generates samples from a fixed length random input. I thus suppose you don't really use a generator with 1 fully connected network followed by up-convolutions but rather have several stages of convolutions followed by a fully connected layer and then up-convolutions ?
- The choice of parametrizing the variance via a sigmoid output unit is somewhat unusual, was there a specific reason for this choice ?
- footnote 1 contains errors: "This allow to" -> "allows to",  "few informations" -> "little information". "This force the network" -> "forces"
- Page 1 error: etc...
- Page 4 error: "operator should to learn"

[1] Markov Chain Monte Carlo and Variational Inference: Bridging the Gap, Tim Salimans and Diedrik P. Kingma and Max Welling, ICML 2015


>>> Update <<<<
Copied here from my response below: 

I believe the response of the authors clarifies all open issues. I strongly believe the paper should be accepted to the conference. The only remaining issue I have with the paper is that, as the authors acknowledge the architecture of the generator is likely highly sub-optimal and might hamper the performance of the method in the evaluation. This however does not at all subtract from any of the main points of the paper.

I am thus keeping my score as a clear accept. I want to emphasize that I believe the paper should be published (just in case the review process results in some form of cut-off threshold that is high due to overall "inflated" review scores).

review_title: Clearly written paper pursuing an interesting idea. Some shortcomings with respect to the evaluation and comparison to prior work

Meta-review: Infusion training is a new, somewhat heuristic, procedure for training deep generative models. It's an interesting novel idea and a good paper, which has also been improved after the authors have been responsive to reviewer feedback.
Decision: Accept (Poster)