Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper is about using denoising autoencoders to improve performance in GANs. In particular, the features as determined by the discriminator, of images generated by the generator, are fed into a denoising AE and we try to have these be reconstructed well. I think it's an interesting idea to use this "extra information" -- namely the feature representations learned by the discriminator. It seems very much in the spirit of ICLR! My main concern, though, is that I'm not wholly convinced on the nature of the improvement. This method achieves higher inception scores than other methods in some cases, but I have a hard time interpreting these scores and thus a hard time getting excited by the results. In particular, the authors have not convinced me that the benefits outweigh the required additional sophistication both conceptually and implementation-wise (speaking of which, will code be released?). One thing I'd be curious to know is, how hard is it to get this thing to actually work? 

Also, I view GANs as a means to an end -- while I'm not particularly excited about generating realistic images (especially in 32x32), I'm very excited about the future potential of GAN-based systems. So it would have been nice to see these improvements in inception score translate into improvements in a more useful task. But this criticism could probably apply to many GAN papers and so perhaps isn't fair here. 

I do think the idea of exploiting "extra information" (like discriminator features) is interesting both inside and outside the context of this paper. 
review_title: GANtastic paper

final_score: rating: 7: Good paper, accept
review_content: The authors present a way to complement the Gerative Adversarial Network traning procedure with an additional term based on denoising autoencoders. The use of denoising autoencoders is motivated by the observation that they implicitly capture the distribution of the data they were trained on. While sampling methods based denoising autoencoders alone don't amount to very interesting generative models (at least no-one could demonstrate otherwise), this paper shows that it can be combined successfully with generative adversarial networks.

My overall assessment of this paper is that it is well written, well reasoned, and presents a good idea motivated from first principles. I feel that the idea presented here is not revolutionary or a very radical departure from what has been done before, I would have liked a slightly more structured experiments section which focusses on and provides insights into the relative merits of different choices one could make (see pre-review questions for details), rather than focussing just on demonstrating that a chosen variant works.

In addition to this general review, I have already posted specific questions and criticism in the pre-review questions - thanks for the authors' responses. Based on those responses the area I am most uncomfortable about is whether the (Alain & Bengio, 2014) intuition about the denoising autoencoders is valid if it all happens in a nonlinear featurespace. If the denoiser function's behaviour ends up depending on the Jacobian of the nonlinear transformation Phi, another question is whether this dependence is exploitable by the optimization scheme.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This paper is well written, and well presented. This method is using denoise autoencoder to learn an implicit probability distribution helps reduce training difficulty, which is neat. In my view, joint training with an auto-encoder is providing extra auxiliary gradient information to improve generator. Providing auxiliary information may be a methodology to improve GAN.  
 
Extra comment:
Please add more discussion with EBGAN in next version. 

review_title: null

final_score: rating: null
review_content: Hi. Thank you for your paper.

1. I'd like to make sure I understand the role of the denoising autoencoder (DAE). Is it correct that the DAE operates entirely in feature space? That is, it takes some features, corrupts them, and then tries to reconstruct them, without ever seeing any raw images? 

2. You mentioned using low-resolution images partly for computational reasons. I'm wondering if you tried just taking a small piece of a big image, instead of downsampling? That would be interesting to me because it would test the ability of your GAN to reproduce the small-scale textures of real images rather than the more "zoomed out" low-res images. 

3. The paper mentions Figure 5.3. This this supposed to say Figure 3?

4. Just to make sure I understand equation (4). Here is my take: The objective function for the generator G is a combination of (i) its ability to fool the discriminator, and (ii) its ability to produce images whose features (as computed by the discriminator!) are well-reconstructed by the DAE. Is that right? 

5. My impression is that the DNE and GAN are trained simultaneously. So the *feature space* on which the DNE operates is constantly shifting as the *discriminator* gets trained. Is that right? Does the generator also have a feature space that could be put to good use? 

6. Do you consider it a major or minor (or trivial) limitation of your work that we can no longer treat the discriminator as a black box?

7. How did you pick which layer of the discriminator to use as your feature space for the DNE? Did you try different layers and compare?

8. This is a comment rather than a question. In a few instances you refer to things as "well known". I did not know these things but was able to enjoy your paper nonetheless. I propose rewording those sentences so that your paper is more welcoming to readers from other fields.
review_title: Some questions

final_score: rating: null
review_content: It is the second interesting work which introduce autoencoder in the discriminator. What is the motivation of using DAE but not AE directly? 
review_title: Pre-review questions

final_score: rating: null
review_content: note 1: using the common notation of P for real and Q for synthetic distribution.
note 2: I don’t really have pre-review questions as such, so this is more like a mini-review in question form

- When designing the loss function, does the discriminator D and the feature space Psi in which the denoising occurs have to be related?
    - Could Psi simply be the identity mapping?
    - Could Psi be a fixed mapping, such as the featurespace of VGG network?
    - Os Psi only trained as part of the discriminator, or also as part of denoising?
    - Do the authors believe that this form of transfer from the discriminator to the denoiser is the essential idea for this to work?
    - Given this is a major part of the paper, could the authors include any control experiments (perhaps in the appendix) comparing these different options?
- How does the method deal with spurious modes/out-of-sample behaviour observed by Alain & Bengio (2014)? When Q and P are very different, Eq (5) only trains the denoiser well around high-P regions.
- Could the authors please elaborate on the validity of the intuition for using the first term of Eqn. (4) in the presence of the nonlinearity Psi? If Psi were the identity, The first term in Eqn. (4) could be seen as an estimate of the energy or negative log probability of the real data distribution up to a constant. However, it is unclear from the text if this intuition remains valid once the data is nonlinearly transformed. (Would the transformation introduce an additional Jacobian term in the probabilities, for example?)
- If D were optimal, it would roughly equal P(x)/(P(x) + Q(x)). Using this formula, with the assumption that the first term approximates something like the cross-entropy -E_Q log P(x), one could express Eqn. (4) in terms of only P and Q. With this in mind:
    - Do the authors agree that the higher \lambda_denoise is compared to \lambda_adv, the less the entropy of Q is taken into account, and this eventually as \lambda_denoise increases, the generator Q would collapse to a delta around the mode of P?
    - Given that undershooting entropy is already a failure mode of GANs, how could one fix this? Would it make sense to also train a denoiser on Q?
- Finally a comment on feature matching in Eqn. (6). A mistake that is often made in modern papers is naively performing stochastic gradient descent on Eqn. (6) (or more generally, maximum mean discrepancy) using biased estimates. To my knowledge, (Dziugaite et al, 2015) is the only modern deep-learning-type paper the correct biased estimator for training a model (Eqn. (8)) Contrast this with (Li, Swersky and Zemel, 2015) Eqns. (2-3). In this light, another relative advantage of the denoising approach is that the naive minibatch-estimate is unbiased as the expectation w.r.t Q is taken outside of the squared error.
review_title: Pre-review questions

Meta-review: The idea of using a denoising autoencoder on features of the discriminator is sensible, and explored and described well here. The qualitative results are pretty good, but it would be nice to try some of the more recent likelihood-based methods for quantitative evaluation, as the inception score is not very satisfying. Also it would be interesting to see if this additional term helps in scaling up to larger images.
Decision: Accept (Poster)