Title: Improving Generative Adversarial Networks with Denoising Feature Matching
Authors: David Warde-Farley
Yoshua Bengio
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning

Abstract: We propose an augmented training procedure for generative adversarial networks designed to address shortcomings of the original by directing the generator towards probable configurations of abstract discriminator features. We estimate and track the distribution of these features, as computed from data, with a denoising auto-encoder, and use it to propose high-level targets for the generator. We combine this new loss with the original and evaluate the hybrid criterion on the task of unsupervised image synthesis from datasets comprising a diverse set of visual categories, noting a qualitative and quantitative improvement in the ``objectness'' of the resulting samples.

TL;DR: Use a denoiser trained on discriminator features to train better generators.

Review discussion:
Hi. Thank you for your paper.
1. I'd like to make sure I understand the role of the denoising autoencoder (DAE). Is it correct that the DAE operates entirely in feature space?
That is, it takes some features, corrupts them, and then tries to reconstruct them, without ever seeing any raw images?
2. You mentioned using low-resolution images partly for computational reasons. I'm wondering if you tried just taking a small piece of a big image, instead of downsampling? That would be interesting to me because it would test the ability of your GAN to reproduce the small-scale textures of real images rather than the more "zoomed out" low-res images.
3. The paper mentions Figure 5.3. This this supposed to say Figure 3?
4. Just to make sure I understand equation (4). Here is my take: The objective function for the generator G is a combination of (i) its ability to fool the discriminator, and (ii) its ability to produce images whose features (as computed by the discriminator!) are well-reconstructed by the DAE. Is that right?
5. My impression is that the DNE and GAN are trained simultaneously.
So the *feature space* on which the DNE operates is constantly shifting as the *discriminator* gets trained. Is that right? Does the generator also have a feature space that could be put to good use?
6. Do you consider it a major or minor (or trivial) limitation of your work that we can no longer treat the discriminator as a black box?
7. How did you pick which layer of the discriminator to use as your feature space for the DNE? Did you try different layers and compare?
8. This is a comment rather than a question. In a few instances you refer to things as "well known". I did not know these things but was able to enjoy your paper nonetheless. I propose rewording those sentences so that your paper is more welcoming to readers from other fields.
Thanks for your careful reading.
1. Yes, entirely in feature space. The reason is partly computational (these convolutional feature hierarchies are being trained by the discriminator, and their reuse is straightforward) and partly synergistic (the discriminative information in features produced as a side effect of discriminator training is ripe for additional exploitation).
2. We did not as yet apply it to patches, although we are currently experimenting with 64x64 ImageNet models. My impression is that, suitably tuning the weighting coefficients, the "classic" adversarial loss term should be able to take care of that aspect.
3. This seems to have been a mysterious LaTeX problem that I will sort out in my next revision of the paper, thank you for catching it.
4.
Indeed. Or phrased in terms of optimality conditions: if r is a (non-parametric, infinite capacity, etc.) optimal denoiser for the data distribution and G reproduces it indistinguishably, then r applied to a sample from G is simply the identity map and the first term vanishes, and the usual theoretical GAN story applies to the remaining term.
5a.
Indeed, simultaneous training means that the denoiser is tracking a nonstationary input distribution. Historical (i.e. Polyak) averaging on the discriminator parameters (as proposed in Salimans et al 2016 simply as a measure to aid convergence/stability) can combat this effect. I experimented with historical averaging for this reason but did not observe a very convincing benefit.
b. The generator indeed has a feature space, though it is less obvious how to exploit it, since we do not have corresponding ground truth. One could conceive of performing an optimization on |G(z) - x|^2 for each example x in order to infer the generator features corresponding most closely to a configuration which reproduces x, but this is obviously a rather expensive affair.
Also, it's not entirely obvious that, for a given setting of G's parameters, a training example is even in the image of the function G, making the optimization somewhat hopeless and the utility of then using the results of this optimization rather unclear.
6. This is a very interesting question and has been an enjoyable one to think about. I think I would say that in the typical GAN formulation, while practitioners may sometimes *act* as though the discriminator is a black box, it really is not and never was. The only training signal that the generator receives is the gradient of a highly nonlinear loss defined by the discriminator, the characteristic twists and turns of which are a very complicated and non-obvious function of the architectural and training decisions made regarding the discriminator. Wider deep learning practice often treats LSTMs (for sequential data) and convolutional nets (for image classification) as black boxes that can be thrown at almost any task fitting a loose mold of sequence prediction/image recognition and used as a subcomponent of a larger system, but the generator's training really is nontrivially dependent on both the architecture and the training-time dynamics of the discriminator, so treating it as a black box is dangerous.
This is all the more the case in light of what theory exists about GANs, given the theoretical assumptions of infinite capacity/optimality that are far from satisfied in practice.
7. In all cases, we used the penultimate layers because they were presumably "most abstract", with such very high level representations being frequently used as "descriptors" in some of the e.g. vision papers we cite. Their limited spatial topology makes it easy and appropriate to use simple, relatively inexpensive, non-convolutional denoisers. We did experiment with both convolutional and non-convolutional denoisers which took as input lower, higher-resolution feature maps, and found that underfitting in the denoiser became much more of a problem, not to mention the additional time expense and complication in choosing relevant architectural details. I would not rule out achieving better results by utilizing higher resolution feature maps, although the architectural search may be quite painful.
8. Thank you for the stylistic feedback, I actually did not realize the rather high frequency of that phrase and will address them and try and provide additional citations if appropriate. If you have specific feedback on any phenomena that are indeed under-explained, I would welcome it.
note 1: using the common notation of P for real and Q for synthetic distribution.
note 2: I don’t really have pre-review questions as such, so this is more like a mini-review in question form
- When designing the loss function, does the discriminator D and the feature space Psi in which the denoising occurs have to be related?
- Could Psi simply be the identity mapping?
- Could Psi be a fixed mapping, such as the featurespace of VGG network?
- Os Psi only trained as part of the discriminator, or also as part of denoising?
- Do the authors believe that this form of transfer from the discriminator to the denoiser is the essential idea for this to work?
- Given this is a major part of the paper, could the authors include any control experiments (perhaps in the appendix) comparing these different options?
- How does the method deal with spurious modes/out-of-sample behaviour observed by Alain & Bengio (2014)?
When Q and P are very different, Eq (5) only trains the denoiser well around high-P regions.
- Could the authors please elaborate on the validity of the intuition for using the first term of Eqn. (4) in the presence of the nonlinearity Psi? If Psi were the identity, The first term in Eqn. (4) could be seen as an estimate of the energy or negative log probability of the real data distribution up to a constant.
However, it is unclear from the text if this intuition remains valid once the data is nonlinearly transformed. (Would the transformation introduce an additional Jacobian term in the probabilities, for example?)
- If D were optimal, it would roughly equal P(x)/(P(x) + Q(x)). Using this formula, with the assumption that the first term approximates something like the cross-entropy -E_Q log P(x), one could express Eqn. (4) in terms of only P and Q. With this in mind:
- Do the authors agree that the higher \lambda_denoise is compared to \lambda_adv, the less the entropy of Q is taken into account, and this eventually as \lambda_denoise increases, the generator Q would collapse to a delta around the mode of P?
- Given that undershooting entropy is already a failure mode of GANs, how could one fix this? Would it make sense to also train a denoiser on Q?
- Finally a comment on feature matching in Eqn. (6). A mistake that is often made in modern papers is naively performing stochastic gradient descent on Eqn. (6) (or more generally, maximum mean discrepancy) using biased estimates. To my knowledge, (Dziugaite et al, 2015) is the only modern deep-learning-type paper the correct biased estimator for training a model (Eqn. (8)) Contrast this with (Li, Swersky and Zemel, 2015) Eqns. (2-3). In this light, another relative advantage of the denoising approach is that the naive minibatch-estimate is unbiased as the expectation w.r.t Q is taken outside of the squared error.
Thank you for your careful reading of the paper. I've tried to respond to all of your points below.
> Could Psi simply be the identity mapping?
In principle yes, although you'd push a lot of complexity into the denoiser (requiring a reconstruction in visible space/potentially expensive decoder).
> Could Psi be a fixed mapping, such as the featurespace of VGG network?
Yes, at additional expense, and perhaps with a less synergistic effect on the D-G training dynamics; see the answer to the "essential idea" question below.
> Os Psi only trained as part of the discriminator, or also as part of denoising?
Only as part of the discriminator. I have pondered the latter but have not implemented it. It's not entirely obvious how to weight the discriminative and denoising objectives for training Psi in that case, but a small search could be performed.
> Do the authors believe that this form of transfer from the discriminator to the denoiser is the essential idea for this to work?
The idea of using a denoising operator to learn an implicit generative model (i.e. sampler) is itself unexplored in the literature (with somewhat good reason, without the addition of e.g. the GAN loss it fails quite badly, unless considering constrained conditional settings as the Sonderby et al work does in service of super-resolution).
As for whether the relationship between the discriminator and the denoiser is essential: besides being computationally convenient, we believe there's a "virtuous cycle" aspect to it. Whether Psi is part of the discriminator or separate from it, the denoiser indirectly influences the discriminator by training the generator on a different objective.
However, in the formulation presented, the denoiser is acting on precisely the representations that the discriminator is using to tell apart real from synthetic, providing an additional signal from which the generator can derive information used to “fool” the discriminator, adopting a more nuanced notion of distributional similarity than linear inseparability in feature space.
> Given this is a major part of the paper, could the authors include any control experiments (perhaps in the appendix) comparing these different options?
I'd be happy to try out the second variant on e.g. CIFAR-10 as it is a fairly straightforward modification to the loss/training procedure, and will do over the holidays. With the first variant, it's not obvious what architecture would make for a sufficiently fair comparison. As it will probably require a separate convolutional feature hierarchy in addition to a convolutional decoder, it will add both a lot of parameters/capacity and a lot of additional computation, making it difficult to unambiguously attribute the source of any difference. If you have additional thoughts on ways to isolate these effects they would be welcome.
> How does the method deal with spurious modes/out-of-sample behaviour observed by Alain & Bengio (2014)?
When Q and P are very different, Eq (5) only trains the denoiser well around high-P regions.
This is an interesting question. One possible explanation is that in such situations, the ordinary adversarial loss term provides the necessary guidance.
Another is that “separate” batch norm throughout the network forces the features at each layer for both samples and training examples into a small enough volume that you are never really traveling too far (in our experiments, we used post-batch-normalized activations at a high layer, but this was not absolutely crucial, perhaps because of “separate” batch norm in the lower layers). This would be an easy control experiment to run.
> Could the authors please elaborate on the validity of the intuition for using the first term of Eqn. (4) in the presence of the nonlinearity Psi?
This is another interesting question. We have viewed it through the lense of providing “hints” to the generator through the manipulation of discriminator features, which we accomplish by implicitly estimating the distribution of phi(x), because it is computationally efficient to do so and because such representations are known to preserve semantically meaningful attributes in trained supervised networks. The phi(x) we choose is not invertible, so it is not a valid change-of-variables in the way Real NVP would be, and the precise relationship between that density and the generator density is unclear, as the objective used to train these features is purely discriminative and not focused on preserving anything else.
It is well known and confirmed by recent work that GANs suffer from mode mass misallocation/dropping, and we make no claim that this method solves these issues (although it does appear to alleviate somewhat at least the concentration/collapsing issue). We do, however, believe it to be superior than the alternatives for allowing the generator to identify modes in complicated distributions (e. g. object categories) without the use of label information. This was our objective, and we believe that we have indeed made progress in this regard.
> Do the authors agree that the higher \lambda_denoise is compared to \lambda_adv, the less the entropy of Q is taken into account, and this eventually as \lambda_denoise increases, the generator Q would collapse to a delta around the mode of P?
Yes. Empirically, optimizing only with the denoiser converges quickly to a few noisy templates (noisy because the features space is insensitive to pixel-perfect realizations). We have layered this loss on top of the original GAN training procedure both because the GAN loss is able to correct, and the complementary sources of information appear (to the naked eye) to mutually regularize against collapse.
> Given that undershooting entropy is already a failure mode of GANs, how could one fix this? Would it make sense to also train a denoiser on Q?
Empirically speaking, with an observed lower tendency towards replicated samples relative to ordinary GANs (at least when trained without minibatch discrimination or pull-away terms that explicitly discourage such replication), we believe that the current method partly addresses it (somewhat unsatisfyingly) by combining two objective functions for G/Q which provide complementary sources of information. We view the main contribution as a method for better unsupervised mode discovery, rather than a standalone silver bullet. I don’t quite understand the question regarding training a denoiser on Q, or what that would accomplish.
> Finally a comment on feature matching in Eqn. (6). A mistake that is often made in modern papers is naively performing stochastic gradient descent on Eqn. (6) (or more generally, maximum mean discrepancy) using biased estimates. To my knowledge, (Dziugaite et al, 2015) is the only modern deep-learning-type paper the correct biased estimator for training a model (Eqn. (8)) Contrast this with (Li, Swersky and Zemel, 2015) Eqns. (2-3). In this light, another relative advantage of the denoising approach is that the naive minibatch-estimate is unbiased as the expectation w.r.t Q is taken outside of the squared error.
Thank you for the pointer to the Dziugaite work, we were unaware of this issue with these other applications of MMD.
Thanks for the detailed responses, please see my review I just posted.
On running more experiments: I leave it to your judgment whether you feel more experiments are justified.
I am still slightly uncomfortable about the effect of the nonlinearity Phi on the (Alain & Bengio, 2014) intuition, but I accept your response to this question. I think it would be nice to point this out explicitly in the paper, and/or to perhaps work out the exact optimal denoiser formula in the presence of nonlinearity (assuming invertability is probably fine, just to gain intuition).
It is the second interesting work which introduce autoencoder in the discriminator. What is the motivation of using DAE but not AE directly?
Thanks for your question. I assume you're referring to the EBGAN work, which hit the arXiv while we were preparing this manuscript and is a parallel submission at ICLR 2017.
We do address the similarities and differences in the "Related work" to EBGAN section of the manuscript. Essentially, EBGAN replaces the discriminator with an autoencoder trained with a hinge loss, which is trained to have low L2 reconstruction error on data and higher (up to a margin) reconstruction error on generator samples. This generalizes GANs in a direction towards other scalar-valued, yet still "implicitly learned" cost functions -- EBGAN tries to minimize the autoencoder's reconstruction error, rather than a classifier's log likelihood.
Our motivation is quite different. We start from the idea of denoising autoencoders as learning another kind of implicit probability distribution, a dynamical system that maps input vectors towards regions of higher probability. We add the idea that discriminatively trained neural networks (of which the discriminator in a typical GAN is an instance) have been empirically shown to learn rich feature distributions that encode a great deal of detailed information about their inputs even at very high levels in the hierarchy. We learn a denoising autoencoder to such features extracted from training data and *apply* the denoising autoencoder to features from samples, thereby using the dynamics defined by the denoising autoencoder to arrive at features that are more likely under the DAE's learned distribution.
Effectively, we treat the features extracted from generated samples as "true features" plus some noise, and ask the DAE to "denoise" them into the correct features. We then train the generator to produce samples that match these features better.
Aside from learning this model in discriminator feature space (and still training a classifier as a discriminator in the first place), perhaps a key technical difference is that at no point do we backpropagate through the autoencoder to the generator. In EBGAN, the generator is interested in manipulating the output of the autoencoder; here we are interested in manipulating the feature space of a discriminator, using the (denoising) autoencoder as a tool to tell us *how* to perform that manipulating (backpropagating through the autoencoder would just seek a fixed point of the DAE dynamics which is not obviously useful).
We view the EBGAN work as a step along a different but interesting path of its own, namely generalizing the concept of an adversary to alternative "contrast functions" (or "energy functions", though the term is somewhat overloaded), but both the motivations and details of the procedure are quite distinct.
The authors present a way to complement the Gerative Adversarial Network traning procedure with an additional term based on denoising autoencoders. The use of denoising autoencoders is motivated by the observation that they implicitly capture the distribution of the data they were trained on.
While sampling methods based denoising autoencoders alone don't amount to very interesting generative models (at least no-one could demonstrate otherwise), this paper shows that it can be combined successfully with generative adversarial networks.
My overall assessment of this paper is that it is well written, well reasoned, and presents a good idea motivated from first principles. I feel that the idea presented here is not revolutionary or a very radical departure from what has been done before, I would have liked a slightly more structured experiments section which focusses on and provides insights into the relative merits of different choices one could make (see pre-review questions for details), rather than focussing just on demonstrating that a chosen variant works.
In addition to this general review, I have already posted specific questions and criticism in the pre-review questions - thanks for the authors' responses. Based on those responses the area I am most uncomfortable about is whether the (Alain & Bengio, 2014) intuition about the denoising autoencoders is valid if it all happens in a nonlinear featurespace. If the denoiser function's behaviour ends up depending on the Jacobian of the nonlinear transformation Phi, another question is whether this dependence is exploitable by the optimization scheme.
Thank you for your review, and happy new year, apologies for the late response, as I said above I have been nursing an injury that makes typing rather unpleasant.
The score of the feature distribution (in the case of an invertible feature extractor) is related to the score of the original distribution via a locally affine function involving the Jacobian of phi, namely a multiplication by the inverse Jacobian of phi plus an additive term that is essentially the gradient of the log determinant of the same with respect to the point x (again multiplied by the Jacobian inverse). This latter term is rather difficult to interpret. The derivation will be included in a revision to be posted shortly. We would stand by the statement that the procedure remains, as presented, a useful heuristic, making more efficient use of computation already being done in the discriminator forward pass.
In addition, I am running two additional experiments (later than I'd hoped due to the aforementioned circumstances), using a pretrained classifier of identical architecture (save for the output layer) as the feature space, and incorporating gradients from the denoising loss into the updates for discriminator parameters. These will be incorporated as soon as possible.
This paper is about using denoising autoencoders to improve performance in GANs.
In particular, the features as determined by the discriminator, of images generated by the generator, are fed into a denoising AE and we try to have these be reconstructed well. I think it's an interesting idea to use this "extra information" -- namely the feature representations learned by the discriminator. It seems very much in the spirit of ICLR! My main concern, though, is that I'm not wholly convinced on the nature of the improvement. This method achieves higher inception scores than other methods in some cases, but I have a hard time interpreting these scores and thus a hard time getting excited by the results.
In particular, the authors have not convinced me that the benefits outweigh the required additional sophistication both conceptually and implementation-wise (speaking of which, will code be released?). One thing I'd be curious to know is, how hard is it to get this thing to actually work?
Also, I view GANs as a means to an end -- while I'm not particularly excited about generating realistic images (especially in 32x32), I'm very excited about the future potential of GAN-based systems.
So it would have been nice to see these improvements in inception score translate into improvements in a more useful task.
But this criticism could probably apply to many GAN papers and so perhaps isn't fair here.
I do think the idea of exploiting "extra information" (like discriminator features) is interesting both inside and outside the context of this paper.
Thank you for your review and happy new year.
I can sympathize with the suspicion that a method adds complexity in a way that may be quite brittle to tuning. I will, however, say that in our case, the method is quite robust; faster convergence towards reasonable-looking samples as well as increased objectness are visible even with small denoisers, although adding depth improved things. We did not extensively tune the architectural hyperparameters of the denoiser, simply using a hidden layer dimension equal to the input dimension in most cases, and using very simple fully connected networks, trained with the same learning rate as everything else (1e-4, approximately the same as Radford et al). This robustness will be further emphasized in a revision to be posted shortly.
I hope to release the code in the next week or so after it is cleaned up and pulled together from a few disparate codebases, I apologize for not releasing it sooner as I have been nursing an injury.
I too see GANs largely as a means to an end; the unconditional generation setting allows us to study some known shortcomings in isolation. It is likely that this method can be extended to the conditional case where most applications lie but there are non-obvious choices to be made about incorporating conditioning information into the denoiser, We demonstrate here that the idea appears to bear fruit in the simple case, making it worth investigating such extensions. The Inception score is certainly far from a perfect measure but provides at least some quantitative backing to our more qualitative claims.
This paper is well written, and well presented. This method is using denoise autoencoder to learn an implicit probability distribution helps reduce training difficulty, which is neat. In my view, joint training with an auto-encoder is providing extra auxiliary gradient information to improve generator. Providing auxiliary information may be a methodology to improve GAN.
Extra comment:
Please add more discussion with EBGAN in next version.
Thank you for your comments. A revision to be posted shortly will incorporate more discussion of the relationship as described in my question reply.
A revision has been posted, addressing several textual concerns, correcting an error in equation 5, and including discussion of the effect of the feature space Phi on the score function learned by the DAE.
A further update will follow later this week if any of the additional experiments bear fruit. Apologies again for the delay.

Meta-review:
role: PC
data: comment: The idea of using a denoising autoencoder on features of the discriminator is sensible, and explored and described well here. The qualitative results are pretty good, but it would be nice to try some of the more recent likelihood-based methods for quantitative evaluation, as the inception score is not very satisfying. Also it would be interesting to see if this additional term helps in scaling up to larger images.

Decision: Accept (Poster)