Title: Unrolled Generative Adversarial Networks
Authors: Luke Metz
Ben Poole
David Pfau
Jascha Sohl-Dickstein
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning
Optimization

Abstract: We introduce a method to stabilize Generative Adversarial Networks (GANs) by defining the generator objective with respect to an unrolled optimization of the discriminator. This allows training to be adjusted between using the optimal discriminator in the generator's objective, which is ideal but infeasible in practice, and using the current value of the discriminator, which is often unstable and leads to poor solutions. We show how this technique solves the common problem of mode collapse, stabilizes training of GANs with complex recurrent generators, and increases diversity and coverage of the data distribution by the generator.

TL;DR: We introduce a method to stabilize Generative Adversarial Networks by defining the generator objective with respect to an unrolled optimization of the discriminator. 

Review discussion:
Have you considered the relation between this approach with other simpler approaches like historical averaging (3.3 in [1]), expert of discriminators (eg. [2])?
Section 2.4 does a good job of explaining why this approach seems to be working well.
However, I'd like to decouple the effect of just using multiple discriminators and the proposed effect (G taking a step taking into account how D will react). To make the narrative in section 2.4 more convincing, it would be useful to do a baseline, where instead of optimizing over the unrolled optimization, we just move G in accordance with a mixture of the last K discriminators (or something similar).
[1]: https://arxiv.org/pdf/1606.03498v1.pdf
[2]: https://arxiv.org/abs/1611.01673
You make a very good point. We have been running several experiments to isolate the effect of the unrolling, and are updating our paper with the results, and with additional comparison plots (in the appendix). We will now briefly describe the results. All experiments are on the 2d toy task.
We tried updating the generator with an ensemble of discriminators taken from the previous K training steps. For K values up to 50, this had no significant effect on training behavior. The generator mode-hopped, as it does for the vanilla GAN.
We additionally tried building the ensemble of discriminators by updating the ensemble with a new discriminator only every N steps, where N is 100 or 1000, rather than every step. In this way the ensemble of discriminators can capture a much longer history. Surprisingly, in this configuration training becomes less stable, with the generator collapsing to and jumping between modes that don't exist in the data distribution. This is likely because having a set of discriminators that changes only very slowly gives the generator many more steps to exploit weaknesses in those discriminators.
In addition we directly explored the importance of the second term in the gradient in Eq. 12. This is the gradient term that stems from backpropagating through the unrolled optimization. We did this by inserting stop_gradient operations in our computation graph, to drop only that term from the gradient.
When this term is dropped from the generator update it takes approximately twice as many unrolling steps, K, to stabilize training. This suggests that backpropagating through discriminator optimization directly aids stability.
The paper introduces a technique for stabilizing the training of Generative Adversrial Networks by unrolling the inner (discriminator) optimization in the GAN loss function several steps and optimizing the generator with respect to the final state of this optimization process.
The experimental evidence that this actually helps is very compelling: the 2d example shows a toy problem where this technique helps substantially, the LSTM MNIST generator example shows that the procedure helps with stabilizing the training of an unusual architecture of generator, and the image generation experiment, while not being definitive, is very convincing.
For future work it would be interesting to see whether a method with smaller memory requirements could be devised based on similar principles.
I strongly recommend to accept this paper.
Thanks so much for your remarks. I totally agree that reducing memory usage and computation would be very useful and we are thinking about these things.
Meanwhile, I appreciate the vote of confidence!
The paper presents an approach for tackling the instability problem that is present in generative adversarial networks. The general idea is to allow the generator to "peek ahead" at how the discriminator will evolve its decision boundary over-time with the premise that this information should prevent the generator from collapsing to produce only samples from a single mode of the data distribution.
This is a very well written paper that clearly motivates its attack on an important open issue. The experiments are well carried out and strongly support the presented idea. The pursued approach is substantially more elegant than current existing "hacks" that are commonly used to make GANs work in practice. I however have three main issues that let me partly doubt the success of the method. If these can be resolved this paper is a clear candidate for acceptance.
1) I am not entirely convinced that the same effect cannot be obtained by the following procedure: simply train the discriminator for an extended number of K steps when updating the generator (say a number equivalent to the unrolling steps used in the current experiments) then, after the generator was updated undo the K updates to the discriminator and do 1 new update step instead. I only briefly glanced at your response to Reviewer2 which seems to imply you now tried something similar to this setup by stopping gradient flow at an appropriate point (although I think this is not exactly equivalent).
2) I tried to reproduce the simple MNIST example but using a fully connected network instead of an RNN generator without much success.
Even when unrolling the discriminator for 30-40 steps the generator still engages in mode seeking behavior or does not train at all. This could either be because of a bug in my implementation or because of some peculiarities of the RNN generator or because I did not use batch normalization anywhere. If it is one of the latter two this would entail a dependence of the proposed approach on specific forms of the discriminator and generator and should be discussed. My code can be found here https://github.com/iclrreproducer/unrolled_gan -- see the comments in the file for switching on unrolling and batch normalization in the generator. This issue could also be addressed simply by opening up the code and setting up a fully connected network example on your side.
3) For the more complicated data distributions the method is only used in combination with many of the existing tricks for training GANs.
As a result the experiments are much less convincing. I personally cannot see a strong difference between the samples generated by any of the CIFAR-10 models. Why did you not try to remove batch-normalization etc. for these examples ? Does training then again become unstable ?
---- UPDATE -----
I am leaving the initial review intact to preserve the conversation. After the authors response all points have been resolved, including a fix to my re-implementation of the ideas presented in the paper.
As a result I have increased my score and believe the paper should be accepted. Please also see my response to the rebuttal below.
Thanks very much for your thoughtful remarks. We are working on a revision now to address these and other considerations.  I will update the PDF and will let you know when it’s complete and will note what has changed. Thanks again.
Thank you for your comments, and for taking the time to implement our technique!!
1. This is a very good point. We have run experiments to test this concept in the toy mixture of Gaussian setting. Running the unrolling algorithm with a stop_gradient preventing gradients from backpropagating through the unrolling is identical to the experiment you suggest. We looked at the effect of including these stop_gradient terms, and additionally at the effect of using an ensemble of historical discriminators. These results can be found in Appendix B of our updated paper. The TLDR is that stabilization also occurs with your suggested approach (without backpropagating through the unrolled optimization), but that it requires more unrolling steps than when the gradient is allowed to backpropagate through the unrolling.
2. Sorry to hear about your results. Sadly GANs are still very finicky and this technique is not a panacea. Your code and implementation was almost perfect; just a few hyper parameters were slightly off [1].
To summarize, we made the relative sizes of the networks the same (500 units each), changed the learning rates, and while probably not critical, moved from using tf.log to the more stable tf.nn.sigmoid_cross_entropy.
In addition to this, we have released a simple implementation in an IPython notebook[2]. It takes a slightly different approach than your implementation, using graph replacement.
3. In these experiments, we wanted to show that there is more to these models than visual sample quality.
In fact, we make the point in the figure caption that these 4 models produce samples that are nearly indistinguishable to the eye.
However, they have different properties when it comes to capturing the diversity of the dataset. Unrolled GANs produce more diverse samples while maintaining the same visual sample quality as standard GANs.
As we discuss in our response to AnonReviewer2, we have additionally added two new measurements of sample diversity for unrolled GANs, presented in Section 3.4.
We believe we have addressed all three of your primary concerns, and we hope you will raise your score as a result.
Thank you very much again for your time and careful review!!
[1] https://github.com/iclrreproducer/unrolled_gan/pull/1
[2] https://github.com/poolio/unrolled_gan
In summary:
Thanks for responding to the issues and examining my re-implementation. I have read through the revised version of the paper and including the additional experiments that you performed as a response to Reviewer 2 I think this paper has significantly been improved and should be published. I have increased my score accordingly.
Detailed response:
Regarding 1) It is good to see that my intuition was correct here; as you write in the paper the additional gradient term should vanish for a discriminator that is unrolled for a large amount of steps between generator steps. The fact that the alternative unrolling experiment without the second term also works does IMO not detract from your idea but rather completes the picture as it makes perfect sense that in cases where the gradient term does not vanish it should be a useful training signal. The only thing that one could question would be the idea that the second gradient term is strictly necessary, rather it seems that it is important to train the discriminator as far to convergence as possible to obtain a reliable learning signal from the "inner optimization".
Regarding 2) Great, I think it is great that we are able to "debug" my understanding of the paper via code directly within the review process and that an available implementation of your method strengthens the submission. I encourage you to open up your code base as well (as you already started to). The difference between running your fixed version of my code with lookahead=0 (no unrolling) and lookahead=5 is quite obvious.
Regarding 3) I read through the additional experiments and found them to clearly support the claims of the paper.
Additional remarks:
There are a few typos/grammar issues in the newly added sections, it would be a shame if they would make it into the final version. Eg.:
- "we use a technique similar to one in" -> the one from Chen ...
- Section 3.4.2 has some tense switching
- Appendix B2: equation 2 -> Equation 2
We are are glad our revised paper made your understanding clearer, and thank you for your raised scores. We also thank you for your additional remarks. The paper is being updated accordingly now.
I missed the deadline for the pre-review question. Please see main review for remaining issues.
This work introduces a novel method for training GANs by displacing simultaneous SGD, and unrolling the inner optimization in the minmax game as a computational graph. The paper is very clearly written, and explains the justification very well. The problem being attacked is very significant and important. The approach is novel, however, similar ideas have been tried to solve problems unrelated to GANs.
The first quantitative experiment is section 3.3.1, where the authors attempt to find the best z which can generate training examples. This is done by using L-BFGS on |G(z) - x|. The claim is that if we're able to find such a z, then the generator can generate this particular training example. It's demonstrated that 0-step GANs are not able to generate many training examples, while unrolled GANs do.
However, I find this experiment unreasonable. Being able to find a certain z, which generates a certain sample does not guarantee that this particular mode is high probability.
In fact, an identity function can potentially beat all the GAN models in the proposed metric. And due to Cantor's proof of equivalence between all powers of real spaces, this applies to smaller dimension of z as well. More realistically, it should be possible to generate *any* image from a generator by finding a very specific z. That a certain z exists which can generate a sample does not prove that the generator is not missing modes. It just proves that the generator is similar enough to an identity function to be able to generate any possible image. This metric is thus measuring something potentially tangential to diversity or mode-dropping.
Another problem with this metric is that that showing that the optimization is not able to find a z for a specific training examples does not prove that such a z does not exist, only that it's harder to find.
So, this comparison might just be showing that unrolled GANs have a smoother function than 0-step GANs, and thus easier to optimize for z.
The second quantitative experiment considers mean pairwise distance between generated samples, and between data samples. The first number is likely to be small in the case of a mode-dropping GAN. The authors argue that the two numbers being closer to each other is an indication of the generated samples being as diverse as the data. Once again, this metric is not convincing. 1. The distances are being measured in pixel-space. 2. A GAN model could be generating garbage, and yet still perform very well in this metric.
There are no other quantitative results in the paper.
Even though the method is optimizing diversity, for a sanity check, scores for quality such as Inception scores or SSL performance would have been useful.
Another metric that the authors can consider is training GAN using this approach on the tri-MNIST dataset (concatenation of 3 MNIST digits), which results in 1000 easily-identifiable modes.
Then, demonstrate that the GAN is able to generate all the 1000 modes with equal probability. This is not a perfect metric either, but arguably much better than the metrics in this paper. This metric is used in this ICLR submission: https://openreview.net/pdf?id=HJKkY35le
Whether this paper is accepted or not, I encourage the authors to investigate this approach further, since the method is promising and interesting.
# Post-rebuttal review
The authors have incorporated changed in the paper by adding more experiments. These experiments now demonstrate the claims of the paper better. The paper was already well-written and introduced a novel idea and addressed an important problem. The only thing holding this paper back was unconvincing experiments, which now has been corrected.
Thus, I would increase my score by 2 points, and recommend accepting the paper.
Thanks very much for your thoughtful remarks. We are working on a revision now to address these and other considerations.  I will update the PDF and will let you know when it’s complete and will note what has changed. Thanks again.
Thank you for your review, and for your specific suggestions for improving our analysis! We have performed additional experiments as you suggest, including on the tri-MNIST dataset. This has significantly improved the quantitative portion of our paper, and we hope you will raise your score as a result.
You make a good point on the difficulty of interpreting the results of the optimization-based inference. Approaches similar to this have recently been used in several applications [1,2,3], so we are not alone in our choice. We believe the results are strongly suggestive of the learned models more completely covering the image manifold, but we are not able to rule out unrolling smoothing the loss landscape.
“More realistically, it should be possible to generate *any* image from a generator by finding a very specific z.”: In current GAN architectures, z-space is actually far lower dimensional than x-space.
As a result (barring edge cases with large fractal dimension), the manifold of achievable images does not fill x-space, and in general there is no z vector that can exactly reconstruct an image. Depending on the learned generator, it will therefore be possible to reconstruct some images with lower error than others.
The pairwise distance histogram could be matched by a high entropy sampling distribution that did not match the data.
However, the combination of high quality samples and large pairwise distances do lead to a meaningful metric of diversity.
Furthermore, we optimize the GAN to match the data distribution not to match pairwise distances, and this correspondence is unlikely to happen purely by chance. This metric additionally bears resemblance to commonly used non-parametric estimates of entropy using nearest neighbors.
Alternative metrics like the inception score suffer from similar flaws: if the generated samples are all from a single class it can have a high inception score (the distribution over classes will be peaked at that one class for all samples) but very poor sample diversity (the generator only produces one class) (see also Odena et al.’s ICLR submission[4]).
Thank you for suggesting the tri-MNIST dataset, and the associated analysis technique of counting the discrete modes covered. This is a particularly appealing approach, since it allows us to directly measure the property of interest, on a (nonlinear) projection of the model distribution. We tested unrolled GANs on this task, and found that they consistently cover more modes than standard GANs. To address the issue of mode collapse for continuous manifolds, we additionally introduced a new colored MNIST dataset, where we can measure the extent to which the generated samples match the continuous distribution of colors. We have updated the paper to include these results (see Section 3.4).
Thank you again for your feedback!
[1] https://arxiv.org/abs/1609.07093
[2] https://arxiv.org/abs/1609.03552
[3] https://arxiv.org/abs/1605.09304
[4] https://arxiv.org/abs/1610.09585
1. "We believe the results are strongly suggestive of the learned models more completely covering the image manifold, but we are not able to rule out unrolling smoothing the loss landscape.". I agree with both points. Perhaps the language of the section should be changed to caution the readers that although this metric is quite substantive, it alone is not conclusive.
2. "the combination of high quality samples and large pairwise distances do lead to a meaningful metric of diversity": Exactly.
So, this paper should ideally include a metric for sample quality as well, such as inception score, so as to not resort to convincing readers and reviewers by subjective judgement of samples. Demonstrating only an improvement in diversity is meaningless, but demonstrating improvement in diversity AND same or better quality is very meaningful and compelling.
3. "inception score suffer from similar flaws". I'm not arguing for inception score alone. See 2.
4. Good job with the 1000 class and colored MNIST. The experiments are much more convincing to me now.
5. (non-technical) The newly added sections need to be formatted better.
Thank you again for your suggested experiments, and we are glad they increased your faith in this paper. We agree that a comment to warn the reader is in order. We have added that and will update the paper soon.

Meta-review:
role: PC
data: comment: The main idea of this paper is clearly presented, and its empirical claims well-demonstrated. It improves our understanding of how to train GANs.

Decision: Accept (Poster)