Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: SUMMARY 
This paper addresses important questions about the difficulties in training generative adversarial networks. It discusses consequences of using an asymmetric divergence function and sources of instability in training GANs. Then it proposes an alternative using a smoothening approach. 

PROS 
Theory, good questions, nice answers. 
Makes an interesting use of concepts form analysis and differential topology. 
Proposes avenues to avoid instability in GANs. 

CONS 
A bit too long, technical. Some parts and consequences still need to be further developed (which is perfectly fine for future work). 

MINOR COMMENTS

- Section 2.1 Maybe shorten this section a bit. E.g., move all proofs to the appendix. 

- Section 3 provides a nice, intuitive, simple solution. 

- On page 2 second bullet. This also means that P_g is smaller than the data distribution in some other x, which in turn will make the KL divergence non zero. 

- On page 2, ``for not generating plausibly looking pictures'' should be ``for generating not plausibly looking pictures''.  

- Lemma 1 would also hold in more generality. 

- Theorem 2.1 seems to be basic analysis. (In other words, a reference could spare the proof). 

- In Theorem 2.4, it would be good to remind the reader about p(z). 

- Lemma 2 seems to be basic analysis. (In other words, a reference could spare the proof). 
Specify the domain of the random variables. 

- relly - > rely 

- Theorem 2.2 the closed manifolds have boundary or not? (already in the questions)

- Corollary 2.1, ``assumptions of Theorem 1.3''. I could not find Theorem 1.3. 

- Theorem 2.5 ``Therefore'' -> `Then'? 

- Theorem 2.6 ``Is a... '' -> `is a' ? 

- The number of the theorems is confusing. 

review_title: review of ``TOWARDS PRINCIPLED METHODS FOR TRAINING GENERATIVE ADVERSARIAL NETWORKS'' 

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: This is a strong submission regarding one of the most important and recently introduced methods in neural networks - generative adversarial networks. The authors analyze theoretically the convergence of GANs and discuss the stability of GANs. Both are very important. To the best of my knowledge, this is one of the first theoretical papers about GANs and the paper, contrary to most of the submissions in the field, actually provides deep theoretical insight into this architecture. The stability issues regarding GANs are extremely important since the first proposed versions of GANs architecture were very unstable and did not work well in practice. Theorems 2.4-2.6 are novel and introduces mathematical techniques are interesting. I have some technical questions regarding the proof of Theorem 2.5 but these are pretty minor.
review_title: very solid contribution to the theoretical understanding of GANS

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: This is a strong submission regarding one of the most important and recently introduced methods in neural networks - generative adversarial networks. The authors analyze theoretically the convergence of GANs and discuss the stability of GANs. Both are very important. To the best of my knowledge, this is one of the first theoretical papers about GANs and the paper, contrary to most of the submissions in the field, actually provides deep theoretical insight into this architecture. The stability issues regarding GANs are extremely important since the first proposed versions of GANs architecture were very unstable and did not work well in practice. Theorems 2.4-2.6 are novel and introduces mathematical techniques are interesting. I have some technical questions regarding the proof of Theorem 2.5 but these are pretty minor.

review_title: very interesting submission

final_score: rating: null
review_content: Very good paper, I hope it will be accepted. I keep my original evaluation.
review_title: final evaluation - strong submission

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper makes a valuable contribution to provide a more clear understanding of generative adversarial network (GAN) training procedure. 

With the new insight of the training dynamics of GAN, as well as its variant, the authors reveal the reason that why the gradient is either vanishing in original GAN or unstable in its variant. More importantly, they also provide a way to avoid such difficulties by introducing perturbation. I believe this paper will inspire more principled research in this direction. 

I am very interested in the perturbation trick to avoid the gradient instability and vanishment. In fact, this is quite related to dropout trick in where the perturbation can be viewed as Bernoulli distribution. It will be great if the connection can be discussed.  Besides the theoretical analysis, is there any empirical study to justify this trick? Could you please add some experiments like Fig 2 and 3 for the perturbated GAN for comparison? 
review_title: good submission

final_score: rating: null
review_content: In Theorem 2.2. what is the definition of an `optimal' discriminator? 
Closed manifolds are intended to mean manifolds without boundary or manifolds with boundary? 
review_title: minor questions

final_score: rating: null
review_content: I'd like to ask a question what you think is the next research direction that should be taken regarding theoretical understanding of GANs.
review_title: next steps

Meta-review: The paper provides a detailed analysis of the instability issues surrounding the training of GANs. They demonstrate how perturbations can help with improving stability. Given the popularity of GANs, this paper is expected to have a significant impact.
Decision: Accept (Oral)