Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors study the ‘non-saturating’ variant of training for GANs and show that it is equivalent to a regular training procedure minimizing a “softened” reverse KL divergence as opposed to Jensen-Shannon divergence. They show a connection between the two training procedures for more general f-divergence losses. For instance, they show that
1. non-saturating training on original GAN loss minimizes KL(p/2+q/2 || p) 
2. non-saturating training on KL(p||q) loss minimizes KL(q||p)

The authors start by arguing about previous analyses of the non-saturating training scheme and present why they do not arrive at the complete picture. Then they go on to introduce f-divergences and f-GAN training before explaining in their notation what precisely non-saturating training means. Then they show that it corresponds to minimization of not the original f-divergence but a hybrid (f,g) divergence. 

Overall the paper presents most of its insights as speculative statements and does not do a good enough job of attempting to concretely formalize them.

Questions:
1. Why is argument of Nowozin et al wrong when q is parametric?
2. What are tail weights of an f-divergence?
3. Unclear why one needs the variational lower bound in f-GAN training.
4. Why does hybrid training (f,g) converge to the minimization of f? Can the discussion in Appendix F be formalized into a Theorem or a Lemma?
5. One of the main takeaways of the paper appears to be that initial phases of JS divergence minimization leads to flat gradients which can be problematic - question: is this an artifact of JS being bounded? Could unbounded divergences avoid running into this issue?

The content organization and highlighting of the main result in the paper can be significantly improved. Since the paper is exposing a theoretical connection as its primary result, I would also recommend a higher level of formalism overall.
1. Figure 2 is references much earlier than it appears
2. Tail weights are referenced before they are defined
3. Formal statement of non-saturating gradient based training captured within a subsection or box. Hard to locate in current draft. Need to read 5 sections to get to it.
4. Appendix C should be in main body.
5. Some definitions need to be defined more formally. For instance,
  a. Hybrid optimization
  b. Non-saturating training

It is unclear how significant the contribution of the paper is. It is a clean mathematical observation but the consequences of the connection are not explored and fleshed out. For instance, can by realizing that the non-saturating gradient training is optimizing a different f-divergence can we explain why non-saturating gradient training is more useful? Any insights backed by some simple example settings of synthesized probability distributions would also be useful. The paper also does not propose any new training methods based on the insights uncovered and it is not clear how significant the connection uncovered is with the information presented in the current draft. I believe it is an interesting direction that the paper probes and with a more deeper look into the phenomenon and related directions can be ready for publishing in a venue such as ICLR. In it’s current form I unfortunately don’t think the contributions of the paper are significant enough for acceptance.

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: ######## Updated Review #########

I would like to thank the author(s) for their rebuttal, which I have carefully read. I also appreciate the effort made to improve the paper. My overall evaluation of the paper stands unchanged. 



#############################





#### My review is based on the updated paper downloaded from the anonymous link. ####

This paper discusses alternative training strategies for f-GANs. While the discussion has some interesting points, the presentation needs to be much improved. It is not easy to follow this paper in its current form, and the main results are not properly emphasized. As such, I am not certain of its real contribution. f-GANs are not routinely used in practice (except for the vanilla JSD and RKL), and as far as I can tell the saturating gradient issue is no longer a central concern (it has been well addressed years ago, with, e.g. WGANs). I am voting to reject this submission, but I am willing to re-evaluate this paper if the author(s) significantly improves their writing. 

- In Section 2, what does it mean by "the Fisher metric of the family"? The concept of Fisher metric is defined anywhere in the text, and there is no reference to it.
- It is not is intuitive why the second derivative of f_R takes the form $u^{-3} f''(u^{-1})$ (given above Eqn (2)). Please elaborate. 
- Please avoid the use of subjective phrases such as "imagining", "get a feel", etc. I am guessing the author(s) are trying to suggest taking a visual inspection of the discrepancies projected on the log-likelihood ratio axis (which is 1D) and figure out which f-div might be more appropriate. 
- Fig. 3 needs legends. There are two solid (dotted, resp) lines in the Figure, and I am guessing one of them is for the saturating and the other for the non-saturating gradient. This needs to be specified because the line specs are identical. 
- After going through the entire paper, I would highly recommend the author(s) to take a course in academic writing. The main contributions are not highlighted and some of the key concepts are not even properly defined. For example, analysis of the non-saturating gradient, which is supposedly the main result of this submission, appeared in pp. 7, by which time most readers have exhausted their patience. The vanilla version of the non-saturating scheme never appeared in the main text. The notation system is also non-standard, where the notation $\bar{\lambda}$, normally reserved for average/expectation, has been used to denote the gradient wrt $\lambda$. The writing can be very unprofessional at times, for example, "the answers to these questions are yes, yes and no respectively".  
- I do not know why the author(s) inserted one toy experiment in the paper, as it serves no purpose. Each model converges to their respectively optimal, as expected. There is no discussion of how to choose an appropriate f-div or (f,g) hybrid in practice. 
- I totally agree with the author(s) on the point that this is a note (sec 7, second paragraph), rather than an academic paper. There is a lot of derivations in the paper, but insightful discussions are limited in the sense that it is not clear how to connect these results to improve practice. The section titles also read like bullet points in a note. 
- Not certain about the significance of this paper. The derivations are fairly standard and I can not find any useful proposals that might benefit the practice of f-GAN training. There is a number of papers discussing non-saturating GAN training (see sec 8), and I do not think this paper adds too much value to this discussion. 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this paper, firstly, a useful expression of the class of f-divergences is proposed. The authors investigate theoretical properties of some popular f-divergences from newly developed tools. Then, the expression is used to investigate GANs with the non-saturating training scheme.

The expressions in Section 3 can be a useful tool to investigate the statistical properties of estimators with f-divergences. However, I think that the usefulness of alternative expressions in Section 3 is not very clear, though an intuitive interpretation is presented. Moreover, the numerical experiment is not sufficient to support some new expressions. Detailed theoretical analysis of the non-saturating training based on the proposed expression would be required for publication from ICLR.  In addition, the authors could shorten the paper within 8 pages that is the standard length of ICLR paper. In the paper "On topological properties of f-divergences" (1967), Csiszar intensively studied non-saturating properties of f-divergences
in the paper. It would be helpful for readers to add some comments on the relation between the theoretical development in this paper and Csiszar's paper.
review_title: Official Blind Review #2

Meta-review: As the reviewers point out, the core contribution might be potentially important but the current execution of the paper makes it difficult to gauge this importance. In the light of this, this paper does not seem ready for appearance in a conference like ICLR.
Decision: Reject