The divergences minimized by non-saturating GAN training

Abstract: Interpreting generative adversarial network (GAN) training as approximate divergence minimization has been
      theoretically insightful, has spurred discussion, and has lead to theoretically and practically interesting
      extensions such as f-GANs and Wasserstein GANs. For both classic GANs and f-GANs, there is an original variant of training and a "non-saturating" variant which uses an alternative form of generator gradient. The original variant is theoretically easier to study, but for GANs the alternative variant performs better in practice. The non-saturating scheme is often regarded as a simple modification to deal with optimization issues, but we show that in fact the non-saturating scheme for GANs is effectively optimizing a reverse KL-like f-divergence. We also develop a number of theoretical tools to help compare and classify f-divergences. We hope these results may help to clarify some of the theoretical discussion surrounding the divergence minimization view of GAN training.

Reviews and author responses:

The authors study the ‘non-saturating’ variant of training for GANs and show that it is equivalent to a regular training procedure minimizing a “softened” reverse KL divergence as opposed to Jensen-Shannon divergence. They show a connection between the two training procedures for more general f-divergence losses. For instance, they show that
1. non-saturating training on original GAN loss minimizes KL(p/2+q/2 || p) 
2. non-saturating training on KL(p||q) loss minimizes KL(q||p)

The authors start by arguing about previous analyses of the non-saturating training scheme and present why they do not arrive at the complete picture. Then they go on to introduce f-divergences and f-GAN training before explaining in their notation what precisely non-saturating training means. Then they show that it corresponds to minimization of not the original f-divergence but a hybrid (f,g) divergence. 

Overall the paper presents most of its insights as speculative statements and does not do a good enough job of attempting to concretely formalize them.

Questions:
1. Why is argument of Nowozin et al wrong when q is parametric?
2. What are tail weights of an f-divergence?
3. Unclear why one needs the variational lower bound in f-GAN training.
4. Why does hybrid training (f,g) converge to the minimization of f? Can the discussion in Appendix F be formalized into a Theorem or a Lemma?
5. One of the main takeaways of the paper appears to be that initial phases of JS divergence minimization leads to flat gradients which can be problematic - question: is this an artifact of JS being bounded? Could unbounded divergences avoid running into this issue?

The content organization and highlighting of the main result in the paper can be significantly improved. Since the paper is exposing a theoretical connection as its primary result, I would also recommend a higher level of formalism overall.
1. Figure 2 is references much earlier than it appears
2. Tail weights are referenced before they are defined
3. Formal statement of non-saturating gradient based training captured within a subsection or box. Hard to locate in current draft. Need to read 5 sections to get to it.
4. Appendix C should be in main body.
5. Some definitions need to be defined more formally. For instance,
  a. Hybrid optimization
  b. Non-saturating training

It is unclear how significant the contribution of the paper is. It is a clean mathematical observation but the consequences of the connection are not explored and fleshed out. For instance, can by realizing that the non-saturating gradient training is optimizing a different f-divergence can we explain why non-saturating gradient training is more useful? Any insights backed by some simple example settings of synthesized probability distributions would also be useful. The paper also does not propose any new training methods based on the insights uncovered and it is not clear how significant the connection uncovered is with the information presented in the current draft. I believe it is an interesting direction that the paper probes and with a more deeper look into the phenomenon and related directions can be ready for publishing in a venue such as ICLR. In it’s current form I unfortunately don’t think the contributions of the paper are significant enough for acceptance.


######## Updated Review #########

I would like to thank the author(s) for their rebuttal, which I have carefully read. I also appreciate the effort made to improve the paper. My overall evaluation of the paper stands unchanged. 



#############################





#### My review is based on the updated paper downloaded from the anonymous link. ####

This paper discusses alternative training strategies for f-GANs. While the discussion has some interesting points, the presentation needs to be much improved. It is not easy to follow this paper in its current form, and the main results are not properly emphasized. As such, I am not certain of its real contribution. f-GANs are not routinely used in practice (except for the vanilla JSD and RKL), and as far as I can tell the saturating gradient issue is no longer a central concern (it has been well addressed years ago, with, e.g. WGANs). I am voting to reject this submission, but I am willing to re-evaluate this paper if the author(s) significantly improves their writing. 

- In Section 2, what does it mean by "the Fisher metric of the family"? The concept of Fisher metric is defined anywhere in the text, and there is no reference to it.
- It is not is intuitive why the second derivative of f_R takes the form $u^{-3} f''(u^{-1})$ (given above Eqn (2)). Please elaborate. 
- Please avoid the use of subjective phrases such as "imagining", "get a feel", etc. I am guessing the author(s) are trying to suggest taking a visual inspection of the discrepancies projected on the log-likelihood ratio axis (which is 1D) and figure out which f-div might be more appropriate. 
- Fig. 3 needs legends. There are two solid (dotted, resp) lines in the Figure, and I am guessing one of them is for the saturating and the other for the non-saturating gradient. This needs to be specified because the line specs are identical. 
- After going through the entire paper, I would highly recommend the author(s) to take a course in academic writing. The main contributions are not highlighted and some of the key concepts are not even properly defined. For example, analysis of the non-saturating gradient, which is supposedly the main result of this submission, appeared in pp. 7, by which time most readers have exhausted their patience. The vanilla version of the non-saturating scheme never appeared in the main text. The notation system is also non-standard, where the notation $\bar{\lambda}$, normally reserved for average/expectation, has been used to denote the gradient wrt $\lambda$. The writing can be very unprofessional at times, for example, "the answers to these questions are yes, yes and no respectively".  
- I do not know why the author(s) inserted one toy experiment in the paper, as it serves no purpose. Each model converges to their respectively optimal, as expected. There is no discussion of how to choose an appropriate f-div or (f,g) hybrid in practice. 
- I totally agree with the author(s) on the point that this is a note (sec 7, second paragraph), rather than an academic paper. There is a lot of derivations in the paper, but insightful discussions are limited in the sense that it is not clear how to connect these results to improve practice. The section titles also read like bullet points in a note. 
- Not certain about the significance of this paper. The derivations are fairly standard and I can not find any useful proposals that might benefit the practice of f-GAN training. There is a number of papers discussing non-saturating GAN training (see sec 8), and I do not think this paper adds too much value to this discussion. 

> The expressions in Section 3 can be a useful tool to investigate the statistical properties of estimators with f-divergences. However, I think that the usefulness of alternative expressions in Section 3 is not very clear, though an intuitive interpretation is presented.

We completely agree that the present paper does not immediately suggest ways to improve current GAN training practice. As mentioned when responding to reviewer #3, we hope that progress over time comes from both better theoretical understanding and better experimental results. In this case, we would argue that our result is simple to state and understand ("non-saturating GAN training doesn't actually optimize Jensen-Shannon") and provides a better understanding of a state-of-the-art method for GAN training (used for example for StyleGAN).

> Moreover, the numerical experiment is not sufficient to support some new expressions. Detailed theoretical analysis of the non-saturating training based on the proposed expression would be required for publication from ICLR. 

The paper outlines a reinterpretation of what current practice is already doing. Implementing optimization of the softened reverse KL using our approach is (intentionally) identical to the standard implementation of current non-saturating GAN training. To try to understand your concern better, is it that the paper does not provide a practically improved way to train GANs, or that you don't feel the paper empirically justifies its theoretical reformulation of current GAN training sufficiently?

> In addition, the authors could shorten the paper within 8 pages that is the standard length of ICLR paper.

Apologies, it's now of the standard length.

> In the paper "On topological properties of f-divergences" (1967), Csiszar intensively studied non-saturating properties of f-divergences
in the paper. It would be helpful for readers to add some comments on the relation between the theoretical development in this paper and Csiszar's paper.

We were unfortunately unable to locate a copy of this paper. We tried searching the back issues of Studia Scientiarum Mathematicarum Hungarica and the usual aggregators. If you have any suggestions we would be very grateful.

> I totally agree with the author(s) on the point that this is a note (sec 7, second paragraph), rather than an academic paper. There is a lot of derivations in the paper, but insightful discussions are limited in the sense that it is not clear how to connect these results to improve practice. The section titles also read like bullet points in a note.

The references to a note are now fixed. We completely agree that the paper does not immediately suggest ways to improve current best practice. However we would argue that progress over time comes from both better theoretical understanding and better experimental results. In this case, the result is simple to state and understand ("non-saturating GAN training doesn't actually optimize Jensen-Shannon") and provides a better understanding of a state-of-the-art method for GAN training (used for example for StyleGAN).

> Not certain about the significance of this paper. The derivations are fairly standard and I can not find any useful proposals that might benefit the practice of f-GAN training. There is a number of papers discussing non-saturating GAN training (see sec 8), and I do not think this paper adds too much value to this discussion. 

We acknowledge that our derivation of f-GANs takes up quite a bit of space in the paper while being similar in substance to that of the original f-GAN paper. The main differences are a more elementary derivation of the variational lower bound, without using Legendre transforms / Fenchel conjugates (but it's completely trivially equivalent), and our use of a standardized form for the output of the critic (in our case the optimal critic is always log p - log q).

In terms of contributions to the general discussion, the present paper addresses a number of deficiencies in the previous literature in this area (this has been made more prominent in the new draft by moving the related work section straight after the introduction), and extends the current literature by showing for the first time that non-saturating GAN training can be viewed as divergence minimization, but for a divergence that has very different properties from Jensen-Shannon. While a relatively straightforward result and pleasingly straightforward to state, we hope / believe it's one that deserves to be well-known by GAN practitioners and theoreticians.

> This paper discusses alternative training strategies for f-GANs. While the discussion has some interesting points, the presentation needs to be much improved. It is not easy to follow this paper in its current form, and the main results are not properly emphasized. As such, I am not certain of its real contribution. f-GANs are not routinely used in practice (except for the vanilla JSD and RKL), and as far as I can tell the saturating gradient issue is no longer a central concern (it has been well addressed years ago, with, e.g. WGANs). I am voting to reject this submission, but I am willing to re-evaluate this paper if the author(s) significantly improves their writing. 

We view the theoretical result that conventional non-saturating GAN training can be viewed as divergence minimization as the main result. We've tried to make that main contribution more prominent in the abstract and moved the section containing the main result earlier in the paper.

The saturation issue is well-addressed practically by both non-saturating training and WGANs, and we would argue both are prominently used in practice. For example, the ground-breaking StyleGAN paper used non-saturating training, finding it performed better than WGANs. This makes the current paper relevant by extending our best theoretical understanding of current practice.

We agree that f-GANs are not routinely used in practice, and consider our results on "non-saturating" variants of f-GANs are secondary. We cover f-GANs in some detail mainly because our main result is framed in those terms.

> In Section 2, what does it mean by "the Fisher metric of the family"? The concept of Fisher metric is defined anywhere in the text, and there is no reference to it.

Clarified the wording in the main text, removing the reference to the Fisher metric, and added detail in an appendix.

> It is not is intuitive why the second derivative of f_R takes the form  (given above Eqn (2)). Please elaborate.

We added an intermediate step in the derivation.

> Please avoid the use of subjective phrases such as "imagining", "get a feel", etc. I am guessing the author(s) are trying to suggest taking a visual inspection of the discrepancies projected on the log-likelihood ratio axis (which is 1D) and figure out which f-div might be more appropriate.

We will address this before camera ready publication.

> Fig. 3 needs legends. There are two solid (dotted, resp) lines in the Figure, and I am guessing one of them is for the saturating and the other for the non-saturating gradient. This needs to be specified because the line specs are identical.

Apologies, we will address this before camera ready publication.

> After going through the entire paper, I would highly recommend the author(s) to take a course in academic writing. The main contributions are not highlighted and some of the key concepts are not even properly defined. For example, analysis of the non-saturating gradient, which is supposedly the main result of this submission, appeared in pp. 7, by which time most readers have exhausted their patience. The vanilla version of the non-saturating scheme never appeared in the main text. The notation system is also non-standard, where the notation , normally reserved for average/expectation, has been used to denote the gradient wrt . The writing can be very unprofessional at times, for example, "the answers to these questions are yes, yes and no respectively".

Addressing specifics first, we moved the main result several sections earlier in the paper. Overlines are used for a lot of different mathematical concepts, and the use for a gradient is fairly standard when talking about adjoints and derivatives. Removed the "yes, yes, no" line specifically (that was indeed a slightly strange choice of register). In terms of the broader point, we're sincerely sorry it was hard to follow, and thank you for helping us try to improve the manuscript.

> I do not know why the author(s) inserted one toy experiment in the paper, as it serves no purpose. Each model converges to their respectively optimal, as expected. There is no discussion of how to choose an appropriate f-div or (f,g) hybrid in practice. 

Moved the toy experiment to the appendix. We included it for two reasons: People are often suspicious of whether theoretical results can really be applied in practice; and this subject area in particular has had several incorrect or questionably correct attempted derivations previously, and we wanted to provide some empirical evidence that we weren't also making a similar sort of error.

In this paper, firstly, a useful expression of the class of f-divergences is proposed. The authors investigate theoretical properties of some popular f-divergences from newly developed tools. Then, the expression is used to investigate GANs with the non-saturating training scheme.

The expressions in Section 3 can be a useful tool to investigate the statistical properties of estimators with f-divergences. However, I think that the usefulness of alternative expressions in Section 3 is not very clear, though an intuitive interpretation is presented. Moreover, the numerical experiment is not sufficient to support some new expressions. Detailed theoretical analysis of the non-saturating training based on the proposed expression would be required for publication from ICLR.  In addition, the authors could shorten the paper within 8 pages that is the standard length of ICLR paper. In the paper "On topological properties of f-divergences" (1967), Csiszar intensively studied non-saturating properties of f-divergences
in the paper. It would be helpful for readers to add some comments on the relation between the theoretical development in this paper and Csiszar's paper.

Meta-review: As the reviewers point out, the core contribution might be potentially important but the current execution of the paper makes it difficult to gauge this importance. In the light of this, this paper does not seem ready for appearance in a conference like ICLR.

Decision: reject