The divergences minimized by non-saturating GAN training
Interpreting generative adversarial network training as approximate divergence minimization has beentheoretically insightful, has spurred discussion, and has lead to theoretically and practically interestingextensions such as f-GANs and Wasserstein GANs.For both classic GANs and f-GANs, there is an original variant of training and a "non-saturating" variant which uses an alternative form of generator gradient.The original variant is theoretically easier to study, but for GANs the alternative variant performs better in practice.The non-saturating scheme is often regarded as a simple modification to deal with optimization issues, but we show that in fact the non-saturating scheme for GANs is effectively optimizing a reverse KL-like f-divergence.We also develop a number of theoretical tools to help compare and classify f-divergences.We hope these results may help to clarify some of the theoretical discussion surrounding the divergence minimization view of GAN training.
Non-saturating GAN training effectively minimizes a reverse KL-like f-divergence.This paper proposes a useful expression of the class of f-divergences, investigates theoretical properties of popular f-divergences from newly developed tools, and investigates GANs with the non-saturating training scheme.