The divergences minimized by non-saturating GAN training
Authors: Matt Shannon
Venue:  (2020)
Keywords: GAN

Abstract: Interpreting generative adversarial network (GAN) training as approximate divergence minimization has been
theoretically insightful, has spurred discussion, and has lead to theoretically and practically interesting
extensions such as f-GANs and Wasserstein GANs. For both classic GANs and f-GANs, there is an original variant of training and a "non-saturating" variant which uses an alternative form of generator gradient. The original variant is theoretically easier to study, but for GANs the alternative variant performs better in practice. The non-saturating scheme is often regarded as a simple modification to deal with optimization issues, but we show that in fact the non-saturating scheme for GANs is effectively optimizing a reverse KL-like f-divergence. We also develop a number of theoretical tools to help compare and classify f-divergences. We hope these results may help to clarify some of the theoretical discussion surrounding the divergence minimization view of GAN training.

Under review as a conference paper at ICLR 2020
THE DIVERGENCES MINIMIZED BY NON-SATURATING GAN TRAINING
Anonymous authors Paper under double-bling review
ABSTRACT
Interpreting generative adversarial network (GAN) training as approximate diver- gence minimization has been theoretically insightful, spurred discussion, and lead to theoretically and practically interesting extensions such as f-GANs and Wasser- stein GANs. In this paper we show that the widely used "non- noonssaurating" training scheme can also be interpreted in this way, specifically minimizing a particular reverse KL-like f-divergence. We also develop a number of theoretical tools to help compare and classify f-divergences. We hope these results may help to clar- ify some of the theoretical discussion surrounding the divergence minimization view of GAN training.
INTRODUCTION
Generative adversarial networks (GANs) (Goodfellow al., 2014) have enjoyed remarkable progress in recent years, producing images of striking fidelity, resolution and coherence (Karras al., 2018; Miyato et al., 2018; Brock et al., 2018: Karras et al., 2019). There has been much progress in both theoretical and practical aspects of understanding and performing GAN training (Nowozin al., 2016; Arjovsky & Bottou, 2017; Arjovsky et al., 2017; Mescheder et al., 2018; Gulrajani et al., 2017; Sønderby et al., 2017; Miyato et al., 2018; Karras et al., 2018; Brock et al., 2018; Karras et al., 2019).
One of the key considerations for GAN training is the scheme used to update the generator and critic. rich avenue of developments has come from viewing GAN training as divergence minimization. Goodfellow et al. (2014) showed the conventional GAN training can be viewed as approximately minimizing the Sensen-Shannon divergence. f-GANs (Nowozin et al., 2016) approximately min- imize f-divergences such reverse KL in principled way. Wasserstein GANs (Arjovsky et al., 2017) approximately minimize the Wasserstein metric, and combine solid theoretical underpinnings with strong practical results. Nevertheless relatively unprincipled "non- mntnssatura scheme (Good- fellow et al., 2014) has continued to obtain groundbreaking results (Karras et al., 2019) and remains thee approach (Lucic al., 2018).
The effect of the non-saturating scheme on training dynamics, and in particular whether it can be viewed as divergence minimization, has been source of discussion and some confusion since the original formulation of GAN training (Goodfellow et al., 2014). The main result of this paper is to show that the non-saturating scheme approximately minimizes the f-divergence 4 KL ( + $ p), which we refer to as the softened reverse KL divergence (36). This puts non-saturating training on similar footing to Wasserstein GANs theoretically sound approach with strong empirical results. We also discuss how our results relate to previous attempts at this problem and attempt to clarify some of the confusion surrounding the divergence minimization view of naturatta training.
In order to better understand the qualitative behavior of different divergences such softened re- verse KL, we develop several tools. We show how to write f-divergences a symmetry-preservinn way, allowing easy visual comparison of f-divergences in way that reflects their qualitative prop- erties ($7). We develop rigorous formulation of tail weight which generalizes the notions of mode- seeking and covering behavior ($8). Using these tools show that the softened reverse KL di- vergence is fairly similar to the reverse KL but very different to the Jensen-Shannol divergence approximately minimized by the original GAN training scheme.

---

Under review as a conference paper at ICLR 2020
2 PREVIOUS DISCUSSION OF NON-SATURATING GRADIENTS
The precise practical effect of the non-saturating scheme and whether it can be motivated in prin- cipled way have been source of discussion and some confusion. In this section we review previous attempts to view non-saturating gradients as a form of divergence minimization.
The original GAN paper claims that, compared to the saturating training scheme based on the Jensen-Shannon divergence, the non-saturating training scheme "results in the same fixed point of the dynamics of G and D but provides much stronger gradients early in learning." (Goodfellow et al., 2014, Section 3). It is true that the original and non-saturating generator gradients give the same final result in the non-parametric case where q is unrestricted, but this is fairly trivial since both gradients lead to q = p, as do all divergences. It is even true that the dynamics of training are essentially the same for the original and non-saturating gradients when q R p, but again this fairly trivial since all f-divergences agree in this regime, as discussed in $3. However the "fixed point of the dynamics" is certainly not the same in the general case of parametric q (see &G for an empirical demonstration). Our results provide a precise way to view the relationship between saturating and non-saturating generator gradients: They are optimizing different f-divergences.
The original f-GAN paper presents a simple argument that the "non-saturating" training scheme has the same fixed points and that the original and non-saturating generator gradients have the same direction (Nowozin et al., 2016, Section 3.2)". However this argument is erroneous. It is true that if p R q then (f")) is approximately 1 everywhere, and so the original and non-saturating generator gradients are approximately equal, but this is true of any f-divergence. There is no guar- antee that the regime p = will ever be approached in the general case where q belongs to a parametric family, it is not the case that the original and non-saturating generator gradients point in approximately the same direction in general (see SG for an empirical demonstration). In fact, the non-saturating form of generator gradient can have completely different qualitative behavior. For example, we show that the non-saturating KI. scheme fact optimizes reverse KI..
A recent paper showed experimentally that the non-saturating generator gradient can successfully learn a distribution in a case where optimizing Jensen-Shannen divergence should fail, and used this to argue that perhaps is not particularly helpful to view GANs as optimizing Censen-Shannon diver- gence (Fedus et al., 2018). The divergence optimized in practice for parametric critics is not exactly the divergence which would be optimized by the theoretically optimal critic, and this distinction seems particularly important in the situation where y and initially have non-overlapping support. However the fact that non-saturating training not optimizing gensen-Shannon is also highly rele- vant to this discussion, since the gradient in the limit of zero noise is zero for Sensen-Shannon but sizeable for softened reverse KL. Thus the success of non-saturating GAN training in practice may be as much due to its optimizing a different divergence as it is to using inexact critic.
Arjovsky and Bottou correctly recognize that the non-saturating generator gradient results in ap- proximately minimizing a different objective function and derive the function for classic GANs (Arjovsky& & Bottou, 2017, Section 2.2.2). The objective function there is expressed as
(I) which is a slightly convoluted form of the expression 2 2LL(5p= 39 p) we derive below. The paper suggests the negative sign of the second term is "pushing for the distributions to be different, which seems like fault in the update", whereas our expression for the divergence makes it clear that this is not an issue.
KL p) 2JS(p, g)
Poole et al. (2016) present a very similar view to that presented in this paper, including recognizing that the generator and critic may be trained to optimize different f-divergences and interpreting the classic non- non-saturating generator gradient as a hybrid scheme of this form where the generator gradient is based
new f-divergence (Poole et al., 2016). However the f-divergence derived there is f()) = log(1 + =-),, which differs from (50) by a factor of u + 1. We refer to this as the improved generator objectives for GANs (IGOG) divergence. It can be written as (p(p.g) 2 (m p) + KL(p|] m) where
1p + 34. It has f"(u)= u 2 (1+ u) and has (2, tail weights, Figure 3 shows that this divergence is qualitatively quite similar to the softened reverse KL but is not identical. The source of the discrepancy between our results and theirs is matching the value instead of the gradient, and is described in detail in SA.
2a-1 (1+
Only in the final NIPS version of the paper, not the arxiv preprint.
2

---

Under review as a conference paper at ICLR 2020
3 THE FAMILY OF F-DIVERGENCES
We start by reviewing the definition of an f-divergence (Ali & Silvey, 1966) and establishing some basic properties. These properties are described in more detail in jB. Throughout the paper we use the convention that is the "true" distribution and is model intended to approximate p.
Given a strictly convex twice continuously differentiable function f Rgo + with ((1) = 0. the f-divergence between probability distributions with densities p and q over TRKK is defined as:
Dy(p. 9) = { {())))
p(x) ((x))
dx
f-divergences satisfy several mathematical properties. Firstly D, is linear in f. Secondly Dy(p. q) 2 O for all distributions p and with equality iff p = This justifies referring to Dr as a divergence. Dj is completely determined by f" As we will see, the algebraic form of f is often simpler than that of f. All f-divergences agree up to an overall scale factor on the divergence between nearby distributions: If p s q then (p, q) RS (1) KL | q) (see (B). This can also be seen in Figure 2, where all f-divergences approximately overlap near zero If f1) 0 and (1) = then we say f is in canonical form. We can always find such an f by appropriately scaling Dj- Using canonical form removes superficial difference in scaling between different f-divergences, making them easier to compare, e.g. in Figure
The definition (2) appears be quite asymmetric in how it treats p and q. but it oheys particular symmetry (Reid& & Williamson, 2011). Let fR(U) = u.(u==1) Then [(() f(---)--- ('( and (u) u- ("( -). It is easy verify that D (p,9) Dylg.p). With (x: : g())) p(z)} and B = {x= : ((x)) < p(x)), we have
Dy(p,g) = SA g(x){
p(x) ((x)
g(x) p(x)
dr
fR
dx
(3)
This is more explicitly symmetric than (2) in the role of p and q. We refer A the set of left mismatches (g > p). and B as the set of right mismatches (q < p). At each point in A, the two distributions p and q are somewhat mismatched, and the penalty paid for this mismatch in terms of the overall divergence D, is governed by the behavior of [u) for 0 < u < 1 (the "left" of the graph of f). Similarly the penalty paid for right mismatches is governed by fu) for u > Note from (3) that a left mismatch can only be heavily penalized if the point is plausible under q. i.e. g(z) is not tiny. Similarly right mismatch can only heavily penalized for points plausible under p.
4 VARIATIONAL DIVERGENCE ESTIMATION
f-GANs are based on an elegant way to estimate the f-divergence between two distributions given only samples from the two distributions (Nguyen et al., 2010). In this section we review this ap- proach to variational divergence estimation. See $E for details how our derivation and notation relates to that of Nowozin et al. (2016).
There is elegant variational bound on the f-divergence Dy(p q) between two densities p and g- Since ff is strictly convex, its graph lies at or above any of its tangent lines and only touches in one place. That is, for k, u > O.
(kk) (u) + -k- ((/ - (w]
[(w)
(4)
with equality iff k u. This inequality is illustrated in the appendix in Figure 4. Substituting p(x)/g(a) for l and u(x) for u, for any continuously differentiable function u R + Ryo we
obtain
Dy(p.g) 2 (lo))(ut)) de df
- a( - dr
(5)
2 Most results also hold for "discrete" prohability distributions. The only difference that the reparameter-
ization trick can no longer be used to reduce variance of the finite sample approximations.
For simplicity, assume the probability distributions suitably nice, e.g. absolutely continuous with respect the Lebesgue measure R p(z). (xx) for e TRK and p and g continuously differentiable.
3

---

The bound (6) leads naturally to variational divergence estimation. The f-divergence between p and S q can be estimated by maximizing E with respect to (Nguyen et al., 2010). Conveniently E, is expressed in terms of expectations and may be approximately computed and maximized with respect to d using only samples from p and q. we parameterize as a neural net dy with parameters v then can approximate the divergence by maximizing Ep(pp q, dz) with respect v. This does not compute the exact divergence because there is no guarantee that the optimal function d' lies in Under review as a conference paper at ICLR 2020
with equality iff u", where u' ux(x) p(z)/q(x)) The function is referred to the critic. It will be helpful to have a concise notation for this bound. Writing u(z) = exp(d(z)) without loss of generality, for any continuously differentiable function d BR R, we have
Dy(p.9) 2 F
(6)
with equality iff d d", where
dy(p.q.d)= p(z)a)(d(x)) dx - ((x)))(d(x)) dz
(7)
(d = f'(ex((d)
(8)
by (d) = exp(d) (exp(dd) - f(aodd)))((xxppie (exp(d))
(9)
(x) = log p(z) log
(10)
Note that both af and by linear in Their derivatives d' (logu) uf"(n) and (log u) u² ("(u) depend on f only through f"
the family {dz ! of functions representable by the neural net, but we hope that for sufficiently flexible neural nets the approximation will be close.
Here briefly summarize the three main f-divergences we consider. The Kullback-Leible (KL) divergence KL(p1) || g) has f"(u) = u ad) = d and (d expd 1. It has (1,2) tail weights and left-bounded and right-unbounded. The reverse KL divergence KL(g p) has ("(u) 2 (d) exp(-d) - 1 and (cd d. It has (2, 1) tail weights and is left- unbounded and right-bounded. Finally the canonicalized Sansen-Shannon divergence 4JJS(p, q) 2 KL m KL m), where m = 1 24 has f") u(u+1) ad(d) = 2og or (d) + log and by(d) log (-d) 2 log has (1, 1) tail weights and bounded. See $D for details.
VARIATIONAL DIVERGENCE MINIMIZATION
f-GANs (Nowozin et al., 2016) generalize classic GANs allow approximately minimizing any f-divergence. In this section we briefly review and discuss the f-GAN formulation.
Consider the task of estimating a probabilistic model from data using f-divergence. Here p is the true distribution and the goal is to minimize I(A) Dy (p, qx) with respect to X, where À + is a parametric family of densities over T We refer to Za as the generator. For implicit generative models such typical GAN generators, the distribution qx is the result of deterministic transform (=) a stochastic latent variable z. However we do not need to assume this specific form for most of our discussion.
We first note that the variational divergence bound E, satisfies a convenient gradient matching property. This is not made explicit in the original f-GAN paper. Denote the optimal given p and ax by d We saw above that Dy qx) and E (p, qx d) match values at = d They also match gradients: From the definitions of Dy and E, we can verify that they have the same gradient with respect to the generator parameters X:
i 0 Dy(p.gx)= Ox Ey
= =-
- ox eN(a)) (d)
(1I)
d=d3
We can minimize Dy(p. qx) using variational divergence minimization, maximizing Ey (p, gx d.) with respect to while minimizing it with respect X. Adversarial optimization such as this lies
4

---

Under review as a conference paper at ICLR 2020
the heart of all flavors of GAN training. Define Ā and 17 as
X 8 0x Eg(p. /
2 ar ea(z) by by(d
(12)
0 D = 0v ()(
(13)
To perform the adversarial optimization, we can feed À and v (or in practice, stochastic approxi- mation to them) as the gradients into any gradient-based optimizer designed for minimization, e.g. stochastic gradient descent or ADAM. There is a simple generalization of the above training proce- dure, which is to base the generator gradients on Ey but the critic gradients on Eg for a possibly different function g (Poole et al., 2016, Section 2.3). We refer to this as using hybrid (f, g) gradients. This also approximately minimizes Dy. See &F for more details.
When training classic GANs in practice. an alternative non non-saturating loss is used as the basis for the generator gradient, and is found to perform much better in practice (Goodfellow et al., 2014). This issue has been discussed in detail previously, so we just give a summary here and discuss in more detail in GF. Early in training, the generator and data distribution are typically not well matched, with samples from p being very unlikely under q and vice versa. This means most of the probability mass of p and q is in regions where d has large magnitude, corresponding the positive and negative tails in Figure 2 and (19). In this regime is not too surprising that this might lead to optimization issues. Similar concerns do not apply to other f-divergences such as KL or reverse KL. but an alternative nonn-stuuatiing generator gradient has still been suggested for use in f-GANs (Nowozin et al., 2016). For both GANs and f-GANs the specific change is replace by by a in the definition of in (12). We are not aware of particular motivation for this procedure in the case of f-GANs other than that it yields the traditional non-saturating GAN scheme in the case of Censen-Shamin
has very flat gradient, and
EFFECT OF NON-SATURATING GRADIENTS
We now discuss the effect of the non-saturating generator gradient on training. We show that, for an optimal critic, the non-saturating generator gradient is the gradient of a globally coherent objective function, that this objective function is f-divergence, and that this f-divergence is not the same as the one optimized by using the original "saturating" gradient. We explicitly derive the divergences optimized by the "non-saturating" KI., reverse KL. and Jensen-Shannon training schemes.
We first establish our main result: "Non-saturating" training based on g is precisely equivalent to a hybrid (f.g) 9) scheme for some f. Consider the f-divergence Dy defined by
("(w) = u "u""((
(14)
This valid f-divergence since ("(u) > 0. It is straightforward to verify that b,, ayy so by = ag k, where the constant ls R does not affect the (reparameterized) gradients. Since the non-saturating gradient uses b instead of a in the definition of its generator gradient X, an original generator gradient using f is the same as a non-saturating generator gradient using g. Since the critic gradient is still based on g. the overall scheme a hybrid (f, g) one, and so approximately minimizes Dj-
We now explicitly compute the corresponding f for some common choices of g. is easy show that if g has (R, tails then D, has (R+1, 1) tails, so the divergence effectively optimized by non-saturating training penalizes left mismatches more strongly and right mismatches less strongly than the original divergence. For the KL divergence, g"(u) u- so f"( = We already saw in 84 that this is the reverse KL divergence. Thus "nto-saturating" training based on the KL divergence is hybrid (reverse KL, KL) scheme, and so in fact approximately minimizes the reverse KL. This equivalence also follows directly from the equality of KL's ag to reverse KL's by. For the reverse KL divergence, g"(w) = 2 so ("(w)= = u -3 The corresponding ʃ may be obtained by integrating twice, choosing constants of integration such that f is canonical. We show in @D that Dy is the canonicalized Pearson x" divergence. It has (3, 0) tail weights and is left-unbounded and right-bounded. Thus "non-saturating" training based on the reverse KL divergence is hybrid (fx², reverse KL) scheme, and so approximately minimizes the Pearson x² ddverrencc. For the canonical- ized
divergence, (u) I[ SO f") ([++1]- The corresponding f may
5

---

Under review as a conference paper at ICLR 2020
again be obtained by integrating twice, choosing constants of integration such that f is canonical. We show in $D that this corresponds to (p q) = 4 p 5 p)). This divergence does not have existing name far we aware. In this paper we have termed it the softened reverse KL (SRKI.) divergence (see $C for details on this terminology). It has (d(d) = -2 exp((-d) -2 log ad) and by 22 log a (d). It has (2, 0) tail weights and is left-unbounder and right-bounded. Thus the non-saturating training scheme described by Goodfellow et al. (2014) is a hybrid (SRKL, JS) scheme, and approximately minimizes the softened reverse KL.
Having derived our main result that the typical non-saturating GAN training scheme effectively optimizes the softened reverse KL divergence, we focus on understanding the qualitative properties of this divergence. We do this by developing some analytic tools applicable to any f-divergence.
7 PUSHFORWARDS AND SYMMETRY-PRESERNIE DIVERGENCE PLOTS
While f-divergences unify many divergences, just plotting the function f is often not informative. The symmetric relationship between divergences such as KL and reverse KL is obfuscated, and may grow quickly even when the divergence is well-behaved. In this section we develop a straight- forward and intuitive way to compare f-divergences visually through a cemmetty-prrservinn diver- gence plot. Our perspective also allows simple summary of the prevalence of mismatches between p and g, through a pushforward plot.
Firstly note that for x q(z), p(xr//((r) is a random variable with some distribution. In fact, since (2) is the expected value of some function of this random variable, (p. q) must depend only on the one-dimensional distribution of this random variable and not on the detailed distribution of p and in space. Formally the distribution of this random variable may be described as the pushforward To obtain more intuitive plots, will work in terms of d' (æ) = log p(x) log q(x) instead of u". We denote the density of the pushforward of
measure of q through the function u" (z:) p(x)//(()))
q through d' by ga- (d). Rewriting the expectation in (2), we obtain
Dy(p. 4) = f dd- (exp dd
(15)
As above we can write this more symmetrically. Define
[(exp d), SR (exp(-d)),
d<0 >>0
(16)
By considering expectations of an arbitrary function of d expressed in x-space and d-space, we can show that
qa- (d) Pa. (d) cxp(-d) Thus, using (3) and (17), we can write the f-divergence
(17)
f qa. (d)s + / pa (d)s
Dy(p.g)
(18)
max (d), qa- (d)) sy(d) dd
(19)
An f-divergence Dy(p, q) involves interaction between the distributions p. q and the function f. and (19) nicely decomposes this interaction in terms of something that only depends on p and (the pushforwards) and something that only depends on (the function sy), connected via a one- dimensional integral. By plotting 8 and imagining integrating against various pushforwards, we can see the properties of different f-divergences in a very direct way. By plotting the pushforwards, we can get feel for what types of mismatch between p and are present in multidimensional r-space, and understand at a glance how badly these mismatches would be penalized for given f-divergence.
Examples of pushforwards for the simple case where p and g multidimensional Gaussians with common covariance are shown in Figure 1. In this case the pushforwards qa- and pa- are themselves one-dimensional Gaussians (since d' is linear), with densities N 0 ( o²) and NN o²) 3
spectively, for some a (this follows from (17)). Examples of s for various f-divergences are shown in Figure 2. We refer 8 symmelry-preservine representation of f. Note that long as f is in

---

Under review as a conference paper at ICLR 2020
pushforward of pushforward of Figure 1: Plots of the pushforward densities Pa- (d) and qu (d) for the case where and are multidimensional Gaussians with common variance. The f-divergence for a given / may be obtained by integrating these pushforwards against s in Figure 2 using (19).
KL
reverse KL
Jensen-Shannon
Jeftreys
0.03
0,00
Plots of sy(d) for various f- divergences. The f-divergence for given p and q may be obtained by integrating sy against the pushforwards of p and g such as those shown in Figure 1 using (19). Symmetries such that be- tween KL and reverse KL are evident.
Figure 2:
canonical form, 8f is twice continuously differentiable at zero. Figure directly expresses several facts about divergences. It shows that left mismatches (regions of space where q > p(x). corre- sponding to d < 0) are penalized by reverse KL much more severely than right mismatches (regions of space where q(x) p(z). corresponding d >> 0). The symmetry between KL and reverse KL is evident. We see that Jensen-Shannon and the Jeffreys divergence (the average of KL and reverse KL) are both symmetric how they penalize left and right mismatches, but differ greatly in how much they penalize small versus large mismatches.
Applying the tools developed this section analyze the non-saturating variant of GAN training, Figure shows the symmetry- prenetripp representation sy (d) for the Sensen-Shannor and softened reverse KL divergences, as well as the reverse KL for comparison. The qualitative behavior of softened reverse KL is quite similar to reverse KL. As discussed in SC, softening has the potential to make large right mismatches much less severely penalized, thus making the divergence more mode- seeking. Here softening increases the slope of the left tail and changes the right tail behavior slightly, but these changes are relatively minor modifications. The Vensen-Shamae is extremely different to the reverse KI. and softened reverse KI.
CLASSIFICATION OF F-DIVERGENCE TAILS
this section we introduce a classification scheme for f-divergences in terms their behavior for large left and right mismatches. While different f-divergences differ in details, this classification determines many aspects of their qualitative behavior.
First we define the notion of tail weight and examine some of its consequences. If f"(() Cu as u + for C. > O and f"(u) ~ S-S-3 as u > oc for (Cu Dus-3 tails and (R, S) tail weights. Here we have used the notation g(u) ~ h(u) as Note that, since fx(u) right tail is equivalent fR having u left tail. Thus tail weights interact simply with symmetry: Dy has (R, S) tail weights then Dr, has (S, R) tail weights. Intuitively, the left tail weight R determines how strongly large left mismatches are penalized compared to small mismatches (which are penalized the same amount by every canonical f-divergence), whereas the right tail weight S determines how strongly large right mismatches penalized compared to small mismatches.
O then we say that Dj has
to mean (u))/((u) + as
ff(("" ((-")) having a 2
Some f-divergences such as Jensen-Shaae Shannon are bounded, while others such as KL are unbounded, and it is useful to have characterization of when boundedness occurs. We say Dy is bounded if

---

Tail weights capture these distinctions in a straightforwwrd Under review as a conference paper at ICLR 2020
softened reverse KL
reverse KL
Densen-Shannon
IGOG paper
tail weight
boundedness
divergence
(left, right)
(left, right)
overall
KL
(1, 2)
(0, xc)
x
RKL
(2,
(oo,
00
Cansen-Shanoon
(1,1)
(0, 0)
0
Jeffreys
(2.2)
(oo, 00)
00
Pearson x²
(3,0)
(oo, 0)
00
0 -6
softened RKL
(2,0)
(00, 0)
00
-2
IGOG
(2,0) (00,
00
Figure 3: Plots of sd) for various reverse KL-like f-divergences. Softened reverse KI. is the divergence effectively minimized by non- saturating GAN training. IGOG is the diver- gence derived by Poole et al. (2016).
Table 1: Tail weights and boundedness for the f- divergences considered in this paper. For boundedness, O denotes bounded and oo denotes unbounded. divergence is bounded if and only if left and right tail weights are both less than 2.
there is an M E R such that Dy(p. < M for all densities p and q. We say S is left-bounded is bounded on (0, 1), and right- lighn-bounded if fR is bounded on (0, 1). equivalently if f(u)/u // is bounded on u > From (3) it is casy to see that if f is left-bounded and right-unboundet then we can find
then D,
is bounded. The converse is also true: If f is deft-unbounded and z with arbitrarily large divergence Dy q). This can be seen for example by partitioning IR. into two sets A and B and considering densities p and q which are constant on A and constant B, or strictly speaking smooth approximations thereof. Tail weight determines boundedness. It can be checked by integrating and bounding that a divergence with (R, S) tail weights is left-bounded iff R < 2 and right-bounded iff S < 2. Thus D, is bounded iff R, S < 2. The tail weights and boundedness properties of various f-divergences considered in this paper are summarized in Table Boundedness properties can also be seen in Figure 2. Left and right boundedness of f is trivially equivalent to left and right boundedness of sy. Thus we can see that reverse KI. is left unbounded but right bounded, for example. The unbounded tails in this plot are all asymptotically linear in d.
Tail weights provide an extension of the typical classification of divergences as mode-seeking or covering (Bishop, 2006, Section 10.1.2). Models trained with reverse KL tend to have distributions which are more compact than the true distribution, sometimes only successfully modeling certain modes (density peaks) of multi-modas true distribution. Models trained with KL tend to have distributions which are less compact than the true distribution, "covering" the true distribution en- tirely even if it means putting density in regions which are very unlikely under the true distribution (Bishop, 2006, Figure 10.3). However there are important qualitative aspects of divergence behav- ior that not captured by these labels. For example, Jensen-Shannol covering: It would be more accurate to say that a model trained using Jensen-Shhe -hannnnn tries to match very closely when it matches, but doesn't worry overly about large mismatches in either direction. The Jeffreys divergence is also symmetric and so neither mode-seeking nor covering, but has very different behavior from but precise way.
neither mode-secking nor
Tail weights and boundedness provide an extremely concise way to see the qualitative effect of using the non-saturating variant of GAN training. The softened reverse KI. divergence effectively opti- mized by conventional non-saturating GAN training has tail weights (2, 0), and so is unbounded, is likely to have strong gradients starting from random initialization where large mismatches are present, and penalizes left mismatches strongly but tolerates large right mismatches and so is mode- seeking. In contrast the Kensen-Shannon divergence effectively optimized by saturating GAN train- ing has tail weights (1, 1), and so is bounded, is likely to have weak gradients in the presence of large mismatches, and tolerates large left and right mismatches.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Syed Mumtaz Ali and Samuel D Silvey. A general class of coefficients of divergence of one distri- bution from another. Journal of the Royal Statistical Society: Series (Methodological), 28(1): 131-142, 1966.
Martin Arjovsky and Lon Bottou. Towards principled methods for training generative adversarial networks. In Proc. ICLR, 2017.
Martin Arjovsky, Soumith Chintala, and Lon Bottou. Wasserstein generative adversarial networks. In Proc. ICML, pp. 214-223, 2017.
David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Grae- pel. The mechanies of n-player differentiable games. In Proc. ICML, 2018.
Christopher M Bishop. Puttern recognition and machine learning. Springer, 2006.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Proc. ICLR, 2018.
Sung-Hyuk Cha. Comprehensive survey on distance/similarity measures between probability den- sity functions. International Journal of Mathematical Models and Methods in Applied Sciences, 2007.
William Fedus, Mihaela Rosca, Balaji Lakshminarayanann, Andrew M. Dai, Shakir Mohamed, and Ian Goodfellow. Many paths to equilibrium: GANs do not need to decrease a divergence at every step. In Proc. ICLR, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, pp. 5767-5777, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In Proc. ICLR, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401-4410 2019.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are GANs created equal? A large-scale study. In Advances in neural information processing systems, pp. 700-709, 2018.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of GANs. Advances in Neural Information Processing Systems, pp. 1825-1835, 2017.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for GANs do actually converge? In Proc. ICML, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In Proc. ICLR, 2018.
Vaishnavh Nagarajan and J Zico Kolter. Gradient descent GAN optimization is locally stable. In Advances in Neural Information Processing Systems, pp. 5585-5595, 2017.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(1 (6(17)5587-5581, 2010.

---

Under review as a conference paper at ICLR 2020
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training generative neural sam- plers using variational divergence minimization. In Advances in Neural Information Processing Systems, pp. 271-279, 2016.
Wei Peng, Yuhong Dai, Hui Zhang, and Lizhi Cheng. Training GANs with centripetal acceleration. 2019.
arXiv preprint
Ben Poole, Alexander A Alemi, Jascha Sohl-Dickstein, and Anelia Angelova. Improved generator objectives for GANs. In Proc. NIPS Workshop on Adversarial Training, 2016.
Mark Reid and Robert C Williamson. Information, divergence and risk for binary experiments. Journal of Machine Learning Research, 12(Mar):731-817, 2011.
Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferenc Huszár. Amortised MAP inference for image super-resolution. In Proc. ICLR, 2017.
A FURTHER COMPARISON TO PREVIOUS WORK
As mentioned in 52, Poole al. (2016) present very similar view that presented in this paper, including recognizing that the generator and critic may be trained to optimize different f-divergences and interpreting the classic non-saturating generator gradient hybrid scheme of this form where the generator gradient is based on a new f-divergence (Poole et al., 2016). We now discuss the discrepancy between our result and theirs.
In the language of the present paper, Poole et al. (2016, Equation (8) define the approximation
Dy(p.g) = (()))(((=)/(()))))) E,,()) q, = (()))) de
(20)
and show that the gradients of Ey for this particular / match the non-saturating GAN gradients. Ep(p, q. d') for the optimal critic
This valid approximation of the value, since Dy d(x) d' dz(x)) log p(z) log q(z). However the gradients are not the same: The partial derivative of the left side of (20) with respect to the parameters q involves two terms, one for each occurrence
of ((x) in the integrand, and the partial derivative of the right side only includes one of these. Thus
it is not the case that optimizing F. using gradient descent (while continually keeping the critical optimal) optimizes Dj.
B PROPERTIES OF F-DIVERGENCES
In this section we go into more detail about some of the properties of f-divergences which were briefly covered in the main text.
Firstly note that Dy is linear f. that is Dy+g = + and Day = kD, where k > 0. If f(1 0 and s(1) 0 then (f (1) Secondly note that adding an affine term to f(u) does not affect Dy: If g(u) for Is e then Dy = Dy. Any affine term added must be of the form k ku in order to respect the f(1) 0 constraint. Thus the second derivative f" determines the divergence completely. This property is also true the various bounds and finite sample approximations" derived in this paper, so we may legitimately consider f" rather than f as the essential quantity of interest for given divergence. Working with f" has the added advantage that for many common f-divergences f" has simpler algebraic form than f. For any densities and q we have Dy q) > 0 with equality iff p = 4. as can be seen by plugging the constant function u(x)) = into (5). If f'(1) = 0 and (1) then we say f is in canonical form. We can put any in canonical form by scaling
and (kf) = so Di+g and D& are valid f-divergences.
f(u) + k ku
and adding a suitable affine term, and this corresponds to a scaling of Dy. Each f-divergence has unique canonical form.
As long as the reparameterization trick is used to compute finite sample approximations, is standard practice. If simpler finite sample approximation such as naive REINFORCE is used then & affects the variance of the generator gradient.
10

---

Under review as a conference paper at ICLR 2020
Different E-divergences may behave very differently when p and q are far apart but essentially identical when @ = p. In fact the divergence between nearby distributions belonging to some family is given by (1) times the Fisher metric of the family. Specifically
Dy (1x. (A+eo)= f² (e F(N) +
(21)
where E E R. U E TRK and F(A) = E. (x) ax log ( ax log (x(x)) is the Fisher infor- mation matrix for the parametric family of distributions specified by qa- Alternatively this may be stated in the non-parametric form
/
(u(x)) de + O(s²)
D, (a, q ev) = 3 ==
(22)
g(x)
where ; RK - satisfies f u(x) dr O. Informally may state this as:
Dy(p.9) 21((1)) (p(x)- q(()) q(x))²
dr
(23)
p(z)
Thus all f-divergences agree up to a constant factor on the divergence between two nearby distribu- tions, and they are all just scaled versions of the Fisher metric in this regime. This can also be seen in Figure 2, where all f-divergences approximately overlap near zero.
C DIVERGENCE SYMMETRIZATION AND SOFTENING
We apply simple operations to divergence to obtain another divergence. In this section we consider the effect of reversing, symmetrizing and softening operations on f-divergences. Many common f-divergences can be obtained from others in this way, and this provides unified way of concisely describing many f-divergences based on KI,, for example.
Consider applying an operation to a divergence D(p, g) to obtain another divergence D(p, q). We already saw the reversing operation D(p, 4) = D(g. p) in $3. If D is an f-divergence In this case
uf(u with function f(u) then DR is f-divergence with function fR(u) 3 p). D is f- Symmetrizationn means D(p.q) 3 D(p + 1
(u) u 3 ff
divergence then f + 1 f enacts symmetrization. Finally (q-) jsoftenennng refers to replacing with m = + $q, i.c D(p, q) 4p m). If D is an f-divergence with function f then setting the new f(u) to be 2(1+ T+x
enacts softening. In this case the new f"(u) is f" (224). The factor of above to ensure that the divergence remains canonical after softening, i.e. (1( 1. Softening has the potential make large right mismatches much less severely penalized, since in gions space where p(x)/q(z) was large because p(z) was moderate and g(z) was tiny, p(=) now approximately 2, so large right mismatch is only penalized by the softened divergence as much as a moderate right mismatch is penalized by the original divergence. This is reflected in the tail weights: It is easy to show using the tools we have developed above that if the original divergence has (R, S) tail weights then the softened divergence has (R, 0) tail weights.
2x
Many f-divergences verse KI. is Reverse(KL.. Jeffreys is Symmetrize(KL), the canonicalized K-divergence 4 KT(p1 | m) (Cha, 2007) is Soften(KL) and canonicalized Jensen-Shannon SynmetrizeSSttennKLL)) In this terminology, the main claim of this paper is that the non-saturating procedure for GAN train- ing is fact effectively minimizing the softened reverse KL divergence 4 [L p)) given by Soften(Reverse(KL))
be written concisely
series of these operations. For example
D DETAILED EXPRESSIONS FOR VARIOUS F-DIVERGENCES
In this section we give more details of the f-divergences considered $4 and 86. The expressions for Dy and F, obtained by plugging the chosen f into (2) and (7) respectively.
11

---

Under review as a conference paper at ICLR 2020
f(u)
3
Tangent line
2
1
0
-1
0.5
1.5
2
u
Figure 4: A strictly convex function f R250 R and a tangent line. The variational bound used by f-GANs is based on the fact that strictly convex function f lies at or above its tangent lines.
The KL divergence satisfies:
f(u) ulog u
(24)
("(u) =1
(25)
Dy(p, ②) KL(pll = / p(xx) log p(x) dr
(26)
g(x)
E = 1 + / p(z))(())dr ((x)) exp(((x)) dr
(27)
(d = d (d) exp(d) -
(28)
(29)
The KL divergence has (u (u-1,,-1) tails, (1, 2) tail weights, and is left-bounded and right- unbounded.
The reverse KL divergence satisfies:
f(u) = log
(30)
("(u)= u2
(31)
Dy(p, q) = KL(9 [1p) = ((x)) log ((x)) dr
(32)
p(x)
Ey = / (xteexp((d(1)) dx - / ((x))(()) do
(33)
ay(d) exp(-d) by(d) =
(34)
(35)
The reverse KL divergence has (u 2u-2,,00 2,24 2) tails, (2, 1) tail weights, and left-unbounded and right- bounded.
12

---

Under review as a conference paper at ICLR 2020
The Jensen-Shannor divergence JS(p, q) has (1) 1/4 and so is not canonical. In most of the paper we therefore consider the canonicalized
divergence 4JS(p, q). This satisfies:
f(u) 2u log 2(u + 1) log(u u- 1) 4
(36)
("(u) = ulu +1)
(37)
Dy(r, = 4JS(p, ①))
(38)
=2KT.(p) pppt + }4) + 2T +p }1)
(39)
p(z)
g(x)
= =4oog +
log)
log (tx) + g(x) dx + 2 g(>) log
p(x) ((x) dx
(40)
g,(p.g.d) 4 2 2 2 ( dx + 2 logo dr
(41)
ad(d) = 2 log o (d) + 2og
(42)
(dd -2 log (-d) 2og
(43)
The canonicalized Jensen-Shanne divergence has (2u-1, 1 2u 2) tails, (1,1) tail weights, and is both left-bounded and right-bounded and so bounded overall.
The Pearson x2 (or Kagan) divergence has (1) 2 so is not canonical. The canonicalized Pearson X² divergence satisfies:
f(u) (u 1)²
(44)
2u
f"(u)= =
(45)
/ / [((ee) - p(x)) 2
Dy(p,q) =
p(1) da
(46)
Ey(p.q.d) = - - p(x) exp (-2d(x)) dr + q(x))xp) (-d(x)) dr (47)
af(d) = 1 eeep(-20
(48)
by(d) = exp(-d)
(49)
The canonicalized Pearson X² divergence has (u (u-3, 3,=",=00 3) tails, (3, 0) tail weights, and is left- unbounded and right-bounded. The expression for f here corrects a swapped definition the origi- nal f-GAN paper" (according to the definitions of the Pearson and Neyman divergences given in the paper, the expression given for the Pearson f is actually the Neyman f and vice versa) (Nowozin al., 2016). In 56 we discussed the equality of the non-saturating reverse KL generator gradient to the conventional canonicalized Pearson x² generator gradient. This can be seen from (14), as we did in $6, or directly by noting that ay for the reverse KL divergence equal to by for the Pearson x2 divergence.
The softened reverse KL divergence satisfies:
f(u) = 2(u + 1) log u u+1
log 2
(50)
2 f"(u)= (2(++)
(51)
Dy(p,g) = 4KL + t p)
(52)
E,(p.q.d) 4log 2 + p(z) [exp (-d(x)) logo (d(z)) dr
2 q(()) log (d(x)) dx
(53)
(d exp(-d) log (d) 2 2og
bd(d) = 2 log (d) + 2 log
'In the the arxiv preprint, the final NIPS version of the paper.
13

---

Under review as a conference paper at ICLR 2020
The SRKL divergence has (2u 2u 3) tails, (2, 0) tail weights, and is left-unbounded and right- bounded. In 86 we discussed the equality of the non saturatiu canonicalized Jensen-Shannon gen- erator gradient to the conventional softened reverse KL generator gradient. This can be seen from (14), as we did in $6, or directly by noting that af for the canonicalized Jensen-Shannon divergence is equal to by for the softened reverse KI. divergence.
E F-GAN NOTATION
The original f-GAN paper (Nowozin et al., 2016) phrases the results presented in $4 in terms of the Legendre transform I of S. The two descriptions T(x) ("((z)) and using the result f f*(((u)) = uf f(u). We find our description helpful since it avoids having to explicitly match the domain of f ensures the optimal d is the same for all f-divergences, and because the Legendre transform is complicated for one of the divergences we consider. An "output activation" was used in the original f-GAN paper to adapt the output d of the neural net to the domain of f*. This is equal to (exx up irrelevant additive constants, for all the divergences we consider, and so our description also matches the original description this respect.
equivalent, as can be seen by setting
F MORE VARIATIONAL DIVERGENCE MINIMIZATION
The gradient matching property shows that performing very many critic updates followed by a single generator update is sensible learning strategy which, assuming the critic is sufficiently flexible and amenable to optimization, essentially performs very slow gradient-based optimization on the true divergence D, with respect A. However in practice performing a few critic updates for each generator update, or simultaneous generator and critic updates, performs well, and it is easy to see that these approaches optima of D,, subject as always to the assumption that the critic is sufficiently richly parameterized. Convergence properties of these schemes are investigated much more thoroughly elsewhere, for example (Nagarajan & Kolter, 2017; Gulrajani et al., 2017: Mescheder et al., 2017; 2018; Balduzzi al., 2018; Peng et al., 2019), and are not the main focus here.
least have the correct fixed points terms of Nash equílibria of E, and
A similar discussion applies to hybrid schemes. Subject as always to the assumption of a richly parameterized critic, if we perform very many critic updates for each generator update, then the used to compute the generator gradient will still be close to d",, and so the generator gradient will be close to the gradient of Dy, even though the path took to approach d' was governed by y rather than f. The fixed points of the two gradients also still correct, and so it seems reasonable again use more general update schemes and we might hope for similar convergence results (not analyzed here).
For an implicit generative model (2) where 2 ~ P()), we have
(p,,, (())b, (dx(xxx))) dz
(54)
Thus there (d) factor in the generator gradient, and in fact this the only way the choice of f-divergence affects the generator gradient. For reverse KL, (d( 1, allowing the gradients from the other factors to pass freely. Most of the contribution to the initial gradient for reverse KL is likely come from regions in space with large negative d due the P(z) factor. For canonicalized Jensen-Shannon, by (d) (d)),, which tends to zero exponentially quickly as - - and tends to as d + oc. Regions of space with large positive d have a tiny contribution to the gradient due to the (z)) factor, while regions with large negative are exponentially suppressed by By (d). Based on these considerations it might be tempting conclude that left- untoundeddesss is the most important factor in being able to learn from a random initialization. A divergence with left tail weight R has (d) exp(-d(F2 R 2)) R. 2 2 ensures that by (d) does not decay exponentially d -co. However the case of KL shows that right-unboundedness is also capable of allowing learning. For KL, (d) = exp d, and the situation is complicated, since it exponentially magnifies gradients from regions with large positive d, which are extremely unlikely under P22). We know the overall gradient can sometimes be a reasonable learning signal, since training models such multivariate
14

---

Under review as a conference paper at ICLR 2020
GAN divergence
AG divergence
1.8
-0.000
0.240
1.8
0.840
0.780
0,720
0.660-
0.540
0,200
0.480
0,600
1.6
1.6
0.420
0.240
Sigma
1.4
sigma
14
00,30
1.2
1.2
0,160
1.0
1.0
0. 0.180
0.220
0,240
0.260
0.300
0.32
-0.400-
-0.3
0.240
1.0
1.2
1.4
1.6
1.8
2.0
1.0
1.2
1.4
1.6
1.8
2.0
mu
mu
Figure 5: Comparing training using the saturating and non-saturating GAN generator gradients on a toy problem. The true distribution p a mixture of 1D Gaussians and the model distribution q is single Gaussian. Contour plots show the Jensen-Shannon (JS) divergence (left), and softened reverse KL divergence KKL + &g p) (right) as function model parameters. Lines show the progression of SGD- based JS training based the original, saturating gradient and based the non-saturating gradient (solid for learned critic; dotted for optimal critic). The original scheme converges the IS divergence minimum. The non-saturating scheme, which by the results of this paper is equivalent to hybrid (SRKI., JS) scheme, converges to the SRKI. divergence minimum as expected.
Gaussian using KL divergence works well. However even the expected gradient allows learning, the stochastic approximation obtained by sampling from q likely have extremely large variance.
The saturation issue sometimes presented as being specific to the loss E used for classic GAN training. but the gradient matching property presented in $5 shows it is fundamental to the Jensen- Shannon divergence. The more critic updates we perform initially, the more saturated d is on samples from g, and the true divergence Dj-
closely the gradient of F, with respect approximates the gradient of the
The typical fix the saturation issue is to the non-saturating generator gradient
/ ax (x) log (d(x)) dr P(())
Ô
T=
ax log (d(z)()))) dr (55)
Since the gradient of log (d) tends to 1 tends -oo, the gradient used for training is now larger.
G EXPERIMENTAL VALIDATION OF MATHEMATICAL RESULT
In order to validate our mathematical conclusions we conducted a simple experiment. Training behavior using the original and non-saturating gradients a toy problem is shown in Figure 5. We see that the two cases minimize different divergences, as expected based on the theoretical arguments presented above.
15


Reviews:
[{"review_id": "SkeyqBiynS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study the ‘non-saturating’ variant of training for GANs and show that it is equivalent to a regular training procedure minimizing a “softened” reverse KL divergence as opposed to Jensen-Shannon divergence. They show a connection between the two training procedures for more general f-divergence losses. For instance, they show that\n1. non-saturating training on original GAN loss minimizes KL(p/2+q/2 || p) \n2. non-saturating training on KL(p||q) loss minimizes KL(q||p)\n\nThe authors start by arguing about previous analyses of the non-saturating training scheme and present why they do not arrive at the complete picture. Then they go on to introduce f-divergences and f-GAN training before explaining in their notation what precisely non-saturating training means. Then they show that it corresponds to minimization of not the original f-divergence but a hybrid (f,g) divergence. \n\nOverall the paper presents most of its insights as speculative statements and does not do a good enough job of attempting to concretely formalize them.\n\nQuestions:\n1. Why is argument of Nowozin et al wrong when q is parametric?\n2. What are tail weights of an f-divergence?\n3. Unclear why one needs the variational lower bound in f-GAN training.\n4. Why does hybrid training (f,g) converge to the minimization of f? Can the discussion in Appendix F be formalized into a Theorem or a Lemma?\n5. One of the main takeaways of the paper appears to be that initial phases of JS divergence minimization leads to flat gradients which can be problematic - question: is this an artifact of JS being bounded? Could unbounded divergences avoid running into this issue?\n\nThe content organization and highlighting of the main result in the paper can be significantly improved. Since the paper is exposing a theoretical connection as its primary result, I would also recommend a higher level of formalism overall.\n1. Figure 2 is references much earlier than it appears\n2. Tail weights are referenced before they are defined\n3. Formal statement of non-saturating gradient based training captured within a subsection or box. Hard to locate in current draft. Need to read 5 sections to get to it.\n4. Appendix C should be in main body.\n5. Some definitions need to be defined more formally. For instance,\n  a. Hybrid optimization\n  b. Non-saturating training\n\nIt is unclear how significant the contribution of the paper is. It is a clean mathematical observation but the consequences of the connection are not explored and fleshed out. For instance, can by realizing that the non-saturating gradient training is optimizing a different f-divergence can we explain why non-saturating gradient training is more useful? Any insights backed by some simple example settings of synthesized probability distributions would also be useful. The paper also does not propose any new training methods based on the insights uncovered and it is not clear how significant the connection uncovered is with the information presented in the current draft. I believe it is an interesting direction that the paper probes and with a more deeper look into the phenomenon and related directions can be ready for publishing in a venue such as ICLR. In it’s current form I unfortunately don’t think the contributions of the paper are significant enough for acceptance.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJgXA_V2FB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, firstly, a useful expression of the class of f-divergences is proposed. The authors investigate theoretical properties of some popular f-divergences from newly developed tools. Then, the expression is used to investigate GANs with the non-saturating training scheme.\n\nThe expressions in Section 3 can be a useful tool to investigate the statistical properties of estimators with f-divergences. However, I think that the usefulness of alternative expressions in Section 3 is not very clear, though an intuitive interpretation is presented. Moreover, the numerical experiment is not sufficient to support some new expressions. Detailed theoretical analysis of the non-saturating training based on the proposed expression would be required for publication from ICLR.  In addition, the authors could shorten the paper within 8 pages that is the standard length of ICLR paper. In the paper \"On topological properties of f-divergences\" (1967), Csiszar intensively studied non-saturating properties of f-divergences\nin the paper. It would be helpful for readers to add some comments on the relation between the theoretical development in this paper and Csiszar's paper.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "BJezD3zRtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "######## Updated Review #########\n\nI would like to thank the author(s) for their rebuttal, which I have carefully read. I also appreciate the effort made to improve the paper. My overall evaluation of the paper stands unchanged. \n\n\n\n#############################\n\n\n\n\n\n#### My review is based on the updated paper downloaded from the anonymous link. ####\n\nThis paper discusses alternative training strategies for f-GANs. While the discussion has some interesting points, the presentation needs to be much improved. It is not easy to follow this paper in its current form, and the main results are not properly emphasized. As such, I am not certain of its real contribution. f-GANs are not routinely used in practice (except for the vanilla JSD and RKL), and as far as I can tell the saturating gradient issue is no longer a central concern (it has been well addressed years ago, with, e.g. WGANs). I am voting to reject this submission, but I am willing to re-evaluate this paper if the author(s) significantly improves their writing. \n\n- In Section 2, what does it mean by \"the Fisher metric of the family\"? The concept of Fisher metric is defined anywhere in the text, and there is no reference to it.\n- It is not is intuitive why the second derivative of f_R takes the form $u^{-3} f''(u^{-1})$ (given above Eqn (2)). Please elaborate. \n- Please avoid the use of subjective phrases such as \"imagining\", \"get a feel\", etc. I am guessing the author(s) are trying to suggest taking a visual inspection of the discrepancies projected on the log-likelihood ratio axis (which is 1D) and figure out which f-div might be more appropriate. \n- Fig. 3 needs legends. There are two solid (dotted, resp) lines in the Figure, and I am guessing one of them is for the saturating and the other for the non-saturating gradient. This needs to be specified because the line specs are identical. \n- After going through the entire paper, I would highly recommend the author(s) to take a course in academic writing. The main contributions are not highlighted and some of the key concepts are not even properly defined. For example, analysis of the non-saturating gradient, which is supposedly the main result of this submission, appeared in pp. 7, by which time most readers have exhausted their patience. The vanilla version of the non-saturating scheme never appeared in the main text. The notation system is also non-standard, where the notation $\\bar{\\lambda}$, normally reserved for average/expectation, has been used to denote the gradient wrt $\\lambda$. The writing can be very unprofessional at times, for example, \"the answers to these questions are yes, yes and no respectively\".  \n- I do not know why the author(s) inserted one toy experiment in the paper, as it serves no purpose. Each model converges to their respectively optimal, as expected. There is no discussion of how to choose an appropriate f-div or (f,g) hybrid in practice. \n- I totally agree with the author(s) on the point that this is a note (sec 7, second paragraph), rather than an academic paper. There is a lot of derivations in the paper, but insightful discussions are limited in the sense that it is not clear how to connect these results to improve practice. The section titles also read like bullet points in a note. \n- Not certain about the significance of this paper. The derivations are fairly standard and I can not find any useful proposals that might benefit the practice of f-GAN training. There is a number of papers discussing non-saturating GAN training (see sec 8), and I do not think this paper adds too much value to this discussion. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: As the reviewers point out, the core contribution might be potentially important but the current execution of the paper makes it difficult to gauge this importance. In the light of this, this paper does not seem ready for appearance in a conference like ICLR.
Decision: Reject