Adversarial Problems for Generative Networks
Authors: Kalliopi Basioti
George V. Moustakides
Venue:  (2021)
Keywords: generative networks
adversarial generative networks

Abstract: We are interested in the design of generative networks. The training of these mathematical structures is mostly performed with the help of adversarial (min-max) optimization problems. We propose a simple methodology for constructing such problems assuring, at the same time, consistency of the corresponding solution. We give characteristic examples developed by our method, some of which can be recognized from other applications and some are introduced here for the first time. We compare various possibilities by applying them to well known datasets using neural networks of different configurations and sizes.

Under review as a conference paper at ICLR 2021
ADVERSARIAL PROBLEMS FOR GENERATIVE NET- WORKS
Anonymous authors Paper under double-blind review
ABSTRACT
We are interested in the design of generative networks. The training of these math- ematical structures is mostly performed with the help of adversarial (min-max) optimization problems. We propose simple methodology for constructing such problems assuring, at the same time, consistency of the corresponding solution. We give characteristic examples developed by our method, some of which can be recognized from other applications and some are introduced here for the first time. We compare various possibilities by applying them well known datasets using neural networks of different configurations and sizes.
INTRODUCTION
The problem we are interested in, can be summarized as follows: We are given two collections of training data {zj} and {x:}. In the first set the samples follow the origin probability density hzz) and in the second the target density fxx). The target density fxx is considered unknown while h(z) can either be known with the possibility to produce samples Zj every time it is necessary or unknown which case we have a second fixed training set [2j]- Our goal is to design determin- G(Z)
istic transformation ((z) so that the data {y,} produced by applying the transformation onto {z,} follow the target density fly).
Of course one may wonder whether the proposed problem enjoys any solution, namely, whether there indeed exists transformation G(z) capable of transforming z into y with the former following the origin density l(z)) and the latter the target density f(y). The problem of transforming random vectors has been analyzed in Box & Cox (1944) where existence is shown under general conditions. Computing, however, the actual transformation is completely different challenge with of the possible solutions relying on adversarial approaches applied to neural networks.
The most well known usage of this result is the possibility generate synthetic data that follow the unknown target density fxx). In this case hzz) is selected to be simple (e.g. i.i.d. standard Gaussian or i.i.d. uniform) so that generating realizations from h(z) is straightforward. As mentioned, the adversarial approach can be applied even if the origin density l((z) is unknown provided that we have a dataset { with data following the origin density.
It was Goodfellow et al. (2014) that first introduced the idea of adversarial (min-max) optimiza- tion and demonstrated that it results in the determination of the desired transformation G(z) (con- sistency). ). Alternative adversarial approaches were subsequently suggested by Martin Arjovsky & Bottou 2017); Binkkwski et 2018) and shown to also deliver the correct transformation G(z).
We must mention the work of Nowozin et al. (2066) in which class of max optimizations, f-GANs, was defined design generatoor/disermmnatto pairs. Then, Liu et (2017) defined the adversarial divergences class of objective function which further combined f-GANs, MMD-GAN (Li et al. 2017). WGAN, WGAN-GP (Gulrajani et al., (207)). and entropic regularized optimal transport problems. Also, they investigated under what conditions the discriminator's class has the effect of matching generalized moments. Next, the work of [Song & Ermon (2019) connected GANs and Wasserstein GANs (WGANs) (Martin Arjovsky & Bottou| |ottou|[2017), and later Birrell et al. 2020) generalized the results by introducing the diverrgneess which allowed to bridge f- divergencies and integral probability metrics.
Our class of generative adversarial problems establishes a one-to-one correspondence with f-gans under the ideal (non data-driven) setup. However, we believe that approach enjoys certain signif-

---

Under review as a conference paper at ICLR 2021
icant advantages: First, the definition of the two functions Φ(2), (b(2) in Equ. equation 8 is straight- forward while Nowozin et al.] (2016) requires to solve an additional optimization problem for the derivation of each GAN loss. An additional benefit of our approach is the complete control over the result of the maximization problem that defines the discriminator. In other words, we can de- cide what function, the discriminator must estimate. In Nowozin et al.] (20T6) such flexibility does not exist. This is important because we can select the approximation function properly so that we avoid the need to impose difficult constraints on the discriminator output (e.g. positivity) since such constraints tend to seriously affect the approximation quality of the corresponding neural network. Further, there is no need for the discriminator be Lipschitz function, as WGAN or WGAN-GP something that needs extra operations to ensure that the discriminator is Lipschitz.
transfor- Furthermore, we will show that the function the discriminator tries to approximate mation of the likelihood ratio r(x)= = g(x)/ and there are important applications in Statistics where one is interested in estimating only the transformation of the likelihood ratio with the most common cases being the likelihood ratio itself, its logarithm (log- (log-likelihood ratio), or the ratio r(x) which plays the role of the posterior probability between two densities. In other words, there are applications where one is interested only in the "max" part of the min-max problem.
[+r(x]
Finally, because we know what transformation of the likelihood function the discriminator tries to approximate, is possible to compare the different GANs on how closely they reach the optimal value of the likelihood ratio rxx meaning (x) g(x).
As in Nowozin et al. (0116), we will show that our methods provides an abundance of adversarial problems that are capable of identifying the appropriate transformation G(z). Furthermore, we will also provide a simple recipe as to how we can successfully construct such problems.
Arguing along the same lines of the existing min-max formulations: We would like to optimally specify a vector transformation G(z), the generator, and a scalar function D(x), the discriminator. To achieve this, for each combination {G(z), D(x)) we define the cost function
J(G, D) E [ [d(((x))] + Ez-hh (+((DG(()))
where o(=), 1=() are two scalar functions of the scalar and Ex-s(:), B.-a(]) denote expec- tation with respect to the density f((x), li(z) respectively. The optimum combination genera- tor/discriminator is then identified by solving the following min-max problem
min max J(G, D) = min max [Exx)(((D(x)] + [e G(z) D(x)
(2)
G(z) D(x)
We must point out that our goal is not solve equation 2 but rather find a class of functions o(2), (z)) so that the transformation G(z) that will come out of the solution of equation 2 is such that y follows the target density f(y) when z follows the origin density h(z).
If z is random following fizz) then = G(zz is also random and we denote with g(y) its corre- sponding probability density. Clearly, there exists correspondence between transformations G(z) and densities g when the density hzz) of z is fixed. Since we can write
Ezznh (D((())) = Ey~a | (D(y)),,
this allows us to argue that the min-max problem equation 2 is equivalent to
min max (Bx- [ (x())) + Ry-- [b (D(y))}
gly) D(x)
is now possible to combine the two expectations by applying a change of measure and a change of variables and equivalently write equation 3 as follows:
min max [Ex~/ (okes(o@DD) [D((x)) + (D(()) g(z)
f())dx
s(y) D(x)
f(x)
min max (Ex-5 (D(x)) + E [x(x) (D(x))]} g(x) D(x)
min max E / (D(x))) + +x(x)x
9(x) D(x)
where r(x) s(x)/[(xx) denotes the corresponding likelihood ratio. Since f(x) is also fixed, there again a correspondence between r(x) and (x)., hence the previous min-max problem becomes equivalent to
min max Ex 0 o (D(x)) + (Dx)))
(4)
rx))LL D(x)
2

---

Under review as a conference paper at ICLR 2021
Here ILy denotes the class of all likelihood ratios r(x) with respect to the density f(x), namely, all the functions r(x) that satisfy
L = r(xx) r(x) 20, [(xx)
(5)
Using these definitions, let us define the cost
J(r, D) = Ex- [x)) + +x(x)x (D(x)]
(6)
and, according equation we are interested the following min-max problem
min max J(r D).
(7)
x(x) P(x)
As mentioned, actual goal is not solve the adversarial problem. Instead, we would like to properly identify pairs of functions {Ø(z), { so that equation 7 accepts as solution the function
r(X) Indeed, if (X) = is the solution equation 7 this means that g(x) f(x is the
solution to equation 3 and, finally, that the optimum G(x) obtained from equation D is such that
y G(x) follows gly) = f(y) which, of course, is our original objective. Even though the min- max problem in equation
[ iss what we attempt to solve, it is through equation 7 that we understand propose simple design method (recipe) for the two functions 6(2), (2) that assures that the solution of equation indeed r(x) = 1. Before we discuss the details of our work, we would like to summarize this paper's contribution.
what its solution entails. In the next section we focus on equation [ equation
We design a family of GANs problems using a likelihood ratio approach. this class, all optimization problems have the desired property that the generator output follows the target distribution of the random vector of interest, x, in other words, that the likelihood ratio of the two distributions is equal to one.
We propose a straightforward recipe explore the GANs family. With this methodology, we were able to identify subgroups characterized by specific transformations of the likeli- hood ratio. In these subgroups, we discovered novel objective functions and classified to them previously introduced GANs (such as Wasserstein, Cross-Entropy GANs).
We propose new online metric for evaluating the performance of our generative model during training.
Our experiments provide insights for the behavior of the different GANs objective func- tions, with some of the novel objective functions performing better than the already known GANs.
2 A CLASS OF FUNCTIONS 6(2), (2)
Suppose that (r) a strictly increasing and (left and right) differentiable scalar function of the nonnegative scalar r, i.e. r E [0, oo). Denote with J., w ([0, oo)) the range of values of w()) and let - (z) be the inverse function of w()) which exists and is defined for Ja Let p(=) > be positive scalar function also defined for E Iu then, using w(r) and p(z), we propose the following pair d(2), (2) (2)
(8) where denotes derivative. Since w(r and p(z) are arbitrary (provided they satisfy the strict increase and positivity constraint respectively). the class pairs defined by equation i very rich allowing for multitude of choices. We show next that any such pair (œ(22, (=)) gives rise min-max problem, as in equation 7 that accepts r(x) = 1 as its unique solution. We prove this claim in two steps. The first, involves thcorem where we consider simplified version of the min-max problem.
= -w))==-
('x)) = p(z),
Theorem Let w(r), ((2). 1822) and Jes be defined as above with the additional constraint (w(1)) 0. Fix > 0 and consider ((D) + rD) as function of the scalar D. Then, for any D E Iw. we have that
(DD) + (D) s ((w(r)) (w(()))),
(9)
3

---

Under review as a conference paper at ICLR 2021
with equality if and only if w(r).
Consider next the minimization with respect to r of the maximal value in equation It is then true that
min ( +rb((r))) o(((1))
(10)
r00
with equality if and only ifr
consequence of Theorem] is the next corollary, which constitutes the second and final step in proving that the adversarial problem defined in equation 7 has as unique solution the function r(X)
Corollary If the functions Φ(2). xx(2 satisfy equation 8 and w(r) is strictly increasing and left and right differentiable, then in the adversarial problem defined equation 7 the maximizer D(X) wr(()) and the minimizer is r(X) = while the resulting min-max value is equal to
min max Ex~, [x(D()) +r(X))++++)))))))) =(((()) (w(()). r(X)eL, D(x)
(11)
2.1 SUBCLASSSES OF THE GANS FAMILY
Let us now present some of the subclasses of the GANs family, where each subclass character- ized by the type of transformation of the likelihood ratio, wr)). Once we fix w()) and give pairs (ó(2), {22) that satisfy equation g we are able to "pick" objective functions laying in the w(r) subregion.
Subclass A: (()) r The first examined subclass the simplest one, consisting just powers of the likelihood ratio. To the best of our knowledge, this is the first work proposing objective functions from this class. To find the pairs {Ø(z), {b(2) we proceed as follows.
We have that w (z) za and Jlw [0, oo). According to equation 8 for E [0, oo) must a Some examples of this subelass
define (=) p(=), (²))) p(z). Also
presented in Table
For the particular selection (r) (corresponding is equivalent to the Bregman cost (Bregman 1967). In fact there is one-to-one correspondence between our p(z) function and the function that defines the Bregman cost. This correspondence however is lost once we switch to different a or different w(r) function, suggesting that the proposed class of pairs (Ø(2), (2)), far richer than the class induced by the Bregman cost.
1) we can show that the resulting cost
Table 1: Subclass optimization problems for GANs
GAN
((=)
(=)
J
Ala
log(z) -2
[,, 00)
Alb
log(z)
00, 00)
A2
-(1+2)
- -(1+2-1)
0,000
A3
-log(1 + 2)
log(1 +1-1)
00,00
MSE
-0.52
z
[0)
Subclass B: (()) = a logr This subelass considers one of the most popular transformations of the likelihood ratio, the log-likelihood ratio. As with subclass for the first time, the next exam- ples presented. They can be used either under min-max setting, for the determination of the generator/discrimnaato pair, or under a pure maximization setting for the direct estimation of the log-likelihood ratio function log rxx).
We have w -2) = ex and Jw = R. As before p(z) must be strictly positive and, according to must define d'(z) -e== p(2), (²(2)) = p(2). And the likelihood ratio In Table 2 we can see some examples of this subclass.
equation 8 for all real
is r
Subclass C: wr)) - As already mentioned, this is another important transform of the likeli-
hood ratio. Interestingly, in this subclass belongs the first introduced GAN (Goodfellow et al. 2014) the Cross Entropy GAN.
4

---

Under review as a conference paper at ICLR 2021
Table 2: Subelass B optimization problems for GANs
GAN
6(2)
(1(2)
Jw
Bla
B1b
R
Exponential
- -.
0.5s
I
B2
logg(1 e²)
log(1 +e=)
R
When w(r) = FII we have w (=) = - and J = [0. 11. For p(=) > 0, 2 E [0, we must
define the functions ( (²) according to equation 8 of (=) = -==) (2) p(z). In this
case the likelihood ratio is r = D-D- Table 3 we see the cross entropy GAN and C2 which presented here for the first time.
Table 3: Subelass C optimization problems for GANs
GAN
(x))
Iw 0, 1
Cross Entropy C2
log(1 1 2) + logg1 -
log(z)
0, 1]
Subclass D: ((r) sign(logr) This is a special case of (() with the corresponding function not being strictly increasing. It turns out that we can still come up with optimization problems, two of which are known and used in practice, by considering w(r) as a limit of a sequence of strictly increasing functions.
Monotone Loss: As a first approximation we propose sign(z) tanh(52) where > parameter. We note that lim tanh fz) = sign(2). Using this approximation we can write
sign(logr) x tanh ( logr)
r r
= w()).
(12)
As we mentioned, we have exact equality for c + 0o. Let us perform our analysis by assuming that e finite. We note that w (z) (1 and Ia = [-1,,]] Consequently, p(=) for
E [1, we must define d'(=)
1-2 p(=), t' p(=).
DI) If we let e + oo in order to converge to the desired sign function, this yields d'(x) = -p(2) and (z) = p(z). This suggests that 6(2) = f p(x) )dx is decreasing and (b2)) = p(z)dz =
-Ø(2)) 2) is increasing. In fact any strictly increasing function (=2) can be adopted provided select ((2) - (2))
There is popular combination that falls under Case D1). In particular, the selection (=)) -+(=) reminds us of Wasserstein GAN Martin Arjovsky & Bottou (2017), with two differences, in our case should lie in [-1, 1] and the discriminator is not constrained to be Lipschitz function.
Hinge Loss: As second approximation we use the expression sign(2) <ign(>))=|#,, c > O, which is strictly increasing, continuous and converges to sgn(2) as c - co. This suggests that
sign(logr) = sign(log r) logrl = wr),
(13)
and (=) e* Since (r) can assume any real value we conclude that J. = R which, clearly, differs from the previous approximation where we had J.m = [-1, If p(z) > 0, E R then, according to equation we must define d'(2) - p(=), v'(2)) p(=). We present the following case that leads to a very well known pair from completely different application.
D2) If we select & (z) = p(2)) {e- +1 > 0 then (z()) - (e- +1 J. If we now -> oo, we obtain the limiting form for the derivatives which become (2) 11 and = 1 By integrating we arrive at Φ(2) = max + z, 0} and v5(2)
max 11 0). we notice that since e + oo we cannot find the likelihood ratio terms of the discriminator. The cost based on this particular pair is called the hinge loss Tang (0013) and is very popular in binary classification where one is interested only in the maximization problem. The corresponding method is known to exhibit overall performance which in practice is considered among the best Rosasco ct al.
& Czarnecki| (2017). Here, as in Zhao et al.] (2016). we propose the hinge loss means perform adversarial optimization for the design of the generator G(x).
5

---

Under review as a conference paper at ICLR 2021
Table 4: Optimization problems for GANs
GAN
((2)
(z))
Ju
Hinge
-(1 + 2)+
-(1 2) +
Wasserstein
R
In Table we can see Hinge and Wasserstein GANs optimization problems.
The detailed derivation of the above optimization problems and some more examples can be found in Appendix A.2
This completes presentation of examples. However, must emphasize, that these only few illustrations of possible pairs {o(z), (x)) } one can construct. Indeed combining, as dictated by equation 8 any strictly increasing function w(r) with any positive function p(z) generates a legit- imate pair {o(2), { and a corresponding min-max problem equation that enjoys the desired
solution r(x) 1.
Remark Given the numerous choices have in defining adversarial optimizations for solving the same problem, one may wonder whether there exists a means to rank these methods and identify the most efficient, at least for classes of data. This is an extremely challenging question which, unfortunately, finds no answer even in simpler problems. For example, in binary classification there are also classes of optimization problems that are used train neural networks Bartlett Bartlet et al.) (2066); Masnadi-Shiin & Vasconcelos (2009). However, until now, no theoretical analysis exists that can order them and designate the most efficient one. This possible only through experience with countless simulations.
3 DATA-DRIVEN SETUP AND NEURAL NETWORKS
Let us now consider the data-driven version of the problem. As mentioned, the target density f(x is unknown. Instead are given collection of realizations {x,} that follow f(x) and a second col- lection {z,} that follows the origin density h(z). These data constitute our training set. Regarding the second set {zy} it can either become available "on the fly" when lu(z) is known by generating realizations every time they are needed, or it can be considered fixed from the start exactly as {x:). if fz(z) is also unknown. As pointed out in Section I are interested in designing generator Gizz) so that when we apply it onto the data Zj. that 18, Y3 = G(zz) the resulting y, will follow density that matches the target density fxx). Since we are now considering the data-driven ver- sion of the problem, we are going to limit G(z), D(x) ) to be the outputs of corresponding neural networks. Therefore the generator replaced by G(z, f) while the discriminator by D(x, i)) where 0, 2 summarize the parameters of the two neural networks. Of course instead of neural networks one could use any other parametric family, as SVMs, capable of efficiently approximating any nonlinear function.
Once we have selected our favorite w() and p(z) functions we can compute from equation the functions (z),, (z) that enter into the min-max problem defined in equation 2 This problem, after limiting the generator and discriminator to neural networks, can be rewritten as follows
min max ((, min max {1Ex~> d (D(x 0))] Ez-h (esssn(((DEE (),
(14)
If A.. i., are the corresponding optimum parameter values, and the structure of the two networks is sufficiently rich, we expect that G(z, 0.), D(x, 8.) will approximate the optimum functions D(x), G(z) of the ideal problem in equation respectively. In particular for 0g, the generator Giz, (o), whenever applied onto any 2j that follows h(z), it will result in a y; G(z,, 0.) that follows a density which is expected to be close to the target density f(y).
When replacing D(x), G(z) with neural networks we must take special care of the corre- sponding outputs. Basically, we must guarantee that they are the the correct form. This is particularly important in the case of the scalar output D(x, 0) of the discriminator. We recall that the optimum discriminator is D(x) (x))).) This implies that we need to assure that D(x, 0) takes values in J., (the range of wr)).. Consequently, we must apply the proper nonlinearity in the output of the discriminator that will guarantee this fact.
Remark 2

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
In this section, we want to examine the performance of the GANs objectives presented in Table for different datasets. For that reason we tested their performance on four different datasets, namely MNIST (LeCun et al. 1998), CelebA (Liu et al. 2015). CIFAR- 10 datasets (Krizhevsky et al. 2009). and Stanford Cars (Krause et al., 2013). We recall that GANs are notorious for their nonrobust behavior [Bengio (2012); Creswell (resselll et al. 2018); Mescheder et al. (2017). For the stabilization of the training process, we used the maximum gradient-penalty methodology which was generalized to class of Lipschitz GANs in Zhou et al. (2019) (implementation details in Appendix (A.)). In this
section we present our results for the regularization parameter 10, in the Appendix we included results for different values of X.
In Tables 5 we present the final attained Frechet Inception Distances (FID) Heusel et al. (2017) and Kernel Inception Distances (KID) Binkkowsi et al. (2018) scores after training. In Table computed the absolute difference of the discriminator estimated likelihood ratio from the optimal likelihood ratio, which is equal
and the variance around the optimal value. For Hinge and Wasserstein GANs we cannot compute the likelihood ratio related metrics, as we mentioned in section
2.1 Also, in Figure show the evolution of FID (second row) and KID (third row) during training. In the first row we see the value of the distance dx, y) = |Ex= [ D(y)) For the expectations estimation, we used 64 examples for MNIST and 128 for Stanford Cars, CelebA, and CIFAR-10. We believe that this is an insightful quantity since its value is an indicator of how accurate the discriminator can distinguish samples from the target distribution and the generator output. Finally, in Appendix A.3 some generated samples of our trained generative models are included.
Table FID Scores
GAN
CARS
CELEBA
CIFAR10
MNIST
Ala
23.30 + 0.10
1.53
9,67
2.18
Alb
24.22
7.68 1.800.00
9.69
2.00
A2
2..40
7.02
9.53
2.17
A3
23.64
0.06
9.02
2.05
MSE
23.99
7.06
9.01
2.15
Bla
+3.60+0.112
8.06
9.67
2.13
Blb
23.91 0.07
0.07
9.67
0.01
Exponential
0.06
0.03
0.09
2.10
B2
23.46
9.09
9.79
2.09
Cross Entropy
25.39 0.16
7.53
9.72
2.08 0.01
C2
24.36 0.122
7.47 0.01
0.03
2.10
Hinge
22.88 0.14
10.91 + 0.05
9.99 0.03
0.06
Wasserstein
23.99
10.08 + 0.04
9.74 0.02.
2.16
Table 6: KID Scores
GAN
CARS x10 +x10-12
CELEBA
CIFARIO 1000xx10
MNIST
x10 x10
-0100 12
x10 x10-+x100
Ala
4.03 2.45
2.57 + 5.05
2.75 8.47
7.80 5.86
Alb
3.33 + 1.89
2.67 + 8.35
2.36
5.86
6.30+4.30
+ 4.32
A2
4.36 3.18
3.38
9.68
3.09 8.35
+
A3
4.49
7.29 5.48
2.40 4.95
8.53
Mean Square
1.71
2.55 9.34
2.09 5.63
4.4+7.51
Bla
3.17
5.56
9.78
2.35
5.52
4.5+3.33
Blb
2.24
7.32
1.18
2.63 8.42
6.85
Exponential
2.89
10.05
6.10
2.11 8.34
3.41
B2
7.06
12.48
7.38
2.47 + 5.42
8.62
Cross Entropy
13.09 8.99
3.53 12.65
2.53
5.58
6.16
C2
8.45
3.11 9.72
41.0
3.32
Hinge
4.97 3.50
26.65
3.97
9.42
8.33 4.16
Wasserstein
4.92 + 2.70
20.45
1.74 + 5.22
7.33 3.333

---

Under review as a conference paper at ICLR 2021
MSE
Bia
Exponential
Crass Entropy
C2
Hinge
Wassersten
CARS
CELEBA
CIFARI0
MNIST
0.06
0.02
0.4
0.02
0.04
0.01
0.02
1.5
Iterations
x10
lterations
x10
Iterations
x10
Iterations
100
2.3
2.25
24
10
2.2
23
10
2.15
22
2.1
1.5
15
0.5
lterations
x10
leeraioons
x10
Iterations
100
Ieeraioons
x10
10
x10
110
2.5
1.5
KD
1.5
KID
0.5
0.5
1.5
0.5
1.5
10
15
0.5
lterations
X100
Itarations
=10
Iterations
Meeations
x10
Figure I: Evolution of the corresponding FID and KID scores during training for the datasets Stan- ford Cars, CelebA, CIFAR10, and MNIST.
Table 7:
Mean absolute difference of the discriminator estimated likelihood ratio from the optimal value and variance around the optimal likelihood ratio.
GAN
CARS
CELEBA x1-
CIFAR1O x10 x10
MNIST
x1, x10-
x10-1 x x10
Ala
0.10, 0.29
0.01,0.17
1.79, 1.33
0.07, 6.49 6.49
Alb
0.38, 0.37
0.01,0.20
5.71,
0.13, 8.05
A2
0.12,0.21
0.01,0.11
1.83, 1.16
0.07,5.25
A3
0.17,0.13
0.01,0.14
1.27,4.79
2.75,
Mean Square
0.13, 0.62
0.01,0.26
1.78,
0.07,6.16
Bla
0.17,0.72
0.01,0.49
1.46, 1.36
0.08,
Blb
0.14,
0.01,0.57
1.63, 1.49
0.07,7.67
Exponential
0.03,0.16
0.01,0.97
0.32,4.34
2.88
B2
0.03,0.14
0.01,0.90
0.55, 4.86
0.04,3.07
Cross Entropy
0.16,3.98
0.03,0.02
0.52, 0.44
1.39, 2.66
C2
0.71,
0.03,0.94
0.38,0.20
1.39, 2.74
For the MNIST dataset, the different GANs have very similar performance, something that is rea- sonable since this is simple dataset. The d(x. y) curves are quickly converging indicating that the discriminator has stabilized. Interestingly, the FID, KID curves for GANS with similar d(x, y) have also close FID, KID scores. In particular, Subclass C GANs (Cross- Entros- with C2); Subclass D (Hinge) with Wasserstein; B2, Bla and A3; Blb with Exponential: Alb with A2; and Ala with MSE. For the CIFAR-10 dataset, the FID and KID scores for different objective functions tend to have similar behavior, but some differences exist. In particular, the objectives Ala, Alb, A2, MSE, Bla, Blb in the initial iterations create a steep valley for the metric d(x, y); and at the same time in the FID score, these objectives attain smaller FID scores faster than the other GANs, Also, the further improvement of the scores during training seems to be related to the behavior of d(x. y). For instance, the d(xx y) of Alb, A2 is still increasing during the last iterations, and the generated image quality (FID score) is improving. Lastly, similar to MNIST, GANs that have very close d(x, y) tend to have close FID, KID curves.
8

---

Under review as a conference paper at ICLR 2021
For Stanford Cars and CelebA, observe some performance gaps between different subclasses. Specifically for CelebA, the Subclass D and Subclass objective functions start to diverge after some training steps, something that is evident from the FID and KID curves where the scores in- crease dramatically after approximately the first half of iterations. Also, this is noticeable for the d(xr, y). where sudden drop is evident around the same period. Interestingly the variance around the optimal likelihood ratio value is increased for the GANS who started to diverge. For Stanford cars, the Subclass C has the worst performance, with the recorded MMD score being an order larger than the other objectives (expect Subclass B B2, Exponential objectives-). Furthermore, for this dataset, the distances d( d(x,y' .y are "noisy" (strong fluctuations) and with approximately ten times larger mag- nitude value when compared to the corresponding curves in the other datasets. It is reasonable since this dataset a complex one, where we have natural scenes with cars, and, importantly, significantly smaller (~ 8000 training samples, when MNIST, CIFAR 10 has 60000 and CelebA 200000).
In summary, our simulations indicate that the Subclass objectives Ala, Alb, A2, MSE have the best performance both in terms of the computed metrics (hence image generation quality) and stability during training. They might not give the exact best score in all datasets, but are very close to the best recorded, and, most importantly, they have an overall more stable behavior when compared to objectives from the other subclasses. Furthermore, the discriminator convergence to the optimal likelihood ratio between the dataset and the generator output can be ad-hoc measure of the behavior/stability of our generative models than can be used during training to provide online, useful insights of the current training condition.
CONCLUSION
In this paper, we provided and demonstrated a straightforward methodology to determine loss func- tions that solve the generative adversarial problem. Our results suggest that there is not a single loss function that achieves the best performance terms of the examined metries for all different datasets. This performance variation among loss functions becomes evident as the increasing com- plexity of the datasets that convolutes the generation task better addressed by some loss functions that clearly outperform others. Specifically, in simpler datasets, such as MNIST, the evaluated loss functions yield very similar performance, whereas, more intricate datasets like CelebA, CIFAR- 10, and Stanford Cars, performance "gaps" between the different loss functions, and different sub- classes, emerges. Our findings also propose that in every generation task unexplored loss functions outperformed the previously proposed ones. Consequently, this function class is worth-exploring to identify new loss functions that can be used and evaluated in different applications. Our method provides a versatile tool that can be exploited in that direction.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-155 2006.
Yoshua Bengio. Practical recommendations for gradient-based training of deep architectures. In Neural networks: Tricks of the trade. pp. 437-478. Springer, 2012.
Mikolaj Binkowski, Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint orXivv:801 00401,111100001101110111 2018.
Jeremiah Birrell, Paul Dupuis, Markos A. Katsoulakis, and Yannis Pantazis. (f. 7) divergences: Interpolating between f-divergences and integral probability metrics. arXiv preprint arXiv:2011.05953 2020.
George EP Box and David R Cox. An analysis of transformations. Journal of the Royal Statistical Society: Series B (Methodological), 26(2):211-243 1964.
Lev M Bregman. The relaxation method of finding the common point of convex sets and its applica- tion to the solution problems in convex programming. USSR computational mathematics and mathematical physics, 7(3):200-217, 1967.
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35 (1):53-65, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Advances in neural information processing systems, pp- 5767-5777, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Katarzyna Janocha and Wojeiech Marian Czarnecki. On loss functions for deep neural networks in classification. arXiv preprint 1rXiv:1702.05659, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.668 69800 2014.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324 1998.
Chun-Liang Li, Wei-Cheng Chang. Yu Cheng, Yiming Yang, and Barnabás Póezos. Mmd gan: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems, pp. 2203-2213, 2017.
Shuang Liu, Olivier Bousquet, and Kamalika Chaudhuri. Approximation and convergence proper- ties of generative adversarial learning. In Advances in Neural Information Processing Systems, pp. 5545-555$ 2017.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pp. 3730-3738, 2015.
10

---

Under review as a conference paper at ICLR 2021
SC Martin Arjovsky and Leon Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34 th International Conference on Machine Learning, Sydney, Australia, 2017.
Hamed Masnadi-Shirazi and Nuno Vasconcelos. On the design of loss functions for classification: theory, robustness outliers, and savageboost. In Advances in neural information processing systems, pp. 1049-1056, 2009.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of gans. In Advances in Neural Information Processing Systems, pp. 1835, 2017.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances neural information processing systems, pp. 271-299, 2016.
Lorenzo Rosasco, Ernesto De Vito, Andrea Caponnetto, Michele Piana, and Alessandro Verri. Are loss functions all the same? Neural Computation, 1665::0063-1076, 2004.
Jiaming Song and Stefano Ermon. Bridging the gap between f-gans and wasserstein gans. arXiv preprint arXiv: 1910.09779 2019.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation: A comprehen- sive review (statistical experiment and its related topics). 2010.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
Yichuan Tang. Deep learning using linear support vector machines. arXiv preprint arXiv:1306.0233, 2013.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprint 1rXiv:1609.03: 03026, 2016.
Zhiming Zhou, Jiadong Liang. Yuxuan Song, Lantao Yu, Hongwei Wang, Weinan Zhang, Yong Yu, and Zhihua Zhang. Lipschitz generative adversarial nets. arXiv preprint arXiv: Ttt:::99222 1902.0567, 2019.

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1 PROOFS
Theorem Proof. We note that the constraint (w(()) 0 does affect the generality of our class of functions since from equation 8 have that (²),,, after integration, is defined up to arbitrary additive constant. We can always select this constant so that the constraint is satisfied. We
would also like emphasize that this constraint is needed only for the proof of this theorem and
is not necessary for the corresponding min-max problem defined in equation 7
For fixed r, to find the maximum of d(D) + (D) we consider the derivative with respect D which, using equation 8 takes the form
d'(D) (r-
The strict increase of wrr) is inherited by its inverse function w- (2) which, combined with the positivity of p(z), implies that the previous expression has the same sign r-w (D) or wr)) D. Consequently D (r)) is the only critical point of ((D) + rD( which is global maximum. Of course there are possibilities for extrema at the two end points of II, but they can only be (local) minima.
Let us now focus on the resulting function ((w(r)) + (w((). Taking its derivative with respect to yields
(ō(w(r)) + +r((())))) (d'(w(r)) + red (w(r)) + ((w(r)) = (w(r)),
where the last equality is due to the specific definition of the two functions ((2), (=) in equation g Since (2) p(z) > O, this implies that (z2) is strictly increasing, being also the integral of p(z) is continuous in z. If we combine this property with the strict increase and continuity (as a result of left and right differentiability) of w()) we conclude that (ww(r) is also strictly increasing and continuous in v. We recall that (=) is selected satisfy (w(()) O, consequently for 1 the function & (w)) + ru (w(() has a unique minimum which global and no other critical points. Of course it can still exhibit extrema at 0 and/or - do but they can only be (local) maxima.
Corollary Proof. First, we observe that
Ex~ [o(((x)) + +x(x)(((xx)) = Exws (o((()) ((xx)((((x))) + (w)())
(15)
with the last equality being true since Ex {xx)(r(xx) = 1 and where = - (w(1)) We start with the maximization problem. Since D(x) is function of x we have
max Exx [o(D(x)) + r(x) + D(x)
(D()) Exws
max { (o(D(x)) + +x) (16) D(x)
The maximization under the expectation can be performed for each fixed However, when fix x then r(x) becomes a constant and the result of the maximization depends only on the actual value of rx)). This suggests that we can limit ourselves to functions the form D(x) = Dr(())). After this observation we can drop the dependence on and perform, equivalently, the maximization maxp (ø (r)()) + ré (D(r)) for each fixed r. The pair (0(2), {(2) satisfies the assumptions of Theorem therefore maximization is achieved for D(r) w(r). This implies that
max Ex- (x(D(x)) + +x(x)()(x)) = Ex-s [o(w(r(x))) (rxps))))))x))))())))))) +(())(((((()))) + (w(1)). D(x)
12

---

Under review as a conference paper at ICLR 2021
We can continue in a similar way for the minimization problem. Specifically
min max Exxr ( -
r(o) D(x)
min Exn r(x)EL,
2
Exw
min [r(x)eL,
{ + +
} 2
Ex-s min (o(w(r) (r((w(r))
((w(1))
with the last inequality being true since the minimization that follows is unconstrained and the last equality being consequence of [heorem[] The final lower bound is clearly attained by r(X) 1, which also a legitimate solution of the constrained minimization, since r(X) = belongs to the class Ly of likelihood ratios. Consequently r(X) 1 is the solution the min-max problem. Returning to the original min-max setup with (z) replacing ((2), we can clearly see that it satisfies equation 11 This completes the proof.
A.2 THE SUBCLASSES OF THE GANS FAMILY
Subelass A: (r) = r The first examined subclass is the simplest one, consisting of just powers of the likelihood ratio. To the best of our knowledge, this is the first work proposing objective functions from this class. To find the pairs {Ø(z), (²()) we proceed follows.
We have that w (2) zá and Is = [0, oo). According to equation 8 for E [0, oo) we must
define o' (2) = -2 p2)) vx(2) = =(z). The following examples can be shown to satisfy these equations.
A1) If we select p(2) = 29, with 3 # -1, 11 1- a. this yields Φ(2) 1+ +
1 : +B and (x) = 1
1+
For B 1, p(2) 2 (22) -azá, xx2) log z. For -1 a p(z) 2
x(z) = - --log 2, (=)) = -a2-# +
A2) If we select a 1, p(2) (1+=) then, (02) (1 + 2) and (2) -(1 (1+ 2 1
A3) If we select a = 1, p(=) [[+T]s then, <0=) log(1 I + z) and (=)) log(1 + =-))
For the particular selection (r) (corresponding to a 1) we can show that the resulting cost is equivalent to the Bregman cost (Bregman 1967). In fact there is one-to-one correspondence between our p(z) function and the function that defines the Bregman cost. This correspondence however is lost once we switch to different or different w(r) function, suggesting that the proposed class of pairs ((()), {=)),, far richer than the class induced by the Bregman cost.
We should mention that in A1) the selection a = 1,3 O is known as the mean square error criterion and if we apply only the maximization problem then this corresponds to a likelihood ratio estimation technique proposed in the literature by SSugiyama et al. (2010) 2012). We will refer to this case as the MSE GAN.
Subclass B: w() a logr This subclass considers one of the most popular transformations of the likelihood ratio, the lig-likelihood ratio. As in the first subclass, for the first time, the next examples are presented. They can be used either under a min-max setting, for the determination of the generatooddiscriiinator pair, under pure maximization setting for the direct estimation of the log-likelihood ratio function log r(x).
We have w w-1 (2) e and J. R. As before p(z) must be strictly positive and, according to equation 8 for all real 2 we must define d'(z) = -0 p()), (x)) = p(z). The following examples satisfy these equations.
13

---

Under review as a conference paper at ICLR 2021
B1) p(2) = 8=
82 with 3 + 0, a, this produces ((() = ela-010
a-p 422) = e-- If 3 0 then
3
p(z) =1, (())
- (b)) 2. If 3 = a then p(z) = 6. as, x(2)) = - and 1b(z))
a
We call the = 1. 0.5 case the Exponential GAN.
B2) If = 1, p(2) = Itc' then, (()) = - log(1 + e*) and (=)) = - -log(1 (1 + +
Subclass C: ((r) = - As we already mentioned, this is another important transform of the likeli-
hood ratio. Interestingly, in this subclass belongs the first introduced GAN (Goodfellow et al. 2014
the Cross Entropy GAN.
When (r) न we haveew-la the functions 6(2), (2) according to equation 8 d'((2 -=g)(z), (²(2) p(z). The next set
-ー= and Jw [0, 1]. For p(z) > 0, E [0.1] we must define
of examples can be seen to satisfy these equations.
CI) If we select p(z) = 1/2 this yields (z)) = log(1 - 2) and (2) = log 2.
C2) Selecting p(z) = (1 - 2)", with a * 0, -. yields (2) = - 1
1ta (1 - 2) 1 + 1 (1 - 2) and
(=) T+a 2) 1 For 0, we have p(=) = 1 and 6(z) = + log(1 2), (z) 2,
while for a = 11 we have p(=) - and ((=) = - log(1 - =) - (=)) = log(1 - 2)-
In CI) we recognize the functions used in the original article by Goodfellow al. 20114) C2) appears for the first time.
Subelass D: (() slgg() This a special case of w(r) with the corresponding function not being strictly increasing. It turns out that we can still come up with optimization problems, two of which are known and used in practice, by considering w(r) as a limit of a sequence of strictly increasing functions.
Monotone Loss: As a first approximation we propose sign(z) ~ tanh (2) where >> 0 parameter. We note that lime-+oo tanh(5 {2) = sign(2). Using this approximation we can write
sign(logr) r) x tanh (2 (ogr)
r - re +
= w()).
(17)
As we mentioned, we have exact equality for c + co. Let us perform our analysis by assuming that e is finite. We note that w (2) = (++) and Ih = [-,, Consequently, if p(z) >> 0 for
E [1, we must define d'(=) =
p(=), (=)) p(2).
DI) If we let do in order to converge to the desired sign function, this yields d'(x) -p(2) and (2) p(=). This suggests that c²=) -
- f p(x)ddx is decreasing and (=)) = p(z)dz
- 2) is increasing. In fact any strictly increasing function (=)) can be adopted provided we select <(z) (2)..
There is a popular combination that falls under Case DI). In particular, the selection (=)) = = -[z) reminds us of Wasserstein GAN Martin Arjovsky & Bottou (2117), with two differences, in our case 2 should lie in [-1, 1] and the discriminator is not constrained to be a Lipschitz function.
Hinge Loss: As a second approximation we use the expression sign(=) ss sign(=)[æ|#,, e >>> O, which is strictly increasing, continuous and converges to sgu(=) as c - oo. This suggests that
sign(log r) x sign(log r) ll w()),
(18)
and (2) c* Since ((r can assume any real value conclude that Ju = R which, clearly, differs from the previous approximation where we had Ih = [-1, If p(z) > 0, E IR then, according to equation we must define dx(x) = -c p(z), v'(2) = p(z). We present the following case that leads to very well known pair from completely different application.
c²
D2) If we select v'(z) p[=)) = -11 +1,--11 } > 0 then ('()) - (e- -1- fe-1=l++,,<11 J. If we now let c - 0o, we obtain the limiting form for the derivatives which become (2) = -1 and d'(2) -2--1 By integrating we arrive at 6(2)
-
max + 2, 0} and x2(2)
max 2, 0).
The cost based on this particular pair is called the hinge loss Tang (2013) and it is very popular in
binary classification where one is interested only in the maximization problem. The corresponding
14

---

Under review as a conference paper at ICLR 2021
method is known to exhibit overall performance which in practice is considered among the best Rosasco et al. (2004); Janocha & Czarnecki (2017). Here, as in Zhao et al. (2016), we propose the hinge loss means to perform adversarial optimization for the design of the generator ((x).
This completes our presentation of examples. However, we must emphasize, that these are only a few illustrations of possible pairs {Ø(=), +>))) one can construct. Indeed combining, as dictated by equation 8 any strictly increasing function (() with any positive function p(2) generates a legit- imate pair (=)} and a corresponding min-max problem equation 7 that enjoys the desired solution r(x) = 1.
A.3 EXPERIMENTS
In our experiments we employed the two neural networks the generator and the discriminator. For the generator, we used four-layer neural network where the first layer is linear and the remaining deconvolutional; with ReLU activation functions between the layers except the final layer where used sigmoid function since the output is an image with pixel values in the range [0. 1]. The generator input is a standard i.i.d. normal vector with dimension 64 for MNIST and 128 for Stanford Cars, CelebA and CIFAR-10. 10. The output of the generator is 784 x vector for the MNIST dataset, whereas for the other datasets is a 3072 x vector.
For the discriminator, we used four-layer neural network with three convolutional layers followed by a linear layer. We applied Leaky ReLUs between the layers except for the final layer where we adopted proper functions based on the range Ju For the training of the two neural networks we applied the Adam algorithm Kingma & Ba (2014) with 31 = 0.5, By = 0.9, learning rate 10-4 and batch size 50 for MNIST and 128 for Stanford Cars, CelebA and CIFAR-10. For all datasets, the training lasted 180000 iterations.
A.4 CELEBA
In Figures 2
5 we can see the estimation of likelihood ratio from the discriminator neural network and the FID,, KID scores of the generated images during training. For every optimization problem, we tested its performance for different values of the regularization parameter of the maximum gradient penalty. The different values of À where (0.01, 0.1, 1, 10) in Zhou et al. (2019). We must mention that in the cases where some À value is missing means that the algorithm diverged during training.
From our simulations, we notice that, especially in the case of the KID score, the likelihood ra- tio approximation accuracy of the discriminator coincides with the quality of the synthetic images produced by the generator. For instance, in Figure 2 for A3, X = 10 at ~ 150000 iterations the likelihood ratio (which should converge to 1) starts be more variant around the optimal value, then we notice the KID, FID scores increase indicating that the generated images quality drops. It is apparent in Figure I where we see some blurry, almost faceless, images, Also, for Alb, Bla, Blb, Exponential, B2 GANs, we notice the same behavior.
Moreover, in all cases, we notice a faster convergence for larger values of X. Interestingly, there is similar behavior, in terms of convergence, between the different X's For example, in B2, for = 10, we notice that the variance around one is increasing. Similarly for X = 1, = 0.1 but more slowly in terms of iterations. Therefore, it seems that the regularization parameter cannot fix the divergence problems of a GAN, but they can influence the period they will become evident.
As already discussed, in the case of Hinge and Wasserstein GANs, the likelihood ratio value is not computable. For this reason in Figure 5 we present the discriminator output for dataset (D(X) and for synthetic (D (D(G(Z))) samples. In the Hinge GAN, the discriminator output gradually in- creases its range of values for X = 10 around 130000 iteration and for X = around 200000 iteration. Accordingly, the synthetic images produced by the generator start look less similar to the ones from the dataset. Similarly, Wasserstein GANs have the same patterns. But in this case, the discriminator output tends to have large values. Our simulations argue, that a way to reduce those large values is to increase the regularization parameter X, in other words to "force" the discriminator being Lipschitz.
15

---

Under review as a conference paper at ICLR 2021
AIEAA
14
ATE.A
0.
75
48
10
1.5
42-
25
740
Rarafonn
Exratono
All
ATBLA
46-
0.5
1.45
2.5
14
10
10
-107
x10f
4.5
Ab
3.-
0.5
10
12
14
18
x10
9.2
2.5
12
0.5
10-
1.5
8.-
7.0
Baraforn
Tarifoor
-10f
USE
11.0
12.5
9.2
Figure 2: Likelihood ratio (r). KID, FID scores for CelebA dataset. Simulation results correspond Subclass A GANs for different values of the maximum gradient penalty hyperparameter X.
A.5 STANFORD CARS
In Figures 6 8 we can see the estimation of likelihood ratio from the discriminator neural network and the FID, KID scores of the generated images during training. For every optimization problem, we tested its performance for different values of the regularization parameter of the maximum gradient penalty. The different values of X where (0.01, 0.1, 1, 10) as in Zhou et al. 2019). Cases where some value is missing means that the algorithm diverged during training.
16

---

Under review as a conference paper at ICLR 2021
000
000
14
833
heradons
Baraforo
x10
Bo128ord
-10"
2210
0.4
0.5
74
210"
Sapenorats
0.4
0.2
001
1.4
0.4
0.0
7.8
78
Besslone
1104
-101
K'00
04
3.5
0177
1.0
00
02-
1.4
2.5
5.4
0.4
75
0.5
2.5
Sersfone
x12
Serslora
Baradora
Figure 3: Likelihood ratio (r), KID, FID scores for CelebA dataset. Simulation results correspond to Subclass B GANs for different values of the maximum gradient penalty hyperparameter A.
A.6 SYNTHETIC EXAMPLES
In the Figures 10 and 2 present some random samples of the trained generators for the datasets Stanford cars, CelebA, and CIFAR- 10 for À = 10.
17

---

Under review as a conference paper at ICLR 2021
Creas Enpopg
Dreae Denag
Crass Enreny
04
14
74-
0.6
7.
0.5
10
1.6
<10"
x10
Figure 4: Likelihood ratio (r), KID, FID scores for CelebA dataset. Simulation results correspond to Subclass C GANs for different values of the maximum gradient penalty hyperparameter X.
Wassensiain 0.1
Whosersben: A 0.01
lerafion
020
Wasserstein
D620
8 t
Wasserstein
88
Baration
Novaloro
000
Panations
Iorations
Wosserstenn A 10
Wassersten:
D00
78
hanations
x10f
Hanalora
Iaruliora
BAT
10.5
300
3.5
3--
1.5
15
18
Figure 5: Discriminator outputs for real (D(X)) and sythetic (D(G(Z))) samples. KID, FID scores for CelebA dataset. Simulation results correspond Hinge and Wasserstein GANs for different values of the maximum gradient penalty hyperparameter A.
18

---

Under review as a conference paper at ICLR 2021
-10
A1a
4.
25
2.
20.2
15
06.
-100
2.00
0m
Barul
Barulorn
X32
0.1
2.5
ASAA
6.0
Roratons
-10
75.
0.5
00
50
4.5
70
50
3.5
60
40
30
Earaloro
Exrutoro
Earatoro
Figure 6: Likelihood ratio (r), KID, FID scores for Stanford cars dataset. Simulation results corre- spond Subclass A GANs for different values of the maximum gradient penalty hyperparameter
19

---

Under review as a conference paper at ICLR 2021
25.
20
16-
1.0
04
10
144
12
22.5
Ioeations
X11
X104
Blbck
Bltck
980
25-
X.12
-10
10.5
16
31
10
Plporentid
25-
1.5
23
10
30
19
16
81
10
Ennidtons
-10"
Figure 7: Likelihood ratio (r), KID, FID scores for Stanford cars dataset. Simulation results cor- respond to Subclass B GANs for different values of the maximum gradient penalty hyperparameter A.
20

---

Under review as a conference paper at ICLR 2021
25 110
25.
2.5
30.5
16-
23
Danatoona
Earatoon
29
Figure 8: Likelihood ratio (r), KID, FID scores for Stanford cars dataset. Simulation results cor- respond to Subclass C GANs for different values of the maximum gradient penalty hyperparameter A.
0000
223
02
20
Heratiors
Wassardriis
29
Handtoon
Wassersbeir.
DOZZA
Wowseesten:c.cco 19
10
Inrotions
Kiof
30-
12 14
Figure 9: Discriminator outputs for real (D(X)) and sythetic (D(G(Z)) samples, KID, FID scores for Stanford cars dataset. Simulation results correspond to Hinge and Wasserstein GANs for differ- ent values of the maximum gradient penalty hyperparameter X.
21

---

Under review as a conference paper at ICLR 2021
Ala
Alb
A2
A3
MSE
Bla
B1b
Exponential
B2
Cross Entropy
C2
Hinge
Wasserstein
Figure 10: Random samples for different GANs objective function (and maximum gradient penalty regularization parameter À = 10) on Stanford cars dataset.
22

---

Under review as a conference paper at ICLR 2021
Ala
A1b
A2
A3
MSE
Bla
B1b
Exponential
B2
Cross Entropy
C2
Hinge
Wasserstein
Figure 11: Random samples for different GANs objective function (and maximum gradient penalty regularization parameter = 10) on CelebA dataset.
23

---

Under review as a conference paper at ICLR 2021
Ala
A1b
A2
A3
MSE
Bla
B1b
Exponential
B2
Cross Entropy
C2
Hinge
Wasserstein
Figure 12: Random samples for different GANs objective function (and maximum gradient penalty
regularization parameter À 10) CIFAR-10 dataset.
24


Reviews:
[{"review_id": "wCy49lcuPyo", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper generalizes the min-max problem of GANs to form a richer family of generative adversarial networks. Interestingly, most of the well-known variants of GANs can be found in the spectrum of formulations covered by the family proposed in this work.\nIn terms of modeling, it is evident that the family proposed in the paper is richer than that of f-GAN. The family in this paper is shown to have a connection to WGAN except that the Lipschitz condition is omitted. \nHowever, under the light of existing works including f-GAN and other relevant works, the obtained theoretical results are not surprising to me. In addition, apart from providing a richer family, this work does not significantly influence the practical aspects of GANs.\nI have some following questions:\n1. If we solve the min-max problem in (2) subjected the fact that \\phi and \\psi satisfy Eq. (9), is it equivalent to minimizing any divergence between two distributions with pdfs f and g?\n2. D(x) is not a typical discriminator whose values between [0;1] providing the probability to distinguish true and fake data, is not it? D is more similar to a critique whose output values are real-valued, is not it?", "extra_scores": {}, "extra_text": {}}, {"review_id": "Tv8JIuD3bl3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, this paper provides impacts on understanding the core of generative models with adversarial optimization problems.\nThis paper shows the diverse possibilities of formulating the generative model optimization problems that the researchers can further investigate for better performances. \nAlso, this paper shows that generative models with unexplored losses achieve the best results in various datasets which demonstrates the possibilities of future improvements of generative models.\nOverall, this paper is valuable to the machine learning community (especially for generative models and adversarial training). \nThe below are some concerns for this paper but those concerns are not bigger than the advantages of this paper.\n\n1. Quantitative experiments\n- Although the authors provided two tables (Table 2 and 3), there were not much analyses about the results.\n- I understand that it is not an easy problem to understand \"when\" should we use \"which\" function. However, it would be great if the authors can discover some trends in the results to demonstrate which type of functions work well with which type of datasets.\n- I think it would be great to use some synthetic data with known characteristics of distributions as the target distribution to analyze for understanding this point.\n\n2. Other types of dataset\n- Generative models are widely utilized in computer vision. \n- However, there are various other types of datasets that can get benefits of generative models such as tabular data and time-series data.\n- It would be good if the authors can provide some simple experiments to demonstrate its generalizability.\n\n3. Minor points\n- It is not clear to transform between equation (3) and (4). I think this is a critical part in this paper; thus, it would be good to explain a little bit more for this part.\n- The authors explain the differences between f-GAN and this paper. However, it is not super clear to understand. It would be good to clarify this point to highlight the novelty of this paper.\n\n--------------------------After reading other reviews are rebuttals---------------------\n\nAfter reading all the reviews from other reviewers and corresponding rebuttals, I think this paper is a good paper and enough to be accepted in ICLR. \n1. I think it has a clear difference from f-GAN. It can provide a new loss function for the generative models which can further extend the success of generative models in the future.\n2. Experiments are not super interesting but at least it has some intuitions corresponding to the authors' claims.\n3. General theoretical results for the generative models (such as when should we use which loss) is a very difficult problem to solve. Maybe this paper can provide some intuitions for solving that large problem. But it seems too much to ask this thing to the authors of this paper. Without that, I think this paper is still worth to present to the ICLR readers and participants.\n\nTherefore, I am standing on my original score (7).", "extra_scores": {}, "extra_text": {}}, {"review_id": "QssmiSKeZhV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary of contributions:**\nThis paper proposes a new framework to design new loss for GANs. The authors show that their framework is quite general and encompass a number of existing approaches (e.g. the original GAN formulation, hinge loss, etc..), they also propose a categorization in three different classes and derive new loss function. They then compare experimentally the different existing loss and the new proposed loss that fall under their framework.\n\n**Main comment**:\nThe framework proposed in the paper is interesting since it's quite general and the authors are able to derive a large number of existing as well as new loss from it. However, I think the framework has several limitations:\n1. The formulation is based on the likelihood ratio which is only defined if the support of $g$ and $f$ match, this is known to not be the case in the context of GANs.\n2. The benefit of the framework is not clear, while it provides a way to derive new loss it's not clear what are the advantages of the new loss. Theoretically the author argue that it is a hard question to answer, and I agree. The authors try to answer this question through experiments but I find the experiments not very convincing. In particular, the authors argue that subclass A objectives are more stable based on the CelebA experiment, however it's not clear to me that the instability is due to a specific choice of objective function, it might just be that the hyper parameters where slightly off for the other objectives. I believe it would be interesting to understand better the results on CelebA, in particular maybe to show that some objectives are indeed more stable, they can vary several hyper-parameters and compare how often each objective is better than the other, that would make the results and conclusion much more convincing.\n\n*Minor comment*:\nThe paper is overall clear but the clarity of some sections could be improved. I think theorem 1 would be more clear if stated a bit differently simply saying that $D=\\omega(r)$ maximize $\\phi(D)+r \\psi(D)$ and that $r=1$ minimize $\\phi(\\omega(r))+r \\psi(\\omega(r))$. Section 3 is a bit dense, the subclasses also seem a bit arbitrary. I believe section 5 could be improved by stating more clearly the different observations, right now it looks more like a description of the figures than a clear statement of the question that the experiments try to answer and how they answer it.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QBIqYE8PKkw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n========\nIn this paper, the authors set out to find what scalar functions will make for a “max” part of the “min-max” GAN objective. They then find such a class of functions, and show that only a ratio between two equal probabilities will be admitted as a solution.\n\nPros:\n====\nThe paper nicely introduces a different way of seeing GANs, not as a difference between the generated and real data, but as a an integer of the ratio between generated and real distribution times the discriminator. Only if the ratio is 1 everywhere is the discriminator unable to maximize the max part of the GAN objective.\n\nFurther, I liked the idea that the discriminator shouldn’t just decide what class data belongs to, but also estimate the probability ratio. Specifically, in the formulation here, the max part is maximized when $D(X) =\\omega(r(X))$, so maximized iff $\\omega^{-1}(D(x))$ doesn’t just classify, but says the probability ratio between the two classes. If this idea is expanded upon, I think the authors could make a novel contribution.\n\nCons:\n=====\n\nUnfortunately, the authors have neglected to carefully explain how their contribution relates to previous work. It’s telling that the paper cites only two papers from 2018, one from 2019 and none from 2020. All other citations are from previous years, even though 2018-2020 has been a time of much GAN research.\n\nA key way in which the author’s work hasn’t been sufficiently compared to previous work is with their main claim “We propose a simple methodology for constructing such [min-max] problems assuring, at the same time, consistency of the corresponding solution.” In [Liu], they show a class of of functions where consistency is also guaranteed, and the class shown by the authors here is a subset of the class in [Liu]. The details are at the bottom of my review\n\nFurther, many of the techniques in this paper seem very similar to [Song], where they also investigate the f*-gan divergence. Specifically, the claims they make in Theorem 1 seem very similar to Prop. 2 in [Song]. Also the change of measure trick in the introduction can be found in [Song]. A detailed comparison of this work to that work would also be helpful.\n\nSince when reading this paper one simply doesn’t know what is previous work which has already been done by others and what is the author’s novel contribution. Once the authors address this, and one is confident the contribution is indeed novel, then the submission would be worth considering.\n\nDetails of why this is a subset of what’s already been shown in [Liu]:\n\nThere, they examine the difference between the target density $d$ (in this paper $d$ is $f$, but Liu uses $f$ for something else) and the generated density $g$ via $\\sup_{f\\in\\mathcal F}\\mathbb E_{x\\sim d,y\\sim g}[f(x,y)]$, so we find the function $f$ in a class $\\mathcal F$ which maximally separates the classes from $d$ and $g$.\n\nNow this work proposes to do the same thing, but with $f(x,y)=\\phi(D(x)) - \\psi(D(y))$ where $\\phi(z) = -\\int_{\\omega^{-1}(0)}^z \\omega^{-1}(t)p(t) dt + C_1 $ and $\\psi(z)=\\int_{\\omega^{-1}(0)}^z p(t) dt + C_2$.\n\nIn [Liu] they then split f(x,y) up into two functions m and r, such that f(x,y)=m(x, y) - r(x,y) where m(x,y) has the form m(x,y)=v(x)-v(y). This can be done in your case too, resulting in (here we drop the constants C_1 and C_2 for simplicity)\n\n$v(x) = \\int_{\\omega^{-1}(0)}^{D(x)} p(t) dt$, $v(y) = \\int_{\\omega^{-1}(0)}^{D(y)} p(t) dt$\n\nand \n\n$r(x,y) = \\int_{\\omega^{-1}(0)}^{D(x)} (\\omega^{-1}(t) + 1) p(t)dt$\n\nSince D(x) must be in $\\mathcal J_\\omega$, this integral has an infimum, and theorem 4 from [Liu] can be applied to achieve the same results as in this paper.\n\n[Song] Song, Jiaming, and Stefano Ermon. \"Bridging the Gap Between $ f $-GANs and Wasserstein GANs.\" arXiv preprint arXiv:1910.09779 (2019).\n\n[Liu] Liu, Shuang, Olivier Bousquet, and Kamalika Chaudhuri. \"Approximation and convergence properties of generative adversarial learning.\" Advances in Neural Information Processing Systems. 2017.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a new family of losses for GANs and showed that this family is quite general and encompasses a number of existing losses as well as some new loss functions. The paper compared experimentally the existing losses and the new proposed losses. But the benefit of this family is not clear theoretically, and this work did not also provide the very helpful insights for the practical application of GANs.

Decision: Reject