SAN: Inducing Metrizability of GAN with Discriminative Normalized Linear Layer
Authors: Yuhta Takida
Masaaki Imaizumi
Takashi Shibuya
Chieh-Hsin Lai
Toshimitsu Uesaka
Naoki Murata
Yuki Mitsufuji
Venue: ICLR 2024 poster (2024)
Keywords: Generative adversarial network
optimal transport
sliced Wasserstein distance
Primary area: generative models

Abstract: Generative adversarial networks (GANs) learn a target probability distribution by optimizing a generator and a discriminator with minimax objectives. This paper addresses the question of whether such optimization actually provides the generator with gradients that make its distribution close to the target distribution. We derive *metrizable conditions*, sufficient conditions for the discriminator to serve as the distance between the distributions, by connecting the GAN formulation with the concept of sliced optimal transport. Furthermore, by leveraging these theoretical results, we propose a novel GAN training scheme called the Slicing Adversarial Network (SAN). With only simple modifications, a broad class of existing GANs can be converted to SANs. Experiments on synthetic and image datasets support our theoretical results and the effectiveness of SAN as compared to the usual GANs. We also apply SAN to StyleGAN-XL, which leads to a state-of-the-art FID score amongst GANs for class conditional generation on CIFAR10 and ImageNet 256$\times$256. The code is available at https://github.com/sony/san.

Published as a conference paper at ICLR 2024
SAN: INDUCING METRIZABILITY OF GAN WITH DISCRIMINATIVE NORMALIZED LINEAR LAYER
Yuhta Takida' Masaaki Imaizumi Takashi Shibuya' Chieh-Hsin Lai'
Toshimitsu Uesaka' Naoki Murata' Yuki Mitsufuji "Sony AI, "The University of Tokyo, 'Sony Group Corporation
ABSTRACT
Generative adversarial networks (GANs) learn target probability distribution by optimizing a generator and a discriminator with minimax objectives. This paper addresses the question of whether such optimization actually provides the gener- ator with gradients that make its distribution close to the target distribution. We derive metrizable conditions, sufficient conditions for the discriminator to serve the distance between the distributions, by connecting the GAN formulation with the concept of sliced optimal transport. Furthermore, by leveraging these theoretical results, we propose a novel GAN training scheme called the Slicing Adversarial Network (SAN). With only simple modifications, a broad class existing GANs can be converted to SANs. Experiments on synthetic and image datasets support our theoretical results and the effectiveness of SAN as compared the usual GANs. We also apply SAN StyleGAN-XL, which leads state- of-the-ar art FID score amongst GANs for class conditional generation on CIFAR10 and ImageNet 256x256 Our implementation is available on the project page https:://taa da . github. so/san/
INTRODUCTION
Utilizing generative adversarial network (GAN) (Goodfellow al.) (014) is a popular approach for generative modeling, and GANs have achieved remarkable performance in image (Brock et al.)
2019 Karras et al. 2019 202T), audio (Kumar et 2019 Donahue et al. 2019 Kong et al. 2020)) and video (Tulyakov et al. 2018; Hao et al. 2021) domains. The aim of GAN is to learn target probability measure via a neural network, called generator. To achieve this, a discriminator is introduced, and the generator and discriminator are optimized in a minimax way.
Here, we pose the question of whether GAN optimization actually makes the generator distribution close to the target distribution. For example, likelihood-batee generative models such variational autoencoders (Kingma & Welling 2014: Higgins et al.] [2017] Zhao et al., [201 Takida et al. 2002). normalizing flows (Tabak & Vanden-Fijnden (2010] Tabak Tabak & Turner 2013) Rezende & Mohamed, 2013), and denoising diffusion probabilistic models (Ho et al.J [020) Song et al. | &
Dhariwal 2021) are optimized via the principle of minimizing the exact Kulbback-eibleer diver-
gence or its upper bound (Jordan et al. 1999). For GANs, in contrast, solving the minimization problem with the optimal discriminator equivalent to minimizing specific dissimilarity (Good- fellow et al.) 2014 Nowozin et al. 2016 Lim & Ye 2017 Miyato et al. 2018). Furthermore, GANs have been analyzed on the basis the optimal discriminator assumption (Chu et al.) (020). How- ever, real-world GAN optimizations hardly achieve maximization (Fiez et al., 2022). and analysis of GAN optimization without that assumption remains a challenge.
To analyze and investigate GAN optimization without the optimality assumption, there are various approaches from the perspectives of training convergence (Mescheder et al. 2017 Nagarajan & Kolter, 2017 Mescheder et al.) 2018 Sanjabi et al.] 2018 Xu et al. 2020), loss landscape around
the saddle points of minimax problems (Farnia Ozdaglar 2020 Berard et al. 2020), and the
smoothness of optimization (Fiez et al. (022). Although these studies have been insightful for GAN stabilization, there has been little discussion as to whether trained discriminators indeed provide generator optimization with gradients that reduce dissimilarities.

---

Published as a conference paper at ICLR 2024
Table 1: Common GAN losses do not simultaneously satisfy all the sufficient conditions given in Theorem 5.3 Thus, we propose SAN to address direction optimality. Even if direction w is the maximizer of the inner problems V. it does not satisfy direction optimality except in Wasser- stein GAN (see Sec. [6. In Appx. H it is empirically demonstrated that discriminator trained on Wasserstein GAN tends not to satisty separability. We put * in the last column since injectivity not directly affected by objective functions but by other factors (see Appx. H for detailed explanations).
Direction optimality
Separability weak
Injectivity
Wasserstein GAN
GAN (Hinge, Saturating, Non soon-saturating) SAN (Hinge, Saturating, Non-saturating)
In this paper, we provide a unique perspective on GAN optimization that helps to consider whether discriminator is metrizable.
Definition 1.1 (Metrizable discriminator). Let po and be measures. Given an objective function [(;:-) for 6, a discriminator f (J,D)- D)- or J-metrizable for μs and po, if [(;; f) is minimized only with 0 E arg min, mino,D(po, (40 for certain distance on measures, D(,,.).
To evaluate the dissimilarity with a given GAN minimization problem J. we are interested in other conditions besides the discriminator's s optimality. Hence, we propose metrizable conditions, namely, direction optimality, separability, and injectivity, that induce a I -mmtriizalee discriminator. We first introduce a divergence, called functional mean divergence (FM*) in Sec. Sed connect the FM" with the minimization objective function of Wasserstein GAN. Then, we obtain the metrizable conditions for Wasserstein GAN by investigating Question [.2] We provide an answer this question See, by relating the FM to the concept of sliced optimal transport (Bonneel et al. 2015) Kolouri et al., 2019). Then, in Sec. 5 we formalize the proposed conditions for Wasserstein GAN and further extend the result to generic GAN.
Question 1.2. Under what conditions is FM a distance?
Based on the derived metrizable conditions, we propose the Slicing Adversarial Network (SAN) in Sec. 6 As seen in Table [ find that optimal discriminators for most existing GANs do not satisfy direction optimality. Hence, we develop a modification scheme for GAN maximization problems to enforce direction optimality on our discriminator. Owing to the scheme's simplicity, GANs can easily be converted to SANs. We conduct experiments to verify our perspective and demonstrate that SANs are superior to GANs in certain generation tasks synthetic and image datasets. In particular, we confirmed that SAN improves the state-of-the-art FID for conditional generation with StyleGAN-XI. (Sauer et al. 2022) CIFAR10 and ImageNet 256x despite the simple modifications.
2 PRELIMINARIES
2.1 NOTATIONS
We consider a sample space C D. and latent space C R.D: Let P(X) be the set of all probability measures on X, and let L (X. RD) denote the L space for functions X + RDD Let μ (or 2) represent a probability measure with probability density function In (or I,). Denote du (M, = Em-y [xxx]] Ey-u (xx)). We use the notation of the pushforward operator which is defined as 9: := ( for B E X with function
Z + X and a probability measure E P(Z). We denote the Euclidean inner product by (.,)). Lastly, (+) denotes a normalized vector.
Please also refer to Appx. A for some notations defined in the rest of the papers.
2.2 PROBLEM FORMULATION IN GANS
Assume that we have data obtained by discrete sampling from a target probability distribution μ4y E P(X). Then, we introduce a trainable generator function ge Z - X with parameter 0 E R Do to model a trainable probability measure as Pa = gor with a E P(Z). The aim of generative modeling here is to learn go that it approximates the target measure as p10 plo-
2

---

Published as a conference paper at ICLR 2024
For generative modeling in GAN, introduce the notion of discriminator F(X) C
(X, We formulate the GAN's optimization problem as a two-player game between the gen- erator and discriminator with V F(X) P(X) - R and J R F(X) + R, as follows:
v(f:pe) and min f) SEF())
max
Regarding the choices of V and J. there are GAN variants (Goodfellow al. 2014 Nowozin et al. 2016 Arjovsky et al., 2017) that lead to different dissimilarities between μoo and μo with the maximizer In this paper, we use representation of the discriminator an inner- aner-productt form:
f(x) = (w> h(z)),
(2)
where w E CDD- and h(x)) E L(X,, P). The form is naturally represented by neural network
2.3 WASSERSTEIN DISTANCE AND Irs USE FOR GANs
We consider the Wasserstein-p distance (Villani 2009) between probability measures μ and
inf f Wp(H, 2) == rEll(H,))) xxx
| - x' ='IIdd((2,
where p E [1, oo) and [[,, v) is the set of all coupling measures whose marginal distributions are a and The idea of Wasserstein GAN is to learn a generator by minimizing the Wasserstein- distance between P4 and μą. For this goal, one could adopt the Kantorovich-Ruubee duality representation rewrite Eq. (3 and obtain the following optimization problem:
(KR)
max (w( == dy (Mo: 11)) (EFu,(X)
(4)
where FLip denotes the class of I-Lipschitz functions. In contrast, we formulate an optimization problem for the generator as minimization of the right side of Eq. (4 w.r.t. the generator parameter:
Jw (0: f):= ---=--a. EEewga [xx)] DERPo
min
(5)
2.4 SLICED OPTIMAL TRANSPORT
The Wasserstein distance is highly intractable when the dimension D2 is large (Arjovsky et al. 2017). However, it is well known that sliced optimal transport can be applied to break this in- tractability by projecting the data on one-dimensional space. That is, for the D2 1 case, the Wasserstein distance has a closed-fome solution:
W,,, = (L,
(6)
where (-) denotes the quantile function for I,.. The closed- form solution for oie-dimensional space prompted the emergence of the concept of sliced optimal transport.
In the original sliced Wasserstein distance (SW) (Bonneel al. 2015). probability density function on the data space X is mapped to a probability density function of & E TR by the standard Radon transform (Natterer 2001 Helgason 2011) as RI(E, w) = fx 1 (x))(E
(zz, (zrw)) dzz where 8(-) is the Dirac delta function and E SD.-1 a direction. The sliced Wasserstein distance between and is defined as sw." (1, v) = (Suegn.-, Wr w), R.,, (.T((, (dww))) Intuitively, the idea behind this distance is to decompose high-dimensional distributions into an infinite number of pairs of tractable distributions by linear projections.
Various extensions of the sliced Wasserstein distance have been proposed (Kolouri et al. 2019 Deshpande et al. 2019 Nguyen al.] 2021). Here, we review extension called augmented sliced Wasserstein distance (ASW) (Chen et al. 2022). Given a measurable injective function
In practice, the discriminator is implemented as Eq. 2 c.g., fa(x) who (ooz 1 o loz, o 1, (x)) with nonlinear layers (lo, (os) Waz RD, and their weights == (6e) E Rdo
3

---

Published as a conference paper at ICLR 2024
RP, the distance is obtained via the spatial Radon transform (SRT), which is defined for any & E RR and w E gd-1, as follows:
8 IE) - 1(x))(5 (we, h())))dx.
The ASW-p is then obtained via the SRT in the same fashion as the standard SW:
ASW (M, = W In(,,w, S"h In (-,w))
W
(8)
Equation (6) can be used to evaluate the integrand Eq. which is usually evaluated via approx- imated quantile functions with sorted finite samples from S II( W) and 8" (',Ww.
3 FORMULATION OF QUESTION 1.2
We introduce a divergence called the functional mean divergence, FM or FM which is defined for given functional space function, respectively. Minimization the FM can be formulated as an optimization problem involving Jw. and we cast Question 1.2 in this context. In Sec. 4 we provide an answer this question, which turn provides the metrizable conditions with Tw Sec.
3.1 PROPOSED FRAMEWORK: FUNCTIONAL MEAN DIVERGENCE
We start by defining the FM with given functional space. Definition 3.1 (Functional Mean Divergence (FM)). We define a family of FMs as
DDM (1,)))
max (H, C (xF(X)
Further, we denote an instance in the family as FM v) E @DM where F(X) S (X,
By definition, the FM family includes the integral probability metric (IPM) (Müller 1997). which includes the Wasserstein distance KR form as a special case.
Proposition 3.2. For F(X) E L (X, (X,
IR. IPM=(:,))
max (. E 9
The FM an extension of the IPM to deal with vector-valued functional spaces. Although the FM with a properly selected functional space yields a distance between target distributions, the maximization in Eq. 4 is generally difficult to achieve. Instead, we use the following metric, which is defined for a given function.
Definition 3.3 (Functional Mean Divergence* (FM')). Given a functional space F(X) - (X, IR D), we define a family of FMs*
S
g = (1,v) (d, 2)ll2/h E F(X)).
(10)
Further, we denote an instance in the family as FM, (H, E 2FM", where h E F(X).
We are interested in Question 3.4 which is a mathematical formulation of Question 1.2 We give an answer to this question in Sec. Since optimization of the FM" is related to Tw Sec. 3.2 the conditions in Question 3.4 enable us to derive (Jw, FM,,) -eelrizable conditions in Sec. 5
Question 3.4. Under what conditions for F(X) E (X, RR)) are FM;,(..) E FM distances?
3.2 DIRECTION OPTIMALITY TO CONNECT FM" AND Jw
Optimization of the FM* with a given h E involving Jw.
returns to optimization problem
Proposition 3.5 (Direction optimality connects FM and Tw). Let be on SD-1, For any E (X, R).), minimization of (1g (o) equivalent to optimization of m max Tw(O; hw). Thus, VoFM, (1o, po) vaJw(0; (w*,h) ,h)), where wt is the optimal solution (direction) given as follows:
arg max d(w.n) (Mo,No). wE3D-1
(11)
4

---

Published as a conference paper at ICLR 2024
h(a))
[Lmmm 4.3)
(M 90%
we optimal ((2) k(a))
Wasserstin distance
7s(X)
W
E gD
Projection ento 3 with
F((X)
S fo (
plue (
Ill Lemma d:ll Every masrASWa// 9P%) is cistence
Fiee (8)
PPropositiin 4.51
Every FMC e S dn
Probubility density functions
Cumulrative density functions
Leuus 5.11
Discriminator decomposition into inner- product form (we, h(z)). Direction w projects h(z) onto R. In this figure, is separable because Fh (E) S plw (E) for all & E R (see Definition 4.2
Figure 1:
Figure 2: Outline of Sec. 4 Proposi- tion 4.5 is major step toward our main theorem.
The proof can be found in Appx. D where the key idea involves using the Cauchy- Sehhar-Schwart in- equality, I|da (No. (a)ll2 2 (w, dz (Ho. (ea) with w E SD-1, and the linearity of d, (Ho.Pe), leading to [|da(oo ua)ll d2 (No, (aa).
Recall that we formulated the discriminator the inner-product form 2). which is aligned with Proposition 3.5 Here, we introduced notion of optimal direction, which most distinguishes Jo and μe in the feature space given by h, intuitively (refer to Figure [ for illustration). We refer to the condition for the direction in Eq. (11 as direction optimality. It is obvious here that, given function h, the maximizer w" becomes dy (Ho+ Mo).
From the discussion this section, Proposition 3.5 supports the notion that investigating Ques-
tion 3.4 will reveal the (Jw, FM,) -metrizabee conditions.
4
CONDITIONS FOR METRIZABILITY: ANALYSIS BY MAX-ASW
4.1 STRATEGY FOR ANSWERING QUESTION 3.4
We consider the conditions of F(X) for Question 3.4 the context of sliced optimal transport. To this end, define variant of sliced optimal transport, called maximum augmented sliced Wasserstein divergence (max-ASW) in Definition 4. Sec. 4.2, we first introduce a condition called the separable condition, where divergences included in the FM family are also included in the maX-ASW family. In Sec. 4.3 we further introduce a condition called the injective condition, where the max-ASW is distance. Finally, imposing these conditions F(X) brings the desired conditions (see Fig. for the discussion flow).
Definition 4.1 (Maximum Augmented Sliced Wasserstein Divergence (maX-ASW)). Given a func- tional space F(X) C L (X, R),,, we define family of max-ASWs as
2TA = (U,D) + max W1 S. Ih WEBD-1
E()) F(X)
(12)
Further, we denote an instance of the family as max-ASW (H, gyr where F(X).
Note that although the formulation the definition includes the SRT, similarly to the ASW in Eq. (8), they differ in terms of the method of direction sampling (w).
4.2 Separability FOR EQUIVALENCE OF FM AND MAX-ASW
We introduce a property for the function h, called separability, which connects the FM and the max-ASW as in Lemma
Definition 4.2 (Separable). Given μ, v E P(X), let w be on SD-1, and let ph)((.) be the cumulative distribution function of Sh [p(,,). If = dr (p, v) satisfies Fl.w" (E) S Fhi (E) for any ç E TR, E L (X,R") RD) is separable for those probability measures. We denote the class of all these separable functions for them Fsta.,i or Fs for notation simplicity.
5

---

Published as a conference paper at ICLR 2024
Injectivity on ensures mar-ASW distance
Wasserstein GAN loss
min Jw f (w" ht) = min FM, (no, po) 0
min A.
Direction optimality on Separability on h Figure 3: An example with Wasserstein GAN loss: the metrizable conditions (direction optimality, separability, and injectivity) ensure that Wasserstein GAN loss evaluates the distance between data and generator distributions.
Lemma 4.3. Given u.u E P(X), every h E Fs(m,)) satisfies FM,, v) E 2zg.
Intuitively, separability ensures that optimal transport maps for all the samples from St Ino (:,w)
Sh no ((,w) share the same direction in Figure ! For general function E L (X, RD), -.) necessarily included in the max-ASW family, i.e., D(.,) (: $ D(,,)) O for general F(X) S L (X, P).. Given h, calculation of the max-ASW via Eq. (6) gen-
erally involves evaluating the sign of the difference between the quantile functions. Intuitively, the equivalence between the FM and maX-ASW distances holds if the sign is always positive regardless
of E [0, 1]: otherwise, the sign's dependence on breaks the equivalence.
4.3 INJECTIVITY FOR MAX-ASW TO BE DISTANCE
Imposing injectivity on guarantees that the induced max-ASW is indeed distance. Intuitively, injectivity prevents the loss of information from the original samples x 140 and x ~ μo-
Lemma 4.4. Every (-x.ASW.(...) E DTh is a distance, where Fi indicates a class of all the injective functions in (X,
Lemmas and 4.4 mmeedately tell us that (M, v) with a separable and injective h is indeed distance because it is included in the family of max-ASW distances, stated in Proposition 4.5 With the result, we now have one the answers to Question 3.4 which hints at our main theorem.
Proposition 4.5. Given v.v P(X), every FM, v) FFM is indeed distance, where Fis indicates Fi n Fs..
DOES GAN TRAINING ACTUALLY MINIMIZE DISTANCE?
We present Theorem 5.3 which is main theoretical result and gives sufficient conditions for the discriminator to be T 7-metrizable. This theorem inspires us to propose modification schemes for GAN training in Sec.
We directly apply the discussion in the previous section to Jw. and by extending the result for Wasserstein GAN general GAN, we derive the main result. First, simple combination of Propositions 3.5 and 4.5 yields the following lemma. Lemma 5.1 provides the conditions for the Jw-metrizable (see Fig.Bi 3g-3 how each condition works).
discriminator
Lemma 5.1 -ww-metrizable). Given h Fi with Fs S (X, let w E SD be dy (Mo,. pa). Then [(x) (w, h(x)) is (Jw, FM,, -metrribbe.......
Next, we generalize this result to more generic minimization problems. Our scope here is the mini- mization problems of general GANs that are formalized in the form [(: f) En [ f with Rg R + IR. We utilize the gradient of such minimization problems w.r.t. 6:
fx)]
VoJ (0; f) Eswa Irs
(13)
where r (-) is the derivative of (.)(),, as listed in Table By ignoring a scaling factor, Eq.
13 can
be regarded gradient of y 1 ,100 _ where 7o is defined via Igeas (æ) x ro (xx) (x).
To examine whether updating the generator with the gradient in Eq. (1) can minimize certain
distance between or and μig, we introduce the following lemma.
Lemma 5.2. For any p X + R, and a distance for probability measures D(.,)) D(pe, (0) indicates distance between l and

---

Published as a conference paper at ICLR 2024
f(x)
lu(x))
Decomposing (inner (inner-praduce product form)
Normalizing
11
last linear layer
Discriminator:
W E RRD
W E SD
Minimax prob.: maxy V(S:H)) m6) ming f):
Converting GAN to SAN
max (h3 ()) ( + A ("s (.)
minl [(;; (J)
Figure 4:
Converting GAN to SAN requires only simple modifications to discriminators.
We finally derive our main result, the J-metrizable conditions for μl and μe. by leveraging Lemma 5.2 and applying Propositions 3.5 and 4.5 to ph pozet and fo The theorem suggests that the discriminator not the optimal solution to the original maximization problem V.
serve distance between the generator and target distributions even if it is
Theorem 5.3 (J -Metrizabiility) Given a functional J(:; := En-ual [Ro fxx)) with R : - whose derivative, denoted R', is positive, let (X, R,) and E SP satisfy the following conditions. Then, f(x) = (w, h(r)) is I --metrizable for po and pa.
(Direction optimality) w = arg max (Po If'of -ft'sf and Pg
Fig
(Separability)) h is separable for Bg (Injectivity) h is injective function.
In particular, direction optimality is the most interesting aspect for investigating GAN among the three conditions. It should be noted that most existing GANs besides Wasserstein GAN do not sat- isfy direction optimality with the maximizer of V. which will be explained in Sec. 6 Therefore, we will focus on direction optimality in the next section. Please refer to Appx. H for a more detailed explanation and empirical investigation about the other conditions. Our observations regarding in- ductive effects of GAN losses to the metrizable conditions are summarized in Table I Note that injectivity cannot be directly controlled by loss designs mentioned in the table.
SLICING ADVERSARIAL NETWORK
This section describes proposed model, the Slicing Adversarial Network (SAN) to achieve the direction optimality of w in Theorem [.3 while keeping separability of h. We develop SAN by modifying the maximization problem V to guarantee that the optimal solution w achieves direction optimality. The proposed modification scheme is applicable to most existing GAN objectives and does not necessitate additional exhaustive hyperparameter tuning or computational complexity.
As mentioned in Sec. 5 given a function h, the maximization problems in most GANs (exclud- ing Wasserstein GAN) cannot achieve direction optimality with the maximum solution of V (see Table We use hinge GAN as an example to illustrate this claim. The objective function to be maximized in hinge GAN is formulated as
Vitinge (ww, h);; (b) En-po [min((, -1 (wi, hxx)() En-yo (min(0, -1 (w, ((x))))- (14)
Given h, the maximizer w becomes dr (Hō. ug),, where nijf and μi denote truncated distributions h(x)) < 1 and (w, l(x))) > -1. respectively. whose supports are restricted by conditioning 1- F1 (Ho. pe). the maximum solution does not Since (Hg (z] is generally different from (F1
satisfy direction optimality for 3 and j
In line with the above discussion, we propose the following novel maximization problem:
max
(SAN(W, (w) = V((W (h) -A diwnt (1th Pg -750f
(15)
WEBA- (EF(X)
5h (h(aaaa))
L (wih.pp)
where (.)- indicates dtop-gradient operator. The first and second terms induce separability on h and direction optimality on w, respectively. We simply set X E R, to in our experiments. The proposed modification scheme in Eq. enables select any maximization objective for h with

---

Published as a conference paper at ICLR 2024
Table Minimization problem and weighting function for direction optimization.
Minimization problem
Weighting ry f(x)
Wasserstein GAN Hinge GAN
Eawyg [[x)]
Saturating GAN Non-saturating GAN
Rasip [log (f(())) E Rizwa [log c(1 f(x))
1 1-s(f(>)) s(f()))
-ee-ing: GAN
-00- GAN Noe-on
Hinge
Saturating
Non-saturating
- -1ege SAN -0 Sataning Nra-akmiing SAN
0.9
GAN
Inner product
0.81
0.5
0.5
0.3
SAN
0.2
10000
Iteratioon
Figure 5: Comparison of the learned distributions (at 10,000 iterations) between GAN and SAN with various objectives. all cases, SANs cover all modes whereas mode collapse occurs in some GAN cases.
Figure 6: Inner product of trained w and numerically estimated optimal direction for 140 and /i during training. The trained w were closer to the optimal di- rection with SAN than with GAN.
Ch while L" enforces direction optimality on w. Hence, we can convert most GAN maximization problems into SAN problems by using Eq. (15) (also see Algorithm I One also easily extend SAN to class conditional generation by introducing directions for respective classes (see Appx. C).
A key strength of SAN is that the model can be trained with just two small modifications existing GAN implementations (see Fig. 4 and Appx. B}. The first is to implement the discriminator" 's final linear layer to be on a hypersphere, i.e., (x) W.p.hepr (z) with Wer @gn1 where po the parameter for the /th layer. The second is to use the maximization problem defined in Eq. (15).
hypr,
Comparison with other generative models. First, we compare SAN with GAN. In GAN, other than Wasserstein GAN, the optimal discriminator is known to involve a dilemma: it induces certain dissimilarity but leads to exploding and vanishing gradients (Arjovsky et al. 2017 Lin al. 2021)
In contrast, in SAN, the metrizable conditions ensure that the optimal discriminator is not necessary to obtain a metrizable discriminator. Next, similarly to variants of the maximum SW (Deshpande) et al. 2019 Kolouri et al., 2019). SAN's discriminator is interpreted and shiing them in the most distinctive direction w as in Fig. In the conventional methods, one- dimensional Wasserstein distances
extracting features h(xr)
approximated by sample sorting (see Remark (4.2). It has been reported that the generation performances get better when the batch size increases (Nguyen let al.) (202T; [Cen al. (2022), but larger batch sizes lead slower training speeds, and the size is capped by memory constraints. In contrast, SAN removes the necessity of the approximation.
7 EXPERIMENTS
We perform experiments with synthetic and image datasets to (1) verify our perspective on GANs as presented in Sec. 5 terms direction optimality, separability, and injectivity, and (2) show the effectiveness of SAN against GAN. For fair comparisons, we essentially use the same architectures in SAN and GAN. However, we modify the last linear layer of SAN's discriminators (see Sec. T).
7.1 MIXTURE OF GAUSSIANS
To empirically investigate optimal direction, we conduct experiments mixture of Gaussian (MoG). Please refer to Appx. for empirical verification of the implications of Theorem 5.3 re- garding separability and injectivity. We use two-dimensional sample space X R The target MoG on comprises eight isotropic Gaussians with variances 0.05² and means distributed evenly on circle of radius 1.0. We utilize 10-dimensional latent space Z to model generator measure.
8

---

Published as a conference paper at ICLR 2024
Table 3: FID scores (U) on DCGAN.
Hinge loss
Saturating loss
Non- Non-saturating loss
Dataset
GAN
SAN
GAN
SAN
GAN
SAN
CIFARTO
24.07
0..2
98
20.62.10.44
04.90-0.21
20.31
CelebA
32.51+2.53
27.79
1..0
28.16
28.22
27.78
Table 4: FID and IS results with the experimental setup of BigGAN (Brock et al. 2019). Scores
marked with * are results from our implementation, which is based on the BigGAN authors' PyTorch implementation. For reference, scores reported in their paper are denoted with f.
CIFARTO
CIFARTOD
Metric
Hinge GAN
Hinge GAN"
Hinge SAN'
Hinge GAN
Hinge SAN'
FID (1) IS (t)
14.73
8,25
6.20
10.7310.16
8.05
9.22
9.05+0055
9.16-00.8
10.5640011
10.72+0.05
Table 5: Numerical results for StyleGAN-XL (Suuer et al. 2022) and StyleSAN-XL XL. Scores marked with i and & are reported in their paper and repository, respectively. Note that our StyleSAN-XI. model trained on CIFARI0 is larger in model size than StyleGAN-XL.
CIFARTO
ImageNet (256x256)
Method
StyleGAN-XI.
StyleSAN-XI.'
StyleGAN-XI. StyleSAN-XI."
FID (L)
(1.85)
1.36
2.30
2.14
IS (t)
265.12
274.20
We compare SAN and GAN with various objectives. As shown in Fig. the generator measures trained with SAN cover all modes, whereas mode collapse (Srivastava et al. 2017) occurs with hinge GAN and non- non-saturating GAN. In addition, Fig. shows a plot of the inner product of the learned direction w (or the normalized weight in GAN's last linear layer) and the estimated optimal direction for 2 and Pig".
Recall that there is no guarantee that non-optimal direction induces distance.
7.2 IMAGE GENERATION
We apply the SAN scheme to image generation tasks to show it scales beyond toy experiments.
DCGAN. We train SANs and GANs with various objective functions on CIFAR10 (Krizhevsky
et al.
2009) and CelebA (128x 128) (Liu et al. 2015). We adopt the DCGAN architectures (Radford apply spectral normalization (Miyato et al. 2018). As shown in Table SANs outperform GANs in terms of the FID score in all cases.
al. 2016), and for the discriminator,
BigGAN. Next, we apply SAN to BigGAN (Brock et al. 2019) on CIFARIO and CIFARI00. In this experiment, we calculate the Inception Score (IS) (Salimans et al., 2016), as well as the FID, evaluate the sample quality by following the experiment the original paper. As in Table 4 the adoption of SAN consistently improves the generation performance in terms of both metrics.
StyleGAN-XL. Lastly, we apply our SAN training framework to StyleGAN-XL (Sauer al. 2022 which is a state-of-the-art GAN-based generative model. We name the StyleGAN-XL model com- bined with our SAN training framework StyleSAN-XL. We train StyleSAN- on CIFAR10 and ImageNet (256x256) (Russakovsky et al. 2015), and report the numerical results, with BigGAN. As shown in T our models improve the state-of-the-art scores on both cases.
CONCLUSION
We have proposed unique perspective on GANs to derive sufficient conditions for the discrimi- nator to serve as distance between the data and generator probability measures. To this end, we introduced the FM" and max-ASW families. By using class of metries that are included both families, we derived the metrizable conditions for Wasserstein GAN. We then extended the result to a general GAN. The derived conditions consist of direction optimality, separability, and injectivity.
We then leveraged the theoretical results to develop the Slicing Adversarial Network (SAN), in which a generator and discriminator are trained with a modified GAN training scheme. Despite the ease of modifications and the generality, this model can impose direction optimality the discriminator. Our experiments on synthetic and image datasets showed that SANs outperformed GANs in terms of both the sample quality and mode coverage.
9

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
The experimental setups are provided in Appx. with a detailed description the network ar- chitectures and hyperparameters. We further provide source code to reproduce results at https: g/g thub. com/ sony/san, Our implementation for all the image generation tasks is based on open-source repositories. The proof' of all the theoretical claims can be found in Appx. [
ACKNOWLEDGEMENTS
We sincerely acknowledge the support of Basavaraj Murali, Riya Kuriakose, Sonal Monteiro, Srinidhi Srinivasa, and Takuya Narihira, who assisted with our experiments during the rebuttal pe- riod. Computational resouree of AI Bridging Cloud Infrastructure (ABCI) provided by National Institute of Advanced Industrial Science and Technology (AIST) was used.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875. 2017.
Wasserstein GAN. arXiv preprint
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (GANs). In Proc. International Conference on Machine Learning (ICML), volume 70, pp. 224-232, 2017.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In Proc. International Conference on Machine Learning (ICML), pp. 573-52,.. 2019.
Hugo Berard, Gauthier Gidel, Amjad Almahairi, Pascal Vincent, and Simon Lacoste-Julien. A closer look at the optimization landscapes of generative adversarial networks. In Proc. International Conference Learning Representation (ICLR), 2020.
Clément Bonet, Paul Berg, Nicolas Courty, François Septier, Lucas Drumetz, and Minh- Tan Pham. Spherical biced-wassersteinn In Proc. International Conference on Learning Representation (ICLR), 2023.
Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and radon Wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 51(1)222-45, 2015.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In Proc. International Conference Learming Representation (ICLR), 2019.
Robin Chan, Sarina Penquitt, and Hanno Gottschalk. Lu-net: Invertible neural networks based on matrix factorization. arXiv preprint arXiv:2302. 10524, 2023.
Tatjana Chavdarova, Gauthier Gidel, François Fleuret, and Simon Lacoste-Julien. Reducing noise in gan training with variance reduced extragradient. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Tatjana Chavdarova, Matteo Pagliardini, Sebastian U Stich, François Fleuret, and Martin Jaggi. Taming gans with aakkaahad-minmaal In Proc. International Conference Learning Represen- tation (ICLR), 2022.
Xiongjie Chen, Yongxin Yang, and Yunpeng Li. Augmented sliced wasserstein distances. In Proc. International Conference Learning Representation (ICLR), 2022.
Casey Chu, Kentaro Minami, and Kenji Fukumizu. Smoothness and stability in GANs. In Proc. International Conference on Learning Representation (ICLR), 2020.
Ishan Deshpande, Yuan- Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth. and Alexander G Schwing. Max-sliced Wasserstein distance and its use for GANs. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10648-10658 2019.
10

---

Published as a conference paper at ICLR 2024
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In Proc. Inter- national Conference on Learning Representation (ICLR), 2019.
Jiaojiao Fan, Qinsheng Zhang, Amirhossein Taghvaei, and Yongxin Chen. Variational wasserstein gradient flow. In Proc. International Conference on Machine Learning (ICML), volume 162, pp. 6185-6215, 2022.
Farzan Farnia and Asuman Ozdaglar. Do GANs always have nash equilibria? In Proc. International Conference on Machine Learning (ICML), pp. 3029-3039, 2020.
Tanner Fiez, Chi Jin, Praneeth Netrapalli, and Lillian Ratliff. Minimax optimization with smooth algorithmic adversaries. Proc. International Conference on Learning Representation (ICLR), 2022.
Yuan Gao, Yuling Jiao, Yang Wang, Yao Wang, Can Yang, and Shunkang Zhang. Deep generative learning via variational gradient flow. In Proc. International Conference on Machine Learning (ICML). pp. 2093-21011 2019.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 2672-2680, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training Wasserstein GANs. Proc. Advances in Neural Information Processing Sys- tems (NeurIPS), pp. 5767-5777, 2017.
Zekun Hao, Arun Mallya, Serge Belongie, and Ming-Yu Liu. GANeraft: Unsupervised 3d neural rendering of minecraft worlds. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14072-14088 14082, 2021.
Sigurdur Helgason. The Radon transform on Rn. Integral Geometry and Radon Transforms, pp- 1-62. Springer, 2011.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with constrained variational framework. In Proc. International Conference on Learning Representation (ICLR), 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 6840-6851, 2020.
Ya-Ping Hsich, Chen Liu, and Volkan Cevher. Finding mixed nash equilibria of generative adversar- ial networks. Proc. International Conference on Machine Learning (ICML), pp. 2810-2819, 2019.
Chi Jin, Praneeth Netrapalli, and Michael Jordan. What local optimality in minimax optimization? In Proc. International Conference Machine Learning (ICMI), volume 119, pp. 4880-4889, 2020.
Rie Johnson and Tong Zhang. A framework of composite functional gradient methods for generative adversarial models. IEEE transactions on pattern analysis and machine intelligence, 43(1):17- 32, 2019.
Michael Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine Learning, 37(2):183-233, 1999.
Mahdi Karami, Dale Schuurmans, Jascha Sohl-Dickstein, Laurent Dinh, and Daniel Duckworth. Invertible convolutional flow. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. Proc. International Conference on Learning Represen- fation (ICIR), 2018.
11

---

Published as a conference paper at ICLR 2024
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative ad- versarial networks. In Proc. JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8110-8111 (00-8119,2200 2020.
Tero Karras, Miik Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free free generative adversarial networks. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. Interna- tional Conference on Learning Representation (ICIR), 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. In Proc. International Conference on Learning Representation (ICLR), 2014.
Soheil Kolouri. Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde. General- ized sliced Wasserstein distances. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. HiFi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 17022-17033, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. MelGAN: Generative adver- sarial networks for conditional waveform synthesis. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of
imagenet classes in fréchet inception distance. In Proc. International Conference on Learning Representation (ICLR), 2023.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, (6(11):2278-21 2324, 1998.
Jose Lezama, Wei Chen, and Qiang Qiu. Run-sort-rerun: Escaping batch size limitations in sliced wasserstein generative models. In Proc. International Conference on Machine Learning (ICML), volume 139, pp. 2275-6285, 2021.
Jerry Li, Aleksander Madry, John Peebles, and Ludwig Schmidt. On the limitations of first-order approximation in gan dynamics. In Proc. International Conference on Machine Learning (ICML), pp. 3005-3013, 2018.
Jae Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint arXiv: 1705:007900844 2017.
Zinan Lin, Vyas Sekar, and Giulia Fanti. Why spectral normalization stabilizes GANs: Analysis and improvements. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 9625-9638, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3730-3738, 2015.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of gans. In Proc. Advances in Neural Information Processing Systems (NeurlPS), volume 30, 2017.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for GANs do actually converge? In Proc. International Conference on Machine Learning (ICML), pp. 3481- 3490, 2018
12

---

Published as a conference paper at ICLR 2024
Paul Milgrom and Ilya Segal. Envelope theorems for arbitrary choice sets. Econometrica, 70(2): 583-601, 2002.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. In Proc. International Conference on Learning Representation (ICLR), 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In Proc. International Conference on Learning Representation (ICLR), 2018.
Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429-44 443, 1997.
Vaishnavh Nagarajan and J Zico Kolter. Gradient descent GAN optimization is locally stable. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017.
Frank Natterer. The mathematics of computerized tomography. SIAM, 2001.
Khai Nguyen and Nhat Ho. Revisiting sliced wasserstein on images: From vectorization to convo- lution. In Proc. Advances Neural Information Processing Systems (NeurlPS), volume 35, pp- 2022.
Khai Nguyen, Nhat Ho, Tung Pham, and Hung Bui. Distributional sliced-Waad Wasserstein and appli- cations to generative modeling. In Proc. International Conference on Learning Representation (ICLR), 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proc. International Conference on Machine Learning (ICML), pp. 8162-8171. PMLR, 2021.
Atsushi Nitanda and Taiji Suzuki. Gradient layer: Enhancing the convergence of adversarial training for generative models. In Proc. International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 1008-1016, 2018.
Sebastian
Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training generative neural sam- plers using variational divergence minimization. Proc. Advances in Neural Information Pro- cessing Systems (NeurIPS), volume 29, 2016.
Michael Puthawala, Konik Kothari, Matti Lassas, Ivan Dokmanić, and Maarten De Hoop. Globally injective relu networks. The Journal of Machine Learning Research, 23(1):4544 4598, 2022.
Chongli Qin, Yan Wu, Jost Tobias Springenberg, Andrew Brock, Jeff Donahue, Timothy P Lillicrap, and Pushmeet Kohli. Training generative adversarial networks by solving ordinary differential equations. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. $599-5609 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proc. International Conference Learning Representation (ICLR), 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. Proc. In- ternational Conference on Machine Learning (ICML), pp. 7768-8763, 2021.
Lillian Ratliff. Samuel A Burden, and S Shankar Sastry. Characterization and computation of local nash equilibria in continuous games. In Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 917 924. IEEE, 2013.
Danilo Jimenez. Rezende and Shakir Mohamed. Variational inference with normalizing flows. In
Proc. International Conference Machine Learning (ICML), pp. 1530-1538, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 15:2111252, 2015.
13

---

Published as a conference paper at ICLR 2024
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Proc. Advances Neural Information Processing Systems (NeurIPS), volume 29, 2016.
Maziar Sanjabi, Jimmy Ba, Meisam Razaviyayn, and Jason D Lee. On the convergence and robust- ness of training gans with regularized optimal transport. In Proc. International Conference on Machine Learning (ICML), volume 31, 2018.
Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 17480-17492 2021.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. In Proc. International Conference on Machine Learning (ICML., 2023.
Samarth Sinha, Anirudh Goyal, Colin Raffel, and Augustus Odena. Top-k training of GANs: Im- proving generators by making critics less critical. In Proc. Advances Neural Information Pro- cessing Systems (NeurIPS), 2020.
Jiaming Song and Stefano Ermon. Bridging the gap between f-gans and wasserstein gans. In Pruc. International Conference on Machine Learning (ICML), pp. 9078-9087, 2020.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Proc. International Conference Learning Representation (ICLR), 2020.
Yang Song, Chenlin Meng, and Stefano Ermon. Mintnet: Building invertible neural networks with masked convolutions. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. VEEGAN: Reducing mode collapse in gans using implicit variational learning. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proc. IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 2818-2822 2016.
Esteban G Tabak and Cristina V Turner. A family of nonparametric density estimation algorithms. Communicationn on Pure and Applied Mathematics, 66(2):145-164, 2013.
Esteban Tabak and Eric Vanden Eijnden. Density estimation by dual ascent of the log-likelihood. Communications in Mathematical Sciences, 8(1) 8(1):217-233, 2010.
Yuhta Takida, Takashi Shibuya, WeiHsiang Liao, Chich-Hsin Lai, Junki Ohmura, Toshimitsu Ue- saka, Naoki Murata, Takahashi Shusuke, Toshiyuki Kumakura, and Yuki Mitsufuji. SQ-VAE: Variational Bayes on discrete representation with self-annealed stochastic quantization. In Proc. International Conference on Machine Learning (ICML), 2022.
Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural net- works. In Proc. International Conference on Machine Learning (ICML), pp. 6105-6114, 2019.
Dávid Terjčk. Adversarial lipschitz regularization. In Proc. International Conference on Learning Representation (ICLR), 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and
Herve Jegou. Training data-efficient image transformers amp; distillation through attention. In Proc. International Conference Machine Learning (ICML), pp. 10347-10357, 2021.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In Proc. IEEE Conference on Computer Vision and Pattenn Recognition (CVPR), pp. 1526-1535, 2018.
14

---

Published as a conference paper at ICLR 2024
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Kun Xu, Chongxuan Li, Jun Zhu, and Bo Zhang. Understanding and stabilizing GANs' training dynamics with control theory. In Proc. International Conference on Machine Learning (ICML), pp. 10566-10575, 2020.
Yasin Yaz, Chuan Saann-Shnggaa Foo, Stefan Winkler, Kim-Hui Yap, Georgios Piliouras, Vijay Chan- drasekhar, et al. The unusual effectiveness of averaging in gan training. In Proc. International Conference on Learning Representation (ICLR), 2019.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Proc. International Conference Machine Learning (ICML), pp. 7354- 7363, 2019.
Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. In Proc. International Conference on Learning Representation (ICLR), 2020.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. InfoVAE, Balancing learning and inference in variational autoencoders. Prec. AAAI Conference on Artificial Intelligence (AAAI), pp. 5885- 5892, 2019.
15

---

Contents
A Notations
16
B
Converting GANs to SANS
16
Class Conditional SAN
17
D
Proofs
18
E
Related Work
22
F.1
Literature on GAN
22
E.2
Literature Sliced Wasserstein Distance
23
E.3
Limitations
23
F
Experimental Setup
24
F.I
Mixture of Gaussians
24
F22
Image Generation
24
G
Supplementary Experiments
27
G.1
BigGAN
27
G.2
StyleGAN XL on FFHQ
27
H
Investigation of Separability and Injectivity
28
H.1
Inductions for Injectivity
28
H.2
Mixture of Gaussians
H.3 Vision Datasets
29
Investigation of VGAN in SW
30
Generated Samples
30
A NOTATIONS
We summarize several notations that are used throughout the paper but not introduced in Sec. 2.1 in
Table 6
B CONVERTING GANs TO SANS
We can convert GANs SANs simply by adding two modifications to existing GAN implementa- tions, as shown in Fig. 4 First, we propose to modify the last linear layer of the discriminator as the following PyTorch-like code snippet. In the code snippet, feature_network represents any feature extractor f(-)) built with nonlinear layers. On top of it, propose using the novel max- imization objective functions _h and L The discriminator's 's outputs, out out_ and out dir, utilized for h and L respectively.
import torch. nn as nn
class GANDiscriminates inn.Medle))
def _init_ _init_(seff, dim dimension of output (b(x)"
):
super (GANDiscriminator,, self) init_ ()
self main feature_netwer netp:ennettrrkkcccmc (dim) nonlinear function "h (x)
10
self.fc nn. Linear (dim, 1) # last linear layer "w"
11
12
def forward(self,
x batch of real ffae samples
16

---

Published as a conference paper at ICLR 2024
feature self. main (x) out self. (feature)
16
17
18
return out
Listing 1: GAN discriminator
import torch
import torch.nn import torch.nn. functional as
class SANciscriminatan (nn.Module):
def fist_(sseff,
dim dimension of output of "b(x)
):
super (SANDiscrimmnattoo self) self)._inite init_() ()
10
self.main feature_network (dim) nonlinear function "h (x)
init torch. randn (dim, initialization for direction self.vec nn. Parameter (vec_init) unnormalizee direction
13
14
15
def (sewa
batch real/fake fake samples
17
):
18
feature self.main(* (x)
19
rirection F. n.rormali ze (self. f. vec, dim-0) direction "omega" out_fun feature directien den.detachh ()
out_dir baatureddetact () @ direction
22
23
dir return out fun, out
Listing 2: SAN discriminator
As an example, converting the maximization objective of Hinge GAN to SAN leads to
VSAN
Hingo (w. h po) = Viinnge (w ,1),, 140) +λ Vw((us, h)) p0)
(16)
8"
For general GANs, the approximations of po and pi p needed. The objective term L can be formulated as
1 EErnools o of (x)(w, (x) Zo
1
L" h, (b)
-B-nnolr o (x)
(17)
Zo
where Zo Exzgan [rJ o f-(x)) and Zo = Eynny [r] (x)). The normalizing terms Zo and Zo can be approximated per batch. However, we found it beneficial to use the following auxiliary objective function instead:
Cammpee (w: h, Ho) Ez~juy [Ry ((w, h))] EEz [Ry (Rv, ((w,))))) h))],
(18)
which ignores the normalization terms Zn and Zo in Eq. 117 The psuedo code can found in Algo- r
C CLASS CONDITIONAL SAN
SAN can easily be extended to conditional generation. Let C' be the number of classes. For conditional case, adopt the usual generator parameterization, go Z {c) + which gives a conditional generator distribution μА := go(,, c)ra. For the discriminator, we prepare trainable directions for the respective classes Ω == [ Under this setup, we define the minimax problem of conditional SAN as follows:
max Bew 02h
(YAAN (wor h; n8)
and min (ee [J (Wo, h:)]
(19)
17

---

Published as a conference paper at ICLR 2024
Table 6: Several notations used throughout the main paper.
Sliced optimal transport
RIn(sw))
The RT of probability density function with direction CSP- given.
The SRT of probability density function / with direction E SP-1 and function X R given.
())
The cumulative distribution function of S* (,(,)))
Functional spaces
F(X)
Any functional space included in L ~ (X,
(X)
The class of all the separable functions for j4, / E P(X), which is often simply denoted as Fs. The class of all the injectivity functions in L (X, which often simply denoted as Fi. The intersection set of Fi and Fs, i.c. Fis Fi
Fi(X)
Fi)(X)
Divergences and distances
The family of FM with D E N given. Its instance is denoted as FM, v) E DD
RM
The family of FM" with F given. Its instance is denoted FM, (H, v) E 2 with EF. The family of max-ASW with F given. Its instance is denoted as max-ASW, E 2 with h E F.
27
Algorithm Training SAN (the blue lines indicate modified steps against GAN training)
Require: Consider sample space X and latent space Z. A target distribution l40 E P(X) and base distribution a E (Z()) are given. Initialize parameter modeling a generator function as go Z X, and discriminator parameters, o and w gD-1, modeling a discriminator function as (w, hp), where h2 : X + Rº.. Step sizes for the trainable parameters are set as 77. npp and n Original minimization and maximization problems for GANs, denoted JGAN and VGAN.
are given.
while not converged do
/+ Discriminator update */
for certain number of steps do
Sample minibatch of data {r:M11 ~ Ao (i.idd).. N where y; = =go(=z) with { (i.i.d).
Get generated samples {
Compute L as the Monte-Carke Carlo estimate of VaNN( (w, h); 0) with {ai LM and {ue
Compute L" as the Monte-Carlo estimate of Eq. 18 with {x:) and (u
Update the parameters end for
Projgo-1 ( V [") and o t xp + Top V.C..
/s Generator update #/
for certain number of steps do
Get generated samples {y where y; = go())) with { a (i.i.d.).
Compute J as the Monte- Monte-Carlo estimate of JGAN (w, h)) with )
Update the parameters as +- 0 110 V.J.
end for end while
where P(c) is a pre-fix probability mass function for c, and pū is the target probability measure conditioned on
GAN can easily be converted into SAN even for conditional generation, as many previous works have implemented a conditional discriminator with the class-conditional last layer as w Th. hrve (x) (Miyato Koyama 2018 Miyato et al. 2018 Brock et al. 2019: Zhang et al. 2019).
D PROOFS
Proposition 3.2
For F(X) E (X, (PMx(s)) max d 2da
Proof. The claim is direct consequence of the definitions of the IPM and the FM.
18

---

Published as a conference paper at ICLR 2024
We introduce a lemma, which will come in handy for the proofs of Proposition 3.5 and Lemma 4.3
Lemma D.1. Given function h E L (X, and probability measures M, E P(X), we huve
(1, max WESD
d(w.A) (),
(20)
Proof. From the Cauchy-Schward Schwarz inequality, we have the following lower bound using w E SD1:
||da (1,, 2)1l2
|| [2)] Be~ (h(x)]1l2
ス (w, E [xx2]] - (x (x())),
(21)
where the equality holds if and only if = d(p v). Recall that (.) denotes a normalization operator. Then, calculating the norm l|dy (H, 2/) formulated the following maximization problem:
|| 2)l22 = max WESD-1
Ex,ualss h(x))1 Ee h(x))]
(22)
max WEBD-1
d(w,)) (1,
(23)
Note that we can interchange the inner product and the expectation operator by the independency of w.r.t. x.
Let be on SD-1 For any h E L (X,X) minimization of FM,, (Ho, Po) is equivalent to optimization of mingge max Tw(0: (w, h.). Thus,
Proposition 3.5
V MyFMi po) = VoJw(0; (w", h)),,
(24)
where w is the optimal solution (direction) given as follows:
w* arg max d(w,)) (Pos Mo)-
(25)
wE8n-1
Proof. From Lemma D.I
we have
(H):Hll)) = max D-
d(w.)) (Ho: po)
(26)
max WESD
Euzapo [(w, h(x))) Exwno (ow h(x))))
(27)
By simple envelope theorem Milgrom & Segal 2002). obtain
V Volld (o)ll2
(Eur Iw"", h(x)))] Ex I(w", h(x))))),
(28)
where at max d) (No: no). Since the first term the right-hand side of Eq. 28 does not depend on taking the gradient of [|c,, (No, (o) w.r.t. 0 becomes
Vollda (Ho; po)ll2 = - VoRaswpole ,((x))]-
(29)
Substituting the definition of Tw and FM, (:)) into Eq.
29 leads to the equality in the statement of
Proposition 3.5
Lemma
4.3
Given p, E P(X), every E (X) satisfies FM, (M, v) E Dmn
Proof. From Lemma D.I
we have
||dn (1.22 = max ExwpllW, h(())](()]]]))))(() - E Kw. (wwh(x)) WERD-1
(30)
"Eznullw* (/,,,("))))))))))))))))))))))))) Eynrll"* h(x))))
(31)
where the maximizer is w* = dr (1, v).
19

---

Published as a conference paper at ICLR 2024
The expected values reformulated by the SRT as
Bewallus (xx))) = Sxx (JB 5ō(E - (oelt,/ntt. h(2))dd I (a))d
(32)
1 & (1 (x)))((5- (w, (w. M()))) dg
(33)
= -,
(34)
The interchange of the integral follows from Fubini's 's theorem. Here, the applicability of the theorem is justified by the absolute integrability of the integrant function. That is, fx [1((2))2dp((() v
[| (XRe)_= = ( < 00.
Further, the right-hand side of Eq. follows the representation of expectation by an integral of quantiles:
Sh (, [E] = L. 2h
(35)
By substituting Eqs. (34) and 35 into Eq. (3Ip, we obtain
da (1, 122 L. (Q (0) Qb (p)) dp.
(36)
The right-hand hand side of Eq. 36
bounded above:
(u, 1)112 S f Qh (0) Quaa (p) dp.
(37)
The equality holds if and only if Q/ (p) Qu 1 (p) 2 0 for all oor Qhi (p) Q/a < for
all p. By considering the fact that selecting Tw changes the sign and Eq. (36) is non-negative, the
necessary and sufficient condition for the equality becomes Qha" (p) Q (p) 0 for E [0, 1].
From the monotonicity quantile functions, the condition for the equality is written as F (p) S Fh(((p)
We here borrow lemma from Chen al. (2022). which is used in the proof of Lemma 4.4 Lemma D.2 (Chen et al. (0022), Lemma 1). Given an injective function h. X + RD and two probability measures μ1, P(X), for all & E IR and w SD1 I'"II( S
(",L(E w) if and
only if V.
Every max-ASWa(ss)) Dnd is distance, where Fi indicates a class of injective functions (X,,)
Lemma
4.4
Proof. One prove this claim by following similar procedures to the proof of Proposition 1 in Kolouri et al. (2019).
First, the non-negativity and symmetry properties are immediately proven by the fact that the Wasserstein distance is indeed a distance satisfying non-negatibity and symmetry.
Next, let denote
w" arg max W -E8D-1
LS (sw),, (-,w))
(38)
Pick P(X) arbitrarily. The triangle inequality with mar-ASWal is satisfied since (HneASWaann,
W Wi(S (5 (m
max WEBD-1
WS (,,w), TT
S W (,,"", ) TT
w') + (S ((,") (hr
(39)
I max WERD
Wh T (:w), ()):w")))
+ max WSth (,('") w),, Shh Iup w))
(40)
wERD-1
mmxx-ASW,,,L, max-ASWW P2)-
(41)
20

---

Published as a conference paper at ICLR 2024
We next show the identity of indiscemibles. For any ,4, from (i(,, O, have = 0; therefore, max-ASW, (jt, p)) = O. Conversely, max-ASW, is equivalent to 8 [,(,,w) = 1. w) for any w E SD-1, This holds if and only if f = from the injectivity assumption on h and Lemma D.2 Therefore, max-ASW, (1,, 0 holds if and only μ =.
Proposition 4.5 indicates m.
Given p.v P(X), every FMR(U.V) E D indeed distance, where Fis
Proof. The claim is direct consequence of the results of Lemmas 4.3 and 4.4
v) E Dmn holds for any D(u, v) A At the same time, from Fi,s Fi and Lemma 4.4 D(u, D is distance. Thus, the claim is proven.
From Fiss == Fi C Fs and Lemma 4.3 D ((u,))
Lemma 5.1
Given FiFFs with Fi, S LXx let w E eDD1 be dr (uo pa). Then f(z) = w hx)) is (Jw, FM,.)
Proof.
From Proposition 4.5 given that injective and separable for μo and μes FM,, (Po, (ea) is
distance between μ4g and μaa
From Proposition
3.5 minimizing FM,, (Ho, ue) w.r.t 0 is equivalent to optimizing the following: min Jw (0, (h)),, #
(42)
where w" the maximizer of d(w.,n k) (po: pa), i.c., w" = (1o, Now, FM,, (Ho, (4) is distance, so x + (w",, h(x))) (Jw. FMG mettrizblle for μo and po-
Lemma 5.2 For any p X R+ and distance for probability measures D(.,)), D(je w0) indicates a distance between μ and v.
Proof. For notational simplicity, we denote D(je, DP) as Dp(u,, v). First, the non-negativity and symmetry properties are immediately proven by the assumption that D(:,) is distance.
The triangle inequality regarding Dp(: is satisfied since the following inequality holds for any (1,, U2; p1 E P(X):
Dp (2) D( S D(7 ( + D(GG, (2))
(43)
where [zς(2) (z) x p (x) for i = 1,2,3. Now, from the positivity assumption on I + Ipe is invertible. In particular, one recover μ4 from ji uniquely by
(x((2)
Tm (x) p(x) Ix d44())
(44)
Thus, with the μr given by Eq.
44 we have
Dp Ha) D(Ri, (i= 1,2).
(B))
(45)
From Eqs.
43 and
45
we have
Dolus < D.(ui. + (us, u2).
(46)
Obviously, (M, n) = 0. Now, (u, p) = 0 is equivalent to jrP = iP. Here, from the invertibility of T Iae again, jJP 5e if and only if
Thus, D Dp(u, v) = -
Theorem 5.3. Given a functional J(:: f) = Exwno Ro f()) with R. + R whose derivative, denoted as R', is positive, let E L (X, RD) and w E SD-1 satisfy the following conditions. Then, f(x h(x)) is J-metrizable for pa and pg-
21

---

Published as a conference paper at ICLR 2024
(Direction optimality) arg max -a (a (To of
-ig of
(Separability) h is separable for Po R'os and P4 R'of (Injectivity) h is an injective function.
Then, -+ (w, h(x)) metrizable with for ps and po-
Proof. For notational simplicity, we omit the superscripts of pi and jig Lemma 5.2 if (Mo, Ho) is distance between fo and Po, then it is also distance between μo and pr.
of in this proof. From
Following the same procedures in Lemma 5.1 leads to the claim.
From Proposition 4.5 given that à injective and separable for ño and jıg. FM,, (Po. (1a) is a distance
between jin and jig.
From Proposition
3.5
minimizing FM,,, (Ba, (10) w.r.t. 0 is equivalent to optimizing the following:
min [w, ,1)),
(47)
where (0, (w", h)) = -Bzppolllu ((=))( and w" = arg maxges0 Id(a,)) (N0 R''of oof Now that w w" from the direction optimality assumption, the gradient of the minimization objec- tive in Eq. (47) w.r.t. 0 becomes
VoJw(O, ,1)) - - DE [w,, h(x)}]
(48)
z E Erno R o
(49)
VoJ (wo ((g)))))),
(50)
where we used an envelope theorem Miilgrom & Segal 2002 to get Eq. f=)) a normalizing constant.
48 and Z = E Ex [r
Now, (Mo (a)) is distance and the gradient of FM, (No. (ao) is equivalent to VoJ (0, (w,h(go(2))) except for the scaling factor Z, so + (10, h(x)) I metrizabble
E RELATED WORK
In this section, we present the related work, some which are described in Secs. and 6 mentioning the limitations of the current work.
while
E.1 LITERATURE ON GAN
Goodfellow et al. 2014) was responsible for pioneering work that developed GAN with various theoretical analyses under the optimal discriminator (maximizer) assumption. Since then, GAN has been applied
variety of tasks, yet the instability of its learning has always been challenge. Arjovsky et al. (017) elaborated on one of the causes by showing that the maximizers of f- f-GANs lead to exploding and vanishing gradients with disjoint supports of the generator and data distribu- tions. Although they proposed Wasserstein GAN, which avoids the gradient issue, it suffers from another instability issue (Chu et al., 2020 Xu al. 2020 Qin et al. (020). We empirically observe that Wasserstein GAN tends not to satisfy separability. which gives us another explanation for the instability. Several techniques to improve the GAN training for general GANs have been proposed, such regularization terms (Nagarajan & Kolter 2017 Terjék 2020 Zhang et al. 2020), guidance losses (Sauer et al., 2022 2023), and optimization schemes (aaz et al. 2019 Chavdarova et al.) 2019 Qin et al.) 2020 Chavdarova et al. 2022). It should be noted that the above techniques orthogonal to our proposed method imposing direction optimality, which means SAN has potential to enhance many existing GANs. Our experiment on StyleGAN-XL, which adopts many of the techniques to achieve the state-of-the-art performance in large-scale generation tasks (Sauer et al. 2022), supports this expectation (see Sec. 7.2)
22

---

Published as a conference paper at ICLR 2024
Several papers have focused on theoretical understanding of GAN optimization. The discriminator optimality is a convenient assumption for theoretical analysis (Goodfellow et al. 2014 Johnson &
Zhang 2019; Gao et al., 2019 Fan al.] 2022), although achieving such optimality in practical
2020). Some studies shed light on the analysis of GAN optimization from the perspective of the local stability and convergence behavior of gradient
cases i8 rarely feasible (Li et al. 2018 Chu et al.
Chu
descent-ascent (Nagarajan & Kolter, |017| Mescheder et al. 20188. In another direction, researches have introduced and characterized variants of equilibrium notions such the local equilibria (Ratiffi et al. 2013 Jin et al. 2020), proximal equilibria (Farnia & Ozdaglar 2020), and mixed Nash equi- libria (Arora et al.J 2017] Hsieh et al. (209). Optimization smoothness, a useful perspective for
looking into stability of gradient-based learning dynamics, is beneficial even for GAN cases under
certain assumptions (Nitanda & Suzuki] 2018 Chu et al., [020 Fiez et al. 2022). In our work, we
analyze GAN optimization in terms of the concept of metrizability (see Definition which does not fall into the above taxonomy.
E.2 LITERATURE ON SLICED WASSERSTRIN DISTANCE
SW was initially proposed to mitigate the course of dimension (Bonneel et al. 2015). The calcula- tion of SW involves Radon transform and direction sampling (w). Kolouri et al. (019)) first extended SW to the generalized SW in the context of deep learning. They replaced Radon transform with a generalized Radon transform (GRT) by using defining functions, which allows the use of broad class of injective functions. Further, Chen et al. (2022) proposed the SRT to make it easier to incor- porate neural functions into variant of Radon transform. There are also some variants of Radon Transform, which take into account the properties of the target data geometry (Bonet et al., 2023) and neural layers (gguyen& Ho 2022).
Regarding the slicing part, the most basic approach is to sample directions according to uniform distribution on the hypersphere to discretely approximate the integral of Eq. (8). Since the random nature of slices could result in an underestimation of the distance (Kolouri et al. 2019). more ef- ficient sampling schemes have been investigated. Deshpande al. (2019) proposed the concept of max-SW, which uses only the direction that maximizes the one-dimensional Wasserstein distance (e.g., the integrand Eq. K)) Nguyen et al.] (2021) generalized the vanilla sliced distances and max-sliced distances by introducing probability distributions on the hypersphere.
Lastly, it is well known that a larger number of samples leads to a better approximation of the one- dimensional Wasserstein distance, thus leading to stronger generation performance (Nguyen al.)
2021 Lezama et al. 202T Chen et al., 2022). Therefore, simply increasing the batch size can en- hance the performance. However, the number of samples utilized for the model training is typically constrained by the memory required to store the model (Lezama et al.) (021). To overcome this lim- itation, Lezama al. T) proposed an algorithm that evaluates the one-dimensional Wasserstein distance with a virtual large batch size, although they did not apply the algorithm to the training of generative models from scratch but to the fine-tuning of GAN models due its high computational cost. This kind of algorithm may be useful even for the fine-tuning of SAN models especially when the obtained h does not satisfy separability rigorously.
E.3 LIMITATIONS
Metrizable conditions. Although our framework provides insights into GAN training without any unrealistic assumptions, a theoretical investigation into the separability is left as future work. In the current work, we conducted empirical studies on separability rather than a mathematical analysis. While our empirical observations on this property are consistent across different datasets, we believe additional theoretical investigation on it will be helpful for further understanding and improving GAN training.
SAN. One limitation of SAN is that it is not applicable to specific GANs (e.g., least-squares GAN) in a straightforward way. Naive least-squares SAN does not satisfy the metrizable condition, as its minimization problem does not satisfy the monotonic property regarding R in Theorem [.3 However, we should emphasize that SAN still covers broad class of GAN objective including f- GANs, and it is also possible to construct a new SAN objective by adopting different GAN variants for JGAN and VEAN.
23

---

Published as a conference paper at ICLR 2024
05
phiw" (5)
0.
0.6
05
64.
04
0.4
044
6M
0D
48
06
(a) linge GAN
(b) Saturating GAN
(c) Non saturating GAN
(d) liinge SAN
Jin
60
D4
Saturating SAN
Non-saturating SAN
Wasserstein GAN
(h) Rotation behavior in Wasserstein GAN
Figure 7: (a)-()) Estimated cumulative density function 5,000 iterations. Separability is satisfied in most GANs and SANs, but the Wasserstein GAN, the discriminator does not satisfy separabil- ity. (h) Data samples and generated samples during training with Wasserstein GAN. Different colors represent different iterations (2,000, 4,000,. 10,000). Rotational behavior is observed.
1.0
1.0
1.0
0.0
0.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
(a) Bascline (ReLU)
(b) + LReLU
(c) GP
Figure 8: Effects of techniques induce injectivity. The of LReLU instead of ReLU and the addition of a GP to the maximization problem both improve the performance.
F EXPERIMENTAL SETUP
F.1 MIXTURE OF GAUSSIANS
In Sec. 7.1 and Appx. H we the same experimental setup below. For both the generator and discriminator, we adopt simple architectures comprising fully connected layers by following the previous works (Mescheder et al. [2117] Nagarajan & Kolter 2017 (inha et al. 2020). We basically use Ieaky ReLU (LRcLU) with a negative slope of 0.1 for the discrıminator.
F.2 IMAGE GENERATION
DCGAN. We borrow most parts of the implementation from a public repository Please refer to Tables [ details of the architectures. Spectral normalization is applied to the discriminator. We use the Adam optimizer (Kigmma & Ba 2015) with betas of (0.0, 0.9) and an initial learning rate of 0.0002 for both the generator and discriminator. To convert GANs into SANs, we add two small modifications to the implementations. First, we apply weight normalization instead of spectral normalization on the last linear layer of the discriminator, which maintains its exact weight on a hypersphere throughout the training. Next, we utilize the propesed objective functions for the maximization problems. We train the models for 2,000 iterations with the minibatch size of 64. We run the training with three different seeds for the models and report the mean and deviation values in Table 3
-normal izat gga
24

---

Published as a conference paper at ICLR 2024
1.0
Real Fake
1.0
1.0
0.4
0.4
0.4
0.2
0.2
0.2
(a) Hinge SAN
(b) Saturating SAN
(c) Non-saturating SAN
1.0
1.0
0.5
08
0.6
0.3
02
0.0
0.0
(d) Wasserstein GAN
(e) Hinge GAN (ReLU)
Figure 9: Estimated cumulative density functions for the MNIST case. We follow the same pro- cedure as in Fig. to plot them. (a)-(c) Separability satisfied by the SAN models. (d) In contrast, Wasserstein GAN does not satisfy the property, leading to bad generation results (same as in Fig. g (e) The use of ReL.U in Hinge SAN does not destroy separability but leads bad generation results due to the non-injectivity caused by the non-injective activation (see Fig. T0 a)).
Table 7: DCGAN architecture on CIFARIO.
Discriminator
Generator
Input: x E R 9x332332
Input: E R 128 ~ 7),
3x3, stride= 1, channel=64, LRel.U
SNConv (4x4, stride= 2, channel-64), LReLU
Deconv (4x4, stride= c512 BN, ReLU Deconv (4x4, stride=2, ch5nn BN, ReLU Deconv (4x4, stride=2, channel=128), BN, ReLU Deconv (4x4, stride=2, channel-64), BN, ReLU Decony (4x4, stride= channel=3), Tanh
SNConv (3x3, stride=1, 1, 128)) LReLU
SNConv (4x4, stride=2, channel= LReLU
SNConv (3x3, stride= 1, channel=256), LRcLU
SNConv (4x4, stride= 2, channcl=256) LReLU
SNConv (3x3, stride=1, channcl=512), LReLU
SNLinear
BigGAN. We use the authors' PyTorch implementation to compare GANs and SANs in the set- ting of BigGAN on CIFAR10. We exactly follow the instructions in the repository to train the GAN in the class conditional generation task. The discriminator constructed with ReLUs, which may impede injectivity as discussed in Appx. H.I Since the aim of the experiment is to validate our training scheme presented in Sec 6 in existing settings, we do not change the activation function. We defer an ablation study injectivity induction by the activation functions to Appx. H.3 Re- garding the unconditional generation task, we simply replace the class embedding for the last linear layer with single linear layer. Similarly to the DCGAN setting, implement the SANs only by modifying the last linear layer of the discriminator and the maximization objective function. We the training with three different seeds for respective cases and report the mean and deviation values in Table 4 Note that it has been reported that there are numerical gaps between the eval- uation scores reported in the original paper and those obtained by reproduction with the PyTorch implementation (Song & Ermon 2020).
StyleGAN-XL. We use the official implementation compare GAN and SAN in the setting of StyleGAN-XL. We follow the instructions in the repository to train StyleSAN-XL in the class
http:///tthob.oco////oooooosssst xlgge/lleesstttttttgg egan-x1
25

---

Published as a conference paper at ICLR 2024
4
4
4
4
4
0
2
2
0
9
3 0
4
4
4
4
4
4
4
4
9
8
9
9
7
6
3
2
4
4
4
4
4
4 4 4 4 4 4
4
65
5
4
6
2
5
5 7 6
2
4
4
3
3
3
2
4
6
4
9
4
4 6
4
4
4
4
4
4
4
3
7
5
O
8
9
2
4
4
4
4
4
4
9
4
0
R
5
2
3
4
2 4
4
2 4
9 6
0
7
2
6
7
4
2
4
4
7
8
5
6
8
?
(a) Baseline (ReLU)
(b) +LReLU
/
6 7
5
4
8
0
6
t
3
2
3
5
7
0
6
3
D
5
0
?
/
/
2
9
3
6
4
3
2
7
5
6
3
9
17
5
9
3
6
0
5
0
3
9
1
3
5
6
6
D 8
O
t
4
7
4
8
of
t
6
6
6
9
0
2
(c)
+GP
(d) Wasserstein GAN
Figure 10: (c)-() Effects of techniques to induce injectivity. (a) The use of ReLU leads to collapse issue. (b) The of LReLU instead of ReLU and (c) the addition of GP to the maximization problem both improve the performance. (d) Generated results for Wasserstein GAN as a reference.
Table 8: DCGAN architecture CelebA.
Discriminator
Generator
Input: x 3x1188228
Input: R 128 (),
3x3, stride= 1, channel= LReLU
Deconv (4x4, stride=1, 5122. BN, ReLU
SNConv (1 x 4, stride= 2, 1hannel=128), LReLU
Deconv (4x4, stride=2, ch5nn BN, ReLU
SNConv (3x3, stride= 1, channel=256), LReLU
Deconv (4x4, stride=2, 2, channel=128), BN, ReLU
SNConv (4x4, stride=2, 2, channel=256), LReL.U
Deconv (4x4, stride=2. 22 64)), BN, ReLU
SNConv (3x3, stride= 1. channel=512), LReLU
Deconv (4x4, stride=2. channel=32), BN, ReLU
SNConv (4x4, stride=2. channel-512), LReLU
Decony (4x4, =tride=2, channel=3), Tanh
SNConv (3x3, stride=1, channel=1024), LReLU
SNLincar
conditional generation task. As in the DCGAN and BigGAN settings, implement the SAN by modifying the last linear layer of the discriminator and the maximization objective function. Besides, since just normalizing the last linear layer may deteriorate the representation ability of the overall network, we introduce the trainable scaling factor a E IR to keep the ability. Note that the scaling scalar is included in h rather than to keep w on the hypersphere. which is consistent with Listing 2. For ImageNet, we train a 256 x 256 generative model on top of the pretrained 128 x 128 generative mode in a progressive growing manner (Karras al., 2018 /auer al. 2022). We set the --k mg option to 11,000 according to an author's comment in the repository We then report the FID and IS calculated with the provided code. For CIFARIO, we train a model from 16 x 16 resolution to the target resolution in a progressive way. We train model with our arguments that
/models/
imagenet 128.
hhttps://qithub. 1ssuecomment-1111100
26

---

Published as a conference paper at ICLR 2024
Table Numerical results for BigGAN Brocck al. 2019) and BigSAN on CIFARIO in uncondi- tional setting.
Method
FID ())
IS (t)
BigGAN 17.1611.34
8.42.10.11
BigSAN
14.45 10.58
8.81 ...04
Table 10: Numerical results for BigGAN Brock et al. scheme.
2019) and the one fine-tuned by SAN training
Method
FID (L)
IS (1)
BigGAN (baseline)
8.25
9.05
BigGAN (fine-tuned with SAN)
7.59.:..23
9.04
should different from the original (see our repository in different model size.
7
for the details of our training), resulting
G SUPPLEMENTARY EXPERIMENTS
G.1 BIGGAN
In this section, we present two supplementary experimental results to further support the effective- ness of SAN training scheme.
We also compare SAN and GAN with BigGAN architecture in an un- conditional generation by following the experimental setting in Song & Ermon (2020). As reported in Table 9 SAN consistently improves the generation performance in terms of FID and IS.
Unconditional BigSAN.
Fine-tuning GAN with SAN. Another interesting question is whether SAN training scheme is valid even for fine-tuning GAN. We fine-tuned the BigGAN models trained on CIFAR-10 with the SAN training objectives for 10k iterations. We found our proposed method is valid for fine-tuning existing GANs as reported in Table I0] Still, training the SAN from scratch benefits from our modification scheme more, leading to better performance than the fine-tuned model.
G.2 STYLEGAN-XL ON FFHQ
Here, we train model on FFHQ (Karras et al. 2019) from 16 x 16 resolution to 1024 x 1024 resolution a progressive way. We train a model with our arguments that should be different from the original, resulting in different model size (see our repository for the details of training). In this experiment, we calculate not only the FID but also the FID which is calculated in the feature space of CLIP image encoder (Radford et al. 2021). because it has been reported that FID is more highly correlated with human assessment on FFHQ (Kynkäääniemm et al. 2023).
We report the numerical results in Table II and show generated images in Fig. II As shown in Tabll TI StyleSAN-XX achieves comparable or better FID and FIDCLIP scores. However, both StyleGAN-XL. and StyleSAN-XL. occasionally generate low-quality samples with poor proportions and artifacts, as observed in the original Projected GAN (Sauer et al.) 2021). Kynkäänniemi et al. (2023) reported that StyleGAN2 (Karras et al. (020) trained on FFHQ can produce high-quality and diverse samples, even though its FID score is not good as that of StyleGAN-XL -L or StyleSAN. XL. We hypothesize that this discrepancy arises because the two encoders used in StyleGAN- XI/StyleSAN-XI. training, EfficientNet (Tan & Le 2019) and DeiT-base (Touvron et al. 2021 are trained on ImageNet, the same dataset used train the Inception-V3s (Szegedy et al.) (016) uti- lized in FID calculation. While this may improve FID scores, it does not consistently enhance actual sample quality, particularly outside the dataset's domain.
https://qittubb.coo/mm/o
Efhg256.pkJ
httt:://gittbbb.cmm/////
27

---

Published as a conference paper at ICLR 2024
Table 11: Numerical results for StyleGAN-XL (Sauer et 2022 and StyleSAN-XL FFHQ.
Scores marked with $ are reported in their repository. Note that our StyleSAN-XL models are larger model size than StyLeGAN-XL.
(D)
FIDEUP (
Resolution
StyleGAN-XL
StyleSAN-XL
StyleGAN-XL
StyleSAN-XL
256 256
(2.19)
1.68
(3.00)
3.01
(2.23)
1.77
(3.21)
3.06
1024 x 1024
(2.02)
1.61
(3.62)
3.11
(a) StyleGAN-XL
(b) StyleSAN-XI.
Figure 11: Generated samples from StyleGAN-XIBann and StyleSAN-XL trained on FFHQ.
H INVESTIGATION OF SEPARABILITY AND INJECTIVITY
In this section, separability and injectivity are empirically investigated.
H.1 INDUCTIONS FOR INJECTIVITY
We present some potential inductions for injectivity, which are empirically verified in the next sub- sections.
There are various ways to impose injectivity on the discriminator, one of which is to implement the discriminator with an invertible neural network. Although this topic has been actively stud- ied (Behrmann al.) 2019] Karami al. 2019; [Sog et al. 2019). such networks have higher computational costs for trainigg (Chen et al. (022). Another way Is to add regularization terms the maximization problem. For example, gradient penalty (GP) (Gulrajani et al. 2017 can
28

---

Published as a conference paper at ICLR 2024
Table 12: Effect of injectivity induction by activation functions on FID and IS in the BigSAN setting. The use of LReLU leads to better or competitive scores than that of
CIFARIOO
CIFARIO0
Metrie
BigSAN w/ ReLU
BigSAN w/ LReLUU
BigSAN w/ ReLU
BigSAN w/ LReLU
FID (1)
6.20
5.97 10.12
8.05 10.04
7.97 0.06
IS (t)
9.16=0008
9. 9.23+0.03
10.72
10.84
promote injectivity by explicitly regularizing the discriminator's gradient. The penalty enforces the norm of the discriminator Jacobian to be unity. This mitigates the occurrence of aero-gradient, one of the prominent factors contributing to non-injectivity.
In contrast, simple removal of operators that can destroy injectivity, such as ReLU activation, can implicitly overcome this issue. It is obvious that invertible activations (such leaky ReLU) con- tribute to layerwise injectivity. Several studies suggest that constructing an injective neural network with ReLU is more challenging compared to invertible activation functions (Puthawala et al. 2022 Chan et al. 2023).
H.2 MIXTURE OF GAUSSIANS
We visualize the generated samples mainly confirm that the generator measures cover all the modes of the eight Gaussians. In addition, we plot the cumulative density functions of S T,o (:)w) and I (.,w) to verify separability. As shown in Fig. 7 the cumulative density function for the generator measures trained with Wasserstein GAN does not satisfy separability, whereas the generator measures trained with other GAN losses do satisfy it. This may be what causes the rotation behavior of the Wasserstein GAN, as seen in Fig. 7h).
To investigate the effect of injectivity on the training results, we train Hinge GAN using a discrimi- nator with ReLU as a baseline. We apply two techniques to induce injectivity: (1) the use of LReLU and (2) the addition of a GP to the discriminator's 's maximization problem. As shown in Fig. 8 with either of these techniques, the training improved and mode collapse does not occur.
H.3 VISION DATASETS
We extend the experiment in Appx. H.2 to MNIST (LeCun et al. 1998) to investigate injectivity and separability on vision datasets. First, we train SANs with various objective functions and Wasser- stein GAN on MNIST. We adopt DCGAN architectures and apply spectral normalization to the discriminator. As in Appx. H.2,, we plot the cumulative density function for the generator measures trained with various objective functions in Fig. 9 The trained generator measures empirically sat- isfy separability except for the case Wasserstein GAN, which is a similar tendency to the case of MoG. We also conducted an ablation study regarding the technique inducing injectivity by following Appx. H. We train Hinge SAN using a discriminator with ReLU as a baseline, and compare this baseline with two cases: (1) the of LReLU and (2) the addition of a GP to the discriminator's objective function. The generated samples for these cases are shown in Fig. a)))))(c) As we can see, the generator trained with the ReLU-based discriminator suffers from the mode collapse issue, while in contrast, the techniques inducing injectivity mitigate the issue and lead to reasonable gen- eration results. As a reference, we depict the generation result from generator that is trained with a L.ReU-based discriminator and Wasserstein GAN in Fig. d), where we can see that the gener- ated samples are completely corrupted. Note that the discriminator with ReLU satisfies separability, the same as in Fig. e). This suggests that the deterioration in Hinge SAN with the ReL.U dis- criminator (Fig. a))a)) and that in the Wasserstein GAN model (Fig. d) are triggered by different causes, ie., from non-injectivity and non-separability, respectively.
To further verify our insights in more realistic setting, we conduct another experiment using Big- GAN architecture, where the discriminator is constructed with ReLUs. For ablation of the injectivity induction by the activation functions, we replace ReLUs with LReLUs in the discriminator, and sub- sequently train BigSAN both CIFAR- and CIFAR-100. The use of the injective activation leads to slightly better scores than the original BigSAN as reported in Table 12 This result consistent with our theoretical and empirical investigations.
29

---

Published as a conference paper at ICLR 2024
50.0
Wasserstein max-SWGAN
Hinge SWAN-
40.0
Saturating max SWGAN
Hinge GAN
Saturating Batch size 64
Hinge SAN
30.0
Saturating SAN
20.0
32
64
128
256
512
Batch size
Figure 12: Empirical studies on the impact batch size on SWGAN trained on CIFARIO. The FID score decreases the batch size increases especially with Vw.
I INVESTIGATION OF VGAN IN SW
The aim of this empirical study compare VGAN in the SW context. To this end, we train max-SW- based generative models on CIFAR 10 using the same experimental setting as in Sec. 7.2 (DCGAN). We apply the ASW to max-SWGAN (Deshpande et al. 2019), which was shown to achieve com- petitive or better generation performances compared to the uniform sampling of w despite reducing the computational complexity and the memory footprint of the model. We train the model by the following optimization problem:
max VGAN((W, h(x))), and
(51a)
min W2(S no S* (1,w)),
(51b)
Note that the maximizer in Eq. (lap does not generally maximize the Wasserstein distance in Eq. (51b II] Since the minimization problem (51b) requires sample sorting according to discrimi- nator values approximate the Wasserstein distance, the larger batch size is expected enable better distance approximation, leading to better generation performance, at the expense of compu- tational cost (refer Fig. in Nguyen et al. (202I) and Fig. 14 in Chen et (2022).. Regarding VGAN, we utilize the maximization problems for Wasserstein GAN, hinge GAN, and saturating GAN (non-saturating GAN) for comparison.
We plot the FID score w.r.t. batch size in Fig. 12 Interestingly, with the maximization problem of Wasserstein GAN, the batch size greatly influences the generation performance, unlike the other
GAN variants. We suspect the smaller sample size causes noisy approximation of the Wasserstein distance, since Vw tends to bring non-separable (as shown in our experiments in Sec. HD. In contrast, the other GAN variants lead to reasonable generative models even with a smaller batch size. The FID scores GAN and SAN reported in Table 3 are also plotted here for reference. When the batch size is larger, the FID score of the max-SWGANs converges to certain value, which is better than that GAN but worse than that of SAN. We suspect this is because, among the comparison methods, SAN training is the only way ensure optimal direction.
J GENERATED SAMPLES
Images generated by GANs and SANs shown in Figs. 13
14
15
and
17
11 Although the idea of max-SW is to select the optimal direction Deshpande et al. 2019), the proposed model does satisfy the property due its surrogate maximization objective VGAN-
30

---

Published as a conference paper at ICLR 2024
Saturating GAN
(b) Saturating SAN
Figure 13: Generated samples from DCGAN trained on CIFARIO.
(a) Non-saturating GAN
(b) Non-saturating SAN
Figure 14: Generated samples from DCGAN trained on CelebA (128x 128).
31

---

Published as a conference paper at ICLR 2024
(a) Hinge GAN (unconditional)
(b) Hinge SAN (unconditional)
(c) Hinge GAN (conditional)
(d) Hinge SAN (conditional)
Figure 15: Generated samples from BigGAN trained on CIFAR1O.
Figure 16: Generated samples from StyleSAN-XL trained on CIFARIO.
http:///a...a.ooooooooo///////aa /1/models/ imagenet256.pk
32

---

Published as a conference paper at ICLR 2024
(a) StyleGAN-XL
(b) StyleSAN-XL
Figure 17: Generated samples from StyleGAN-XI and StyleSAN-XI. trained on ImageNet.
33


Reviews:
[{"review_id": "50ILol9dWZ", "reviewer": "Reviewer_xjKk", "summary": "The authors focus on evaluating the metrizability of discriminators in Generative Adversarial Networks (GANs) training by determining if the discriminator's measure can constitute a metric, defined by properties such as directionality, separability, and injectivity. Initially, they establish a connection between the Wasserstein distance and the Functional Mean Divergence (FM*). Then the Separability is proposed to link FM* to the maximum Average Sliced Wasserstein (max-ASW) distance. Subsequently, they introduce the injectivity for the design of effective discriminators. They examine commonly used GAN frameworks, including Wasserstein GAN and other losses GANs, and find that most GAN architectures, including those using Hinge-loss, Saturating, and Non-Saturating loss functions, typically do not fulfill the criterion of directional optimality. In response to this, the authors propose a straightforward modification to enhance this property within these GANs. To substantiate their theoretical claims, they conduct experiments that demonstrate the significance of these three properties. They also apply their modifications to contemporary models such as DCGAN, BigGAN, and StyleGAN-XL. The results affirm that their simple yet effective alterations lead to improvements over the baseline performance, thereby confirming the practical value of their theoretical insights.", "questions": "1. Could the authors cite studies or evidence that show how regularizing the gradient of the discriminator or switching from ReLU to LeakyReLU might enhance injectivity?\n\n2.  In Table 4, are the results shown for BigGAN the ones obtained after replacing ReLU with LeakyReLU?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors present a theoretically grounded analysis that yields a straightforward method for enhancing GAN discriminators.\n\n2. Their theoretical contributions are insightful and offer practical guidance for discriminator design. \n\n3. The experiments conducted are well-aligned with the theory, and the clarity of the writing effectively conveys the study's findings and implications.", "weaknesses": "1. While the study shows experimental progress, the improvement in FID (Frechet Inception Distance) is modest.\n\n2. The authors are encouraged to conduct experiments on CIFAR-100 using BigGAN and FFHQ using StyleGAN2 to further verify the effectiveness of the proposed method.\n\n3. To enhance accessibility, the authors should consider simplifying the mathematical notation to cater to a wider audience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O3t4dvzXGn", "reviewer": "Reviewer_EquL", "summary": "This article introduces a novel theoretical perspective to shed light on the optimization process, which determines whether the generator receives gradients that bring its distribution closer to the target distribution. The article introduces three metrizable conditions: direction optimality (evaluating the gradient between two distributions), separability (examining the distinctness between two distributions), and injectivity (measuring the metric as a distance). These conditions aim to provide clarity on the underlying motivation. Furthermore, the article includes a theoretical proof and introduces a new loss function and a normalized linear layer. These enhancements are designed to boost the discriminative capability of the discriminator and enable the generation of diverse, high-fidelity samples from the generator.", "questions": "I have read the paper carefully, so i want the author to clarify the questions as follows:\n\nIf the author can clarify the following questions well, I'd like to raise my rating score.\n\n1. If I understand correctly, \"direction optimality\" implies that a negative gradient of the metric between the generated distribution and the target distribution , which guides the generated distribution toward the target. \"Separability\" pertains to two disjointed distributions, and \"injectivity\" suggests that the metric between the generated and target distributions is a distance. Is this interpretation accurate?\n\n2. Could you clarify the symbol \"$*$\" in Table 1?\n\n\n3. I recommend that when the article first mentions metrizable conditions, namely, direction optimality, separability, and injectivity, it would be more reader-friendly to provide an intuitive understanding of these terms and indicate which paragraphs explain these definitions. This would help readers quickly grasp the key concepts before delving into the detailed explanations.\n\n4. I have a question about the statement as follows:\nmost existing GANs besides Wasserstein GAN do not satisfy direction optimality with the maximizer $w$ of $\\mathcal{V}$. Although most existing GANs do not  satisfy direction optimality proposed in the article, but the discriminator of these gans actually provides the generator with gradients that make its distribution close to the target distribution which has been discussed in other articles[1-5]. What do you think about this question?\n\n\n\n5. There might be an error in the symbol  $g\\# $     $=\\sigma(g^{-1}(B))$. \n\nThe function $g$ is defined as $g: Z \\rightarrow X$, so the inverse function $g^{-1}$ should be $g^{-1}: X \\rightarrow Z$. The term $\\sigma(\\cdot) \\in \\mathcal{P}(Z)$. However, $g\\#$ \n\nis meant to represent the generator $g: Z \\rightarrow X$. Thus, there appears to be a discrepancy.\n\n\n6. I recommend that the author provide a clear explanation of why Equation (16) satisfies Theorem 5.3. I find it somewhat confusing and would appreciate a more comprehensible clarification.\n\n7. I have reviewed the code in the supplementary materials. There is a discrepancy between the code provided in Listing 2 (SAN discriminator) and lines 92-96 in the file \"STYLESAN-XL/pg_modules/san_mondules.py/SANConv2d.\" To ensure accuracy, please verify which version is the correct one.\n8. Should the SAN be trained from scratch, or can it be fine-tuned, especially in the last layer, using the Eq.16 loss function?\n\n9. I find it quite confusing to understand the importance of the three conditions in your theorem 5.3. In my opinion, direction optimality should be the most important condition, while the separability and injectivity conditions may not carry the same weight. This is because metrics used in GANs, such as f-divergences or IPM functions, inherently satisfy the distance property. Maintaining the separability condition becomes increasingly challenging as GAN training progresses and the generated distribution approaches the real data distribution.\nTherefore, I recommend that the author clearly outline the relationships and significance of all three conditions to provide a better understanding.\n\n\n1.Variational inference via Wasserstein gradient flows\n\n2.Variational Wasserstein gradient flow\n\n3.Deep Generative Learning via Variational Gradient Flow\n\n4.A Framework of Composite Functional Gradient\nMethods for Generative Adversarial Models\n\n5.Gradient Layer: Enhancing the Convergence of Adversarial Training for Generative Models", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This article introduces a novel theoretical perspective aimed at addressing whether the optimization process truly provides the generator with gradients that lead its distribution closer to the target distribution. The article proposes three metrizable conditions, namely direction optimality (evaluating the gradient between two distributions), separability (evaluating disjointedness), and injectivity (measuring the metric as a distance).\n\nAdditionally, the article provides both theoretical and empirical evidence to demonstrate that only Wasserstein GANs fulfill these conditions. To enhance the discriminative capability of the discriminator and facilitate the generation of diverse, high-fidelity samples from the generator, the authors introduce a new loss function and a normalized linear layer. This normalized linear layer serves as a plug-in component that can be seamlessly incorporated into various GAN models.\n\nIn experimental evaluations, the SAN model achieves the best FID score when compared to numerous GAN baseline models.", "weaknesses": "The article's motivation, which revolves around the examination of whether trained discriminators effectively supply gradient information to optimize generators for reducing dissimilarities, appears to have been explored in prior articles [1-5]. However, the article falls short in providing a clear connection between the theoretical results and their practical implications. Moreover, the significance of the theoretical findings might benefit from more detailed explanation.\n\n\n1.Variational inference via Wasserstein gradient flows\n\n2.Variational Wasserstein gradient flow\n\n3.Deep Generative Learning via Variational Gradient Flow\n\n4.A Framework of Composite Functional Gradient\nMethods for Generative Adversarial Models\n\n5.Gradient Layer: Enhancing the Convergence of Adversarial Training for Generative Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)