WASSERSTEIN-BOUNDED GENERATIVE ADVERSAR-
Year: 2019

Abstract:
In the field of Generative Adversarial Networks (GANs), how to design a stable training strategy remains an open problem. Wasserstein GANs have largely promoted the stability over the original GANs by introducing Wasserstein distance, but still remain unstable and are prone to a variety of failure modes. In this paper, we present a general framework named Wasserstein-Bounded GAN (WBGAN), which improves a large family of WGAN-based approaches by simply adding an upper-bound constraint to the Wasserstein term. Furthermore, we show that WBGAN can reasonably measure the difference of distributions which almost have no intersection. Experiments demonstrate that WBGAN can stabilize as well as accelerate convergence in the training processes of a series of WGAN-based variants.

heading: 1 INTRODUCTION
text: Over the past few years, Generative Adversarial Networks (GANs) have shown impressive results in many generative tasks. They are inspired by the game theory, that two models compete with each other: a generator which seeks to produce samples from the same distribution as the data, and a discriminator whose job is to distinguish between real and generated data. Both models are forced stronger simultaneously during the training process. GANs are capable of producing plausible synthetic data across a wide diversity of data modalities, including natural images (Karras et al., 2017; Brock et al., 2018; Lucic et al., 2019), natural language (Press et al., 2017; Lin et al., 2017; Rajeswar et al., 2017), music (Yang et al., 2017; Mogren, 2016; Dong et al., 2017; Dong & Yang, 2018), etc.
Despite their success, it is often difficult to train a GAN model in a fast and stable way, and researchers are facing issues like vanishing gradients, training instability, mode collapse, etc. This has led to a proliferation of works that focus on improving the quality of GANs by stabilizing the training procedure (Radford et al., 2015; Salimans et al., 2016; Zhao et al., 2016; Nowozin et al., 2016; Chen et al., 2016; Qi, 2017; Deshpande et al., 2018). In particular, Arjovsky et al. (2017) introduced a variant of GANs based on the Wasserstein distance, and releases the problem of gradient disappearance to some extent. However, WGANs limit the weight within a range to enforce the continuity of Lipschitz, which can easily cause over-simplified critic functions (Gulrajani et al., 2017). To solve this issue, Gulrajani et al. (2017) proposed a gradient penalty method termed WGAN-GP, which replaces the weight clipping in WGANs with a gradient penalty term. As such, WGAN-GP provides a more stable training procedure and succeeds in a variety of generating tasks. Based on WGAN-GP, more works (Wei et al., 2018; Petzka et al., 2017; Wu et al., 2018; Mescheder et al., 2018; Thanh-Tung et al., 2019; Kodali et al., 2017; Kim et al., 2018) adopt different forms of gradient penalty terms to further improve training stability. However, it is often observed that such gradient penalty strategy sometimes generate samples with unsatisfying quality, or even do not always converge to the equilibrium point (Mescheder et al., 2018).
In this paper, we propose a general framework named Wasserstein-Bounded GAN (WBGAN), which improve the stability of WGAN training by bounding the Wasserstein term. The highlight is that the instability of WGANs also resides in the dramatic changes of the estimated Wasserstein distance during the initial iterations. Many previous works just focused on improving the gradient penalty term for stable training, while they ignored the bottleneck of the Wasserstein term. The proposed training strategy is able to adaptively enforce the Wasserstein term within a certain value, so as to balance the Wasserstein loss and gradient penalty loss dynamically and make the training process more stable.
WBGANs are generalized, which can be instantiated using different kinds of bound estimations, and incorporated into any variant of WGANs to improve the training stability and accelerate the convergence. Specifically, with Sinkhorn distance (Cuturi, 2013; Genevay et al., 2017) for bound estimation, we test three representative variants of WGANs (WGAN-GP (Gulrajani et al., 2017), WGANdiv (Wu et al., 2018), and WGAN-GPReal (Mescheder et al., 2018)) on the CelebA dataset (Liu et al., 2015). As shown in Fig. 1, WBGANs outperform the corresponding counterparts, which demonstrates that the bounded strategy results in more stable training and accelerates the convergence.

heading: 2 BACKGROUNDS
text: Wasserstein GANs (WGANs). WGANs (Arjovsky et al., 2017) were primarily motivated by unstable training caused by the gradient vanishing problem of the original GANs (Goodfellow et al., 2014). They proposed to use 1-Wasserstein distance W1 (Pr,Pg) to measure the difference between Pr and Pg, the real and generated distributions, given that W1 (Pr,Pg) is continuous everywhere and differentiable almost everywhere under mild assumptions. The objective of WGAN is formulated using the KantorovichRubinstein duality (Villani, 2008):
min G max D∈L1 Ex∼Pr [D(x)]− Ex̃∼Pg [D(x̃)], (1)
where L1 is the function space of all D satisfying the 1-Lipschitz constraint ‖D‖L ≤ 1. D is a critic and G is the generator, both of which are parameterized by a neural network. Under an optimal critic, minimizing the objective with respect to G is to minimize W1 (Pr,Pg). To enforce the 1-Lipschitz constraint on the critic, WGAN used a weight clipping on the critic to constrain the weights within a compact range, [−c, c], which guarantees the set of critic functions is a subset of the k-Lipschitz functions for some k. With weight clipping, the critic tends to learn over-simplified functions (Gulrajani et al., 2017), which may lead to unsatisfying results. Gulrajani et al. (2017); Wei et al. (2018); Petzka et al. (2017); Wu et al. (2018) proposed different forms of gradient penalty as a regularization term, so that a generalized loss function with respect to the critic can be written as:
LD = −[Ex∼Pr [D(x)]− Ex̃∼Pg [D(x̃)]] + λGP ·GP, (2) where Ex∼Pr [D(x)]−Ex̃∼Pg [D(x̃)] stands for the Wasserstein term, and GP for the gradient penalty term. LD is actually posing a tradeoff between these two objectives.
Wasserstein Distance between Empirical Distributions. In practice, we approximate W1 (Pr,Pg) using W1 ( P̂r, P̂g ) , where P̂r and P̂g denote the empirical version of Pr and Pg with N samples,
i.e., P̂r = 1N ∑N i=1 δyi , and P̂g = 1 N ∑N i=1 δG(zi). Here, yi is randomly sampled from the real
image dataset, and δyi is the Dirac delta function at location yi. Computing W1 ( P̂r, P̂g ) is a typical problem named discrete optimal transport. We denote B as the set of probabilistic couplings between two empirical distributions defined as:
B := {Γ ∈ RN×N+ | Γ1N = P̂g,Γ>1N = P̂r}, (3)
where 1N is a N -dimensional all-one vector. Then we have W1(P̂r, P̂g) = minγ∈B〈Γ,C〉F , where 〈·, ·〉F is the Frobenius dot-product and C is the cost matrix, with each element Ci,j = c(G(zi),yj) denoting the cost to move a probability mass from G(zi) to yj . The optimal coupling is the solution of this minimization problem: Γ0 = arg minΓ∈B〈Γ,C〉F . The Sinkhorn Algorithm. Despite Wasserstein distance has appealing theoretical properties in measuring the difference between distributions, its computational costs for linear programming are often high in particular when the problem size becomes large. To alleviate this burden, Sinkhorn distance (Cuturi, 2013) was proposed to approximate Wasserstein distance:
dα(P̂r, P̂g) := min P∈Uα(P̂r,P̂g) 〈P,C〉, (4)
where Uα(P̂r, P̂g) is a subset of B defined in Eq. 3:
Uα(P̂r, P̂g) := {Γ ∈ B|H(Γ) > H(P̂r) + H(P̂g)− α} ⊂ B, (5) where H(·) is the entropy defined as H(Γ) = − ∑N i,j=1 Γi,j log Γi,j and H(P̂r) = − ∑N n=1 p̂n log p̂n where p̂n is the probability of the n-th sample. Compared to Wasserstein distance, Sinkhorn distance restricts the search space of joint probabilities to those with sufficient smoothness. To compute Sinkhorn distance, a Lagrange multiplier was used:
dλ(P̂r, P̂g) = 〈 Γλ,C 〉 , Γλ = arg min
Γ∈B 〈Γ,C〉 − 1 λ H(Γ). (6)
Each α corresponds a λ ∈ [0,∞) such that dα(P̂r, P̂g) = dλ(P̂r, P̂g) holds for that pair (P̂r, P̂g). dλ(·, ·) can be computed with a much cheaper cost than the original Wasserstein distance using matrix scaling algorithms. For λ > 0, the solution Γλ is unique and has the form Γλ = diag(u)K diag(v), where K is the element-wise exponential of −λC. u and v are two non-negative vectors uniquely defined up to a multiplicative factor (Cuturi, 2013).

heading: 3 WASSERSTEIN-BOUNDED GANS

heading: 3.1 BOUND CONSTRAINT ON 1-WASSERSTEIN DISTANCE
text: We start with Eq. 2 and denote W = Ex∼Pr [D(x)]− Ex̃∼Pg [D(x̃)]. W is often referred to as the Wasserstein term, which is unbounded during the training process. In a wide range of WGAN’s variants such as WGAN-GP (Gulrajani et al., 2017), the critic defined by LD is to maximize the Wasserstein term W while satisfying the gradient penalty GP. However, in practice, we find that W often rises rapidly to a tremendous value which is far from rational during the initial training procedure. A possible reason may lie in that the critic function does not satisfy the Lipschitz constraint during the initial training stage. As shown in Fig. 2, this leads to dramatic instability in optimization and finally results in unsatisfying performance in image generation.
Our idea is thus straightforward, i.e., setting an upper-bound for W . The modified critic loss function is written as: LD = −W + [W −ĎW ]+ + λGP ·GP, (7) where [·]+ = max{·, 0} is the ramp function that ignores negative inputs, and ĎW denotes the upper bound of W, and will be discussed later. If W 6 ĎW , the term [W −ĎW ]+ simply vanishes and Eq. 7 is equivalent to Eq. 2; otherwise, we have−W +[W −ĎW ]+ = −ĎW , which implies that the modified Wasserstein term is bounded by ĎW .
Our formulation brings a benefit to the numerical stability of the Wasserstein term. In practice, it remains comparable to the other term, λGP ·GP, so that both W and GP can be optimized in a ‘mild’ manner, i.e., without any one of them dominating or being ignored during training. Note that the ĎW term cannot be chosen arbitrarily. Setting it too small, ĎW will limit the capacity of the critic function, resulting in a poor generation. Setting it too large, there will be no effect of bounding the W term.
The proposed bounded strategy is a general framework. We name it general in two folds: First, WBGAN can be applied to almost all gradient penalty based WGANs, such as WGAN-GP (Gulrajani et al., 2017), WGAN-GPReal (Mescheder et al., 2018), etc. Moreover, there are different ways to estimate the value of ĎW . For example, the linear programming was applied successfully to some existing WGANs like WGAN-TS (Liu et al., 2018). In what follows, we present an example which uses Sinkhorn distance to estimate ĎW , while we believe other ways of estimation are also possible.

heading: 3.2 WBGAN WITH SINKHORN DISTANCE
text: In this section, we give an instantiation, Sikhorn distance (Cuturi, 2013), to effectively compute the bounded term ĎW . The motivation of using Sinkhorn distance lies in that in theory, the Wasserstein term of WGAN will eventually converge to the 1-Wasserstein distance between the real distribution Pr and the generated distribution Pg (Arjovsky et al., 2017; Gulrajani et al., 2017). Therefore, we can use the 1-Wasserstein distance between the empirical distributions, P̂r and P̂g, as the upper-bound ĎW . Since the computation of Wasserstein distance involves a large linear programming which
Algorithm 1 WBGAN with Sinkhorn distance Require: learning rate α, batch size M , the number of iterations of the critic per generator iteration
Ncritic, weight of gradient penalty λGP, weight of Sinkhorn distance λs, initial parameters θ and φ0, other hyper-parameters;
1: while φt has not converged do 2: for n = 1, . . . , Ncritic do 3: Sample a batch {x(m)}Mm=1 ∼ Pr from real data; 4: Sample a batch {z(m)}Mm=1 ∼ Pz of prior samples; 5: W ← 1M ∑M m=1Dθ(x (m))− 1M ∑M m=1Dθ(Gφt(z
(m))); 6: Calculate Sinkhorn distance dλ(P̂r, P̂g) between {x(m)}Mm=1 and {Gφt(z(m))}Mm=1; 7: Lθ ← −W + [W − dλ(P̂r, P̂g)]+ + λGP ·GP; 8: θ ← Adam(Lθ, θ, α, β1, β2); 9: end for
10: Sample a batch {z(m)}Mm=1 ∼ Pz of prior samples; 11: Calculate Sinkhorn distance dλ(P̂r, P̂g) between {x(m)}Mm=1 and {Gφt(z(m))}Mm=1; 12: Lφt ← −Ez∼Pz [Dθ(Gφt(z))] + λs · dλ(P̂r, P̂g); 13: φt+1 ← Adam(Lφt , φt, α, β1, β2); 14: end while Ensure: trained parameters θ and φT (converged).
suffers heavy computational costs, we replace it by Sinkhorn distance instead – the Sinkhorn distance between P̂r and P̂g can be computed using Sinkhorn’s matrix scaling algorithm (Cuturi, 2013), which is orders of magnitude faster than the linear programming solvers.
Mathematically, consider a generator function Gφ(z) that produces samples by transforming noise input z drawn from a simple distribution Pz , e.g., Gaussian distribution. Dθ stands for a critic function parameterized by θ. The objective of the critic is:
Lθ(Pr,Pg) = max Dθ∈L1 Ex∼Pr [Dθ(x)]− Ex∼Pg [Dθ(x)] − [ Ex∼Pr [Dθ(x)]− Ex∼Pg [Dθ(x)]− dλ(P̂r, P̂g) ] + ,
(8)
where dλ(P̂r, P̂g) is the Sinkhorn distance defined in Eq. 6. On the other hand, given a fixed critic function Dθ? , considering that Sinkhorn distance allows gradient back-propagation (Genevay et al., 2017), we can find the optimal generator Gφ? by solving:
φ? = arg min φ −Ez∼Pz [Dθ?(Gφ(z))] + λs · dλ(P̂r, P̂g), (9)
where λs is a balancing hyper-parameter, which we set λs = 0.5 in this paper. In Algorithm 1, we summarize the flowchart of training WBGAN with Sinkhorn distance.
3.2.1 RELATIONSHIP BETWEEN dλ(P̂r, P̂g) AND W1(Pr,Pg)
We employ dλ(P̂r, P̂g) as an approximation of 1-Wasserstein distance W1(Pr,Pg). Let (X, d) be a separable metric space. P(X) denotes the set of Borel probability measures. Pp(X) denotes the set of all µ ∈ P(X) such that ∫ X d(x,y)pdµ(x) < +∞ for some y ∈ X . We can suppose real data distribution Pr, generated data distribution Pg and their empirical distribution P̂r and P̂g all in Pp(X). Proposition 1. Let Pr and Pg be real data distribution and generated data distribution. Suppose that P̂r and P̂g are empirical measures of Pr and Pg . Then we have 0 6W1(Pr,Pg) 6 E[W1(P̂r, P̂g)].
Proof. Please refer to Appendix A.
Proposition 1 tells us that as E[W1(P̂r, P̂g)] → 0, W1(Pr,Pg) is forced to 0. Cuturi (2013) has pointed out that if λ is chosen large enough, dλ(P̂r, P̂g) coincides with W1(P̂r, P̂g). So, it is reasonable to use dλ(P̂r, P̂g) to constrain the Wasserstein term.

heading: 3.3 ANALYSIS OF WBGAN
text: Most GANs measure the distance between distributions based on probability divergence. We will prove that the Eq. 8 is indeed a valid divergence. First, we have the following definition.
Definition 1. Given probability measures p and q, D is a functional of p and q. If D satisfies the following properties:
1. D(p, q) ≥ 0; 2. p = q ⇐⇒ D(p, q) = 0, (10)
then we say D is a probability divergence between p and q. Remark 1. The following W (Pr,Pg) satisfies the Definition 1 and is therefore a probability divergence.
W (Pr,Pg) = max D∈L1 Ex∼Pr [D(x)]− Ex̃∼Pg [D(x̃)], (11)
where L1 is the 1-Lipschitz constraint. Please see the proof and detailed discussion in Su (2018). This is the objective of critic used by WGAN (Arjovsky et al., 2017).
Remark 2. Equation 8 satisfies the Definition 1 and is a probability divergence.
Proof. The proof is given in Appendix B.
Remark 3. Consider two distributions Pr(x) = δ(x − α), Pg(x) = δ(x − β) that have no intersection (α 6= β). δ is the Dirac delta function. In such an extreme case, Eq. 8 can still be optimized by gradient descent.
Proof. The proof is in Appendix C
Remark 2 tells us that Eq. 8 is a valid divergence. Since the real data distribution is supported by lowdimensional manifolds, the supports of generated distribution and real data distribution are unlikely to have a non-negligible intersection. Remark 3 shows that compared to the standard GAN (Goodfellow et al., 2014), WBGAN can continuously measure the difference between two distributions, even if there is almost no intersection between the distributions.

heading: 4 EXPERIMENTS

heading: 4.1 SETTINGS AND BASELINES
text: To verify that WBGAN is a generalized approach, we select three variants of WGAN, namely, WGAN-GP (Gulrajani et al., 2017), WGAN-div (Wu et al., 2018) and WGAN-GPReal (gradient penalty on real data only) (Mescheder et al., 2018) as our baselines. By adding bound constraints to these WGAN variants, we obtain the counterparts WBGAN-GP, WBGAN-div, and WBGANGPReal, respectively. Two different network architectures are used, i.e., DCGAN (Radford et al., 2015) and BigGAN (Brock et al., 2018). For DCGAN, we directly output the activation before the sigmoid layer. BigGAN is a conditional GAN (Mirza & Osindero, 2014) architecture, in which class conditioning is passed to generator by supplying it with class-conditional gains and biases in the batch normalization layer (Ioffe & Szegedy, 2015; de Vries et al., 2017; Dumoulin et al., 2017). In addition, the discriminator is conditioned (Miyato & Koyama, 2018) by using the cosine similarity between its features and a set of learned class embedding. We use the spectral norm (Miyato et al., 2018) in BigGAN, but for the sake of simplicity, we do not use the self-attention module (Wang et al., 2017; Zhang et al., 2018). Other hyper-parameters and the network architecture of BigGAN simply follow the original paper.
We choose the Fréchet Inception Distance (FID) (Heusel et al., 2017) for quantitative evaluation, which has been proven to be more consistent with individual assessment in evaluating the fidelity and variation of the generated image samples.

heading: 4.2 MID-RESOLUTION EXPERIMENTS
text: We first investigate mid-resolution image generation on the CelebA dataset (Liu et al., 2015), a large-scale face image dataset with more than 200K face images. During training, we crop 108× 108 face from the original images and then resize them to 64× 64.
FID Stability. We first use DCGAN to build our generator and discriminator. Training curves are shown in Fig. 2, and quantitative results are summarized in Table 1. Each approach is executed for 5 times and the average is reported. All FID curves are obtained from generators directly without using the moving average strategy (Karras et al., 2017; Mescheder et al., 2018; Brock et al., 2018; Yazıcıet al., 2018) to avoid over-smoothing the FID curves, such that we can diagnose the underlying oscillating properties of different methods during training. One can see that WBGAN-based counterparts improve the stability during training, and achieve superior performance over the WGAN-based baselines. We emphasize that the converged FID values reported by WBGAN-div and WBGANGPReal are lower than those reported by WGAN-div and WGAN-GPReal. In particular, WGAN-div suffers several FID fluctuation unexpectedly, and WGAN-GPReal has not ever achieved FID convergence during the entire training process. Regarding WGAN-GP, although the final FID is slightly better than that of WBGAN-GP (6.76 vs. 7.32), we observe a much slower convergence rate in Fig. 2(a). For the generated face images by different approaches, please refer to Fig. 10 in Appendix F for details.
We also investigate a stronger backbone by replacing the network with BigGAN, a conditional GAN architecture that uses spectral normalization on both generator and discriminator. We set the number of labels to be 1 since the CelebA dataset only contains face images. Training curves are shown in Fig. 3 and quantitative results are summarized in Table 1. Among three WGAN-based methods, only WGAN-GP achieves convergence, but its convergence speed and the FID value are inferior to those reported by WBGAN-GP. In opposite, both WGAN-div and WGAN-GPReal fails to converge while the counterparts equipped with WBGAN perform well. For the generated face images by different approaches, please refer to Fig. 11 and Fig. 12 in Appendix F for details.
Under review as a conference paper at ICLR 2020
0 20 40 60 80 100 120 140 160 180 200
Epoch
10
20
30
40
50 60 70 80 90 100 F ID WGAN-GP_exp1 WGAN-GP_exp2 WGAN-GP_exp3 WGAN-GP_exp4 WGAN-GP_exp5 WBGAN-GP_exp1 WBGAN-GP_exp2 WBGAN-GP_exp3 WBGAN-GP_exp4 WBGAN-GP_exp5
20 40 60 80 100 120 140 160 180 200
Epoch
0
20
40
60
80
100 120 140 160 180 200
F ID
WGAN-div_exp1 WGAN-div_exp2 WGAN-div_exp3 WGAN-div_exp4 WGAN-div_exp5 WBGAN-div_exp1 WBGAN-div_exp2 WBGAN-div_exp3 WBGAN-div_exp4 WBGAN-div_exp5
20 40 60 80 100 120 140 160 180 200
Epoch
0
20
40
60
80
100
120
140
160
180
200
F ID
WGAN-GPReal_exp1 WGAN-GPReal_exp2 WGAN-GPReal_exp3 WGAN-GPReal_exp4 WGAN-GPReal_exp5 WBGAN-GPReal_exp1 WBGAN-GPReal_exp2 WBGAN-GPReal_exp3 WBGAN-GPReal_exp4 WBGAN-GPReal_exp5
0 5000 10000 15000
Iterations
-10
0
10
20
30
40
W te rm
WGAN-GP WBGAN-GP
(a) (b)
(c) (d)
Figure 4: Curves of the Wasserstain term, produced by WGAN-GP and WBGAN-GP.
0 10 20 30 40 50 60 70 80 90 100
Epoch
40
80
120
160
200
F ID
Hingeloss WGAN-GP WBGAN-GP WGAN-div WBGAN-div WGAN-GPReal WBGAN-GPReal
0 20 40 60 80 100 120 140 160 180 200
Epoch
20
40
60
80
100
F ID
WGAN-GP WGAN-GP+D-bound WGAN-GP+G-Sinkhorn WGAN-GP+D-bound+G-Sinkhorn
(a) (b)
Figure 5: FID curves of DCGAN-based
ablation study on the CelebA dataset.0 2000 4000 6000 8000 10000
Iterations
-20
-15
-10
-5
0
5
10
15
20
W te
rm
WGAN-GP WBGAN-GP
0 100 200 300 400 500
Iterations
-15
-10
-5
0
5
10
15
W te
rm
WGAN-div WBGAN-div
0 100 200 300 400 500 -10
-5
0
5
10
W te
rm
WGAN-GPReal WBGAN-GPReal
0 2000 4000 6000 8000 10000
Iterations
-150
-100
-50
0
50
100
150
200
G lo
ss
WGAN-GP WBGAN-GP
0 500 1000 1500 2000
Iterations
-20
0
20
40
60
80
G lo
ss
WGAN-div WBGAN-div
0 500 1000 1500 2000 -10
0
10
20
30
40
G lo
ss
WGAN-GPReal WBGAN-GPReal
(a) (b) Iterations (c)
Figure 6: Generator loss in the beginning iterations. BigGAN on CelebA. (a) WGAN-GP vs WBGAN-GP, (b) WGAN-div vs WBGAN-div, (c) WGAN-GPReal vs WBGAN-GPReal.
Wasserstein Loss and Generator Loss Stability. Next, we evaluate the stability of WBGAN in terms of the Wasserstein term and generator loss. In Fig. 4, we evaluate the impact on WGAN-GP (DCGAN on CelebA). One can see that, after the bound is applied, the Wasserstein term W is stablized especially during the start of training. Due to space limit, more results using BigGAN on CelebA are provided in Appendix E. In addition, we compute a new term named the generator loss, which is defined as Gloss = −Ez∼Pz [Dθ(Gφ(z))]. Fig. 6 shows the curves of this statistics during the starting iterations. Compared to WGAN-based approaches, WBGAN-based approaches produce more stable Gloss terms, which verifies that the training process of GAN becomes more stable.
Ablation Study. Before continuing to high-resolution experiments, we conduct an ablation study to investigate the contribution made by different components of WBGAN. The backbone network is DCGAN, and the dataset is CelebA. We compare four configurations, i.e., WGAN-GP, with the original loss term used in WGAN-GP; WGAN-GP+D-bound, which adds a bound (Sinkhorn distance) to the Wasserstein term of the critic D of WGAN-GP; WGAN-GP+G-Sinkhorn, which adds Sinkhorn distance to the loss function of the generator G in WGAN-GP; and WGAN-GP+Dbound+G-Sinkhorn, which is equivalent to the final WBGAN-GP, with Sinkhorn distance added to both critic D and generator G. Fig. 5 plots the FID curves of all four settings. One can see that, although the FID curves of WGAN-GP and WGAN-GP+G-Sinkhorn descend quickly in the first 10 epochs, they begin to fluctuate between 20 to 40 epochs. On the other hand, when WGAN-GP is combined with D-bound, FID is able to descend smoothly (without fluctuation), showing that it is the bounded constraint that stablizes the training process. Finally, by integrating both D-bound and G-Sinkhorn into WGAN-GP, the FID curve descends not only smoothly but also fast, which is what we desire in real-world applications.

heading: 4.3 HIGH-RESOLUTION EXPERIMENTS AND REMARKS
text: In this section, we evaluate our approach on higher-resolution (128 × 128) images. We use the CelebA-HQ dataset (Karras et al., 2017), and use BigGAN (Brock et al., 2018) as the backbone. As
the target become larger (128× 128), the number of images we can feed into a single batch becomes smaller (64). Since we are using an empirical way of estimating Sinkhorn distance, it becomes less accurate in the scenario of small batch size and large image size. In other words, it is no longer the best choice to use Sinkhorn distance to estimate the upper-bound ĎW .
Returning to our generalized formulation, Eq. 7, we note that other forms of bound to constrain the critic. Here we consider a very simple bound, which is also based on empirical study. Note that the baseline methods, though not converging very well, can finally arrive at a stablized W value. Heuristically, we use this constant value (there is no need to be accurate) as the bound, which is 10 for WGAN-GP, 5 for WGAN-div and 3 for WGAN-GPReal, respectively. In Appendix D, we provide the curves of the Wasserstein term for these baselines, which lead to our estimation.
FID curves and quantitative results using these constant bounds are shown in Fig. 7 and Table 2, respectively. We find that WBGAN-GP produces a similar convergence rate with WGAN-GP, WBGAN-div is slightly better than WGAN-div, and WBGAN-GPReal outperforms WGAN-GPReal and produces the best results. For the generated face images by different approaches, please refer to Fig. 13 and Fig. 14 in Appendix F for details.
Discussions. From the above experiments, we can see that Sinkhorn distance is just one way of upper-bound estimation. In case that it becomes less accurate, we can freely replace it with other types of estimation. Besides the constant bound used above, there also exist other examples, such as the two-step computation of the exact Wasserstein distance (Liu et al., 2018). However, it is still a challenge to estimate the Wasserstein distance between high-resolution (1024× 1024) image distributions efficiently. Nevertheless, the most important deliveries of our work are that a bounded Wasserstein term can bring benefits on training stability, and that we can use it to a wide range of frameworks based on WGAN.

heading: 5 CONCLUSIONS
text: This paper introduced a general framework called WBGANs, which can be applied to a variety of WGAN variants to stabilize the training process and improve the performance. We clarify that WBGANs can stabilize the Wasserstein term at the beginning of the iterations, which is beneficial for smoother convergence of WGAN-based methods. We present an instantiated bound estimation method via Sinkhorn distance and give a theoretical analysis on it. It remains an open topic on how to set a better bound for higher resolution image generation tasks.

heading: A PROOF OF PROPOSITION 1
text: Proof. Suppose µ, υ1, υ2∈ Pp(X), t1, t2 ≥ 0, t1 + t2 = 1, then there exist γ1(x, y) and γ2(x, y) with marginals (µ, υ1) and (µ, υ2) satisfying:
W1(µ, υ1) = ∫ X×X ‖x− y‖1dγ1(x, y), (12)
W1(µ, υ2) = ∫ X×X ‖x− y‖1dγ2(x, y). (13)
Let υ = t1υ1 + t2υ2, γ(x, y) = t1γ1(x, y) + t2γ2(x, y), then γ(x, y) has marginals (µ, υ). We can derive: W1(µ, υ) ≤ ∫ X×X ‖x− y‖1dγ(x, y)
= t1 ∫ X×X ‖x− y‖1dγ1(x, y) + t2 ∫ X×X ‖x− y‖1dγ2(x, y)
= t1W1(µ, υ1) + t2W1(µ, υ2).
(14)
This conclusion can be extended to a general form:
W1(µ, υ) ≤ t1W1(µ, υ1) + t2W1(µ, υ2) + . . . tnW1(µ, υn), (15)
where υ1, υ2, . . . υn ∈ Pp(x), t1, t2, . . . tn ≥ 0, t1+t2+ · · ·+tn = 1, υ = t1υ1+t2υ2+ · · ·+tnυn. Suppose P̂gi(1 ≤ i ≤ n) are the independent empirical measures drawn from P̂g. From Eq. 15, we can get
W1(Pr, 1
n n∑ i=1 P̂gi) ≤ 1 n n∑ i=1 W1(Pr, P̂gi). (16)
According to the strong law of large numbers, we can derive that with probability 1, 1n ∑n i=1 P̂gi → Pg as n → ∞ (assuming Pg has finite first moments). Since W1 is continuous in Pp(X), we can derive that with probability 1, W1(Pr, 1n ∑n i=1 P̂gi)→W1(Pr,Pg) as n→∞. By the law of large
numbers again, with probability 1, 1n ∑n i=1W1(Pr, P̂gi)→ E[W1(Pr, P̂g)] as n→∞. Thus we can deduce that: W1(Pr,Pg) ≤ E[W1(Pr, P̂g)]. (17)
Similarly, suppose P̂ri(1 ≤ i ≤ n) are the independent empirical measures drawn from P̂r. Since the symmetry of Wasserstein distance, we can deduce that:
W1(Pr, P̂g) ≤ E[W1(P̂r, P̂g)]. (18)
Therefore, combining Eq. 17 and Eq. 18, we can get W1(Pr,Pg) ≤ E[W1(P̂r, P̂g)].

heading: B PROOF OF REMARK 2
text: Proof. First, let Dθ(x) ≡ 0, then
Lθ(Pr,Pg) = max Dθ∈L1 Ex∼Pr [Dθ(x)]− Ex∼Pg [Dθ(x)] − [ Ex∼Pr [Dθ(x)]− Ex∼Pg [Dθ(x)]− dλ(P̂r, P̂g) ] +
≥ Ex∼Pr [0]− Ex∼Pg [0]− [ Ex∼Pr [0]− Ex∼Pg [0]− dλ(P̂r, P̂g) ] +
= 0,
(19)
where dλ(P̂r, P̂g) ≥ 0 is the Sinkhorn distance defined in Eq. 6. Next, if Pr = Pg, then we have Lθ(Pr,Pg) = 0. So we only need to show Lθ(Pr,Pg) > 0 if Pr 6= Pg .
Let Dθ(x) = sign(Pr(x)− Pg(x)), we have w = Ex∼Pr [Dθ(x)]− Ex∼Pg [Dθ(x)]
= ∫ (Pr(x)− Pg(x)) · sign(Pr(x)− Pg(x))dx
> 0
(20)
Applying this into Eq. 8 leads to
Lθ(Pr,Pg) = max Dθ∈L1 Ex∼Pr [Dθ(x)]− Ex∼Pg [Dθ(x)] − [ Ex∼Pr [Dθ(x)]− Ex∼Pg [Dθ(x)]− dλ(P̂r, P̂g) ] +
≥ w − [ w − dλ(P̂r, P̂g) ] +
= { w, if w ≤ dλ(P̂r, P̂g) dλ(P̂r, P̂g). otherwise
(21)
Since Pr 6= Pg, we know that dλ(P̂r, P̂g) > 0. Therefore, we have Lθ(Pr,Pg) > 0 while Pr 6= Pg. We finish the proof.

heading: C PROOF OF REMARK 3
text: Proof. Let Pr(x) = δ(x−α), Pg(x) = δ(x− β) and α 6= β, then we have Lθ(Pr,Pg) = max
Dθ∈L1 Ex∼Pr [Dθ(x)]− Ex∼Pg [Dθ(x)] − [ Ex∼Pr [Dθ(x)]− Ex∼Pg [Dθ(x)]− dλ(P̂r, P̂g) ] +
= max Dθ∈L1
Dθ(α)−Dθ(β)− [ Dθ(α)−Dθ(β)− dλ(P̂r, P̂g) ] +
= max Dθ∈L1 { Dθ(α)−Dθ(β), if Dθ(α)−Dθ(β) ≤ dλ(P̂r, P̂g) dλ(P̂r, P̂g). otherwise
(22)
We know that Wasserstein distance W (Pr,Pg) = maxDθ∈L1 Dθ(α) − Dθ(β). Since Pr,Pg are Dirac distributions, then we have W (Pr,Pg) = dλ(P̂r, P̂g). Combining this into Eq. 22 leads to Lθ(Pr,Pg) = dλ(P̂r, P̂g).
Considering that Sinkhorn distance dλ(P̂r, P̂g) (Cuturi, 2013) allows gradient back-propagation, we finish the proof.

heading: D CURVES OF THE WASSERSTEIN TERM ON CELEBA-HQ DATASET
text: Fig. 8 shows the convergence curves of Wasserstein term for three WGAN methods. Convergence values are different for different WGANs. For example, WGAN-GP converges to 10. WGAN-div is 5. WGAN-GPReal is 3. We use these values as bound.

heading: E ADDITIONAL STABILITY EXPERIMENTS ON THE WASSERSTEIN TERM
text: Fig. 9 shows the curves of Wasserstein term in the beginning iterations. Compared to the WGANbased method, WBGAN-based method is more stable. The network architecture is BigGAN and the dataset is CelebA.

heading: F SAMPLES AND INTERPOLATIONS FROM FACE MODELS
text: Here we display a few generated samples of face images by different approaches on the CelebA and CelebA-HQ datasets.
0 2000 4000 6000 8000 10000
Iterations
-150
-100
-50
0
50
100
G lo
ss
0 500 1000 1500 2000
Iterations
-20
0
20
40
G lo
ss
0 500 1000 1500 2000
Iterations
-10
0
10
20
G
lo
ss
(d) (e)
(f )
(a) WGAN-GP (b) WBGAN-GP
(a) WGAN-GP (b) WBGAN-GP
(c) WGAN-div (d) WBGAN-div
(a) WGAN-GP (b) WBGAN-GP
(a) WGAN-GP (b) WBGAN-GP
(c) WGAN-div (d) WBGAN-div
(a) WGAN-GP
(b) WBGAN-GP
(c) WGAN-div
(d) WBGAN-div

Review summaries:
The paper proposes a new way of stabilizing Wasserstein GANs by using Sinkhorn distance to upper-bound the objective of WGAN's critic's loss during the training. GAN stabilization is a well-motivated problem and limiting the dramatic changes of discriminator loss clearly helps achieving this goal. Experiments show that in few settings the proposed method successfully addresses this issue. However, the theoretical insights have multiple flaws and incorrect proofs, while the choice of experiments raises few questions. Finally, the resolution limitations suggest the overall contribution is rather incremental.