Wasserstein-Bounded Generative Adversarial Networks
Authors: Peng Zhou
Bingbing Ni
Lingxi Xie
Xiaopeng Zhang
Hang Wang
Cong Geng
Qi Tian
Venue:  (2020)
Keywords: GAN
WGAN
GENERATIVE ADVERSARIAL NETWORKS

Abstract: In the field of Generative Adversarial Networks (GANs), how to design a stable training strategy remains an open problem. Wasserstein GANs have largely promoted the stability over the original GANs by introducing Wasserstein distance, but still remain unstable and are prone to a variety of failure modes. In this paper, we present a general framework named Wasserstein-Bounded GAN (WBGAN), which improves a large family of WGAN-based approaches by simply adding an upper-bound constraint to the Wasserstein term. Furthermore, we show that WBGAN can reasonably measure the difference of distributions which almost have no intersection. Experiments demonstrate that WBGAN can stabilize as well as accelerate convergence in the training processes of a series of WGAN-based variants.

Under review as a conference paper at ICLR 2020
WASSERSTEIN- GENERATIVE ADVERSAR- IAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
In the field of Generative Adversarial Networks (GANs). how to design stable training strategy remains an open problem. Wasserstein GANs have largely pro- moted the stability over the original GANs by introducing Wasserstein distance, but still remain unstable and are prone a variety of failure modes. In this paper, we present general framework named which improves a large family of WGAN-bbsed approaches by simply adding show that
Beasserstein-Boundee
GAN (WBGAN),
upper-bound constraint to the Wasserstein term. Furthermore, WBGAN can reasonably measure the difference of distributions which almost have intersection. Experiments demonstrate that WBGAN can stabilize well as accelerate convergence in the training processes of a series of WGAN-based variants.
INTRODUCTION
Over the past few years, Generative Adversarial Networks (GANs) have shown impressive results in many generative tasks. They are inspired by the game theory, that two models compete with each other: generator which seeks produce samples from the same distribution as the data, and a discriminator whose job is to distinguish between real and generated data. Both models are forced stronger simultaneously during the training process. GANs are capable of producing plausible synthetic data across a wide diversity of data modalities, including natural images (Karras et al. 2017)] Brock et 2018 Lucic et al.] 2019)). natural language (Press et al., (2017: Lin et al.] 2 Rajeswar et al. 2017 music (Yang et al. 2017 Mogren 2016 Dong et al. 2017 Dong & Yang 2018), etc.
Despite their success, it is often difficult train a GAN model in a fast and stable way, and researchers are facing issues like vanishing gradients, training instability, mode collapse, etc. This has led to proliferation of works that focus on improving the quality of GANs by stabilizing the training procedure (Radford et al. 2015 Salimans et al., 2016 Zhao et al., 2016 Nowozin et al.) particular, Arjovsky et al. (2017)
2016 Chen et al., 2016, Qi 2017: Deshpande et al., 2018).
introduced variant of GANs based on the Wasserstein distance, and releases the problem of gradient disappearance to some extent. However, WGANs limit the weight within a range to enforce the continuity of Lipschitz, which can easily cause over-simplified critic functions (Gulrajaani al.) (017). To solve this issue, Gulrajani et al. (2017) proposed a gradient penalty method termed WGAN-GP, which replaces the weight clipping in WGANs with gradient penalty term. As such, WGAN-GP provides a more stable training procedure and succeeds in a variety of generating tasks. Based
on WGAN-GP, more works (Wei al. 2018 Petzka et al. 2017 Wu et al. 2018 Mescheder et al. 2018 Thanh-Tung et al., 2019 Kodali et al., [017: Kim et al. 2018) adopt different forms
of gradient penalty terms to further improve training stability. However, it is often observed that
such gradient penalty strategy sometimes generate samples with unsatisfying quality, or even do not always converge to the equilibrium point (Mescheder et al., 2018).
In this paper, we propose a general framework named Wasserstein- Boundeed GAN (WBGAN), which improve the stability of WGAN training by bounding the Wasserstein term. The highlight is that the instability of WGANs also resides in the dramatic changes of the estimated Wasserstein distance during the initial iterations. Many previous works just focused on improving the gradient penalty term for stable training, while they ignored the bottleneck of the Wasserstein term. The proposed training strategy is able to adaptively enforce the Wasserstein term within a certain value, so as to balance the Wasserstein loss and gradient penalty loss dynamically and make the training process more stable.

---

Under review as a conference paper at ICLR 2020
WBGANs generalized, which can be instantiated using different kinds of bound estimations, and incorporated into any variant of WGANs to improve the training stability and accelerate the conver- gence. Specifically, with Sinkhorn distance (Cuturi 2013) Genevay et al. 2017) for bound estimation, we test three representative variants of WGANs (WGAN-GP (Gulrajani et al., 2017), WGAN- div (Wu al. 2018), and WGAN-GPRent (Mescheder et al. 2018) on the CelebA dataset (Liu iet al., (2015). As shown in Fig. | WBGANs outperform the corresponding counterparts, which demonstrates that the bounded strategy results in more stable training and accelerates the convergence.
160
WIGA-GPP
2 BACKGROUNDS
HIGGA
VICAN 2i
120
WIBGAN-dy
WIAN--PPRel
Wasserstein GANs (WGANs), WGANs (Arjovsky
WEGAN GPReal
al. 2017) were primarily motivated by unstable train- ing caused by the gradient vanishing problem of the original GANs (Goodfellow et al. 2014). They pro- posed to use 1-Wasserstein distance Wi (Ir, Pg) to measure the difference between P, and p.. the real and generated distributions, given that W1 (P,,, Fg) continuous everywhere and differentiable almost everywhere under mild assumptions. The objec- tive of WGAN formulated using the Kantorovich- Rubinstein duality (Villani 2008):
40
160
Epoch
Figure 1: Instability in the training process of three variants of WGANs, iee. WGAN- GP. WGAN-div and WGAN-GPReal.
min max EE-P, [D(x)) [=) - Ea-P, (D(r,((())), DEC,
(1)
where Li is the function space of all D satisfying the 1-Lipschitz constraint ||] S 1. D is a critic and G is the generator, both of which are parameterized by a neural network. Under an optimal critic, minimizing the objective with respect to G' is to minimize W1 (Ir P,). To enforce the I-Lipschitz constraint on the critic, WGAN used a weight clipping on the critic to constrain the weights within compact range, cc.,c]. which guarantees the set of critic functions is a subset of the k-Lipschitz functions for some k. With weight clipping, the critic tends to learn over-simplified functions (Gulrajani et al. 2017), which may lead unsatisfying results. Gulrajani al. (2017): : Weil et al. (Ptzka et al. (017)) Wu et al. (2018) proposed different forms of gradient penalty as regularization term, so that generalized loss function with respect to the critic can be written as:
LD Ee- [D()] Ex-P, 3 [D) + AGP GP,
(2)
where Ep E [D(x) - [@)] stands for the Wasserstein term, and GP for the gradient penalty term. LD is actually posing tradeoff between these two objectives.
Wasserstein Distance between Empirical Distributions. In practice, we approximate W1 (P,, Pg)) using Wi (P,,, P3). where T-, and Pg denote the empirical version of P, and Pg with N samples, i.., P., N Ein dy., and Pr, N Σ 6G(x.). Here, y: is randomly sampled from the real image dataset, and 8y, is the Dirac delta function at location ye. Computing W1 (P...P.) is typical problem named discrete optimal transport. We denote B as the set of probabilistic couplings between two empirical distributions defined as:
B = [ PERRN NXN
FI, IN 1r
(3)
where 1 v is a N-dimensional all-one vector. Then we have W1 (P,,,, min (T. C) where ( is the Frobenius dot- product and C is the cost matrix, with each element Ct, = c c()()))) y
denoting the cost to move a probability mass from G(z,) to yz. The optimal coupling is the solution of this minimization problem: To- = arg minres F.
The Sinkhorn Algorithm. Despite Wasserstein distance has appealing theoretical properties in measuring the difference between distributions, its computational costs for linear programming are often high in particular when the problem size becomes large. To alleviate this burden, Sinkhorn distance (Cuturi 2013) was proposed to approximate Wasserstein distance:
d, Pg) == min (P.C),
(4)
(f...)
2

---

Under review as a conference paper at ICLR 2020
where Ua (Pr, Py) is subset of B defined in Eq.
Ua (F.. = (T (T)
H(P,) a C,
(5)
where H.(-) is the entropy defined as HT) = - ش.j F log I. and = n Pm log Pr. where P.. is the probability of the n-th sample, Compared to Wasserstein distance, Sinkhorn distance
restricts the search space of joint probabilities those with sufficient smoothness. To compute Sinkhorn distance, a Lagrange multiplier was used:
d (ir, Py) -(*, r arg min (T, C) HT).
(6)
Each a corresponds À E [0, o0) such that d P.) d (P,,,P)) holds for that pair (Ir, pg). (r,-) can be computed with much cheaper cost than the original Wasserstein distance using matrix scaling algorithms. For À > O, the solution r is unique and has the form r = diag(u K diag(u), v), where K is the element- wise exponential of -XC. and two non-negative vectors uniquely defined up to a multiplicative factor (Cuturi 2013).
3
WASSERSTEIN-BOUNEE GANS
3.1 BOUND CONSTRAINT ON 1- WASSERSTEIN DISTANCE
We start with Eq. and denote W E.p.P [D(x)] Eg [D())]. W is often referred as the Wasserstein term, which is unbounded during the training process. In a wide range of WGAN's variants such as WGAN-GP (Gulrajani al.) 2017). the critic defined by Lp is to maximize the Wasserstein term W while satisfying the gradient penalty GP. However, in practice, we find that W often rises rapidly to a tremendous value which is far from rational during the initial training procedure. A possible reason may lie in that the critic function does not satisfy the Lipschitz constraint during the initial training stage. shown in Fig. [ this leads dramatic instability in optimization and finally results in unsatisfying performance in image generation.
Our idea is thus straightforward, i.e., setting an upper-bound for W. The modified critic loss function is written as:
LD = WW W + IW - W]+ + AGP GP.
where - max (-: (-,0) 0} is the ramp function that ignores negative inputs, and W denotes the upper bound of W, and will be discussed later. If W < W, the term [W WW,, simply vanishes and Eq. equivalent to Eq. 2, otherwise, we have W + [W W]+ W, which implies that the modified Wasserstein term is bounded by W.
Our formulation brings benefit the numerical stability of the Wasserstein term. In practice, remains comparable to the other term, AGP GGP, so that both W and GP can be optimized in 'mild' manner, i.e., without any one of them dominating or being ignored during training. Note that the W term cannot be chosen arbitrarily. Setting it too small, W will limit the capacity of the critic function, resulting in poor generation. Setting it too large, there will be no effect of bounding the W term.
The proposed bounded strategy is general framework. We name it general in two folds: First, WBGAN can be applied to almost all gradient penalty based WGANs, such as WGAN-GP (Gulrajani et al. 2017), WGAN-GPRes (Mescheder et al. 2018), etc. Moreover, there are different ways to estimate the value of W. For example, the linear programming was applied successfully to some existing WGANs like WGAN-TS (Liu et al. 2018). In what follows, we present example which uses Sinkhorn distance to estimate W, while we believe other ways of estimation are also possible.
3.2 WBGAN WITH SINKHORN DISTANCE
In this section, we give an instantiation, Sikhorn distance (Cuturi 2013). to effectively compute the bounded term W. The motivation of using Sinkhorn distance lies in that in theory, the Wasserstein term of WGAN will eventually converge to the 1-Wasserstein distance between the real distribution P, and the generated distribution IP, (Arjovsky et al. 2017 Gulrajani et al. 2017). Therefore, we can use the I-Wasserstein distance between the empirical distributions, IP, and I,,,, as the upper-bound W. Since the computation of Wasserstein distance involves large linear programming which
3

---

Under review as a conference paper at ICLR 2020
Algorithm 1 WBGAN with Sinkhorn distance
Require: learning rate a, batch size M, the number of iterations of the critic per generator iteration
Neritic. weight of gradient penalty AGP, weight of Sinkhorn distance Xa, initial parameters 0 and Po. other hyper-parameters:
1: while &e has not converged do
2: for 1..... Naritic do
3: Sample a batch (x(m) } M
Pr from real data;
m==1
4:
Sample batch {=(m) LM
m== P= of prior samples; Em=1 Dø Da(ar (m)) W M
5:
1/ >m== Do (Z, Calculate Sinkhorn distance d (IP,, JiPr,Pa) Pg) between (x((m) jmm and {Go, ( }
M
6:
7:
- + IW -d (P,,,P Py]]+ + AGP GPP
8:
+ Adam(Le 0, a, 31, 82):
9:
end for
10: Sample a batch (z (z(m) | ~ P2 of prior samples;
11: Calculate Sinkhorn distance d (Apr. Pr. P.) between (m(m)) Jm=1 and {Go, )
12:
Lo. EE--, &G (2))] + A d (P.,
13:
Φ1+1 - Adam( : Pt. at 31, 32);
end while Ensure:
trained parameters and or (converged).
suffers heavy computational costs, we replace it by Sinkhorn distance instead the Sinkhorn distance between Pr, and P can be computed using Sinkhorn's 's matrix scaling algorithm (Cuturi 2013). which is orders of magnitude faster than the linear programming solvers.
Mathematically, consider generator function Gø(z) that produces samples by transforming noise input drawn from simple distribution IPz. e.g., Gaussian distribution. Da stands for critic function parameterized by 0. The objective of the critic is:
(Prr Pg) DoELi
Ex~P, [Da(x)] Ex [D(
(8)
Emap, [a(#)) - Ex-r, (Du()) (P,,,
where dA PPy) is the Sinkhorn distance defined in Eq. 6
On the other hand, given a fixed critic function Dø., considering that Sinkhorn distance allows gradient back-propagation (Genevay et al. 2017). we can find the optimal generator Gø- by solving:
arg min
-Eg-r, (D- (G& (z))] + Ag (P,,
(9)
where >ę is a balancing parameer which we set A 0.5 this paper. Algorithm I we summarize the flowchart of training WBGAN with Sinkhorn distance.
3.2. RELATIONSHIP BETWEEN d' (Tr., P,)) AND Wi (P.,,,,)
We employ (f,r, PP separable metric space. P(X) denotes the set of Borel probability measures. (x) denotes the set of all μ E P(X) such that Jx d(æ, dz)x) +oo for some y X. We can suppose real data distribution Pr,, generated data distribution FF, and their empirical distribution iž,, and IP, all in (X).
approximation of 1-Wasserstein distance Wi (P,, P.). Let (X, d) be
Proposition 1. Let P., and P., be real data distribution and generated data distribution. Suppose that P, and P, empirical measures of IP, and Pg. Then have 0 & W1 (P,P,,Pp) < E(W, (P,, 1 .)
Proof. Please refer to Appendix
Proposition tells us that as W ( P.)] + 0, (P,,,, forced to 0. Cuturi 2013 has pointed out that if À is chosen large enough, dd (f,,, (,P,,,,) Py) coincides with W, (IP, P.). So, it is reasonable d (Tp,. IP,)) constrain the Wasserstein term.
4

---

Under review as a conference paper at ICLR 2020
3.3 ANALYSIS OF WBGAN
Most GANs measure the distance between distributions based on probability divergence. We will indeed valid divergence. First, have the following definition.
prove that the Eq. 8
Definition 1. Given probability measures p and D is a functional of p and q. If 'D satisfies the following properties:
1. D(p.g) 2 0;
(10)
2. p= = D(p, 0) O,
then we say D is probability divergence between p and q.
Remark 1. The following W W(Pr; (p)) satisfies the Definition and is therefore probability diver- gence.
W (FF,, max Ex-F, DEC:
[D())] [=-r,[((x))
Ea-r, [b))],
(11)
where Li is the 1-Lipschitz constraint. Please see the proof and detailed discussion in Su (0018). This is the objective of critic used by WGAN (Arjovsky et al. 2017).
Remark Equation
X satisfies the Definition and a probability divergence.
Proof. The proof is given in Appendix B
Remark 3. Consider two distributions (,(x) 8(æ a),, P2(2) 5(x (3) that have no intersection (a * (). 5 is the Dirac delta function. In such an extreme case, Eq. 8 can still be optimized by gradient descent.
Proof. The proof is in Appendix
Remark 2 tells us that Eq. 8 is a valid divergence. Since the real data distribution is supported by low- dimensional manifolds, the supports of generated distribution and real data distribution are unlikely to have a non-negligible intersection. Remark shows that compared to the standard GAN (Goodfellow et al. 2014), WBGAN can continuously measure the difference between two distributions, even if there is almost no intersection between the distributions.
EXPERIMENTS
4.1 SETTINGS AND BASELINES
To verify that WBGAN is generalized approach, we select three variants of WGAN, namely, WGAN-GP (Gulrajani et al. 2017). WGAN-div (Wu et al., 2018) and WGAN-GPReal (gradient baselines. By adding bound constraints penalty real data only) (Mescheder al. 2018) WBGAN-GP, WBGAN-div, and WBGAN- to these WGAN variants, we obtain the counterparts
GPReal, respectively. Two different network architectures are used, i.e., DCGAN (Radford et al.) 2013) and BigGAN (Brock et al. 2018). For DCGAN, we directly output the activation before the sigmoid layer. BigGAN is conditional GAN (Mirza & Osindero 2014) architecture, which class conditioning is passed to generator by supplying it with flass-conditionall gains and biases in the batch normalization layer (lofte & Szegedy (015 de Vries al. 2017 Dumoulin al. (017). In addition, the discriminator is conditioned (Miyato & Koyama 2018) by using the cosine similarity between its features and set of learned class embedding. We use the spectral norm (Miyato et al.) 2018 in BigGAN, but for the sake of simplicity, we do not use the self-attention module (Wang et al. 2017 Zhang et al. 2018). Other hyper saper-pprameters and the network architecture of BigGAN simply
follow the original paper.
We choose the Fréchet Inception Distance (FID) (Heusel et al. 2017) for quantitative evaluation, which has been proven to be more consistent with individual assessment in evaluating the fidelity and variation of the generated image samples.
4.2 MID- RES-LUSSONT EXPERIMENTS
We first investigate mid-resolution image generation on the CelebA dataset (Liu et al. 2015), large-scale face image dataset with more than 200K face images. During training, we crop 108 x 108 face from the original images and then resize them to 64 x 64.
5

---

Under review as a conference paper at ICLR 2020
80
130
100
50
100
100
20-
49-
20
40
00
80 100 120
140 300 200 200
20 40
50
140 100 180 Eoch (b)
100
129
200
40
0d
100
120
140
100
100 200
Epoch
Epoch
(e)
Figure 2: FID curves on the CelebA dataset, with WGAN-GP, WGAN-div and WGAN-GPReal as baselines, respectively. Each figure contains 5 individual runs for both each counterpart.
Table 1: FIID comparison between WGAN-based methods and WBGAN- methods. The BigGAN architecture uses spectral normalization in the generator and discriminator, and the number of conditional labels set to be because the training dataset only contains face images.
Network urchitecture Loss
Dataset
Resolution
Batch
Puram(M)
Param(M)
WGAN-GP
6.76 :: 0.17
WBGAN-GP (ours)
7.32 0.55
DCGAN
WGAN-div
CelebA
64 64
128
5.1
4.3
13.94 2.67
WBGAN-divv (ours)
6.26 0.30
WGAN GiRRaa
34.92 - 6.84
WBGAN-GPReal (ours)
6.01 0.33
WGAN-GP
13.39
WBGAN-GP (ours)
6.97
BigGAN
WGAN-div
WBGAN-div (ours)
CelebA
64 x
64
128
8.4
4.9
15.93
7.23
WGGAN-PPRa
42.71
WBGANN-GPRE (ours)
9.61
FID Stability. We first use DCGAN to build our generator and discriminator. Training curves are shown in Fig. 2 and quantitative results are summarized in Table [ Each approach is exccuted for 5 times and the average is reported. All FID curves are obtained from generators directly without using the moving average strategy (Karras et al.] 2017 Mescheder al. 2018 Brook al. 2018 YazIcret al. 2018) to avoid over-smoothing the FID curves, such that we can diagnose the underlying oscillating properties of different methods during training. One can see that WBGAN-based counter- parts improve the stability during training, and achieve superior performance over the WGAN-basee baselines. We emphasize that the converged FID values reported by WBGAN- and WBGAN- GPReal are lower than those reported by WGAN-div and WGAN-GPReal. In particular, WGAN-div suffers several FID fluctuation unexpectedly, and WGAN-GPReal has not ever achieved FID conver- gence during the entire training process. Regarding WGAN-GP, although the final FID is slightly bet- ter than that of WBGAN-GL (6.76 vs. 7.32), we observe a much slower convergence rate in Fig. 2[).. For the generated face images by different approaches, please refer to Fig. 10 Appendix For details. We also investigate a stronger backbone by replacing the network with BigGAN, conditional GAN archi- tecture that uses spectral normalization on both gen- erator and discriminator. We the number of labels to be 1 since the CelebA dataset only contains face images. Training curves are shown in Fig. and quanti- tative results are summarized in Table Among three WGAN-based methods, only WGAN-CA achieves con- vergence, but its convergence speed and the FID value are inferior to those reported by WBGAN-GP. In op- posite, both WGAN-div and WGAN-GPReal fails to converge while the counterparts equipped with WB- GAN perform well. For the generated face images by different approaches, please refer to Fig. II and Fig. [2 in Appendix F for details.
200
MANN
WRAN OPtee
120
20
10
50
10
100
poch
Figure FID curves of BigGAN-based approaches on the CelebA dataset.

---

Under review as a conference paper at ICLR 2020
100
WGAN-GP WBGAN-GP
40-bound
30
80
WGAA ---dhoom
VIG.M--22 CP 40-000U1H6 Srhhom
W term
FID
20
-10
5000
10000
15000
200
Iterations
Epoch
Figure 4: Curves of the Wasserstain term, pro- duced by WGAN-GP and WBGAN-GP.
Figure 5: FID curves of DCGAN-based ablation study on the CelebA dataset.
200
80
40
AGAR-GP OF MEGANGA dP
IGGA ON WEGAN-99 dV
150
WEGAN GFRed
60
30
100
losss
50
G tooos
40
20
20
10
30
-102
152
-10
2000
4003
6000
5000
10000
500
1000 Iterations (b)
1500
2000
500
1000
1500
2000
Iterations
Iterations (6)
(8)
Figure Generator loss in the beginning iterations. BigGAN on CelebA. (a) WGAN-GP vs WBGAN-GP, (b) WGAN-div vs WBGAN-div, (c) WGAN-GPReal vs WBGAN-GPReal.
Wasserstein Loss and Generator Loss Stability. Next, we evaluate the stability of WBGAN in terms of the Wasserstein term and generator loss. In Fig. we evaluate the impact on WGAN-GP (DCGAN on CelebA). One see that, after the bound is applied, the Wasserstein term W is stablized especially during the start of training. Due to space limit, more results using BigGAN on CelebA are provided in Appendix E In addition, we compute
term named the generator loss, which is defined as Ginss = - Eg-P, (=)() Fig. shows the curves of this statistics during the starting iterations. Compared WGAN-based approaches. WBGAN-basse approaches produce more stable Gloes terms, which verifies that the training process of GAN becomes more stable.
Ablation Study. Before continuing to high-resolution experiments, we conduct an ablation study to investigate the contribution made by different components of WBGAN. The backbone network DCGAN, and the dataset is CelebA. We compare four configurations, i.e., WGAN-GP, with the original loss term used in WGAN-GP; WGAN-GP+D GP+D-boundd which adds bound (Sinkhorm CRANN-GP-G-Sinhoom which
distance) the Wasserstein term of the critic of WGAN-GP; adds Sinkhorn distance to the loss function of the generator G in WGAN-GP: and WGAN-GP+D+ Moun+GG-Sinhoome which is equivalent to the final WBGAN-GP, with Sinkhorn distance added to both critic D and generator G. Fig. plots the FID curves of all four settings. One can see that, although the FID curves of WGAN-GP GP and "GGAN-GPG--Sinhhor descend quickly in the first 10 epochs, they begin to fluctuate between 20 to 40 epochs. On the other hand, when WGAN-GP is combined with D-bound, FID is able to descend smoothly (without fluctuation), showing that is the bounded constraint that stablizes the training process. Finally, by integrating both D-bound and G-Sinkhorn into WGAN-GP, the FID curve descends not only smoothly but also fast, which is what we desire in real-world applications.
4.3 RESOLUTINO EXPERIMENTS AND REMARKS
In this section, we evaluate our approach on higher-resolution (128 x 128) images. We use the CelebA-HQ dataset (Karras et al. 2017). and use BigGAN (Brock al. 2018 the backbone. As
7

---

Under review as a conference paper at ICLR 2020
Table 2: FID comparison in high-dimensional experiments.
Network urchitecture
Loss
Dataset
Resolution
Batch
Parum(M)
D Puram(M)
FID
WGAN-GP
17.58
WBGAN- CP (ours)
18.32
BigGAN
WGAN-div
CelebA-HQ
128 x 128
64
8.4
9.6
21.05
WBGAN-div (ours)
17.26
WGAN-GPReal
21.33
WBGGN-GGPeea (ours)
12.87
110
110
110
100
WGAR-O WBGAMIGT
100
RGAR
WBCA
100
RCARRGFRIC WAGANGGPRAN
70
60
62
50
20
22
10
10
Epoch
Epoch
Epoch
(5)
(b)
(c)
Figure 7: Curves of FID using BigGAN network architecture on the CelebA-HQ dataset. (a) WGAN- GP vs WBGAN-GP, (b) WGAN-div vs WBGAN-div, (c) WGAN-GPReal vs WBGAA-GPReal. Lower is better.
the target become larger (128 X 128), the number of images we can feed into a single batch becomes smaller (64). Since we are using an empirical way of estimating Sinkhorn distance, it becomes less accurate in the scenario of small batch size and large image size. In other words, it is no longer the best choice to use Sinkhorn distance to estimate the upper-bound W.
Returning to our generalized formulation, Eq. 7 we note that other forms of bound to constrain the critic. Here we consider very simple bound, which also based on empirical study. Note that the baseline methods, though not converging very well, can finally arrive at stablized W value. Heuristically, we use this constant value (there is no need to be accurate) as the bound, which is 10 for WGAN-GP, 5 for WGAN-div and 3 for WGAN-GPRee provide the curves of the Wasserstein term for these baselines, which lead to our estimation.
respectively. In Appendix D we
FID curves and quantitative results using these constant bounds are shown in Fig. 7 and Table 2 respectively. We find that WBGAN-GP produces a similar convergence rate with WGAN-GI WBGAN-div is slightly better than WGAN-div, and WBGAN-GPReal outperforms WGAN-GPReal and produces the best results. For the generated face images by different approaches, please refer to Fig. 13 and Fig. 14 in Appendix F for details.
From the above experiments, we can see that Sinkhorn distance is just one way of upper-bound estimation. In case that it becomes less accurate, we can freely replace it with other types of estimation. Besides the constant bound used above, there also exist other examples, such as the two-step computation of the exact Wasserstein distance (Liu et al. 2018). However, it is still a challenge to estimate the Wasserstein distance between high-resolution (1024 x 1024) image distributions efficiently. Nevertheless, the most important deliveries of our work are that a bounded Wasserstein term can bring benefits on training stability, and that we can use it to wide range of frameworks based on WGAN.
Discussions.
CONCLUSIONS
This paper introduced general framework called WBGANs, which can be applied to a variety of WGAN variants to stabilize the training process and improve the performance. We clarify that WBGANs can stabilize the Wasserstein term at the beginning of the iterations, which is beneficial for smoother convergence of WGAN based methods. We present instantiated bound estimation method via Sinkhorn distance and give a theoretical analysis on it. It remains an open topic on how to set better bound for higher resolution image generation tasks.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein Generative Adversarial Networks. In ICML. pp. 214-223, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. 1r09v 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, llya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. In NIPS, 2016.
Marco Cuturi. Sinkhorn Distances: Lightspeed Computation of Optimal Transport. In NIPS, pp- 2292-2300, 2013.
Harm de Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron Courville. Modulating early visual processing by language. In NIPS, 2017.
Ishan Deshpande, Ziyu Zhang, and Alexander Schwing. Generative Modeling Using the Sliced Wasserstein Distance. In CVPR, pp. 3483-3491 IEEE, 2018.
Hao-Wen Dong and Yi-Hsuan Yang. Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation. arXiv: 09099999999.11 [cs, eess, stat], April 2018.
Wao-Wen Dong, Wen-Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang. MuseGAN: Multi-track Se- quential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment. 17Xiv:/709.06298 [es, eess), September 2017.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A Learned Representation For Artistic Style. ICLR, 2017.
Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning Generative Models with Sinkhorn Divergences. /7Xiv:/706.00022 [stat], June 2017.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), NIPS, pp. 2672-2680. Curran Associates, Inc., 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In NIPS, pp. 5767-5777, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NIPS, 2017.
Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv:1502.03/66 [cs], February 2015.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive Growing of GANs for Improved Quality, Stability, and Variation. 17X1v:/710.10190 [cs, stat), October 2017.
Cheolhyeong Kim, Scungtae Park, and Hyung Ju Hwang. Local Stability and Performance of Simple
Gradient Penalty mu- Wasserstein GAN. arXiv: 10t::/8/..000 18..002222 [es, stat), October 2018.
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On Convergence and Stability of GANs. arXiv: 1705.07215 [cs], May 2017.
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-ting Sun. Adversarial Rank- ing for Language Generation. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), NIPS, pp. 3155-3165. Curran Associates, Inc., 2017.
Huidong Liu, GU Xianfeng, and Dimitris Samaras. A Two-Step Computation of the Exact GAN Wasserstein Distance. In ICMI., pp. 3165-3174, 2018.

---

Under review as a conference paper at ICLR 2020
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, pp. 3730-373333 3738, 2015.
Mario Lucic, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly. High-Fidelity Image Generation With Fewer Labels. arXiv:1903.022) [cs, stat], March 2019.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for GANs do actually Converge? In ICML, 2018.
Mehdi Mirza and Simon Osindero. Conditional Generative Adversarial Nets. arXivv1411.1784 [cs,
star), November 2014.
Takeru Miyato and Masanori Koyama. cGANs with Projection Discriminator. ICLR, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks, arXiv:1802.05957 [cs, February 2018.
Olof Mogren. C-RNN-GAN:
Continuous recurrent neural networks with adversarial training.
arXiv: 161111111000 1.09904 [es], November 2016.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. F-GAN: Training Generative Neural Samplers using Variational Divergence Minimization. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), NIPS, pp. 271-279. Curran Associates, Inc., 2016.
Henning Petzka, Asja Fischer, and Denis Lukovnicov. On the regularization of Wasserstein GANs. arXiv: 08894 [es, stat], September 2017.
Ofrr Press, Amir Bar, Ben Bogin, Jonathan Berant, and Lior Wolf. Language Generation with Recurrent Generative Adversarial Networks without Pre-training, arXiv:1706.0139) [cs], June 2017.
Guo-Jun Qi. 17110:1001.06264 [cs], January 2017.
Sensitive Generative Adversarial
Networks on Lipschitz Densitics.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv: 151 1.0634 [cs], November 2015.
Sai Rajeswar, Sandeep Subramanian, Francis Dutil, Christopher Pal, and Aaron Courville. Adversarial Generation of Natural Language. arXiv: 10929 [es, stat], May 2017.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved Techniques for Training GANs. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), NIPS, pp. 2234-2242. Curran Associates, Inc., 2016.
Jianlin Su. GAN-QP: A Novel GAN Framework without Gradient Vanishing and Lipschitz Constraint. arX1v:1811.072% [cs, stat], November 2018.
Hoang Thanh Truyen Tran, and Svetha Venkatesh. Improving Generalization and Stability of Generative Adversarial Networks. arXiv: /902203984 [cs, stat), February 2019.
Cédric Villani. Optimal Transport: Old and New, volume 338. Springer Science & Business Media, 2008.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local Neural Networks. arXiv:1711.09971 [cs], November 2017.
Xiang Wei, Boqing Gong, Zixia Liu, Wei Lu, and Liqiang Wang. Improving the Improved Training of Wasserstein GANs: Consistency Term and Its Dual Effect. arXiv:1803.0155: 0151 {cs, stat], March 2018.
Jiging Wu, Zhiwu Huang, Janine Thoma, Dinesh Acharya, and Luc Van Gool. Wasserstein divergence for gans. In ECCV, pp. 653-668, 2018
Li-Chia Yang, Szu- Yu Chou, and Yi-Hsuan Yang. MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation. arXiv: /703. 10847 [cs], March 2017.
10

---

Under review as a conference paper at ICLR 2020
Yasin YazIeı, Chuan-Sheng Foo, Stefan Winkler, Kim-Hui Yap, Georgios Piliouras, and Vijay Chandrasekhar. The Unusual Effectiveness of Averaging in GAN Training. 10X1v18866004498 [cs, stat], June 2018
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-Attention Generative Adversarial Networks. arXiv:/805.088/8 [cs, stat], May 2018.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based Generative Adversarial Network. 16X.v:1609.0312 (es, stat], September 2016.

---

Under review as a conference paper at ICLR 2020
A PROOF OF PROPOSITION 1
Proof. Suppose μ4, U2 Pp (x(X), 11, 12 0, 11 + 12 = 1, then there exist Y (2,, y) and 72(2, v) with marginals (ju, v1) and (1, U2) satisfying:
wi(m, = -Sxxx || - vll v),
(12)
wi(H, = / 1|x- 10llvlldd1(( v).
(13)
xxx
Let v 11 + t2v2, ((0, 5) = (x, + t272(2, y). then 7(2, y) has marginals (H, v). We can derive:
u)( S / ull-ylldd
Xxx
= xxx 1| dle-vllidhi yllidhi (2r, v) / xxx
(14)
||00 vll1dr2((,
= WWWWi vi) + (2W (1, U2).
This conclusion can be extended general form:
W (1,,)) W (14 v1) to 02) + W, (1,, un),
(15)
where v1, 1/2-.. Un (3)) 11, 12, In 0, +2 1 li 1202
Un-
Suppose Ps (1 S i n) are the independent empirical measures drawn from P. From Eq.
we
can get
WIPP 17
W (P.. P..)
(16)
94
According to the strong law of large numbers, we can derive that with probability 1, 1 > P. 9.
Fg as > oo (assuming l9 has finite first moments). Since W1 continuous in P (X))), we can
derive that with probability 1, W, (Pr. Ei= ) + Wi Fg)) as n + co. By the law of large numbers again, with probability 1, n E Wi Pg.) (WW(PP,, Py)l - Thus deduce that:
(P. < (F,
(17)
Similarly, suppose (1 n) are the independent empirical measures drawn from P,.. Since the symmetry of Wasserstein distance, we can deduce that:
W WIP,P,,P) < B[W, (f,,, P,)].
(18)
Therefore, combining Eq. 17 and Eq. 18 we can get Wi (P,, P.) S (P,, P.)..
B PROOF OF REMARK 2
Proof. First, let Dø(ar) = 0, then
(P,, PP,) = max DoEL,
Ex-P, [Do(x)] - Em--, [Do(x)]
Eunwe [Do(z))
Eawr, [Do(z))] d (PP. P.. Po,)
(19)
> Exnp, [0] - 10 Em - 00] d (IP Po))
0,
where d (P,,,)))) 2 0 is the Sinkhorn distance defined in Eq.
Next, if IP, = Fg then we have (P,,PPP Pg) = 0. So we only need to show L (P,,PPP)) >
P. * p
12

---

Under review as a conference paper at ICLR 2020
Let Dy(æ:) (ign(P,, - -P)(x)), we have
En-p, [Du(x)] Bm-F, [2x())]
(P, (2) (2)())) sigm (x)(d)
(20)
> 0
Applying this into Eq. leads to Lg (F, Fg)
max
E Bowt [D(=)]] E Ex,, [D,(]
DoECi
(oxx)] - EENP, Da(x)) - d' (Pr,, (Pr.Pa)
(Pr.
(21)
w
w
w - d (P, Pgg otherwise
w (B, (Pr,PP))
Since Prr Pg. we know that d (P,, Pg) > 0. Therefore, we have La (Pr,Fa) > 0 while P. + Fg. We finish the proof.
C PROOF OF REMARK
Proof. Let (xx = 5(x a), a). Px()))))))))))))))))){)) = 6( B3 and a + B, then we have
Lo(Pr, Pg) max DoELI
ExrP, [Do(x)] EawP, Do(x))
EEWP [Bxx),DD(x)) - Ex- [os(x)) - (1r,, (Prr..p))
Do(a) Do( (o((a) Do(B) d (f...Pp))
(22)
max DoEC:
[ (o(a)) Do(g), dA
if Dola) - Do(3) - (PP,),
max DoEL:
otherwise
We know that Wasserstein distance W(P,,, Py), maxD Do(a) Dz( Since P., Py Dirac distributions, then we have ([P,, Pg) = d' (P,, Pg). Combining this into Eq. 22 leads to (PP,,PP,) = (P.,..))))
Considering that Sinkhorn distance d² (P,,, P,))) Cuturi 2013 allows gradient back pock-propagation, we finish the proof.
D CURVES OF THE WASSERSTEIN TERM ON CELEBA-HQ DATASET
Fig. & shows the convergence curves of Wasserstein term for three WGAN methods. Convergence values are different for different WGANs. For example, WGAN-GP converges to 10. WGAN-div is WGAN-GPReal is 3. We use these values as bound.
E ADDITIONAL STABILITY EXPERIMENTS ON THE WASSERSTEIN TERM
Fig. shoows the curves of Wasserstein term in the beginning iterations. Compared to the WGAN- based method, WBGAN-based method more stable. The network architecture is BigGAN and the dataset is CelebA.
F SAMPLES AND INTERPOLATIONS FROM FACE MODELS
Here we display a few generated samples face images by different approaches on the CelebA and CelebA-HQ datasets.
13

---

Under review as a conference paper at ICLR 2020
100
term
-40
-30
100
Iterations (8)
x10
Iterations (b)
X15
Iterations (c)
x30
Figure 8: Wasserstein term for all training iterations. (a) WGAN-GP vs WBGAN-GP, (b) WGAN-div WBGAN-div, (c) WGAN-GPReal WBGAN-GGReal.
DC
15
W temm
W torm
-10
-5
-10
2000
6000
0000
3000
10000
100
200
300
400
500
100
20c
300
400
500
Iterations
Iterations (b)
Iterations (c)
Figure 9: Wasserstein term in the beginning iterations. (a) WGAN-GP vs WBGAN-GP, (b) WGAN- div vs WBGAN-div, (c) WGAN-GPReal vs WBGANNGGReal.
(a) WGAN-GP
(b) WBGAN-GP
14

---

Under review as a conference paper at ICLR 2020
(c) WGAN-div
(d) WBGAN-div
(c) WGAN-GPReal
1 WBGAN-GPReal
Figure 10: Samples of DCGAN CelebA64
15

---

Under review as a conference paper at ICLR 2020
WGAN-GP
(b) WBGAN-GP
(c) WGAN-div
(d) WBGAN-div
16

---

Under review as a conference paper at ICLR 2020
(c) WGAN-GPReal
(f) WBGAN-GPReal
Figure 11: Samples of BigGAN CelebA64
WGAN-GP
(b) WBGAN-GP
17

---

Under review as a conference paper at ICLR 2020
T2S
25
25
25
2
2
WGAN-div
(d) WBGAN-div
(c) WGAN-GPReal
(f) WBGAN-GPReal
Figure 12: Interpolations of BigGAN between on CelebA64
18

---

Under review as a conference paper at ICLR 2020
(a) WGAN-GP
(b) WBGAN GG
(c) WGAN-div
(d) WBGAN-
19

---

Under review as a conference paper at ICLR 2020
(c) WGAN-GPReal
(f) WBGAN-GPReal
Figure 13: Samples of BigGAN on CelebA-HQ122
20

---

Under review as a conference paper at ICLR 2020
(a) WGAN-GP
2
(b) WBGAN-GP
21

---

Under review as a conference paper at ICLR 2020
(c) WGAN-div
(d) WBGAN-div
22

---

Under review as a conference paper at ICLR 2020
(e) WGAN-GPReal
( WBGAN-GPReal
Figure 14: Interpolations of BigGAN between z on CelebA-HOU -QQ188
23


Reviews:
[{"review_id": "rye7_6u1cS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes bounding the Wasserstein term in WGAN with an aim to stabilize training. The basic framework of the proposal, termed WBGAN, is presented in Section 3.1, and an instantiation using the Sinkhorn distance is described in Section 3.2.\n\nThe proposal is demonstrated to empirically improve stability over the baselines in the mid-resolution experiments presented in Section 4.2. On the other hand, in the high-resolution experiments in Section 4.3, the authors suggest that WBGAN with Sinkhorn does not scale well, and show some results with the bound determined on the basis of a separate run to investigate the converged value of the Wasserstein term, which is impractical. Moreover, the empirical performance seems comparable to the baselines in Figure 7. Because of these I would judge the contribution of this paper not strong, so that I would recommend weak acceptance.\n\nIn page 3, lines 20-21, the authors argue \"a possible reason,\" which I do not understand. Take WGAN as an example. With weight clipping taking place over the whole training process, the Lipschitz constraint should automatically be satisfied even in the initial training phase, which would invalidate the authors' argument here.\n\nIn Section 3.2.1, I do not understand why the authors assume that the relevant distributions are in P_p(X) without specifying the value of p. Is it meant to be P_1(X)?\n\nIn Appendix A, I do not understand the sentence \"Suppose \\hat{\\mathbb{P}}_{g_i} are the independent empirical measures drawn from \\hat{\\mathbb{P}}_g.\" It says as if samples from \\hat{\\mathbb{P}}_g were measures, but they are actually values in X.\n\nThose figures showing the FID (Figures 1, 2, 5, and 7) are drawn versus the \"epochs\", whereas those showing the Wasserstein terms (Figure 4) and the generator loss (Figure 5) are drawn versus the iterations, which makes comparison between them impossible.\n\nPage 1, line 43: which improve(s) the stability\nPage 2, equation (3): Here a vector \\Gamma 1_N is equated with an empirical distribution \\hat{\\mathbb{P}}, which does not make sense.\nPage 2, line 40: where 1_N is a(n) N-dimensional\nPage 3, line 8: Each \\alpha corresponds (to) a \\lambda\nPage 3, line 21: As shown in Fig. (2->1)\nPage 3, line 27: W should be italicized.\nPage 3, line 42: Si(n)khorn distance\nPage 6, caption of Figure 2: both each counterpart -> each of both counterparts\nPage 6, line 1: discriminator -> critic\nPage 6, line 24: both WGAN-div and WGAN-GPReal fail(s) to converge\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkgl4zLaKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work represents some new effort to stabilize & improve the training of Wasserstein-based generative networks. To this end, the author(s) have proposed to combine both the primal and dual formulation of Wasserstein distance. By \"clipping\" the dual estimate wrt the primal estimate, or simply an arbitrarily set upper bound,  the author(s) hope to overcome the volatile training of WGANs which might be the consequence of an unbounded dual. Despite the fact that the overall idea is new and the results supported the claim, I am not convinced of its significance. It's an okay paper, but I am expecting more from an ICLR paper. As such, I am willing to endorse this submission. A more detailed review can be found below to help the author(s) improve their work. \n\nStrength. \n+ I like the idea of \"bounding\" the dual W-distance estimate, although, in theory, the observed degeneracy should not happen if the network is always properly regularized. \n+ The overall writing is clear and well-motivated. \n\nWeakness. \n- The key observation that motivates this study is that W-distance often rises rapidly during the early training phase. This can be addressed by alternative tricks, say hot start the discriminator by pretraining it to be close to Lipschitz-1, or clipping the discriminator gradient to avoid drastic changes to the function. An ablation study is justified to pin down the real cause of this gain. \n- It is not clear from the paper whether the  []_+ op back-propagates gradient or not, which will make a big difference here. If it backpropagates gradients, then when the dual estimate is larger than the threshold, then the computation is wasted (zero gradients). If not so, then the proposed solution does not affect gradient at all. \n- It's worth noticing that vanilla Sinkhorn distance does not work well for generative purposes (see recent MMD-GAN and Sinkhorn-GAN papers, which all computes Sinkhorn distance in the feature space rather than data space). This questions the necessity of using the Sinkhorn estimate and why it works here. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gjePeUFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new way of stabilizing Wasserstein GANs by using Sinkhorn distance to upper-bound the objective of WGAN's critic's loss during the training. GAN stabilization is a well-motivated problem and limiting the dramatic changes of discriminator loss clearly helps achieving this goal. Experiments show that in few settings the proposed method successfully addresses this issue. However, the theoretical insights have multiple flaws and incorrect proofs, while the choice of experiments raises few questions. Finally, the resolution limitations suggest the overall contribution is rather incremental.\n\nThe paper thus requires further work in terms of design, theory and experiments; for instance the choice of Sinkhorn distance as a boundary-heuristic might be too limiting. Although the main bounding strategy seems to be a promising idea, this work does not meet the quality requirements of ICLR. \n\nPros:\nA. Conceptual simplicity of the general method of bounding discriminator's objective.\nB. The method works well for a few variants of Wasserstein GAN on medium resolutions (64x64, 128x128).\n\nIssues:\n1. Resolution limitation. The upper-bound cannot be reasonably computed for higher resolutions in mini-batch training.\n2. It is unclear how costly the method is in terms of resources and training time.\n3. Experiments were done with CelebA/CelebA-hq with DCGAN and BigGAN architectures at 64x64 resolution and BigGAN architecture for 128x128 resolution. However, they lack comparison with vanilla BigGAN (trained with hinge loss) and the explanation of the architectual differences. Authors only mention the lack of self-attention, yet the used architecture has much less parameters than the original BigGAN; it is also unclear how BigGAN for 64x64 resolution should look like. In fact, the choice of BigGAN for 64x64 resolution is questionable, as this model has been designed for datasets of much higher complexity. \n4. In two out of three experiments, WBGAN-GP does not beat WGAN-GP. Given BigGAN's results at 64x64, it might be the case that bounding strategy just helps in the case of heavily-overparametrized model.\n5. Definiton (8) is unclear: firstly, the objective L_theta is parametrized by parameters of discriminator (theta), over which the maximum on the rhs is taken (this follows in Appendices). Secondly, it has non-deterministic term d^lambda(..) which depends on random empirical distributions \\hat(P_r), \\hat(P_g) (in Proposition 8. which follows, authors take expectation over a Wasserstein distance between these distributions).\n6. Remark 2's proof is incorrect. At the beginning of p.13 authors take D_theta(x) = sign(P_r(x) - P_g(x)), which is not a Lipschitz function. Also P_r and P_g may not be discrete so the entire expression does not make sense.\n7. Remark 2 is not true non-deterministic formulation of (8). It is possible that P_g != P_r and \\hat(P_g) = \\hat(P_r); in such case sinkhorn distance is 0 and the remaining terms of rhs of (8) cancell out (e.g. P_r is a Dirac delta at 0 and P_g = Bernoulli(0.1), then it is possible that \\hat(P_r) = \\hat(P_g) = delta(0)). Perhaps re-shaping definition (8) so that it is deterministic would help achieving correctness of Remark 2.\n8. Remark 3. is unclear: in what sense expression (8) can be optimized by gradient descent?\n9. The paper seems rougly written (see below) and the text needs some polishing.\n\nTypos/ requires clarification:\n[p2.] Backgrounds -> background\n[p.3] corresponds a \\lambda -> corresponds to \\lambda\n[p.3] 'The proposed bounded strategy' -> bounding\n[p.3] 'We name it general in two folds' (?)\n[p.7] 'High-resolution' - 128x128 is not high.\n[p.8] Discussions -> Discussion", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a framework named Wasserstein-bounded GANs which generalizes WGAN. The paper shows that WBGAN can improve stability.

The reviewers raised several questions about the method and the experiments, but these were not addressed.

I encourage the authors to revise the draft and resubmit to a different venue.
Decision: Reject