Title: IMPROVING THE IMPROVED TRAINING OF WASSERSTEIN GANS: A CONSISTENCY TERM AND ITS DUAL EFFECT

Abstract: Despite being impactful on a variety of problems and applications, the generative adversarial nets (GANs) are remarkably difficult to train. This issue is formally analyzed by Arjovsky & Bottou (2017), who also propose an alternative direction to avoid the caveats in the minmax two-player training of GANs. The corresponding algorithm, called Wasserstein GAN (WGAN), hinges on the 1-Lipschitz continuity of the discriminator. In this paper, we propose a novel approach to enforcing the Lipschitz continuity in the training procedure of WGANs. Our approach seamlessly connects WGAN with one of the recent semi-supervised learning methods. As a result, it gives rise to not only better photo-realistic samples than the previous methods but also state-of-the-art semi-supervised learning results. In particular, our approach gives rise to the inception score of more than 5.0 with only 1,000 CIFAR-10 images and is the first that exceeds the accuracy of 90% on the CIFAR-10 dataset using only 4,000 labeled images, to the best of our knowledge.

1. INTRODUCTION:

We have witnessed a great surge of interests in deep generative networks in recent years (Kingma & Welling, 2013; Goodfellow et al., 2014; Li et al., 2015) . The central idea therein is to feed a random vector to a (e.g., feedforward) neural network and then take the output as the desired sample. This sampling procedure is very efficient without the need of any Markov chains.

In order to train such a deep generative network, two broad categories of methods are proposed. The first is to use stochastic variational inference (Kingma & Welling, 2013; Rezende et al., 2014; Kingma et al., 2014) to optimize the lower bound of the data likelihood. The other is to use the samples as a proxy to minimize the distribution divergence between the model and the real through a two-player game (Goodfellow et al., 2014; Salimans et al., 2016) , maximum mean discrepancy (Li et al., 2015; Dziugaite et al., 2015; Li et al., 2017b) , f-divergence (Nowozin et al., 2016; Nock et al., 2017) , and the most recent Wasserstein distance (Arjovsky et al., 2017; Gulrajani et al., 2017) .

With no doubt, the generative adversarial networks (GANs) among them (Goodfellow et al., 2014) have the biggest impact thus far on a variety of problems and applications (Radford et al., 2015; Denton et al., 2015; Im et al., 2016; Isola et al., 2016; Springenberg, 2015; Sutskever et al., 2015; Odena, 2016; Zhu et al., 2017) . GANs learn the generative network (generator) by playing a twoplayer game between the generator and an auxiliary discriminator network. While the generator has no difference from other deep generative models in the sense that it translates a random vector into a desired sample, it is impossible to calculate the sample likelihood from it. Instead, the discriminator serves to evaluate the quality of the generated samples by checking how difficult it is to differentiate them from real data points. However, it is remarkably difficult to train GANs without good heuristics (Goodfellow et al., 2014; Salimans et al., 2016; Radford et al., 2015) which may not generalize across different network architectures or application domains. The training dynamics are often unstable and the generated samples could collapse to limited modes. These issues are formally analyzed by Arjovsky & Bottou (2017) , who also propose an alternative direction (Arjovsky et al., 2017) to avoid the caveats in the minmax two-player training of GANs. The corresponding algorithm, namely, Wasserstein GAN (WGAN), shows not only superior performance over GANs but also a nice correlation between the sample quality and the value function that GANs lack.

1.1 BACKGROUND: WGAN AND THE IMPROVED TRAINING OF WGAN WGAN (Arjovsky et al., 2017) aims to learn the generator network G(z), for any random vector z ∼ P z , such that the Wasserstein distance is minimized between the resulting distribution P G of the generated samples {G(z)} and the real distribution P r underlying the observed data points {x}; in other words, min G W (P r , P G ). The Wasserstein distance W (P r , P G ) is shown a more sensible cost function for learning the distributions supported by low-dimensional manifolds than the other popular distribution divergences and distances -for example, the Jensen-Shannon (JS) divergence implicitly employed in GANs (Goodfellow et al., 2014) .

Due to the Kantorovich-Rubinstein duality (Villani, 2008) for calculating the Wasserstein distance, the value function of WGAN is then written as min

EQUATION

where D is the set of 1-Lipschitz functions. Analogous to GANs, we still call D the "discriminator" although it is actually a real-valued function and not a classifier at all. Arjovsky et al. (2017) specify this family of functions D by neural networks and then use weight clipping to enforce the Lipschitz continuity. However, as the authors note, the networks' capacities become limited due to the weight clipping and there could be gradient vanishing problems in the training.

Improved training of WGAN. Gulrajani et al. (2017) give more concrete examples to illustrate the perils of the weight clipping and propose an alternative way of imposing the Lipschitz continuity.

In particular, they introduce a gradient penalty term by noting that the differentiable discriminator D(•) is 1-Lipschitz if and only if the norm of its gradients is at most 1 everywhere,

GP | x := E x ∇ x D( x) 2 -1 2 (2)

where x is uniformly sampled from the straight line between a pair of data points sampled from the model P G and the real P r , respectively. A similar regularization is used by Kodali et al. (2017) .

Potential caveats. Unlike the weight clipping, however, by no means one can penalize everywhere using this term through a finite number of training iterations. As a result, the gradient penalty term GP takes effect only upon the sampled points x, leaving significant parts of the support domain not examined at all. In particular, consider the observed data points and their underlying manifold that supports the real distribution P r . At the beginning of the training stage, the generated sample G(z) and hence x could be distant from the manifold. The Lipschitz continuity over the manifold is not enforced until the generative model P G becomes close enough to the real one P r , if it can.

1.2. OUR APPROACH AND CONTRIBUTIONS:

In light of the above pros and cons, we propose to improve the improved training of WGAN by additionally laying the Lipschitz continuity condition over the manifold of the real data x ∼ P r . Moreover, instead of focusing on one particular data point at a time, we devise a regularization over a pair of data points drawn near the manifold following the most basic definition of the 1-Lipschitz continuity. In particular, we perturb each real data point x twice and use a Lipschitz constant to bound the difference between the discriminator's responses to the perturbed data points x , x .

Figure 1 illustrates our main idea. The gradient penalty GP | x often fails to check the continuity of region near the real data x, around which the discriminator function can freely violate the 1-Lipschitz continuity. We alleviate this issue by explicitly checking the continuity condition using the two perturbed version x , x near any observed real data point x.

( ) G z x D x x' x'' [ ]

Figure 1 : Illustration of our main idea. In addition to the gradient penalty over x, we also examine x and x around the real data point x in each iteration.

In this paper, we make the following contributions.

(1) We propose an alternative mechanism for enforcing the Lipschitz continuity over the family of discriminators by resorting to the basic definition of the Lipschitz continuity. It effectively improves the gradient penalty method (Gulrajani et al., 2017) and gives rise to generators with more photo-realistic samples and higher inception scores (Salimans et al., 2016) . (2) Our approach is very data efficient in terms of being less prone to overfitting even for very small training sets. We do not observe obvious overfitting phenomena even when the model is trained on only 1000 images of CIFAR-10 ( Krizhevsky & Hinton, 2009) .

(3) Our approach can be seamlessly integrated with GANs to be a competitive semi-supervised training technique (Chapelle et al., 2009) thanks to that both inject noise to the real data points.

As results, we are able to report the state-of-the-art results on the generative model with a inception score of 8.81 ± 0.13 on CIFAR-10, and the semi-supervised learning results of 9.98 ± 0.21 on CIFAR-10 using only 4,000 labeled images -especially, they are significantly better than all the existing GAN-based semi-supervised learning results, to the best of our knowledge.

2. APPROACH:

We firstly review the definition of the Lipschitz continuity and then discuss how to use it to regularize the training of WGAN. We then arrive at an approach that can be seamlessly integrated with the semi-supervised learning method (Laine & Aila, 2016) . By bringing the best of the two worlds, we report better semi-supervised learning results than both (Laine & Aila, 2016) and existing GANbased methods.

2.1. IMPROVING THE IMPROVED TRAINING OF WGAN:

Let d denote the 2 metric on an input space used in this paper. A discriminator D : X → Y is Lipschitz continuous if there exists a real constant M 0 such that, for all x 1 , x 2 ∈ X ,

EQUATION

Immediately, we can add the following soft consistency term (CT ) to the value function of WGAN in order to penalize the violations to the inequality in eq. ( 3),

EQUATION

Remarks.

Here we face the same snag as in (Gulrajani et al., 2017) , i.e., it is impractical to substitute all the possibilities of (x 1 , x 2 ) pairs into eq. ( 4). What pairs and which regions of the input set X should we check for eq. ( 4)? Arguably, it is fairly safe to limit our scope to the manifold that supports the real data distribution P r and its surrounding regions mainly for two reasons. First, we keep the gradient penalty term and improve it by the proposed consistency term in our overall approach. While the former enforces the continuity over the points sampled between the real and generated points, the latter complement the former by focusing on the region around the real data manifold instead. Second, the distribution of the generative model P G is virtually desired to be as close as possible to P r . We use the notation M in eq. ( 3) and a different M in eq. ( 4) to reflect the fact that the continuity will be checked only sparsely at finite data points in practice.

Perturbing the real data. To this end, the very first version we tried was to directly add Gaussian noise δ to each real data point, resulting in a pair of x + δ 1 , x + δ 2 , where x ∼ P r . However, as noted by Arjovsky et al. (2017) and Wu et al. (2016) , we found that the samples from the generator become blurry due to the Gaussian noise used in the training. We have also tested the dropout noise that is applied to the input and found that the resulting MNIST samples are cut off here and there. for N of iterations do 3:

for i = 1, ..., b do 4:

Sample real data x ∼ P r , random vector z ∼ P z , and a random number ∼ U [0, 1].

5::

x ← x + (1 -)G(z)

6:

L (i) ← D(G(z)) -D(x) + λ 1 GP | x + λ 2 CT | x ,x 7:

end for 8:

θ d ← Adam(∇ θ d 1 b b i=1 L (i) , θ d , γ) 9:

end for 10:

Sample b random vectors z (i) 11:

θ g ← Adam(∇ θg 1 b b i=1 -D(G(z (i) )), θ g , γ) 12: end for

The success comes after we perturb the hidden layers of the discriminator using dropout, as opposed to the input x. When the dropout rate is small, the perturbed discriminator's output can be considered as the output of the clean discriminator in response to a "virtual" data point x that is not far from x. Thus, we denote by D(x ) the discriminator output after applying dropout to its hidden layers. In the same manner, we find the second virtual point x around x by applying the (stochastic) dropout again to the hidden layers of the discriminator, and denote by D(x ) the corresponding output.

Note that, however, it becomes impossible to compute the distance d(x , x ) between the two virtual data points. In this work, we assume it is bounded by a constant and absorb the constant to M . Accordingly, we tune M in our experiments to take account of this unknown constant; the best results are obtained between M = 0 and M = 0.2. For consistency, we use M = 0 to report all the results in this paper.

A consistency regularization. Our final consistency regularization takes the following form,

CT | x ,x = E x∼Pr [max (0, d (D(x ), D(x )) + 0.1 • d (D -(x ), D -(x )) -M )]

(5)

where D(x ) is the output of the discriminator given the input x and after we apply dropout to the hidden layers of the discriminator. We envision this is equivalent to passing a "virtual" data point x through the clean discriminator. We find that it slightly improves the performance by further controlling the second-to-last layer D -(•) of the discriminator, i.e., the d (D -(x ), D -(x )) above.

This new consistent regularization CT | x ,x enforces the Lipschitz continuity over the data manifold and its surrounding regions, effectively complementing and improving the gradient penalty GP | x used in the improved training of WGAN. Putting them together, our new objective function for updating the weigts of the discriminator is

EQUATION

Algorithm 1 shows the complete algorithm for learning a WGAN in this paper. For the hyperparameters, we borrow λ 1 = 10 from Gulrajani et al. (2017) and use λ 2 = 2 for all our experiments no matter on which dataset. Another hyper-parameter is M in eq. ( 5). As stated previously, M taking values between 0 and 0.2 gives rise to about the same results in our experiments.

2.2. A SEAMLESS CONNECTION WITH A SEMI-SUPERVISED LEARNING METHOD:

In this section, we extend the WGAN to a semi-supervised learning approach by drawing insights from two related works.

• Following (Salimans et al., 2016) , we modify the output layer of the discriminator such that it has K + 1 output neurons, where K is the number of classes of interest and the (K + 1)-th neuron is reserved for contrasting the generated samples with the real data using the Wasserstein distance in the WGAN. We use a (K + 1)-way softmax as the activation function of the last layer. Figure 2 shows the framework for training the discriminator, with the objective function below,

EQUATION

where the first three terms are the same as in (Salimans et al., 2016) , while the last consistency regularization is calculated after we apply dropout to the discriminator. The last term essentially leads to a temporal self-ensembling scheme to benefit the semi-supervised learning. Please see (Laine & Aila, 2016) for more insightful discussions about it.

The generator loss matches the expected features of the generated sample and the real data points,

EQUATION

)

3. EXPERIMENTAL RESULTS:

We conduct experiments on the prevalent MNIST (LeCun et al., 1998) , and CIFAR-10 ( Krizhevsky & Hinton, 2009) datasets. The code is available on https://github.com/biuyq/CT-GAN to facilitate the reproducibility of our results.

3.1. MNIST:

The MNIST dataset provides 70,000 handwritten digits in total and 10,000 of them are often left out for the testing purpose. Following (Gulrajani et al., 2017) , we use only 1,000 of them to train the WGAN for a fair comparison with it. We use all the 60,000 training examples in the semi-supervised learning experiments and reveal the labels of 100 of them (10 per class). This setup is the same as in (Rasmus et al., 2015) . No data augmentation is used. Please see Appendix A for the network architectures of the generator and discriminator, respectively.

Qualitative results. Figure 3 shows the generated samples with improved training of WGAN by the gradient penalty (GP-WGAN) and ours with the consistency regularization (CT-GAN), respectively, after 50,000 generator iterations. It is clear that our approach gives rise to more realistic samples than GP-WGAN. The contrasts of our samples between the foreground and the background are in general sharper than those of GP-WGAN.

Overfitting. We find that our approach is less prone to overfitting. To demonstrate this point, we show the convergence curves of the discriminator's value functions by GP-WGAN and our CT-GAN in Figure 4 . The red curves are evaluated on the training set and the blue ones are on the test set. We can see that the results on the test set become saturated pretty early in GP-WGAN, while ours can consistently decrease the costs on both the training and the test sets. This observation also holds for the CIFAR-10 dataset (cf. Appendix E). Semi-supervised learning results. We compare our semi-supervised learning results with those of several competitive methods in Table 1 . Our approach is among the best on this MNIST dataset.

3.2. CIFAR-10:

CIFAR-10 (Krizhevsky & Hinton, 2009) contains 50,000 natural images of size 32×32. We use it to test two networks for the generative model: a small CNN and a ResNet (cf. Appendix A for the network structures). For the former we use only 1,000 images to train the model and we use the whole training set to learn the ResNet.

Qualitative results. Figure 5 contrasts the samples we generated to those by GP-WGAN when the generator is the small CNN. Figure 7 shows the results by a larger-scale ResNet. Our results are more photo-realistic.

Additionally, we also draw the histograms of the discriminator's weights in Figure 6 after we train it using GP-WGAN and CT-GAN, respectively. It is interesting to see that ours controls the weights within a smaller and more symmetric range [-0.67, 0.96] than the [-2.00, 10.12] by GP-WGAN, partially explaining why our approach is less prone to overfitting.

Table 1 : Comparing our semi-supervised learning approach with state-of-the-art ones on MNIST.

Method:

Test error (%) Ladder (Rasmus et al., 2015) 1.06±0.37 VAT (Miyato et al., 2017) 1.36 CatGAN (Springenberg, 2015) 1.39±0.28 Improved GAN (Salimans et al., 2016) 0.93 ± 0.065 Triple GAN (Li et al., 2017a) 0.91 ± 0.58 Our CT-GAN 0.89 ± 0.13 

Supervised IS Unsupervised IS Accuracy(%) SteinGANs (Wang & Liu, 2016) 6.35 --DCGANs (Radford et al., 2015) 6.58 6.16 ± 0.07 -Improved GANs (Salimans et al., 2016) 8.09 ± 0.07 --AC-GANs (Odena et al., 2016) 8.25 ± 0.07 --GP-WGAN (Gulrajani et al., 2017) 8.42 ± 0.10 7.86 ± 0.07 91.85 SGANs (Huang et al., 2016) 8.59 ± 0.12 --ALI (Warde-Farley & Bengio, 2016) -5.34 ± 0.05 -BEGAN (Berthelot et al., 2017) -5.62 -EGAN-Ent-VI (Dai et al., 2017) -7.07 ± 0.10 -DFM (Warde-Farley & Bengio, 2016) -7.72 ± 0.07 -Our CT-GAN 8.81±0.13 8.12±0.12 95.91

Comparison of the inception scores. Finally, we compare our approach with GP-WGAN on the whole training set for both unsupervised and supervised generative-purpose task using ResNet. For model selection, we use the first 50,000 samples to compute the inception scores (Salimans et al., 2016) , then choose the best model, and finally report the "test" score on another 50,000 samples. The experiment follows the previous setup in (Odena et al., 2016) . From the comparison results in Tables 2, we conclude that our proposed model achieves the highest inception score on the CIFAR-10 dataset, to the best of our knowledge. Some generated samples are shown in Figure 7 .

For the small CNN based generator, the inception scores of GP-WGAN and our CT-GAN are 2.98 ± 0.11 and 5.13 ± 0.12, respectively. Table 3 : Comparing our semi-supervised learning approach with state-of-the-art ones on CIFAR-10

Test error (%) Ladder (Rasmus et al., 2015) 20.40 ± 0.47 VAT (Miyato et al., 2017) 10.55 TE (Laine & Aila, 2016) 12.16 ± 0.24 Teacher-Student (Tarvainen & Valpola, 2017) 12.31 ± 0.28 CatGANs (Springenberg, 2015) 19.58 ± 0.58 Improved GANs (Salimans et al., 2016) 18.63 ± 2.32 ALI (Dumoulin et al., 2016) 17.99 ± 1.62 CLS-GAN (Qi, 2017) 17.30 ± 0.50 Triple GAN (Li et al., 2017a) 16.99 ± 0.36 Improved semi-GAN (Kumar et al., 2017) 16.78 ± 1.80 Our CT-GAN 9.98 ± 0.21

Semi-supervised learning. For the semi-supervised learning approach, we follow the standard training/test split of the dataset but use only 4,000 labels in the training. A regular data augmentation with flipping the images horizontally and randomly translating the images within [-2,2] pixels is used in our paper (No ZCA whitening). We report the semi-supervised learning results in Table 3 . The mean and standard errors are obtained by running the experiments 5 rounds. Comparing to several very competitive methods, ours is able to achieve state-of-the-art results. Notably, our CT-GAN outperfroms all the GAN based methods by a large margin. Please see Appendix A for the network architectures and Appendix C for the ablation study of our algorithm.

4. CONCLUSION:

In this paper, we present a consistency term derived from Lipschitz inequality, which boosts the performance of GANs model. The proposed term has been demonstrated to be an efficient manner to ease the over-fitting problem when data amount is limited. Experiments show that our model obtains the state-of-the-art accuracy and Inception score on CIFAR-10 dataset for both the semisupervised learning task and the learning of generative models. MNIST, the number of training epochs is set to 300 with a constant learning rate of 0.003. The other hyper-parameters are exactly the same as in the improved GAN (Salimans et al., 2016) .

For the experiments on the generative models, to have a fair comparison for our results with the existing ones, we keep the network structure and hyper-parameters the same as in the improved training of WGAN (Gulrajani et al., 2017 ) except that we add three dropout layers to some of the hidden layers as shown in tables 6 to 8.

If we remove the CT term, the test error goes up to 14.98, signifying the effectiveness of the CT regularization.

If we remove GAN from our approach, it almost reduces to TE; in fact, all the settings here are the same as TE except that we use an extra regularization (D (., .) in CT) over the second-to-last layer. We can see that the error is still significantly larger than our overall method.

We use the weight normalization as in Salimans et al. (2016) , which becomes a core constituent of our approach. The batch normalization would actually invalidate the feature matching in Salimans et al. (2016) .

Finally, we also test the version without the regularization to the second-to-last layer and observe a little drop in the performance. In our experiments, we find that although the GP-WGAN (Gulrajani et al., 2017) has applied a Lipschitz constraint in the form of the gradient penalty over the input sampled between a real data point and a generated one, the actual effect on the 2 norm of the gradient is not as good as our CT-GAN model in the real data points, as illustrated in Figure 1 . We empirically verify this fact by Figure 8 , which shows the 2 norms of the gradients of the discriminator with respect to the real data points. The closer to 1 the norms are, the better the 1-Lipschitz continuity is preserved. Figure 8 further demonstrates that our consistency (CT) regularization is able to improve GP-WGAN (Gulrajani et al., 2017) . Definition of the Lipschitz continuity. Additionally, we check how much the 1-Lipschitz continuity is satisfied according to its vanilla definition (cf. Eq. ( 3)). Figures 9 and 10 plot the CTs of Eq. ( 4) and Eq. ( 5), respectively, over different iterations of the training using 1000 CIFAR-10 images. Figure 10 is the actual CTs used to train the generative model. Figure 9 is drawn as follows.

For every 100 iterations, we randomly pick up 64 real examples and split them into two subsets of the same size. We compute d(D(x 1 ) -D(x 2 ))/d(x 1 -x 2 ) for all the (x 1 , x 2 ) pairs, where x 1 is from the first subset and x 2 is from the second. The maximum of d(D(x 1 ) -D(x 2 ))/d(x 1 -x 2 ) is plotted in Figure 9 . We can see that the CT-GAN curve converges under a certain value much faster than GP-WGAN. The 1-Lipschitz continuity is better maintained by CT-GAN than GP-WGAN over the whole course of the training procedure on the manifold of the real data. 4)) over different iterations of the training using 1000 CIFAR-10 images. In each iteration, we randomly pick up two real data points to compute the CT. 

APPENDIX E GP-WGAN WITH DROPOUT:

We run another experiment of GP-WGAN by adding the same dropout layers used in our method to the networks of GP-WGAN, denoted by GP-WGAN+dropout. This may help understand the contribution of our approach in preventing the overfitting problem -is it due to the CT regularization, or merely due to the dropout? This experiment is done by removing the CT term out of our value function for training WGAN using 1,000 CIFAR-10 images and yet keep the dropout layers.

The Inception score of GP-WGAN+dropout is 4.29 ± 0.12, and the generated samples are shown in Figure 11(a) . We also plot the curve of CT in the training procedure of GP-WGAN+dropout and contrast it to the curve of our . It is clear that GP-WGAN+dropout outperforms GP-WGAN (2.98 ± 0.11), and our method (5.13 ± 0.12) outperforms GP-WGAN with a large margin. Finally, we plot the convergence curves of the discriminators' negative cost function learned by GP-WGAN, GP-WGAN+Dropout, and our CT-GAN in Figure 12 . We can see that dropout is able to reduce the overfitting of GP-WGAN, but it is not as effective as our CT-GAN. 

APPENDIX F EXPERIMENTS ON LARGE DATASET:

In this section, we further present experimental results on the large-scale ImageNet (Deng et al., 2009) and LSUN bedroom (Yu et al., 2015) datasets. The experiment setup (e.g., network architecture, learning rates, etc.) is exactly the same as in the GP-WGAN work (Gulrajani et al., 2017) . We refer readers to (Gulrajani et al., 2017) or our code on GitHub (https://github.com/biuyq/CT-GAN) for the details of the setup.

Our experiment on ImageNet is conducted using images of the size 64×64. After 200, 000 generator iterations, the inception score of the proposed CT-GAN is 10.27±0.15, whereas GP-WGAN's is 9.85±0.17. In addition, the inception score comparison of GP-WGAN and our CT-GAN in each generator iteration is shown in Figure 13 . We can observe that the inception score of our proposed CT-GAN becomes higher than GP-WGAN's after early generator iterations. Finally, Figure 14 shows the samples generated by GP-WGAN and CT-GAN, respectively. 

:



Figures and Tables:

our proposed CT-GAN for training a generative neural net. Most of our experiments are conducted with the default values λ 1 = 10,λ 2 = 2, N = 5, γ = 0.0002, b = 64, iter = 10 6 Require: b, the batch size. λ 1 ,λ 2 , weights. γ, the Learning rate. iter, number of iterations.1: for iter of training iterations do 2:

Figure 2: Framework for the semi-supervised training. For clarity, we have omitted the generator.

Figure 3: Images generated by (a) GP-WGAN (Gulrajani et al., 2017) and (b) Our CT-GAN, respectively.

Figure 5: Generated samples by (a) GP-WGAN and (b) our CT-GAN. Here the generator is a small CNN. See Figure 7 for the samples by a ResNet.

Figure 7: Generated samples by our ResNet model: (a) Generated samples by unsupervised model and (b) Generated samples by supervised model. Each column corresponds to one class in the CIFAR-10 dataset.

32*32 Image x, Input: Noise z 128 5*5 conv. 128, Pad = same, Stride = 2, lReLU MLP 8192, ReLU, Batch norm 0.5 Dropout Reshape 512*4*4 5*5 conv. 256, Pad = same, Stride = 2, lReLU 5*5 deconv. 256*8*8 0.5 Dropout ReLU, Bach norm 5*5 conv. 512, Pad = same Stride = 2, lReLU 5*5 deconv. 128*16

Figure 8: The maximum 2 norm of the gradients of the discriminator with respect to the input on CIFAR-10 testing set in each iteration of the training using 1000 CIFAR-10 training images.

Figure 9: CT (cf. Eq. (4)) over different iterations of the training using 1000 CIFAR-10 images. In each iteration, we randomly pick up two real data points to compute the CT.

Figure 10: CT (cf. Eq. (5)) over different iterations of the training using 1000 CIFAR-10 images.

Figure 11: The results of GP-WGAN+dropout.

Figure 12: Convergence curves of the discriminator cost: (a) GP-WGAN, (b) GP-WGAN+Dropout, and (c) CT-GAN (ours).

Figure 13: Inception score of GP-WGAN and our CT-GAN with respect to iterations

Inception score and accuracy of different models on CIFAR-10

Table content: <table/>

Ablation study of our semi-supervised learning method

Table content: <table><tr><td>Method</td><td>Test Error</td></tr><tr><td>w/o CT</td><td>14.98±0.43</td></tr><tr><td>w/o GAN</td><td>11.98±0.32</td></tr><tr><td>w batch norm</td><td>-</td></tr><tr><td colspan="2">w/o D (., .) over the second-to-last layer 10.70±0.24</td></tr><tr><td>Ours</td><td>9.98±0.21</td></tr><tr><td colspan="2">APPENDIX D EXAMINING THE 1-LIPSCHITZ CONTINUITY</td></tr><tr><td>Norm of gradient.</td><td/></tr></table>

Peer review (iclr2018):
 Summary: The paper proposes a new regularizer for wgans, to be combined with the traditional gradient penalty. The theoretical motivation is bleak, and the analysis contains some important mistakes. The results are very good, as noticed by the comments, the fact that the method is also less susceptible to overfitting is also an important result, though this might be purely due to dropout. One of the main problems is that the largest dataset used is CIFAR, which is small. Experiments on something like bedrooms or imagenet would make the paper much stronger. If the authors fix the theoretical analysis and add evidence in a larger dataset I will raise the score. Detailed comments: - The motivation of 1.2 and the sentence "Arguably, it is fairly safe to limit our scope to the manifold that supports the real data distribution P_r and its surrounding regions" are incredibly wrong. First of all, it should be noted that the duality uses 1-Lip in the entire space between Pr and Pg, not in Pr alone. If the manifolds are not extremely close (such as in the beginning of training), then the discriminator can be almost exactly 1 in the real data, and 0 on the fake. Thus the discriminator would be almost exactly constant (0-Lip) near the real manifold, but will fail to be 1-lip in the decision boundary, this is where interpolations fix this issue. See Figure 2 of the wgan paper for example, in this simple example an almost perfect discriminator would have almost 0 penalty. - In the *Potential caveats* section, the implication that 1-Lip may not be enforced in non-examined samples is checkable by an easy experiment, which is to look for samples that have gradients of the critic wrt the input with norm > 1. I performed the exp in figure 8 and saw that by taking a slightly higher lambda, one reaches gradients that are as close to 1 as with ct-gan. Since ct-gan uses an extra regularizer, I think the authors need some stronger evidence to support the claim that ct-gan better battles this *potential caveat*. - It*s important to realize that the CT regularizer with M* = 1 (1-Lip constraint) will only be positive for an almost 1-Lip function if x and x* are sampled when x - x* has a very similar direction than the gradient at x. This is very hard in high dimensional spaces, and when I implemented a CT regularizer indeed the ration of eq (4) was quite less than the norm of the gradient. It would be useful to plot the value of the CT regularizer (the eq 4 version) as the training iterations progresses. Thus the CT regularizer works as an overall Lipschitz penalty, as opposed to penalizing having more than 1 for the Lipschitz constant. This difference is non-trivial and should be discussed. - Line 11 of the algorithm is missing L^(i) inside the sum. - One shouldn*t use MNIST for anything else than deliberately testing an overfitting problem. Figure 4 is thus relevant, but the semi-supervised results of MNIST or the sample quality experiments give hardly any evidence to support the method. - The overfitting result is very important, but one should disambiguate this from being due to dropout. Comparing with wgangp + dropout is thus important in this experiment. - The authors should provide experiments in at least one larger dataset like bedrooms or imagenet (not faces, which is known to be very easy). This would strengthen the paper quite a bit.