ChainGAN: A sequential approach to GANs

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) are a class of generative models that have shown promising results in multiple domains , including image generation , text generation , and style-transfer ( Goodfellow et al. , 2014 ; Zhang et al. , 2017 ; Zhu et al. , 2017 ) . GANs use a game-theoretic framework pitting two agents – a discriminator and a generator – against one another to learn a data distribution Pd . The generator attempts to transform a noise vector z ∼ N into a sample x̃ that resembles the training data . Meanwhile , the discriminator acts as a classifier trying to distinguish between real samples , x ∼ Pd , and generated ones , x̃ ∼ Pg . Equation 1 is the original minimax objective proposed by Goodfellow et al . ( 2014 ) , which can also be viewed as minimizing the Jensen-Shannon divergence between two distributions , Pd and Pg . These agents are most often implemented as neural networks and Goodfellow et al . ( 2014 ) proved that , under suitable conditions , the generator distribution Pg will match the data distribution , Pd . min G max D E x∼Pd [ log ( D ( x ) ] + E z∼N [ 1− log ( D ( G ( z ) ) ] ( 1 ) Despite their success , GANs are notoriously hard to train and many variants have been proposed to tackle shortcomings . These shortcomings include mode collapse ( i.e. , the tendency for the generator to produce samples lying on a few modes instead of over the whole data space ) , training instability , and convergence properties . In most of these modified GANs , however , the generator is realized as a single network that transforms noise into a generated data sample . In this paper , we propose a sequential generator architecture composed of a number of networks . In the first step , a traditional generator ( in this paper called the base generator ) is used to transform a noise sample z into a generated sample x̃0 . The sample is then fed into a network ( called editor ) which attempts to enhance its quality , as judged by the discriminator . We have a number of these editor networks connected in a chain , such that the output of one is used as the input of the next . Together , the base generator and the editors ahead of it are called the chain generator . Any preexisting generator architecture can be used for the base generator , making it quite flexible . For the editors , architectures designed for sample enhancement work well . The whole chain generator network can be quite small ; in fact , it is smaller than most existing generator networks , yet produces equivalent or better results ( Section 4.2 ) . Each network in the chain generator is trained independently based on scores they receive from the discriminator . This is done instead of an end-to-end backpropagation through the whole base generator + editor chain ( Section 3.3 ) . This allows each network in the chain to get direct feedback on its output . A similar approach has been quite successful in classification tasks . Huang et al . ( 2017 ) showed that in a classification architecture composed of a chain of ResNet blocks , training each of these blocks independently , produces better results than an end-to-end training scheme ( He et al. , 2015 ) . In this paper , we propose a sequential generator architecture and a corresponding training regime for GANs . The benefits of this approach include : • Better performance on some datasets when compared with relevant models ( Table 1 ) • Efficient in memory and in the number of parameters ( Section 7.1 ) . • Base generator itself performs better when trained alongside editors ( Figure 4a ) . • It is flexible and can be adapted to existing network architectures 2 BACKGROUND AND RELEVANT WORK . 2.1 GAN OBJECTIVES . The formulation by Goodfellow et al . ( 2014 ) ( Equation 1 ) is equivalent to minimizing the JensenShannon divergence between the data distribution Pd and the generated distribution Pg . However , Arjovsky et al . ( 2017 ) showed that , in practice , this leads to a loss that is potentially not continuous , causing training difficulty . To address this , they presented the Wasserstein GAN ( WGAN ) which minimized the Wasserstein ( or ‘ Earth mover ’ ) distance between two distributions . This is a weaker distance than Jensen-Shannon and Arjovsky et al . ( 2017 ) showed that , under mild conditions , it resulted in a function that was continuous everywhere and differentiable almost everywhere . The WGAN objective can be expressed as : min G max D∈D E x∼Pd [ D ( x ) ] − E x̃∼Pg [ D ( x̃ ) ] ( 2 ) where D is the set of all 1-Lipchitz functions . D no longer discerns between real and fake samples but is used to approximate the Wasserstein distance between the two distributions ( it is here called a ‘ critic ’ ) . G in turn looks to minimize this approximated distance . Several approaches have been proposed to enforce the Lipchitz constraint on the critic , the most notable being gradient penalty ( Gulrajani et al. , 2017 ) . This method penalizes the norm of the critic ’ s gradient with respect to its input and was shown to have stable training behavior on various architectures . Due to its strong theoretical and empirical results , we use a modified version of this objective for our work . The objective for WGAN with gradient penalty can be expressed as : L = E x∼Pd [ D ( x ) ] − E x̃∼Pg [ D ( x̃ ) ] + λ E x̂∼Px̂ [ ( ||∇x̂D ( x̂ ) ||2 − 1 ) 2 ] , ( 3 ) where Px̂ samples uniformly along straight lines between pair of points sampled from the data distribution Pd and the generator distribution Pg Gulrajani et al . ( 2017 ) . λ is the gradient penalty coefficient . 2.2 GAN ARCHITECTURES . There have been a diverse range of architectures proposed for the generator and discriminator networks . DCGAN has shown promising results by using a deep convolutional architecture for both the generator and discriminator ( Radford et al. , 2015 ) . However , this uses a single large generator network instead of the sequential architecture that we have developed . Our architecture provides similar results as DCGAN while being more efficient , both in network parameters ( Section 4.2 ) and in memory . EnhanceGAN , proposed by Deng et al . ( 2017 ) , successfully use GANs for unsupervised image enhancement . In this model , the generator is given an image , and it tries to improve its aesthetic quality . The editor part of our network is similar to this as they are responsible for enhancing images in an unsupervised fashion . Sequential generator architectures have been proposed for tasks where the underlying data has some inherent sequential structure . TextGAN uses an LSTM architecture for the generator to generate a sentence , one word at a time ( Zhang et al. , 2017 ) . Ghosh et al . ( 2016 ) used an RNN-style discriminator and generator to solve abstract reasoning problems . This attempts to predict the next logical diagram , given a sequence of past diagrams . C-RNN-GAN is another example that uses GANs on continuous sequential data like music ( Mogren , 2016 ) . Unlike these works , we do not assume the data is inherently sequential nor do we produce a single sample by generating a number of smaller units ; rather , the output of any network in the chain generator is a whole sample , and the succeeding ones try to enhance it . Moreover , each editor network has its own set of parameters , unlike LSTMs or RNNs where these are shared . This allows each editor to be independent and makes our model efficient as we do not have to backpropagate across time . The concept of using multiple generators has been proposed previously . Hoang et al . ( 2017 ) used multiple generators which share all but the last layer , trained alongside a single discriminator and a classifier . Their work has shown resistance to mode collapse and achieves state-of-the-art results on some datasets . Their work was influenced by MIX+GAN , which showed strong theoretical and empirical results by using a mixture of GANs instead of a single generator and discriminator ( Arora et al. , 2017 ) . Notably , they showed that using a mixture of GANs guarantees existence of approximate equilibrium , leading to stable training . However , a drawback of their work was the large memory cost of using multiple GANs - they suggest a mixture size of 5 . Our work is similar to these in that we use multiple generators . The output of each editor is a different transformation on the noise vector z and can be viewed as originating from different generators . However , the generators in our model are not independent ; rather , they are connected in a chain and meant to build upon the previous ones . 3 MODEL . 3.1 MOTIVATION . A clear advantage of a sequential approach is that each output in the sequence is conditioned on the previous one , simplifying the problem of generating a complicated sample by mapping them to a sequence of simpler problems . A sequential approach also means that loss or other latent information in the intermediate steps can be used to guide the training process , which is not possible in one-shot generation . As such , we propose a sequential generator architecture and an associated training objective for GANs . We observe that the challenge of generating a data sample from noise can be divided into two components : generating a crude representation of what that sample might be , and then iteratively refining it ( Figure 1 ) . A generator trained on images may first generate a rough picture of a car or a boat , e.g. , but it can then be successively edited and enhanced . Intuitively , this is akin to the editorial process wherein a writer produces a rough sketch of an article , which then passes through a number of editors before the finished copy is published . 3.2 ARCHITECTURE . We use a WGAN with gradient penalty as the starting point for our architecture due to its theoretical robustness and training stability ( Section 2.1 ) . However , instead of using a single generator , we use the chain generator consisting of a base generator , denoted by Gb and several editors , denoted by Ei . The base generator takes in z ∼ N , and produces a sample x̃0 attempting to fool the critic , as in the traditional GAN architecture . The editors are connected in a chain where the ith one takes as input a sample x̃ ( i−1 ) and tries to produce an enhanced version as the output , x̃i ( Figure 1 ) . Thus , the output of each editor is conditioned on the previous one . Our network can be expressed recursively as : x̃0 = Gb ( z ) x̃i = Ei ( x̃ ( i−1 ) ) ( 4 ) Each of the intermediate samples in the chain , x̃i , is an attempt to fool the critic . In the WGAN formulation , the critic is used to approximate the Wasserstein distance between the data distribution Pd and the generated distribution Pg , where Pg is obtained by transforming z ∼ N using the network G. In our formulation , each output x̃i is obtained by the following transformation on z : Ei ( Ei−1 ( . . . E1 ( Gb ( z ) ) ) ) ) . As such , xi can be viewed as being sampled from distribution Pgi and a critic is needed to approximate the Wasserstein distance between Pd and Pgi ( see Section 7 for a detailed explanation ) . For a chain of n editors plus the base generator , n+1 critics are needed . For the ith editor and critic , the objective can be expressed as : Li = E x∼Pd [ Di ( x ) ] − E x̃i∼Pgi [ Di ( x̃i ) ] + λ E x̂∼Px̂i [ ( ||∇x̂iDi ( x̂ ) ||2 − 1 ) 2 ] ( 5 )

Summary: The paper proposes a GAN variant, called ChainGAN, which expresses the generator as a "base generator" -- which maps the noise vector to a rough model sample -- followed by a sequence of "editors" -- which progressively refine the sample. Each component of the generator is trained independently to fool its own separate discriminator, without backpropagating through the entire chain of editors. The proposed ChainGAN model is trained on MNIST, CIFAR10, and CelebA. The paper presents model samples for all three datasets, as well as Inception scores for CIFAR10.