Title: 3C-GAN: AN CONDITION-CONTEXT-COMPOSITE GENERATIVE ADVERSARIAL NETWORKS FOR GEN-ERATING IMAGES SEPARATELY

Abstract: We present 3C-GAN: a novel multiple generators structures, that contains one conditional generator that generates a semantic part of an image conditional on its input label, and one context generator generates the rest of an image. Compared to original GAN model, this model has multiple generators and gives control over what its generators should generate. Unlike previous multi-generator models use a subsequent generation process, that one layer is generated given the previous layer, our model uses a process of generating different part of the images together. This way the model contains fewer parameters and the generation speed is faster. Specifically, the model leverages the label information to separate the object from the image correctly. Since the model conditional on the label information does not restrict to generate other parts of an image, we proposed a cost function that encourages the model to generate only the succinct part of an image in terms of label discrimination. We also found an exclusive prior on the mask of the model help separate the object. The experiments on MNIST, SVHN, and CelebA datasets show 3C-GAN can generate different objects with different generators simultaneously, according to the labels given to each generator.

1. INTRODUCTION:

Recently Generative adversarial network (GAN) is gaining much attention because it has shown promising results for generating natural images (Goodfellow et al. (2014) ; Goodfellow (2016) ). Still, it is limited in generating globally coherent, or high-resolution images. Therefore, to have GAN generate better quality images is the main research topic. There have been many works tackling this topic, including using novel network structures, (Radford et al. (2015) ), Denton et al. (2015) ), using novel objective functions for training ( Zhao et al. (2016) , Arjovsky et al. (2017) , Gulrajani et al. (2017) ), using multi-stage network generation (Im et al. (2016) , Kwak & Zhang (2016) , Yang et al. (2017) ), applying 3D structure information for image generation ( Wang & Gupta (2016) ).

Another main research avenue of GAN is to make a generator more controllable, that is, the generator can generate the image given the criterion we want. Conditional GAN serves this purpose by ensuring the generation is conditional on the given criterion (Odena et al. (2016) , Mansimov et al. (2015) , Reed et al. (2016a) , Kaneko et al. (2017) , Zhu et al. (2016) ). Specifically, in these conditional models, the generator not only accepts the noise code, but also a conditional code, either from a one-hot representation of class labels, or a sentence embedding from a text, or an image, such that this given side information encodes the criterion, and the generated images are conditional on that. Aside from the conditional model, that is a supervised approach, there are models employed an unsupervised approach. InfoGAN (Chen et al. (2016) ) does not need any labels: it discovered the high-level meaning of the image purely from data, and assign the meanings to its noise code. For example, for MNIST dataset, InfoGAN model learn to have one code control the width of the generated digits, and one code controls the rotation of the digits. As an unsupervised approach, InfoGAN discovered the criteria that we might think is useful for the dataset.

Both the supervised and unsupervised models are one-generator approach, there is another multigenerators approach that also aims to make the model more controllable (Kwak & Zhang (2016 ) Yang et al. (2017) ). They generate images part-by-part, such that each part has its semantic mean-ing, and combing all parts results in a realistic image. Since a part of an image has some persistent characteristics, these model learn which parts of the data should be generated together in an unsupervised way. In addition, the model of Yang et al. (2017) includes a transformer network that learns an affine transformation for the foreground in an image. Since each part of an image is generated by different generators, these model provide a high-level meaning of the generation for each generator and possess a finer control over original GAN.

We develop a model that also aims to improve the controllability over original GAN. Specifically, our model is a multi-generators approach, where one generator is a "context" generator that does not take any conditional labels as input, and another is a conditional generator that takes its label information. The conditional generator generates the part of the image that is related to its labels, whereas the context generator generates the other part of the image. Both generators learn where to generate (a mask), and what to generate (an appearance). The context generator shared its input codes with the conditional generator so it "knows" what context it is on when generating its own part. However, there is no restriction for the conditional generator to not generate the context. We propose a cost function on the label discrimination to penalize the conditional generator to do so. In addition, we proposed an exclusive prior on the mask such that any pixels in an image should be generated from one generator only. All in all, our contributions can be listed as follows.

2. METHOD:

The algorithm we proposed is based on Wasserstein GAN, conditional GAN, and model an image with a layered structure, we introduce them in the background section.

2.1. BACKGROUND:



2.1.1. WASSERSTEIN GAN:

A generative adversarial network (GAN) consists of two neural networks trained simultaneously and in opposition to one another. Assume a real sample x follows an unknown distribution p data . The generator network G takes as input a random code vector z and output the fake data x f , such that x f = G(z), whereas the discriminator network D takes as input either a training sample or a synthesized sample from G and outputs a probability that the input is real or fake. The discriminator is trained to maximize the probability of assigning the correct source to both training samples and samples from G, where the generator is trained to minimize the probability that D assigns its generated sample as fake. The objective function is

EQUATION

, where D and G are parameterized by θ D and θ G respectively.

Compared to original GAN, Wasserstein GAN is more stable and resilient to hyper-parameters changes (Arjovsky et al. (2017) ). Another advantage over original GAN is the cost function of it decreases steadily among training. Wasserstein GAN also has two networks D and G that are trained simultaneously and adversarially. The main difference is instead outputting a probability, D outputs a real value. The objective function becomes:

EQUATION

The difference between the first and the second term is the estimated Wasserstein distance (W ). While D maximizes the distance, G minimizes it. Importantly, D needs to be a 1-Lipschitz function so that the estimation is accurate. A penalty term involves the first-order derivative of D is added to the objective function to encourage this requirement. Readers can find the details of it in Gulrajani et al. (2017) .

2.1.2. CONDITIONAL GAN:

Conditional GAN is an extension of GAN that makes it conditional on the prior information of the data, and here we focus on a type of the conditional GAN that is conditional on image labels or attributes. More specifically, we applied the conditional GAN model proposed in Odena et al. (2016) (AC-GAN). Here we briefly introduce this model. Assume a real sample x and its label l x follows an unknown distribution P data . The label is represented as a one-hot representation. The generator G takes as input a random code vector z u concatenated with a one-hot representation of a random label z l to generate a fake sample, such that

z c = [z u , z l ], x f = G(z c ).

To encourage G to generate a sample that is conditional on z l , we have an auxiliary classifier C, such that D and C share all layers except the last layer. While D outputs the Wasserstein distance mentioned above, C takes the input sample and its label, and outputs a cross-entropy loss between the label and the probability distribution over the class labels. The objective function becomes:

EQUATION

Where the first two terms correspond to the Wasserstein distance in equation 2, the third term corresponds to the cross-entropy loss for a real image (L r c ), and the fourth term corresponds to the cross-entropy loss for the fake image (L f c ).

2.1.3. LAYERED-STRUCTURE MODELING FOR IMAGES:

Since an image is taken from our 3D world, it usually contains a layered structure. Modeling an image as with a mask is common. An example of that is a two-layered foreground/background modeling,

x = f m + b (1 -m)

, where f is the foreground image, b is the background, m is the mask for the foreground, and is element-wise multiplication. This modeling has already been applied in synthesize natural images (Isola & Liu (2013) , Reed et al. (2016b) , Yan et al. (2016) ). This modeling has also been applied to videos (Darrell & Pentland (1991) , Wang & Adelson (1994) , Jojic & Frey (2001) , Kannan et al. (2005) , Vondrick et al. (2016) .

Recently there are two GAN models also applied the foreground/background layer setting Kwak & Zhang (2016) , Yang et al. (2017) , the generation process for them is the second layer (foreground) is generated subsequently after the first layer (background). In contrast, we model an image as a context layer and a conditional layer and generate the two layers together.

2.2 CONDTION-CONTEXT-COMPOSITE GAN (3C-GAN)

2.2.1. MODEL DEFINITION:

Our model is based on the conditional GAN and Wasserstein GAN. It has one Discriminator D, one auxiliary classifier C, but two generators G 1 and G 2 . G 1 is an ordinary generator, but G 2 is a conditional generator. While G 1 generates the context of an image, G 2 generates the part of an image that is conditional on its input label. The final output, (X f ), is the composite image from G 1 and G 2 . The model definition is shown below.

m 1 , f 1 = G 1 (z u , z l ) m 2 , f 2 = G 2 (z u , z v , z l ) m n 1 , m n 2 = softmax(m 1 , m 2 ) o 1 = f 1 m n 1 o 2 = f 2 m n 2 x f = G c (z u , z v , z l ) = o 1 + o 2

Figure 1 shows the architecture of 3C-GAN. A part of input noise codes (z u ) that is for context generation. z u is shared for both G 1 and G 2 because G 2 also needs to know the context to generate. The same z u for G 1 and G 2 inputs enable them to "communicate" with each other. At the same time, G 2 takes additional input codes (z v ) and label codes (z l ). z v let G 2 has additional power to control over the conditional part.

We denote G c as the composite model of G 1 and G 2 . Each generator generates a mask (m 1 , m 2 ) and an appearance (f 1 , f 2 ). The masks generated from G 1 and G 2 are further normalized by softmax. Each appearance element-wisely multiplies by each mask and summed together. This way the mask controls to generate which part of an image. The composite fake image x f is then given to discriminator and the optimization is as same as equation 3.

There is an auxiliary classifier C helps the model conditional on its label. As AC-GAN, Both x r and its label l x pair, and x f , z l pair are given to C to compute the classification loss. The losses for two pairs are the (L r c ) and (L f c ) in equation 3, respectively. The loess encourage G c as a whole to be conditional on the label; However, they do not encourage G 2 to be a conditional generator. Intuitively, if G 2 generates the conditional part, G 2 should have to be discriminated as the same label as G c ; therefore, we proposed an additional classification loss (L p c ) for the input pair o 2 , z l , given the same classifier C. That is:

EQUATION

In practice, there is no restriction on the structure of D and G. To have good generating quality and fast convergence, we applied the structure of DCGAN (Radford et al. (2015) ) . To save computational power, G 1 and G 2 can share all the layers except the first(bottom) layer, that is a fully-connected layer computing the 4*4 feature map from the input code.

Besides the basic GAN cost and conditional cost (classifier loss), there are two additional costs that are essential for our purpose.

2.2.2. LABEL DIFFERENCE COST:

In the model defined above, G 2 is not restricted to generate the unconditional part of an image, that is, G 2 could take the job G 1 does. In the extreme case, G 2 could generate all images, that is, m n 1 is zero everywhere. Here we proposed a cost to penalize G 2 such that it generates only the succinct part when changing the condition of an image.

EQUATION

, where z f l (label flip) is any label code that is different from z l , and the summation is over all pixels in an image. Since only G 2 is accessible to the condition information, G 2 is forced to generate the succinct part of condition changes when we apply this cost.

2.2.3. EXCLUSIVE PRIOR:

In the above model, except that m 1 + m 2 = 1, we do not have any prior on these masks. In practice, if one pixel is generated by a generator, we want it to take a full responsibility for generating it, that is, the value of its mask for that pixel should be close to 1. Here we proposed an exclusive prior on the two masks.

L ex = E zu,zv,z l |m 1 m 2 | (6)

, where the summation is over all pixels in an image. We found adding this prior makes G 1 and G 2 generate a part of an image separately.

2.2.4. OBJECTIVE FUNCTION:

The overall loss function for 3C-GAN is:

L all = W + L r c + L f c + L p c + αL ld + βL ex (7)

The definition of W, L r c , L f c is the same as that in conditional GAN. L p c is defined in the model definition for 3C-GAN. α and β are additional hyper-parameters that need to be tuned. In practice, we first tune α with β fixed to 0, find the best alpha, then tune β with the best α. We can tune both α, β qualitatively or quantitatively. If we visualize x f , we can see if it is generated according to labels. If not, alpha for that setting is too high. Also, if we visualize m 1 or m 2 , we can see if they are near zero or one. On the other hand, if the loss L f c and L p c becomes too high, we know the alpha is set too high.

3. EXPERIMENT:

We conduct experiments on three datasets 1)MNIST LeCun et al. (1998) 2) SVHN Netzer et al. (2011) 3)CelebA Liu et al. (2015) . To display our method can separately generate the part of an image with its label, we modify MNIST data making the image background a uniform grayscale value between [0, 200] , and resize it to 32*32. We name this modified version "MNIST-BACKGROUND". For SVHN, we use the extra set of the cropped digits. For CelebA, we focus on the label of smile/not smile, and use the set of aligning face. We implement our method based on open source code1 . The details of parameters of each experiment will be discussed in each section.

3.1. RESULTS FOR MNIST-BACKGROUND DATASET:

MNIST-BACKGROUND is similar to MNIST, with 10 different digits and 50000 images for training. We set the channel number of the model to be 32, and the input noise code number to be 30. There are 15 context codes (z u ), 15 object codes (z v ), and 10 label codes (z l ). Therefore, the input dimensionality for G 1 is 15 and for G 2 is 40. The weight for label difference cost (L l d) is set to be 1 (α = 1), and the weight for exclusive cost (L e x) is set to be 0 (β = 0). The two generators shared all structure except the bottom layer, and model is trained for 100000 iterations.

The results are shown in figure 2 to figure 4. The three figures are generated with the same input noise code so we can compare them. The samples are generated conditional on digit class 0, 1, ... to 9. There are totally 128 samples, each digit sample repeats 12 times, and the rest is fed with digit class 0. Since this dataset is simple, the training is converged, and the model shows good generation result (2). In addition, we show the results generated by G 1 and G 2 in figure 3 and figure 4 . For visualization purpose, we let a checkerboard pattern c composite with the output from G1, such that figure 3 shows

f 1 m n 1 + c (1 -m n 1 ). Similarly, figure 4 shows f 2 m n 2 + c (1 -m n 2 )

. The obvious checkerboard pattern in both figures means the mask is zero in those pixels. We can see G 1 only generate the context while G 2 generate the part of the image related to the label. SVHN is also a dataset with 10 different digits classes. In an image, a digit with the class label is in the center; however, there could be digits on the side of an image that does not related to the image's label. When training on this dataset, we set the channel number of the model to be 64, and the input noise code number to be 64. There are 32 context codes (z u ), 32 object codes (z v ), and 10 label codes (z l ). Therefore, the input dimensionality for G 1 is 32 and for G 2 is 74. α is set to 10, and β is set to 0.5. The two generators also shared all structure except the bottom layer, and model is trained for 100000 iterations.

The results are shown in figure 5 to figure 7. The setting is the same as before: the three figures are generated with the same input noise code, and the samples are generated conditional on digit class 0, 1, ... to 9. Figure 5 shows the model generates fairly reasonable images. In figure 6 and figure 7 , we see G 2 only generates the center digit in an image that is labeled, other digits are left to G 1 . Compared to the results for MNIST-BACKGROUND, one major difference is the digits in G 2 include the context other than the digit itself. We suspect this happens because sometimes the digits are dark while the background is light, and sometimes vice versa. And the digits generated by G 2 need to include the context around to have the classifier C give it a high probability. 

3.3. RESULTS FOR CELEBA DATASET:

There are 20599 images on this dataset. We pre-process the dataset by resizing the images to 64by-64. We choose the attribute of smiling as image label for the conditional generator in our model because the number of images of smiling and not-smiling is similar. When training on this dataset, we set the channel number of the model to be 128, and the input noise code number to be 128 ( 64 context codes (z u ), 64 object codes (z v ), and 2 label codes (z l ). α is set to 100, and β is set to 0.5.

The results are shown in figure 8 to figure 10 . There are 100 samples, with the first fifty samples belong to the class "not smiling", and the second fifty belongs to "smiling". The results show G 2 focus on face generation while G 1 focus on background generation. Since the condition of smiling/not smiling is determined by the face, it makes sense that G 2 take the responsibility to generate the faces. However, the separation is not as clear as the results on SVHN and MNISTB. We suspect this happens because we use the attribute smiling/not smiling as a condition, which only affects a small fraction of pixels in an image, and that is not strong enough to provide G 2 a solution to generate the whole face. 

:

We are the first one showing that by applying label information, the model and the cost function we proposed, we can have a generator to learn to generate the part of the image only related to the label.

We show that a simpler multi-generators structure, without using Recurrent Neural Net to generate different part of the image subsequently, works to generate images part-by-part simultaneously.

https://github.com/igul222/improved_wgan_training

Figures and Tables:

Figure 1: The architecture of 3C-GAN. The σ stands for softmax normalization, and the stands for element-wise multiplication. Note that classifier network C is reused three times for different input and label pairs.

Figure 2: composite generated samples x f for MNIST-BACKGROUND dataset

Figure 4: generated sample from G 2 for MNIST-BACKGROUND dataset

Figure 6: generated samples from G 1 for SVHN dataset

Figure 7: generated sample from G 2 for SVHN dataset

Figure 8: composite generated samples x f for CelebA dataset

Figure 9: generated samples from G 1 for CelebA dataset

Peer review (iclr2018):
 [Overview] This paper proposed a new generative adversarial network, called 3C-GAN for generating images in a composite manner. In 3C-GAN, the authors exploited two generators, one (G1) is for generating context images, and the other one (G2) is for generating semantic contents. To generate the semantic contents, the authors introduced a conditional GAN scheme, to force the generated images to match the annotations. After generating both parts in parallel, they are combined using alpha blending to compose the final image. This generated image is then sent to the discriminator. The experiments were conducted on three datasets, MNIST, SVHN and MS-CelebA. The authors showed qualitative results on all three datasets, demonstrating that AC-GAN could disentangle the context part from the semantic part in an image, and generate them separately. [Strenghts] This paper introduced a layered-wise image generation, which decomposed the image into two separate parts: context part, and semantic part. Corresponding to these two parts are two generators. To ensure this, the authors introduced three strategies: 1. Adding semantic labels: the authors used image semantic labels as the input and then exploited a conditional GAN to enforce one of the generators to generate semantic parts of images. As usual, the label information was added as the input of generator and discriminator as well. 2. Adding label difference cost: the intuition behind this loss is that changing the label condition should merely affect the output of G2. Based on this, outputs of Gc should not change much when flipping the input labels. 3. Adding exclusive prior: the prior is that the masks of context part (m1) and semantic part (m2) should be exclusive to each other. Therefore, the authors added another loss to reduce the sum of component-wise multiplication between m1 and m2. Decomposing the semantic part from the context part in an image based on a generative model is an interesting problem. However, to my opinion, completing it without any supervision is challenging and meaningless. In this paper, the authors proposed a conditional way to generate images compositionally. It is an interesting extension of previous works, such as Kwak & Zhang (2016) and Yang (2017). [Weaknesses] This paper proposed an interesting and intuitive image generation model. However, there are several weaknesses existed: 1. There is no quantitative evaluation and comparisons. From the limited qualitative results shown in Fig.2-10, we can hardly get a comprehensive sense about the model performance. The authors should present some quantitative evaluations in the paper, which are more persuasive than a number of examples. To do that, I suggest the authors exploited evaluation metrics, such as Inception Score to evaluate the overall generation performance. Also, in Yang (2017) the authors proposed adversarial divergence, which is suitable for evaluating the conditional generation. Hence, I suggest the authors use a similar way to evaluate the classification performance of classification model trained on the generated images. This should be a good indicator to show whether the proposed 3C-GAN could generate more realistic images which facilitate the training of a classifier. 2. The authors should try more complicated datasets, like CIFAR-10. Recently, CIFAR-10 has become a popular dataset as a testbed for evaluating various GANs. It is easy to train since its low resolution, but also means a lot since it a relative complicated scene. I would suggest the authors also run the experiments on CIFAR-10. 3. The authors did not perform any ablation study. Apart from several generation results based on 3C-GAN, iIcould not found any generation results from ablated models. As such, I can hardly get a sense of the effects of different losses and know about the relative performance in the whole GAN spectrum. I strongly suggest the authors add some ablation studies. The authors should at least compare with one-layer conditional GAN. 4. The proposed model merely showed two-layer generation results. There might be two reasons: one is that it is hard to extend it to more layer generation as I know, and the other one reason is the inflexible formulation to compose an image in 2.2.1 and formula (6). The authors should try some datasets like MNIST-TWO in Yang (2017) for demonstration. 5. Please show f1, m1, f2, m2 separately, instead of showing the blending results in Fig3, Fig4, Fig6, Fig7, Fig9, and Fig10. I would like to see what kind of context image and foreground image 3C-GAN has generated so that I can compare it with previous works like Kwak & Zhang (2016) and Yang (2017). 6. I did not understand very well the label difference loss in (5). Reducing the different between G_c(z_u, z_v, z_l) and G_c(z_u, z_v, z_l^f) seems not be able to force G1 and G2 to generate different parts of an image. G2 takes all the duty can still obtain a lower L_ld. From my point of view, the loss should be added to G1 to make G1 less prone to the variation of label information. 7. Minor typos and textual errors. In Fig.1, should the right generator be G2 rather than G1? In 2.1.3 and 2.2.1, please add numbers to the equations. [Summary] This paper proposed an interesting way of generating images, called 3C-GAN. It generates images in a layer-wise manner. To separate the context and semantic part in an image, the authors introduced several new techniques to enforce the generators in the model undertake different duties. In the experiments, the authors showed qualitative results on three datasets, MNIST, SVHN and CelebA. However, as I pointed out above, the paper missed quantitative evaluation and comparison, and ablation study. Taking all these into account, I think this paper still needs more works to make it solid and comprehensive before being accepted.