Title: 3C-GAN: AN CONDITION-CONTEXT-COMPOSITE GENERATIVE ADVERSARIAL NETWORKS FOR GEN-ERATING IMAGES SEPARATELY

Abstract: We present 3C-GAN: a novel multiple generators structures, that contains one conditional generator that generates a semantic part of an image conditional on its input label, and one context generator generates the rest of an image. Compared to original GAN model, this model has multiple generators and gives control over what its generators should generate. Unlike previous multi-generator models use a subsequent generation process, that one layer is generated given the previous layer, our model uses a process of generating different part of the images together. This way the model contains fewer parameters and the generation speed is faster. Specifically, the model leverages the label information to separate the object from the image correctly. Since the model conditional on the label information does not restrict to generate other parts of an image, we proposed a cost function that encourages the model to generate only the succinct part of an image in terms of label discrimination. We also found an exclusive prior on the mask of the model help separate the object. The experiments on MNIST, SVHN, and CelebA datasets show 3C-GAN can generate different objects with different generators simultaneously, according to the labels given to each generator.

1. INTRODUCTION:

Recently Generative adversarial network (GAN) is gaining much attention because it has shown promising results for generating natural images (Goodfellow et al. (2014) ; Goodfellow (2016) ). Still, it is limited in generating globally coherent, or high-resolution images. Therefore, to have GAN generate better quality images is the main research topic. There have been many works tackling this topic, including using novel network structures, (Radford et al. (2015) ), Denton et al. (2015) ), using novel objective functions for training ( Zhao et al. (2016) , Arjovsky et al. (2017) , Gulrajani et al. (2017) ), using multi-stage network generation (Im et al. (2016) , Kwak & Zhang (2016) , Yang et al. (2017) ), applying 3D structure information for image generation ( Wang & Gupta (2016) ).

Another main research avenue of GAN is to make a generator more controllable, that is, the generator can generate the image given the criterion we want. Conditional GAN serves this purpose by ensuring the generation is conditional on the given criterion (Odena et al. (2016) , Mansimov et al. (2015) , Reed et al. (2016a) , Kaneko et al. (2017) , Zhu et al. (2016) ). Specifically, in these conditional models, the generator not only accepts the noise code, but also a conditional code, either from a one-hot representation of class labels, or a sentence embedding from a text, or an image, such that this given side information encodes the criterion, and the generated images are conditional on that. Aside from the conditional model, that is a supervised approach, there are models employed an unsupervised approach. InfoGAN (Chen et al. (2016) ) does not need any labels: it discovered the high-level meaning of the image purely from data, and assign the meanings to its noise code. For example, for MNIST dataset, InfoGAN model learn to have one code control the width of the generated digits, and one code controls the rotation of the digits. As an unsupervised approach, InfoGAN discovered the criteria that we might think is useful for the dataset.

Both the supervised and unsupervised models are one-generator approach, there is another multigenerators approach that also aims to make the model more controllable (Kwak & Zhang (2016 ) Yang et al. (2017) ). They generate images part-by-part, such that each part has its semantic mean-ing, and combing all parts results in a realistic image. Since a part of an image has some persistent characteristics, these model learn which parts of the data should be generated together in an unsupervised way. In addition, the model of Yang et al. (2017) includes a transformer network that learns an affine transformation for the foreground in an image. Since each part of an image is generated by different generators, these model provide a high-level meaning of the generation for each generator and possess a finer control over original GAN.

We develop a model that also aims to improve the controllability over original GAN. Specifically, our model is a multi-generators approach, where one generator is a "context" generator that does not take any conditional labels as input, and another is a conditional generator that takes its label information. The conditional generator generates the part of the image that is related to its labels, whereas the context generator generates the other part of the image. Both generators learn where to generate (a mask), and what to generate (an appearance). The context generator shared its input codes with the conditional generator so it "knows" what context it is on when generating its own part. However, there is no restriction for the conditional generator to not generate the context. We propose a cost function on the label discrimination to penalize the conditional generator to do so. In addition, we proposed an exclusive prior on the mask such that any pixels in an image should be generated from one generator only. All in all, our contributions can be listed as follows.

2. METHOD:

The algorithm we proposed is based on Wasserstein GAN, conditional GAN, and model an image with a layered structure, we introduce them in the background section.

2.1. BACKGROUND:



2.1.1. WASSERSTEIN GAN:

A generative adversarial network (GAN) consists of two neural networks trained simultaneously and in opposition to one another. Assume a real sample x follows an unknown distribution p data . The generator network G takes as input a random code vector z and output the fake data x f , such that x f = G(z), whereas the discriminator network D takes as input either a training sample or a synthesized sample from G and outputs a probability that the input is real or fake. The discriminator is trained to maximize the probability of assigning the correct source to both training samples and samples from G, where the generator is trained to minimize the probability that D assigns its generated sample as fake. The objective function is

EQUATION

, where D and G are parameterized by θ D and θ G respectively.

Compared to original GAN, Wasserstein GAN is more stable and resilient to hyper-parameters changes (Arjovsky et al. (2017) ). Another advantage over original GAN is the cost function of it decreases steadily among training. Wasserstein GAN also has two networks D and G that are trained simultaneously and adversarially. The main difference is instead outputting a probability, D outputs a real value. The objective function becomes:

EQUATION

The difference between the first and the second term is the estimated Wasserstein distance (W ). While D maximizes the distance, G minimizes it. Importantly, D needs to be a 1-Lipschitz function so that the estimation is accurate. A penalty term involves the first-order derivative of D is added to the objective function to encourage this requirement. Readers can find the details of it in Gulrajani et al. (2017) .

2.1.2. CONDITIONAL GAN:

Conditional GAN is an extension of GAN that makes it conditional on the prior information of the data, and here we focus on a type of the conditional GAN that is conditional on image labels or attributes. More specifically, we applied the conditional GAN model proposed in Odena et al. (2016) (AC-GAN). Here we briefly introduce this model. Assume a real sample x and its label l x follows an unknown distribution P data . The label is represented as a one-hot representation. The generator G takes as input a random code vector z u concatenated with a one-hot representation of a random label z l to generate a fake sample, such that

z c = [z u , z l ], x f = G(z c ).

To encourage G to generate a sample that is conditional on z l , we have an auxiliary classifier C, such that D and C share all layers except the last layer. While D outputs the Wasserstein distance mentioned above, C takes the input sample and its label, and outputs a cross-entropy loss between the label and the probability distribution over the class labels. The objective function becomes:

EQUATION

Where the first two terms correspond to the Wasserstein distance in equation 2, the third term corresponds to the cross-entropy loss for a real image (L r c ), and the fourth term corresponds to the cross-entropy loss for the fake image (L f c ).

2.1.3. LAYERED-STRUCTURE MODELING FOR IMAGES:

Since an image is taken from our 3D world, it usually contains a layered structure. Modeling an image as with a mask is common. An example of that is a two-layered foreground/background modeling,

x = f m + b (1 -m)

, where f is the foreground image, b is the background, m is the mask for the foreground, and is element-wise multiplication. This modeling has already been applied in synthesize natural images (Isola & Liu (2013) , Reed et al. (2016b) , Yan et al. (2016) ). This modeling has also been applied to videos (Darrell & Pentland (1991) , Wang & Adelson (1994) , Jojic & Frey (2001) , Kannan et al. (2005) , Vondrick et al. (2016) .

Recently there are two GAN models also applied the foreground/background layer setting Kwak & Zhang (2016) , Yang et al. (2017) , the generation process for them is the second layer (foreground) is generated subsequently after the first layer (background). In contrast, we model an image as a context layer and a conditional layer and generate the two layers together.

2.2 CONDTION-CONTEXT-COMPOSITE GAN (3C-GAN)

2.2.1. MODEL DEFINITION:

Our model is based on the conditional GAN and Wasserstein GAN. It has one Discriminator D, one auxiliary classifier C, but two generators G 1 and G 2 . G 1 is an ordinary generator, but G 2 is a conditional generator. While G 1 generates the context of an image, G 2 generates the part of an image that is conditional on its input label. The final output, (X f ), is the composite image from G 1 and G 2 . The model definition is shown below.

m 1 , f 1 = G 1 (z u , z l ) m 2 , f 2 = G 2 (z u , z v , z l ) m n 1 , m n 2 = softmax(m 1 , m 2 ) o 1 = f 1 m n 1 o 2 = f 2 m n 2 x f = G c (z u , z v , z l ) = o 1 + o 2

Figure 1 shows the architecture of 3C-GAN. A part of input noise codes (z u ) that is for context generation. z u is shared for both G 1 and G 2 because G 2 also needs to know the context to generate. The same z u for G 1 and G 2 inputs enable them to "communicate" with each other. At the same time, G 2 takes additional input codes (z v ) and label codes (z l ). z v let G 2 has additional power to control over the conditional part.

We denote G c as the composite model of G 1 and G 2 . Each generator generates a mask (m 1 , m 2 ) and an appearance (f 1 , f 2 ). The masks generated from G 1 and G 2 are further normalized by softmax. Each appearance element-wisely multiplies by each mask and summed together. This way the mask controls to generate which part of an image. The composite fake image x f is then given to discriminator and the optimization is as same as equation 3.

There is an auxiliary classifier C helps the model conditional on its label. As AC-GAN, Both x r and its label l x pair, and x f , z l pair are given to C to compute the classification loss. The losses for two pairs are the (L r c ) and (L f c ) in equation 3, respectively. The loess encourage G c as a whole to be conditional on the label; However, they do not encourage G 2 to be a conditional generator. Intuitively, if G 2 generates the conditional part, G 2 should have to be discriminated as the same label as G c ; therefore, we proposed an additional classification loss (L p c ) for the input pair o 2 , z l , given the same classifier C. That is:

EQUATION

In practice, there is no restriction on the structure of D and G. To have good generating quality and fast convergence, we applied the structure of DCGAN (Radford et al. (2015) ) . To save computational power, G 1 and G 2 can share all the layers except the first(bottom) layer, that is a fully-connected layer computing the 4*4 feature map from the input code.

Besides the basic GAN cost and conditional cost (classifier loss), there are two additional costs that are essential for our purpose.

2.2.2. LABEL DIFFERENCE COST:

In the model defined above, G 2 is not restricted to generate the unconditional part of an image, that is, G 2 could take the job G 1 does. In the extreme case, G 2 could generate all images, that is, m n 1 is zero everywhere. Here we proposed a cost to penalize G 2 such that it generates only the succinct part when changing the condition of an image.

EQUATION

, where z f l (label flip) is any label code that is different from z l , and the summation is over all pixels in an image. Since only G 2 is accessible to the condition information, G 2 is forced to generate the succinct part of condition changes when we apply this cost.

2.2.3. EXCLUSIVE PRIOR:

In the above model, except that m 1 + m 2 = 1, we do not have any prior on these masks. In practice, if one pixel is generated by a generator, we want it to take a full responsibility for generating it, that is, the value of its mask for that pixel should be close to 1. Here we proposed an exclusive prior on the two masks.

L ex = E zu,zv,z l |m 1 m 2 | (6)

, where the summation is over all pixels in an image. We found adding this prior makes G 1 and G 2 generate a part of an image separately.

2.2.4. OBJECTIVE FUNCTION:

The overall loss function for 3C-GAN is:

L all = W + L r c + L f c + L p c + αL ld + βL ex (7)

The definition of W, L r c , L f c is the same as that in conditional GAN. L p c is defined in the model definition for 3C-GAN. α and β are additional hyper-parameters that need to be tuned. In practice, we first tune α with β fixed to 0, find the best alpha, then tune β with the best α. We can tune both α, β qualitatively or quantitatively. If we visualize x f , we can see if it is generated according to labels. If not, alpha for that setting is too high. Also, if we visualize m 1 or m 2 , we can see if they are near zero or one. On the other hand, if the loss L f c and L p c becomes too high, we know the alpha is set too high.

3. EXPERIMENT:

We conduct experiments on three datasets 1)MNIST LeCun et al. (1998) 2) SVHN Netzer et al. (2011) 3)CelebA Liu et al. (2015) . To display our method can separately generate the part of an image with its label, we modify MNIST data making the image background a uniform grayscale value between [0, 200] , and resize it to 32*32. We name this modified version "MNIST-BACKGROUND". For SVHN, we use the extra set of the cropped digits. For CelebA, we focus on the label of smile/not smile, and use the set of aligning face. We implement our method based on open source code1 . The details of parameters of each experiment will be discussed in each section.

3.1. RESULTS FOR MNIST-BACKGROUND DATASET:

MNIST-BACKGROUND is similar to MNIST, with 10 different digits and 50000 images for training. We set the channel number of the model to be 32, and the input noise code number to be 30. There are 15 context codes (z u ), 15 object codes (z v ), and 10 label codes (z l ). Therefore, the input dimensionality for G 1 is 15 and for G 2 is 40. The weight for label difference cost (L l d) is set to be 1 (α = 1), and the weight for exclusive cost (L e x) is set to be 0 (β = 0). The two generators shared all structure except the bottom layer, and model is trained for 100000 iterations.

The results are shown in figure 2 to figure 4. The three figures are generated with the same input noise code so we can compare them. The samples are generated conditional on digit class 0, 1, ... to 9. There are totally 128 samples, each digit sample repeats 12 times, and the rest is fed with digit class 0. Since this dataset is simple, the training is converged, and the model shows good generation result (2). In addition, we show the results generated by G 1 and G 2 in figure 3 and figure 4 . For visualization purpose, we let a checkerboard pattern c composite with the output from G1, such that figure 3 shows

f 1 m n 1 + c (1 -m n 1 ). Similarly, figure 4 shows f 2 m n 2 + c (1 -m n 2 )

. The obvious checkerboard pattern in both figures means the mask is zero in those pixels. We can see G 1 only generate the context while G 2 generate the part of the image related to the label. SVHN is also a dataset with 10 different digits classes. In an image, a digit with the class label is in the center; however, there could be digits on the side of an image that does not related to the image's label. When training on this dataset, we set the channel number of the model to be 64, and the input noise code number to be 64. There are 32 context codes (z u ), 32 object codes (z v ), and 10 label codes (z l ). Therefore, the input dimensionality for G 1 is 32 and for G 2 is 74. α is set to 10, and β is set to 0.5. The two generators also shared all structure except the bottom layer, and model is trained for 100000 iterations.

The results are shown in figure 5 to figure 7. The setting is the same as before: the three figures are generated with the same input noise code, and the samples are generated conditional on digit class 0, 1, ... to 9. Figure 5 shows the model generates fairly reasonable images. In figure 6 and figure 7 , we see G 2 only generates the center digit in an image that is labeled, other digits are left to G 1 . Compared to the results for MNIST-BACKGROUND, one major difference is the digits in G 2 include the context other than the digit itself. We suspect this happens because sometimes the digits are dark while the background is light, and sometimes vice versa. And the digits generated by G 2 need to include the context around to have the classifier C give it a high probability. 

3.3. RESULTS FOR CELEBA DATASET:

There are 20599 images on this dataset. We pre-process the dataset by resizing the images to 64by-64. We choose the attribute of smiling as image label for the conditional generator in our model because the number of images of smiling and not-smiling is similar. When training on this dataset, we set the channel number of the model to be 128, and the input noise code number to be 128 ( 64 context codes (z u ), 64 object codes (z v ), and 2 label codes (z l ). α is set to 100, and β is set to 0.5.

The results are shown in figure 8 to figure 10 . There are 100 samples, with the first fifty samples belong to the class "not smiling", and the second fifty belongs to "smiling". The results show G 2 focus on face generation while G 1 focus on background generation. Since the condition of smiling/not smiling is determined by the face, it makes sense that G 2 take the responsibility to generate the faces. However, the separation is not as clear as the results on SVHN and MNISTB. We suspect this happens because we use the attribute smiling/not smiling as a condition, which only affects a small fraction of pixels in an image, and that is not strong enough to provide G 2 a solution to generate the whole face. 

:

We are the first one showing that by applying label information, the model and the cost function we proposed, we can have a generator to learn to generate the part of the image only related to the label.

We show that a simpler multi-generators structure, without using Recurrent Neural Net to generate different part of the image subsequently, works to generate images part-by-part simultaneously.

https://github.com/igul222/improved_wgan_training

Figures and Tables:

Figure 1: The architecture of 3C-GAN. The σ stands for softmax normalization, and the stands for element-wise multiplication. Note that classifier network C is reused three times for different input and label pairs.

Figure 2: composite generated samples x f for MNIST-BACKGROUND dataset

Figure 4: generated sample from G 2 for MNIST-BACKGROUND dataset

Figure 6: generated samples from G 1 for SVHN dataset

Figure 7: generated sample from G 2 for SVHN dataset

Figure 8: composite generated samples x f for CelebA dataset

Figure 9: generated samples from G 1 for CelebA dataset

Peer review (iclr2018):
 Summary: This paper studied the conditional image generation with two-stream generative adversarial networks. More specifically, this paper proposed an unsupervised learning approach to generate (1) foreground region conditioned on class label and (2) background region without semantic meaning in the label. During training, two generators are competing against each other to hallucinate foreground region and background region with a physical gating operation. An auxiliary “label difference cost” was further introduced to encourage class information captured by the foreground generator. Experiments on MNIST, SVHN, and CelebA datasets demonstrated promising generation results with the unsupervised two-stream generation pipeline. == Novelty/Significance == Controllable image generation is an important task in representation learning and computer vision. I also like the unsupervised learning through gating function and label difference cost. However, considering many other related work mentioned by the paper, the novelty in this paper is quite limited. For example, layered generation (Section 2.2.1) has been explored in Yan et al 2016 (VAEs) and Vondrick et al 2016 (GANs). == Detailed comments == The proposed two-stream model is developed with the following two assumptions: (1) Single object in the scene; and (2) Class information is provided for the foreground/object region. Although the proposed method learns to distinguish foreground and background in an unsupervised fashion, it is limited in terms of applicability and generalizability. For example, I am not convinced if the two-stream generation pipeline can work well on more challenging datasets such as MS-COCO, LSUN, and ImageNet. Given the proposed method is controllable image generation, I would assume to see the following ablation studies: keeping two latent variables from (z_u, z_l, z_v) fixed, while gradually changing the value of the other latent variable. However, I didn’t see such detailed analysis as in the other papers on controllable image generation. In Figure 7 and Figure 10, the boundary between foreground and background region is not very sharp. It looks like equation (5) and (6) are insufficient for foreground and background separation (triplet/margin loss could work better). Also, in CelebA experiment, it is not a well defined experimental setting since only binary label (smiling/non-smiling) is conditioned. Is it possible to use all the binary attributes in the dataset. Also, please either provide more qualitative examples or provide some type of quantitative evaluations (through user study , dataset statistics, or down-stream recognition tasks). Overall, I believe the paper is interesting but not ready for publication. I encourage authors to investigate (1) more generic layered generation process and (2) better unsupervised boundary separation. Hopefully, the suggested studies will improve the quality of the paper in the future submission. == Presentation == The paper is readable but not well polished. -- In Figure 1, the “G1” on the right should be “G2”; -- Section 2.2.1, “X_f” should be “x_f”; -- the motivation of having “z_v” should be introduced earlier; -- Section 2.2.4, please use either “alpha” or “.alpha” but not both; -- Section 3.3, the dataset information is incorrect: “20599 images” should be “202599 images”; Missing reference: -- Neural Face Editing with Intrinsic Image Disentangling, Shu et al. In CVPR 2017. -- Domain Separation Networks, Bousmalis et al. In NIPS 2016. -- Unsupervised Image-to-Image Translation Networks, Liu et al. In NIPS 2017.