Generate Anime Style Face Using DCGAN and Explore Its Latent Feature Representation | by Haryo Akbarianto Wibowo | Towards Data Science

Hello everyone , It’s been a while ! Today, I want to write about my result of learning and experimenting another Deep Learning technique , which is Generative Adversarial Network (GAN). I studied and learned about it recently . I think it would be nice if share my experiment to everyone.
GAN is mostly about generating something. In this article I want to share about the experiment on generating anime character faces. Not only generate, I’ve also experimented that the image can be manipulated by linear algebra operation of its latent variable (a vector that is used on generating the faces). I also see that the faces that are generated follow a statistic distribution, which is really awesome.
This article will be focused on the tutorial how to do GAN with each steps explained (with source code). It will be targeted for anyone who is interested in AI, especially who want to practice on using Deep Learning. It also targets everyone who want to learn how to do GAN for the first time. I will write this article as easy as possible to understand about it. I hope for the reader, by reading this article, they know how general GAN works.
If you want to grasp better understanding on reading this article, I suggest that you know at least neural network and Convolution Neural Network (CNN).
There is a GitHub link at the end of this article if you want to know about the complete source code. For now, I will give the python notebook and Colaboratory link in the repository.
Image 0 is one of the generated anime character faces that we will create by using the picture formed by the model. The first and second picture from the left is generated with GAN. The third is the addition of the first and second faces (You can call it a fusion of the first and second faces).
TechnologyIntroductionBrief Description About GANImplementationResultsLesson LearnedConclusionAfterwordsRepositorySources
Technology
Introduction
Brief Description About GAN
Implementation
Results
Lesson Learned
Conclusion
Afterwords
Repository
Sources
Python 3.7Colaboratory : Free Jupyter notebook environment that requires no setup and runs entirely in the cloud. Have GPU Tesla K80 or even TPU! Sadly Tensorflow v2.0 alpha still does not support TPU at the moment of this writing. Sadly, DCGAN cannot be trained via TPU.Keras : Python Library for doing Deep Learning.Data is taken from here
Python 3.7
Colaboratory : Free Jupyter notebook environment that requires no setup and runs entirely in the cloud. Have GPU Tesla K80 or even TPU! Sadly Tensorflow v2.0 alpha still does not support TPU at the moment of this writing. Sadly, DCGAN cannot be trained via TPU.
Keras : Python Library for doing Deep Learning.
Data is taken from here
One of the topic that is hot in the Deep Learning field is Generative Adversarial Network (GAN). Introduced by Ian Goodfellow et al., It can generate something from scratch unsupervised. In Computer Vision. There are many researchers out there researching and improving it. For example NVIDIA create realistic face generator by using GAN. There are also some research on the music domain on using GAN. My previous article that shows about generating music can also be done by using GAN.
There are many variant type of GAN developed by the researchers out there. One of the newest (by the time I write this article) is HoloGAN that can generate 3D representation from natural images. If you look at how it can do, it is actually amazing. Actually, these advanced GAN follow basic fundamental of how GAN works. Every GAN out there have two agents as its learner, discriminator and generator (We will dive into these terms later) . To know more about advanced GAN techniques, one must know how the basic GAN works.
This article will focus on implementing Deep Convolutional GAN (DCGAN), one of the variant of GAN proposed by A Radford et al,. Basically, it is a GAN with many Convolution Layers. It is one of the popular GAN neural network . We will build a different architecture from the proposed architecture in their paper. Although different, it still yield some good results.
One of the interesting things about GAN is that it will build its latent variables (a 1-D vector of any length) which can be linear algebra operated. The example on Image 0 is one of the example. The first face’s vector (from the left) is added to the second face’s vector. Then, it yield the third face.
It also yield some interesting data distribution. Every point in the distribution have different kind of faces. For example, the data who is centered at mean -0.7 will have face with yellow hair.
We will start from knowing a brief description about GAN.
To make it simpler, it is one of the Deep Learning technique used to generate some new data from scratch. It runs in unsupervised way meaning that it can run without labelled by human. It will make the data based on the pattern that it learns.
There are some characteristics aspects on GAN who is a generative model, which is:
Learns joint probability P(x,y) where x is the input and y the output. It will do inference based on P(x|y) , given output y, it will infer the x. You can say that y is the the real data in GAN.
When the model is given the training real data y, it will learn the characteristic of the real data. It will learn by identifying the real data latent feature representation variable. To make it simpler, it learns the base constructor feature of the images in the real data. For example, the model can learn that faces constructed by the color of the eyes and hair. These two will be one of the base which will be used on generating the faces. By tweaking its variable, it can also alter the generated faces. For example, by raising the variable of the eyes, the eyes will be blacker. Lowering it will make the opposite otherwise.
It can build a probability distribution such as normal distribution which can be used on avoiding the outlier. Since outlier usually very rare in the distribution, it will be very rare to generate it. So GAN functions well on real data that has outlier.
GAN composes into two neural networks, the Discriminator and Generator. GAN will make these two networks fight each others on a Zero-Sum game framework (Game Theory). This is a game between these agents (the networks). The adversarial name in GAN comes from this concept.
Generator will generate some fake data and the Discriminator will identify a couple of data which has the fake data generated by Generator and data sampled from real data. The objective of the Generator is mainly generating some fake data that is intended similar to the real data and fooling the Discriminator on identifying what data is real and fake. The objective of the Discriminator is to make it smarter on identifying the real and fake data. Each agents will move alternately. By dueling these agents, We hope that these agents will get stronger, especially the Generator.
You can say that they are rival destinied to each others. The main character is the Generator who strive better and better to make our purpose realized by learning from the fight from its rival.
Okay, in another word, the Generator will mimic the real data by sampling the distribution that is learned and intended to be the same distribution as the real data. It will train its neural network which can generate it. Whereas, The Discriminator will train its neural network in supervised technique on detecting the fake and real data. Each network will train its network alternately.
Here is the rough steps on how GAN works:
Generate random noise in a probability distribution such as normal distribution.Make it as an input of our Generator neural network. It will output generated fake data . These steps can also mean that we sample some data from the distribution that the generator learned. We will notate the noise as z_n and the generated data as G(z_n) . G(z_n) means the result of the noise processed by the generator G.We combine the generated fake data with the data that is sampled from the dataset (which is real data). Make them to be input of our Discriminator. We will notate it as D. Discriminator will try to learn by predicting whether the data is fake or not. Train the neural network by doing forward pass and followed by back propagation. Updates the D weights.Then, we need to train the Generator. We need to make the G(z_n) or the fake data generated from random noises as the input of the D. Note that this steps only input the fake data into the Discriminator. Forward pass the G(z_n) in D. By using the Discriminator neural network, on doing forward pass, predict whether the fake data is fake or not (D(G(z_n))). Then do back propagation where we will only update the G weights.Repeat these steps until we can see that the generator provide good fake data or maximum iteration has been reached.
Generate random noise in a probability distribution such as normal distribution.
Make it as an input of our Generator neural network. It will output generated fake data . These steps can also mean that we sample some data from the distribution that the generator learned. We will notate the noise as z_n and the generated data as G(z_n) . G(z_n) means the result of the noise processed by the generator G.
We combine the generated fake data with the data that is sampled from the dataset (which is real data). Make them to be input of our Discriminator. We will notate it as D. Discriminator will try to learn by predicting whether the data is fake or not. Train the neural network by doing forward pass and followed by back propagation. Updates the D weights.
Then, we need to train the Generator. We need to make the G(z_n) or the fake data generated from random noises as the input of the D. Note that this steps only input the fake data into the Discriminator. Forward pass the G(z_n) in D. By using the Discriminator neural network, on doing forward pass, predict whether the fake data is fake or not (D(G(z_n))). Then do back propagation where we will only update the G weights.
Repeat these steps until we can see that the generator provide good fake data or maximum iteration has been reached.
The illustration is as follow:
By updating the distribution of the Generator to match the Discriminator. It’s the same as minimizing the JS Divergence. For further information, you can read this article.
To make our agents learn, make sure to make the discriminator and generator dominate each other. Make them in a balance as possible and make discriminator and generator learn at the same time. When the discriminator is too strong (can differentiate between fake and real 100%), Generator become unable too learn anything. If at the process of training we reach this point, better to end it. The opposite also have effect where the Generator is stronger than Discriminator. It causes Mode Collapse where our model will always predict the same outcome for any random noises. This is one of the hard and difficult part of the GAN that can make someone frustrated.
If you want to understand more, I suggest to take a peek and read this awesome article.
This depends on the variant of the GAN that we will develop. Since we will be using DCGAN, we will use a sequential couple of CNNs layer.
We will use a custom architecture that is different from the original paper. I follow the architecture used in the François Chollet Deep Learning with Python book with some changes.
The configuration that we used for building the DCGAN is as follow:
latent_dim = 64height = 64width = 64channels = 3
This means that we will have a 64 dimension of latent variables. The height and width of our images are 64. Every images have 3 channels (R, G, B)
Here is the imported library and how the data is prepared:
Here is the architecture :
It consists of Convolution Layers where one of them is Convolution Transpose Layer. To upsample the size of the image (32 -> 62), we will use strides parameter in the Convolution Layer. This is done to avoid unstable training of the GAN.
Code
It also consists of Convolution Layers where we use strides to do downsampling.
Code
To make the backpropagation possible for the Generator, we create new network in Keras, which is Generator followed by Discriminator. In this network, we freeze all the weights so that its weight do not changes.
This is the network :
The config of the training is as follow:
iterations = 15000 batch_size = 32
The configuration means that we will do 15000 iterations. Every iteration we process 32 batches of real data and fake data (64 in total for training the discriminator).
Following the rough steps that I’ve explained above, here is how we train the DCGAN step by step:
Iterate until max iterations the following steps
Iterate until max iterations the following steps
for step in tqdm_notebook(range(iterations)):
2. Generate random noise in a probability distribution such as normal distribution.
random_latent_vectors = np.random.normal(size = (batch_size, latent_dim))generated_images = generator.predict(random_latent_vectors)
3. Combine the generated fake data with the data that is sampled from the dataset.
stop = start + batch_sizereal_images = x_train[start: stop]combined_images = np.concatenate([generated_images, real_images])labels = np.concatenate([np.ones((batch_size,1)),                                     np.zeros((batch_size, 1))])
Note that we use sequential sampler where each data will be sampled sequentially until the end of the data. The number that will be sampled is equal to the batch size.
4. Add noise to the label of the input
labels += 0.05 * np.random.random(labels.shape)
This is an important trick on training GAN.
5. Train the Discriminator
d_loss = discriminator.train_on_batch(combined_images, labels)
6. Train the Generator
random_latent_vectors = np.random.normal(size=(batch_size,                                                  latent_dim))misleading_targets = np.zeros((batch_size, 1))a_loss = gan.train_on_batch(random_latent_vectors,                               misleading_targets)
Note that we create a new latent vectors . Don’t forget that we need to swap the label. Remember we want to minimize the loss resulted from the discriminator on failing predicting the fake . The label should be 1 for the misleading_targets.
7. Update the start index of the real dataset
start += batch_size    if start > len(x_train) - batch_size:    start = 0
That’s it, here is the complete code on training the DCGAN:
Okay, let the fun begins! We will begin on visualizing generated images on different mean point. Before we do that, let me tell you that this is the result of above model who is trained 20000 steps (iterations) and trained with 30000 steps. The model is trained around 7 hours ( ~4300 steps per hours) . I will give name to the fewer steps model as Model-A and the other as Model-B.
Here we go!
How to read
N ~ (x, y) : latent vectors randomly generated by following the normal distribution that has mean x and Standard Deviation y
Results on latent vectors on N ~ (0,0.4) on Model-A:
Not bad huh, although there are some images who has asymetric face.
Results on latent vectors on N ~ (0,1) on Model-A:
Look at it.. the model produced some abomination faces out there. This turns out that this model doesn’t really grasp the distribution of the real data. It can do better when the standard deviation is lower. The DCGAN that I trained has not yet grasped how to represent the data point which is not too close to the mean point. I think, it needs more training or more powerful architecture.
Let’s change the architecture to Model-B
Results on latent vectors on N ~ (0,0.4) on Model-B:
Its okay, but the faces become darker. I wonder what happened to the generator.
Results on latent vectors on N ~ (0,1) on Model-B:
Uhmm, well.. most of them still contains abomination faces. Some of them produced okay faces. The quality still almost the same as Model-A. Okay.. For the next batches of images, let’s change the standard deviation as close as the mean. 0.4 would be the best.
Let’s check if our latent vectors is generated with centered mean of -0.3 and 0.3 using the same standard deviation.
Results on latent vectors on N ~ (-0.3,0.4) on Model-A:
Results on latent vectors on N ~ (0.3,0.4) on Model-A:
Results on latent vectors on N ~ (-0.3,0.4) on Model-B:
Results on latent vectors on N ~ (0.3,0.4) on Model-B:
See the differences?
Yes, look at their hair. At mean 0.3 the hair is mostly black (some of them are brown). On the contrary, at mean -0.3 the hairs are mostly yellow. Yeah, our model can put the faces at its corresponding points. Also the Model-B generate faces that are darker than A.
From what we’ve done above, we can get the intuition on how the data distribution that our model has learnt.
Lets plot it:
From the result has shown above, I think the less the latent vector mean, it will make the face have brighter hair and the more the latent vector is, the faces will have purpler hair.
To make sure, let’s see the mean of face in each mean points:
We plot these latent vector whose mean is:
[-1, -0.8, -0.6, -0.4, -0.2, 0, 0.2 0.4, 0.6, 0.8 ]
The first row is MODEL-A and the second row is MODEL-B. By manipulating the mean of the latent vectors, we can see the face that it generate at that point. We can see that:
The lower the point of the vector is, the yellower the hair is.
It has darker face in the middle. It means that average faces in the dataset have these style.
The positive the point of the vector is , the bluer the hair is. The positive latent vector also has more open mouth on smiling.
Amazing isn’t it?
Not yet, We can do linear algebra operation to the latent vector. The result of the equation can also be generated and have interesting result. Take a result from our first faces before the introduction section:
G + D
GAN faces is the result of addition of G with D. You can see that the hair become a bit brown. and the hair follow the D style on the right and G on the left.
Below are the result of others operation:
G — D (Absolute)
Component-Wise multiplied (G, D)
We will see how is the generated images if we manipulate a dimension in the latent vector. As I said earlier, the model will learn latent feature representation. So, every element in the latent vector has a purpose on generating the image.
To do the visualization, we will freeze all elements in vectors and change tthe chosen dimension that want to be checked.
For example we want to examine the 1st elements in the latent vector, we change that dimension and keep the others the same.
We will generate some faces that has the mean in these points:
[-0.6, -0.3, 0.1, 0.3, 0.6]
For every mean points, we will generate faces whose a dimension in latent vector is changed with these values iteratively:
[-1.08031934, -0.69714143, -0.39691713, -0.12927146,  0.12927146, 0.39691713,  0.69714143,  1.08031934]
Let visualize on our chosen dimension: ( This section will only use MODEL-A)
28th dimension:
What are the 28th latent variable’s purposes?
I think, it make the hair become brighter, change the left eye shape, and also small changes on the right eye. Since it compresses the feature into 64 length latent vector, one dimension can have several purposes.
Let’s see another one!
5th dimension
What are this latent variable purposes?
I think it has something to do with the left eyes even though the treatment on how the left eyes is different for each mean points. It also make the hair a bit darker. What do you think?
11th dimension
I think, this dimension care about the mouth and the right eye.
Another example on faces that generated from a mean points by only tweaking a latent variable:
That’s it. We can plot any dimensions in the latent vectors and see what are its purposes. Although sometimes it’s hard to see what are a latent variable’s purposes.
Let’s sample 8 real faces from dataset:
And sample 8 from generator N ~ (0,1) for model A and B:
So, if we act as Discriminator, can we differentiate between the real and fake faces?
No doubt that we can still differentiate between what faces are fake and real. The model need more training or powerful architecture to make it happen. Even so, our model can still generate anime style faces shape, which is great.
Below is the lesson that I learnt after researched DCGAN:
Training GAN is hard. It’s hard on making a stable architecture without seeing tips and trick from the one who has experienced it. Especially on balancing the power of the Discriminator and Generator. Making the GAN not become collapse is also a challenge.
Actually, these model is still not good at generating the fake images. Nevertheless, it can build some good faces although not as good as the real one. We can still differentiate the fake images and real images. This is because the model hasn’t grasped the data distribution of the real data yet.
The model degrades its quality around 26000 steps. This is where in my experiment, the generator became weak. This is an instability in the GAN. I need to search a better architecture to do it. We can see the result on the Model B becomes darker.
And so, I’ve developed another architecture with the Batch Normalization and even Dropout Layer. Guess what? There are two result that I have in tweaking the architecture. Model collapse and Discriminator dominance. I guess developing the GAN architecture is not easy.
Yet, there are many tips and tricks on developing a good GAN that I haven’t implemented. Maybe the instability of the model can be reduced by following these tips.
There are many GAN variant that are more stable such as WGAN-DC and DRAGAN, and SAGAN. I need to use different architecture who might do better than DCGAN.
This article have tell us about what GAN is doing and tell us step by step on how to do it. After that, it tell us an interesting characteristic of its latent vector that show the data distribution learned by the generator. It shows us that It can form a data distribution.
The latent vector can be linear algebra operated. It can show us some interesting things such as addition of two latent vectors will combine the features each of these faces. It can also be manipulated to change the face based on the changed element in the latent vector.
Even so, our model still cannot make a face that can make us wondering whether that face is fake or not. It can still form an anime style faces though.
That’s it, my first experience on doing GAN. I got better on grasping what GAN is doing. I also want to explore my curiosity about what GAN has learned. There they are, It is really amazing what it actually do. The generator can map the random vector generated by normal random noise into a data distribution. It cluster the faces into designated data points.
On doing GAN, I actually run several model that I handcrafted. Well, they have failed miserably. One time I thought the model can success, but it go into mode collapse (the predicted face will be the same regardless of the latent vector). I found fchollet repository about DCGAN and follow its architecture.
Since this is my first time designing a GAN, I expect a lot feedback from everyone about this. Just point me the mistake that I’ve done since this is my first time doing it. Forgive me if the result is not that good. I just want to share my excitement on doing GAN. And share how to do it.
Even so, It is really fun and I want to experiment another variant of GAN such as WGAN-GP, DRAGAN, or SAGAN. I only skim a little what they are about and want to experiment it. Expect an article from doing these experiments 😃.
This meme actually picture this experiment 😆.
I welcome any feedback that can improve myself and this article. I’m in the process of learning on writing and learning about Deep Learning. I appreciate a feedback to make me become better. Make sure to give feedback in a proper manner 😄.
See ya in my next article!
See this GitHub repository:
github.com
Currently, I only provide IPython Notebook for training the GAN from scratch. Note if the model don’t output a face-shaped images on around 200 iterations, restart the training (run from ‘Create Model’ section).
Later, I will create a playground notebook to experiment on manipulating the latent variables.
medium.com
Thanks Renu Khandelwal for the awesome article.
https://www.cs.toronto.edu/~duvenaud/courses/csc2541/slides/gan-foundations.pdf
https://github.com/fchollet/deep-learning-with-python-notebooks
lilianweng.github.io
https://arxiv.org/pdf/1511.06434.pdf
medium.com
Thanks Jonathan Hui for the awesome article.