Generating Anime Characters with StyleGAN2 | by Fathy Rashad | Towards Data Science

Generative Adversarial Network (GAN) is a generative model that is able to generate new content. The topic has become really popular in the machine learning community due to its interesting applications such as generating synthetic training data, creating arts, style-transfer, image-to-image translation, etc.
GAN consisted of 2 networks, the generator, and the discriminator. The generator will try to generate fake samples and fool the discriminator into believing it to be real samples. The discriminator will try to detect the generated samples from both the real and fake samples. This interesting adversarial concept was introduced by Ian Goodfellow in 2014. There are already a lot of resources available to learn GAN, hence I will not explain GAN to avoid redundancy.
I recommend reading this beautiful article by Joseph Rocca for understanding GAN.
towardsdatascience.com
The StyleGAN paper, “A Style-Based Architecture for GANs”, was published by NVIDIA in 2018. The paper proposed a new generator architecture for GAN that allows them to control different levels of details of the generated samples from the coarse details (eg. head shape) to the finer details (eg. eye-color).
StyleGAN also incorporates the idea from Progressive GAN, where the networks are trained on lower resolution initially (4x4), then bigger layers are gradually added after it’s stabilized. By doing this, the training time becomes a lot faster and the training is a lot more stable.
StyleGAN improves it further by adding a mapping network that encodes the input vectors into an intermediate latent space, w, which then will have separate values be used to control the different levels of details.
Why add a mapping network? One of the issues of GAN is its entangled latent representations (the input vectors, z). For example, let’s say we have 2 dimensions latent code which represents the size of the face and the size of the eyes. In this case, the size of the face is highly entangled with the size of the eyes (bigger eyes would mean bigger face as well). On the other hand, we can simplify this by storing the ratio of the face and the eyes instead which would make our model be simpler as unentangled representations are easier for the model to interpret.
With entangled representations, the data distribution may not necessarily follow the normal distribution where we want to sample the input vectors z from. For example, the data distribution would have a missing corner like this which represents the region where the ratio of the eyes and the face becomes unrealistic.
If we sample the z from the normal distribution, our model will try to also generate the missing region where the ratio is unrealistic and because there Is no training data that have this trait, the generator will generate the image poorly. Therefore, the mapping network aims to disentangle the latent representations and warps the latent space so it is able to be sampled from the normal distribution.
Additionally, Having separate input vectors, w, on each level allows the generator to control the different levels of visual features. The first few layers (4x4, 8x8) will control a higher level (coarser) of details such as the head shape, pose, and hairstyle. The last few layers (512x512, 1024x1024) will control the finer level of details such as the hair and eye color.
For full details on StyleGAN architecture, I recommend you to read NVIDIA's official paper on their implementation. Here is the illustration of the full architecture from the paper itself.
StyleGAN also allows you to control the stochastic variation in different levels of details by giving noise at the respective layer. Stochastic variations are minor randomness on the image that does not change our perception or the identity of the image such as differently combed hair, different hair placement and etc. You can see the effect of variations in the animated images below.
StyleGAN also made several other improvements that I will not cover in these articles such as the AdaIN normalization and other regularization. You can read the official paper, this article by Jonathan Hui, or this article by Rani Horev for further details instead.
When there is an underrepresented data in the training samples, the generator may not be able to learn the sample and generate it poorly. To avoid this, StyleGAN uses a “truncation trick” by truncating the intermediate latent vector w forcing it to be close to average.
The Ψ (psi) is the threshold that is used to truncate and resample the latent vectors that are above the threshold. Hence, with higher Ψ, you can get higher diversity on the generated images but it also has a higher chance of generating weird or broken faces. For this network Ψ value of 0.5 to 0.7 seems to give a good image with adequate diversity according to Gwern. Though, feel free to experiment with the threshold value.
I will be using the pre-trained Anime StyleGAN2 by Aaron Gokaslan so that we can load the model straight away and generate the anime faces. So, open your Jupyter notebook or Google Colab, and let’s start coding.
Note: You can refer to my Colab notebook if you are stuck
So first of all, we should clone the styleGAN repo.
$ git clone https://github.com/NVlabs/stylegan2.git
If you are using Google Colab, you can prefix the command with ‘!’ to run it as a command: !git clone https://github.com/NVlabs/stylegan2.git
Next, we would need to download the pre-trained weights and load the model. Make sure you are running with GPU runtime when you are using Google Colab as the model is configured to use GPU.
Now, we need to generate random vectors, z, to be used as the input fo our generator. Let’s create a function to generate the latent code, z, from a given seed.
Then, we can create a function that takes the generated random vectors z and generate the images.
Now, we can try generating a few images and see the results.
The function will return an array of PIL.Image. In Google Colab, you can straight away show the image by printing the variable. Here is the first generated image.
Let’s show it in a grid of images, so we can see multiple images at one time.
And then we can show the generated images in a 3x3 grid.
One of the nice things about GAN is that GAN has a smooth and continuous latent space unlike VAE (Variational Auto Encoder) where it has gaps. Hence, when you take two points in the latent space which will generate two different faces, you can create a transition or interpolation of the two faces by taking a linear path between the two points.
Let’s implement this in code and create a function to interpolate between two values of the z vectors.
Let’s see the interpolation results. You can see that the first image gradually transitioned to the second image.
Now that we’ve done interpolation. We can finally try to make the interpolation animation in the thumbnail above. We will use the moviepy library to create the video or GIF file.
When you run the code, it will generate a GIF animation of the interpolation. You can also modify the duration, grid size, or the fps using the variables at the top.
If you made it this far, congratulations! You have generated anime faces using StyleGAN2 and learned the basics of GAN and StyleGAN architecture.
Now that we have finished, what else can you do and further improve on? Here are a few things that you can do.
Other DatasetsObviously, StyleGAN is not limited to anime dataset only, there are many available pre-trained datasets that you can play around such as images of real faces, cats, art, and paintings. Check out this GitHub repo for available pre-trained weights. On the other hand, you can also train the StyleGAN with your own chosen dataset.
Conditional GANCurrently, we cannot really control the features that we want to generate such as hair color, eye color, hairstyle, and accessories. Conditional GAN allows you to give a label alongside the input vector, z, and hence conditioning the generated image to what we want. Alternatively, you can try making sense of the latent space either by regression or manually. If you want to go to this direction, Snow Halcy repo maybe be able to help you, as he done it and even made it interactive in this Jupyter notebook.
I’d like to thanks Gwern Branwen for his extensive articles and explanation on generating anime faces with StyleGAN which I strongly referred to in my article. I fully recommend you to visit his websites as his writings are a trove of knowledge. Additionally, check out ThisWaifuDoesNotExists website which hosts the StyleGAN model for generating anime faces and a GPT model to generate anime plot.
If you enjoy my writing, feel free to check out my other articles!
towardsdatascience.com
towardsdatascience.com
[1] Karras, T., Laine, S., & Aila, T. (2019). A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4401–4410).