Generating Anime Characters with StyleGAN2
By: Fathy Rashad
Tags: Deep Learning, Artificial Intelligence, Machine Learning, Art, Data Science

Generating Anime Characters with StyleGAN2

Learn how to generate this cool anime faces interpolation

Generated StyleGAN Interpolation [Image by Author]

Generative Adversarial Network

Generative Adversarial Network (GAN) is a generative model that is able to generate new content. The topic has become really popular in the machine learning community due to its interesting applications such as generating synthetic training data, creating arts, style-transfer, image-to-image translation, etc.

GAN Architecture [Image by Author]

GAN consisted of 2 networks, the generator, and the discriminator. The generator will try to generate fake samples and fool the discriminator into believing it to be real samples. The discriminator will try to detect the generated samples from both the real and fake samples. This interesting adversarial concept was introduced by Ian Goodfellow in 2014. There are already a lot of resources available to learn GAN, hence I will not explain GAN to avoid redundancy.

I recommend reading this beautiful article by Joseph Rocca for understanding GAN.

StyleGAN2

The StyleGAN paper, “A Style-Based Architecture for GANs”, was published by NVIDIA in 2018. The paper proposed a new generator architecture for GAN that allows them to control different levels of details of the generated samples from the coarse details (eg. head shape) to the finer details (eg. eye-color).

StyleGAN also incorporates the idea from Progressive GAN, where the networks are trained on lower resolution initially (4x4), then bigger layers are gradually added after it’s stabilized. By doing this, the training time becomes a lot faster and the training is a lot more stable.

Progressive Growing GAN [Source: Sarah Wolf]

StyleGAN improves it further by adding a mapping network that encodes the input vectors into an intermediate latent space, w, which then will have separate values be used to control the different levels of details.

StyleGAN Generator Architecture [Image by Author]

Why add a mapping network?

One of the issues of GAN is its entangled latent representations (the input vectors, z). For example, let’s say we have 2 dimensions latent code which represents the size of the face and the size of the eyes. In this case, the size of the face is highly entangled with the size of the eyes (bigger eyes would mean bigger face as well). On the other hand, we can simplify this by storing the ratio of the face and the eyes instead which would make our model be simpler as unentangled representations are easier for the model to interpret.

With entangled representations, the data distribution may not necessarily follow the normal distribution where we want to sample the input vectors z from. For example, the data distribution would have a missing corner like this which represents the region where the ratio of the eyes and the face becomes unrealistic.

If we sample the z from the normal distribution, our model will try to also generate the missing region where the ratio is unrealistic and because there Is no training data that have this trait, the generator will generate the image poorly. Therefore, the mapping network aims to disentangle the latent representations and warps the latent space so it is able to be sampled from the normal distribution.

Additionally, Having separate input vectors, w, on each level allows the generator to control the different levels of visual features. The first few layers (4x4, 8x8) will control a higher level (coarser) of details such as the head shape, pose, and hairstyle. The last few layers (512x512, 1024x1024) will control the finer level of details such as the hair and eye color.

Variations in coarse level details (head shape, hairstyle, pose, glasses) [Source: Paper]

Variation in fine level details (hair color) [Source: Paper]

For full details on StyleGAN architecture, I recommend you to read NVIDIA's official paper on their implementation. Here is the illustration of the full architecture from the paper itself.

Stochastic Variation

StyleGAN also allows you to control the stochastic variation in different levels of details by giving noise at the respective layer. Stochastic variations are minor randomness on the image that does not change our perception or the identity of the image such as differently combed hair, different hair placement and etc. You can see the effect of variations in the animated images below.

Coarse Stochastic Variation [Source: Paper]

Fine Stochastic Variation [Source: Paper]

StyleGAN also made several other improvements that I will not cover in these articles such as the AdaIN normalization and other regularization. You can read the official paper, this article by Jonathan Hui, or this article by Rani Horev for further details instead.

Truncation Trick

When there is an underrepresented data in the training samples, the generator may not be able to learn the sample and generate it poorly. To avoid this, StyleGAN uses a “truncation trick” by truncating the intermediate latent vector w forcing it to be close to average.

The 𝚿 (psi) is the threshold that is used to truncate and resample the latent vectors that are above the threshold. Hence, with higher 𝚿, you can get higher diversity on the generated images but it also has a higher chance of generating weird or broken faces. For this network 𝚿 value of 0.5 to 0.7 seems to give a good image with adequate diversity according to Gwern. Though, feel free to experiment with the threshold value.