Tips On Training Your GANs Faster and Achieve Better Results
By: Tushar Mittal
Tags: Gans, Machine Learning, Data Science, Deep Learning, Generative Adversarial

Input Noise:

The generator takes in noise as input and to my surprise, the size of the noise vector also affected the results. Initially, I was using 100 as the size of noise vector but when I changed it to 128 and 256 my results got better and the training speed also increased.

Batch Size:

The biggest mistake that I was doing initially when I started was to use a very big value for batch size. While training your GAN use a batch size smaller than or equal to 64. Using a bigger batch size might hurt the performance because during the initial training the discriminator might get a lot of examples to train on and it might overpower the generator, which would have a negative effect on training. For me, using batch sizes of 8 and 16 worked out best.

Loss Functions:

Loss function plays a huge role in the training process and researchers have been experimenting with various custom loss functions to be used in GAN training. I started out with a simple binary_crossentropy loss function which gave out decent results. Then I tried out the Wasserstein loss function which is used in the WGAN paper and this slightly improved the results. During the contest, most of the participants seemed to use the RaLS (Relativistic Average Least Square) Loss function, but despite many efforts, I could not get it working in Tensorflow. The takeaway from this point is trying out different loss functions can help.

Over Training might Hurt:

While training your network you might be tempted to train your GAN for as long as you possibly can, but overtraining might degrade the quality of the generated samples. Most of the contest participants reported that the best images were produced in between the training loops and not towards the end. In the contest the MiFID (Memorization-informed Fréchet Inception Distance) metric was used to evaluate the generated samples, so I monitored this metric and used Early Stopping to achieve the best results.

Mode Collapse:

Most of the times the data we have is multimodal, which means it has multiple modes where each mode represents data which share similar features.

A very common problem while training GANs is Mode Collapse in which the Generator produces images belonging to only a few modes of the data and ignores all other modes, hence the generated images are very similar and lack variety. There are a few suggested ways to solve this problem but I could not try all of them out. Experimenting with the Learning Rate of both Generator and Discriminator helped me to overcome this problem. So while training your GANs you should monitor the generated samples to see if your model is suffering from this problem.

Other Tips:

There is also this GAN Hacks repository which enlists many other techniques that you can use and most of them helped me to converge faster and achieve better results. Some more components that you can experiment with:

The architecture of your GAN is the most important thing. You can take inspiration from various famous GANs and implement your own version of it. Although most of the GANs use ADAM optimizer for training, don’t be afraid to try out the good old Gradient Descent optimizer. For the activation functions, instead of using the simple ReLU you can try out its other variants such as Leaky ReLU, ELU (Exponential Linear Unit) or PReLU (Parametric Rectified Linear Unit). You can also use various image augmentation techniques while training.

During the training of various GANs, I also found out that they were very sensitive to the changes and most of the times even a small change anywhere changed the results drastically. Despite the extensive research going on in the field of Deep Learning, many topics still remain a black box. I realized this during the contest, most of the things that I was implementing and testing, I could not find any reasonable explanation for most of them. That is why the configurations that worked for me are not guaranteed to work you as well but these are some of the topics that you can focus on while training your GANs because these things helped me to achieve good results. The configurations of all these features will depend on the type of problem that you working on and also the data you have.