# Enhanced Generative Adversarial Networks with Restart Learning Rate in Discriminator
## 4 Evaluation
### Result Analysis
However in the experiments of Table 2, what we want to prove is that our restart schedule could improve the performance in various GANs despite of different initial learning rates and training steps. So we have different parameter settings for experiments in Table 1 and experiments in the DCGAN model of Table 2.  
The learning rate for GANs training is usually around \(1\times 10^{-4}\) (or 0.0001), and a large learning rate always causes oscillation over training steps. In Table 2, when we set stable \(lr_{d}\) to be \(1\times 10^{-3}\) (or 0.001), the more steps we take, the worse results we obtain in DCGAN and SNGAN (i.e., column 1 of Table 2), which indicates the oscillation. Application of the restart learning rate in discriminator will avoid the oscillation problem effectively and brings more meaning for gradients for training, even though the maximum of Restart \(lr_{d}\) is also set to be \(1\times 10^{-3}\) (i.e., the column 2 of Table 2). When we set learning rate to be a relatively small value \(1\times 10^{-4}\), the restart learning rate could still promote the performance of DCGAN and SNGAN (shown in column 3 and column 4 of Table 2). In practice, \(lr_{d}=1\times 10^{-4}\) is usually our choice for training process. Thus, under this value, we run experiments of each GAN five times to calculate its average and standard deviation of FID to check the impact of our proposed method on stability. When we use stable \(lr_{d}\), WGAN-GP performed well in both cases (i.e., \(lr_{d}=1\times 10^{-3}\) and \(lr_{d}=1\times 10^{-4}\)) which means WGAN-GP stabilizes training process efficiently while the use of restart \(lr_{d}\) in it lead to worse results (i.e., line 5 and line 6 of Table 2). We think there are two reasons: (1) WGAN-GP has already stable the training, hence periodical restart learning rate brings interference (can be indicated by the standard deviation of FID). (2) The reason why the restart learning rate is effective in DCGAN and SNGAN is that it avoids the disappearance of \(D\_loss\), while the loss in the discriminator of WGAN-GP depends on Earth Mover (EM) distance which is continuous and differentiable till optimality. Bold numbers in Table 2 means the best results in three GANs respectively. WGAN-GP with stable \(lr_{d}\)(FID = 59.245) and DCGAN with restart \(lr_{d}\)(FID = 58.735) have obtained the highest  
Figure 3: Visual results trained with Restart \(lr\) of DCGAN on CIFAR-10 and Align Celeba. (**a**) DCGAN trained with Stable \(lr\) on CIFAR-10. (**b**) DCGAN trained with Restart \(lr\) on CIFAR-10. (**c**) DCGAN trained with Stable \(lr\) on Align Celeba. (**d**) DCGAN trained with Restart \(lr\) on Align Celeba. comparable performance in 40k iterations. Table 2 shows that restart \(lr_{d}\) is beneficial to DCGAN and SNGAN (with low average FID and comparable standard deviation value) but is not available for WGAN-GP. Exploring the deep reason why restart learning rate is useful or useless in various GAN frameworks is one of our future research directions. Table 3 records the time spent for DCGAN and WGAN-GP to reach the best results (shown in bold) in Table 2. As we have mentioned in the above paragraph, DCGAN with this restart \(lr_{d}\) and WGAN-GP with stable \(lr_{d}\) have comparable results in 40k iterations while DCGAN with this restart \(lr_{d}\) spent less time. The reason why WGAN-GP with stable \(lr_{d}\) cost more time is that it needs to calculate penalty in every iteration. We also note that the restart schedule does not bring too many calculations (compare line 1 and line 2 of Table 3). All in all, the restart learning rate improves the generation performance of DCGAN without too much computational complexity and it is meaningful for GANs training. DCGAN with restart \(lr_{d}\) is a reasonable choice for GANs generation work. We also compare our method with the Two Time-Scale Update Rule [24]. In Figure 4, 'Original' means the DCGAN model in [2], 'TTUR' means DCGAN trained by a Two Time-Scale Update Rule [24], and 'Restart' means our method proposed in this paper.