# Enhanced Generative Adversarial Networks with Restart Learning Rate in Discriminator
## 4 Evaluation
### Result Analysis
\(\eta=0.8\) exhibited the optimal result as shown in the third row of Table 5 (corresponding FID = 63.793). In conclusion, we can see that values slightly less than 1.0 are suitable for parameter \(\eta\). Figure 5: \(D\) loss line with different \(\beta\). Figure 6: \(D\) loss line with different \(\eta\). \(\alpha\) is the minimum value that the learning rate can decay to (shown in line 5 of Algorithm 1). When \(\alpha\) is 0.0, it means there is no baseline, and the learning rate could decrease to zero to provide no gradient for updating (it is obvious that we need gradients for training GANs). In this way, we set its values between 0.0 and 1.0 to be the baseline of learning rate to explore its impact. When \(\alpha\) is slightly greater than 0.0, the corresponding generated results are more improved than those of \(\alpha=0\) (i.e., row 2, 3, 4 in Table 6). \(\alpha=0.2\) has obtained the optimal performance and the representative line in Figure 7 shows the highest \(D\) loss. Thus, the following conclusions can be inferred: appropriate increase \(D\) loss can effectively prevent the disappearance of \(D\) loss, so as to make GAN generate improved results. However, excessive values can lead to an increase in FID scores (i.e., row 5, 6 in Table 6). Additionally, we also noted that the value \(\alpha=0.5\) makes \(D\) loss nearly disappearing (orange line in Figure 7). We conjecture that it is because the range (1.0 to 0.5) of the learning rate is limited, which is contrary to our original intention for restart. All in all, we consider \(\alpha=0.2\) to be most suitable for our method. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & **Line** & \(\beta\) & \(\eta\) & \(\alpha\) & \(P\) & **FID** \\ \hline
1 & Blue & 1.0 & 1.0 & 0.0 & 40 & 69.448 \\
2 & & 1.0 & 1.0 & 0.1 & 40 & 67.987 \\
3 & Green & 1.0 & 1.0 & 0.2 & 40 & **65.413** \\
4 & & 1.0 & 1.0 & 0.3 & 40 & 66.547 \\
5 & Orange & 1.0 & 1.0 & 0.5 & 40 & 70.681 \\
6 & & 1.0 & 1.0 & 0.7 & 40 & 82.516 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: FID results in terms of different \(\alpha\). Bold denotes the smallest distance. Figure 7: \(D\) loss line with different \(\alpha\). \(P\) is updated by \(P=P*\beta\) in the algorithm (line 6 in Algorithm 1). \(P\) is the initial number of steps in the first cycle. As was previously discussed, \(\beta\) to be 1.0 or slightly greater than 1.0 is a moderate choice for training GANs. Hence too small values of \(P\) (i.e., rows 1, 2, and 3 in Table 7) provide limited steps to find potential optima. When the value of \(P\) is too large, the period in the loss curve is too long, and the sudden increase of learning rate at the beginning of the next period will bring too high fluctuations to the training process (the orange line in Figure 8). Heavy fluctuation makes the optimization of the previous cycle meaningless which is shown with underperformed generated results (i.e., rows 5 and 6 in Table 7). When the initial value of \(P\) is set to 40, it allows the restart learning rate to exercise its full performance and exhibits the satisfactory results (the blue line in Figure 8 and the row 4 in Table 7). Overall, appropriate parameter selection is essential for demonstrating the functionality of our new method. Especially, it is shown that appropriate tuning of \(\eta\) and \(\alpha\) significantly improves the performance. Through various experiments, we found the setting of parameters (\(\beta=1.0\), \(\eta=0.8\), \(\alpha=0.2\), \(P=40\)) which gives satisfiable results. It also could be further explained that remarkable \(D_{loss}\) performance corresponds to low FID results. Restart learning prevents the fast disappearance of \(D_{loss}\)(Figure 1a) and the rapid growth of \(G_{loss}\) (Figure 1b). As explained above, preventing the rapid decline in the loss of the discriminator can effectively improve the performance of GAN, and restarting learning is an effective way for achieving the above-mentioned functions.