# Enhanced Generative Adversarial Networks with Restart Learning Rate in Discriminator
## 4 Evaluation
### Result Analysis
For fairness, the learning rate and updates settings of Original and TTUR are the same as settings in Table 1 of [24]. In our proposed method ('Restart' in Figure 4), the learning rate of the generator is the same as 'Original' while the maximum learning rate of the discriminator is also equal to \(1\times 10^{-4}\). In Figure 4, compared with the round-trip fluctuation of DCGAN with original learning rates, the FID results indicate that the DCGAN trained with TTUR decreases steadily and outperforms the original DCGAN in 80k updates. However, the DCGAN with our method obtained the highest performance and converges faster than the other two methods. In order to analyze our method in detail, we have further explored the influence of four important parameters in the method on the experiment results and which are shown in the following parts. The four parameters are \(\beta\) (cycle length control coefficient), \(\eta\) (peak value control coefficient), \(\alpha\) (the baseline of amplitudes), and \(P\) (the number of steps in the first cycle), respectively. As we have discussed in Section 2.2, the line of Figure 1a shows the convergence of \(D\) which will influence the final output. Therefore, we have investigated the influence of our method in two ways: (1) the convergence of loss of \(D\) and (2) the result in terms of FID. Besides that, we consider the modification of the parameter settings that can prevent the fast convergence of \(D\) and ensure the effectiveness during the training process. \begin{table}
\begin{tabular}{c c c} \hline  & **GANs** & **Time/s** \\ \hline
1 & DCGAN with Stable \(lr_{d}\) & 479.2910 \\ \hline
2 & DCGAN with Restart \(lr_{d}\) & 495.3721 \\ \hline
3 & WGAN-GP & 662.9115 \\ \hline \end{tabular}
\end{table} 
Table 3: Time spent in 40k iterations on CIFAR-10. In the ablation study of four parameters (\(\beta\), \(\eta\), \(\alpha\), and \(P\)), we always explore the impact of different values of a parameter when the other three parameters remain unchanged. Based on the analysis of generated results (FID score), we select three representative values to see their impact on the discriminator loss curves. Table 4 gives different values related to the cycle length control coefficient \(\beta\) (row 6 in Algorithm 1) and Figure 5 shows three corresponding \(D\) loss lines. \(\beta\) defines the multiple of the current period length relative to the previous one. According to the update formula of period length (\(P=P*\beta\)), \(\beta\) with a value less than 1.0 (i.e., row 1 in Table 4) shortens the length of a period. When the training process can not run properly, 'NaN' situation can occur (pink line in Figure 5). 'NaN' represents that the program produces none value. Conversely, a large value (i.e., row 5 and 6 in Table 4) may cause heavy disturbance as shown in the green line of Figure 5 due to the sudden increase of the learning rate in the next period. However, this disturbance will not lead to heavy worse results (i.e., FID score in row 5) because of fast convergence. As shown in Table 4, restart learning rate with \(\beta=1.0\) performs most effectively (i.e., row 2). However, at the same time, we also note that when we take values between 1.0 and 2.0 for \(\beta\), the differences among them are small (i.e., row 3, row 4, and row 5). Even if \(\beta\)  
\(\eta\) is the multiple of the current peak value relative to the previous one (note line 5 in Algorithm 1). We also set different values to parameter \(\eta\) (Table 5). We have observed that \(\eta=1.1\) makes the peak value cumulative as the epoch increases (orange line in Figure 6), and the increasing fluctuation is not conducive to the convergence of the whole training process. Additionally, larger values lead to worse results (compare row 5 and row 6 in Table 5). However, values set to 1.0 or slightly less than 1.0 are beneficial for generated results (i.e., rows 2,3, and 4). For example, blue line and red line in Figure 6 provide more gentle restart for training process.