# Enhanced Generative Adversarial Networks with Restart Learning Rate in Discriminator
## 3 DCGAN with Restart Learning Rate
### Restart Learning Rate in Discriminator  
In GANs, if the convergence speed of the discriminator is too fast, then it can lead to the failure of model training. If the loss value of the discriminator is closer to 0, then the loss value neither guides the generator to find the local minimum nor explore more areas. This affects both the quality and the diversity of the generated results. Therefore, we introduce the restart learning rate which can be reset periodically during the learning process. This method does not only prevent the loss value of the discriminator to be closer to 0, but it also provides an effective gradient for \(G\) during the back-propagation procedure. Besides that, the sudden increase in the learning rate at the beginning of each cycle is conducive to the model to explore more areas and increases the diversity of results. Overall, applying the restart learning algorithm in the discriminator can overcome the generator's problem and improve the quality and diversity of the results. In the practical experiments, we must admit that a sudden increase in the restart learning rate in the next cycle will cause significant interference to the training process. In the early stage of training, the advantages of this interference outweigh the disadvantages, similar to algorithms that add noise to the training samples. However, during the training process, the model gradually finds an area near the optimal point in a flat place, and then this interference will cause shocks, which gives more disadvantages than benefits. Therefore, we use the parameter \(\eta\) to gradually reduce the peak learning rate of each cycle to relieve the shock at the end of training.  
\[lr_{t+1}=lr_{t}*\eta*\{[1+cos(\pi*steps/restart\ decay\ steps)]*(1-\alpha)+\alpha\} \tag{7}\]  
\[restart\ decay\ steps=\beta*i*decay\ steps \tag{8}\]  
where \(i\) is the index of number of period, and \(\beta\) is a parameter used to increase (i.e., 2) or decrease (i.e., 0.5) the cycle length. Noted that \(\pi*steps/restart\ decay\ steps\) in Equation (7) returns only integer value.  
We provide the pseudo-code of our proposed method in Algorithm 1. The inputs include the number of updates in the first cycle denoted as \(P\), and the number of all updates denoted as \(t\). We initialize hyper-parameters used in the restart learning rate method, such as the original learning rate \(lr_{d}\), an amplitude control parameter \(\eta\), and the baseline of amplitude \(\alpha\). Parameter \(\theta\) in the DCGAN network is initialized as 0. We use ADAM optimizer [22] during the back-propagation procedure. Noted that the restart learning rate can be applied to any optimizer. In ADAM, \(\widehat{m}\) is 'bias-corrected first moment estimate' and \(\widehat{v}\) is 'bias-corrected second raw moment estimate'.  
For each step, the discriminator is updated by the ADAM optimizer. In the meanwhile, the restart learning rate, introduced at line 5, will be updated at every step. The steps of the first cycle are determined by the initial \(P\). After that, the length of the current cycle will be determined by parameter \(\beta\) (at line 6). Note that the cyclical restart learning rate does  
Figure 2: Restart Learning Rate: (a) learning rate with different restart cycles; (b) learning rate with different restart peaks; (c) Learning rate with different minimum learning rate decays.  
not restart completely (i.e., not from zero) in each cycle, but its update is based on the last learning rate from the previous cycle.  
```
0: first decay steps \(P\), steps \(t\)
0:\(lr_{d}\)
1: Initialize learning rates \(lr_{d}=0.001\), \(\eta=0.9\alpha=0.5\), weights \(\theta=0\)
2:The procedure of updating \(lr_{d}\):
3:for steps \(t=0,1,2,\cdots\)do
4:\(\tau\gets t\mod P\)
5:\(lr_{d_{t+1}}=lr_{d_{t}}*\eta*\{[1+cos(\pi*\tau)]*(1-\alpha)+\alpha\}\)
6:\(P=P*\beta\)
7:Backpropagation with ADAM Optimizer
8:\(\theta_{t+1}=\theta_{t}-lr_{d_{t+1}}*\widehat{m}_{t+1}/\sqrt{\widehat{v}_{t+ 1}+\varepsilon}\)
9:End for
```  
**Algorithm 1** Restart Learning Rate in discriminator of DCGAN