## 3 Method
### Training Method
#### 3.4.2 Adaptive Learning Rate Strategy  
In the training process of the neural network, the learning rate is a crucial hyperparameter that determines the update stride of network parameters. A large learning rate may lead to easy divergence, while a small learning rate may lead to slow convergence [41]. In the prior experiments, the conventional fixed learning rates and exponential learning rates failed to balance the early exploration and the later convergence, and a slightly higher learning rate could bring the reward curve to a higher level but make it difficult to converge. Therefore, the learning rate should be adjusted dynamically, and its appropriate decay and increase can make the reward curve converge to a higher level. To improve the training effect of the actor network, an adaptive learning rate strategy is proposed that dynamically adjusts the learning rate according to the training performance. The reward curve can intuitively reflect the training performance. However, the neural network is trained on the four kinds of samples with significantly different optimal rewards, resulting in the reward curve appearing as a step shape. Hence, the learning rate cannot be adjusted according to the reward value, but the trend of the reward curve can be selected as the indicator for adjusting the learning rate. The adjustment process of the adaptive learning rate contains two steps: first, the learning rate overall decays by a fixed decay rate to ensure the convergence of the reward curve; then, for every ten training steps, if the trend of the reward curve does not reach the expected value, the learning rate will increase slightly to help the reward curve reach a higher level. After the learning rate is initialized or increases, the reward curve may be at a lower level in the early stage, but it is expected to rise rapidly in the middle stage of the following training and converge stably in the later stage, similar to a cosine wave of half a cycle. Thus, a cosine curve is used as the reference curve, and the expected value is calculated according to its slope. Details of the adaptive learning rate strategy are elaborated below. Firstly, some variables need to be defined to describe the adaptive learning rate: at training step \(\kappa\), given a batch of samples, (1) \(lr^{\kappa}\) is the learning rate of the actor network; (2) \(R^{\kappa}=\sum_{b=1}^{BS}R_{b}^{\kappa}\) is the average reward of this batch of solutions, denoting the reward at the current step; (3) given \(\{R^{\kappa+\tau-11}|\tau=1,\cdots,10\}\) which is the reward set of the previous ten steps, \(\overline{R^{\kappa}}\) is its mean value; (4) \(V^{\kappa}=\sum_{b=1}^{BS}V_{b}^{\kappa}\) is the average evaluation value of this batch of samples, denoting the evaluation value at the current step; (5) \(\overline{V^{\kappa}}\) is the mean value of \(\{V^{\kappa+\tau-11}|\tau=1,\cdots,10\}\); (6) \(sl^{\kappa}\) is the slope of the reward curve; (7) \(e^{\kappa}\) represents the expected slope; (8) if \(sl^{\kappa}<e^{\kappa}\), \(\kappa\) will be recorded as \(\hat{\kappa}\) whose initial value is zero. (9) \(K\) is the total training steps; and (10) \(\gamma\) is the decay rate. Secondly, the trend of the reward curve is expected to be close to that of a cosine curve, as shown in Figure 6. After ten training steps, the slope of the reward curve is compared with the expected value, which is gained according to the slope of the reference cosine curve. If the slope of the reward curve is less than the expected value, the learning rate of the actor network will increase slightly; otherwise, the learning rate will keep decaying at the following ten training steps. Once the learning rate increases, the reward curve is expected to rise like a new cosine curve at the following training steps. Thirdly, the reference cosine curve function \(c(\kappa,\hat{\kappa})\) is formulated in Equation (33), and its derivative is formulated in Equation (34).