## 3 Method
### Training Method
#### 3.4.2 Adaptive Learning Rate Strategy
The derivative function cannot be directly set as the expected slope for two reasons: (1) \(V^{\hat{\kappa}}\) is just an evaluation value and not the actual optimal value, and it may be inaccurate and larger than 1; (2) \(V^{\hat{\kappa}}\) and \(R^{\hat{\kappa}}\) are the values at one step and cannot accurately represent training results. Thus, some improvements are made based on the derivative function. First, \(|V^{\hat{\kappa}}-R^{\hat{\kappa}}|\) is replaced with \(\frac{|V^{\hat{\kappa}}-R^{\hat{\kappa}}|}{V^{\hat{\kappa}}}\) to ensure the outcome is between 0 and 1. Then, \(V^{\hat{\kappa}}\) and \(R^{\hat{\kappa}}\) are, respectively, replaced with the mean values \(\overline{V^{\kappa}}\) and \(\overline{R^{\hat{\kappa}}}\). In addition, \(2\cdot(K-\hat{\kappa})\) is replaced with \(K\), resulting in that the expected slope can be larger in the early stage of training and be smaller in the later stage. In this way, the learning rate can have more chances to rise in the early stage but can keep decaying in the later stage. The final expected slope \(e^{\kappa}\) is formulated in Equation (35) through these improvements. \[c(\kappa,\hat{\kappa})=R^{\hat{\kappa}}+\frac{|V^{\hat{\kappa}}-R^{\hat{\kappa }}|}{2}\cdot[1-\cos(\frac{\kappa-\hat{\kappa}}{K-\hat{\kappa}}\pi)]. \tag{33}\]  
\[\frac{\text{d}c(\kappa,\hat{\kappa})}{\text{d}\kappa}=\frac{\pi\cdot|V^{\hat{ \kappa}}-R^{\hat{\kappa}}|}{2\cdot(K-\hat{\kappa})}\cdot\sin(\frac{\kappa- \hat{\kappa}}{K-\hat{\kappa}}\pi). \tag{34}\]  
\[e^{\kappa}=\frac{\pi\cdot|\overline{V^{\kappa}}-\overline{R^{\hat{\kappa}}}|} {K\cdot\overline{V^{\kappa}}}\cdot\sin(\frac{\kappa-\hat{\kappa}}{K-\hat{ \kappa}}\pi). \tag{35}\]  
Fourthly, the reward curve slope \(sl^{\kappa}\) is calculated according to the reward values of the previous ten steps through the least square method, which is formulated as outlined below:  
\[sl^{\kappa}=\frac{\sum_{\tau=1}^{10}(\tau-\overline{\tau})(R_{a}^{\kappa+ \tau-11}-\overline{R^{\hat{\kappa}}})}{\sum_{\tau=1}^{10}(\tau-\overline{\tau })^{2}}, \tag{36}\]  
where \(\overline{\tau}\) is the mean value of one to ten. Figure 6: Diagram of the reward curve and the reference cosine curves. In this diagram, a fold line is used to represent the reward curve. However, the actual reward curve is a fluctuating curve. The appropriate increase in the learning rate can accelerate the rise of the reward curve in the mid-term of training. A new cosine curve is set as the reference cosine curve whenever the learning rate increases. Eventually, the adaptive learning rate \(lr^{\kappa+1}\) of the next step is calculated as follows:  
\[lr^{\kappa+1}= \begin{cases}\gamma\cdot lr^{\kappa},&\kappa\text{ mod }10\
eq 0\\ fl(lr^{\kappa}),&\kappa\text{ mod }10=0^{\prime}\end{cases}  \tag{37}\]  
where  
\[fl(lr^{\kappa})= \begin{cases}\gamma\cdot lr^{\kappa},&sl^{\kappa}\geq e^{ \kappa}\\ lr^{\kappa}+\frac{|\
abla^{2}-\frac{R^{\kappa}}{|\
abla^{2}}}{\
abla^{\kappa}} \cdot(lr^{\kappa}-lr^{\kappa}),&sl^{\kappa}<e^{\kappa}.\end{cases}  \tag{38}\]  
As formulated in Equations (37) and (38), if the step number \(\kappa\) is the multiple of 10 and the reward curve slope \(sl^{\kappa}\) is less than the expected value \(e^{\kappa}\), the learning rate will increase slightly; otherwise, the learning rate \(lr^{\kappa}\) will decay by the decay rate \(\gamma\). As formulated in Equation (38), the increasing extent of the learning rate is determined by the difference between the evaluation value and the reward value and the difference between the last increased learning rate and the current one, and the learning rate cannot exceed the last increased learning rate. Through the adaptive learning rate strategy, the learning rate shows a downward trend but rises slightly at a few training steps, improving the exploration and convergence of the training algorithm.