## 4 Experiment
### Convergence and Rationality Are Combined to Determine the Learning Rate  
In this section, we further select the appropriate learning rate interval. Different from the traditional simple setting, we not only consider the accumulated income of the algorithm as the negative feedback to form a closed loop. We also consider adding convergence as another condition for the negative feedback loop. In machine learning and deep neural network training, the error value is usually used to determine whether the algorithm converges or not [34; 35]. Although unlike traditional supervised learning, reinforcement learning does not have a deterministic target value (supervised learning), the purpose of the algorithm is to approximate the optimal value function. Therefore, in the TD algorithm, the TD error can be similar to the error used in the network parameter update in supervised learning. Therefore, it can be used as a measure of convergence performance of the algorithm [23]. By comparing the typical values of controller returns and convergence results of learning rate training of different orders, it can be seen that even when the return values are close, the convergence performance of the algorithm is still quite different. As can be seen from Figure 4, when the learning rate is 0.001, the algorithm can quickly obtain better results, but in the later stage of interaction with the environment, the performance stability of the algorithm is poor. This may be because the learning rate is more sensitive to the gradient in the later stage of the algorithm. The mean square error when the learning rate is 0.0001 and the convergence effect is good can be seen in Figure 5.  
At the same time, when the learning rate is 0.0001, the convergence speed of the algorithm is slow. However, it can be seen that once the algorithm converges, the error was better in a large number of experiments. However, in most cases in our experiment, when the learning rate is 0.0001, the typical variation value of the mean square error is shown in Figure 6. \begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline
**Learning Rate** & **1** & **2** & **3** & **4** & **5** & **6** & **7** & **8** & **9** & **10** & **Average** \\ \hline \(lr=0.1\) & \(-198.52\) & \(-197.91\) & \(-200\) & \(-199.64\) & \(-200\) & \(-200\) & \(-200\) & \(-200\) & \(-200\) & \(-200\) & \(-199.607\) \\ \(lr=0.01\) & \(-170.57\) & \(-153.68\) & \(-168.26\) & \(-168.82\) & \(-159.43\) & \(-169.99\) & \(-172.21\) & \(-183.17\) & \(-166.16\) & \(-182.69\) & \(-169.498\) \\ \(lr=0.001\) & \(-118.23\) & \(-116.03\) & \(-114.3\) & \(-120.0\) & \(-124.69\) & \(-123.25\) & \(-120.29\) & \(-126.11\) & \(-110.97\) & \(-130.76\) & \(-120.463\) \\ \(lr=0.0001\) & \(-119.94\) & \(-123.38\) & \(-125.16\) & \(-130.38\) & \(-126.16\) & \(-129.12\) & \(-110.19\) & \(-122.44\) & \(-117.18\) & \(-118.78\) & \(-122.273\) \\ \(lr=0.0001\) & \(-200\) & \(-200\) & \(-200\) & \(-200\) & \(-179.53\) & \(-114.09\) & \(-200\) & \(-200\) & \(-200\) & \(-200\) & \(-189.362\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Comparison of optimal return values obtained by different learning rates. Figure 4: General performance of the mean square error convergence with a learning rate of 0.001. It can be seen that when the learning rate is at the magnitude of 0.0001, although the algorithm can have a better return value, it is difficult for the algorithm to converge in the later stage of training. It makes it hard to stabilize the result. At the same time, we also conducted a test in this order of the learning rate. Comparatively speaking, the learning rate at 0.0002 can appear to have more stable convergence training results. Therefore, when we dynamically adjust the learning rate, we add 0.0002 to maintain the stability of the learning rate at this order. Our ultimate goal is that the algorithm can approach the optimal result quickly in the early stage and maintain stable convergence results in the later stage.