## 3 The Method of Dynamic Adjustment Learning Rate and Convergence Proof
### Convergence Relation between Approximation Method and Dynamic Regulation Learning Rate  
In the process of algorithm updating, the incremental method instead of the Monte Carlo method [30; 31] is more in line with the requirements of dynamic programming. It is also more applicable to the controlled objects in the experimental environment.  
The process of the algorithm is described as follows: The interaction samples observed in \(c-1\) are: \(g_{1},g_{2},...,g_{c-1}\). Then it is easy to understand that the value function before \(c-1\) is: \(\bar{g}_{c-1}=\frac{1}{c-1}\sum_{i=1}^{c-1}g_{i}\). So we can get the value function at \(c\) observations is \(\bar{g}_{c}=\frac{1}{c}\sum_{i=1}^{c}g_{i}\). It can be proved in Equation (11):  
\[\bar{g}_{c}=\bar{g}_{c-1}+\frac{1}{c}\left(g_{c}-\bar{g}_{c-1}\right) \tag{11}\]which is the theoretical basis of Equation (3). At the same time, it is proved that the algorithm can converge to the value of the objective function. At this point, we may try another way to understand Equation (4). In practical reinforcement learning applications we usually do not record the number of sample interactions \(c\), but instead use a smaller value \(\alpha\) (\(\alpha\in(0,1)\)) instead of \(1/c\). This is because the default algorithm samples \(c\rightarrow\infty\). At this point, \(\frac{1}{c}\rightarrow\infty\). It can be seen that the learning rate \(\alpha\) in reinforcement learning is an approximation in the theoretical basis. This provides a possibility for us to dynamically adjust the value of learning rate.  
To prove that dynamically adjusting learning rate \(\alpha\) can guarantee the convergence of the algorithm, we introduce the important Robbins-Monro algorithm in the approximation algorithm. The algorithm reckons that: we need to pass a number of observations of some bounded random variable G \(g_{1},g_{2}...,g_{c-1}\) to estimate the expected value of the random variable \(q=E\left[G\right]\), the iterative formula can be used in Formula (12):  
\[q_{k}\gets q_{k-1}+\alpha_{k}\left(g_{k}-q_{k-1}\right)\;k=1,2,... \tag{12}\]  
to estimate the value of \(q\). Where the \(q_{0}\) initialization is random because it takes multiple iterations to approximate the exact value. Usually we set it to \(0\). The \(\alpha_{k}\) here is similar to the learning rate in the reinforcement learning update iteration. If the learning rate sequence satisfies \(\left\{\alpha_{k}:k=1,2...\right\}\) to ensure the convergence of the Robbins-Monro approximation algorithm, the following three conditions should be met:  
1. \(\alpha_{k}>=0,k=1,2,3,...\);
2. The condition under which an arbitrary point of convergence can be reached without any initial restriction: \(\sum_{k=1}^{\infty}\alpha_{k}^{2}<\infty\);
3. Finally, the convergence point can be reached without noise restriction.  
If the above three requirements are met, \(k\rightarrow\infty\rho_{k}\to q\). The temporal-difference of the reinforcement learning algorithm is equivalent to \(\alpha_{k}=\frac{1}{k}\;\left(k=1,2,...\right)\), such a learning rate sequence also meets all the conditions of the Robbins-Monro algorithm convergence, then the Robbins-Monro algorithm is used as the convergence basis of the sequential difference algorithm. When the reinforcement learning algorithm is used in a variety of vectors \(\alpha_{k}=\frac{1}{k}\;\left(k=1,2,...\right)\) (as long as these learning rates meet condition 3), the convergence of the sequential difference algorithm [27] can be guaranteed during the iterative update. Therefore, we can adopt different learning rates to adjust the algorithm based on the different advantages of learning rates at different stages.