## 3 The Method of Dynamic Adjustment Learning Rate and Convergence Proof
### Mathematics Model and Convergence of Temporal-Difference
 \tag{5}\]  
Suppose it is also true for n:  
\[\underset{\left(s,a\right)}{\max}\left|E\left[\hat{Q}_{k+1}^{\left(n+1\right) }\left(s,a\right)\right]-Q^{\pi}\left(s,a\right)\right|<=\gamma\underset{ \left(s,a\right)}{\max}\left|\hat{Q}_{k+1}^{\left(n\right)}\left(s^{\prime},a ^{\prime}\right)-Q^{\pi}\left(s^{\prime},a^{\prime}\right)\right| \tag{6}\]  
The following proof of \(\hat{Q}_{k+1}^{\left(n+1\right)}\left(s,a\right)\) also set up:\[\hat{Q}\left(s,a\right) =R_{t+1}+\gamma\sum_{a^{\prime}\in A}\pi\left(s_{t+1},a^{\prime} \right)\left(\hat{Q}_{k}\left(s_{t+1},a^{\prime}\right)\right) \tag{7}\] \[\cdot\left(1-I\left(a^{\prime},a_{t+1}\right)+I\left(a^{\prime},a _{t+1}\right)\hat{Q}_{k}^{n}\left(s_{t+1},a^{\prime}\right)\right)\]  
where \(I\left(a^{\prime},a_{t+1}\right)\) is an indicator function [27]:  
\[I\left(a^{\prime},a_{t+1}\right)= \begin{cases}1\ if\ a^{\prime}=a_{t+1}\\ 0\ else\end{cases}  \tag{8}\]  
\[ \begin{split}\max_{\left(s,a\right)}\left|\mathbb{E}\left[\hat{Q }_{k+1}^{\left(n+1\right)}\left(s,a\right)\right]-Q^{\pi}\left(s,a\right) \right|&=\max_{\left(s,a\right)}\left|\begin{array}{c}R_{s}^{\alpha}+ \gamma\sum_{a^{\prime}}\ p_{s^{\prime}}^{\alpha}\sum_{a^{\prime}}\pi\left(s^ {\prime},a^{\prime}\right)\cdot\left\{E\left[\left(\begin{array}{c}1\\ -I\left(a^{\prime},a_{t+1}\right)\end{array}\right)\cdot\hat{Q}_{k}^{\left(n \right)}\left(s^{\prime},a^{\prime}\right)\right]\right\}\\ +I\left(a^{\prime},a_{t+1}\right)\hat{Q}_{k}^{\left(n\right)}\left(s^{\prime},a^{\prime}\right)-R_{s}^{\alpha}-\gamma\sum_{a^{\prime}}p_{s^{\prime}}^{ \alpha}\sum_{a^{\prime}}\pi\left(s^{\prime},a^{\prime}\right)Q^{\alpha}\left( s^{\prime},a^{\prime}\right)\\ \left.\sum_{a^{\prime}}\ p_{s^{\prime}}^{\alpha}\sum_{a^{\prime}}\pi\left(s^ {\prime},a^{\prime}\right)\left\{E\left[\left(1-I\left(a^{\prime},a_{t+1} \right)\right)\cdot\left(\begin{array}{c}\hat{Q}_{k}^{\left(n\right)}\left( s^{\prime},a^{\prime}\right)\\ -Q^{\pi}\left(s^{\prime},a^{\prime}\right)\end{array}\right)\right]\right\} \\ +I\left(a^{\prime},a_{t+1}\right)\cdot\left(\hat{Q}_{k}^{\left(n\right)}\left( s^{\prime},a^{\prime}\right)-Q^{\pi}\left(s^{\prime},a^{\prime}\right)\right) \end{split} \right|\\ <=\gamma\max_{\left(s,a\right)}\left|\hat{Q}_{k}^{\left(n\right)}\left(s,a \right)-Q^{\pi}\left(s,a\right)\right|\end{split} \tag{9}\]  
**Theorem 2**: _Under the MDP structure, for any initial \(Q\left(s,a\right),\forall\gamma\in\left(0,1\right)\), \(\forall\left(s,a\right)\in s\times a\). \(Q\) is updated according to Equation (3). So \(\hat{Q}_{t}^{\left(n\right)}\left(s,a\right)\) converges to \(\hat{Q}^{\pi}\left(s,a\right)\) with probability._  
_In fact, if \(\ pi\left(s_{t+1}\text{ in Theorem }1,a^{\prime}\right)\) satisfies Formula (10):_  
\[\pi\left(s_{t+1},a^{\prime}\right)= \begin{cases}1&a^{\prime}=a_{t+1};\\ 0&else.\end{cases}  \tag{10}\]  
_This is the special case of the algorithm in Theorem 1, and the convergence of the algorithm can be proved by the convergence of Theorem 1._  
**Theorem 3**: _If \(Q\left(\sigma\right)\)meets the conditions:_  
1. _Finite state space;_
2. \(\sum_{i=1}^{\infty}\alpha_{i}=\infty,\sum_{i=1}^{\infty}\alpha^{2}<+\infty\)_._  
_Then the \(Q_{t}\left(s,a\right)\)generated by the above iteration converges probabilistically to \(Q^{\pi}\left(s,a\right)\)._  
**Proof. **\(Q\left(\sigma\right)\) is a convex combination of Theorem 1 and Theorem 2 algorithms. It can be obtained that \(Q\left(\sigma\right)\)generated \(Q_{t}\left(s,a\right)\) converges probabilistically \(Q^{\pi}\left(s,a\right)\).