## 3 The Method of Dynamic Adjustment Learning Rate and Convergence Proof
### Mathematics Model and Convergence of Temporal-Difference  
The temporal-difference algorithm is one of the most important algorithms in reinforcement learning. This algorithm works well in both model-based and non-model-based environments. The algorithm needs to sample the trajectory [27] generated by the policy \(\pi\):  
\[S_{0},A_{0},R_{1},S_{1},...,S_{T-1},A_{T-1},R_{T},S_{T}\]  
This sequence is also called eligibility traces [28]. Compared with the Monte Carlo method [28], which requires a complete eligibility trace, the temporal-difference method only needs to select a section of qualification trace between the existing state and a certain state to update the strategy. This provides the conditions for the online learning of algorithms. Let us take the \(Q\left(\sigma\right)\) algorithm for example. The parameter \(\sigma\) represents the percentage of sampling during the operation of the algorithm. Because one-step iteration is widely used in online learning, we take one-step iteration as an example. \[G_{t}^{\left(1\right)}=r_{t+1}+\gamma\left[\sigma Q_{t}\left(s_{t+1},a_{t+1} \right)+\left(1-\sigma\right)\overline{Q_{t+1}}\right] \tag{3}\]  
When \(\sigma=0\), it means no sampling, iterate with the existing \(Q\) value, where \(\overline{Q_{t+1}}=\sum_{a^{\prime}}\pi\left(s_{t+1},a^{\prime}\right)Q_{t} \left(s_{t},a^{\prime}\right)\). When \(\sigma=1\), it means that the update is done iteratively through sampling. Thus, \(\sigma\) can be thought of as the ratio of sampling to updating. \[ \begin{split} Q_{t+1}\left(s_{t},a_{t}\right)&=Q \left(s_{t},a_{t}\right)+\alpha\left[G_{t}^{\left(1\right)}-Q_{t}\left(s_{t},a _{t}\right)\right]\\ &=Q\left(s_{t},a_{t}\right)+\alpha\delta_{t}^{\sigma}\end{split}  \tag{4}\]  
From the derivation above, it is not difficult to find that \(\delta_{t}^{\sigma}=r_{t+1}+\gamma\left[\sigma Q_{t}\left(s_{t+1},a_{t+1} \right)+\left(1-\sigma\right)\overline{Q_{t+1}}\right]-Q_{t}\left(s_{t},a_{t}\right)\). **Theorem 1**. : _In the MDP, for any initialized \(Q\left(s,a\right),\forall\gamma\in\left(0,1\right)\). \(\forall\left(s,a\right)\in S^{\prime}\times A\). The update mode of \(Q\) can be expressed as: \(\hat{Q}_{k+1}^{\left(n+1\right)}\left(s,a\right)=r_{t+1}+\gamma\sum_{a}\pi \left(s_{t+1},a\right)\hat{Q}_{k}^{\left(n\right)}\left(s_{t+1},a\right)\), where \(\hat{Q}\) represents the estimate of \(Q\), and \(k\) represents the update frequency of \(Q\)._  
Proof. : If \(\Delta_{n}=E\left[\hat{Q}_{k+1}^{\left(1\right)}\left(s,a\right)\right]-Q^{ \pi}\left(s,a\right)\), you get \(\|\Delta_{n+1}\|<\gamma\|\Delta_{n}\|\) is the maximum norm compression sequence, that is, \(\hat{\Delta}_{n}\) converges to 0 probabilistically [29]. \[ \begin{split}&\text{For }n=1:\\ &\max_{\left(s,a\right)}\left|E\left[\hat{Q}_{k+1}^{\left(1 \right)}\left(s,a\right)\right]-Q^{\pi}\left(s,a\right)\right|=\\ &\left|R_{s}^{a}+\gamma\sum_{\left(s^{\prime},a^{\prime}\right)} p_{ss^{\prime}}^{a}\pi\left(s^{\prime},a^{\prime}\right)\hat{Q}_{k+1}^{\left(1 \right)}\left(s^{\prime},a^{\prime}\right)-R_{s}^{a}-\gamma\sum_{\left(s^{ \prime},a^{\prime}\right)}P_{ss^{\prime}}^{a}\pi\left(s^{\prime},a^{\prime} \right)Q^{\pi}\left(s^{\prime},a^{\prime}\right)\right|\\ &<=\gamma\underset{\left(s,a\right)}{\max}\left|\hat{Q}_{k+1}^{ \left(1\right)}\left(s^{\prime},a^{\prime}\right)-Q^{\pi}\left(s^{\prime},a^{ \prime}\right)\right|\end{split}