# Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
## 3 Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
### Oaktdc
Then, we can obtain \(v_{t}=C^{-1}\mathbb{E}[\delta_{t}\phi_{w}(s_{t})|\theta_{c}]\). From the perspective of \(v\), we consider \(\theta\) as a constant \(\theta_{c}\); however, bringing \(v\) into the update rule of Equation (18), it will be reformulated as follows:

\[\frac{d}{dt}\bar{w}(t)=\hat{\Gamma}^{\dot{W}}_{\mathfrak{w}(t)}(-\frac{1}{2} \
abla_{\dot{W}}\mathcal{L}_{slow})(\bar{w}(t)), \tag{23}\]6

let

\[\frac{d}{dt}\bar{w}(t)=\hat{\Gamma}^{\dot{W}}_{\mathfrak{w}(t)}(-\frac{1}{2} \
abla_{\dot{W}}\mathcal{L}_{slow})(\bar{w}(t)), \tag{23}\]7

be sigma fields generated by the following quantities:

\[\frac{d}{dt}\bar{w}(t)=\hat{\Gamma}^{\dot{W}}_{\mathfrak{w}(t)}(-\frac{1}{2} \
abla_{\dot{W}}\mathcal{L}_{slow})(\bar{w}(t)), \tag{23}\]8

let

\[ \begin{split} Z_{k+1}&=(\phi_{w}(s_{t})-\gamma\phi_ {w}^{\prime}(s_{t}))\phi_{w}^{\top}(s_{t})C^{-1}\mathbb{E}[\delta_{t}\phi_{w}( s_{t})|\theta_{t}]\\ &\quad-\mathbb{E}[(\phi_{w}(s_{t})-\gamma\phi_{w}^{\prime}(s_{t} ))\phi_{w}^{\top}(s_{t})C^{-1}\mathbb{E}[\delta_{t}\phi_{w}(s_{t})|\theta_{t}] |\mathcal{H}_{t}]\\ &=(\phi_{w}(s_{t})-\gamma\phi_{w}^{\prime}(s_{t}))\phi_{w}^{\top}( s_{t})C^{-1}\mathbb{E}[\delta_{t}\phi_{w}(s_{t})|\theta_{t}]\\ &\quad-\mathbb{E}[(\phi_{w}(s_{t})-\gamma\phi_{w}^{\prime}(s_{t} ))\phi_{w}^{\top}(s_{t})]C^{-1}\mathbb{E}[\delta_{t}\phi_{w}(s_{t})|\theta_{t} ].\end{split}  \tag{41}\]

It is easy to see that \(Z_{k},k\geq 0\) are integrable random variables and \(\mathbb{E}[Z_{t+1}|\mathcal{H}_{t}]=0\), \(\forall k\geq 0\), further:

\[\mathbb{E}[||Z_{t+1}||^{2}|\mathcal{H}_{t}]\leq c_{2}(1+||\theta_{t}||^{2}),t \geq 0, \tag{42}\]

for some constants, \(c_{2}>0\), again because \(r_{t}\), \(\phi_{w}(s_{t})\) and \(\phi_{w}^{\prime}(s_{t})\) have uniformly bounded second moments. Consider now the following ODE associated with Equation (18):

\[\dot{\theta}_{t}=\mathbb{E}[(\phi_{w}(s_{t})-\gamma\phi_{w}^{\prime}(s_{t})) \phi_{w}^{\top}(s_{t})]C^{-1}\mathbb{E}[\delta_{t}\phi_{w}(s_{t})|\theta_{t}]. \tag{43}\]

Let \(\bar{h}(\theta_{t})\) be the driving vector field of the ODE (43). Note that:

\[ \begin{split}\bar{h}(\theta_{t})&=\mathbb{E}[(\phi _{w}(s_{t})-\gamma\phi_{w}^{\prime}(s_{t}))\phi_{w}^{\top}(s_{t})]C^{-1}\mathbb{ E}[\delta_{t}\phi_{w}(s_{t})|\theta_{t}]\\ &=(\mathbb{E}[\phi_{w}(s_{t})\phi_{w}^{\top}(s_{t})]-\mathbb{E}[ \gamma\phi_{w}^{\prime}(s_{t})\phi_{w}^{\top}(s_{t})])C^{-1}\mathbb{E}[\delta_{t }\phi_{w}(s_{t})|\theta_{t}]\\ &=A^{\top}C^{-1}(-A\theta_{t}+b),\end{split}  \tag{44}\]

we can bring Equation (19) to \(\mathbb{E}[\delta_{t}\phi_{w}(s_{t})|\theta_{t}]\), then, we can see that it is equal to \(-A\theta_{t}+b\). Now, \(\theta^{*}=A^{-1}b\) is shown to be a unique, globally asymptotically stable equilibrium for (43). Let \(\bar{h}_{\infty}(\theta)=\lim_{r\to\infty}\bar{h}_{\infty}(r\theta)/r\). Then, \(\bar{h}_{\infty}(\theta)=-A^{\top}C^{-1}A\theta\). Consider the ODE:

\[\dot{\theta}=-A^{\top}C^{-1}A\theta_{t}. \tag{45}\]

Because \(C^{-1}\) is a positive definite and \(A\) has full rank (as it is nonsingular by assumption), the matrix \(A^{\top}C^{-1}A\) is also a positive definite. The ODE (45) has its unique globally asymptotically stable equilibrium as its point of origin. Assumptions 1 and 2 of [40] are once again verified and the claim can follow from these. #### 4.2.2 Convergence Analysis of OAKTDC and OAKETD

In the proof of the previous section, we innovatively utilized dual timescales and ODE fixed-point theory to demonstrate the convergence of parameter \(\theta\). Similarly, we can obtain convergence guarantees for OAKTDC, which can be referenced using the proof from the previous section.