# Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
## 3 Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
### Oaktdc
Furthermore, the stochastic sequence \(\{\theta_{t}\}_{t\in\mathbb{N}}\) generated by the semi-gradient algorithm within the two-timescale setting converges almost exactly to the limit \(\theta^{*}\), which satisfies the following equation:_

\[\Phi_{\varpi^{*},D}\theta^{*}=\Pi_{\varpi^{*},D}\mathbb{T}(\Phi_{\varpi^{*},D }\theta^{*}), \tag{24}\]

_where \(\bar{w}^{*}\in\mathcal{W}\), \(\Phi_{\varpi^{*},D}\) is the attentive kernel-based feature matrix with the \(\phi_{\varpi^{*},D}(s)\) as its rows, and \(\Pi_{\varpi^{*},D}\) is a projection operator according to \(\Pi_{\varpi^{*}}V=\arg\min_{\bar{V}\in\mathcal{F}}||\bar{V}-V||_{D}^{2}\) with \(\mathcal{F}=\{\Phi_{\varpi^{*},D}\theta\}\)._

The proof of Theorem 2 closely follows the approach used in two-timescale networks [29]. Specifically, we use a fixed dictionary to eliminate the disturbances introduced by the dictionary construction process [37]. In the two-timescale optimization of the proposed OAKGTD2, OAKTDC, and OAKETD, under Assumption 4, it is implied that the optimization processes of the two timescales are asynchronous, with the feature parameters \(\bar{w}\) in the slow timescale converging more slowly than the value parameters \(\theta\) in the fast timescale. This characteristic gives rise to two perspectives: from the perspective of the fast timescale, the update of the slow-timescale appears static and nearly constant, while from the slow-timescale perspective, the updates of the fast timescale seem to have reached equilibrium. Therefore, we only need to prove that the parameters of each of the two timescales can converge. The convergence of parameter \(\bar{w}\) in the slow timescale was n established in OAKTD [12]. An analysis is provided in that reference; this section will not review it further. In the following subsection, we will specifically analyze the convergence guarantees of parameter \(\theta\) in the fast timescale for the three proposed algorithms. ### Fast-Timescale Convergence Analysis

The three algorithms proposed in this paper differ in their parameter updates at the fast time scale. Specifically, OAKGTD2 and OAKTDC introduce an auxiliary variable \(v\) on a third timescale, \(\eta\), to update \(\theta\). Therefore, for these two algorithms, the analysis is further divided into a convergence analysis of \(v\) and \(\theta\). In contrast, for OAKETD, only the convergence of \(\theta\) needs to be analyzed. #### 4.2.1 Convergence Analysis of OAKGTD2

In this subsection, we first conduct an analysis of the timescale of the auxiliary variable \(v\) based on ODE, and derive its expression. Using this expression, we will prove that, under the assumptions stated earlier, the value function parameter \(\theta\) in the OAKGTD2 algorithm will converge to the TD fixed point in Equation (25) with a probability of one. \[V_{\theta,w}=\Pi TV_{\theta,w}, \tag{25}\]

where \(T\) is the Bellman operator and \(\Pi\) is the projection operator, which is defined as follows:

\[\Pi=\Phi(\Phi^{\top}G\Phi)^{-1}\Phi^{\top}G, \tag{26}\]

where \(\Phi\) is the matrix whose rows are the \(\phi_{w}(s)\) and the norm \(||v||_{\text{G}}^{2}=v^{\top}Gv\) is weighted by the matrix \(G\) that has the \(d(s)\) (see Equation (11)) on its diagonal. **Theorem 3** (Convergence of OAKGTD2). : _Consider the update rules in Equations (17) and (18) of OAKGTD2. Let the step-size sequences \(\{\eta_{t}\}\) and \(\{\beta_{t}\}\) satisfy Assumption 4. Furthermore, assume that \((\phi_{w}(s_{t}),r_{t},\phi_{w}^{\prime}(s_{t}))\) is an i.i.d. sequence with uniformly bounded second moments. Let \(A=\mathbb{E}[(\phi_{w}(s_{t})-\gamma\phi_{w}^{\prime}(s_{t}))\phi_{w}^{\top}(s _{t})]\), \(b=\mathbb{E}[r_{t}\phi_{w}(s_{t})]\), and \(C=\mathbb{E}[\phi_{w}(s_{t})\phi_{w}(s_{t})^{\top}]\). Assume that \(A\) and \(C\) are non-singular matrices. Then, parameter vector \(\theta_{t}\) converges to the TD fixed point in Equation (25) with a probability of one._

Proof.