# Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
## 3 Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
### Oakgtd2  
The TD learning methods in RL, particularly early algorithms such as TD(0) and TD(\(\lambda\)) [33], perform well in the on-policy setting. However, they often encounter stability and convergence issues in off-policy settings. Off-policy learning involves using a behavior policy that is different from the target policy to generate data, which limits the applicability of standard TD learning methods. To address this problem, Gradient Temporal Difference 2 (GTD2) [31] represents a further optimization and development based on GTD [34].  
Specifically, GTD2 introduces an auxiliary weight vector \(v\) to assist in updating the main weight vector \(\theta\). This mechanism effectively separates the target and behavior policies of the value function, thereby reducing the impact of off-policy data and enhancing the algorithm's stability. Different from the transition \((s_{t},s_{t+1},r_{t})\) in the on-policy setting, in off-policy setting, we define a probability over independent triples of state and the next state, and reward random variables, denoted as \((s_{t},s^{\prime}_{t},r_{t})\), with associated online attentive kernel-based feature-vector random variables \(\phi_{w}(s_{t})=K(s_{t})a_{w}(s_{t})\) and \(\phi^{\prime}_{w}(s^{\prime}_{t})=K(s^{\prime}_{t})a_{w}(s^{\prime}_{t})\), which are defined according to Equation (5).  
To address the limitation of GTD2 in only performing a linear value function approximation, we introduce the value function learning process of GTD2 under the online attentive kernel-based framework, forming a new algorithm called Online Attention Kernel-Based Gradient Temporal Difference Learning 2 (OAKGTD2). In OAKGTD2, the updates for the auxiliary variable \(v\) and value function parameter \(\theta\) are as follows:  
\[v_{t+1}=v_{t}+\eta_{t}(\delta_{t}-v^{\top}_{t}\phi_{w}(s_{t}))\phi_{w}(s_{t}), \tag{17}\]  
\[\theta_{t+1}=\theta_{t}+\beta_{t}(\phi_{w}(s_{t})-\gamma\phi^{\prime}_{w}(s^{ \prime}_{t}))(\phi^{\top}_{w}(s_{t})v_{t}), \tag{18}\]  
where \(\eta_{t}\) is the learning rate introduced for the auxiliary variable \(v\), which will be heuristically updated, together with \(\alpha_{t}\) in Equation (12) and \(\beta_{t}\) in Equation (18), in each iteration. \(\delta_{t}\) is the temporal difference error in off-policy settings like that in Equation (4), defined as follows:  
\[\delta_{t}=r_{t}+\gamma\theta_{t}^{\top}\phi^{\prime}_{w}(s^{\prime}_{t})- \theta_{t}^{\top}\phi_{w}(s_{t}). \tag{19}\]  
OAKGTD2 combines the advantages of OAKTD with the stability improvements in GTD2 to address challenges in off-policy reinforcement learning. By incorporating the dual-weight update mechanism from GTD2, it effectively balances learning stability and computational efficiency, making it particularly suitable for large-scale or complex state spaces. The design of the OAKGTD2 lies in the learning processes across two timescales. In the slow timescale, the algorithm retains the characteristics of OAKTD, gradually adjusting its parameters to ensure learning stability and reduce accumulated variance. The key point in OAKGTD2 is the update process in the fast timescale, which quickly adapts to environmental changes and reduces variance during the value learning process. Refer to Algorithm 1 for the pseudocode of OAKGTD2.  
```
Input: the policy \(\pi\) to be evaluated, the parameters \(\alpha\), \(\beta\), \(\eta\), \(\epsilon\), \(\mu_{1}\)

Initialize \(D=\varnothing\), \(\theta\), \(\bar{\theta}\), \(w\)
```
1:for each episode do
2: Initialize state \(s\)
3:while\(s\) is not the termination state do
4: Sample \(a\sim\pi(s)\)
5: Take action \(a\), observe \(r\) and \(s^{\prime}\)
6: Update dictionary \(D\) according to Equation (7)
7: Update \(w\), \(\bar{\theta}\) according to Equations (12) and (14)
8: Update \(v\), \(\theta\) according to Equations (17) and (18)
9: Update parameters \(\alpha\), \(\beta\), \(\eta\) and \(\epsilon\)
10:\(s\gets s^{\prime}\)
11:endwhile
12:endfor ```
**Algorithm 1** OAKGTD2