# Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
## 3 Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
### Oaktdc  
The Temporal Difference with Correction (TDC) [31] is a further development based on the GTD2. It improves the TD update rule by introducing a gradient correction term. This correction term allows the algorithm to utilize off-policy data more effectively while maintaining the same linear time complexity as traditional TD learning. Specifically, TDC introduces an auxiliary weight vector that directly corrects the gradient bias caused by the discrepancy between the target policy and the behavior policy. This design significantly enhances the stability of learning and enables the algorithm to work effectively in a wider range of environments, particularly in complex environments where the data sampling strategy frequently changes. Similar to the OAKGTD2 algorithm, we introduce the value function learning process of TDC under the online attentive kernel-based framework, forming a new algorithm called Online Attention Kernel-Based Temporal Difference with Correction (OAKTDC). In OAKTDC, the update rule for the auxiliary variable \(v\) is the same as in OAKGTD2, and the update rule for the value function parameter \(\theta\) can be expressed as follows:  
\[\theta_{t+1}=\theta_{t}+\beta_{t}(\delta_{t}\phi_{w}(s_{t})-\gamma\phi_{w}^{ \prime}(s_{t}^{\prime})(\phi_{w}^{\top}(s_{t})v_{t})), \tag{20}\]  
where the auxiliary vector \(v_{t}\) is the same as the auxiliary vector that is updated in Equation (17). OAKTDC enhances the stability of off-policy learning processes while retaining the advantages of OAKTD by introducing corrective terms. Its core idea combines online learning and attention mechanisms with the corrective terms of the TDC algorithm to improve stability and efficiency during learning. Compared to the OAKGTD2 algorithm, OAKTDC demonstrates unique advantages in addressing non-stationarity issues. While OAKGTD2 focuses on reducing variance during learning through importance-weighting and gradient descent, OAKTDC directly corrects parameter updates using TDC's corrective terms to mitigate the effects of non-stationarity. This approach enables the OAKTDC algorithm to learn more stably in rapidly changing environments and allows for better adaptation to sparse reward problems. Refer to Algorithm 2 for the pseudocode of the OAKTDC. Input: the policy \(\pi\) to be evaluated, the parameters \(\alpha\), \(\beta\), \(\eta\), \(\epsilon\), \(\mu_{1}\)  
Initialize \(D=\varnothing\), \(\theta\), \(\bar{\theta}\), \(w\)
```
1: Follow steps 1-7 of Algorithm 1
2: Update \(v\), \(\theta\) according to Equations (17) and (20)
3: Follow steps 9-12 of Algorithm 1

### Oaketd

The Emphatic Temporal Difference (ETD) [25] is designed to enhance the stability and accuracy of TD learning in non-stationary environments. ETD introduces a mechanism called the \"emphasis factor\" to adjust the importance of updates for each state. These weights consider not only the current state transition probabilities but also the long-term importance of each state under the policy. ETD is particularly suitable for scenarios where the policy itself may change over time. In such cases, traditional TD learning may become unstable due to policy changes. ETD maintains focus on important states under the old policy through its emphasis mechanism, even if these states are less frequent under the new policy. We introduce the value function learning process of ETD under the online attentive kernel-based framework, forming a new algorithm called Online Attention Kernel-Based Emphatic Temporal Difference (OAKETD). The design of the OAKETD algorithm preserves the slow-timescale process of OAKTD. For the fast timescale, OAKETD adopts the update mechanism from the ETD. ETD enhances the learning signal by introducing an emphasis factor \(F_{t}\), thereby improving the stability and efficiency of the learning process.