# Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
## 2 Background and Preliminaries
### Online Attentive Kernel-Based TD Learning  
The OAKTD algorithm is an on-policy RL algorithm that combines kernel methods and attention mechanisms, aimed at improving the accuracy and stability of value function estimation. OAKTD is particularly suitable for handling high-dimensional state spaces and non-linear problems, achieving efficient learning through kernel-based value function estimation, online dictionary construction, and two-timescale optimization. The value obtained in Equation (1) using the kernel-based VFA can be written as follows:  
\[V_{\theta,w}(s)=\theta^{\top}K(s)a_{w}(s)=\sum_{s_{i}\in D}\theta_{i}k(s,s_{i} )a_{w}(s,s_{i}), \tag{5}\]  
where \(K(s)\) is the diagonal matrix of kernel function \(k(s,s_{i})\), parameters \(w\) and \(\theta\) denote the learning parameter vectors, \(a_{w}(s)\) represents the attention function vector of elements \(a_{w}(s,s_{i})\), and \(D\) denotes the online dictionary constructed by MNC [27]. The online attentive kernel-based temporal difference learning framework is illustrated in Figure 1.  
In TD learning, a key challenge is to approximate the true value function. Using kernel methods can avoid the complex errors that are linearly related to the horizon of the MDP [23; 26]. Kernel methods form the basis of the OAKTD algorithm by mapping data from the original space to a high-dimensional feature space, where linear decision boundaries can effectively solve nonlinear problems in the original space, thus eliminating the approximation errors that generally linearly increase with the range of MDP. The kernel function \(k(s,s^{\prime})\) computes the similarity between any two states, \(s\) and \(s^{\prime}\). In this paper, we use the Gaussian kernel, defined as follows:  
\[k(s,s^{\prime})=\exp\left(-\frac{\left\|s-s^{\prime}\right\|^{2}}{2\sigma^{2}} \right), \tag{6}\]  
where \(\sigma\) is the bandwidth parameter of the kernel function. Online dictionary construction selects a representative set of states (dictionary entries) from the state space to achieve efficient value function estimation. The goal of online dictionary construction is to maintain coverage of the states that are most relevant to the current policy while avoiding unnecessary redundancy. It utilizes a novelty criterion [27] based on the distances between data points to select and update dictionary entries. Specifically, at each time  
Figure 1: The framework of attentive kernel-based VFA, where the dictionary is \(D=\{s_{1},s_{2},...,s_{n}\}\) and the multiplication of \(k(s,\cdot)\) and \(a_{w}(s\cdot)\) is used as a sparse representation. Then, a dot product with the value function parameters \(\theta\) can yield the value of the value function. step \(t\), the process involves computing the distance \(d_{t}\) between the new state \(s_{t}\) and existing states in the dictionary and comparing this against a predefined threshold \(\mu_{1}\). If the distance is less than the threshold, the state is not added to the dictionary; otherwise, it is added as a new dictionary entry. This process can be represented by the following formula:  
\[D_{t}= \begin{cases}D_{t-1},&\text{if }d_{t}<\mu_{1},\\ D_{t-1}\cup\{\phi_{w}(s_{t})\}&\text{otherwise}.\end{cases}  \tag{7}\]  
The attention mechanism allows the algorithm to focus on learning the most valuable state features for the current task, thereby enhancing learning efficiency and prediction accuracy. Inspired by the general attention mechanism [28], a novel scoring function based on the \(L_{1}\)-norm is defined as follows:  
\[e(s,s_{i})=w^{\mathsf{T}}\psi(s,s_{i}), \tag{8}\]  
where the learning parameter \(w\in\mathbb{R}^{|s|}\) and \(\psi(s,s_{i})\in\mathbb{R}^{|s|}\) is a vector in which each element represents the Euclidean distance between each value in the state and the corresponding value in the dictionary state.