# Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
## 1 Introduction  
Reinforcement Learning (RL), as an important branch of machine learning, has garnered extensive attention recently [1; 2; 3; 4; 5]. In RL, an agent continuously interacts with the unknown environment to optimize its policy for maximizing cumulative rewards [6]. Advanced RL methods are capable of handling tasks with large-scale state-action spaces [7; 8; 9], primarily because they use Value Function Approximation (VFA) to represent the mapping from states to policy values. A major challenge for VFA is catastrophic forgetting or interference [10; 11]. To address this challenge, VFA methods can be categorized into three main classes: linear methods, neural network-based methods, and kernel-based methods [12]. In linear methods, techniques like Tile Coding [13] and \(n\)-tuple networks [14] have been applied in VFA algorithms, offering good convergence guarantees under TD learning. However, they are predefined and suffer from the curse of dimensionality [6]. Neural network methods, such as Rectified Linear Unit (ReLU) [15] and dropout [16], induce sparsity, helping to mitigate catastrophic forgetting. \(k\)-sparse autoencoders [17] and Winner-Take-All autoencoders [18] also promote sparsity but may truncate meaningful outputs [10]. Regularization techniques, such as \(L_{1}\)[19] and \(L_{2}\)[20], as well as distributional regularizers [10], further enhance sparsity. Neural methods do not require predefined features but lack convergence guarantees under TD learning. Kernel-based approaches have gained attention for their powerful function approximation capabilities [21; 22]. These methods achieve linear learning and are well-suited to high-dimensional state and action spaces. A key advantage over neural networks is their theoretical convergence guarantees. Specifically, universal kernels [23] ensure that TDlearning converges to the true value function [24; 25], eliminating the need for complex error analysis [26]. To address catastrophic forgetting, OSKTD [27] manually defines a selective function for sparse representation, while OAKTD [12] uses attention mechanisms [28] and two-timescale optimization [29] for stability. However, these methods only apply to on-policy TD learning, where the policies used for decision-making and evaluation are the same. Off-policy TD learning [30] allows for learning from data generated by different policies, improving efficiency and robustness. Efficient feature representation and convergence guarantees for off-policy learning remain challenges for kernel-based VFA methods. To enhance the adaptability of kernel-based VFA methods in off-policy settings, this paper improves upon widely used off-policy algorithms with strong convergence properties, such as Gradient Temporal Difference Learning 2 (GTD2) [31], Temporal Difference with Correction (TDC) [31], and Emphatic Temporal Difference (ETD) [25]. We design and integrate an online attention kernel to automatically and efficiently provide the sparse representation of state features, resulting in three algorithms: Online Attention Kernel-Based Gradient Temporal Difference Learning 2 (OAKGTD2), Online Attention Kernel-Based Temporal Difference with Correction (OAKTDC), and Online Attention Kernel-Based Emphatic Temporal Difference (OAKETD). In these algorithms, we employ two-timescale optimization [29] to learn the value function, ensuring stability. Specifically, the OAKGTD2 and OAKTDC algorithms introduce an auxiliary weight vector and refine the fast-timescale parameter updates in the two-timescale framework, enhancing the stability and speed of learning. In the OAKETD algorithm, an emphasis function is incorporated into the fast-timescale parameter updates, further improving the stability and efficiency of learning. Notably, we theoretically prove that the value functions in all three algorithms converge to the optimal solution. We conducted an experimental analysis of the three proposed algorithms across different test tasks. Compared to baseline algorithms, the proposed methods demonstrated superior performance.