# Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
## 3 Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
### Oaktdc
OAKETD performs well in the early stages but exhibits slow growth and large fluctuations in the later stages. As a baseline algorithm, OAKTD shows overall stability but has lower cumulative rewards. In Acrobot, both OAKTDC and OAKTD exhibit good stability in the later stages of training, with cumulative rewards gradually increasing and stabilizing. OAKETD shows a rapid increase in the early stages but continues to improve in the later stages. The performance of OAKGTD2 in the Acrobot task remains relatively volatile compared to other algorithms, although it is slightly better than OAKTD, indicating that its adaptability varies across different tasks. Figure 4: Learning curves in Mountain Car, Acrobot, and CartPole and a comparison with other off-policy algorithms. The abscissa is the number of training steps, and the ordinate is the cumulative reward. Comparison with other off-policy algorithms: In the MountainCar environment, compared with the classic off-policy algorithms Q-Learning and DQN, this Figure shows that all three improved algorithms outperform both Q-Learning and DQN. In this study, an in-depth experimental analysis of the OAKTD algorithm and its variants, OAKGTD2, OAKETD, and OAKTDC, was conducted across three classic reinforcement learning tasks: Mountain Car, Cart Pole, and Acrobot. The experimental results indicate that different algorithms exhibit varying strengths and weaknesses in reinforcement learning. The OAKGTD2 algorithm performs well initially but shows some instability in later stages, making it suitable for tasks requiring rapid initial adaptation. The OAKTDC algorithm stands out in the Mountain Car and Acrobot environments due to its strong stability and fast convergence, making it ideal for efficient training scenarios. The OAKETD algorithm excels in its handling of off-policy and non-stationary distribution issues, showing good stability during long-term training, and is suitable for long-duration tasks. In contrast, while the OAKTD algorithm is stable, it does not achieve the cumulative reward improvements seen with other algorithms, which validates the need for improvements to adapt it to off-policy learning environments. ### Ablation Study and Parameter Analysis

In the experiments presented in this section, we also conducted a set of ablation studies to demonstrate the performance gain of the online attentive kernel-based feature representation in OAKGTD2, OAKTDC, and OAKETD, as shown in Figure 5. The results indicate that this feature representation provides significant improvements to the algorithms. Additionally, we performed a parameter analysis of the dictionary construction parameter \(\mu_{1}\) in the proposed algorithm. It can be seen that selecting appropriate parameters is also beneficial to the algorithm's performance. ## 6 Conclusions and Future Work

In this paper, we proposed a novel framework, online attentive kernel-based off-policy TD learning, to address the challenges in feature representation and stability in off-policy

Figure 5: Learning curves of the ablation study in the three scenarios, along with the experimental results obtained under different parameters in the Acrobot environment. reinforcement learning. Within this framework, we introduced three improved algorithms: OAKGTD2, OAKTDC, and OAKETD. By employing a two-timescale optimization approach, the framework effectively integrates online attentive kernel-based methods to allow for sparse feature representation with efficient value function updates. We rigorously proved the convergence of all proposed algorithms using TD fixed-point theory, ensuring their theoretical validity. Extensive experiments in standard reinforcement learning environments demonstrated that our framework outperforms existing methods, achieving cumulative reward improvements of 15% over on-policy algorithms and 25% over common off-policy algorithms. These results highlight the framework's potential for both theoretical research and practical applications in reinforcement learning.