# Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
## 3 Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
### Oaktdc
In OAKETD, the random sequence \(\{\theta_{t}\}_{t\in\mathbb{N}}\) generated by the ETD algorithm under double timescales almost certainly converges to the limit \(\theta^{*}\), where \(\theta^{*}\) satisfies

\[\Pi_{\oplus}T(\Phi_{\oplus}\theta^{*})=\Phi_{\oplus}\theta^{*}, \tag{46}\]

where \(\Phi_{\oplus}\) is the feature matrix, \(\Pi_{\oplus}\) is a projection operator derived from \(\Pi_{\oplus^{*}}V=\underset{\tilde{V}\in\mathcal{F}}{\text{arg min}}\,|| \tilde{V}-V||_{\text{D}}^{2}\), and \(\mathcal{F}=\{\Phi_{\oplus^{*}}\theta\}\). Since the step-size sequences \(\{\alpha_{t}\}_{t\in\mathbb{N}}\) and \(\{\beta_{t}\}_{t\in\mathbb{N}}\) satisfy Assumption 4, the different learning rates lead to asymptotic non-synchronous convergence, resulting in multiple perspectives. From the viewpoint of the faster timescale recursion, the slower timescale recursion appears quasi-static, whereas from the viewpoint of the slower timescale recursion, the faster timescale recursion seems to be balanced. Therefore, it is permissible to analyze the convergence of the fast and slow timescales independently. The convergence proof for the slower timescale is consistent with the OAKGTD2 algorithm, and the stability analysis for the faster timescale was provided by [25], indicating convergence to the aforementioned fixed point. ## 5 Experimental Results and Analysis

In this section, we present experimental comparisons of the OAKGTD2, OAKTDC, and OAKETD algorithms in three environments: Mountain Car, Cart Pole, and Acrobot. The goal was to test and compare the performance of these algorithms in standard reinforcement learning environments. ### Experimental Environments and Comparison Algorithms

First, we briefly introduce the three experimental environments, all of which are continuous control tasks. Then, we present the comparison algorithms used in the experiments alongside the proposed methods. In the Mountain Car environment, a car was driven to the top of a hill (see Figure 3a). The challenge lies in the fact that the car's engine is too weak to overcome gravity, preventing it from directly accelerating up the hill. The car has three possible actions: full throttle forward (\(+1\)), full throttle reverse (\(-1\)), and no throttle (\(0\)). The car's position \(x_{t+1}\) is updated based on its previous position \(x_{t}\) and velocity \(\dot{x}_{t}\), as follows:

\[x_{t+1}=\text{bound}[x_{t}+\dot{x}_{t+1}],\]

where \(\dot{x}_{t+1}=\text{bound}[\dot{x}_{t}+0.001a_{t}-0.0025\cos(3x_{t})]\); the bound operation enforces \(-1.2\leq x_{t+1}\leq 0.5\) and \(-0.07\leq\dot{x}_{t+1}\leq 0.07\). In addition, when \(x_{t+1}\) reaches the left bound, \(\dot{x}_{t+1}\) will be reset to zero. The reward is \(-1\) for all states except the goal state at the top of the hill, in which the episode ends with a reward of \(0\). The discount factor is set to \(0.99\), and each episode starts from a random position \(x_{0}\in[-0.6,-0.4)\) with zero velocity. The Cart Pole task is a classic control task involving an inverted pendulum (see Figure 3b), where the pendulum's center of mass is above the pivot point, making it inherently unstable. The system can be stabilized by adjusting the position of the pivot beneath the center of mass. The state of the Cart Pole is defined by the cart's position, the cart's velocity, the pole's angle, and its angular velocity. There are two available actions: moving the cart left or right. A reward of \(+1\) is awarded for each timestep in which the pole remains upright. The discount factor is set to \(1\), and the objective is to maintain the balance of the pole by applying suitable forces at the pivot. The Acrobot is a two-link underactuated robotic system (see Figure 2(c)). The objective is to swing the end-effector to a height that is at least one link length above the base. The system offers three discrete actions: applying a positive torque (\(+1\)), applying a negative torque (\(-1\)), and applying no torque (\(0\)). The state space is defined by the two joint angles and their respective angular velocities.