# Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
## 3 Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
### Oaktdc
A reward of \(-1\) is given for all states except for the goal state, where the episode concludes with a reward of \(0\). The discount factor is set to \(0.99\), and the agent starts from a vertically downward position. ### Experimental Setup

To ensure fairness and the comparability of the results, common parameters were set across all algorithms. During the learning process, an \(\epsilon\)-greedy exploration strategy was used as the behavior policy. Using this strategy, the agent randomly selects an action with probability \(\epsilon\) and chooses the action with the highest estimated value with probability \(1-\epsilon\). In this section, \(\epsilon\) is set to exponentially decrease, i.e., \(\epsilon=0.01\times 0.9999^{t}\). This decay simulates the agent's gradual familiarity with the environment and continuous optimization of the policy as the learning process progresses. In our methods, the Gaussian kernel function \(k(s,s^{\prime})=e^{-||s-s^{\prime}||^{2}/(2\sigma^{2})}\) is used, with a bandwidth \(\sigma=0.1\). For the online dictionary construction process, the parameter \(\mu_{1}=0.3\) is set, and in the OAKETD algorithm, the interest value is \(I=1\). In Mountain Car, the parameters for each algorithm are set as follows: the learning rate in the slow timescale is set to \(\alpha_{t}=0.05\times 0.99999^{t}\), the learning rate in the fast timescale is \(\beta_{t}=0.001\times 0.999999^{t}\), and the learning rate of the auxiliary parameter is \(\eta_{t}=0.0001\times 0.999999^{t}\). In Cart Pole, the learning rate in the slow timescale is set to \(\alpha_{t}=0.05\times 0.99999^{t}\), the learning rate in the fast timescale is \(\beta_{t}=0.01\times 0.999999^{t}\), and the learning rate of the auxiliary parameter is \(\eta_{t}=0.005\times 0.999999^{t}\). Comparison with other off-policy algorithms: In the Mountain Car environment, we conducted comparisons with the traditional off-policy algorithm, Q-learning, as well as the DQN algorithm [7], which incorporates deep feature representation. To maintain fairness in Q-learning, we utilized the same online attentive kernel-based features for feature representation as in our proposed methods, with the learning rate set to \(\alpha=0.1\). Regarding DQN, the learning rate was configured to \(0.001\), with a discount factor of \(0.99\). The target network was updated every \(500\) steps, the replay buffer had a capacity of \(100\),\(000\), and the batch size was set to \(32\). ### Experimental Results

In the experiments, each algorithm was run at least \(50\) times independently for each control task, with each experiment involving \(1\),\(000\),\(000\) training steps. Additionally, the

Figure 3: Visualization of the Mountain Car, Cart Pole and Acrobot experiments. performance of the algorithm was evaluated every 1000 training steps. Figure 4 displays the learning curves of different algorithms across various control tasks. The horizontal axis represents the number of training steps, while the vertical axis represents the cumulative reward. A higher cumulative reward indicates that the algorithm has a better performance at that step. In Mountain Car, OAKTDC and OAKTD still exhibit relatively good stability, particularly OAKTDC, which shows a relatively smooth cumulative reward curve and performs quite well. OAKETD improves rapidly in the early stages but performs similarly to OAKTD in the later stages. OAKGTD2 shows considerable fluctuation in performance; although the cumulative reward improves at certain stages, it does not clearly outperform the other algorithms overall. In Cart Pole, OAKGTD2 stands out the most, with its cumulative reward gradually increasing and reaching a high reward value in the later stages of training. However, the performance of OAKGTD2 exhibits considerable fluctuations, with significant uncertainty in its performance being indicated during certain training steps. OAKTDC performs stably, with cumulative rewards that gradually increase and stabilize; although its peak values are not as high as those of OAKGTD2, it shows better consistency.