# Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
## 3 Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
### Oaktdc
In OAKETD, the parameter update rule for the fast process is as follows:

\[F_{t+1}=\gamma_{t}\rho_{t}F_{t}+1, \tag{21}\]

\[\theta_{t+1}=\theta_{t}+\beta_{t}F_{t}\rho_{t}\delta_{t}\phi_{w}(s_{t}), \tag{22}\]

where \(\delta_{t}\) represents the TD error, which is defined in Equation (19), \(\beta_{t}\) is the fast-timescale learning rate, \(\rho_{t}\) denotes the importance sampling ratio \(\rho_{t}=\pi_{\text{T}}(a_{t}|s_{t})/\pi_{\text{B}}(a_{t}|s_{t})\) where \(\pi_{\text{T}}\) is the target policy to be evaluated, and \(\pi_{\text{B}}\) is the behavior policy that executes the actions. This subsection proposes the OAKETD algorithm, building upon the slow-timescale process of OAKTD. OAKETD replaces the Mean Squared Value Error (MSVE) with the ETD algorithm in the fast-timescale process, aiming to achieve more stable and efficient learning in off-policy environments. The main advantage of OAKETD lies in its ability to effectively handle off-policy data and maintain stable learning processes in non-stationary environments. The slow-timescale learning process ensures long-term stability and variance control, while the fast-timescale process enhances the focus on critical state transitions, thereby accelerating learning speed and improving learning quality. Refer to Algorithm 3 for the pseudocode of the OAKETD. ```
1:the policy \(\pi\) to be evaluated, the parameters \(\alpha\), \(\beta\), \(\rho\), \(\epsilon\), \(\mu_{1}\)
2:Initialize \(D=\varnothing\), \(\theta\), \(\bar{\theta}\), \(w\)
3:Follow steps 1-7 of Algorithm 1
4:Update \(F\), \(\theta\) according to Equations (21) and (22)
5:Update parameters \(\alpha\), \(\beta\), \(\rho\) and \(\epsilon\)
6:Follow steps 10-12 of Algorithm 1
```

**Algorithm 3** OAKETD

### Comparison of Existing Algorithms

In this section, we will compare various algorithms to highlight their differences in terms of update rules. The compared algorithms include TD and OAKTD, both of which are on-policy algorithms, with the key distinction being that OAKTD is kernel-based. On the other hand, the off-policy algorithms GTD2, TDC, and ETD are not kernel-based. We summarize these algorithms, along with the three proposed algorithms, in Table 1, emphasizing the differences in their update rules. ## 4 Convergence Analysis

In this section, we first present the convergence conditions for online dictionary construction, and combine the online dictionary convergence conditions with some mild assumptions, under which the convergence guaranteed to occur at the slow timescale for the proposed methods is the same as that achieved with OAKTD [12] and will not be elaborated here. Second, we provide details of the convergence over the fast timescale for OAKGTD2, OAKTDC, and OAKETD. ### Assumptions and Convergence Conditions

**Assumption 1**. : _The kernel function \(k\) in the dictionary construction is a Lipschitz continuous Mercer kernel and state space \(S\) is a compact subset of a Banach space._

**Assumption 2**. : _The Markov chain induced by the given policy \(\pi\) is ergodic, i.e., aperiodic and irreducible._

**Assumption 3**. : _An instance of the dynamic transition of the given MDP is \(O_{\pi}=\{s_{0},a_{0},r_{1},s_{1},a_{1},r_{2},s_{2},\ldots\}\), where the initial state \(s_{0}\) is chosen arbitrarily, the action is \(a_{t}\sim\pi(s_{t})\), the next state is \(s_{t+1}\sim P(s_{t},a_{t},\cdot)\), and the reward is \(r_{t}=R(s_{t},a_{t},s_{t+1})\)._

**Assumption 4**.