# Online Attentive Kernel-Based Off-Policy Temporal Difference Learning
## 2 Background and Preliminaries
### Online Attentive Kernel-Based TD Learning
After being further normalized through a Softmax function, the attention function is defined as follows:  
\[a_{w}(s,s_{i})=\frac{\exp(w^{\top}\psi(s,s_{i}))}{\sum_{s_{j}\in D}\exp(w^{ \top}\psi(s,s_{j}))}. \tag{9}\]  
In OAKTD, two-timescale optimization can achieve an accurate estimation of the value function parameters and allow for the timely adjustment of attention weights. The core idea of two-timescale optimization is to decompose the learning process into two independently evolving time scales--a fast-timescale and a slow-timescale--as illustrated in Figure 2.  
In the slow timescale, we employ an auxiliary parameter \(\bar{\theta}\) to learn sparse representations \(\phi_{w}(s)\), which captures the features of the state space. This part learns slowly to stabilize sparse representations, aiming to obtain more accurate and stable feature representations over long-term training. This part also updates the parameters of the attention mechanism, adjusting attention weights to influence the estimation and updating of the value function. The fast-timescale builds upon well-learned sparse representations using parameter \(\theta\) to estimate the function \(V_{w,\theta}\), which approximates the value function. This part learns  
Figure 2: Online attentive kernel-based temporal difference learning two-timescale optimization; the dashed lines in the figure represent the back-propagation process, specifically illustrating how model parameters are updated through back-propagation to minimize the loss function. rapidly to quickly approximate the true values of the value function, yielding good estimation results in a short period and using online dictionary construction and value function estimation to approximate the value function of the current state. This process includes updating the dictionary based on the current state, calculating the estimated value function, and updating parameters based on the estimated value and immediate rewards. Structurally, the algorithm divides the learning and updating process of the value function into two layers: a sparse representation layer and a linear estimation layer. This design allows the algorithm to handle different learning tasks effectively at different timescales, balancing rapid adaptation with stability. In the sparse representation layer, the algorithm uses attention kernels to construct sparse representations of states. In the linear estimation layer, the algorithm estimates the value function of states \(V_{\theta,w}(s)\) using a linear combination approach. For convenience, we express \(K(s)a_{w}(s)\) as \(\phi_{w}(s)\); then, Equation (5) can be rewritten as follows:  
\[V_{\theta,w}(s)=\theta^{\top}\phi_{w}(s). \tag{10}\]  
In the slow timescale, the algorithm aims to stably learn sparse representations of attention kernels. It employs a Naive Residual-Gradient method based on the Mean Squared Temporal Difference Error (MSTDE) [6], minimizing the MSTDE to gradually adjust the parameters of attention kernel features to improve the long-term accuracy of value function estimation. The objective function in this process is defined as follows:  
\[\mathcal{L}_{slow}(\bar{w})=\sum_{s\in S}d(s)\big{(}r+\gamma V_{\bar{w}}(s^{ \prime})-V_{\bar{w}}(s)\big{)}^{2}, \tag{11}\]  
where \(\bar{w}\) represents attention kernel parameters and \(V_{\bar{w}}\) denotes value function estimation based on the current parameters. \(d(s)\) is the distribution over states \(\sum_{s\in S}d(s)=1\).