# Hybrid Deep Reinforcement Learning for Pairs Trading
## 4 Hybrid Deep Reinforcement Learning for Pairs Trading
### Hybrid Reinforcement Learning
#### 4.4.4 Hybrid Reinforcement Learning Algorithm
\[d=(e\bmod c)+b \tag{28}\]  
Figure 8: Neural network structures. (**a**) Actor. (**b**) Regression. (**c**) Critics. (**d**) Stop-Loss. In the extended DDQN algorithm (red parts in Algorithm 1), the stop-loss network \(Q_{\Psi}\), whose structure is shown in Figure 8d, is combined with the SRL network \(\kappa_{i}\) shown in Figure 7. To support discrete stop-loss boundaries, a softmax layer is used as the output layer of the stop-loss network. An action \(a_{t}^{sl}\) is selected using the \(\varepsilon\)-greedy strategy (lines 10 and 11). The target value \(Y_{sl}\) is computed as in the DDQN, and then, the stop-loss network is updated using \(Y_{sl}\) (lines 20 and 21). When the stop-loss network is updated, the SRL network \(\kappa_{i}\) is also updated by backpropagation. ```
0: Sliding-window data \(w_{t}^{data}\leftarrow\{x_{t-N_{w}+1},\ \ldots,\ x_{t-1},\ x_{t}\}\) obtained from the data preprocessing
0: actor network \(\mu_{\Psi}\), SRL network \(\lambda_{\Psi}\) for \(\mu_{\Psi}\), stop-loss network \(Q_{\Psi}\), SRL network \(\kappa_{i}\) for \(Q_{\Psi}\)
1. Initialize an actor network \(\mu_{\Psi}\), an SRL network \(\lambda_{\Psi}\) for \(\mu_{\Psi}\), and a regression network
2. Initialize critic networks \(Q_{\Psi}\), \(Q_{\Psi}\), SRL networks \(\sigma_{\eta_{1}}\), \(\sigma_{\eta_{2}}\) for \(Q_{\Phi}\), \(Q_{\Phi}\)
3. Initialize a stop-loss network \(Q_{\Psi}\), an SRL network \(\kappa_{i}\) for \(Q_{\Psi}\)
4. Initialize target networks : \(\phi^{\prime}\leftarrow\phi\), \(v^{\prime}\gets v\), \(\theta^{\prime}_{1,2}\leftarrow\theta_{1,2}\), \(\eta^{\prime}_{1,2}\leftarrow\eta_{1,2}\), \(\psi^{\prime}\leftarrow\psi\), \(\iota^{\prime}\leftarrow\iota\)
5. Initialize a prioritized replay buffer \(\mathcal{B}\)
6.for\(\epsilon=1\) to \(N_{epochs}\)do
7. Compute the delay value for each epoch: \(d\leftarrow(e\bmod c)+b\)
8.for\(t=1\) to \(T-1\)do
9. Select a trading action \(a_{t}^{ta}\) with exploration noise \(\epsilon\sim\mathcal{N}(0,\sigma):\ a_{t}^{ta}\leftarrow\mu_{\phi}(s_{t})+\epsilon\) where \(s_{t}=\lambda_{v}\Big{(}w_{t}^{data}\Big{)}\)
10. With probability \(\varepsilon\), select a random stop-loss boundary \(a_{t}^{sl}\)
11. Otherwise, select \(a_{t}^{sl}\leftarrow\underset{a^{sl}}{\arg\max}Q_{\Psi}\Big{(}s_{t},a^{sl} \Big{)}\) where \(s_{t}=\kappa_{i}\Big{(}w_{t}^{data}\Big{)}\)
12. Observe rewards \(r_{t}^{la}\), \(r_{t}^{sl}\) and the next input \(w_{t+1}^{data}\)
13. Store a transition \(\langle w_{t}^{data},a_{t}^{ta},a_{t}^{sl},r_{t}^{la},a_{t}^{sl},w_{t+1}^{data },p_{t}\rangle\) to \(\mathcal{B}\) where \(p_{t}=\max_{i<t}p_{i}\)
14. Sample a minibatch of \(B\) transitions \(\langle w_{i}^{data},a_{i}^{ta},a_{i}^{sl},r_{i}^{la},r_{i}^{sl},w_{i+1}^{data },p_{i}\rangle\) from \(\mathcal{B}\) with \(P(i)\) in Equation (8)
15. Smooth the target policy with \(\epsilon\sim\text{clip}(\mathcal{N}(0,\sigma^{\prime}),-c,c):\ a_{t}^{sl} \leftarrow\mu_{\varphi}\Big{(}s_{i+1}^{\prime}\Big{)}+\epsilon\) where \(s_{i+1}^{\prime}=\lambda_{\varphi}\Big{(}w_{i+1}^{data}\Big{)}\)
16. Compute temporal difference errors: \(\delta_{i}^{j}\leftarrow\gamma_{critic}-Q_{\theta_{i}}\Big{(}s_{i}^{j},a_{i} ^{ta}\Big{)}\) where \(s_{i}^{j}=o_{\eta_{i}}\Big{(}w_{i}^{data}\Big{)}\)
17. Update the critics \(\theta_{j}\) by the MSE loss \(\frac{1}{B}\sum\ \big{(}\delta_{i}^{j}\big{)}\) with the importance sampling weight \(\omega_{i}\) in Equation (9)
19. Update the transition priority: \(p_{i}\leftarrow\underset{j=1,2}{\min}\Big{(}\Big{|}\delta_{i}^{j}\Big{|}+\xi \Big{)}\)
20. \(Y_{sl}\leftarrow r_{i}^{sl}+\gamma\text{Q}_{\Psi^{\prime}}(s_{i+1}^{\prime}, \underset{a^{sl}}{\arg\max}Q_{\Psi}\Big{(}s_{i+1},a^{sl}\Big{)})\) where \(s_{i+1}^{\prime}=\kappa_{\ell}\Big{(}w_{i+1}^{data}\Big{)}\), \(s_{i+1}=\kappa_{i}\Big{(}w_{i+1}^{data}\Big{)}\)
21. Update the stop-loss network \(\psi\) by the MSE loss: \(\frac{1}{B}\sum\ \big{(}Y_{sl}-Q_{\Psi}\Big{(}s_{i},a_{i}^{sl}\Big{)}\Big{)}^{2}\) where \(s_{i}=\kappa_{i}\Big{(}w_{i}^{data}\Big{)}\)
22.if\(t\bmod d\)then
23.