# Hybrid Deep Reinforcement Learning for Pairs Trading
## 4 Hybrid Deep Reinforcement Learning for Pairs Trading
### Hybrid Reinforcement Learning
#### 4.4.4 Hybrid Reinforcement Learning Algorithm
Update the actor \(\phi\) by the deterministic policy gradient: \(\frac{1}{B}\sum\ \
abla\text{Q}_{\theta_{1}}\big{(}s_{i}^{1},a_{i}^{ta}\big{)}\) where \(s_{i}^{1}=o_{\eta_{i}}\Big{(}w_{i}^{data}\Big{)}\), \(a_{i}^{ta}=\mu_{\phi}\Big{(}\lambda_{v}\Big{(}w_{i}^{data}\Big{)}\Big{)}\)
24. Update the regression network by the MSE loss: \(\frac{1}{B}\sum\ (spread_{i+1}^{\prime}-spread_{i+1}^{predicted}\big{)}^{2}\)
25. Update the actor \(\phi\) by the CE loss for the behavior cloning: \(\frac{1}{B}\sum\ \
abla\text{CE}\Big{(}a_{i}^{ta},a_{expert}^{a}\Big{)}\) where \(a_{i}^{ta}=\mu_{\phi}\Big{(}\lambda_{v}\Big{(}w_{i}^{data}\Big{)}\Big{)}\)
26. Soft-update the target networks: \(\phi^{\prime}\leftarrow\tau\phi+(1-\tau)\phi^{\prime}\), \(v^{\prime}\leftarrow\tau v+(1-\tau)v^{\prime}\), \(\theta_{1,2}^{\prime}+\tau\theta_{1,2}+(1-\tau)\theta_{1,2}^{\prime}\), \(\psi^{\prime}\leftarrow\tau\psi+(1-\tau)\psi^{\prime}\), \(\iota^{\prime}\leftarrow\tau+(1-\tau)\iota^{\prime}\)
27.endif
28.endfor
29.endfor
```  
**Algorithm 1**The hybrid reinforcement learning algorithm of HDRL-Trader  
In the extended DDQN algorithm (red parts in Algorithm 1), the stop-loss network \(Q_{\Psi}\), whose structure is shown in Figure 8d, is combined with the SRL network \(\kappa_{i}\) shown in Figure 7. To support discrete stop-loss boundaries, a softmax layer is used as the output layer of the stop-loss network. An action \(a_{t}^{sl}\) is selected using the \(\varepsilon\)-greedy strategy (lines 10 and 11). The target value \(Y_{sl}\) is computed as in the DDQN, and then, the stop-loss network is updated using \(Y_{sl}\) (lines 20 and 21). When the stop-loss network is updated, the SRL network \(\kappa_{i}\) is also updated by backpropagation. ```
0: Sliding-window data \(w_{t}^{data}\leftarrow\{x_{t-N_{w}+1},\ \ldots,\ x_{t-1},\ x_{t}\}\) obtained from the data preprocessing
0: actor network \(\mu_{\Psi}\), SRL network \(\lambda_{\Psi}\) for \(\mu_{\Psi}\), stop-loss network \(Q_{\Psi}\), SRL network \(\kappa_{i}\) for \(Q_{\Psi}\)
1. Initialize an actor network \(\mu_{\Psi}\), an SRL network \(\lambda_{\Psi}\) for \(\mu_{\Psi}\), and a regression network
2. Initialize critic networks \(Q_{\Psi}\), \(Q_{\Psi}\), SRL networks \(\sigma_{\eta_{1}}\), \(\sigma_{\eta_{2}}\) for \(Q_{\Phi}\), \(Q_{\Phi}\), \(Q_{\Phi_{2}}\)
3. Initialize a stop-loss network \(Q_{\Psi}\), an SRL network \(\kappa_{i}\) for \(Q_{\Psi}\)
4. Initialize target networks : \(\phi^{\prime}\leftarrow\phi\), \(v^{\prime}\gets v\), \(\theta^{\prime}_{1,2}\leftarrow\theta_{1,2}\), \(\eta^{\prime}_{1,2}\leftarrow\eta_{1,2}\), \(\psi^{\prime}\leftarrow\psi\), \(\iota^{\prime}\leftarrow\iota\)
5. Initialize a prioritized replay buffer \(\mathcal{B}\)
6.for\(\epsilon=1\) to \(N_{epochs}\)do
7. Compute the delay value for each epoch: \(d\leftarrow(e\bmod c)+b\)
8.for\(t=1\) to \(T-1\)do
9. Select a trading action \(a_{t}^{ta}\) with exploration noise \(\epsilon\sim\mathcal{N}(0,\sigma):\ a_{t}^{ta}\leftarrow\mu_{\phi}(s_{t})+\epsilon\) where \(s_{t}=\lambda_{v}\

### Discussion

Table 2 summarizes the pairs trading methods in terms of their data preprocessing, SRL, and reinforcement learning abilities. The notation \"\(\bigcirc\)\" in Table 2 indicates that the corresponding technique is used, and \"\(\times\)\" indicates that it is not used. HDRL-Trader is the only method that integrates all novel techniques (dimensionality reduction, clustering, gate structure, regression model, behavior cloning, PER, dynamic delay, and hybrid reinforcement learning). ## 5 Experiments and Results

### Experimental Setup

#### 5.1.1 Datasets for Training and Testing

We evaluate pairs trading methods using two datasets with different numbers of stock pairs in the S&P 500 index. The stock data are obtained from Yahoo Finance [52]. Stock pairs are selected if their absolute Pearson's correlation coefficient is greater than or equal to 0.85, and the \(p\)-value of their augmented Dickey-Fuller test is less than or equal to 0.05 for the training period.