# Hybrid Deep Reinforcement Learning for Pairs Trading
## 4 Hybrid Deep Reinforcement Learning for Pairs Trading
### Hybrid Reinforcement Learning
#### 4.4.4 Hybrid Reinforcement Learning Algorithm
In the hybrid reinforcement learning, we set the trading window size in Equation (22) to 60; transaction cost rate \(\zeta\) in Equation (23) to 0.3%; trading boundary \(tb\) for the behavior cloning to 1.0; \(c\) and \(b\) in Equation (28) for the dynamic delay to 4 and 2, respectively; \(\alpha\), \(\beta\), and \(\xi\) for the PER to 0.6, 0.4, and 0.0001, respectively; probability \(\varepsilon\) to 0.3; noise size \(\sigma\) for exploration to 0.7; noise size \(\sigma^{\prime}\) for regularization to 0.7; clipping size \(c\) to 1; mini-batch size to 32; and learning rate to 0.0001. ### Experimental Results

#### 5.2.1 Comparison with Other Methods

To compare HDRL-Trader with the other methods in detail, the smaller dataset with various price trends is used. Table 3 shows that the pairs trading methods (P-DDQN, PTDQN, P-Trader, and HDRL-Trader) achieve good profits for all price trends because pairs trading is a market-neutral trading strategy. HDRL-Trader has the highest return rate, highest SR, and lowest MDD for all price trends because it integrates all of the novel techniques in Figure 7:1 Compared with the second-best method, P-Trader, which directly determines trading actions without a stop-loss boundary, HDRL-Trader shows significantly higher performance because of the hybrid reinforcement learning with the stop-loss network. For the downward trend, the MDDs are higher than those for the other trends for all methods because they suffer from exits with a great loss due to the large divergence, as shown in Figure 7:2 Even in this situation, HDRL-Trader has the lowest MDD because it reduces significant losses using hybrid reinforcement learning with the stop-loss network. To verify the generalization ability of the methods, the larger dataset is used. As shown in Figure 12 and Table A1 in Appendix A, HDRL-Trader outperforms all other methods in terms of the minimum, maximum, and average return rate, SR, and MDD. HDRL-Trader archives an average return rate of 82.4%, which is 25.7%P higher than that of the second-best method. The average SR of HDRL-Trader is 1.24, which is 0.26 higher than the second-best method. The average MDD of HDRL-Trader is 0.36, which is 0.08 lower than the second-best method. These results indicate that the hybrid reinforcement learningalgorithm of HDRL-Trader with its novel techniques is very effective for generalization. The experimental results can be statistically analyzed using the metrics for measuring machine intelligence [54], and we leave this as future work. #### 5.2.2 Ablation Studies

To evaluate the contribution of each technique used in HDRL-Trader, ablation studies are conducted using the smaller dataset. The techniques excluded one-by-one are the stop-loss network (SL), PER, behavior cloning (BC), clustering (Clu), dimensionality reduction (Dim), regression network (Reg), and gate structure (Gat). Figure 13 shows the results, where \"All\" denotes HDRL-Trader with all techniques. All the techniques contribute significantly to the performance improvement. In particular, the hybrid with the stop