# Hybrid Deep Reinforcement Learning for Pairs Trading
## 4 Hybrid Deep Reinforcement Learning for Pairs Trading
### Hybrid Reinforcement Learning
#### 4.4.4 Hybrid Reinforcement Learning Algorithm
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multicolumn{6}{c}{**Rate of Return (the Higher the Better)**} \\ \hline \multicolumn{2}{c}{**Stock Pairs**} & \multicolumn{1}{c}{**B\&H**} & \multicolumn{1}{c}{**P-DDQN**} & \multicolumn{1}{c}{**PTDQN**} & \multicolumn{1}{c}{**P-Trader**} & \multicolumn{1}{c}{**HDRL-Trader**} \\ \hline \multirow{2}{*}{Upward Trending} & A, PLD & 76.8\% & 92.9\% & 115.3\% & 109.2\% & **151.7\%** \\ \cline{2-6}  & ARE, DRL & 46.5\% & 42.9\% & 75.6\% & 91.8\% & **129.0\%** \\ \hline \multirow{2}{*}{Sideways Trending} & HIG, BK & \(-0.4\%\) & 27.8\% & 51.8\% & 49.6\% & **79.8\%** \\ \cline{2-6}  & O, EVRG & 0.8\% & 59.2\% & 65.8\% & 73.5\% & **104.0\%** \\ \hline \multirow{2}{*}{Downward Trending} & COG, MRO & \(-42.8\%\) & 107.0\% & 104.5\% & 99.1\% & **132.3\%** \\ \cline{2-6}  & GPS, MRO & \(-38.2\%\) & 37.8\% & 13.7\% & 37.5\% & **64.0\%** \\ \hline \multicolumn{2}{c}{Minimum} & \(-42.8\%\) & 27.8\% & 13.7\% & 37.5\% & **64.0\%** \\ \hline \multicolumn{2}{c}{Maximum} & 76.8\% & 107.0\% & 115.3\% & 109.2\% & **151.7\%** \\ \hline \multicolumn{2}{c}{Average} & 7.1\% & 61.2\% & 71.1\% & 76.8\% & **110.1\%** \\ \hline \hline \multicolumn{6}{c}{**Sharpe Ratio (the Higher the Better)**} \\ \hline \multicolumn{2}{c}{**Stock Pairs**} & \multicolumn{1}{c}{**B\&H**} & \multicolumn{1}{c}{**P-DDQN**} & \multicolumn{1}{c}{**PTDQN**} & \multicolumn{1}{c}{**P-Trader**} & \multicolumn{1}{c}{**HDRL-Trader**} \\ \hline \multirow{2}{*}{Upward Trending} & A, PLD & 1.57 & 1.88 & 1.82 & 2.06 & **2.25** \\ \cline{2-6}  & ARE, DRL & 1.13 & 1.26 & 1.72 & 1.49 & **1.74** \\ \hline \multirow{2}{*}{Sideways Trending} & HIG, BK & 0.25 & 0.82 & 0.98 & 0.86 & **1.25** \\ \cline{2-6}  & O, EVRG & 0.30 & 0.66 & 0.84 & 1.20 & **1.64** \\ \hline \multirow{2}{*}{Downward Trending} & COG, MRO & \(-0.62\) & 1.49 & 1.16 & 1.33 & **1.61** \\ \cline{2-6}  & GPS, MRO & \(-0.12\) & 0.54 & 0.48 & 0.63 & **0.93** \\ \hline \multicolumn{2}{c}{Minimum} & \(-0.62\) & 0.54 & 0.48 & 0.63 & **0.93** \\ \hline \multicolumn{2}{c}{Maximum} & 1.57 & 1.88 & 1.82 & 2.06 & **2.25** \\ \hline \multicolumn{2}{c}{Average} & 0.42 & 1.11 & 1.17 & 1.26 & **1.57** \\ \hline \multicolumn{2}{c}{**Maximum Drawdown (the Lower the Better)**} \\ \hline \multicolumn{2}{c}{**Stock Pairs**} & \multicolumn{1}{c}{**B\&H**} & \multicolumn{1}{c}{**P-DDQN**} & \multicolumn{1}{c}{**PTDQN**} & \multicolumn{1}{c}{**P-Trader**} & \multicolumn{1}{c}{**HDRL-Trader**} \\ \hline \multirow{2}{*}{Upward Trending} & A, PLD & 0.45 & 0.38 & 0.40 & 0.27 & **0.24** \\ \cline{2-6}  & ARE, DRL & 0.36 & 0.30 & 0.35 & 0.33 & **0.22** \\ \hline \multirow{2}{*}{Sideways Trending} & HIG, BK & 0.43 & 0.25 & 0.39 & 0.21 & **0.16** \\ \cline{2-6}  & O, EVRG & 0.51 & 0.49 & 0.35 & 0.36 & **0.21** \\ \hline \multirow{2}{*}{Downward Trending} & COG, MRO & 0.65 & 0.70 & 0.59 & 0.46 & **0.43** \\ \cline{2-6}  & GPS, MRO & 0.81 & 0.51 & 0.53 & 0.52 & **0.48** \\ \hline \multicolumn{2}{c}{Minimum} & 0.36 & 0.25 & 0.35 & 0.21 & **0.16** \\ \hline \multicolumn{2}{c}{Maximum} & 0.81 & 0.70 & 0.59 & 0.52 & **0.48** \\ \hline \multicolumn{2}{c}{Average} & 0.53 & 0.44 & 0.43 & 0.36 & **0.29** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Experimental results on the smaller dataset. loss network is very crucial because it reduces significant losses. For the dynamic delay technique, we compare its performance with those of static delays ranging from two to five. Figure 14 shows that the dynamic delay outperforms all the static delays, demonstrating its contribution. #### 5.2.3 Comparison with Other Hyperparameter Values

Figure 15 shows the effect of the major hyperparameters used in HDRL-Trader, i.e., the dimensionality reduction threshold, number of clusters, sliding window size, and noise size for exploration. The average return rate is evaluated on the smaller dataset. As shown in Figure 15a-c, the performance is degraded if the dimensionality threshold, number of clusters, or sliding window size is set too small or too large.