# Hybrid Deep Reinforcement Learning for Pairs Trading
## 4 Hybrid Deep Reinforcement Learning for Pairs Trading
### Hybrid Reinforcement Learning
#### 4.4.4 Hybrid Reinforcement Learning Algorithm
A smaller dataset with diverse price trends is used to compare HDRL-Trader with other methods in detail. The smaller dataset consists of six stock pairs with three different price trends in the test period (upward, sideways, and downward), as shown in Figures 9-11, respectively. A larger dataset of 20 stock pairs (Table A1 in Appendix A) is used to verify the generalization ability of the methods. For the two datasets, the period of the training data is from January



#### 5.1.2 Evaluation Metrics

We evaluate the methods in terms of the rate of return and risk indicators, such as the Sharpe ratio (SR) [53] and maximum drawdown (MDD) [2]. The rate of return is computed by Equation (29), where \(PV_{start}\) is the initial money of $10,000, and \(PV_{end}\) is the portfolio value at the end of the test period, which is the sum of the values of long and short positions and the remaining money. \[(PV_{end}-PV_{start})/PV_{start} \tag{29}\]

The SR in Equation (30) measures the return of an investment compared to its risk. In Equation (30), \(\mathbb{E}[R]\) is the expected return, and \(\sigma[R]\) is the standard deviation of the return, which measures fluctuation (i.e., risk). A higher SR indicates a higher risk-adjusted return. We use the change rate of the portfolio value at each day as the return for the day. \[\text{SR}=\frac{\mathbb{E}[R]}{\sigma[R]} \tag{30}\]

The MDD in Equation (31) measures the maximum loss rate from the peak to the trough of a portfolio over a specified period \(T\). In Equation (31), the inner max term computes the drawdown for time \(\tau\). A lower MDD indicates a lower risk. \[\text{MDD}(T)=\max_{\tau\in(0,T)}\bigg{[}\max_{t\in(0,\tau)}\frac{PV_{t}-PV_{ \tau}}{PV_{t}}\bigg{]} \tag{31}\]

Figure 11: Stock pairs trending downward. (**a**) COG, MRO. (**b**) GPS, MRO. Figure 7:0 Stock pairs trending sideways. (**a**) HIG, BK. (**b**) O, EVRG. #### 5.1.3 Baseline Methods

HDRL-Trader is compared with the state-of-the-art methods listed below. For a fair comparison, we optimize the network structures and hyperparameters for each method, as described below. The Adam optimizer is used for all methods with momentum parameters \(\beta_{1}\) of 0.9 and \(\beta_{2}\) of 0.999, an epsilon of \(10^{-7}\), and a decay of 0.99. * Buy and Hold (B&H) buys two stocks in a pair on the first day of the test period, holds them, and then sells them on the last day of the test period. * PTDQN [1] determines trading and stop-loss boundaries using a DQN. The \(Q\)-network comprises two ReLU dense layers with 15 units and a softmax output layer. We set the sliding window size to 30, mini-batch size to 32, learning rate to 0.001, and \(\varepsilon\) to 0.5 with a decay of 0.95. * P-DDQN [3] determines trading actions using a DDQN with a negative reward multiplier. The \(Q\)-network comprises two ReLU dense layers with 50 units and a softmax output layer. We set the mini-batch size to 32, learning rate to 0.0001, and \(\varepsilon\) to 0.3. * P-Trader [5] determines trading actions using a DDQN and employs techniques such as clustering, a gate structure, a temporal attention mechanism, and a regression network. The \(Q\)-network comprises two ReLU dense layers with 128 and 64 units and a softmax output layer. The regression network comprises a ReLU dense layer with 128 units and a linear output layer. We set the sliding window size to 15, mini-batch size to 32, learning rate to 0.0001, and \(\varepsilon\) to 0.5 with a decay of 0.95. #### 5.1.4 Implementation Details of HDRL-Trader

In the data preprocessing, we set the tumbling window size for the z-score normalization to 20, the dimensionality threshold \(F\) in Figure 4 to 8, and the number of clusters to 10. In the SRL, we set the sliding window size for input to 10 and the number of units of the LSTM layer to 128.