# Hybrid Deep Reinforcement Learning for Pairs Trading
## 4 Hybrid Deep Reinforcement Learning for Pairs Trading
### Hybrid Reinforcement Learning
#### 4.4.4 Hybrid Reinforcement Learning Algorithm  
Figure 7 illustrates the proposed architecture for hybrid reinforcement learning, which combines the extended TD3 and DDQN algorithms. The extended TD3 algorithm uses an actor network \(\mu_{\phi}\), SRL network \(\lambda_{v}\) for \(\mu_{\phi}\), regression network that shares \(\lambda_{v}\) with the actor network, two critic networks \(Q_{\theta_{1}}\) and \(Q_{\theta_{2}}\), and two SRL networks \(o_{\eta_{1}}\) and \(o_{\eta_{2}}\) for \(Q_{\theta_{1}}\) and \(Q_{\theta_{2}}\), respectively. The actor network is trained with behavior cloning. The extended DDQN algorithm uses a \(Q\)-network \(Q_{\psi}\) for the stop-loss boundary (called the stop-loss network) and the SRL network \(\kappa_{t}\) for \(Q_{\psi}\). All networks, except the regression network, have corresponding target networks. In Figure 7, the SRL networks correspond to the sensors, and the trading system corresponds to the effector. Figure 8 shows the structure of each network. Figure 6: An example of the expert’s actions. Figure 7: Architecture of the hybrid deep reinforcement learning. Algorithm 1 presents the proposed hybrid reinforcement learning algorithm. The input for the SRL networks is a sliding window \(w_{t}^{data}\) of weighted feature vectors \(\{x_{t-N_{w}+1},\ldots,\)\(x_{t-1},x_{t}\}\) obtained from data preprocessing. After interacting with the environment, a transition of \(\left<w_{t}^{data},a_{t}^{ta},a_{t}^{sl},r_{t}^{la},\)\(r_{t}^{sl},w_{t+1}^{data},p_{t}\right>\) is stored to \(\mathcal{B}\) where the priority \(p_{t}\) is set to the maximum (line 13). Using the sampling probability in Equation (8), a minibatch of transitions is sampled (line 14). In the extended TD3 algorithm, the actor network \(\mu_{\phi}\) is combined with the SRL network \(\lambda_{v}\), and updates on \(\mu_{\phi}\) are back-propagated to \(\lambda_{v}\). In the actor network shown in Figure 8a, a softmax layer is used as the output layer to support discrete actions. To select an action with exploration noise, a different amount of noise is added to each output of the softmax layer (line 9), and then argmax is applied to the outputs. To avoid overfitting, we employ the target policy smoothing technique of TD3 (line 15). To provide accurate state information to the actor network, we train the regression network, whose structure is shown in Figure 8b, by minimizing the MSE between the real and predicted closing spreads (line 24). The actor network is indirectly affected by this training through the underlying SRL network \(\lambda_{v}\), which is shared with the regression network. To learn the expert's action \(a_{expert}^{ta}\), we train the actor network by minimizing the cross-entropy (CE) loss between the softmax output vector of \(a_{i}^{ta}\) and the expert's action \(a_{expert}^{ta}\), which is represented as a one-hot vector (line 25). The critic networks \(Q_{\theta_{1}}\) and \(Q_{\theta_{2}}\), whose structures are shown in Figure 8c, are combined with the corresponding SRL networks \(o_{\eta_{1}}\) and \(o_{\eta_{2}}\), respectively. To avoid overestimation, we employ the clipped double \(Q\)-learning technique of TD3 when computing the target value \(Y_{critic}\) (line 16). \(Y_{critic}\) is used in computing the temporal difference errors (line 17). The critic networks are updated by minimizing the MSE loss with the importance sampling weight \(\omega_{i}\) in Equation (9) (line 18). The updates on each critic network are back-propagated to the corresponding SRL network. The transition priority \(p_{i}\) is updated using the minimum of the temporal difference errors (line 19). In contrast to the static delay used in the delayed policy updates technique of TD3, the dynamic delay technique [36] is applied when updating the actor and target networks for more stable and efficient training. At each epoch, the delay value \(d\) is computed using Equation (28), where \(c\) and \(b\) are parameters for controlling the variance and minimum delay value, respectively (line 7).