# Hybrid Deep Reinforcement Learning for Pairs Trading
## 4 Hybrid Deep Reinforcement Learning for Pairs Trading
### Hybrid Reinforcement Learning
#### 4.4.2 Reward Functions for Pairs Trading  
The reward function for the trading actions is designed to consider both the risk of exit and transaction cost. An exit usually results in a loss because the positions are closed by force at the end of the trading window (or the exit time), even if the spread has not returned to the mean. In each trading window, the ratio \(RT_{t}^{exit}\) of time interval between the exit time \(t_{exit}\) and position closing time \(t_{close}\) is computed by Equation (22). If the positions are closed near the exit time (i.e., \(RT_{t}^{exit}\) is small), we consider that the risk of exit is high.  
\[RT_{t}^{exit}=\frac{t_{exit}-t_{close}}{|\text{trading window}|} \tag{22}\]  
In a real trading environment, there are transaction costs, such as fees, taxes, and trading slippages. Here, slippage is the difference between the expected price and actual price at which a trade is executed. The transaction cost is factored into the stock price,  
Figure 5: State representation learning. (**a**) Gate structure. (**b**) Architecture of the SRL.  
as in Equation (23), where \(\zeta\) is the transaction cost rate used to discount the price. The discounted price \(price_{t}\) is applied when the stocks are sold.  
\[price_{t}=price_{t}\times(1-\zeta) \tag{23}\]  
We use the returns of the long and short positions to define the reward function. The return \(R_{t}^{long}\) for the long position is defined in Equation (24), where \(n^{long}\) is the number of shares, and \(t_{close}\) and \(t\) are the position closing time and current time, respectively. The difference in stock prices is normalized by the stock price at \(t\). The stock price at \(t_{close}\) is discounted by the transaction cost rate because the stock is sold at \(t_{close}\). Similarly, the return \(R_{t}^{short}\) for the short position is defined in Equation (25).  
\[R_{t}^{long}=n^{long}\times\left(\vec{price}_{t_{close}}^{long}-price_{t}^{ long}\right)/price_{t}^{long} \tag{24}\]  
\[R_{t}^{short}=n^{short}\times\left(\vec{price}_{t}^{short}-price_{t_{close}}^ {short}\right)/\vec{price}_{t}^{short} \tag{25}\]  
We define the reward \(r_{t}^{ta}\) for trading actions in Equation (26). For a normal close, the reward is the portfolio return (\(R_{t}^{long}+R_{t}^{short}\)) multiplied by \(RT_{t}^{exit}\), which is designed to reduce the risk of exit. For an exit or stop-loss close, the reward is the portfolio return itself, which is usually negative.  
\[r_{t}^{ta}=\left\{\begin{array}{ll}\left(R_{t}^{long}+R_{t}^{short}\right) \times RT_{t}^{exit}&\text{if a normal close occurs}\\ R_{t}^{long}+R_{t}^{short}&\text{if an exit or stop-loss close occurs}\\ 0&\text{otherwise}\end{array}\right. \tag{26}\]  
We define the reward \(r_{t}^{sl}\) for the stop-loss boundary in Equation (27). When the current spread \(S_{t}\) touches or transgresses the stop-loss boundary \(a_{t}^{sl}\) (i.e., \(S_{t}/a_{t}^{sl}\geq 1\)), the reward is computed as the rate of change between the current spread \(S_{t}\) and next spread \(S_{t+1}\). If \(S_{t+1}\) diverges more than \(S_{t}\) (i.e., \(S_{t+1}/S_{t}>1\)), the reward is positive, which indicates that the stop-loss boundary is correctly determined. If \(S_{t+1}/S_{t}<1\), then the reward is negative. When the current spread \(S_{t}\) diverges more than the spread \(S_{o}\) at the position opening (i.e., \(S_{t}/S_{o}>1\)), the reward is computed as the rate of change between \(S_{o}\) and \(S_{t}\). In this case, the reward is always negative, which indicates that the stop-loss boundary is incorrectly determined.  
\[r_{t}^{sl}=\left\{\begin{array}{ll}\frac{S_{t+1}-S_{t}}{S_{t}}&\text{if} \ \frac{S_{t}}{a_{t}^{sl}}\geq 1\\ \frac{S_{o}-S_{t}}{S_{o}}&\text{if}\ \frac{S_{t}}{S_{o}}>1\\ 0&\text{otherwise}\end{array}\right. \tag{27}\]