# Hybrid Deep Reinforcement Learning for Pairs Trading
## 1 Introduction  
Pairs trading is a statistical arbitrage strategy that exploits short-term price divergences between a pair of assets that have historically moved together. In the stock market, pairs traders simultaneously open a short position in an overvalued stock and a long position in an undervalued stock. When the prices of the two stocks converge, the opened positions are closed by taking the opposite positions. Figure 1 illustrates the similar price movements of a stock pair of A. O. Smith Corp. (AOS, Milwaukee, WI, USA) and Carnival Corp. (CCL, Miami, FL, USA) in the Standard & Poor's (S&P) 500 index, which is a collection of the stocks of 500 large companies in the United States. As shown in the figure, pairs trading exploits divergence and convergence movements using both long and short positions. To find stock pairs with arbitrage opportunities, we check whether the spread, which is the difference between the prices of two stocks, forms a mean-reverting stationary process. Figure 2 illustrates the normalized spread of the stock pair shown in Figure 1. Traditionally, pairs trading methods open positions when the spread touches a trading boundary (e.g., \(T_{1}\) in Figure 2), which is predetermined as a constant multiple of the standard deviation of the spread. The constant value may vary for different pairs-trading methods. When the spread returns to the mean (e.g., \(T_{2}\) in Figure 2) or the specified trading window ends, the positions are closed. The spread may diverge too far from the mean (e.g., \(T_{3}\) in Figure 2), possibly resulting in a great loss. To limit potential losses in such situations, pairs trading methods set stop-loss boundaries to close the positions by force. Algorithmic trading enables pairs traders to execute complex trading strategies without human intervention. In recent years, deep reinforcement learning has received considerable attention in the field of algorithmic trading because it has both the perception ability of deep learning and the decision-making ability of reinforcement learning. Pairs trading methods based on deep reinforcement learning can be classified into two approaches: (1) indirectly determining trading actions, such as long and short, based on the trading and stop-loss boundaries [1,2] and (2) directly determining trading actions based on the spread [3, 4, 5]. The former approach has a fixed gap between the trading and stop-loss boundaries, as shown in Figure 2, which is certainly not optimal because the trading boundary is completely dependent on the stop-loss boundary. The latter approach has no stop-loss boundaries, which presents the risk of a great loss. In this paper, we propose a hybrid deep reinforcement learning method for pairs trading called HDRL-Trader, which overcomes the disadvantages of the two aforementioned approaches. HDRL-Trader uses two independent reinforcement learning networks: one for determining trading actions and the other for determining stop-loss boundaries. First, it extracts robust features from observations by applying dimensionality reduction and clustering. Second, an accurate state representation is derived from the preprocessed features using state representation learning (SRL). This SRL method co-trains a regression model that predicts the next spread alongside a reinforcement learning model. Third, reinforcement learning and behavior cloning are combined to learn the behavior of a prophetic expert who sees future spread movements. Fourth, the twin delayed deep deterministic policy gradient (TD3) algorithm [6] is extended to determine trading actions for pairs trading. The SRL method, behavior cloning, prioritized experience replay (PER), and dynamic delay are all incorporated into the extended TD3 algorithm. Fifth, the double deep \(Q\)-network (DDQN) algorithm [7] is extended to determine stop-loss boundaries and combined with the extended TD3 algorithm. Compared with state-of-the-art pairs trading  
Figure 1: Similar price movements of a stock pair (AOS and CCL).