# Hybrid Deep Reinforcement Learning for Pairs Trading
## 2 Background
### Reinforcement Learning
#### 2.2.1 Deep Q-Network  
In value-based reinforcement learning, the agent estimates the expected discounted return, or value, for each state according to Equation (3) or for each state and action pair according to Equation (4). The \(\epsilon\)-greedy strategy is widely used to derive a new policy \(\pi^{\prime}\) from \(Q^{\pi}(s,a)\). With probability \((1-\epsilon)\), the agent performs the action with the maximum \(Q\)-value (i.e., greedy action), that is, \(\pi^{\prime}(s)=\operatorname*{argmax}_{a\in A}Q^{\pi}(s,a)\). With probability \(\epsilon\), the agent performs a random action for exploration.  
\[V^{\pi}(s)=\mathbb{E}_{a\sim\pi}[G_{t}|S_{t}=s] \tag{3}\]\[Q^{n}(s,a)=\mathbb{E}_{a\sim\pi}[G_{t}|S_{t}=s,\ A_{t}=a] \tag{4}\]  
For a large state and action space, a deep \(Q\)-network (DQN) [16] employs deep neural networks to approximate \(Q\)-values. The techniques of experience replay and, using a target network, enable stable learning. An experience replay buffer stores transitions as tuples of \(\langle s_{t},a_{t},r,s_{t+1}\rangle\), and the agent learns by sampling batches from the buffer. The online network \(Q_{\theta}\) is periodically copied to the target network \(Q_{\theta^{\prime}}\) during learning. The mean squared error (MSE) between the target value \(Y_{\text{DQN}}\) in Equation (5) and \(Q_{\theta}(s_{t},a_{t})\) is the loss function of the DQN (Equation (6)), which uses the Bellman equation [17].  
\[Y_{\text{DQN}}=r+\gamma\underset{a}{\max}Q_{\theta^{\prime}}(s_{t+1},a) \tag{5}\]  
\[Loss_{\text{DQN}}=\mathbb{E}\Big{[}(Y_{\text{DQN}}-Q_{\theta}(s_{t},a_{t}))^ {2}\Big{]} \tag{6}\]  
The DDQN solves the overestimation problem of the DQN by selecting the greedy action using \(Q_{\theta}\) and estimating the \(Q\)-value of the action using \(Q_{\theta^{\prime}}\), as shown in Equation (7).  
\[Y_{\text{DDQN}}=r+\gamma Q_{\theta^{\prime}}\bigg{(}s_{t+1},\ \underset{a}{ \text{argmax}}\ Q_{\theta}(s_{t+1},a)\bigg{)} \tag{7}\]  
The PER [18] samples important transitions more frequently to learn more efficiently. The probability \(P(i)\) of transition \(i\) being sampled is defined in Equation (8). The \(p_{i}\) in Equation (8) is the priority of transition \(i\), which is computed using the temporal difference error \(\delta_{i}=Y_{\text{DDQN}}-Q_{\theta}(s,a)\). There are two variants of prioritization: proportional and rank-based prioritization. In proportional prioritization, \(p_{i}=|\delta_{i}|+\xi\), where \(\xi\) is a small positive number used to ensure a non-zero probability for transitions with \(\delta_{i}=0\). In rank-based prioritization, \(p_{i}=\frac{1}{rank(i)}\), where \(rank(i)\) is the rank of transition \(i\) when the replay buffer is sorted by \(|\delta_{i}|\). To guarantee that every transition is sampled at least once, new transitions have the maximum priority. The \(\alpha\) in Equation (8) determines the extent to which the probability is affected by priority. By adjusting \(\alpha\), we can interpolate between greedy prioritization and uniform sampling (\(\alpha=0\)).  
\[P(i)=\frac{p_{i}^{\alpha}}{\sum_{k}p_{k}^{\alpha}} \tag{8}\]  
Non-uniform sampling in the PER results in a biased estimate of \(Q\)-values. To correct this bias, the PER employs importance sampling weights according to Equation (9), where \(B\) is the size of the replay buffer, \(\beta\) is the bias-annealing factor for annealing the amount of importance sampling correction over time, and \(\max_{k}\omega_{k}\) is the maximum weight for normalizing weights. The weights are incorporated into the \(Q\)-learning updates by using \(\omega_{i}\delta_{i}\) instead of \(\delta_{i}\).  
\[\omega_{i}=(B\cdot P(i))^{-\beta}/\max_{k}\omega_{k} \tag{9}\]  
Rainbow DQN [19] is a comprehensive improvement of the DQN that combines several techniques, such as the DDQN, PER, dueling networks [20], multistep learning [21], distributional reinforcement learning [22], and noisy networks [23].