# A Stock Prediction Method Based on Deep Reinforcement Learning and Sentiment Analysis
## 3 Preliminaries
### Deep Reinforcement Learning
#### 3.3.2 DQN and DDQN  
DQN (Algorithm A1. This algorithm is shown in Appendix A), as one of the most classical DRL algorithms, has been applied in several fields with good results. In RL, \(Q(s,\ a)\) is called the action value function, which represents the expected value of the future return in the case of being in state \(s\) and taking action \(a\). Using the Q-Learning algorithm, we can get a Q-table (Table 1), which contains the Q-values for taking different actions \(a\) (\(a\in A\)) in all states \(s\) (s \(\in S\)). After initializing the Q-table, the Q-learning algorithm uses the Bellman equation to update the Q-table by iterating during the interaction between the agent and the environment until convergence. Before explaining the DQN algorithm further, we explain the Bellman equation. MDP can be modeled using Equation (6). \[p\left(s^{{}^{\prime}},r\Big{|}s,a\right)=Pr\left(s_{t+1}=s^{{}^{\prime}},r_{t +1}=r\Big{|}s_{t}=s,a_{t}=a\right), \tag{6}\]  
where \(Pr\) refers to the probability, \(s\) refers to the state at a particular moment, \(s^{{}^{\prime}}\) refers to the state at the next moment, \(a\) is the action performed at this particular moment, and \(r\) is the reward that the agent receives for performing \(a\) this action. Therefore, Equation (7) can be obtained by deriving Equation (4). \[ \begin{array}{rcl}v_{\pi}(s)&\triangleq&\mathbb{E}_{\pi}[G_{t}\ |\ S_{t}=s]\\ &=&\mathbb{E}_{\pi}[R_{t+1}+\gamma G_{t+1}\ |\ S_{t}=s]\\ &=&\sum\limits_{a}\pi(a|s)\sum\limits_{s^{{}^{\prime}}}\frac{\gamma}{Pr}\left( s^{{}^{\prime}},r\Big{|}s,a\right)\left[r+\gamma\mathbb{E}_{\pi}\Big{[}G_{t+1} \Big{|}S_{t+1}=s^{{}^{\prime}}\Big{]}\right]\\ &=&\sum\limits_{a}\pi(a|s)\sum\limits_{s^{{}^{\prime}}}r\ p\left(s^{{}^{ \prime}},r\Big{|}s,a\right)\left[r+\gamma\upsilon_{\pi}\Big{(}s^{{}^{\prime}} \Big{)}\right]\text{, for all }s\in S\text{,}\end{array}  \tag{7}\]  
where \(S\) is the state space, \(s\) refers to the state at a particular moment, \(s^{{}^{\prime}}\) refers to the state at the next moment, \(a\) is the action performed at this particular moment, and \(r\) is the reward that the agent receives for performing \(a\) this action. The last line of Equation (7) is the Bellman equation for \(v_{\pi}(s)\). With the final Q-table obtained, the agent can then determine what action to take in a certain state \(s\) to obtain the maximum Q-value. The iteration is as follows:\[Q(s_{t},a_{t})\gets Q(s_{t},a_{t})+\alpha\Big{[}R_{t+1}+\gamma\underset{a}{max}Q(s _{t+1},a)-Q(s_{t},a_{t})\Big{]}, \tag{8}\]  
where \(\alpha\) is the learning rate, which is used to specify the magnitude of the update, and \(\gamma\) is the discount rate, which is used to ensure that the iteration can converge. The \(r_{t+1}\) in Equation (8) is the reward given to the agent by the environment after the agent performs the action. However, most real-life RL problems possess infinite states and it is not possible to solve such RL problems using Q-table. To solve this type of problem, we can parameterize the value function and approximate \(Q(s,a)\) by \(Q(s,a;\theta_{i})\). The function \(Q(s,a;\theta_{i})\) can be generated using a neural network, which we call Deep Q-network, where \(\theta_{i}\) is the parameter on which the neural network is to be trained. The parameter update process is as follows:  
1. Set up two networks, the behavior network \(Q(s_{t},a_{t};\theta_{i})\) and the target network \(Q\big{(}s_{t+1},a_{t+1};\theta_{i}^{-}\big{)}\). The behavior network is responsible for controlling agent and collecting experience; target network is responsible for computing \(Y_{i}\): \[Y_{i}=r_{t+1}+\gamma\underset{a_{t+1}}{max}Q\big{(}s_{t+1},a_{t+1};\theta_{i} ^{-}\big{)}\ ;\] (9) where i is the number of iterations and \(\gamma\) is the discount rate. The \(r_{t+1}\) in Equation (9) is the reward given to the agent by the environment after the agent performs the action. 2.