# A Stock Prediction Method Based on Deep Reinforcement Learning and Sentiment Analysis
## 4 Methodology
### Methods  
1. DQN and DDQN  
DQN and DDQN are set up with the same CNN structure, State, Action, Reward, and Course of dealing. State, Action, and Course of dealing are described in Section 3.1, and the CNN setup is described in Section 3.2. Reward is shown below:  
\[r_{c}^{t}=l_{c}^{t+1}\times a_{c}^{t}. \tag{15}\]  
where \(c\) denotes the company, \(t\) denotes the date, and \(l_{c}^{t+1}\) is the percentage increase in closing price. There are two ways to add sentiment analysis to the model. We can add the sentiment analysis index average \(h_{c}^{t}\) to state or we can add the sentiment analysis index average \(h_{c}^{t}\) to reward. Until we get the experimental results, we do not know which joining way is better. Therefore, we designed two methods. We first add \(h_{c}^{t}\) to the reward, with the state remaining the same as the DQN. Then, we add the \(h_{c}^{t}\) to the state, with the reward remaining the same as the DQN. These two methods are referred to as SADQN-R and SADQN-S. Specific descriptions of the two methods are shown below. 2. SADQN-R  
We add the sentiment analysis index \(h_{c}^{t}\) to the Reward of DQN and call the resulting new model Sentiment Analysis Deep Q-Network-R (SADQN-R). State, Action and Course of dealing for SADQN-R are described in Section 3.1, and the CNN setup for SADQN-R is described in Section 3.2. Reward is shown below:  
\[\mathrm{r}_{\mathrm{c}}^{t}=\mathrm{I}_{\mathrm{c}}^{t+1}\times\mathrm{a}_{ \mathrm{c}}^{t}+\left(\mathrm{h}_{\mathrm{c}}^{t+1}-0.45\right)\times\mathrm{a }_{\mathrm{c}}^{t}, \tag{16}\]  
where \(c\) denotes the company, \(t\) denotes the date, and \(\mathrm{I}_{\mathrm{c}}^{t+1}\) is the percentage increase in closing price. \(h_{c}^{t}\) is in the range [0, 1], so we set 0.45 to be the sentiment-neutral value. As shown in Figure 2, it is the model diagram of SADQN-S. The model diagram of SADQN-R differs from that of SADQN-S in that the middle two rows of the input State are all zeros. Figure 2: For the sake of convenience, we take the 10 th order matrix as an example to make a diagram. In the behavior network, the blue color in the top and bottom 4 rows of the input \(S_{t}\) represents the number 1, the yellow colour in the middle 2 rows represents the number 1, and the rest of the white color represents the number 0. The input State-0-1 matrix, through the convolutional, ReLu, and FC layers, finally outputs a value vector \(p_{\mathrm{c}}^{t}\), which predicts the value function of each of the three actions. The agent selects the action corresponding to the maximum value and then updates the network weights according to the loss functions. Sadqn-S  
We add the sentiment analysis index \(h_{c}^{t}\) to the State of DQN, and call the resulting new model Sentiment Analysis Deep Q- Network-S (SADQN-S). Action and Course of dealing for SADQN-S are described in Section 3.1, and the CNN setup for SADQN-S is described in Section 3.2. Reward is shown below:  
\[\mathbf{r}_{\text{c}}^{t}=\mathbf{I}_{c}^{t+1}\;\times\;\mathbf{a}_{\text{c}}^ {t}. \tag{17}\]  
where \(c\) denotes the company, \(t\) denotes the date, and \(\mathbf{I}_{c}^{t+1}\) is the percentage increase in closing price. In SADQN-S, State is no longer the same as described in Section 3.1. This is because in SADQN-S, the sentiment analysis index averages are added to State, and the Reward remains the same as DQN. We averaged \(h_{c}^{t}\) into 8 intervals based on data preprocessing. The sentiment index average is a number between 0 and 1. We divide the interval [0, 1] into 8 equal parts, i.e., [0, 0.125], [0.125, 0.25], [0.25, 0.375], \(\ldots\), [0.875, 1]. If \(h_{c}^{t}\) belongs to the first interval, columns 1-4 of rows 16-17 of the State-0-1 matrix are 1, and the rest are unchanged; if \(h_{c}^{t}\) belongs to the second interval, columns 5-8 of rows 16-17 of the State-0-1 matrix are 1, and the rest are unchanged; and so on. The State-0-1 matrix is shown in Figure 3 (using the State-0-1 matrix for stock #601318 on 22 February 2024 as an example, \(N=32\)).