# A Stock Prediction Method Based on Deep Reinforcement Learning and Sentiment Analysis
## 4 Methodology
### Modeling the Environment for RL Task  
* State Most previous stock methods use raw data from stocks as inputs to the model, and few studies used images as inputs in stock prediction. Research [44] tried to use images as inputs in their paper and improved the performance of the model. Research [18] applied DRL to stock prediction by using images as input to the model and greatly improved the model's return. Therefore, we believe that further research on the direct use of images as input is necessary. We call the input State the State-0-1 matrix. The State-0-1 matrix consists of the closing price, volume after max-min normalization. The closing price of a stock is the price at which the last trade in that stock was made on a business day. If there are no transactions on the day, the closing price of the previous business day will be the closing price of the day. Stock volume is the number of trades made between buyers and sellers of a stock and is unilateral. For example, a stock with a volume of 100,000 shares means that the buyer bought 100,000 shares while the seller sold 100,000 shares. The State-0-1 matrix (Algorithm A2. This algorithm is shown in Appendix A) is an \(N\)-order square matrix of 0 s and 1 s (\(N\) columns represent data for the last \(N\) days), and the matrix is categorized into upper, middle, and lower matrices. The middle matrix (2 rows and \(N\) columns) is full of zeros and is used to help the agent distinguish between closing price and volume. The upper matrix \(((N-2)/2\) rows and \(N\) columns) is the closing price: the normalized closing price is divided into m intervals, if the closing price on day \(x\) is in the maximum interval, the first row of column \(x\) is 1, and each of the rest of column \(x\) is 0; if the closing price on day \(x\) is in the second-largest interval, the second row of column \(x\) is 1, and each of the rest of the columns \(x\) is 0; if the closing price on day \(x\) is in the minimum interval, the \((N-2)/2\)th row of column \(x\) is 1, and each of the rest of column \(x\) is 0; and so on. The lower matrix \((N-2)/2\) rows and N columns) is the volume and is generated in the same way as the upper matrix. In this paper, we set \(N\) to 32. The State-0-1 matrix is shown in Figure 1 (using the State-0-1 matrix for stock #601318 on 22 February 2024 as an example, \(N=32\)). We use blue for the number 1 and white for the number 0. The State of company \(c\) at moment \(t\) is denoted as \(s_{c}^{t}\) (c denotes the company and \(t\) denotes the date). * Action The three actions are long (\(a_{c}^{t}=1\)), neutral (\(a_{c}^{t}=0\)), and short (\(a_{c}^{t}=-1\)) (c denotes the company and \(t\) denotes the date). * Reward DQN, DDQN, SADQN-R, and SADQN-S all have different reward settings, as described in Section 3.3. * Course of dealing Input the stock's data for the past 32 days to the agent, which outputs a vector of value functions \(\rho_{c}^{t}\) (c denotes the company and t denotes the date). The value function vector \(\rho_{c}^{t}\) is a three-dimensional column vector with three values approximating the value functions of each of the three actions. We can benefit the next day by choosing the action that has the largest value function. When investing in stocks, it is too risky to invest in only one stock, so it is necessary to set up a portfolio that is invested according to weight \(\alpha_{c}\). Assuming that the total number of stocks at the time of conducting the portfolio is \(M\), at  
Figure 1: State-0-1 matrix for Stock #601318 at 22 February 2024.  
moment \(t\), our model processes \(M\) stocks simultaneously and constructs the weights using the output vector of value functions \(\rho_{c}^{t}\). The weights are positive if the value function of the long action is the largest, and negative if the value function of the short action is the largest. In order to have an intuitive result, we set the total assets to 1. A positive weight represents the execution of a long action. A negative weight represents the execution of a short action.