# A Stock Prediction Method Based on Deep Reinforcement Learning and Sentiment Analysis
## 3 Preliminaries
### Deep Reinforcement Learning  
#### 3.3.1 Basic Knowledge  
RL is a computational approach to understanding and automating goal-directed learning and decision making. It is distinguished from other computational approaches by its emphasis on learning by an agent from direct interaction with its environment, without relying on exemplary supervision or complete models of the environment [38]. From a mathematical point of view, the RL problem is an idealized Markov Decision Process (MDP) with a theoretical framework for achieving goals through interactive learning. In RL, the agent is responsible for learning and making decisions, and everything outside the agent that interacts with it is called the environment. The agent is in a \"state\" at every moment in the process of continuous exploration. In RL, the interaction between the agent and the environment is called \"action\". In other words, \"action\" is an action that the agent can take. The agent and the environment interact over a series of discrete time steps. The time step is denoted by \(t\), \(t=1,2,3,\cdots\). The interaction between the agent and the environment is continuous: in the current state (\(S_{t}\)), the agent chooses action (\(A_{t}\)), the environment responds to the action and presents a new state (\(S_{t+1}\)) to the agent, and the environment generates a reward (\(R_{t+1}\)), the goal of the agent is to maximize the cumulative reward it receives in the long run. Cumulative reward refers to the return (\(G_{t}\)):  
\[G_{t}\triangleq R_{t+1}+R_{t+2}+R_{t+3}+\cdots+R_{T}. \tag{2}\]  
In Equation (2), \(t\) refers to the current moment and \(T\) is the moment when the interaction between the agent and the environment ends. However, in many cases, the interaction between the agent and the environment continues without restriction. At this point \(G_{t}\) does not necessarily converge. Therefore, we need to introduce another concept called discounting. Adding the discount rate allows \(G_{t}\) to be convergent when the interaction goes on indefinitely. As shown below:  
\[G_{t}\triangleq R_{t+1}+\gamma R_{t+2}+\gamma^{2}R_{t+3}+\cdots=\sum_{k=0}^{ \infty}\gamma^{k}R_{t+k+1}, \tag{3}\]  
where \(\gamma\) is a parameter, \(0\leq\gamma\leq 1\), called the discount rate. The value function is divided into state-value function and action-value function. The state-value function is used to estimate the expected return of the agent in the current state. The action-value function is used to estimate the expected return of the agent in the current state-action pair. The magnitude of the expected return depends on the action chosen by the agent, so the value functions are related to the particular ways of acting, called policies (\(\pi\)). A policy is a mapping from the states to the probabilities of choosing each possible action in that state. We denote the value function of the state \(s\) under the policy \(\pi\) as \(V_{\pi}(s)\) (state-value function):  
\[V_{\pi}(s)\triangleq E_{\pi}[G_{t}|S_{t}=s]=E_{\pi}\left[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1}|S_{t}=s\right]. \tag{4}\]  
We denote the value of taking action a in state s under policy \(\pi\) as Q\({}_{\pi}\)(s,a) (action-value function). The mathematical expression is shown below:\[Q_{\pi}(s,\ a)\triangleq E_{\pi}[G_{t}|S_{t}=s,\ A_{t}=a\ ]=E_{\pi}\left[\sum_{k=0}^{ \infty}\gamma^{k}R_{t+k+1}|S_{t}=s,\ A_{t}=a\right]. \tag{5}\]  
Finite MDP problems can be well solved by methods based on value functions such as dynamic programming (DP), Monte Carlo methods (MC), and temporal-difference learning (TD). However, most of the real-life RL problems are continuous with infinite states, so the methods based on value functions cannot evaluate the value function well. To solve this problem, researchers propose to use function approximation for prediction, using parameterization to approximate the value function. Although this is an improvement over traditional RL, experimental results show that the performance is still poor for complex problems.