# Empirical Analysis of Automated Stock Trading Using Deep Reinforcement Learning
## 3 Background  
### Reinforcement Learning Overview  
Reinforcement learning is a field of machine learning that seeks to select actions in systems in chronological order to achieve desired goals. In RL, the time variable is typically discrete-time and forms a sequential decision-making problem because actions are applied at every time step. In this case, the decision maker is called an agent, and the system is called an environment. The agent observes the state representing the change in the environment and selects an action with discontinuous or continuous values under a certain policy, which updates the environment by applying being applied to the environment. This process is briefly described in Figure 1. Deep Reinforcement learning is a particular type of RL with deep neural networks for state representation or function approximation for value functions, policy, transition models, or reward functions [18], and now, RL usually refers to DRL. The fundamentals of RL are as follows, and Equations (1)-(5) are described in detail in the equation Appendix A. * Agent: a software program that learns to make intelligent decisions. It is also called a decision-maker. * Environment: the world the agent stays within. * State \(s\): a snapshot of a moment in the environment, including the agent. * Action \(a\): a move performed by the agent. * State set denoted as \(S\): a set of possible \(s\). * Action set denoted as \(A\): a set of possible \(a\). * Transition probability, denoted as \(P(s^{\prime}|s,a)\): probability of moving to state \(s^{\prime}\) when the agent takes action \(a\) in state \(s\). * Policy \(\pi\): defines the agent's behavior in an environment. * Reward \(r\): feedback is given for the action. * Trajectory \(\tau\): an interaction between the agent and the environment by performing some actions, starting from the initial state until it reaches the terminal state. It is often called an episode. \[\tau=(s_{0},a_{0},r_{0},s_{1},a_{1},r_{1},\ldots,s_{T},a_{T})\] (1)
* Discount factor \(\gamma\): a value between 0 and 1 multiplied by the future reward to convert future rewards into present rewards. * Return \(R\): the sum of rewards obtained in an episode. RL aims to maximize the expected \(R\) of \(\tau\). \[R(\tau)=\gamma^{0}r_{0}+\gamma^{1}r_{1}+\gamma^{2}r_{2}+\ldots+\gamma^{T}r_{T} =\sum_{t=0}^{T}\gamma^{t}r_{t}\] (2)
* \(Q\)-function denoted as \(Q(s,a)\): Q-function defines the value of a state-action pair, which is the return the agent would obtain starting from state \(s\) and performing the action following policy \(\pi\). \[Q^{\pi}(s,a)=[R(\tau)|s_{0}=s,a_{0}=a]\] (3)  
RL can be classified into value-based and policy-based methods, among which policy-based methods do not compute the Q function. Instead, the optimal policy is calculated directly by parameterizing the policy using some parameter \(\theta\). Some function maps the state to the probability of actions in that state. As for the function, a neural network that can be used as a function approximator is called a policy network. The parameters of the policy network are \(\theta\), which are trained using the gradient descent algorithm, and this method is called the policy gradient method. In the policy gradient method, the state is fed as an input to the network and returns the probability distribution over an action space. The policy gradient method aims to find the optimal parameter \(\theta\) of the neural network so that the network returns the correct probability distribution over the action space. The objective of the network is to assign high probabilities to actions that maximize the expected return of \(\tau\). The objective function can be described as:  
\[J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}(\tau)}[R(\tau)] \tag{4}\]  
where \(\tau\sim\pi_{\theta}(\tau)\) is the sampling of \(\tau\) based on policy \(\pi\) given by the network parameterized by \(\theta\). Figure 1: Agent–environment interface. To maximize the objective function, we use the gradient of the objective function.