## 2 Methodology
### Formal Definition  
The Bellman equations [19] can be extended to calculate the optimal Q-value for the given state-action pair as follows:  
\[Q^{*}(s,a)=\mathbb{E}[r+\gamma\max_{a^{\prime}}Q^{*}(s^{\prime},a^{\prime})] \tag{2}\]  
\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Algorithm** & **Description** & **Policy** & **Action Space** & **State Space** \\ \hline DQN & Deep Q Network & Off-policy & Discrete & Continuous \\ DDPG & Deep Deterministic Policy Gradient & Off-policy & Continuous & Continuous \\ A3C & Asynchronous Advantage Actor-Critic Algorithm & On-policy & Continuous & Continuous \\ TRPO & Trust Region Policy Optimization & On-policy & Continuous & Continuous \\ PPO & Proximal Policy Optimization & On-policy & Continuous & Continuous \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparision of deep reinforcement algorithms.  
Here \(s,a\) are the current state and action, whereas \(s^{\prime},a^{\prime}\) are the state-action in the next timestep and \(r,\gamma\) are reward and discount factor, respectively. The intution behind the Equation (2) is that if the optimal value of \(Q^{*}(s^{\prime},a^{\prime})\) is known for all possible actions \(a^{\prime}\), then the optimal strategy is to select the action that maximizes the expected value of \(r+\gamma Q^{*}(s^{\prime},a^{\prime})\).  
The basic idea here is to update the Q-values iteratively  
\[Q_{i+1}(s,a)=\mathbb{E}[r+\gamma\max_{a^{\prime}}Q_{i}(s^{\prime},a^{\prime})] \tag{3}\]  
such equations converge when \(Q_{i}\to Q^{*}\) as \(i\to\infty\). In reality, this approach becomes impractical as the action-value function is estimated seperately for each sequence with out any generalization. Therefore a function approximator is used estimate the action-value function. This can be represented as  
\[Q(s,a;\theta)\approx Q^{*}(s,a) \tag{4}\]  
In deep Q learning we use the neural network as the function approximator and \(\theta\) represents the weights of the Q-network. This network can be trained by minimizing a sequence of loss functions \(L_{i}(\theta_{i})\) that are obtained at each iteration \(i\),  
\[L_{i}(\theta_{i})=\mathbb{E}[(y_{i}-Q(s,a;\theta_{i}))^{2}] \tag{5}\]  
where \(y_{i}=\mathbb{E}[r+\gamma\max_{a^{\prime}}Q_{i}(s^{\prime},a^{\prime};\theta_ {i})]\) is the target for iteration \(i\). The parameters of the previous iteration \(\theta_{i-1}\) are held fixed when the loss function is being optimized, but the parameters of target \(\theta_{i}\) change as they depend on the network weights. By differentiating the loss function with respective weights \(\theta_{i}\) we arrive at the gradient of the loss function in equation  
\[\
abla_{\theta_{i}}L_{i}(\theta_{i})=\mathbb{E}[(r+\gamma\max_{a^{\prime}}Q(s ^{\prime},a^{\prime};\theta_{i-1})-Q(s,a;\theta_{i}))\
abla_{\theta_{i}}Q(s,a ;\theta_{i})]. \tag{6}\]  
It is computationally expensive to calculate the full expectations for the above gradient, and therefore this loss function is optimized by stochastic gradient descent. Based on the frequency of the weights being updated the expectations are replaced by the samples from the environment and its action Selection strategies.  
The DQN Agent observes the environment then takes an action in it to achieve a reward. The reward achieved depends on the state it reached by taking a particular action in the environment. Here the environment includes both the observations and states, though the DQN agent can only see observations before taking the actions. The states here are merely to calculate the reward for the agent. The following sections explains them in detail.