## 2 Materials and Methods
### Proposed Solution: Deep Reinforcement Learning
#### 2.2.3 DRL Algorithm  
Any RL agent receives a reward \(R_{\tau}\) for each action taken, \(a_{\tau}\). The function that RL algorithms seek to maximize is called the _discounted future reward_ and is defined as:  
\[G_{\tau}=R_{\tau+1}+\gamma R_{\tau+2}+...=\sum_{k=0}^{\infty}\gamma^{k}R_{\tau +k+1} \tag{20}\]  
where \(\gamma\) is a fixed parameter known as the _discount factor_. The objective is then to learn an action _policy_ that maximizes such the discounted future reward. Given a specific policy \(\pi\), the _action value function_, also called _Q-value function_ indicates how much valuable it is to take a specific action \(a_{\tau}\) being at state \(s_{\tau}\):  
\[Q_{\pi}(s,a)=E_{\pi}[G_{\tau}\|s_{\tau}=s,a_{\tau}=a] \tag{21}\]  
from (21) we can derive the _recursive Bellman equation_:  
\[Q(s_{\tau},a_{\tau})=R_{\tau+1}+\gamma Q(s_{\tau+1},a_{\tau+1}) \tag{22}\]  
Notice that if we denote the final state with \(s_{final}\), then \(Q(s_{final},a)=R_{a}\). The Temporal Difference learning mechanism uses (22) to approximate the Q-values for state-action pairsin the traditional _Q-learning algorithm_. However, in large state or action spaces, it is not always feasible to use tabular methods to approximate the Q-values. To overcome the traditional Q-learning limitations, [PERSON] et al. [53] proposed the usage of a Deep Artificial Neural Network (ANN) approximator of the Q-value function. To evict convergence to local-optima, they proposed to use an \(\epsilon\)-greedy policy where actions are sampled from the ANN with probability \(1-\epsilon\) and from a random distribution with probability \(\epsilon\), where \(\epsilon\) decays slowly at each MDP transition during training. They also used the _Experience Replay_ (ER) mechanism: a data structure \(\mathcal{D}\) keeps \((s_{\tau},a_{\tau},r_{\tau},s_{\tau+1})\) transitions for sampling uncorrelated training data and improve learning stability. ER mitigates the high correlation presented in sequences of observations during online learning. Moreover, authors in [54] implemented two neural network approximators for (21), the Q-network and the Target Q-network, indicated by \(Q(s,a,\theta)\) and \(Q(s,a,\theta^{-})\), respectively. In [54], the target network is updated only periodically to reduce the variance of the target values and further stabilize learning with respect to [53]. Authors in [54] use stochastic gradient descent to minimize the following loss function:  
\[\mathcal{L}(\theta)=E_{(s_{\tau},a_{\tau},r_{\tau},s_{\tau+1})\sim U(\mathcal{ D})}\{[r+\gamma\max_{a}Q(s_{\tau},a,\theta^{-})-Q(s_{\tau},a_{\tau}; \theta)]^{2}\} \tag{23}\]  
where minimization of (23) is done with respect to the parameters of \(Q(s,a,\theta)\). [PERSON] et al. [55] applied the concepts of Double Q-Learning [56] on large-scale function approximators. They replaced the target value in (23) with a more sophisticated target value:  
\[\mathcal{L}(\theta)=E_{(s_{\tau},a_{\tau},r_{\tau},s_{\tau+1})\sim U(\mathcal{ D})}\{[r_{\tau}+\gamma Q(s_{\tau+1},\underset{a}{\text{argmax}}Q(s_{\tau+1},a; \theta),\theta^{-})-Q(s_{\tau},a_{\tau};\theta)]^{2}\} \tag{24}\]  
Doing such a replacement, authors in [55] avoided over-estimations of the Q-values which characterized (23). This technique is called Double Deep Q-Learning (DDQN), and it also helps to decorrelate the noise introduced by \(\theta\), from the noise of \(\theta^{-}\). Notice that \(\theta\) are the parameters that approximate the function used to choose the best actions, while \(\theta^{-}\) are the parameters of the approximator used to evaluate the choices. Such a differentiation in the _learning_ and _acting_ policies is also called _off-policy_ learning. Instead, [PERSON] et al.