## 3 Methodological Approach
### Algorithms
#### 3.1.2 DDPG  
DDPG [33] combines elements of value function based and policy gradient based algorithms, following the actor-critic architecture. This algorithm aims to learn a deterministic policy \(\pi_{\theta}(s)=a\) and it is derived from the deterministic policy gradient theorem [34].  
Following the actor-critic architecture, DDPG uses an action-value function \(Q_{w}(s,a)\) as critic to guide the learning process of the policy and it is based on the deep Q-network (DQN) [35]. Prior to DQN, it was believed that learning value functions with large and nonlinear function approximators was difficult. DQN is able to learn robust value functions due to two innovations: First, the network is trained off-policy getting experience samples from a replay buffer to eliminate temporal correlations. In addition, target networks are used, which are updated more slowly, and this gives consistent targets during temporal difference learning.  
The critic is updated according to the gradient of the objective defined in Equation (7).  
\[L(w)=\mathbb{E}[(Q_{w}(s_{t},a_{t})-y_{t})^{2}] \tag{7}\]  
where  
\[y_{t}=r(s_{t},a_{t})+\gamma Q_{w}(s_{t+1},a_{t+1})|_{a_{t+1}=\pi_{\theta}(s_{t +1})} \tag{8}\]The actor is updated following the deterministic policy gradient theorem, defined in Equation (9). The intuition is to update the policy in the direction that improves \(Q_{w}(s,a)\) most.  
\[\
abla J(\theta)=\mathbb{E}[\
abla_{\theta}\pi_{\theta}(s_{t})\
abla_{a}Q_{w}(s_ {t},a_{t})|_{a=\pi_{\theta}(s_{t})}] \tag{9}\]  
As mentioned before, the target value defined in Equation (8) is calculated using the target networks \(\pi^{\prime}_{\theta^{\prime}}\) and \(Q^{\prime}_{w^{\prime}}\), which are updated more slowly and this gives more consistent targets when learning the action-value function.  
As DDPG is an off-policy algorithm, the policy used to get experience is different from the behaviour policy. Despite the behaviour policy being deterministic, typically, a stochastic policy is used to get experience, being able to better explore the environment. This exploratory policy is usually achieved adding noise to the behaviour policy. Although there are common noises such as normal noise or Ornstein-Uhlenbeck noise [36], which are added directly to the action generated by the policy, in [37], [PERSON] et al. proposed adding noise to the neural network's parameter space to improve the exploration and to reduce the training time. The DDPG algorithm is described in Algorithm 2.  
```
1:Initialise the actor \(\pi_{\theta}(s)\) and the critic \(Q_{w}(s,a)\) networks.
2:Initialise the target networks \(Q^{\prime}\) y \(\pi^{\prime}\) with the weights \(\theta^{\prime}\leftarrow\theta\), \(w^{\prime}\gets w\)
3:Initialise the replay buffer
4:for\(\varepsilon\in\textit{episodes}\)do
5: Initialise noise generation process
6: Get the first observation
7:for\(t\in\textit{steps}\)do
8: Select the action \(a_{t}=\pi_{\theta}(s_{t})+N\)
9: Execute the action \(a_{t}\), get the reward \(r_{t}\) and the observation \(s_{t+1}\)
10: Store the transition \(<s_{t},a_{t},r_{t},s_{t+1}>\) in replay buffer
11: Get M experience samples \(<s_{i},a_{i},r_{i},s_{i+1}>\) from the replay buffer
12:\(y_{i}=r_{i}+\gamma Q^{\prime}_{w^{\prime}}(s_{i+1},\pi^{\prime}_{\theta^{ \prime}}(s_{i+1}))\)
13: Update the critic minimising the loss : \[L=\frac{1}{M}\sum_{i}(y_{i}-Q_{w}(s_{i},a_{i}))^{2}\]
14: Update the policy of the actor: \[\
abla_{\theta}J\approx\frac{1}{M}\sum_{i}\
abla_{a}Q_{w}(s,a)|_{s=s_{i},a=\pi _{\theta}(s_{i})}\
abla_{\theta}\pi_{\theta}(s)|_{s=s_{i}}\]
15: Update target networks: \[\theta^{\prime} \leftarrow\tau\theta+(1-\tau)\theta^{\prime}\] \[w^{\prime} \leftarrow\tau w+(1-\tau)w^{\prime}\]
16:endfor
17:endfor
```  
**Algorithm 2** Deep Deterministic Policy Gradient (DDPG).