## 3 Quadcopter Control
### Deep Reinforcement Learning for Online Parameter Estimation and Tuning
2: Set target policy network \(u_{target}\) and target critic network \(Q_{target}\) with weights \(\theta^{H_{target}}\)and \(\theta^{Q_{target}}\)
3: Set target parameters weights equal to main parameters weights:

\(\theta^{H_{target}}\ \leftarrow\ \theta^{H}\,\ \theta^{Q_{target}}\ \leftarrow\ \theta^{Q}\)
4:for episode = 1, M do
5: Initialize a random process noise \(\mathcal{N}\) for action
6: exploration. 7: Receive initial observation state \(s_{1}\). 8:for t = 1, T do
9: Select actions \(a=\mu_{a_{1}}+\xi_{noise}\) where \(\xi_{noise}\ \sim\mathcal{N}\)
10: Observe a vector of states \(s\)
11: Apply actions (\(a_{2}\) to \(a_{7}\)) to outer loop PID controllers as follow:
12:\(u1_{x}=(a_{2}\cdot e_{x})+(a_{3}\cdot\hat{e}_{x})+(a_{4}\ \int e_{x}dt)\)
13:\(u1_{y}=(a_{5}\cdot e_{y})+(a_{6}\cdot e_{y})+(a_{7}\ \int e_{y}dt)\)
14: Use the updated \(u1_{x}\) and \(u1_{y}\) to generate \(\theta_{des}\), \(\vartheta_{des}\)
15: Use \(\theta_{des}\), \(\varphi_{des}\) as setpoints for inner loop PIDs and generate \(u_{2}\),\(u_{3}\) and \(u_{4}\). Compute \(u_{MPC}\) by minimizing the MPC cost function. 16:\(\min\ \{J(d_{k})=\sum\limits_{i=0}^{P-1}e_{RT}^{T}(k+i)Q_{MPC}e_{RT}(k+i)+e_{u}^{T} (k+i)Re_{u}(k+i)+\)
17:\(\Delta u^{T}(k+i)R_{\Delta u}(k+i)\}\)
18: Compute \(u_{1}\) using the generated manipulated variable from MPC and the
19: scalling factor of the compensator \(u_{1}=u_{MPC}(k+i|k)-g(nonimal\ mass\ of\ the\ robot+a_{1})\)
20: Apply control inputs \(u_{1}\), \(u_{2}\), \(u_{3}\) and \(u_{4}\) to the drone dynamics model. 21: Observe the next vector of states \(s_{t+1}\), and the next reward \(r\)
22: Store \((s,a,r,s_{t+1})\) in reply buffer \(D\). 23: Randomly sample a minibatch of \(N\) transitions \((s_{i},a_{i},r,s_{i+1})\) from \(D\). 24: Compute targets: \(y_{RLi}=r_{i}+\gamma Q_{target}(s_{i+1},u_{target}(s_{i+1}\Big{|}\theta^{H_{ target}})\big{|}\theta^{Q_{target}})\)
25: Update critic by minimizing the loss: \(L=\frac{1}{N}\sum\limits_{i}\left(y_{RLi}-Q\big{(}s_{i},a_{i}\big{|}\theta^{Q} \big{)}\right)^{2}\)
26: Update the actor policy using a sampled policy gradient:
27:\(\
abla_{\theta^{H}}\!\approx\frac{1}{N}\sum\limits_{i}\
abla_{a}Q\big{(}s,a \big{|}\theta^{Q}\big{)}\Big{|}_{s=s_{i}\,a=\mu(s_{i})}\
abla_{\theta^{H}}\mu(s|\theta^{H})\Big{|}_{s_{i}}\)
28: Update the target networks:
29:\(\theta^{Q_{target}}\leftarrow\tau\theta^{Q}+(1-\tau)\theta^{Q_{target}}\)
30:\(\theta^{H_{target}}\leftarrow\tau\theta^{H}+(1-\tau)\theta^{H_{target}}\)
31:endfor
32:endfor
```  
**Algorithm 1**The proposed algorithm. The actor outputs are actions (\(a=\mu_{\theta}(s)+\xi_{noise}\)) selected from a continuous action space by considering the current state of the environment. In this study, the action is comprised of a vector of actions \(\mathbf{a}=[a_{1}\ a_{2}\ a_{3}\ a_{4}\ a_{5}\ a_{6}\ a_{7}]\). Here, \(a_{2}\) to \(a_{7}\) are used to actively estimates and tune the gain coefficients of the outer loop PID controllers for x and y positions, which are two controllers located in outer loop PID control block. The action (\(a_{1}\)) generates an updated estimation of the proper scaling factor for updating the compensator. The defined states were errors \(\left(e_{x},e_{y},e_{z}\right)\) and \(\left(\int e_{x},\ \int e_{y},\ \int e_{z}\right)\). The Q function is computed using the rewards from the environments. The notion is that the best policy can be found by maximizing the true Q-value of all actions. The architecture of the proposed adaptive RL-based controller is shown in Figure 5.  
To find the optimal architecture for the actor-critic network, many simulations were run with different numbers of layers and neuron counts in neural networks. According to the simulations findings, it was observed that increasing the size of hidden layers encourages polynomial exploration, whereas increasing the number of layers promotes exponential exploration. The number of layers in the actor network varied from 1 to 5 for the purpose of the experiment.