# Single-Parameter-Tuned Attitude Control for Quadrotor with Unknown Disturbance
## 3 Quadrotor Attitude Control System Design
### Estimation of the CGPs
Moreover, the zero state-action point \(E_{o}=\left(s_{o},a_{o},s_{o}^{\prime}\right)=\left[0,0,b_{0}\right],\left[0 \right],\left[0,0,b_{0}\right]\) is an obvious result, but may never be explored in the training process, which is an important reason for the steady-state error problem. These samples should be considered to accelerate the training process and eliminate steady-state errors. #### 3.3.3 Training Algorithm

The training algorithm in this paper is the DPG algorithm [26], and we add some Gaussian noise to the action as the exploration strategy. Algorithm 3 lists the whole algorithm. Figure 3: Network structures. ```
\(e_{max}\): max number of episodes  
\(s_{max}\): max number of steps in an episode  
\(n_{s}\): max number of samples  
\(R\left(\mathbf{s}^{\prime}\right)\): reward function in Equation (13)  
\(\tau\): soft update rate
```
1: initialize weights of online critic network \(\mathbf{\theta}_{v}\) and online actor network \(\mathbf{\theta}_{\pi}\) randomly
2: initialize weights of target critic network \(\mathbf{\theta}^{\prime}_{v}\) and target actor network \(\mathbf{\theta}^{\prime}_{\pi}:\mathbf{\theta}^{\prime}_{v}\leftarrow\mathbf{\theta}_{v}, \mathbf{\theta}^{\prime}_{\pi}\leftarrow\mathbf{\theta}_{\pi}\)
3: initialize memory box \(B\)
4:for\(epi=1,2...e_{max}\)do
5: initialize the agent with random state
6: put the zero state-action point \(E_{o}\) into the memory box \(B\gets B\bigcup E_{o}\)
7:for\(s=1,2...s_{max}\)do
8: take action \(\mathbf{a}\) according to the exploration strategy and current state \(\mathbf{s}\)
9: receive the next state \(\mathbf{s}^{\prime}\) from the environment
10: put \(E\) and the mirror sample \(E_{m}\) into the memory box \(B\gets B\bigcup E\bigcup E_{m}\)
11:\(T=T\cup T_{i}\);
12:for\(i=1,2...n_{s}\)do
13: sample experience \(\mathbf{e}_{i}=\left(\mathbf{s}_{i},\mathbf{a}_{i},\mathbf{s}^{\prime}_{i}\right)\) from \(B\)
14: compute reward of experience \(\mathbf{e}_{i}:\tau_{i}\gets R\left(\mathbf{s}^{\prime}_{i}\right)\)
15:\(y_{i}\gets r_{i}+\lambda V\left(\left(\mathbf{s}^{\prime}_{i},\tau\left(\mathbf{s} ^{\prime}_{i}|\mathbf{\theta}_{\pi}\right)\right)|\mathbf{\theta}_{\tau}\right)\)
16: update \(\mathbf{\theta}_{v}\) by minimizing the loss: \(L=\frac{1}{N}\sum_{i}\left(y_{i}-V\left(\mathbf{s}_{i},\mathbf{a}_{i}|\mathbf{\theta}_{v} \right)\right)^{2}\)
17: update \(\mathbf{\theta}_{\pi}\) using the sampled policy gradients: \(\bigtriangledown_{\mathbf{\theta}_{\pi}}\approx\frac{1}{N}\sum_{i}\bigtriangledown_ {\mathbf{a}}V\left(\left(\mathbf{s},\mathbf{a}|\mathbf{\theta}_{\pi}\right)|_{s=s_{i}} \bigtriangledown_{\mathbf{\theta}_{\pi}}\pi\left(\mathbf{s}|\mathbf{\theta}_{\pi}\right) \right)|_{s=s_{i}}\)
18: soft update \(\theta^{\prime}_{v}\) with \(\mathbf{\theta}_{v}:\mathbf{\theta}^{\prime}_{\pi}\leftarrow\tau\mathbf{\theta}_{v}+(1- \tau)\,\mathbf{\theta}^{\prime}_{v}\)
19: soft update \(\theta^{\prime}_{\pi}\) with \(\mathbf{\theta}_{\pi}:\mathbf{\theta}^{\prime}_{\pi}\leftarrow\tau\mathbf{\theta}_{\pi}+( 1-\tau)\,\mathbf{\theta}^{\prime}_{\pi}\)
20:endfor
21:endfor
```  
**Algorithm 3** Training algorithm with mirror sampling.