# Automatic Control Optimization for Large-Load Plant-Protection Quadrotor
## 2 Materials and Methods
### Attitude Controller Design for Large-Load Plant-Protection Quadrotors
#### 2.2.2 Network Structure and Training Algorithm
To improve the the robustness for sensor noise of the quadrotor, sensor noise are added to the agents in the training process. In our algorithm, we run several agents in parallel and immediately employ their experience at each time step, which has some advantages:  
1. Different agents have different physical parameters with different initial states and take different actions, thus avoiding the experience correlation problem. 2. Compared to traditional RL methods such as deep Q learning (DQN) [25], parallel training does not need experience replay skills and thus requires much less memory. 3. Several agents can provide more experience at the same time, greatly accelerating the training process. The whole algorithm is listed as Algorithm 1. \(n_{agent}\): max number of agents  
\(e_{max}\): max number of episodes  
\(s_{max}\): max number of steps during an episode  
\(R(\mathbf{s}^{\prime})\): The reward function  
\(\tau\): soft update rate  
1: create \(n_{agent}\) agents with different system parameters  
2: randomly initialize the weights of the online critic network \(\mathbf{\theta}_{v}\) and the online actor network \(\mathbf{\theta}_{\pi}\)  
3: initialize the weights of the target critic network \(\mathbf{\theta}^{\prime}_{v}\) and the target actor network \(\mathbf{\theta}^{\prime}_{\pi}:\mathbf{\theta}^{\prime}_{v}\leftarrow\mathbf{\theta}_{v}, \mathbf{\theta}^{\prime}_{\pi}\leftarrow\mathbf{\theta}_{\pi}\)  
4: **for \(epi=1,2...e_{max}\) do**  
5: initialize all agents with random states  
6: **for \(step=1,2...s_{max}\) do**  
7: clear the memory box \(B\leftarrow\varnothing\)  
8: **for \(n=1,2...n_{agent}\) do**  
9: recieve the current state \(\mathbf{s}\)  
10: take action \(\mathbf{a}\) according to \(\mathbf{s}\)  
11: receive the next state \(\mathbf{s}^{\prime}\)  
12: put \(E=(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})\) into the memory box \(B\gets B\bigcup E\)  
13: **end for**  
14: **for \(i=1,2...n_{agent}\) do**  
15: obtain experience \(\mathbf{e}_{i}=\big{(}\mathbf{s}_{i},\mathbf{a}_{i},\mathbf{s}^{\prime}_{i}\big{)}\) from \(B\)  
16: compute the reward of experience \(\mathbf{e}_{i}:r_{i}\gets R\big{(}\mathbf{s}^{\prime}_{i}\big{)}\)  
17: \(y_{i}\gets r_{i}+\lambda V\big{(}(\mathbf{s}^{\prime}_{i},\pi\big{(}\mathbf{s}^{ \prime}_{i}|\mathbf{\theta}_{\pi}\big{)})|\mathbf{\theta}_{v}\big{)}\)  
18: update \(\mathbf{\theta}_{v}\) by minimizing the loss: \(L=\frac{1}{N}\sum_{i}(y_{i}-V(\mathbf{s}_{i},\mathbf{a}_{i}|\mathbf{\theta}_{v}))^{2}\)  
19: update \(\mathbf{\theta}_{\pi}\) using the sampled policy gradients:  
\(\bigtriangledown_{\mathbf{\theta}_{\pi}}\approx\frac{1}{N}\sum_{i}\bigtriangledown _{\mathbf{a}}V((\mathbf{s},\mathbf{a}|\mathbf{\theta}_{v})|_{\mathbf{s}=s_{i},\mathbf{a}=\mathbf{a}_{i}} \bigtriangledown_{\mathbf{\theta}_{\pi}}\pi(s|\mathbf{\theta}_{\pi}))|_{\mathbf{s}=\mathbf{s }_{i}}\)  
20: soft update \(\mathbf{\theta}^{\prime}_{\pi}\) with \(\mathbf{\theta}_{v}:\mathbf{\theta}^{\prime}_{\pi}\leftarrow\tau\mathbf{\theta}_{\pi}+(1- \tau)\mathbf{\theta}^{\prime}_{\pi}\)  
21: soft update \(\mathbf{\theta}_{\pi}\) with \(\mathbf{\theta}_{\pi}:\mathbf{\theta}^{\prime}_{\pi}\leftarrow\tau\mathbf{\theta}_{\pi}+(1- \tau)\mathbf{\theta}^{\prime}_{\pi}\)  
22: **end for**  
23: **end for**  
**Algorithm 1** Extended-state RL parallel training algorithm