# Error Dynamics Based Dual Heuristic Dynamic Programming for Self-Learning Flight Control
## 3 Online Tracking Controller Based on ED-DHP
### ED-DHP Framework
The intermediate variables for the \(j\)th hidden layer node and the outputs of the neural networks can be written as:  
\[p_{c,j}=\sum_{i=1}^{n}e_{i}(t)w_{\text{critical},ij}(t)+\sum_{i=n+1}^{2n}x_{ref,i-n}(t)w_{\text{critical},ij}(t),\ \ j=1,\ldots,10 \tag{23}\]  
\[q_{c,j}=\tanh(p_{c,j})=\frac{e^{p_{c,j}}-e^{-p_{c,j}}}{e^{p_{c,j}}+e^{-p_{c,j} }},\ \ j=1,\ldots,10 \tag{24}\]  
\[\hat{\lambda}_{k}=\sum_{j=1}^{10}q_{c,j}w_{\text{critic}2,jk}(t),\ \ k=1,\ldots,2n \tag{25}\]  
\[p_{a,j}=\sum_{i=1}^{n}e_{i}(t)w_{\text{actor1},ij}(t)+\sum_{i=n+1}^{2n}x_{ref,i-n}(t)w_{\text{actor1},ij}(t),\ \ j=1,\ldots,10 \tag{26}\]  
\[q_{a,j}=\tanh(p_{a,j})=\frac{e^{p_{a,j}}-e^{-p_{a,j}}}{e^{p_{a,j}}+e^{-p_{a,j} }},\ \ j=1,\ldots,10 \tag{27}\]  
Figure 2: Schematic of ED-DHP. It is desired to update the critic NN by minimizing the square Bellman error:  
\[E_{\text{C}}(t)=\frac{1}{2}e_{\text{C}}^{\text{T}}(t)e_{\text{C}}(t) \tag{29}\]  
where  
\[ \begin{split}& e_{\text{C}}(t)=\frac{\partial[V_{\text{X}}(t-1)-J_{ \text{X}}(X(t-1),u_{a}(t-1))-\gamma V_{\text{X}}(t)]}{\partial X(t-1)}=\lambda( X(t-1))-\frac{\partial J_{\text{X}}(X(t-1),u_{a}(t-1))}{\partial X(t-1)}-\gamma \frac{\partial V_{\text{X}}(t)}{\partial X(t-1)}\\ &=\lambda(X(t-1))-\Big{(}X(t-1)^{\text{T}}Q_{\text{X}}+u_{a}(t-1) ^{\text{T}}R\frac{\partial u_{a}(t-1)}{\partial X(t-1)}\Big{)}-\gamma\lambda(X (t))\Big{(}\frac{\partial X(t)}{\partial X(t-1)}+\frac{\partial X(t)}{ \partial u_{a}(t-1)}\frac{\partial u_{a}(t-1)}{\partial X(t-1)}\Big{)}\end{split}  \tag{30}\]  
In Equation (30), \(\lambda(X(t-1))\) and \(\lambda(X(t))\) can be approximated by critic NN. \(\partial u_{a}(t-1)/\partial X(t-1)\) is calculated by actor NN. The transition matrices \(\partial X(t)/\partial X(t-1)\) and \(\partial X(t)/\partial u_{a}(t-1)\) can be obtained from Equation (9). Because the partial derivative \(f_{1}\) with respect to the angle rate of other channels is close to 0, we can obtain the transition matrices as follows: (The input dynamics are identified by RLS.) \[\frac{\partial X(t)}{\partial X(t-1)}\approx \begin{bmatrix}1&0\\ 0&1\end{bmatrix} _{2n\times 2n} \tag{31}\]  
\[\frac{\partial X(t)}{\partial u_{a}(t-1)}\approx \begin{bmatrix}\Delta tg(t-1) \\ 0\end{bmatrix} _{2n\times m}\approx \begin{bmatrix}\hat{5}_{1}(t-1)\\ 0\end{bmatrix} _{2n\times m}\] (3  
\[ \begin{split} w_{\text{actor2}}(t+1)&=w_{\text{actor2}}(t)- \eta_{A}\frac{\partial V_{X}(t-1)}{\partial u_{x}(t-1)}\frac{\partial u_{x}(t-1 )}{\partial w_{\text{actor2}}(t)}\approx w_{\text{actor2}}(t)-\eta_{\text{A}} \Big{[}\frac{\partial J(t-1)}{\partial u_{x}(t-1)}+\gamma\hat{\lambda}(X(t)) \frac{\partial X(t)}{\partial u_{x}(t-1)}\Big{]}\frac{\partial u_{x}(t-1)}{ \partial w_{\text{actor2}}(t)}\\ &\approx w_{\text{actor2}}(t)-\eta_{A}\bigg{[}u_{a}(t-1)^{T}R+ \gamma\hat{\lambda}(X(t))\bigg{[}\begin{array}{c}\xi_{1}(t-1)\\ 0\end{array}\bigg{]}_{2n\times m}\bigg{]}\frac{\partial u_{a}(t-1)}{\partial w _{\text{actor2}}(t)}\end{split}  \tag{35}\]  
\[ \begin{split} w_{\text{actor1}}(t+1)&=w_{\text{actor1}}(t )-\eta_{A}\frac{\partial V_{X}(t-1)}{\partial u_{x}(t-1)}\frac{\partial u_{x}( t-1)}{\partial w_{\text{actor1}}(t)}\approx w_{\text{actor1}}(t)-\eta_{\text{A}} \Big{[}\frac{\partial J(t-1)}{\partial u_{x}(t-1)}+\gamma\hat{\lambda}(X(t)) \frac{\partial X(t)}{\partial u_{x}(t-1)}\Big{]}\frac{\partial u_{x}(t-1)}{ \partial w_{\text{actor1}}(t)}\\ &\approx w_{\text{actor1}}(t)-\eta_{\text{A}}\bigg{[}u_{a}(t-1)^{T}R+ \gamma\hat{\lambda}(X(t))\bigg{[}\begin{array}{c}\xi_{1}(t-1)\\ 0\end{array}\bigg{]}_{2n\times m}\bigg{]}\frac{\partial u_{a}(t-1)}{\partial u _{a}}\frac{\partial u_{x}}{\partial p_{x}}\frac{\partial p_{x}}{\partial w_{ \text{actor1}}(t)}\end{split}  \tag{36}\]  
```
0: Simulation parameters: sampling time \(\Delta t\) and the total simulation time \(T\); The controller parameters for the slow loop subsystem; The forgetting factor of RLS; The positive definite matrices; The discount factor and the learning rates of two neural networks.