# Error Dynamics Based Dual Heuristic Dynamic Programming for Self-Learning Flight Control
## 3 Online Tracking Controller Based on ED-DHP
### ED-DHP Framework  
For the system dynamics (9), the perfect control input contains the feedforward input \(u_{d}^{*}\) and the feedback input \(u_{e}^{*}\) if the system dynamics is known [27]. \[\left\{ \begin{array}{c}u_{d}^{*}=(\Delta tg)^{-1}(\phi(x_{ref}(t))-\Delta tf _{1}(x_{ref})-\Delta t\Delta f)\\ u_{e}^{*}=-\frac{\gamma}{2}R^{-1}(\Delta tg(t))^{\mathrm{T}}\frac{\partial V( e(t+1))}{\partial t(t+1)}\\ u_{a}^{*}=u_{d}^{*}+u_{e}^{*}\end{array} \right. \tag{14}\]  
The symbols in Equation (14) are given as follows. Function \(J\) is known as the utility and is selected based on the quadratic energy function in this paper. \(V\) is defined as a performance measure function or value function in RL. It represents the sum of expected future reward from the current state. Matrices \(Q\in\mathbb{R}^{n\times n}\) and \(R\in\mathbb{R}^{m\times m}\) are positive definite. \(\gamma\) is a discount factor. \[J(e(t),u_{e}(t))=\frac{1}{2}\Big{[}e(t)^{\mathrm{T}}Qe(t)+u_{e}(t)^{\mathrm{T }}Ru_{e}(t)\Big{]} \tag{15}\]  
\[V(t)=\sum_{i=t}^{+\infty}\gamma^{(i-t)}J(e(i),u_{e}(i)) \tag{16}\]  
For learning the control strategy of the attitude system without some prior knowledge, the augmented system is constructed in terms of \(e(t)\) and \(x_{ref}(t)\)[28, 29] as:  
\[X(t)= \begin{bmatrix}e(t)\\ x_{ref}(t)\end{bmatrix} \in\mathbb{R}^{2n} \tag{17}\]  
\[ \begin{bmatrix}e(t+1)\\ x_{ref}(t+1)\end{bmatrix} = \begin{bmatrix}e(t)-Z(t)+\Delta tf_{1}(x(t))\\ \phi(x_{ref}(t))\end{bmatrix} + \begin{bmatrix}\Delta tg(t)\\ 0\end{bmatrix} u_{a}(t)\equiv F(X(t))+G(t)u_{a}(t) \tag{18}\]  
So we could have the following controller for the fast loop subsystem as follows: (Here, the second part \(u_{a}\) is the output vector of actor NN, and the third \(u_{b}=-\xi_{1}{}^{-1}\xi_{2}\) is designed to compensate for the total uncertainty.) \[ \begin{array}{c}u^{*}=u_{p}+u_{d}^{*}+u_{e}^{*}-(\Delta tg)^{-1}(\Delta t \Delta f)\\ \approx u_{p}+\underbrace{u_{d}+u_{e}}_{u_{e}\text{-ED-DHP}}+\underbrace{-\xi_ {1}{}^{-1}\xi_{2}}_{u_{p}\text{-RLS}}\\ =u_{p}+u_{a}+u_{b}\end{array}  \tag{19}\]  
For the augmented system (18), the cost function \(J\) and value function \(V\) can be rewritten as:  
\[J_{X}(X(t),u_{a}(t))=\frac{1}{2}\Big{[}X(t)^{T}Q_{X}X(t)+u_{a}(t)^{T}Ru_{a}(t) \Big{]} \tag{20}\]\[V_{X}(t)=\sum_{i=1}^{+\infty}\gamma^{(i-t)}J_{X}(X(t),u_{a}(t))\] \[=J_{X}(X(t),u_{a}(t))+\gamma\sum_{i=t+1}^{+\infty}\gamma^{(i-t-1)}J_{X}(X(i ),u_{a}(i)) \tag{21}\]  
\[=J_{X}(X(t),u_{a}(t))+\gamma V_{X}(t+1)\]  
where  
\[Q_{X}= \begin{bmatrix}Q&0\\ 0&0\end{bmatrix} \in\mathbb{R}^{2n\times 2n} \tag{22}\]  
Figure 2 shows the schematic of ED-DHP. The critic NN approximates the derivatives of the value function \(V(X(t))\) concerning the augmented state \(X(t)\), defined as \(\lambda(X(t))\). As can be observed from this figure, the previous derivatives of the value function \(\hat{\lambda}(X(t-1))\) and the augmented state \(X(t-1)\) are required to be stored and used in updating the NNs, without predicted ones. The control policy estimated by the actor NN is updated by minimizing the value function. A two-layer neural network with one hidden layer containing 10 neurons is used as the critic NN and actor NN in this article, shown in Figure 3. The hidden-layer activation function is tanh. For the critic NN, \(w_{\text{critical},ij}\), and \(w_{\text{critic},2,k}\) represent the weights from the \(i\)th input node to the \(j\)th hidden layer node and the \(j\)th hidden node to the \(k\)th output node, respectively. For the actor NN, \(w_{\text{actor1},ij}\), and \(w_{\text{actor2},k}\) represent the weights from the \(i\)th input node to the \(j\)th hidden layer node and the \(j\)th hidden node to the \(k\)th output node.