# Error Dynamics Based Dual Heuristic Dynamic Programming for Self-Learning Flight Control
## 4 Numerical Simulation and Discussion
### Example 1  
The scheme firstly controls the angle of attack \(\alpha\), sideslip angle \(\beta\), and bank angle \(\gamma_{V}\) to track the sinusoidal reference signals under the condition of a fixed operating point. The initial conditions are \(\alpha(0)=2^{\circ}\), \(\beta(0)=\gamma_{V}(0)=0^{\circ}\), and \(\omega_{x}(0)=\omega_{y}(0)=\omega_{z}(0)=0^{\circ}\) / s. The primary control surfaces are \(\delta_{z}(0)=-11^{\circ}\) and \(\delta_{x}(0)=\delta_{y}(0)=0^{\circ}\), which are close to the sketchy trim values. The reference commands are designed as:  
\[\left\{\begin{array}{c}\alpha_{ref}=2+2\times\sin(0.4\pi t)\\ \beta_{ref}=1\times\sin(0.4\pi t)\\ \gamma_{Vref}=1\times\sin(0.4\pi t)\end{array}\right. \tag{37}\]  
The controller parameters for the slow loop subsystem are set as \(K_{\alpha}=K_{\beta}=K_{\gamma_{V}}=3.5\). The positive parameters of TD are designed as \(r=120\) and \(\tau=0.9\). The forgetting factor of RLS is \(\gamma_{\text{TRLS}}=1.0\), to ensure consistency of the identification. It was found that if \(\gamma_{\text{RLS}}\) is set to a value less than 1, the identification speed is improved with the reduction of stability of ED-DHP. So the parameters identified onlineshould have the correct sign and, to some extent, the correct relative magnitude in the self-learning process. The initial parameter matrix of RLS is set as follows, (which only conforms to the polarity of the vehicle):  
\[\varsigma(0)=\begin{bmatrix}-0.3&0.0&0.0\\ 0.0&-0.3&0.0\\ 0.0&0.0&-0.3\\ 0.05&0.05&0.05\end{bmatrix}\in\mathbb{R}^{(m+1)\times n}|_{m=n=3}\]  
The initial estimation covariance matrix is set as:  
\[Cov(0)=\begin{bmatrix}100&0&0&0\\ 0&100&0&0\\ 0&0&100&0\\ 0&0&0&100\end{bmatrix}\in\mathbb{R}^{(m+1)\times(m+1)}|_{m=n=3}\]  
The ED-DHP is implemented with the sampling time \(\Delta t=0.01s\), and the integration period of RK4 is 0.001s. The weights of the critic NN and actor NN are randomly initialized in \([-0.5,0.5]\). The positive definite matrices of ED-DHP are \(Q=\text{diag}\{800,800,800\}\) and \(R=\text{diag}\{0.1,0.2,0.2\}\). The learning rate of critic NN is \(\  
Figure 5: Tracking errors. It takes a little longer to learn to control the sideslip angle and bank angle than the attack angle. This can be attributed to the stronger coupling characteristics between the roll and yaw channels.  
Figure 6: Control signals. It is worth noting that the equivalent surface of the pitch channel is initialized near the rough trim value of -11 degrees. If the trim value of your plant is close to 0, you can initialize it at 0 degrees.  
Figure 7: Weights updating of actor NN and critic NN. The parameters of neural networks converge to a steady state at about 1.4 s.  
stability even though the learning rate does not change throughout the tracking process. However, it is worth noting that the angular rate loop has a large amplitude oscillation, especially the yaw rate loop and the roll rate loop. The phenomenon is mainly aroused by the cross-channel coupling and characteristics of the air vehicle's roll channel.