## 3 Reinforcement Learning-Based Results and Discussion
### Linking Manoeuvres  
Prior to demonstrating the sequential response of the two separate manoeuvres from a single agent, it was retrained with an increased pitch angle weight, and with noise applied to the starting conditions. As with the previous example, the selected manoeuvre was encoded with a 'one-hot' vector appended to the truncated longitudinal state vector. The increase in the pitch angle weight reduced the 'bleed' between the manoeuvres and allowed for a smoother transition between the two manoeuvres when linked sequentially. The added noise allowed for imperfect state conditions at the start of the episode, which would arise when linking together the manoeuvres sequentially. Although these combined changes could result in a poorer overall performance, signified by a lower overall reward function, the agent was more robust to noise in the starting state and demonstrated clearer separation between the two manoeuvres.  
The noise added to the starting conditions was distributed uniformly within a range defined by the run metadata. The values given for airspeed and flight path angle noise in Table 9 were the half-range of this distribution. For example, the airspeed value used for each episode was distributed uniformly within a range \(\pm 1.0\,\mathrm{m}\,\mathrm{s}^{-1}\) about the trimmed airspeed.  
Figure 11 shows the new response for the agent trained on two manoeuvres. The improvement can clearly be seen in the pitch angle response where the aircraft was much closer to having a pitch angle of zero for the descent at the end of the episode, and much closer to the vertical at the end of the episode when that manoeuvre was demanded. It is this instance of the trained agent that was used in the final demonstration where both manoeuvres were demonstrated sequentially in the same simulation.  
Figure 10: Run F: Simulation results showing the Multiple Manoeuvres from a Single Agent.  
The manoeuvres were chained together by transforming the vehicle's position for the second manoeuvre, such that the manoeuvre appeared to start at position \((x,z)=(0,0)\), replicating the conditions experienced during training. This transformation is represented in Equation (9). The remainder of the state vector remained untransformed. The simulation of this run can be seen in Figure 12, where the aircraft initially descended, and, at the conclusion of this first manoeuvre, the agent was signalled to transition into the vertical. It should be noted that the entry speed into the hover was faster than the agent experienced during the training episodes, which was why the aircraft was close to the vertical, but still at non-zero airspeed at the conclusion of both manoeuvres.  
\[(x^{\prime},z^{\prime})=(x-x_{T_{1}},z-z_{T_{1}}) \tag{9}\]  
The scenario shown here could represent transition through an opening into a confined space. It should be noted that the obstacles shown in the figure were not used within the simulation, but were used to demonstrate how these types of agents could be used in practice. With robust responses, multiple manoeuvres and agents could be chained together to create complex, nonlinear paths, capable of fast responses with lightweight controller implementation.  
Figure 11: Run G: Simulation demonstrating Multi-manoeuvre Agent with noise.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Timesteps & 1,000,000 \\ \(x_{\text{lim}}\) & 30 m \\ \(u_{\text{lim}}\) & 25 m s\({}^{-1}\) \\ \(w_{\theta}\) & 1 \\ \(w_{\text{climb}}\) & 1 \\ Use reduced observation & True \\ Network depth & 3 \\ Network width & 64 \\ Airspeed noise & 1.0 m s\({}^{-1}\) \\ Flight path angle noise & 0.2 rad \\ \hline \hline \end{tabular}
\end{table}
Table 9: Run G: Multi-manoeuvre with noise metadata used.