## 3 Reinforcement Learning-Based Results and Discussion
### Hover Manoeuvre  
The next manoeuvre explored and presented here is a hover entry. The main objective was for the agent to control the vehicle from level trimmed flight through to a hover. There was an 'easy' solution in the vehicle pitching to the vertical and maintaining that attitude until the speed reduced to 0. In order to provide a more interesting environment and challenge, a secondary objective was also encoded into the reward function. Initially, this secondary objective was to minimise the height gain during the manoeuvre. To motivate this, again consider the manoeuvre as part of an approach to a landing sequence, this time for a tail-sitter landing. The hover condition is much less efficient than \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Timesteps & 500,000 \\ \(x_{\text{lim}}\) & 45 m \\ \(u_{\text{lim}}\) & 25 m s\({}^{-1}\) \\ \(w_{\text{fo}}\) & 0 \\ \(w_{\text{climb}}\) & 1 \\ \(w_{\text{wp}}\) & 0.15 \\ Waypoint positions & \((10,-2)\) and \((30,5)\) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Run B: Descent with Waypoints metadata used. Figure 6: Run B: Simulation results showing the Descent Response With Waypoints. conventional flight, so to minimise energy expenditure during the final descent phase, the transition to hovering flight should take place as low as possible. The initial approach was to check at each timestep if the vehicle had achieved a state within some acceptable bounds of a hover, with the assumption that a conventional control system would then be able to stabilise the hover. The bounds used for the hover state are listed in Table 4. If the vehicle was deemed to be in the hover, the episode was ended and the episode reward was proportional to the final height of the vehicle: namely, \(R=z_{\text{end}}+c\). The constant offset was tuned to ensure that heights above the starting position (that is a negative \(z\)-coordinate) did not result in a negative reward. This initial approach proved difficult to train. Most runs resulted in the vehicle simply falling until the episode was terminated by the time or position limits. The reward signal was only given in the constrained part of the state space near the hover condition, which made it very sparse for the learning algorithm, with the gradient in the reward space difficult to determine. In order to improve this, a time limit was added to the episodes, after which the reward would be based on the vehicle's progress towards the hover state, as shown in the reward function  
\[R_{\text{end}}=100\bigg{(}\frac{1}{1+|q|}\cdot\frac{1}{1+|90^{\circ}-\theta|} \cdot\frac{1}{1+|u|}\cdot\frac{1}{1+|w|}\bigg{)}.\]  
For a perfect hover, this function returned a value of 100, with any deviation from hover discounting this. The constant offset above was also tuned to ensure a successful hover resulted in a reward greater than 100. Figure 7a,b show the evaluated outputs for an agent trained with the hover reward function. The parameters used for training the agent are listed in Table 5. The horizontal velocity at the end of the episode was still a little high for the hover condition, so the episode timed out. Of note was the initial throttle input, which was likely to increase elevator effectiveness. This was followed by a period of no throttle to minimise height gain. The throttle then increased at the end of the episode to maintain the hover. The overall height gain throughout the manoeuvre was \(\approx\)10 m. Note also the rapid transition into the high alpha regime as the throttle came back on at approximately 1.8 s.  
During training for this manoeuvre, another solution was generated in addition to the expected aggressive pitch-up, this being a nose-down manoeuvre. In the nose-down variation, the vehicle pitches down, passing through extreme negative angles of attack before ending the episode pointing upwards, usually significantly below the starting point and, potentially, very close to the hover condition.