## 3 Reinforcement Learning-Based Results and Discussion  
### Descent Manoeuvre  
The initial manoeuvre explored and presented here was constrained descent. For this manoeuvre the objective is to achieve the maximum descent within a fixed forward distance, with the vehicle starting in trimmed level flight. The naive implementation simply provides a reward proportional to the final \(z\)-coordinate value when the vehicle crosses the \(x\)-coordinate limit. In typical aerospace convention, positive \(z\) is downwards so \(R=w_{z}z\) gives an increasing reward for increasing descent, assuming \(w_{z}>0\). Such an implementation induces descent behaviour, with one potential solution being a vertical dive. This can achieve an unlimited descent within a fixed forward distance. However, it has a number of problems. Firstly, increasing airspeed in the dive likely exceeds the validity of the model, and in the real world could potentially exceed airframe limits. Secondly, as the episode ends when the forward distance limit is reached, the agent seeks to maximise downwards velocity in the moments before episode end in order to maximise the reward. This likely leads to an extreme nose-down attitude, which is not conducive to continued controlled flight. To address these problems, consider the manoeuvre in the context of an approach to landing. The exit velocity should be limited, lest the vehicle overrun the runway, and the exit attitude should be close to level, assuming a conventional landing. Therefore, the exit attitude condition can be encouraged with the reward function  
\[R_{\text{end}}=(1-w_{\theta}|\theta-\theta_{\text{target}}|)w_{z}z,\]  
where the overall reward is weighted by the pitch error. In this case, \(\theta_{\text{target}}\) is 0, with a weighting term \(w_{\theta}\) left for tuning the reward function. The airspeed condition can be incorporated in a similar fashion by weighting the overall reward by the ratio of the vehicle's initial kinetic energy to that at the end of the episode:  
\[R_{\text{end}}=w_{z}z\frac{|v_{b,0}|^{2}}{|v_{b,\text{end}}|^{2}}.\]  
In this formulation, an increase in the vehicle's kinetic energy over the initial value causes a decrease in the reward, while a reduction in energy increases the reward. This has the \begin{table}
\begin{tabular}{c c c} \hline \hline
**Property** & **Measured** & **Unit** \\ \hline Mass, \(m\) & 1.221 & kg \\ \(I_{xx}\) & 0.019 & kg m\({}^{2}\) \\ \(I_{yy}\) & 0.091 & kg m\({}^{2}\) \\ \(I_{zz}\) & 0.121 & kg m\({}^{2}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Mass and inertial properties of the MXS2 airframe. effect of encouraging a bleed-off of energy during the manoeuvre, but this is ideal as part of a landing approach manoeuvre. With both of these modifications in place, there is still an edge case that the agent can exploit. Assuming there is sufficient forwards distance, the vehicle can be put into a potentially unlimited vertical dive and pull up into a climb to bleed off the excess energy prior to hitting the forward limit. This is not a particularly useful solution in an approach to landing scenario, unless the landing site is located on a cliff edge. To combat this, the reward function is further augmented to penalise the amount of climb the vehicle carries out during the manoeuvre:  
\[R_{\text{end}}=\frac{w_{z}z}{1+w_{\text{climb}}(z_{\text{max}}-z)}.\]  
This reduces the award for any end position above the lowest point in the manoeuvre. The \(w_{\text{climb}}\) parameter is used for tuning the reward. There are a few further edge cases that the agent could exploit that are imposed with hard limits, rather than modifying the shape of the end-of-episode reward. If any of the conditions are breached, the episode is ended immediately and a negative reward given to the agent. First, there is a lower limit on the \(x\)-coordinate to prevent the agent using more forwards space than intended by flying the aircraft back past the initial position. Second, there is a set of conditions to prevent the agent looping the vehicle.