## 4 Conclusions and Further Work  
A new nonlinear aircraft model was used within a reinforcement learning framework to train agents to perform different longitudinal manoeuvres, including those at a high angle of attack where response is highly nonlinear. The approach taken is shown to be able to encode for manoeuvres including transition into the hover and the descent. Furthermore, a single agent was trained for two manoeuvres, which were subsequently demonstrated in a single simulation where both were performed sequentially.  
There are many avenues available for further work, including additional randomisation, both on the input state and in the simulation environment in which the agents are trained. This should include the effects of gusts and steady state wind. The results shown here are for the longitudinal response. It would be interesting to extend the training to include the lateral-directional response, to allow for nonlinear flight regimes such as the spin.  
Whilst the model is based on both wind tunnel data and additional resources, it would be useful to further explore the fidelity at high alpha. This would also benefit from taking the model and the trained agents into free-flight. One reason for the selection of the baseline aircraft is that it is available for free flight tests in the same configuration that it was tested in the wind tunnel. Comparison of simulated and free-flight data through manoeuvres and model identification from free-flight data will help to highlight continuing model inaccuracies and the remaining reality gap.  
Training the model for multiple manoeuvres comes at the expense of increased training time. The trade-off between the use of linked multiple agents for simplicity, against the use of a larger network single agent needs to be evaluated through manoeuvre performance, training time, overall weight size and robustness. Increased randomisation of the initial conditions would allow a single agent to cope with variation in the end points. Increased training with stricter conditions on the endpoints would also help to facilitate robust transition between manoeuvres. The networks would also benefit from further hyper-parameter optimisation. The use of eXplainable AI (XAI) methods may help in this optimisation effort, and in assessing the efficacy of the current approach. The use of such methods in a control context, with coupled input states, is an area of active research [5; 20].  
Finally, reinforcement learning-based agents were shown to be capable of exploiting the high-alpha part of the flight envelope through rapid manoeuvres. This was achieved through the use of complex reward structures, and our future work will consider the inclusion of these agents in a multi-modal form of control where there are transitions between more traditional model predictive control and the rapid manoeuvres that this type  
Figure 12: Run G: multi-manoeuvre with noise sequentially evaluated output. Shown with representative obstacles.  
of agent can generate. Flight into, and out of, confined spaces with fixed-wing vehicles will benefit from rapid, nonlinear, agile control structures.  
Investigation and methodology, R.C. and L.F.; software and visualization, R.C.; supervision T.R. and S.E.; writing R.C., T.R. and S.E. All authors have read and agreed to the published version of the manuscript.  
This work was supported by the Engineering and Physical Sciences Research Council grant number EP/N509619/1.  
The code used in generating these results is available on GitHub at: [[https://github.com/rob-clarke/pymxs](https://github.com/rob-clarke/pymxs)]([https://github.com/rob-clarke/pymxs](https://github.com/rob-clarke/pymxs)) (accessed on 14 July 2023). The repository also includes the wind tunnel data and the longitudinal model implementation under the wind_tunnel_data and models directories respectively. Run data and trained models are available in the same repository in the archive branch.  
The authors declare no conflict of interest.