Automated algorithmic trading: machine learning and agent-based modelling in complex adaptive financial markets

Over the last three decades, most of the world's stock exchanges have transitioned to electronic trading through limit order books, creating a need for a new set of models for understanding these markets. In this thesis, a number of models are described which provide insight into the dynamics of modern financial markets as well as providing a platform for optimising trading and regulatory decisions. The first part of this thesis proposes an autonomous system that uses novel machine learning techniques to predict the price return over well documented seasonal events and uses these predictions to develop a profitable trading strategy. The DAX, FTSE 100 and S&P 500 are explored for the presence of seasonality events before an automated trading system based on performance weighted ensembles of random forests is introduced and shown to improve the profitability and stability of trading such events. The performance of the models is analysed using a large sample of stocks and the results show that the system described in this section produces superior results in terms of both profitability and prediction accuracy compared with other ensemble techniques. The second part of this thesis explores price impact. For many players in financial markets, the price impact of their trading activity represents a large proportion of their transaction costs. This section of the thesis proposes an adaptation of the system introduced in the first part for predicting the price impact of order book events. The system's performance is benchmarked using ensembles of other popular regression algorithms including: linear regression, neural networks and support vector regression using depth-of-book data from the BATS Chi-X exchange. The results show that recency weighted ensembles of random forests produce over 15% greater prediction accuracy on out-of-sample data, for 5 out of 6 timeframes studied, compared with all benchmarks. Finally, a novel procedure for extracting the directional effects of features is proposed and used to explore the features most dominant in the price formation process. The final part of this thesis addresses the requirement for testing algorithmic trading strategies laid out in the Markets in Financial Instruments Directive (MiFID) II by describing an agent-based simulation. Five types of agent operate in a limit order market producing a model that is able to reproduce a number of stylised market properties including: clustered volatility, autocorrelation of returns, long memory in order flow, concave price impact and the presence of extreme price events. The model is found to be insensitive to reasonable parameter variations. Finally, the model is used to explore how trading strategy affects the implementation shortfall of trading a large order. A number of execution strategies with various order types, are evolved and evaluated in the agent-based market. It is shown that the evolved strategies outperform the simple, well known strategies significantly, suggesting that execution strategy plays an important role in determining the implementation shortfall of trading large orders.