ML Optimisation for Portfolio Allocation
By: Machine Learning, Nus Fintech Society
Tags: Machine Learning, Artificial Intelligence, Market Research Reports, Portfolio, Trading

Authors: Darius Oh (Tech Lead), Mayve Chua, Luke Lau, Sng Hao Jun, Jamais Ang, Kexin Wang, Jacob Pang

Introduction

Portfolio Optimisation is the process of selecting the best portfolio, or rather the best set of possible weights for a basket of assets, given some sort of objective function. Common objective functions include maximising returns, minimising risk, maximising return-risk ratio or minimising tail risk.

The aim of our study is to understand deeper into how Machine Learning can be used to tackle such a complex asset allocation problem, and potentially devise a quantitative strategy that exceeds traditional methods.

The most common strategy that comes to mind when we talk about Portfolio Optimisation would be Markowitz’s Mean-Variance Optimisation. It aims to maximise portfolio returns while minimising portfolio risk, or rather the portfolio’s volatility. For example, investing in two separate negatively correlated stocks instead of a single stock is likely to reduce the risk, or in this case the volatility, of the portfolio. Such an action can also potentially generate greater returns even with the reduced risk. However, Markowitz’s model requires the forecasted portfolio return and volatility as an input to the model. Instead of a simple moving average forecast, we see an opportunity to use Machine Learning to gather a more accurate forecast of the future expected portfolio returns and volatility. With the assumption that the underlying theory of Mean-Variance Optimisation works, we would then be able to come up with an optimal portfolio. This strategy would, hence, combine more advanced Machine Learning techniques with traditional financial theory.

On top of this, we also wanted to explore a second approach to Portfolio Optimization through a pure Machine Learning strategy. This would allow us to come up with a new strategy that is independent of the reliance of traditional financial theory. Recently, much focus has been placed on Reinforcement Learning for Dynamic Portfolio Optimisation, as it is able to handle the complex sequential decision-making process. Hence, we wish to explore building our own Reinforcement Learning model to handle the dynamic optimization process and maximise portfolio returns.

Dataset Used

Our models will be trained purely with historical OHLC (Open-High-Low-Close) price data. This means that it would not account for changes in the macroeconomic environment such as Interest Rates or Inflation. We collected the historical price data from the year of 2001 until the year of 2019 using yfinance Python API. We excluded earlier time periods as we believe that the price signals in the market performed much more differently as compared to more recent times.

Our stock universe would be the constituent stocks under the Dow Jones Industrial Average Index (DJI) as of 29 August 2021, excluding stocks listed after 2001 due to the lack of data. We chose DJI as it would provide us with a small enough stock universe that can cover a diverse array of industries. In addition, we would also be able to benchmark our portfolio performance against the performance of the index over time. This eventually left us with 27 stocks in our stock universe, coming from various fields of industry.

Approach 1: Mean-Variance Optimization

Asset Returns Forecast Model

Motivation

To support Markowitz’s model for portfolio optimization, we aim to explore using machine learning models to forecast the returns for each of the 27 chosen stocks. In which, our team has narrowed down to three more popular stock predictive machine learning models namely Decision Tree, Long Short-Term Memory (LSTM) and Autoregressive Integrated Moving Average (ARIMA) to kickstart our exploration. Our team has chosen to explore LSTM and ARIMA as it has proven to support time-series prediction well. On the other hand, we have also chosen a decision tree as our team is interested in exploring the performance of a regression tree in the context of price prediction.

Model Exploration

All three models will predict for the target variable, log returns (generated using the adjusted close prices) of each stock ticker and the performance of each model will be compared and evaluated using mean square error (MSE) and root mean square error (RMSE). Adjusted close price is used for determining the log returns as it is more comprehensive and takes into account dividends, stock splits and factors that happen after the market closes.

For the Decision tree model, our team used Adjusted Close Price and Volume as input variables to predict for the log returns of each ticker. Meanwhile for LSTM and ARIMA, for every 50 time steps in the historical log returns dataset of each ticker, our team will utilize the log returns in these time steps as training data to predict for the next time step (testing data). Relative Strength Index (RSI) was also included. After training our model, we evaluated the performance of the model, and the results are shown in the result table below.

Table 1: RMSE and MSE of models

From the results, it can be seen that the decision tree model performed the best as it has the lowest RMSE and MSE. However, this is because the training data contain Adjusted Close Prices which is used to calculate the target variable, log returns and hence the model trained can be said to be biased and the results are overfitted. This is undesirable as the model may not be able to perform as well when fitted with unseen data. However, our team could not fit the training data of LSTM and ARIMA to the decision tree as the decision tree is unable to support the ratio of having many of the same input variables to 1 target variable (1: n) like that of LSTM and ARIMA. Hence, it can be concluded that the decision could only support 1:1 training data making it unable to support the context of our project. As such, despite the decision tree performing the best, our team has decided not to move forward with this model.

On the other hand, it can be seen that the ARIMA model performed the worst. Theoretically speaking, this could be due to the fact that ARIMA is powered by linear regression and hence it assumes that the time series training data being fed is stationary. This means that ARIMA assumes that the training data has no seasonality and has a constant mean, variance and covariance over time. This would thus not support the prediction well as the training data used in this project are long-term (at least 15 years) and contain seasonality in the data itself. As such, it can be said that the ARIMA model is more suited for prediction of short-term time-series data where there is no seasonality as compared to long-term dataset where there exists seasonality. With that, our team has also decided not to move forward with this model.

Therefore, we have decided to move forward with the LSTM model as it is able to handle seasonality in data which the ARIMA model cannot handle as well as handle 1: n training data which the decision tree model cannot handle.

Hyperparameter tuning for LSTM

To further improve on our chosen model, our team has decided to perform hyperparameter tuning to our LSTM model by tuning the learning rate, number of dropout layers and the number of neurons in the first layer. Tuning the learning model will allow the model to be more efficient as it controls how quickly the model can converge to a local minima. On the other hand, tuning the number of dropout layers and the number of neurons in the first layer will improve the performance of the model as it prevents the model from being under or overfitted. To do so, Keras in-built hyper tuner was used to tune the learning rate and number of neurons.

Evaluation Results for LSTM (Apple Ticker)

With that, our team has used the LSTM model to predict for the log returns of each ticker and the actual vs predicted result of the model when predicting Apple stock can be seen in the figure below.

Figure 1: Actual vs Predicted Result of Apple Ticker using LSTM model

Limitations and Further improvements

Due to the limitation of time, our team kept our maximum epochs to 10 to reduce training time which could compromise the performance of our model. This is because the number of epochs is related to the number of rounds of optimization that are applied during training and hence with more epochs, the error on training data will reduce which will in turn improve performance of the model. Thus, it can be said that by limiting the maximum number of epochs to 10 can result in a non-optimal number of rounds of optimization and thus a less optimal performance LSTM model being trained. Having said that, a room for improvement would be to tune the number of epochs when training the model so as to further improve the performance of our model. Furthermore, we can look into using a bi-directional LSTM which has been shown to be able to capture additional features from the data which is not able to be captured by a unidirectional LSTM.

Considering the scope of our project, a more in-depth time series analysis could be completed to further improve the final performance of our model. By analysing and accounting for possible seasonality present in the various time series of the stock prices, it would be easier for the neutral network to learn and hence potentially achieve better performance and more accurate results. For example, to get the model to learn the seasonality of the data, we can use the difference (P(t +1) -P(t)) as the input feature instead, this would allow the generation of a leading model and at the same time prevent the model from having to solely reference yesterday’s price, effectively mitigating the problem of a lagging model.

Going beyond the scope of our project, future work could look into including qualitative data to improve the robustness of our overall model. Sentiment analysis done on diverse data sources such as tweets, news articles, company reports and financial statements could also be included and combined into the deep learning model using feature engineering techniques. By doing so, there is a high potential to develop a more comprehensive prediction system that considers both quantitative and qualitative data.

Asset Covariance Forecast Model

Motivation

The covariance matrix is the other essential input for portfolio optimisation. In order to formulate a predicted rolling covariance matrix, we had to explore how to develop a predicted rolling volatility and rolling correlation matrix. Since a rolling correlation matrix is easily generated, the difficulty here was mainly exploring a logical way to predict rolling volatility.

Harry Markowitz in Portfolio Selection paper (1952) introduces the concept of volatility. Volatility of a portfolio is the standard deviation of the returns of this portfolio. This concept continues to be used in financial markets to define uncertainty/risk. However, most allocation and option pricing models assume that volatilities are constant through time. This assumption is flawed in that volatility clustering is observable in practice: periods of low volatility tend to be followed by periods of low volatility and periods of high volatility tend to be followed by periods of high volatility.

Volatility clustering represents a challenge being an example of heteroskedasticity. In order to alleviate this, Engle (1982) proposed the Autoregressive Conditional Heteroskedasticity (ARCH model). Then, Bollerslev (1986) built upon Engle’s model and published his General Autoregressive Conditional Heteroskedasticity (GARCH model).

Thus, we decided to train a GARCH model to predict future volatility as this would address volatility clustering. We create a methodology that enables the estimation of a GARCH model while using the most robust parameters.

Model Exploration

Bollerslev (1986) proposes an extension of the previous model, in which he adds an MA (moving average) part to the equation:

Such a process is written GARCH (p, q). β is a new vector of weights deriving from the underlying MA process, we now have γ + ∑α +∑β = 1. For example, a GARCH(1,1) process has p = 1 and q = 1. It can be written as:

In order to estimate ω, α and β, we usually use the maximum likelihood estimation method. In this case, it consists of maximizing:

In our exercise, we tested GARCH models with parameters p=1 to 5 and q=1 to 5, resulting in 25 possible combinations of p’s and q’s on each of the assets. In order to not overfit the GARCH model, we selected the model that is able to do well in the scoring metrics across all assets.

Evaluation Results

In order to select the set of hyperparameters that are optimized for the smallest extent of errors, we first chose 3 common error metrics — Root Mean Squared Error (RMSE), Mean Squared Error (MSE) and Mean Absolute Error (MAE) in our GARCH modelling analyses performed on each of the 30 assets in our portfolio.

Figure 2: Results of GARCH Modelling using MSE, RMSE and MAE

As the GARCH modelling may possibly yield differing sets of hyperparameters p and q for each asset in the portfolio, we explored a naïve way of taking the mean of the values of p and q across all 30 assets in the portfolio for each error metric. As seen in the figure above, all 3 error metrics return the same set of mean hyperparameters (p = 3, q = 5).

Following this, we used p = 3 and q = 5 as inputs for our hyperparameters in the GARCH modelling to obtain the predicted volatility values which would then allow us to derive the rolling volatility values.

Figure 3: Covariance Matrix of our Portfolio (showing only 10/30 assets)

Together with the rolling correlation values, the covariance matrix can then be calculated.

Limitations & Further improvements

Naïve Mean method

The method of taking the average of the hyperparameters across all the assets of the portfolio may bring about inaccuracies in the GARCH modelling as the hyperparameters p and q represent the extent of which the past values have an influence on the current value. Taking the average of p and q across assets is essentially implying that — if I have a portfolio of 2 assets where it is optimal to take the past x and y values to derive their current respective values, I can proceed to take the past (average of x and y) values to derive the current values for both assets in the portfolio. This is a naïve method as it banks on the assumption that the extent of errors that result from each asset is the same for all assets in the portfolio, which is not necessarily true in reality.

An alternative way that can be explored in future to obtain values p and q might involve running the GARCH modelling on the portfolio returns instead of on individual assets. For example, we can first aggregate across all 30 assets in the portfolio to obtain portfolio returns values at each timepoint. Thereafter, we can run the GARCH modelling on the portfolio to observe what sets of values of p and q are returned with the lowest error values using the same 3 error metrics as illustrated above. This set of values of p and q may arguably represent more accurately the extent of which past values have an influence on the current value.

Optimization

Model Implementation

Having obtained the forecasted daily asset returns from the LSTM model as well as the forecasted daily asset covariance from the GARCH model, we are now able to use these data as input for mean-variance optimisation. We make use of the PyPortfolioOpt Python Library which helps us to perform such optimisations efficiently.

Figure 4: Efficient Frontier Plot using 20000 randomly generated weights

The efficient frontier line is the set of optimal portfolios that offer the highest expected return for a defined level of risk or the lowest risk for a given level of expected return. Portfolios above the line are unattainable regardless of the asset allocation, while portfolios below the line are attainable but are less optimal. In order to decide on which optimal portfolio on the efficient frontier to choose, we have to fix an objective function for the optimisation. This could be maximising return-risk ratio, minimising volatility, maximising return with a risk constraint or minimising volatility with a returns target.

For each trading day, we will generate the Efficient Frontier using the current day’s forecasted returns and forecasted covariance. Based on the objective function that we set, we will obtain an optimal portfolio allocation that optimizes for the objective function. This would then give us the weights to hold for each asset for the day. This implies that there will be a rebalancing of the portfolio at the start of each trading day.

In this study, we will be exploring maximising return-risk ratio as well as maximising return with a given risk constraint. This will be further discussed in the Performance Comparison section.

Limitations

The key limitation of the mean-variance optimization strategy is its assumption that returns follow a normal distribution. However, this may not be applicable to some assets which display skewness in the returns data. Since the variability below and above is no longer the same (i.e. downside variance could be much greater than upside variance), it would now make more sense for an investor to work towards minimising downside variance rather than the portfolio variance as a whole. A research paper by Andrea Rigamonti discusses this more elaborately, and pushes for a new strategy known as mean-semivariance strategy. Semivariance is similar to variance, but it only accounts for observations below the mean. This potentially makes it a more useful measure of portfolio risk, especially when returns are not normally distributed.

Another key limitation of our optimisation strategy is that it is not extendable to other assets outside of our stock universe. As we are training the machine learning forecast models specific to only the 27 assets of interest, we would not be able to use the same models to forecast the data for other assets which can exhibit very different behaviours.

Approach 2: Reinforcement Learning

Brief Overview

Reinforcement Learning is an area in machine learning where we train an agent to perceive and interpret its environment, take actions and learn through trial and error. Given a specific situation or environment, the agent would be able to make the best possible decision to take that can result in the maximum reward attained. In the context of our problem statement, our agent would be making decisions on what stocks our portfolio should hold on each trading day in order to maximise the final cumulative portfolio returns.

Environment

For every step in each episode, the Environment supplies an observation to the Agent, accepts the corresponding action,

which represents the portfolio weights to adopt prior to open, and executes the interactions between the internal portfolio state and the changes in the financial market. Transaction costs of 0.2% were applied on the transition of weights from the prior close, wt-1 to, at to simulate actual trading. Using the vector of assets’ returns rt, the relative portfolio weights at close is then given by

The effective portfolio return (with respect to the prior close) for the step is equivalent to

. Since the rewards are additively accumulated across the episodes, applying logarithm enables geometric returns to be captured in the rewards. For the purposes of setting a common basis for comparison across episodes starting from different dates, the return relative to a benchmark return rate, bt is used as a measurement for the reward ut. S&P 500 was used as the benchmark for this project.

The observations were comprised of the following features, separated into components:

a. Asset prices, returns, expected returns for the previous 20 days.

b. Asset covariance based on the previous 180 days, naive markowitz tangent portfolio weights, and the prior close portfolio weights.

Agent

In order to work with the continuous space of the action required (portfolio weights), Deep Deterministic Policy Gradient (DDPG) was used for the RL agent. A custom actor network was used, consisting of a convolutional asset scoring network and secondary dense network. The first component of the observations is fed into the asset scoring network, on an individual asset basis, to obtain an asset score. The asset scores, along with the second component of the observations are then fed into the secondary network to obtain the action. This design prevents the network from overfitting on underlying asset-specific trends which therefore trades some performance for ability to generalize outside the sample.

Actor Network Architecture

Evaluation Results

The results for the k-fold validation are as shown for the model after 50 episodes of training and 100 episodes of training. With the exception of 2020, the model with 100 episodes of training performs better as compared to the model with 50 episodes of training. The models generally outperform the benchmark during stable periods, but underperform during the recovery periods post-shocks.

Limitations

As deep RL requires extensive training and computation, time and investment into the project poses as the primary limitation, which is constrained further by applying k-fold validation. As each action influences the environment state, a permutation of sequences of actions over the same episode provides different training to the agent. In the context of the portfolio optimization problem, where the initial state already varies based on the starting date, it is improbable for an agent trained on 50 or 100 episodes to have covered sufficient exposure. We expect that given sufficient time, the RL model can perform better.

Performance Comparison

Portfolio Performance Results

Our portfolios are generated via three methods —

● Baseline: Mean-Variance Optimisation using rolling average of past returns as forecast

● ML: Mean-Variance Optimisation using forecasts from LSTM and GARCH

● RL: Reinforcement Learning

We include the Baseline portfolio as a comparison to observe the advantages that Machine Learning can bring as compared to a much simpler strategy. In addition, both the Baseline & ML are optimized to maximise return-risk ratio.

Additionally, we decided to filter away the portfolio performance past 2020 for this analysis because our Machine Learning Models have not been able to provide reliable forecasts during periods of unforeseen market shock.

Figure 5: Cumulative Returns of portfolios

Table 2: Summary Statistics of portfolios

From the table above, we can first see that our RL portfolio performs significantly worse than the other two portfolios. It has the lowest annualized returns as well as the lowest return/risk ratio as well. The potential reason for this poor performance has been discussed under Limitations in the previous section.

When comparing the Baseline portfolio against the ML portfolio, we can see that the ML portfolio has a much higher return/risk ratio. While it does have a slightly lower annualized return, it has a significantly lower annualized volatility in comparison to the baseline portfolio. This means that while both portfolios can attain a similar yield, the ML portfolio is able to provide us with a less risky option as compared to the Baseline portfolio. We also wish to explore the flip side where we fix a risk constraint for both portfolios and aim to maximise returns given the same set of constraints.

Figure 6: Cumulative Returns of portfolios

Table 3: Summary Statistics of portfolios

From the table above, we can now see that in order to adhere to the risk constraints, the Baseline Portfolio has a reduced Annualized Volatility together with a reduced Annualized Returns. In contrast, our ML Portfolio is able to take up slightly more risk but has resulted in a significant increase in Annualized Returns. With this objective function, our ML Portfolio achieves a significant higher yield for a similar risk level as compared to the Baseline Portfolio.

Further Discussions

It is useful to note that when we changed our objective function from maximising return/risk ratio to maximising returns, it actually resulted in an increase in the return/risk ratio of the portfolio. This is very unexpected as it means that the ML portfolio generated when maximising return/risk ratio is still not optimal. The reason for this is because we optimize for the objective function using forecasted data. However, due to our forecasts not being able to follow closely enough to the actual numbers, our “forecasted optimal” portfolio will not be able to match the “actual optimal” portfolio.

However, we can see that for the optimisation of the two different objective functions, using the forecasts from our Machine Learning models has proved to achieve better portfolio performance as compared to using a naive method.

Conclusion

From this study, we can observe the potential advantages of using more advanced Machine Learning techniques to make investment decisions. While there has been much skepticism and distrust due to the black-box nature of these techniques, it is without a doubt that many investors and institutions are moving towards such strategies due to the upsides that it brings. Whether it is to complement an existing traditional strategy or to devise an entirely new strategy, Machine Learning is definitely the area that the quantitative finance industry is moving towards and increasingly placing emphasis on.

Sources:

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3465717

https://towardsdatascience.com/free-stock-data-for-python-using-yahoo-finance-api-9dafd96cad2e

https://pyportfolioopt.readthedocs.io/en/latest/MeanVariance.html

https://medium.com/analytics-vidhya/why-you-should-not-use-arima-to-forecast-demand-196cc8b8df3d

https://arxiv.org/pdf/1911.09512.pdf

https://medium.com/analytics-vidhya/analysis-of-stock-price-predictions-using-lstm-models-f993faa524c4