End To End Python Implementation Of Finding Optimised Efficient Investment Portfolios
By: Farhad Malik
Tags: Data Science, Finance, Fintech, Programming, Python

End To End Python Implementation Of Finding Optimised Efficient Investment Portfolios

Explaining How To Build The Efficient Investment Portfolios From Start To End

One of the milestones of the investment management application was to implement an end to end solution that starts by fetching company stock prices and builds a set of efficient and optimum portfolios using optimisation routines.

This article will demonstrate how to achieve it.

Please read FinTechExplained disclaimer. This application is based on my opinions and they could be wrong. Always seek advice from a professional financial advisor before investing your money. This article aims to explain portfolio optimisation concept at an abstract level and it should not be considered as an investment advice.

1. Article Aim

This article will document the following key points:

The Theory Of Efficient Frontier The Process Flow Of The End To End Solution Implementation in Python Path Of The Complete Code Next Steps

2. The Theory Of Efficient Frontier

We hear the term “efficient frontier” quite a lot but what does it mean exactly? And what is the Modern Portfolio Theory all about?

Burton G Malkiel expressed it in a perfect manner:

“The theory provides a firm foundation for the intuition that you should not put all your eggs in one basket and shows investors how to combine securities to minimize risk” (Burton G. Malkiel).

Key Note: Never put all of your eggs in one basket. Make sure you diversify!

Photo by Diggity Marketing on Unsplash

Let’s consider you have $10'000 of cash available and you are interested in investing it.

Your aim is to invest the money for a year.

Like any rational investor, you expect the final amount in a years time to be higher than the $10'000 amount you want to invest.

The last thing you want is to lose the $10'000 you started your investment with.

So far, so good!

Key Note: I am a strong believer in maintaining focus when we are attempting to reach a goal. And the key to meet that goal is to adapt swiftly to the changing environment.

Read this article if you want to gain an understanding of investment management:

Photo by Samuele Errico Piccarini on Unsplash

2.1 Understanding Risk-Return Relationship

There are many investment options available, such as buying a T-bill or company shares, etc. Some of the investment options are riskier than the others because they attract us to gain higher returns.

Hence, the point to note is that there exists a risk-return trade-off. As an investor, our sole aim might be to invest in the least risky investment option that yields the highest return.

If we buy a number of assets such as shares of different companies then the total risk of the portfolio can be reduced due to diversification. This means that an investor can reduce the total risk and increase the return by mixing assets with different proportions. This is due to the fact that the assets can be correlated with each other and combining them together can reduce the risk. Hence it lets us produce a good portfolio.

This brings the question: What Is A Good Portfolio?

A good portfolio is more than a long list of good stocks and bonds. It is a balanced whole, providing the investor with protections and opportunities with respect to a wide range of contingencies. — Harry Markowitz

The Modern Portfolio Theory of Nobel prize winner, Harry Markowitz can be implemented by using optimisation routines.

Photo by Will Tribe on Unsplash

3. The 7 Steps Of Efficient Frontier Process

Before I present the end-to-end implementation of the solution, I wanted to highlight a key technical note here. The core of the architecture of this application revolves around just one concept:

When the application runs, it produces a large amount of data such as stock prices, their returns, covariance matrix, portfolios and their allocations along with their risk, return and Sharpe ratio. The Python code saves all of the required data in an Excel spreadsheet. It’s located is specified in the settings file.

We need to perform the following 7 steps:

The three stages of seven steps

The first stage (steps 1–3) is to calculate the daily returns of our stocks. The second stage is step 4 whereby we need to optimise the allocations to retrieve the best portfolio for each of our target returns. The last stage is about using the allocations to calculate the risk and return and plot the portfolios.

Photo by Ion Şipilov on Unsplash

4. Starting The Implementation In 3 Stages

This section will document the Python code with an explanation of how I have implemented the solution.

The following packages are required to support the application:

Pandas: for data manipulation

Numpy: for calculations

Matplotlib: for charting

SciPy: for optimisation

4.1 First Step Is To Fetch The Stock Prices

We built a component that fetched the historical stock prices:

end_date = settings.get_end_date()

start_date = settings.get_start_date(end_date)

closing_prices = price_extractor.get_prices(settings.PriceEvent, start_date, end_date) #plot stock prices & save data to a file

cp.plot_prices(closing_prices)

fr.save_to_file(closing_prices, 'StockPrices')

The stock prices are time-series data.

This is the snapshot of the data. The code stored the data in the StockPrices sheet of the excel spreadsheet as shown below.

As you can see, we are unable to compare ZM with SYMC because the stock prices are of different magnitude. Hence, we need to compute the returns to standardise the stock prices.

4.2 We Need To Generate The Returns From The Prices

We need to compute their geometric returns by calculating the following equation:

The returns are generated so that we can standardise the stock prices and compare them.

returns = settings.DailyAssetsReturnsFunction(closing_prices, settings.ReturnType) #plot stock prices & save data to a file

cp.plot_returns(returns)

fr.save_to_file(returns, 'Returns')

The returns are time-series data. This is the snapshot of the data which is stored in the Returns sheet of the excel spreadsheet as shown below.

4.3 Asset Expected Mean Returns And Covariance Matrix

There are essentially two steps involved:

expected_returns = settings.AssetsExpectedReturnsFunction(returns)

covariance = settings.AssetsCovarianceFunction(returns)

#Plot & Save covariance to file

cp.plot_correlation_matrix(returns)

fr.save_to_file(covariance, 'Covariances')

The first step is to compute a single number to represent the returns of the assets.

One of the most common ways to represent the returns as a singl number is to compute the mean of the returns which is known as the expected mean returns.

To compute the asset expected mean return, we need the mean of the returns of each stock calculated as:

From the asset returns, we can compute the portfolio returns. The assets within the portfolio have been allocated a proportion of the total investment amount. As an instance, the portfolio might hold 40% of asset ABC and 60% of asset DEF.

Note: Are historical returns the right choice? Should we instead generate different measures? I will implement a superior application in the future that demonstrates how we can implement time-weighted returns instead.

2. The second step is about generating the Covariance Matrix from the asset returns

The code then prepares the covariance matrix:

The covariance matrix is stored in the Covariances sheet of the Excel spreadsheet as shown below:

The volatility of the portfolio is the risk of the portfolio. The volatility is computed by calculating the standard deviation of the returns of each stock along with the covariance between each pair of the stocks by using this formula:

Volatility in this instance is the standard deviation i.e. the total risk of the portfolio.

Standard deviation measures the dispersion of the values around the mean.

4.4 Now Important Point: The Monte-Carlo Simulation & The Efficient Portfolios Line

Let’s first generate 100’000+ portfolios via Monte Carlo simulation first. The allocations will be generated randomly.

This technique relies upon the theory that as we generate a portfolio a large number of times, we will eventually reach the formation of the true optimum portfolio. Remember, here the generation of a portfolio implies randomly generating the allocations (weights).

This article demonstrates the Monte-Carlo simulation in detail:

If we plot the risk and return for each of the portfolios on a chart then we will see an arch line at the top of the portfolios.

This yellow line is essentially pointing at the portfolios that are the most efficient. This line is known as the efficient frontier.

The efficient frontier is a set of portfolios that give us the highest return for the lowest possible risk.

Every other portfolio that does not reside on the efficient frontier is not as efficient because it offers the same return as a portfolio on the efficient frontier but by taking a higher risk.

Any other portfolio is therefore inferior to the portfolios on the efficient frontier. As a result, we can literally ignore the portfolios that are not on the efficient frontier line.

Executing Monte-Carlo Simulator:

portfolios_allocations_df = mcs.generate_portfolios(expected_returns, covariance, settings.RiskFreeRate) portfolio_risk_return_ratio_df = portfolios_allocation_mapper.map_to_risk_return_ratios(portfolios_allocations_df)

The portfolios are stored in the MonteCarloPortfolios sheet of the Excel spreadsheet as shown below:

5. The Key Step — Portfolio Optimisation

Now let’s produce an optimum portfolio by running the optimisation techniques.

We are going to use the SciPy package. The scipy.optimize module, within the SciPy package, offers a variety of optimisation algorithms. The module includes un/constrained algorithms, global optimisation routines, least-squares minimisation, scalar and multivariate minimisers etc.

It offers Newton-Krylov, Newton Conjugate Gradient, SLSQP, curve fit and dual annealing algorithms amongst others.

5.1 SciPy minimize function

There is a minimize function within the scipy.optimize module that performs the minimisation of a scalar function of one or more variables. The minimize function makes it easier for us to execute the required algorithm on an objective function.

The signature of the method is:

scipy.optimize.minimize(fun, x0, args=(), method=None, jac=None, hess=None, hessp=None, bounds=None, constraints=(), tol=None, callback=None, options=None)[source] funx0args=()method=Nonejac=Nonehess=Nonehessp=Nonebounds=Noneconstraints=()tol=Nonecallback=Noneoptions=None

The key arguments are:

fun: This is the objective function which we want to minimise. In our case, our portfolio is composed of a number of assets. We want to find the allocation of each asset that gives us the least amount of risk. The first parameter of the function is the 1-D array of allocations. We can also pass in a tuple of other parameters to the function too.

This is the objective function which we want to minimise. In our case, our portfolio is composed of a number of assets. We want to find the allocation of each asset that gives us the least amount of risk. The first parameter of the function is the 1-D array of allocations. We can also pass in a tuple of other parameters to the function too. x0 : This is the initial guess. We can assign a value of 1/number of assets, as the initial value of x.

: This is the initial guess. We can assign a value of 1/number of assets, as the initial value of x. args : This is a tuple that can contain the extra optional arguments. As an instance, if our target risk function takes in the covariance matrix then we can pass it in the args tuple.

: This is a tuple that can contain the extra optional arguments. As an instance, if our target risk function takes in the covariance matrix then we can pass it in the args tuple. method : This is a string argument where we can pass in the algorithm name such as SLSQP

: This is a string argument where we can pass in the algorithm name such as SLSQP constraints : This is where we will pass in the constraints such as the sum of allocations should be 1.

: This is where we will pass in the constraints such as the sum of allocations should be 1. bounds: These are essentially the minimum and maximum pairs of each element in x. As an instance, we can indicate that we don’t want the value of allocation to be negative implying that we don’t want to sell an asset.

5.2 Portfolio Optimisation Routine

To perform constrained minimisation for multivariate scalar functions, we can use the minimize function with the SLSQP algorithm.

SLSQP stands for Sequential Least SQuares Programming.

def solve(self, x0, constraints, bounds, covariance):

return minimize(self.__risk_function, x0,

args=(covariance), method='SLSQP',

#prints covergence msgs

options={'disp': True},

constraints=constraints,

bounds=bounds)

5.3 Initial Guess And Bounds

We specified an initial guess of an equal proportion portfolio and a lower bound of 0 along with an upper bound of 1 on each of the allocations.

x0 = np.ones(self.__portfolio_size) * (1.0 / self.__portfolio_size)

bounds = ((0, 1),) * (self.__portfolio_size)

5.4 Constraints

We also specified the constraints. The idea is to minimise a function of x subject to functions of constraints. The constraints can be linear and non-linear. They are defined as dictionaries.

There are two constrains:

Constraints on the allocations are added which ensure that we maximise return Constraints are added to ensure that the sum of the allocations is 1.

Now the optimiser has to produce a portfolio with minimum risk within the bounds that are subject to these constraints.

The keys of the dictionary of each constraint include:

type: This specifies the type of constraints such as ineq (inequality) or eq (equality). If you have an equality constraint such as:

A + B = C then it will be represented as A+B-C

def con():

return A + B - C

It would be equality ( type='eq' ) constraint, where you make a function that must equal zero:

def con(t):

return A + B - C

If you have an inequality constraint such as:

A + B ≥ C then it will be represented as A+B-C

def con():

return A + B - C

It would be an inequality ( type='ineq' ) constraint, where you make a function that must be greater than C:

def con(t):

return A + B - C

fun: The function of the constraint such as the sum of allocations of assets in the portfolio should be equal to 1.

constraints=[] constraints.append({'type': 'eq', 'fun': lambda inputs: 1.0 - np.sum(inputs)})

constraints.append({'type': 'eq', 'args': (returns,),

'fun': lambda allocations, returns:

my_return - self.__return_function(returns, allocations)})

5.5 Execution Of Portfolio Optimiser

optimiser = obj_factory.get_optimiser(targets, len(expected_returns.index))

portfolios_allocations_df = optimiser.generate_portfolios(expected_returns, covariance, settings.RiskFreeRate)

portfolio_risk_return_ratio_df = portfolios_allocation_mapper.map_to_risk_return_ratios(portfolios_allocations_df) #plot efficient frontiers

cp.plot_efficient_frontier(portfolio_risk_return_ratio_df)

cp.show_plots() #save data

print('7. Saving Data')

fr.save_to_file(portfolios_allocations_df, 'OptimisationPortfolios')

fr.close()

We specified a list of our target returns.

def get_my_targets():

return np.arange(0, 1.5, 0.05)

For each of the returns, the optimiser ran the optimisation routine and produced the most optimum and best portfolios which are known as the efficient frontier.

For each target, it returned the most optimum portfolio. The array of allocations are the weights (proportions) of the assets which make up the portfolio for each of our target return

The red line above is the efficient frontier. These are the optimised target portfolios.

6. Complete Code

The complete code is uploaded on GitHub

7. Next Steps:

We have a long way to go but we are getting there! The next set of steps is about:

Adding superior measures to calculate the return Enhanced risk metrics to calculate the risk of the portfolio Implementing a forward-looking correlation matrix Allocation of weights to the actual amount Company extractor to fetch us the right companies and enriching additional information such as sector, volume, ratings etc Adding additional constraints on the sector, currency, ratings, etc Appropriate allocations by units of shares

8. Notes Worth Mentioning

We need to consider transaction costs.

Additionally, we need to remember that the past does not always dictate the future accurately.

The risk and return measures are not superior and have many flaws. We need to explore other measures such as weighted average returns and expected shortfall risk measures along with forward looking covariance matrix.

We need to start considering the factor models.

9. Summary

This article documented the following sections:

The Theory Of Efficient Frontier The Process Flow Of The End To End Solution Implementation in Python Path Of The Complete Code Next Steps

If you are interested in the next phases then please let me know. Also, let me know if you have any other ideas.