Portfolio Optimization With SciPy | by Tony Yiu | Towards Data Science

Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.
Not intended to be investment advice. You can find my code on my GitHub here.
Time for another article on portfolio optimization since I’ve been doing a lot of work around it lately. In case you need a refresher, here are my previous writings on the topic:
towardsdatascience.com
towardsdatascience.com
As fun as it is to optimize a portfolio with just matrix algebra and NumPy, sometimes we need to add constraints. For example, many investors don’t want to or are not allowed to short investments. We can’t guarantee that the optimal portfolio produced using matrix algebra won’t include short positions (negative weights). So instead, we turn to optimization.
In case you didn’t read my previous articles on optimization, optimization refers to the process of solving for the combination of assets that maximizes return for a given amount of risk (or minimizes risk for a specified level of return). The desired output from an optimization is a set of portfolio weights (for each asset) that would produce the optimal portfolio.
The two key inputs to a portfolio optimization are:
Expected returns for each asset being considered.The covariance matrix of asset returns. Embedded in this are information on cross-asset correlations and each asset’s volatility (the diagonals).
Expected returns for each asset being considered.
The covariance matrix of asset returns. Embedded in this are information on cross-asset correlations and each asset’s volatility (the diagonals).
Expected returns are hard to estimate — some people like to use historical averages (dangerous as the past is often not representative of the future), others have their own methodology for estimating return forecasts. I am planning a whole post on this so I won’t go into the details this time. Today we will focus on the optimization process itself. Thus, we will pretend like we got expected return estimates from a consultant. By the way, these were very roughly estimated by me, so they should definitely not be taken as investment advice.
So as you can see from the plot above, we have a portfolio of eight investments ranging from stocks and bonds to commodities. For those unfamiliar with the abbreviation, TIPS are Treasury Inflation Protected Securities — in other words, Treasury bonds with no inflation risk.
Here are the correlations estimated using the past 10 years of returns. The lower the better, so the bluer shaded cells represent potential opportunities for diversification. Technically, the optimization needs the covariance matrix, but the correlation matrix is much more informative to look at.
Now that we have our inputs, let’s get to the code.
I used the same data set as I did in my previous optimization post, except that there are now eight assets instead of four. The labels of each asset are:
factors = ['S&P 500','Emerging Markets','Small Cap',           'Treasury Bonds',           'High Yield','TIPS','Gold','Oil']
I store the daily returns in a Pandas dataframe called factor_returns. Calculating the covariance matrix of our assets is as simple as:
cov =  factor_returns.cov()
As I mentioned above, we will use a return forecast that I created (the returns are specified in the same order as they are listed in factors):
expected_returns = np.array([[ 0.080],                             [ 0.092],                             [ 0.092],                             [-0.017],                             [ 0.034],                             [ 0.001],                             [ 0.010],                             [ 0.079]])
Now let’s import the Python libraries that we need for optimization:
from scipy.optimize import minimize, Bounds, LinearConstraint
I’m going to explain things slightly out of order of how they are actually coded because it’s easier to understand this way. The next block of code shows a function called optimize that runs an optimization using SciPy’s minimize function.
Look at where minimize is called (I bolded it). The first argument func is the function that we want to minimize. The second argument W is the input that the optimizer is allowed to vary — W is what we are solving for and corresponds to the weights of the assets in our portfolio. We need to start it off with a guess — without any strong priors, I just equal weight each asset.
The exp_ret (expected return) and cov (covariance) variables in the args argument are provided inputs that the optimizer is not allowed to vary.
So what the optimizer does is it searches for the vector of portfolio weights (W) that minimize func given our supplied expected returns and covariance matrix.
At the end of the function, you will see that I return optimal_weights[‘x’]. That’s because in the optimizer object we get back from mimimize, ‘x’ pertains to the optimized weights, W.
W = np.ones((factor_moments.shape[0],1))*(1.0/factor_moments.shape[0])# Function that runs optimizerdef optimize(func, W, exp_ret, cov, target_return):    opt_bounds = Bounds(0, 1)    opt_constraints = ({'type': 'eq',                        'fun': lambda W: 1.0 - np.sum(W)},                       {'type': 'eq',                        'fun': lambda W: target_return - W.T@exp_ret})    optimal_weights = minimize(func, W,                                args=(exp_ret, cov),                               method='SLSQP',                               bounds=opt_bounds,                               constraints=opt_constraints)    return optimal_weights['x']
You might notice that minimize also can take two optional arguments — bounds and constraints. These are important so let’s go over them one by one. Bounds is pretty straightforward. We want each of our weights in W to be between 0 and 1, in other words no negative weights or leverage:
opt_bounds = Bounds(0, 1)
For constraints, we have two — we want our weights to sum to 1 (in order for it to be a proper portfolio) and we want to achieve a pre-specified target return (you can set this as a target risk as well or leave it out to altogether). The reason we have a return target is to avoid the case where we optimize and obtain a sweet and diversified portfolio, but the expected return of that portfolio is 3% or some other value that’s too low.
opt_constraints = ({'type': 'eq',                        'fun': lambda W: 1.0 - np.sum(W)},                       {'type': 'eq',                        'fun': lambda W: target_return - W.T@exp_ret})
Now let’s take a look at the part of the code I skipped over earlier. First, what is func? Well, when we solve for the optimal portfolio, we are trying to find the portfolio that gives us the most return per unit of risk (where risk is the portfolio’s standard deviation).
So we want to maximize the ratio of return, which we can calculate as W.T@exp_ret (this multiplies each asset’s return by its weight and sums them), and risk, which we can calculate as W.T@cov@W. The @ character denotes matrix multiplication.
Since we are using an optimizer function that minimizes things, we need to slap a negative sign on our return to risk ratio — that way when we minimize it, we are actually maximizing it.
# Function to optimizedef ret_risk(W, exp_ret, cov):    return -((W.T@exp_ret) / (W.T@cov@W)**0.5)
Now all that’s left to do is to optimize our portfolio:
x = optimize(ret_risk, W, expected_returns, cov,              target_return=0.055)
Here’s what the optimal weights look like:
A few things that jump out in terms of weights:
Small cap and Emerging Markets have the highest expected returns but are not highly weighted. That’s because their volatility, a.k.a. risk, is significantly higher than that of the S&P 500 (see bar chart below). Also they are highly correlated to the S&P 500 and thus offer very little diversification.
Treasury bonds have a negative expected return and yet are the second highest weighted asset class. That’s because it’s also the least correlated asset class to stocks (negative correlation of approximately -0.40). So in effect, the portfolio relies on stocks and high yield bonds for return and Treasury bonds for insurance when stock markets crash.
Oil despite its relatively high return and low correlation to the other assets is just too volatile, hence its low weighting.
And that’s it for optimization with SciPy. The next step in the portfolio strategy process would be to do our best to measure the impact of model and estimation error. Namely, we want to know which inputs are likely to be mis-specified (measurement error) and how sensitive our model outputs are to such errors. Hint: expected returns are very noisy and the outputs of the optimization are highly sensitive to even small changes in expected returns. Next time, we will see how we can get around this issue. Cheers!
If you liked this article and my writing in general, please consider supporting my writing by signing up for Medium via my referral link here. Thanks!