Portfolio Optimization With NumPy | by Tony Yiu | Towards Data Science

Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.
If you would like to see my code, you can find it on my GitHub here.
Last time we talked about how and why portfolio optimizations works:
towardsdatascience.com
Today, we will dive into the technicalities of actually optimizing a portfolio. The cool think about portfolio optimization is that it can be done purely with matrix algebra, no optimization software needed.
Before we start, let’s refresh ourselves on our objective. Quoting my previous post:
What we want to make sure of (as much as we can) is that for a specified level of risk, we are investing in a portfolio that maximizes our chances of earning a positive return. The portfolio that does that, a.k.a. the optimal portfolio, is the one with the highest expected return (or in statistical terms, the one with the highest Z-score).
For a given level of risk, solve for the weights, W, that:Maximize W.T @ ESubject to:  W.T @ Cov @ W = (target risk)^2   and sum(W) = 1Where W is a vector representing the weights of the asset in our portfolio.E is a vector representing the expected returns of the asset.Cov is the covariance matrix of the asset's returns.@ denotes matrix multiplication..T denotes the transpose operation.
W @ E is the expected return of the portfolio.The portfolio’s variance is calculated as W.T @ Cov @ W). Variance is the square of the portfolio’s standard deviation (a.k.a. its risk). In our objective function, we want our portfolio’s variance to be equal to the target variance (the square of our target risk level).
W @ E is the expected return of the portfolio.
The portfolio’s variance is calculated as W.T @ Cov @ W). Variance is the square of the portfolio’s standard deviation (a.k.a. its risk). In our objective function, we want our portfolio’s variance to be equal to the target variance (the square of our target risk level).
Minimization problems are often easier to solve than maximization problems so let’s flip our problem around:
For a given level of risk, solve for the weights, W, that:Minimize W.T @ Cov @ WSubject to:  W.T @ E = target return = mu  and sum(W) = 1Where W is a vector representing the weights of the asset in our portfolio.E is a vector representing the expected returns of the asset.Cov is the covariance matrix of the asset's returns.@ denotes matrix multiplication..T denotes the transpose operation.
Now instead of maxing return for a given level of variance (a.k.a. the square of risk), we are minimizing variance for a given level of return.
To solve this analytically, we can take advantage of the Lagrange Multiplier and rewrite our problem as follows:
L(W,h1,h2) = W.T@Cov@W + h1*(W.T@E - mu) + h1*(W.T@ones - 1)ones denotes a vector of ones with the same length as W
Notice that we have now a single equation that includes the thing we want to minimize (W.T@Cov@W) as well as our two constraints — that the expected return of the portfolio must equal the target return (W@E — mu) and that the portfolio weights must sum to 1 (W@ones — 1).
We can solve this by solving:
gradient{L(W,h1,h2)} = 0
This is basically gradient descent, where we want to find the point where the partial derivatives (a.k.a. slope) with respect to each of the variables is zero. When we reach that point, we know we are the minimum.
We can solve the previous equation with matrix algebra. First let’s write out each partial derivative:
gradient L with respect to W  = 2*Cov@W + h1*mu + h2*ones = 0gradient L with respect to h1 = W.T@E - mu = 0gradient L with respect to h2 = W.T@ones - 1 = 0
We can rewrite our system of equations as:
2*Cov@W + h1*mu + h2*ones = 0W.T@E                     = muW.T@ones                  = 1
The nice thing about matrices is that they allow us to represent systems of equations like this one easily (and solve them easily). In Python matrix notation, the previous three equations would be:
[[2*Cov  h1 h2],     [[W ],       [[0 ], [E.T.   0  0 ],  @   [h1],   =    [mu], [ones.T 0. 0 ]]      [h2]]        [1 ]]       A          @    X      =     b
So we can capture our entire problem with three matrices A, X, and b and using the matrix equation A@X=b.
Now all we need to do is solve for X, which we can do easily by multiplying b by the inverse of A:
X = inverse(A)@bThe first n elements of X are the optimal weights where n is the number of different assets whose weights we are optimizing for.
First we start with some returns. I downloaded mine from Shardar via the QUANDL API. My returns data which I stored in a dataframe called final looks like this:
             S&P 500  Treasury Bonds      TIPS      Golddate                                                    2020-10-07  0.017407       -0.007293 -0.000080 -0.0004512020-10-08  0.008863        0.005400  0.003977  0.0035552020-10-09  0.008930       -0.000187  0.000317  0.0181612020-10-12  0.016088        0.003186  0.000634 -0.0028722020-10-13 -0.006526        0.007161  0.000000 -0.01572
There are four assets, and I have several years of daily returns for each. We can calculate the inputs we need for our optimization with the following code:
# final is a dataframe of daily returns for the assets# I use the historical mean return for my expected returnE = np.array(final.mean(axis=0)).reshape(-1,1)# Calculate the covariance matrix of the asset's returnscov_matrix = np.array(final.cov())# Ones vectorones = np.ones((E.shape[0],1))zeros = np.zeros((2,2))
Next we create the matrixes from our equation (A@X=b):
# Put together the A matrixA = 2*cov_matrixA = np.append(A, E.T, axis=0)A = np.append(A, ones.T, axis=0)temp = np.append(E, ones, axis=1)temp = np.append(temp, zeros, axis=0)A = np.append(A, temp, axis=1)# Put together the b vectorb = np.array([[0],              [0],              [0],              [0],              E[0],  # I set the target return to be              [1]])  # the expected return of stocks# So in essense, I am looking for an optimal portfolio# that is expected to give the same return as I get from# investing in stocks (but with lower risk)
Here’s what each of them look like:
A =   0.000237 -0.000096 -0.000015  0.000004  0.000555  1.0 -0.000096  0.000170  0.000046  0.000038  0.000371  1.0 -0.000015  0.000046  0.000024  0.000022  0.000154  1.0  0.000004  0.000038  0.000022  0.000200  0.000228  1.0  0.000555  0.000371  0.000154  0.000228  0.000000  0.0  1.000000  1.000000  1.000000  1.000000  0.000000  0.0b =   0.000000  0.000000  0.000000  0.000000  0.000555  1.000000# The 0.000555 number in b is the historical daily mean return of # the S&P 500
Finally, we can calculate the optimal weights by inverting matrix A and multiplying it against matrix b:
# Optimize using matrix algebrafrom numpy.linalg import invresults = inv(A)@b# Grab first 4 elements of results because those are the weights# Recall that we optimize across 4 assets so there are 4 weightsopt_W = results[:final.shape[1]]
Let’s take a look at our optimal weights. The optimal portfolio consists primarily of stocks and bonds with a short to TIPS and a tiny allocation to gold. The analytical solution can only produce unconstrained weights (meaning shorts are allowed). If we want only positive weights, we will have to use gradient descent.
These weights were calculated using very naive assumptions and purely historical data. These should definitely not be taken as investment recommendations!
                Optimal WeightsS&P 500                0.602329Treasury Bonds         0.726293TIPS                  -0.357301Gold                   0.028680
Finally, let’s see if our supposedly optimal portfolio is actually optimal. We should expect the return of our optimal portfolio to be similar to that of stocks (S&P 500) but with less volatility. We can see that our optimal portfolio’s return (the purple line) was both higher and less volatile than that of stocks (the blue line).
Bear in mind that this simple analysis is highly biased. The purpose is to show you how portfolio optimization with matrix algebra works. I optimized the portfolio and backtested it over the same timeframe (there was no train-test split), so of course the portfolio will look really good.
Once you finish calculating the optimal weights, there’s still more work to do. The portfolio optimization process is notoriously sensitive to small changes in the inputs (especially expected returns).
So shocking the optimization inputs (via resampling or even adding some artificial noise) to see how the weights change is a good idea.
Finally, it’s good to think about what the optimal portfolio really represents. It’s a portfolio designed to be held for a long time (years). That’s because the data (time series of returns) we use to estimate our inputs need to cover as much time as possible, and ideally multiple business cycles.
It’s not a process designed to help you time the market in the short-term. Expected returns and covariance matrices estimated over short timeframes would introduce substantial risk of estimation error — remember, garbage in garbage out. Cheers!