Understanding Portfolio Optimization | by Tony Yiu | Towards Data Science

Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.
Quantitative finance (at least on the portfolio management side) is all about finding optimal portfolios. For a given level of risk, we want to make sure that we are getting as much return as possible.
In quantitative finance, risk is viewed like a resource. Exposing your portfolio to risk generates returns over time. In other words, expected return is the compensation that we get paid in return for taking on uncertainty (uncertainty around inflation, the economy, etc.).
A few things to keep in mind:
The risk and uncertainty around an investment’s return is traditionally proxied by the standard deviation (a.k.a. volatility) of its historical returns.
There is an approximate and positive relationship between risk and return. The more volatile an asset is, the higher its historical returns have usually been.
This is not always the case though. There are some highly volatile assets whose realized returns have been lackluster relative to those of other similarly volatile assets. One potential reason for the lower return is that these assets are valuable for other reasons, such as low correlation to the stock market. Investments that zig when stock markets zag (and offer a positive expected return) are highly demanded for their ability to hedge.
I covered investment risk in depth in the following post, so if you want to dive deep, please give it a read:
towardsdatascience.com
If you had just one day to tour Tokyo (having never visited there before) and you needed to create an itinerary for that day, how would you do it? Chances are you would start with a list of possible activities. Then you would pick the four or five activities that are likely to be the most fun.
But being unfamiliar with Tokyo, you are worried that you might try an activity and not like it — that would be a waste of your precious time. But you also want to try out some new things. Doing stuff that you already do at home (like reading at a cafe) is kind of fun, but doesn’t have the potential to be super fun. Doing something completely novel could be super fun, but it could also turn out to be terrible.
So for each activity, you not only need to consider how fun it potentially could be (expected return), but also the risk of it turning out to be not much fun after all (uncertainty).
You would also pick activities that are unrelated to each other. Because if you end up doing the same type of activity over and over again, and you find that it’s actually not fun at all, then your day would be ruined. So having a variety of activity types makes it more likely that you will end up having fun overall.
So we can summarize your itinerary creation process as follows — you want to create a basket of activities that maximizes the amount of fun that you are likely to have subject to your personal risk tolerance (your personal desire for new experiences versus familiar ones).
And the the way you assess and trade off one activity versus the others is as follows:
Weigh each activity on its own in terms of its potential for fun-ness versus how uncertain you are about your estimate of its fun-ness.
Weigh each activity against the others to make sure that you have a variety of activity types (diversification). For example, you wouldn’t want every activity to involve food (in case you find that you don’t like Japanese food). Plus, the more you eat and the more full you become, the less you’ll enjoy the next meal. In other words, you want the correlation across activities to be low — so that doing one activity does not impact your enjoyment of the next one.
Once you’re done, assuming you put some effort in, you’ll likely believe that your final list maximizes your probability of having fun that day.
And that in a nutshell is portfolio optimization —
For a given level of risk, we want to maximize our portfolio’s probability of earning a positive return.
And we do so by sifting through financial assets much like we did for activities:
By weighing their expected returns against their volatilities (remember volatility is how we approximate risk). The higher the expected return relative to the volatility, the more sure we are that the return of the investment will be positive over time.
By considering the correlations between the assets’ returns. All things equal, we want assets that are as uncorrelated as possible. This maximizes the benefits of diversification. The next section details why we care about diversification.
Diversification is sometimes called the only free lunch in economics and finance. To see why, let’s use an example.
If we own $100 in a stock that is expected to return 10% over the next year, then our expected return is $10. If we add another $100 of a different stock that also has an expected return of 10%, then our $200 portfolio’s return has now doubled to $20.
The portfolio risk (standard deviation) of a two asset portfolio can be calculated as:
stdev = sqrt(w1^2*stdev1^2 + w2^2*stdev2^2 + 2*w1*w2*p*stdev1*stdev2)where w1 is weight of stock 1w2 is weight of stock 2stdev1 is standard deviation of stock 1stdev2 is standard deviation of stock 2p is correlation between stock 1 and stock 2
Let’s say that the two stocks in our portfolio both have a standard deviation of 10%.
So when we had only $100 invested in a single stock, our portfolio’s risk was $100*10% = $10. This means that we expect our return to be $10 but with a cone of uncertainty around it.
The cone of uncertainty’s width is described by its sigma (a.k.a. standard deviation). In this case, sigma equals $10. The bigger the sigma, the less certain we can be of our portfolio’s return in any specific year (and the wider our portfolio swings in value would be).
If you are unfamiliar with the normal distribution and its parameters, check out the post below:
towardsdatascience.com
And let’s also assume that the two stocks move in exact lockstep (meaning that they have a correlation of 1.0). We can calculate our portfolio’s risk using the formula above:
stdev = sqrt(w1^2*stdev1^2 + w2^2*stdev2^2 + 2*w1*w2*p*stdev1*stdev2)Portfolio Risk= sqrt(0.5^2*0.1^2 + 0.5^2*0.1^2 + 2*0.5*0.5*1*0.1*0.1)= 0.1
So the risk of our two stock portfolio is still 10%. Putting it in dollar terms, our sigma is now $200*10% = $20. So we went from a $100 portfolio with an expected return of $10 and sigma of $10 to a $200 portfolio with an expected return of $20 and a sigma of $20. (I use sigma and risk interchangeably)
So basically nothing really changed. If we had just doubled our holdings in stock 1 (instead of buying stock 2), we would have achieved the same exact result. That’s because of our assumption that stock 1 and stock 2 are perfectly correlated (p=1). So owning either one or both all amount to the same thing.
Now let’s see what happens if stocks 1 and 2 are uncorrelated.
The expected return of our $200 portfolio remains $20. But now our risk is:
stdev = sqrt(w1^2*stdev1^2 + w2^2*stdev2^2 + 2*w1*w2*p*stdev1*stdev2)Since p=0, the third term drops out!Portfolio Risk= sqrt(0.5^2*0.1^2 + 0.5^2*0.1^2 + 2*0.5*0.5*0*0.1*0.1)= 0.0707
Our portfolio risk has dropped to 7.1%. This means in dollar terms our $200 portfolio has a sigma (a.k.a. risk) of $14.14, significantly lower than our previous sigma of $20.
So compared to both the perfectly correlated case and the hold $200 in stock 1 cases, we were able to drop our risk by almost a third while keeping the same expected return! A free lunch indeed.
This is a stylized example obviously. In reality, it’s quite hard to find a bunch of completely uncorrelated assets that all have high expected returns. For a more in depth discussion of correlation and diversification, you can read my previous blog on it below:
towardsdatascience.com
So the optimal portfolio is the one that combines our candidate assets in such a way that, for a given level of risk, the probability of the portfolio earning a positive return is maximized.
Why for a given level of risk? If the only thing that we wanted to do was to maximize our probability of a positive return, we could just invest in super low risk assets like money market funds. We are virtually guaranteed to make money technically, but it would be just a few pennies. Remember you need to take on some risk to get returns.
Rather, what we want to make sure of (as much as we can) is that for a specified level of risk, we are investing in a portfolio that maximizes our chances of earning a positive return. The portfolio that does that, a.k.a. the optimal portfolio, is the one with the highest expected return (or in statistical terms, the one with the highest Z-score).
So the optimization problem we need to solve is:
For a given level of risk, solve for the weights, W, that:Maximize W.T @ ESubject to:  W.T @ Cov @ W = (target risk)^2   and sum(W) = 1Where W is a vector representing the weights of the asset in our portfolio.E is a vector representing the expected returns of the asset.Cov is the covariance matrix of the asset's returns.@ denotes matrix multiplication..T denotes the transpose operation.
In plain English, we want to maximize the expected return (W.T @ E) subject to the following conditions:
Portfolio variance (W.T @ Cov @ W) is equal to target variance. Remember that variance is the square of risk (since risk is equal to standard deviation). So this is the constraint that gets us to our specified risk target.Weights sum to 1. In this case, we allow for shorts (negative weights). If we don’t want negative weights, we would need to add an additional constraint. If we didn’t have this condition, the optimization would blow up or spit out gibberish.
Portfolio variance (W.T @ Cov @ W) is equal to target variance. Remember that variance is the square of risk (since risk is equal to standard deviation). So this is the constraint that gets us to our specified risk target.
Weights sum to 1. In this case, we allow for shorts (negative weights). If we don’t want negative weights, we would need to add an additional constraint. If we didn’t have this condition, the optimization would blow up or spit out gibberish.
We can solve this one of two ways — with an optimizer or analytically. The cool thing about portfolio optimization with shorts allowed is that it has an analytical solution that we can implement with just some matrix algebra.
And that’s it for this article. I’ve already written up some code to run the optimization analytically. But I wanted to keep this article at the conceptual level. Code and examples coming soon in a future post! Until then cheers!
The next part: