Stock Market Analytics with PCA. From Principal Component Analysis to... | by Yao Lei Xu | Towards Data Science

Principal Component Analysis (PCA) is a powerful data analytics tool used in many areas of machine learning. However, despite its versatility and effectiveness, its application in finance is not as widely discussed.
Today, I will talk about how PCA can be used in the stock market, how it relates to the Capital Asset Pricing Model (CAPM), and how we can use PCA to analyse the impact of COVID19.
(You can find the full code and additional resources here)
The first principal component explains most of the variance in the data.
In a nutshell, Principal Component Analysis (PCA) decomposes the data into many vectors called principal components that essentially “summarise” the given data. More specifically, these summaries are linear combinations of the input features that try to explain as much variance in the data as possible. By convention, these principal components are ordered by the amount of variance they can explain, with the first principal component explaining most of the data.
The returns of a stock can be decomposed into: (1) the returns of the risk-free asset, (2) the returns of the market factor, and (3) the idiosyncratic returns of the stock. Overall, the market factor is the primary driver of all stock returns.
The Capital Asset Pricing Model (CAPM) is a famous framework for pricing the returns of an asset such as a stock, with many interesting connections to the modern portfolio theory, which I will discuss in a future post.
Before diving into the details of the CAPM, it is important to understand the notion of risk-free assets and the market factor. A risk-free asset is essentially an asset than can give you returns at virtually no risk (e.g. a government bond). The market factor instead monitors the state of the overall stock market as a whole and is often measured through an index such as the S&P500. Generally speaking, the overall market is more volatile/risky than government bonds, but it also provides more returns to the investors.
With those definitions in mind, let’s look at the concept of the Security Market Line (SML) from CAPM. In practice, SML decomposes the returns of a stock r_i into three main factors:
r_f: risk-free returnbeta_i * (r_m-r_f): market factor returne_i: idiosyncratic return
r_f: risk-free return
beta_i * (r_m-r_f): market factor return
e_i: idiosyncratic return
The intuition behind this equation is that:
(1) the return of a stock should be at least equal to the return of the risk-free asset (otherwise why take the extra risk in the first place?)
(2) the return of the asset is also explained by the market factor, which is captured by the term (r_m-r_f) (measures the excess return of the market with respect to the risk-free asset) and beta_i (measures the degree to which the asset is affected by the market factor).
(3) the return of a stock is also affected by idiosyncratic factors, which are stock specific factors (e.g. the earnings release of a stock affects that individual stock only, but not the overall market).
Empirically speaking, the market factor is the primary driver of the stock market returns, as it tends to explain most of the returns of any given stock in any given day.
When applying PCA to daily stock returns, the first principal component approximates the market factor.
Let’s consider the 500 stocks in the S&P500 index, and compute their daily returns, as shown in the figures below.
rs = prices.apply(np.log).diff(1) rs.plot(title='Daily Returns of the Stocks in the S&P500')
crs = rs.cumsum().apply(np.exp)crs.plot(title='Cumulative Returns of the Stocks in the S&P500')
The figures above show the daily returns and the cumulative returns of the 500 stocks in the S&P500 since the beginning of 2020. The amount of raw data can look quite overwhelming, so let’s process them via PCA by computing the 1st principal component of the daily returns. The figure below shows the values of the 1st principal component, which is essentially a vector of dimension 500 that contains a value for each of the 500 stocks.
from sklearn.decomposition import PCApca = PCA(1).fit(rs.fillna(0))pc1 = pd.Series(index=rs.columns, data=pca.components_[0])pc1.plot(xticks=[], title='First Principal Component of the S&P500')
Recall that (1) the first principal component represents the linear combination of the input data that explains most of the variance, and (2) the primary driver of stock returns is the overall market factor. This implies that if we formulate a portfolio of stocks by allocating the cash proportionally to the 1st principal component (i.e. linear combination of the input data), we can replicate the returns of the S&P500 approximately (i.e. the primary driver of stock returns).
weights = abs(pc1)/sum(abs(pc1)) # l1norm = 1myrs = (weights*rs).sum(1)rs_df = pd.concat([myrs, market_rs], 1)rs_df.columns = ["PCA Portfolio", "S&P500"]crs_df = rs_df.cumsum().apply(np.exp)crs_df.plot(subplots=True);
As shown in the figure above, our PCA portfolio can work as a proxy for the market factor, which is the primary driver of stock returns (hence explaining most of the variance!). Note that although they are similar, the PCA portfolio doesn’t replicate the S&P500 exactly, since the S&P500 is a market-capitalisation weighted average of the 500 stocks, while the weights in the PCA portfolio is influenced by the explained variance.
Using PCA, we can cluster together businesses that were most/least affected by the COVID19 pandemic, without any prior knowledge of their fundamentals.
As you probably know, 2020 has been a wild ride for the stock market due to the COVID19 pandemic. Using PCA, we can analyse how this pandemic affected the individual stocks.
For instance, let’s look a the 1st principal component, and select the stocks that have the most and the least negative PCA weights, as shown below.
fig, ax = plt.subplots(2,1)pc1.nsmallest(10).plot.bar(ax=ax[0], color='green', grid=True, title='Stocks with Most Negative PCA Weights')pc1.nlargest(10).plot.bar(ax=ax[1], color='blue', grid=True, title='Stocks with Least Negative PCA Weights')
Notice from the figure above how the most negative stocks are in the tourism and the energy sector. This makes sense since COVID19 heavily impacted the travelling business, as well as the energy companies that provide fuel for those businesses. On the other hand, the least impacted companies fall into the consumer goods sectors, which also makes sense since this sector benefited from the boost in sales of consumer goods due to the quarantine measures.
Therefore, by applying PCA, we were able to cluster together the best and worst businesses that were affected by the COVID19 pandemic, without any prior knowledge of their fundamentals!
In addition, we can formulate a winning portfolio that is long the top 10 companies according to the PCA weights. As shown in the figure below, the resulting portfolio would have performed significantly better than the market, since it invested in companies that actually benefited from the pandemic.
Note that this portfolio is formed with look-ahead bias, where portfolio weights are computed using future data that were not available at the time of market downturn. The PCA used in this way is therefore a backward looking analytics tool. For more information about look-ahead bias and how to avoid them, check this article out.
myrs = rs[pc1.nlargest(10).index].mean(1)mycrs = myrs.cumsum().apply(np.exp)market_crs = market_rs.cumsum().apply(np.exp)mycrs.plot(title='PCA Portfolio vs. S&P500')market_crs.plot()plt.legend(['PCA Selection', 'S&P500'])
I hope you enjoyed the article! Follow me if you would like to see more content like this.
Also, check out my website for the full code and additional resources.