Index Replication using Portfolio Optimization Methods | by Jason Yip | Towards Data Science

It’s been a whole year since I last posted and I thought to write about a project I’ve done recently during the stay-home period. It was difficult finding practical resources online. There were many math-heavy and theoretical academic papers, but I wanted to support others out there who were looking out for code and a layman’s understanding since I’ve already spent the past 4 days researching and working on this project.
I will leave most of the code out from this post but you can access everything here:
👉🏻Link to Code / Notebook
👉🏻Link to Repo
is a passive investment strategy that attempts to re-create a portfolio that generates returns similar to a broad market index. This can be achieved in these ways:
Full Replication
A Full Replication of an index requires the fund to hold the shares of all the assets in the index and replicate as close as possible each asset’s weight in the index. Trading illiquid assets in the index could add to higher transaction costs for the fund, resulting in higher expense ratios and a poorer fund performance.
Portfolio Optimization
The aim of this project is to replicate the index using an optimal portfolio selection of the largest assets in the index by reducing Tracking Error. Ultimately, we will produce a weight for each of the largest assets in the portfolio to replicate the performance of the index with only a few assets.
Tracking Error
Information Ratio
Russell 1000 is a stock market index maintained by FTSE Russell that tracks the highest-ranking 1,000 stocks in the entire U.S stock market that we aim to replicate for the following. Test Period: 2019–04–01 to 2019–04–30.
Market indices go through periodic rebalancing to ensure they keep up with the current market. Back when FTSE Russell started the index in 1984, Standard Oil of Indiana was among the largest companies in the U.S by market cap. Today, the company doesn’t exist as it had been dissolved.
According to the annual reconstitution schedules, there is a ‘rank day’ where the rankings for the top 1000 companies by market cap are computed at market close. This is followed by a transition period before the reconstitution takes place on the ‘effect day’, where markets open with the newly reconstituted index. We should be careful to account for this in approximating the index with the top companies because they (or their weights) might change across the years. I used the Top 10 components of the index published by FTSE’s reconstitution recap report in 2018 and 2017. The training data is taken from 2018–06–25 to 2019–02–28.
Market Cap of Top 10 in 2018 {'AAPL': 5016475823, 'AMZN': 483381822, 'GOOG': 694989876, 'MSFT': 7898764549, 'FB': 2917587472, 'BRK-B': 2444334034, 'JPM': 3601578394, 'XOM': 4614039868, 'JNJ': 2875386042, 'BAC': 10598321272}
We will reconstruct the Index with the Top 10 components using the Price Return Index from Russell (page 4). The Index value (page 34) is then obtained by compounding the daily return percentages of the Price Return Index. The starting value of the index is equal to the base and this was set as 100 on 12/31/78. Therefore on our base date: 2017–06–27 and we will use the ^RUI’s Close = 1341.03 (from Yahoo Finance) as our base value. The results don’t look close at all using only the Top 10.
The idea is to fit a linear combination of the current top 10 components against index prices in a constrained linear least squares model for the training period. The coeffs obtained will then be non-negative since it is a Long only portfolio.
Each coeff is taken over the sum of coeffs as weightings for each component to be taken in the test period. The sum of weightings will be the amount of leverage that the portfolio requires to replicate the index price.
The optimization called from the library is essentially:
from scipy.optimize import nnlsresult = nnls(trainX, trainY)
The coeffs are then used as weights in this linear combination, where the leverage factor is the sum of coefficients:
While I have validated the model on a holdout period of 1 month before the test period as well, I will only show the results on the test set.
The replicated index seems to visually match up well to the actual index in terms of price and daily returns. The Tracking Error is 43 basis points (0.43%) and RMSE (from the actual) seems pretty low.
The idea is to allocate more weight to a component if it is more correlated with the index’s actual close. Partial correlation correlates the residuals from 1) least-squares regression of Index on the other IVs & 2) least-squares regression of individual IV on other IVs. This accounts for confounding & 3rd party effects. I also ignored negatively correlated components by setting it to 0. Similarly, we will take each correlation over the sum to get the weightings
The optimization called from the library is essentially (where train includes both the historic component & actual index prices:
import pandas as pdtrain.pcorr()
As above, the similar linear combination approach will be used to simulate the actual.
Tracking Error and Returns RMSE are higher than the benchmark. However, the Price RMSE is a lot lower. This is probably due to the oversimplification made by this model in considering the effects of each component independently.
Pearson correlation is unable to capture shape similarity and will determine that a pair of timeseries are uncorrelated with each other even though they maybe are very similar and simply out of phase. This could be apparent between leading/lagging variables and their counterparts. The idea is to allocate more weight to a component if it has a relatively shorter DTW distance with the index and thus, a more similar shape.
The goal is to optimize this for each component (x) and the actual index (y):
The optimization called from the library is essentially:
from tslearn.metrics import dtwtrainX.apply(lambda x: dtw(x, trainY))
The Tracking error, Price RMSE & Returns RMSE performed worse than the Partial Correlation Model. This could mean that the shape similarity might not have been a huge factor in differentiating each component from each other.
While NNLS is a supervised learning approach, the unsupervised NMF method is able to learn the representation of the data via a single principal component. The principal component will be a linear combination of the assets such that the coeffs will be non-negative due to the constraints in this algo. The idea is to find the relative importance of each asset in contributing to the overall variation in the data and use that as our weightings in a linear combination again.
The optimization called from the library is essentially:
from sklearn.decomposition import NMFnmf = NMF(n_components=1)nmf.fit(trainX)coeffs = nmf.components_.tolist()[0]
The problem with this approach is that the prices cannot be properly replicated as we will not know the leverage factor in an unsupervised learning method, thus the price RMSE is very high.
PSO emulates ‘social creatures’ that are scattered around the search space and moving to find the optima. The optimization function here is to minimize Tracking Error.
Bounds = [0,1] since coeffs are non-negative and the value of the component cannot be more than the index.
Grid search is performed over c1, c2, w. The code for the grid search is almost the same as the chunk below and it will give you best_pos to be used in the chunk below to for the final optimization.
from pyswarms.utils.search.grid_search import GridSearchfrom pyswarms.utils.search.random_search import RandomSearchfrom pyswarms.single.global_best import GlobalBestPSOfrom pyswarms.utils.plotters import (plot_cost_history, plot_contour, plot_surface)def find_particle_loss(coeffs):    trainX_returns = trainX.pct_change().dropna()    trainY_returns = trainY.pct_change().dropna()    benchmark_tracking_error = np.std(trainX.dot(coeffs) - trainY)    return benchmark_tracking_errordef swarm(x):    n_particles = x.shape[0]    particle_loss = [find_particle_loss(x[i]) for i in range(n_particles)]    return particle_lossfeature_count = len(trainX.columns)min_bound = feature_count*[0]max_bound = feature_count*[1]optimizer = GlobalBestPSO(n_particles=1000,               dimensions=len(trainX.columns),               options=best_pos,               bounds=(min_bound,max_bound))# Perform optimization, cost=lowest particle_loss among all iterationscost, pos = optimizer.optimize(swarm,iters=100)
The particle (set of coefficients) with the lowest cost (Tracking Error) will be used as coefficients, taken over the sum of coefficients as weights, and used in a linear combination to reconstruct the index.
NMF (Tracking Error: 56 bps, Price RMSE: 368.7, Returns RMSE: 0.006032)
NNLS (Tracking Error: 43 bps, Price RMSE: 27.37, Returns RMSE: 0.004587)
Tracking Error for the PSO model is the lowest.
The following quantitative methods were used on prices of the Top 10 components and the actual Index price in the training period:
Non-negative Least Squares (NNLS) OptimizationRelative Partial CorrelationDynamic Time Warping (DTW) OptimizationNon-negative Matrix Factorization (NMF)Particle Swarm Optimisation (PSO)
Non-negative Least Squares (NNLS) Optimization
Relative Partial Correlation
Dynamic Time Warping (DTW) Optimization
Non-negative Matrix Factorization (NMF)
Particle Swarm Optimisation (PSO)
The weights obtained were then used to allocate a portfolio to simulate the index prices and daily returns in the test period. The performance was then measured by the Tracking Error and Price/Daily Returns RMSE from the actual index.
I have learnt a lot from this project in such a short amount of time and it has really opened my eyes up to more applications of quantitative methods in finance. Hope you’ve enjoyed reading this as much as I’ve enjoyed creating it.