DBSCAN Clustering for Trading
By: Roshan Adusumilli
Tags: Finance, Unsupervised Learning, Stocks, Ai In Finance, Artificial Intelligence

Artificial Intelligence in Finance

DBSCAN Clustering for Trading

Image credit: Sharpshot

Pioneered in the 80’s by quantitative analysts at Morgan Stanley, pairs trading is a trading strategy that allows traders to profit in almost any market conditions. This strategy involves monitoring two historically correlated securities. Once a correlation discrepancy such as one stock going up and the other remaining stagnant is identified, the investor goes long on the underperforming security and short on the outperforming security, hoping that the securities return to their historical correlation. In the case that this does happen, the investor makes a profit from the convergence of the prices.

For example, General Motors and Ford create a similar product (cars), so they share similar stock price movements based on the overall car market. If the price of General Motors stock increases significantly while the price of Ford stock remains the same, someone implementing a pairs trading strategy would short General Motors stock and buy Ford stock, assuming the prices will return to their historical balance point. If General Motors’ stock falls, the investor profits and if Ford’s stock rises, the investor also profits. However, past prices aren’t always indicative of future trends, so this strategy still carries risk. To help us find historically correlated securities, we can use DBSCAN clustering.

DBSCAN Clustering : A Brief Explanation

Images via Google Image Search, labelled for reuse

Machine learning can be utilized in many trading strategies and pairs trading is no different. Density-based spatial clustering of applications with noise (DBSCAN) can be used to cluster stocks and exclude the stocks that don’t fit into a cluster. The two parameters for DBSCAN are eps (how close points should be to each other to be considered part of a cluster) and minPoints (the minimum number of points to form a dense region). From these parameters, the DBSCAN algorithm then creates clusters from the set of points we feed it. Points that are in low density regions are classified as outliers. K-means is another popular clustering algorithm, but it is much more sensitive to outliers, which is why we use DBSCAN instead. Now that we understand pairs trading and DBSCAN, let’s jump into the implementation.

Imports/Data

We start off by making the necessary imports. It’s important to note that the imports from Quantopian are intended to be used in the Quantopian IDE.

import matplotlib.pyplot as plt

import matplotlib.cm as cm import numpy as np

import pandas as pd from sklearn.cluster import DBSCAN

from sklearn.decomposition import PCA

from sklearn.manifold import TSNE

from sklearn import preprocessing from statsmodels.tsa.stattools import coint from scipy import stats from quantopian.pipeline.data import morningstar

from quantopian.pipeline.filters.morningstar import Q500US, Q1500US, Q3000US

from quantopian.pipeline import Pipeline

from quantopian.research import run_pipeline

The pre-defined universe we’ll be working with is the Q1500 which is a large set of liquid stocks.

study_date = "2019-12-31"

universe = Q1500US()

To create our pipeline with we use information from Morningstar.

pipe = Pipeline(columns = {

'Market Cap': morningstar.valuation.market_cap.latest.quantiles(5),

'Industry':morningstar.asset_classification.morningstar_industry_group_code.latest,

'Financial Health':morningstar.asset_classification.financial_health_grade.latest},

screen = universe) res = run_pipeline(pipe, study_date, study_date)

res.index = res.index.droplevel(0) # drop the single date from the multi-index

Let’s take a look at our data. We see that it contains 1486 stocks and a description of financial health, industry, and market cap.

print res.shape

res.head()

We can’t work with stocks without a financial health grade, so we remove them. Two stocks are lacking a financial health grade so they’re removed.

res = res[res['Financial Health']!= None]

print res.shape

Now we’ll create a dictionary to replace the categorical data for financial health with numerical scores.

res['Financial Health'] = res['Financial Health'].astype('object')

health_dict = {u'A': 0.1,

u'B': 0.3,

u'C': 0.7,

u'D': 0.9,

u'F': 1.0}

res = res.replace({'Financial Health': health_dict})

For a better understanding of our data, we use .describe() to view a few basic metrics such as mean and standard deviation.

res.describe()

Daily Returns

Get the daily returns (the dollar change in a stock’s price as a percentage of the previous day’s closing price) to use in our strategy. Then we look at the daily returns for Apple as an example.

pricing = get_pricing(

symbols=res.index,

fields='close_price',

start_date=pd.Timestamp(study_date) - pd.DateOffset(months=24),

end_date=pd.Timestamp(study_date)

) returns = pricing.pct_change()

returns[symbols(['AAPL'])].plot()

Similar to what we did with stocks lacking a financial health grade, we’ll have to remove stocks that don’t have a full return series.

returns = returns.iloc[1:,:].dropna(axis=1)

print returns.shape

PCA and DBSCAN

Our next steps are to use principal component analysis to reduce the dimensionality of the returns data and extract the historical latent common factor loadings for each stock, add the market cap/financial health values, and use DBSCAN.

prin_components = 50

pca = PCA(n_components=prin_components)

pca.fit(returns)

X = np.hstack(

(pca.components_.T,

res['Market Cap'][returns.columns].values[:, np.newaxis],

res['Financial Health'][returns.columns].values[:, np.newaxis])

) X = preprocessing.StandardScaler().fit_transform(X) clf = DBSCAN(eps=1.9, min_samples=3)

print clf clf.fit(X)

labels = clf.labels_

n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)

print "

Clusters discovered: %d" % n_clusters_ clustered = clf.labels_

The total number of possible pairs are over 900,000 so we’ll significantly reduce that number to only pairs that are among the better candidates for our pairs trading strategy.

ticker_count = len(returns.columns)

print "Total pairs possible in universe: %d " % (ticker_count*(ticker_count-1)/2)

clustered_series = pd.Series(index=returns.columns, data=clustered.flatten())

clustered_series_all = pd.Series(index=returns.columns, data=clustered.flatten())

clustered_series = clustered_series[clustered_series != -1] CLUSTER_SIZE_LIMIT = 9999

counts = clustered_series.value_counts()

ticker_count_reduced = counts[(counts>1) & (counts<=CLUSTER_SIZE_LIMIT)]

print "Clusters formed: %d" % len(ticker_count_reduced)

print "Pairs to evaluate: %d" % (ticker_count_reduced*(ticker_count_reduced-1)).sum()

Visualization

Since the data is currently is clustered in 52 dimensions, we can use the T-SNE algorithm to visualize the data in 2D and see the 15 clusters.