Data Transformation
By: Ke Gui
Tags: Machine Learning, Trading, Artificial Intelligence, Data Science, Pandas

In the previews article, I briefly introduced the Volume Spread Analysis(VSA). After we did feature-engineering and feature-selection, there were two things I noticed immediately, the first one was that there were outliers in the dataset and the second issue was the distribution were no way close to normal. By using the method described here, here and here, I removed most of the outliers. Now is the time to face the bigger problem, the normality.

There are many ways to transfer the data. One of the well-known examples is the one-hot encoding, even better one is word embedding in natural language processing (NLP). Considering one of the advantages of using deep learning is that it completely automates what used to be the most crucial step in a machine-learning workflow: feature engineering. Before we get into the deep learning in the later articles, let’s have a look at some simple ways to transfer data to see if we can make it closer to normal distribution.

In this article, I would like to try a few things. The first one is to transfer all the features to a simple percentage change. The second one is to do a Percentile Ranking. In the end, I will show you what happens if I only pick the sign of all the data. Methods like Z-score, which are standard pre-processing in deep learning, I would rather leave it for now.

1. Data preparation

For consistency, in all the 📈Python for finance series, I will try to reuse the same data as much as I can. More details about data preparation can be found here, here and here or you can refer back to my previous article. Or if you like, you can ignore all the code below and use whatever clean data you have at hand, it won’t affect the things we are going to do together.

#import all the libraries

import pandas as pd

import numpy as np

import seaborn as sns

import yfinance as yf #the stock data from Yahoo Finance

import matplotlib.pyplot as plt #set the parameters for plotting

plt.style.use('seaborn')

plt.rcParams['figure.dpi'] = 300 #define a function to get data

def get_data(symbols, begin_date=None,end_date=None):

df = yf.download('AAPL', start = '2000-01-01',

auto_adjust=True,#only download adjusted data

end= '2010-12-31')

#my convention: always lowercase

df.columns = ['open','high','low',

'close','volume']



return df prices = get_data('AAPL', '2000-01-01', '2010-12-31') #create some features

def create_HLCV(i):

#as we don't care open that much, that leaves volume,

#high,low and close

df = pd.DataFrame(index=prices.index)

df[f'high_{i}D'] = prices.high.rolling(i).max()

df[f'low_{i}D'] = prices.low.rolling(i).min()

df[f'close_{i}D'] = prices.close.rolling(i).\

apply(lambda x:x[-1])

# close_2D = close as rolling backwards means today is

# literly the last day of the rolling window.

df[f'volume_{i}D'] = prices.volume.rolling(i).sum()



return df # create features at different rolling windows

def create_features_and_outcomes(i):

df = create_HLCV(i)

high = df[f'high_{i}D']

low = df[f'low_{i}D']

close = df[f'close_{i}D']

volume = df[f'volume_{i}D']



features = pd.DataFrame(index=prices.index)

outcomes = pd.DataFrame(index=prices.index)



#as we already considered the different time span,

#here only day of simple percentage change used.



features[f'volume_{i}D'] = volume.pct_change()

features[f'price_spread_{i}D'] = (high - low).pct_change()

#aligne the close location with the stock price change

features[f'close_loc_{i}D'] = ((close - low) / \

(high - low)).pct_change() #the future outcome is what we are going to predict

outcomes[f'close_change_{i}D'] = close.pct_change(-i)



return features, outcomes def create_bunch_of_features_and_outcomes():

'''

the timespan that i would like to explore

are 1, 2, 3 days and 1 week, 1 month, 2 month, 3 month

which roughly are [1,2,3,5,20,40,60]

'''

days = [1,2,3,5,20,40,60]

bunch_of_features = pd.DataFrame(index=prices.index)

bunch_of_outcomes = pd.DataFrame(index=prices.index)



for day in days:

f,o = create_features_and_outcomes(day)

bunch_of_features = bunch_of_features.join(f)

bunch_of_outcomes = bunch_of_outcomes .join(o)



return bunch_of_features, bunch_of_outcomes bunch_of_features, bunch_of_outcomes = create_bunch_of_features_and_outcomes() #define the method to identify outliers

def get_outliers(df, i=4):

#i is number of sigma, which define the boundary along mean

outliers = pd.DataFrame()

stats = df.describe()



for col in df.columns:

mu = stats.loc['mean', col]

sigma = stats.loc['std', col]

condition = (df[col] > mu + sigma * i) | \

(df[col] < mu - sigma * i)

outliers[f'{col}_outliers'] = df[col][condition]



return outliers #remove all the outliers

features_outcomes = bunch_of_features.join(bunch_of_outcomes)

outliers = get_outliers(features_outcomes, i=1) features_outcomes_rmv_outliers = features_outcomes.drop(index = outliers.index).dropna() features = features_outcomes_rmv_outliers[bunch_of_features.columns]

outcomes = features_outcomes_rmv_outliers[bunch_of_outcomes.columns]

features.info(), outcomes.info()

Information of features dataset

Information of outcomes dataset

In the end, we will have the basic four features based on Volume Spread Analysis (VSA) at different time scale listed below, namely, 1 day, 2 days, 3 days, a week, a month, 2 months and 3 months.

Volume: pretty straight forward

Range/Spread: Difference between high and close

Closing Price Relative to Range: Is the closing price near the top or the bottom of the price bar?

The change of stock price: pretty straight forward

2. Percentage Returns

I know that’s a whole lot of codes above. We have all the features transformed into a simple percentage change through the function below.

def create_features_and_outcomes(i):

df = create_HLCV(i)

high = df[f'high_{i}D']

low = df[f'low_{i}D']

close = df[f'close_{i}D']

volume = df[f'volume_{i}D']



features = pd.DataFrame(index=prices.index)

outcomes = pd.DataFrame(index=prices.index)



#as we already considered the different time span,

#here only 1 day of simple percentage change used.



features[f'volume_{i}D'] = volume.pct_change()

features[f'price_spread_{i}D'] = (high - low).pct_change()

#aligne the close location with the stock price change

features[f'close_loc_{i}D'] = ((close - low) / \

(high - low)).pct_change() #the future outcome is what we are going to predict

outcomes[f'close_change_{i}D'] = close.pct_change(-i)



return features, outcomes

Now, let’s have a look at their correlations using cluster map. Seaborn’s clustermap() hierarchical clustering algorithm shows a nice way to group the most closely related features.

corr_features = features.corr().sort_index()

sns.clustermap(corr_features, cmap='coolwarm', linewidth=1);

Based on this cluster map, to minimize the amount of feature overlap in selected features, I will remove those features that are paired with other features closely and having less correlation with the outcome targets. From the cluster map above, it is easy to spot that features on [40D, 60D] and [2D, 3D] are paired together. To see how those features are related to the outcomes, let’s have a look at how the outcomes are correlated first.

corr_outcomes = outcomes.corr()

sns.clustermap(corr_outcomes, cmap='coolwarm', linewidth=2);

From top to bottom, 20 days, 40 days and 60 days price percentage change are grouped together, so as the 2 days, 3 days and 5 days. Whereas, 1-day stock price percentage change is relatively independent of those two groups. If we pick the next day price percentage change as the outcome target, let’s see how those features are related to it.

corr_features_outcomes = features.corrwith(outcomes. \

close_change_1D).sort_values()

corr_features_outcomes.dropna(inplace=True)

corr_features_outcomes.plot(kind='barh',title = 'Strength of Correlation');

The correlation coefficients are way too small to make a solid conclusion. I will expect that the most recent data have a stronger correlation, but that is not the case here.

How about the pair plot? We only pick those features based on a 1-day time scale as a demonstration. At the meantime, I transferred the close_change_1D to sign base on it’s a negative or positive number to add extra dimensionality to the plots.

selected_features_1D_list = ['volume_1D', 'price_spread_1D', 'close_loc_1D', 'close_change_1D']

features_outcomes_rmv_outliers['sign_of_close'] = features_outcomes_rmv_outliers['close_change_1D']. \

apply(np.sign) sns.pairplot(features_outcomes_rmv_outliers,

vars=selected_features_1D_list,

diag_kind='kde',

palette='husl', hue='sign_of_close',

markers = ['*', '<', '+'],

plot_kws={'alpha':0.3});

The pair plot builds on two basic figures, the histogram and the scatter plot. The histogram on the diagonal allows us to see the distribution of a single variable while the scatter plots on the upper and lower triangles show the relationship (or lack thereof) between two variables. From the plots above, we can see that price spreads are getting wider with high volume. Most of the price change locate at a narrow price spread, in another word, wider spread doesn’t always come with bigger price fluctuation. Either low volume or high volume can cause price change at almost all scale. And we can apply all those conclusions to both up days and down days.

you can also use the close location of bars to add more dimensionality, simply apply

features[‘sign_of_close_loc’] = np.where( \

features[‘close_loc_1D’] > 0.5, \

1, -1)

to see how many bars’ close location above the 0.5 or below 0.5.

One thing that I don’t really like in the pair plot is all the plots with the close_loc_1D condensed, looks like the outliers still there, even I know I used one standard deviation as the boundary which is a very low threshold and 338 outliers were removed. I realize that because the location of close is already a percentage change, adding another percentage change on top doesn’t make much sense. Let’s change it.

def create_features_and_outcomes(i):

df = create_HLCV(i)

high = df[f'high_{i}D']

low = df[f'low_{i}D']

close = df[f'close_{i}D']

volume = df[f'volume_{i}D']



features = pd.DataFrame(index=prices.index)

outcomes = pd.DataFrame(index=prices.index)



#as we already considered the different time span,

#simple percentage change of 1 day used here.



features[f'volume_{i}D'] = volume.pct_change()

features[f'price_spread_{i}D'] = (high - low).pct_change()

#remove pct_change() here

features[f'close_loc_{i}D'] = ((close - low) / (high - low))

#predict the future with -i

outcomes[f'close_change_{i}D'] = close.pct_change(-i)



return features, outcomes

With pct_change() removed, let’s see how the cluster map looks like now.

corr_features = features.corr().sort_index()

sns.clustermap(corr_features, cmap='coolwarm', linewidth=1);

The cluster map makes more sense now. All four basic features have pretty much the same pattern. [40D, 60D], [2D, 3D] are paired together.

and in terms of the features correlations with the outcome.

corr_features_outcomes.plot(kind='barh',title = 'Strength of Correlation');

The longer-range time scale features have weak correlations with stock price return, while the more recent events have more effects on the price returns.

By removing pct_change() of the close_loc_1D , the biggest difference is laid on the pairplot() .

Finally, the close_loc_1D variable plots at the right range. This illustrates that we should be careful with over-engineering. It may lead to a totally unexpected way.

3. Percentile Ranking

According to Wikipedia, the percentile rank is

“The percentile rank of a score is the percentage of scores in its frequency distribution that are equal to or lower than it. For example, a test score that is greater than 75% of the scores of people taking the test is said to be at the 75th percentile, where 75 is the percentile rank.”

The below example returns the percentile rank (from 0.00 to 1.00) of traded volume for each value as compared to a trailing 60-day period.