Feature Engineering & Feature Selection
By: Ke Gui
Tags: Machine Learning, Python, Feature Engineering, Trading, Feature Selection

2. Feature Engineering

The key point of combining VSA with modern data science is through reading and interpreting the bars' own actions, one (hopefully algorithm) can construct a story of the market behaviours. The story might not be easily understood by a human, but works in a sophisticated way.

Volume in conjunction with the price range and the position of the close is easy to be expressed by code.

Volume: pretty straight forward

Range/Spread: Difference between high and close

def price_spread(df):

return (df.high - df.low)

Closing Price Relative to Range: Is the closing price near the top or the bottom of the price bar?

def close_location(df):

return (df.high - df.close) / (df.high - df.low) #o indicates the close is the high of the day, and 1 means close

#is the low of the day and the smaller the value, the closer the #close price to the high.

The change of stock price: pretty straight forward

Now comes the tricky part,

“When viewed in a larger context, some of the price bars take on a new meaning.”

That means to see the full pictures, we need to observe those 4 basic features under a different time scale.

To do that, we need to reconstruct a High(H), Low(L), Close(C) and Volume(V) bar at varied time span.

def create_HLCV(i):

'''

#i: days

#as we don't care about open that much, that leaves volume,

#high,low and close

''' df = pd.DataFrame(index=prices.index) df[f'high_{i}D'] = prices.high.rolling(i).max()

df[f'low_{i}D'] = prices.low.rolling(i).min()

df[f'close_{i}D'] = prices.close.rolling(i).\

apply(lambda x:x[-1])

# close_2D = close as rolling backwards means today is

#literally, the last day of the rolling window.

df[f'volume_{i}D'] = prices.volume.rolling(i).sum()



return df

next step, create those 4 basic features based on a different time scale.

def create_features(i):

df = create_HLCV(i)

high = df[f'high_{i}D']

low = df[f'low_{i}D']

close = df[f'close_{i}D']

volume = df[f'volume_{i}D']



features = pd.DataFrame(index=prices.index)

features[f'volume_{i}D'] = volume

features[f'price_spread_{i}D'] = high - low

features[f'close_loc_{i}D'] = (high - close) / (high - low)

features[f'close_change_{i}D'] = close.diff()



return features

The time spans that I would like to explore are 1, 2, 3 days and 1 week, 1 month, 2 months, 3 months, which roughly are [1,2,3,5,20,40,60] days. Now, we can create a whole bunch of features,

def create_bunch_of_features():

days = [1,2,3,5,20,40,60]

bunch_of_features = pd.DataFrame(index=prices.index)

for day in days:

f = create_features(day)

bunch_of_features = bunch_of_features.join(f)



return bunch_of_features bunch_of_features = create_bunch_of_features()

bunch_of_features.info()

To make things easy to understand, our target outcome will only be the next day’s return.

# next day's returns as outcomes

outcomes = pd.DataFrame(index=prices.index)

outcomes['close_1'] = prices.close.pct_change(-1)

3. Feature Selection

Let’s have a look at how those features correlated with outcomes, the next day’s return.

corr = bunch_of_features.corrwith(outcomes.close_1)

corr.sort_values(ascending=False).plot.barh(title = 'Strength of Correlation');

It is hard to say there are some correlations, as all the numbers are well below 0.8.

corr.sort_values(ascending=False)

Next, let’s see how those features related to each other.

corr_matrix = bunch_of_features.corr()

Instead of making heatmap, I am trying to use Seaborn’s Clustermap to cluster row-wise or col-wise to see if there is any pattern emerges. Seaborn’s Clustermap function is great for making simple heatmaps and hierarchically-clustered heatmaps with dendrograms on both rows and/or columns. This reorganizes the data for the rows and columns and displays similar content next to one another for even more depth of understanding the data. A nice tutorial about cluster map can be found here. To get a cluster map, all you need is actually one line of code.

sns.clustermap(corr_matrix)

If you carefully scrutinize the graph, some conclusions can be drawn:

Price spread closely related to the volume, as clearly shown at the centre of the graph. And the location of close related to each other at different timespan, as indicated at the bottom right corner. From the pale colour of the top left corner, close price change does pair with itself, which makes perfect sense. However, it is a bit random as no cluster pattern at varied time scale. I would expect that 2Days change should be paired with 3Days change.

The randomness of the close price difference could thank to the characteristics of the stock price itself. Simple percentage return might be a better option. This can be realized by modifying the close diff() to close pct_change() .

def create_features_v1(i):

df = create_HLCV(i)

high = df[f'high_{i}D']

low = df[f'low_{i}D']

close = df[f'close_{i}D']

volume = df[f'volume_{i}D']



features = pd.DataFrame(index=prices.index)

features[f'volume_{i}D'] = volume

features[f'price_spread_{i}D'] = high - low

features[f'close_loc_{i}D'] = (high - close) / (high - low)

#only change here

features[f'close_change_{i}D'] = close.pct_change()



return features

and do everything again.

def create_bunch_of_features_v1():

days = [1,2,3,5,20,40,60]

bunch_of_features = pd.DataFrame(index=prices.index)

for day in days:

f = create_features_v1(day)#here is the only difference

bunch_of_features = bunch_of_features.join(f)



return bunch_of_features bunch_of_features_v1 = create_bunch_of_features_v1() #check the correlation

corr_v1 = bunch_of_features_v1.corrwith(outcomes.close_1)

corr_v1.sort_values(ascending=False).plot.barh( title = 'Strength of Correlation')

a little bit different, but not much!

corr_v1.sort_values(ascending=False)

What happens to the correlation between features?

corr_matrix_v1 = bunch_of_features_v1.corr()

sns.clustermap(corr_matrix_v1, cmap='coolwarm', linewidth=1)

Well, the pattern remains unchanged. Let’s change the default method from “average” to “ward”. These two methods are similar, but “ward” is more like K-MEANs clustering. A nice tutorial on this topic can be found here.

sns.clustermap(corr_matrix_v1, cmap='coolwarm', linewidth=1,

method='ward')

To select features, we want to pick those that have the strongest, most persistent relationships to the target outcome. At the meantime, to minimize the amount of overlap or collinearity in your selected features to avoid noise and waste of computer power. For those features that paired together in a cluster, I only pick the one that has a stronger correlation with the outcome. By just looking at the cluster map, a few features are picked out.

deselected_features_v1 = ['close_loc_3D','close_loc_60D',

'volume_3D', 'volume_60D',

'price_spread_3D','price_spread_60D',

'close_change_3D','close_change_60D'] selected_features_v1 = bunch_of_features.drop \

(labels=deselected_features_v1, axis=1)

Next, we are going to take a look at pair-plot, A pair plot is a great method to identify trends for follow-up analysis, allowing us to see both distributions of single variables and relationships between multiple variables. Again, all we need is a single line of code.

sns.pairplot(selected_features_v1)

The graph is overwhelming and hard to see. Let’s take a small group as an example.

selected_features_1D_list = ['volume_1D', 'price_spread_1D',\ 'close_loc_1D', 'close_change_1D'] selected_features_1D = selected_features_v1\

[selected_features_1D_list] sns.pairplot(selected_features_1D)

There are two things I noticed immediately, one is there are outliers and another is the distribution are no way close to normal.

Let’s deal with the outliers for now. In order to do everything in one go, I will join the outcome with features and remove outliers together.

features_outcomes = selected_features_v1.join(outcomes)

features_outcomes.info()

I will use the same method described here, here and here to remove the outliers.

stats = features_outcomes.describe()

def get_outliers(df, i=4):

#i is number of sigma, which define the boundary along mean

outliers = pd.DataFrame()



for col in df.columns:

mu = stats.loc['mean', col]

sigma = stats.loc['std', col]

condition = (df[col] > mu + sigma * i) | (df[col] < mu - sigma * i)

outliers[f'{col}_outliers'] = df[col][condition]



return outliers outliers = get_outliers(features_outcomes, i=1)

outliers.info()

I set 1 standard deviation as the boundary to dig out most of the outliers. Then remove all the outliers along with the NaN values.

features_outcomes_rmv_outliers = features_outcomes.drop(index = outliers.index).dropna()

features_outcomes_rmv_outliers.info()

With the outliers removed, we can do the pair plot again.

sns.pairplot(features_outcomes_rmv_outliers, vars=selected_features_1D_list);

Now, the plots are looking much better, but it is barely to draw any useful conclusions. It would be nice to see which spots are down moves and which are up moves in conjunction with those features. I can extract the sign of stock price change and add an extra dimension to the plots.

features_outcomes_rmv_outliers['sign_of_close'] = features_outcomes_rmv_outliers['close_1'].apply(np.sign)

Now, let’s re-plot the pairplot() again with a bit of tweak to make the graph pretty.

sns.pairplot(features_outcomes_rmv_outliers,

vars=selected_features_1D_list,

diag_kind='kde',

palette='husl', hue='sign_of_close',

markers = ['*', '<', '+'],

plot_kws={'alpha':0.3});#transparence:0.3

Now, it looks much better. Clearly, when the prices go up, they (the blue spot) are denser and aggregate at a certain location. Whereas on down days, they spread everywhere.

I would really appreciate it if you could shed some light on the pair plot and leave your comments below, thanks.

Here is the summary of all the codes used in this article:

#import all the libraries

import pandas as pd

import numpy as np

import seaborn as sns

import yfinance as yf #the stock data from Yahoo Finance import matplotlib.pyplot as plt #set the parameters for plotting

plt.style.use('seaborn')

plt.rcParams['figure.dpi'] = 300 #define a function to get data

def get_data(symbols, begin_date=None,end_date=None):

df = yf.download('AAPL', start = '2000-01-01',

auto_adjust=True,#only download adjusted data

end= '2010-12-31')

#my convention: always lowercase

df.columns = ['open','high','low',

'close','volume']



return df prices = get_data('AAPL', '2000-01-01', '2010-12-31') #create some features

def create_HLCV(i): #as we don't care open that much, that leaves volume,

#high,low and close df = pd.DataFrame(index=prices.index)

df[f'high_{i}D'] = prices.high.rolling(i).max()

df[f'low_{i}D'] = prices.low.rolling(i).min()

df[f'close_{i}D'] = prices.close.rolling(i).\

apply(lambda x:x[-1])

# close_2D = close as rolling backwards means today is

# literly the last day of the rolling window.

df[f'volume_{i}D'] = prices.volume.rolling(i).sum()



return df def create_features_v1(i):

df = create_HLCV(i)

high = df[f'high_{i}D']

low = df[f'low_{i}D']

close = df[f'close_{i}D']

volume = df[f'volume_{i}D']



features = pd.DataFrame(index=prices.index)

features[f'volume_{i}D'] = volume

features[f'price_spread_{i}D'] = high - low

features[f'close_loc_{i}D'] = (high - close) / (high - low)

features[f'close_change_{i}D'] = close.pct_change()



return features def create_bunch_of_features_v1():

'''

the timespan that i would like to explore

are 1, 2, 3 days and 1 week, 1 month, 2 month, 3 month

which roughly are [1,2,3,5,20,40,60]

'''

days = [1,2,3,5,20,40,60]

bunch_of_features = pd.DataFrame(index=prices.index)

for day in days:

f = create_features_v1(day)

bunch_of_features = bunch_of_features.join(f)



return bunch_of_features bunch_of_features_v1 = create_bunch_of_features_v1() #define the outcome target

#here， to make thing easy to understand, i will only try to predict #the next days's return

outcomes = pd.DataFrame(index=prices.index) # next day's returns

outcomes['close_1'] = prices.close.pct_change(-1) #decide which features are abundant from cluster map

deselected_features_v1 = ['close_loc_3D','close_loc_60D',

'volume_3D', 'volume_60D',

'price_spread_3D','price_spread_60D',

'close_change_3D','close_change_60D']

selected_features_v1 = bunch_of_features_v1.drop(labels=deselected_features_v1, axis=1) #join the features and outcome together to remove the outliers

features_outcomes = selected_features_v1.join(outcomes)

stats = features_outcomes.describe() #define the method to identify outliers

def get_outliers(df, i=4):

#i is number of sigma, which define the boundary along mean

outliers = pd.DataFrame()



for col in df.columns:

mu = stats.loc['mean', col]

sigma = stats.loc['std', col]

condition = (df[col] > mu + sigma * i) | (df[col] < mu - sigma * i)

outliers[f'{col}_outliers'] = df[col][condition]



return outliers outliers = get_outliers(features_outcomes, i=1) #remove all the outliers and Nan value

features_outcomes_rmv_outliers = features_outcomes.drop(index = outliers.index).dropna()

I know this article goes too long, I am better off leaving it here. In the next article, I will do a data transformation to see if I have a way to fix the issue of distribution. Stay tuned!