Data Transformation. How to apply modern Machine Learning on... | by Ke Gui | Towards Data Science

Warning: There is no magical formula or Holy Grail here, though a new world might open the door for you.
Identifying OutliersIdentifying Outliers — Part TwoIdentifying Outliers — Part ThreeStylized FactsFeature Engineering & Feature SelectionData TransformationFractionally Differentiated FeaturesData LabellingMeta-labeling and Stacking
Identifying Outliers
Identifying Outliers — Part Two
Identifying Outliers — Part Three
Stylized Facts
Feature Engineering & Feature Selection
Data Transformation
Fractionally Differentiated Features
Data Labelling
Meta-labeling and Stacking
In the previews article, I briefly introduced the Volume Spread Analysis(VSA). After we did feature-engineering and feature-selection, there were two things I noticed immediately, the first one was that there were outliers in the dataset and the second issue was the distribution were no way close to normal. By using the method described here, here and here, I removed most of the outliers. Now is the time to face the bigger problem, the normality.
There are many ways to transfer the data. One of the well-known examples is the one-hot encoding, even better one is word embedding in natural language processing (NLP). Considering one of the advantages of using deep learning is that it completely automates what used to be the most crucial step in a machine-learning workflow: feature engineering. Before we get into the deep learning in the later articles, let’s have a look at some simple ways to transfer data to see if we can make it closer to normal distribution.
In this article, I would like to try a few things. The first one is to transfer all the features to a simple percentage change. The second one is to do a Percentile Ranking. In the end, I will show you what happens if I only pick the sign of all the data. Methods like Z-score, which are standard pre-processing in deep learning, I would rather leave it for now.
For consistency, in all the 📈Python for finance series, I will try to reuse the same data as much as I can. More details about data preparation can be found here, here and here or you can refer back to my previous article. Or if you like, you can ignore all the code below and use whatever clean data you have at hand, it won’t affect the things we are going to do together.
#import all the librariesimport pandas as pdimport numpy as npimport seaborn as sns import yfinance as yf  #the stock data from Yahoo Financeimport matplotlib.pyplot as plt #set the parameters for plottingplt.style.use('seaborn')plt.rcParams['figure.dpi'] = 300#define a function to get datadef get_data(symbols, begin_date=None,end_date=None):    df = yf.download('AAPL', start = '2000-01-01',                     auto_adjust=True,#only download adjusted data                     end= '2010-12-31')     #my convention: always lowercase    df.columns = ['open','high','low',                  'close','volume']         return dfprices = get_data('AAPL', '2000-01-01', '2010-12-31')#create some featuresdef create_HLCV(i):#as we don't care open that much, that leaves volume, #high,low and close     df = pd.DataFrame(index=prices.index)    df[f'high_{i}D'] = prices.high.rolling(i).max()    df[f'low_{i}D'] = prices.low.rolling(i).min()    df[f'close_{i}D'] = prices.close.rolling(i).\                        apply(lambda x:x[-1])     # close_2D = close as rolling backwards means today is     # literly the last day of the rolling window.    df[f'volume_{i}D'] = prices.volume.rolling(i).sum()        return df# create features at different rolling windowsdef create_features_and_outcomes(i):    df = create_HLCV(i)    high = df[f'high_{i}D']    low = df[f'low_{i}D']    close = df[f'close_{i}D']    volume = df[f'volume_{i}D']        features = pd.DataFrame(index=prices.index)    outcomes = pd.DataFrame(index=prices.index)        #as we already considered the different time span,     #here only day of simple percentage change used.       features[f'volume_{i}D'] = volume.pct_change()    features[f'price_spread_{i}D'] = (high - low).pct_change()    #aligne the close location with the stock price change    features[f'close_loc_{i}D'] = ((close - low) / \                             (high -   low)).pct_change()    #the future outcome is what we are going to predict    outcomes[f'close_change_{i}D'] = close.pct_change(-i)           return features, outcomesdef create_bunch_of_features_and_outcomes():    '''    the timespan that i would like to explore     are 1, 2, 3 days and 1 week, 1 month, 2 month, 3 month    which roughly are [1,2,3,5,20,40,60]    '''    days = [1,2,3,5,20,40,60]    bunch_of_features = pd.DataFrame(index=prices.index)    bunch_of_outcomes = pd.DataFrame(index=prices.index)        for day in days:        f,o = create_features_and_outcomes(day)        bunch_of_features = bunch_of_features.join(f)        bunch_of_outcomes = bunch_of_outcomes .join(o)            return bunch_of_features, bunch_of_outcomesbunch_of_features, bunch_of_outcomes = create_bunch_of_features_and_outcomes()#define the method to identify outliersdef get_outliers(df, i=4):     #i is number of sigma, which define the boundary along mean    outliers = pd.DataFrame()    stats = df.describe()        for col in df.columns:        mu = stats.loc['mean', col]        sigma = stats.loc['std', col]        condition = (df[col] > mu + sigma * i) | \                  (df[col] < mu -   sigma * i)         outliers[f'{col}_outliers'] = df[col][condition]        return outliers#remove all the outliersfeatures_outcomes = bunch_of_features.join(bunch_of_outcomes)outliers = get_outliers(features_outcomes, i=1)features_outcomes_rmv_outliers = features_outcomes.drop(index = outliers.index).dropna()features = features_outcomes_rmv_outliers[bunch_of_features.columns]outcomes = features_outcomes_rmv_outliers[bunch_of_outcomes.columns]features.info(), outcomes.info()
In the end, we will have the basic four features based on Volume Spread Analysis (VSA) at different time scale listed below, namely, 1 day, 2 days, 3 days, a week, a month, 2 months and 3 months.
Volume: pretty straight forward
Range/Spread: Difference between high and close
Closing Price Relative to Range: Is the closing price near the top or the bottom of the price bar?
The change of stock price: pretty straight forward
I know that’s a whole lot of codes above. We have all the features transformed into a simple percentage change through the function below.
def create_features_and_outcomes(i):    df = create_HLCV(i)    high = df[f'high_{i}D']    low = df[f'low_{i}D']    close = df[f'close_{i}D']    volume = df[f'volume_{i}D']        features = pd.DataFrame(index=prices.index)    outcomes = pd.DataFrame(index=prices.index)        #as we already considered the different time span,     #here only 1 day of simple percentage change used.       features[f'volume_{i}D'] = volume.pct_change()    features[f'price_spread_{i}D'] = (high - low).pct_change()    #aligne the close location with the stock price change    features[f'close_loc_{i}D'] = ((close - low) / \    (high -    low)).pct_change()#the future outcome is what we are going to predict    outcomes[f'close_change_{i}D'] = close.pct_change(-i)           return features, outcomes
Now, let’s have a look at their correlations using cluster map. Seaborn’s clustermap() hierarchical clustering algorithm shows a nice way to group the most closely related features.
corr_features = features.corr().sort_index()sns.clustermap(corr_features, cmap='coolwarm', linewidth=1);
Based on this cluster map, to minimize the amount of feature overlap in selected features, I will remove those features that are paired with other features closely and having less correlation with the outcome targets. From the cluster map above, it is easy to spot that features on [40D, 60D] and [2D, 3D] are paired together. To see how those features are related to the outcomes, let’s have a look at how the outcomes are correlated first.
corr_outcomes = outcomes.corr()sns.clustermap(corr_outcomes, cmap='coolwarm', linewidth=2);
From top to bottom, 20 days, 40 days and 60 days price percentage change are grouped together, so as the 2 days, 3 days and 5 days. Whereas, 1-day stock price percentage change is relatively independent of those two groups. If we pick the next day price percentage change as the outcome target, let’s see how those features are related to it.
corr_features_outcomes = features.corrwith(outcomes. \                                close_change_1D).sort_values()corr_features_outcomes.dropna(inplace=True)corr_features_outcomes.plot(kind='barh',title = 'Strength of Correlation');
The correlation coefficients are way too small to make a solid conclusion. I will expect that the most recent data have a stronger correlation, but that is not the case here.
How about the pair plot? We only pick those features based on a 1-day time scale as a demonstration. At the meantime, I transferred the close_change_1D to sign base on it’s a negative or positive number to add extra dimensionality to the plots.
selected_features_1D_list = ['volume_1D', 'price_spread_1D', 'close_loc_1D', 'close_change_1D']features_outcomes_rmv_outliers['sign_of_close'] = features_outcomes_rmv_outliers['close_change_1D']. \                                                  apply(np.sign)sns.pairplot(features_outcomes_rmv_outliers,              vars=selected_features_1D_list,             diag_kind='kde',             palette='husl', hue='sign_of_close',             markers = ['*', '<', '+'],              plot_kws={'alpha':0.3});
The pair plot builds on two basic figures, the histogram and the scatter plot. The histogram on the diagonal allows us to see the distribution of a single variable while the scatter plots on the upper and lower triangles show the relationship (or lack thereof) between two variables. From the plots above, we can see that price spreads are getting wider with high volume. Most of the price change locate at a narrow price spread, in another word, wider spread doesn’t always come with bigger price fluctuation. Either low volume or high volume can cause price change at almost all scale. And we can apply all those conclusions to both up days and down days.
you can also use the close location of bars to add more dimensionality, simply apply
features[‘sign_of_close_loc’] = np.where( \ features[‘close_loc_1D’] > 0.5, \ 1, -1)
to see how many bars’ close location above the 0.5 or below 0.5.
One thing that I don’t really like in the pair plot is all the plots with the close_loc_1D condensed, looks like the outliers still there, even I know I used one standard deviation as the boundary which is a very low threshold and 338 outliers were removed. I realize that because the location of close is already a percentage change, adding another percentage change on top doesn’t make much sense. Let’s change it.
def create_features_and_outcomes(i):    df = create_HLCV(i)    high = df[f'high_{i}D']    low = df[f'low_{i}D']    close = df[f'close_{i}D']    volume = df[f'volume_{i}D']        features = pd.DataFrame(index=prices.index)    outcomes = pd.DataFrame(index=prices.index)        #as we already considered the different time span,     #simple percentage change of 1 day used here.        features[f'volume_{i}D'] = volume.pct_change()    features[f'price_spread_{i}D'] = (high - low).pct_change()    #remove pct_change() here    features[f'close_loc_{i}D'] = ((close - low) / (high - low))    #predict the future with -i    outcomes[f'close_change_{i}D'] = close.pct_change(-i)           return features, outcomes
With pct_change() removed, let’s see how the cluster map looks like now.
corr_features = features.corr().sort_index()sns.clustermap(corr_features, cmap='coolwarm', linewidth=1);
The cluster map makes more sense now. All four basic features have pretty much the same pattern. [40D, 60D], [2D, 3D] are paired together.
and in terms of the features correlations with the outcome.
corr_features_outcomes.plot(kind='barh',title = 'Strength of Correlation');
The longer-range time scale features have weak correlations with stock price return, while the more recent events have more effects on the price returns.
By removing pct_change() of the close_loc_1D, the biggest difference is laid on the pairplot().
Finally, the close_loc_1D variable plots at the right range. This illustrates that we should be careful with over-engineering. It may lead to a totally unexpected way.
According to Wikipedia, the percentile rank is
“The percentile rank of a score is the percentage of scores in its frequency distribution that are equal to or lower than it. For example, a test score that is greater than 75% of the scores of people taking the test is said to be at the 75th percentile, where 75 is the percentile rank.”
The below example returns the percentile rank (from 0.00 to 1.00) of traded volume for each value as compared to a trailing 60-day period.
roll_rank = lambda x: pd.Series(x).rank(pct=True)[-1]# you only pick the first value [0]# of the 60 windows rank if you rolling forward.# if you rolling backward, we should pick last one,[-1].features_rank = features.rolling(60, min_periods=60). \                apply(roll_rank).dropna()outcomes_rank = outcomes.rolling(60, min_periods=60). \                apply(roll_rank).dropna()
Pandasrolling(), by default, the result is set to the right edge of the window. That means the window is backward-looking windows, from the past rolls towards current timestamp. That is why, to rank() in that window frame, we pick the last value [-1].
More information about rolling(), please check the official document.
First, we have a quick look at the outcomes’ cluster map. It is almost identical to the percentage change one with a different order.
corr_outcomes_rank = outcomes_rank.corr().sort_index()sns.clustermap(corr_outcomes_rank, cmap='coolwarm', linewidth=2);
The same pattern goes to the features’ cluster map.
corr_features_rank = features_rank.corr().sort_index()sns.clustermap(corr_features_rank, cmap='coolwarm', linewidth=2);
Even with a different method,
# using 'ward' methodcorr_features_rank = features_rank.corr().sort_index()sns.clustermap(corr_features_rank, cmap='coolwarm', linewidth=2, method='ward');
and of course, the correlation of features and outcome are the same as well.
corr_features_outcomes_rank = features_rank.corrwith( \                              outcomes_rank. \                              close_change_1D).sort_values()corr_features_outcomes_rank
corr_features_outcomes_rank.plot(kind='barh',title = 'Strength of Correlation');
Last, you may guess the pair plot will be the same as well.
selected_features_1D_list = ['volume_1D', 'price_spread_1D', 'close_loc_1D', 'close_change_1D']features_outcomes_rank['sign_of_close'] = features_outcomes_rmv_outliers['close_change_1D']. \                                                  apply(np.sign)sns.pairplot(features_outcomes_rank,              vars=selected_features_1D_list,             diag_kind='kde',             palette='husl', hue='sign_of_close',             markers = ['*', '<', '+'],              plot_kws={'alpha':0.3});
Because of the percentile rank (from 0.00 to 1.00) we utilized in the set window, the spots are evenly distributed across all features. The distribution of all the features is more or less close to a normal distribution than the same data without transformation.
The last not least, I would like to remove all the data grain and see how those features related under this scenario.
features_sign = features.apply(np.sign)outcomes_sign = outcomes.apply(np.sign)
Then calculate the correlation coefficiency again.
corr_features_outcomes_sign = features_sign.corrwith(                              outcomes_sign. \                              close_change_1D).sort_values(ascending=False)corr_features_outcomes_sign
corr_features_outcomes_sign.plot(kind='barh',title = 'Strength of Correlation');
It turns out a bit weird now, like volume_1D and price_spread_1D has a very weak correlation with the outcome now.
Luckily, the cluster map remains pretty much the same.
corr_features_sign = features_sign.corr().sort_index()sns.clustermap(corr_features_sign, cmap='coolwarm', linewidth=2);
And the same goes for the relationship between outcomes.
corr_outcomes_sign = outcomes_sign.corr().sort_index()sns.clustermap(corr_outcomes_sign, cmap='coolwarm', linewidth=2);
As for pair plot, as all the data are transferred to either -1 or 1, it doesn’t show anything meaningful.
It is sometimes vital to “standardize” or “normalize” data so that we get fair comparisons between features of differing scale. I am tempted to use Z-score to normalize the data set.
The formula of Z-score requires the mean and standard deviation, by calculating these two parameters across the entire dataset, we have the chance to peek to the future. Of course, we can take advantage of the rolling window again. But generally, people will normalize their data before injecting them into their model.
In summary, by utilizing 3 different data transformations methods, now we are pretty confident we can select the most related features and discard those abundant ones as all 3 methods pretty much share the same patterns.
The last question can the transformed data pass the stationary/normality test? Here, I will use the Augmented Dickey-Fuller test1, which is a type of statistical test called a unit root test. At the meantime, I want to see the skewness and kurtosis as well.
import statsmodels.api as smimport scipy.stats as scsp_val = lambda s: sm.tsa.stattools.adfuller(s)[1]def build_stats(df):    stats = pd.DataFrame({'skew':scs.skew(df),                 'skew_test':scs.skewtest(df)[1],                 'kurtosis': scs.kurtosis(df),                 'kurtosis_test' : scs.kurtosistest(df)[1],                 'normal_test' : scs.normaltest(df)[1]},                  index = df.columns)    return stats
The null hypothesis of the test is that the time series can be represented by a unit root, that it is not stationary (has some time-dependent structure). The alternate hypothesis (rejecting the null hypothesis) is that the time series is stationary.
Null Hypothesis (H0): If failed to be rejected, it suggests the time series has a unit root, meaning it is non-stationary. It has some time dependent structure.
Alternate Hypothesis (H1): The null hypothesis is rejected; it suggests the time series does not have a unit root, meaning it is stationary. It does not have time-dependent structure.
Here is the result from Augmented Dickey-Fuller test:
For features and outcomes:
features_p_val = features.apply(p_val)outcomes_p_val = outcomes.apply(p_val)outcomes_p_val,features_p_val
The test can be interpreted by the p-value. A p-value below a threshold (such as 5% or 1%) suggests we reject the null hypothesis (stationary), otherwise, a p-value above the threshold suggests we cannot reject the null hypothesis (non-stationary).
p-value > 0.05: cannot reject the null hypothesis (H0), the data has a unit root and is non-stationary.
p-value <= 0.05: Reject the null hypothesis (H0), the data does not have a unit root and is stationary.
From this test, we can see that all the results are well below 5%, that shows we can reject the null hypothesis and all the transformed data are stationary.
Next, let’s test the normality.
build_stats(features_outcomes_rmv_outliers)
For normally distributed data, the skewness should be about zero. For unimodal continuous distributions, a skewness value greater than zero meansthat there is more weight in the right tail of the distribution and vice versa.
scs.skewtest() tests the null hypothesis that the skewness of the population that the sample was drawn from is the same as that of a corresponding normal distribution. As all the numbers are below 5% threshold, we have to reject the null hypothesis and say the skewness doesn’t correspond to normal distribution. The same thing goes to scs.kurtosistest().
scs.normaltest() tests the null hypothesis that a sample comes from a normal distribution. It is based on D’Agostino and Pearson’s test2 3 that combines skew and kurtosis to produce an omnibus test of normality. Again, all the numbers are below 5% threshold. We have to reject the null hypothesis and say the data transformed by percentage change is not normal distribution.
We can do the same tests on data transformed by Percentile Ranking and Signing. I don’t want to scare people off by complexing thing further. I am better off ending here before this article goes way too long.
MacKinnon, J.G. 1994. “Approximate asymptotic distribution functions for unit-root and cointegration tests. `Journal of Business and Economic Statistics` 12, 167–76.D’Agostino, R. B. (1971), “An omnibus test of normality for moderate and large sample size”, Biometrika, 58, 341–348D’Agostino, R. and Pearson, E. S. (1973), “Tests for departure from normality”, Biometrika, 60, 613–622
MacKinnon, J.G. 1994. “Approximate asymptotic distribution functions for unit-root and cointegration tests. `Journal of Business and Economic Statistics` 12, 167–76.
D’Agostino, R. B. (1971), “An omnibus test of normality for moderate and large sample size”, Biometrika, 58, 341–348
D’Agostino, R. and Pearson, E. S. (1973), “Tests for departure from normality”, Biometrika, 60, 613–622