Feature Engineering
By: Ke Gui
Tags: Machine Learning, Python, Finance, Fractional Calculus, Marco Prato

Introduction

Fractional differentiation (or Fractional derivative or Fractional calculus) is a great idea once you understood all the equations. But the first time when I was reading Chapter 5 of Advances in Financial Machine Learning by Marcos Prado. All the mathematical symbols and equations just push me away from it.

I was like, what is this?

Believe me, if you feel lost, you are not alone. The reactions of most people just like what Homer Simpson said: “If something’s hard to do, then it’s not worth doing.” Luckily, I was one of the most people until now.

Let’s face it and it wasn’t as hard as it looks like. We can break it down and figure it out step by step with the first principle.

Ok, let’s break down the whole chapter and talk it through.. together, emmm.…I mean together…

Background

If you have no idea of what derivative is or need to refresh your memory of derivative learned in school. There are a few really great tutorials online.

The paradox of the derivative | Essence of calculus, chapter 2 by 3BLue1brown.

Fractional Calculus ~ (FC01) An Introduction to Fractional Calculus by Let’s Learn, Nemo!

well, I watched those 2 videos, but I am still having no idea…. what???? well, you can always watch the third one, I guess.

Ok, let’s do it slowly.

Why Fractional Differentiation?

In previous articles, we have explored the Feature Engineering & Feature Selection and Data Transformation etc.. Why Fractional Differentiation?

Definition: A fractionally differenced time series (as opposed to integer differencing) retains the memory of the original series (as indicated by the high correlation), but also it is stationary, the mean and variance of those time series should be time-invariant (or not change with time). (e.g. have the same variance, mean, skew, etc).

What is stationary: A stationary time series is one whose properties do not depend on the time at which the series is observed. Thus, time series with trends, or with seasonality, are not stationary — the trend and seasonality will affect the value of the time series at different times.

Why stationary is important?

If the data is not stationary, it would be wrong that all the assumptions we made based on the financial market data are Independent and Identically Distributed (IID). If the data (features) are not “stationary” (in other words, their underlying data generation process changes its characteristics with time) it would be hard to predict expected future date.

Sadly, in financial time series data, the volatility of prices can change rapidly from time to time by events, such as on important information releasing day. Both simple percentages return and log returns as well demonstrated in previous articles in these series, remove most of these relationships (low correlation).

This leads to an enormous challenge — how can one make the time series stationary while keeping its predictive power (or memory or information for prediction).

In the extreme, consider transforming any features into strictly 1’s (using np.sign() ) you’ve successfully attained stationarity, but at the cost of all information in the original series. On the other hand, the original stock price has zero differentiation, it has all the information but not stationary.

Lopez de Prado proposes an alternative method, named fractional differentiation, that aims to find the optimal balance between zero differentiation and fully differentiated. The goal of the fractional differentiation is to find the fraction d, which is the minimum number necessary to achieve stationarity, while keeps the maximum amount of memory in our data.

Intuition

My 1st reaction to Fractional Differentiation really reminds me of Exponential Moving Average (EMA)….

More accurate name of EMA is exponentially weighted moving average (EWMA). Todays’ stock price is the sum of weighted N days stock price before.

In general, a weighted moving average is calculated as

The EMA is a moving average that places a greater weight and significance on the most recent data points by decreasing the weighting factors exponentially.

EMA weights N = 15 from Wikipedia

Calculating the EMA requires one more observation than the simple moving average (SMA). Suppose that you want to use 20 days as the number of observations for the EMA. Then, you must wait until the 20th day to obtain the SMA. On the 21st day, you can then use the SMA from the previous day as the first EMA for yesterday. However, EMA doesn’t remove the trend, it will be non-stationary.

More of Exponential Moving Average (EMA) can be found on Pandas official document.

Note for each paragraph

NOW please go to chapter 5.

Like EWM, a feature at current timestamp can be expressed as the sum of all the past values with an assigned weight for each value. Mathematically, it is the sum of a dot product (see 3blue1brown for dot product).

The current value is the function of all the past value before this time point, this is where the long memory comes from:

The question is how to decide each weight for each corresponding value?

The weight is calculated by fractional derivative, and can be expressed as:

When d is a positive integer number, like 1,2,3… etc. As ∀k > d, (∀ is a math symbol means for all), there must be a point where d equal to k and d-k=0:

and memory beyond that point will be removed because of dot product with 0. For example, if d=1, put d into 𝜔:

then, 𝜔={1, −1, 0, 0, …}, if d=2, 𝜔={1, −2, 1, 0, …} and so on so forth.

On the contrary, if d=0.1, as k is always an integer number, wight 𝜔 will be some value other than 0 and the memory will be preserved.

Next question is how to transfer the weight 𝜔 equation into code? If you have watched the fractional derivative video above, the weight can be expressed as :

Now, iterative is one thing code is good at. The weight can be expressed as a simple for loop.

def getWeights(d,size):

'''

d:fraction

k:the number of samples

w:weight assigned to each samples



'''

# thres>0 drops insignificant weights w=[1.]

for k in range(1,size):

w_ = -w[-1]/k*(d-k+1)

w.append(w_)

w=np.array(w[::-1]).reshape(-1,1) #sort and reshape the w

return w

let’s get some real stock data and try it out. For consistency, in all the 📈Python for finance series, I will try to reuse the same data as much as I can. More details about data preparation can be found here, here and here.

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

import seaborn as sns

plt.style.use('seaborn')

plt.rcParams['figure.figsize'] = [16, 9]

plt.rcParams['figure.dpi'] = 300 import yfinance as yf def get_data(symbols, begin_date=None,end_date=None):

df = yf.download('AAPL', start = '2000-01-01',

auto_adjust=True,#only download adjusted data

end= '2010-12-31')

#my convention:always lowercase

df.columns = ['open','high','low',

'close','volume']



return df

Apple_stock = get_data('AAPL', '2000-01-01', '2010-12-31')

price = Apple_stock['close']

price.head()

We get the weight for each day’s stock price of 10 years Apple stock price.

w = getWeights(0.1, price.shape[0])

w

Next, we can set different fraction d and see how the weight changes.

def weight_by_d(dRange=[0,1], nPlots=11, size=6):

'''

dRange: the range of d

nPlots: the number of d we want to check

size: the data points used as an example

w: collection of w by different d value

'''



w=pd.DataFrame()



for d in np.linspace(dRange[0],dRange[1],nPlots):

w_=getWeights(d,size=size)

w_=pd.DataFrame(w_,index=range(w_.shape[0])\

[::-1],columns=[d])

w=w.join(w_,how='outer')



return w

weight_by_d = weight_by_d()

weight_by_d

Then plot those weights into a graph for comparison.

def plotWeights(w, v=2):

ax=w.plot()

ax.axvline(x=v, c='c', linestyle='-.', lw=2.5, alpha=0.4)

ax.axhline(y=0, c='c', linestyle='-.', lw=2.5, alpha=0.4)

ax.legend(loc='upper right');

ax.set_ylabel('Weight')

ax.set_xlabel('Number of data points') plotWeights(weight_by_d_0_1);

The weight converges to near zero after 2 data point, which means the past 2 days have much more weight on the current day (more memories). For example, for d = 0, all weights are 0 except for 𝜔0 = 1 (the 1st blue line on the top). That is the case where the differentiated series coincides with the original one. For d = 1, all weights are 0 except for 𝜔0 = 1 and 𝜔1 =−1(the last yellow line at the bottom). That is the standard first-order integer differentiation, which is used to derive log price returns.

Anywhere in between these two cases, all weights after 𝜔0 (𝜔0 = 1) are negative and greater than −1. Here, instead of calculating the return as the difference of today’s price and yesterday’s price(log return), we calculate the return as the difference of today’s price and all the days before today’s price.

The fraction d doesn’t have to be in the range of [0,1]. We can try different like range [1, 2].

weight_by_d_1_2 = weight_by_d([1,2])

weight_by_d_1_2

plotWeights(weight_by_d_1_2, v=3);

For d > 1, we observe d ∈ [1, 2], 𝜔1∈ [-1, -2], 𝜔2∈ [0, 1] and 𝜔k> 0, ∀k ≥ 2 (I guess this sentence is quit straight forward, no need to explain).

Let’s move to the next paragraph. This convergence part confused me again.

Let’s try a few more alternated range of d to get some intuition.

First, we go to the negative regime.

weight_by_d_2_0 = weight_by_d([-2, 0])

plotWeights(weight_by_d_2_0);

Obviously, no convergence!

Let’s go back to the positive range side.

weight_by_d_2_3 = weight_by_d([2,3])

plotWeights(weight_by_d_2_3);

and [3, 4]

weight_by_d_3_4 = weight_by_d([3,4], size=8)

plotWeights(weight_by_d_3_4);

How about range [0, 10]?

weight_by_d_0_10 = weight_by_d([0,10], size=12)

plotWeights(weight_by_d_0_10);

From the above result, we can see that for k > d, if

then

and 𝜔k = 0 otherwise.

Consequently, the weights converge asymptotically to zero, as an infinite product of factors within the unit circle.

This part can be understood by this plot when d is fixed at one fractional number:

weight_by_d_06_07 = weight_by_d([0.6, 0.7], nPlots=1)

plotWeights(weight_by_d_06_07, v=1);

It is from the cyan colour dash line where k > d (d=0.6). From that point, the weights converge asymptotically to zero, as an infinite product of factors within the unit circle. The absolute value of weight is getting smaller. That is why

If

then