Approaching Time-Series with a Tree-based Model | by Agnis Liukis | Towards Data Science

In this article, I’m going to show a generic way how to approach a time-series prediction problem with a machine learning model. I’m using a tree-based model LightGBM as an example here, but practically it could be almost any classical machine learning model, including linear regression or some other. I’ll cover needed data transformation, basic feature engineering and, modeling itself.
Time series is basically a list of values at known points in time. Some examples: daily purchases at a store, the daily number of website visitors, hourly temperature measurements, etc. Often the interval between data points is fixed, but that is not a mandatory requirement. And the problem we want to solve in a time series is to predict the value at points in the future.
There are many specialized ways how to approach this problem. Exponential smoothing, special models like ARIMA, Prophet, and of course Neural Networks (RNN, LSTM), just to name some. But in this article, I will focus on using generic tree-based models for time series forecasting.
Tree models out of the box don’t support forecasting times series on raw data. Therefore data must be first pre-processed in a special way and corresponding features have to be generated.
Typical raw time series data looks as follows.
There is one value for each time series step. In this particular example, this step is a day, but it could be an hour, a month, or any other time interval. We have some known training data from the past (blue color in the graph above). And we have unknown data in the future we want to forecast (yellow color in the example).
For tree-based models we need many training samples with target values, but here there’s only one long line with time-series data. In order to make it usable, let’s transform it by going through all the values and taking each of them as a target for one sample and prior data points as training data as in the visualization below.
It is very important not to include future data points as training data, otherwise there will be data leak and model will perform poorly on future data. Sample Python code for the reference on how to do it:
SIZE = 60 #we take 2 past months here for each time-series pointCOLUMNS = ['t{}'.format(x) for x in range(SIZE)] + ['target']df_train = []for i in range(SIZE, df.shape[0]):    df_train.append(df.loc[i-SIZE:i, 'value'].tolist())df_train = pd.DataFrame(df_train, columns=COLUMNS)
With this, we transform time series data line with length N into a data frame (table) with (N-M) rows and M columns. Where M is our chosen length of past data points to use for each training sample (60 points = 2 months in the example above). Data table now looks as follows:
Basically now it is already in a format any classical model can take as an input and produce forecasts for the target variable (next data point of time series). However, on raw data like this tree-based models won’t be able to achieve good results. It is necessary to make special features, allowing tree model to capture different components of time series like trend and seasonality.
To better understand, what features we need, let’s remember the main components of time series data:
base level: this is the mean value of time-series data
trend: this is the change of base-level over time
seasonality: these are cyclic changes repeating over some time period (week, month, year)
remainder: this is everything else not covered by 3 other components. The remainder may consist of the predictable part (when changes are triggered by some external irregular event which we can observe and use in our model) and non-predictable part (basically just noise or changes triggered by events we don’t have access to).
Let’s further try to capture the first three time-series components in features usable for our tree-based model.
We will create features in a window-based approach, looking at N successive data points at a time. First simplest case is for window size 1. This simply means that we are taking the last value or our available training data for each sample, which is data point right before target. There’s not much we can do with just one value, so let’s add it as a feature as-is. This is a good feature as this is the most recent known data point. Drawback is that this feature will be very noisy.
df_feats['prev_1'] = df_train.iloc[:,-2] #Here -2 as -1 is a target
Next we take window size 2, having 2 most recent data points before target. Then window size 3 and so on. Example code snippet:
for win in [2, 3, 5, 7, 10, 14, 21, 28, 56]:    tmp = df_train.iloc[:,-1-win:-1]    #General statistics for base level    df_feats['mean_prev_{}'.format(win)] = tmp.mean(axis=1)    df_feats['median_prev_{}'.format(win)] = tmp.median(axis=1)    df_feats['min_prev_{}'.format(win)] = tmp.min(axis=1)    df_feats['max_prev_{}'.format(win)] = tmp.max(axis=1)    df_feats['std_prev_{}'.format(win)] = tmp.std(axis=1)    #Capturing trend    df_feats['mean_ewm_prev_{}'.format(win)] =             tmp.T.ewm(com=9.5).mean().T.mean(axis=1)    df_feats['last_ewm_prev_{}'.format(win)] =            tmp.T.ewm(com=9.5).mean().T.iloc[:,-1]        df_feats['avg_diff_{}'.format(win)] =            (tmp - tmp.shift(1, axis=1)).mean(axis=1)    df_feats['avg_div_{}'.format(win)] =            (tmp / tmp.shift(1, axis=1)).mean(axis=1)
Let’s go through the code. The first line defines window sizes used. Next for each window size, we create temporary variable tmp, with selected data points in the current window. This variable is later used to create all the features.
General statistics for base level
The first 5 features are general statistics — mean, median, min, max, and std. This kind of feature will help to determine the base level of time-series for different window sizes. Depending on the characteristics of time-series, some window sizes may be more important. For example:
for noisy data bigger intervals are better, as they won’t be influenced by the noise so much;
for data with sharp trend, smaller intervals may be better, as they will capture the latest trend better.
Not always it is obvious, which window sizes to use, so it is smart to include several of them and let the model to choose itself.
Capturing trend
The next two lines define Exponentially Weighted Mean (EWM) features. Weighted mean differs from simple mean by giving bigger weights to more recent data points allowing to capture trend better.
And the last two features will also help in capturing trend — they compute the average difference of two following data points. This allows evaluating how fast data is changing and in what direction.
As with previous features group, the window size is important for trend features, too. Small windows will be able to capture fast-changing short term trends. However, there is a risk of capturing noise.
There are many other features which could be helpful in predicting trend. Some ideas to mention:
computing weighted mean of the difference between data points (giving more impact on trend to more recent data points);
taking min, max, std, and other statistics on differences between points.
computing slope of each two following data points and take the average or other statistics on them;
many more.
Features for seasonality
So far we generated features only for the first two components of time series — base value and trend. What about seasonality? There are special features to use for this component, too. Consider the following code example:
for win in [2, 3, 4, 8]:    tmp = df_train.iloc[:,-1-win*7:-1:7] #7 for week    #Features for weekly seasonality    df_feats[‘week_mean_prev_{}’.format(win)] = tmp.mean(axis=1)    df_feats[‘week_median_prev_{}’.format(win)] = tmp.median(axis=1)    df_feats[‘week_min_prev_{}’.format(win)] = tmp.min(axis=1)    df_feats[‘week_max_prev_{}’.format(win)] = tmp.max(axis=1)    df_feats[‘week_std_prev_{}’.format(win)] = tmp.std(axis=1)
Notice tmp variable in this example takes only each 7th value, therefore it contains data only for the same weekdays as the target variable. This way we are calculating statistics features specific for given weekday. If given time-series has strong weekly seasonality component, such features will be very important.
We now have prepared basic features, ready to be used in a tree-based machine learning model. LightGBM model will be used as an example and code for modeling is provided below for the reference:
import lightgbm as lgbparams = {    'objective': 'regression',    'metric': 'mae',    'boosting': 'gbdt',    'learning_rate': 0.06,    'num_leaves': 64,    'bagging_fraction': 0.9,    'feature_fraction': 0.9}x_train = lgb.Dataset(df_feats, df_train['target'])model = lgb.train(params, x_train, num_boost_round=500)preds = model.predict(df_feats_validation)
First we import LightGBM model and prepare couple of it’s most important parameters. Then model is trained on previously prepared features from df_feats variable. Similarly features for validation period are prepared in df_feats_validation variable, which is used for obtaining predictions in preds variable. We can now plot obtained predictions to see, if it works.
Predictions are far from perfect, as we used only very basic features, but still, we can notice that predictions follow the trend of real data. With more similar features and model parameters tuning, results can be improved further.
In this article, I showed a generic approach on how to transform time-series data and gave example features calculated for the main time-series components. Tree-based models are not the first choice for achieving state-of-the-art accuracy on time-series data, however, they have many other good characteristics, like interpretability, simplicity, which often dictates to choose exactly these models for real time-series predictions.
Thanks for reading and stay tuned for more articles about Data Science and Machine Learning.