Using Decision Trees, Random Forest and Gradient Boosting for Time Series Prediction
By: Prerana Jakhotia
Tags: Decision Tree, Time Series Forecasting, Boosting, Bagging, Machine Learning

A time series is a series of data points indexed in time order. A time series has can be decomposed into three parts — trend (the long term direction) seasonality (calendar related movements) and error (unsystematic fluctuations).

Decomposition of time series

There are often other external features that affect the time series. For example, if we look at the sales of clothes, we can see a spike in the volume if there is a promotion going on. Predicting future values of a time series can be one of the most important steps for a business. For example, having an accurate forecast of demand for a product can help a business plan its supply chain better.

There are different statistical forecasting techniques like Moving Averages, Exponential Smoothing, Holt-Winter, ARIMA etc that we use for time series prediction. However, in this blog, I will cover how we can use machine learning techniques like decision trees and other ensemble methods for the same.

Decision Trees: All of us have used a decision tree at some point to make a decision. It is a tree-like structure where at every node we make a decision and continue doing it till we reach a conclusion.

Fig 1. Example of a decision tree (The image is taken from web)

A decision tree uses a training set of different predictors and target. The core algorithm for building decision trees is called ID3. The ID3 algorithm can be used to construct a decision tree for regression by replacing Information Gain with Standard Deviation Reduction.

Ensemble methods is a machine learning technique that combines several base models in order to produce one optimal predictive model. There are different types of ensemble methods —

Bagging: Decision trees are sensitive to the specific data on which they are trained. If the training data is changed (e.g. a tree is trained on a subset of the training data) the resulting decision tree can be quite different and in turn, the predictions can be quite different. To overcome this problem, we create N learners(decision trees) and produce N new training data sets by random sampling with replacement from the original set. The final prediction is the average of prediction from N decision trees. Random Forest is also a bagging method with an extra step. Along with creating N learners that use a randomly sampled training data set, it also does a random selection of features rather than using all features to grow trees.

Fig 2. Bagging for decision trees (The image is taken from web)

2. Boosting: We create N learners(decision trees) and produce N new training data sets by random sampling with replacement from the original set. However, in boosting we training the individual models in a sequential way. Each individual model learns from mistakes made by the previous model. AdaBoost, Gradient Boost etc are different types of boosting methods.

Fig 3. Boosting for decision trees (The image is taken from web)

Using decision trees and ensemble methods for time series prediction -

The goal of the problem is to accurately predict the energy consumption of a house in order to effectively direct power. The energy usage was found to be a highly nonlinear function of the data, requiring a nonlinear prediction method.

Minute by minute household data aggregated at an hourly level as the target variable was combined along with features like external historical weather data to prepare the training data.

Different models like linear regression, decision trees, random forest and gradient boosting in R along with hyperparameter optimization were used to predict the energy usage.

Fig 4. Actual Vs Predicted for Random Forest

We can clearly see from the graph above that a random forest model is able to predict the energy usage with great accuracy even if the energy usage signal has a lot of variability in it.