ML time series forecasting the right way
By: Mario Dagrada
Tags: Machine Learning, Time Series Forecasting, Energy

In the top left figure above we see the original behavior over time of the energy consumption. The p-value of the augmented Dick-Fuller (ADF) test, shown in the figure title, is a good indication that the time series is stationary.

The top right panel shows the value distribution of energy consumption. The data values seem to follow a slow-decaying Poisson distribution where high values of the energy consumption above 50 KWh are much less likely than lower ones (10 to 40 KWh). This is expected behavior since the energy consumption of industrial utilities operating continuously has usually a fairly constant average with sporadic peaks due to production increase or maintenance windows.

The bottom figures show the autocorrelation (ACF) and partial autocorrelation (PACF) functions. See here for more details on their importance for time series analysis. The ACF shows that the time series has a seasonal component at 24 hours (grey horizontal lines) with autocorrelation peaks which do not decrease over time, indicating the strength of this component. This is confirmed by the PACF.

Machine learning forecast with XGBoost

For our forecasting problem, we choose the XGBoost algorithm using this popular Python implementation. XGBoost is fast and accurate compared to other tree-based ML methods for time series problems, as shown by several Kaggle competitions and other works available online (see for instance here or here).

In order to transform a time series forecasting task into a supervised machine learning problem, we need to generate the features. The actual time series values are instead used as a target for the ML model. The types of features we can consider for a time series are divided into 3 categories:

lag features: they use the original time series itself as a feature with a certain shift usually called lag. Lags can be chosen automatically looking at the values of the partial autocorrelation function. In particular, we take as features only the lags where the PACF is greater than 0.2, equivalent to a 5% relevance for the lag. Here the Python snippet to create these features.

standard time-series features : standard time-series features such as hour, weekend, season, and so on. Using some domain knowledge on the data we can also build additional time-series features such as, in our case, the current worker shift per each timestamp.

: standard time-series features such as hour, weekend, season, and so on. Using some domain knowledge on the data we can also build additional time-series features such as, in our case, the current worker shift per each timestamp. exogenous features: exogenous features are themselves time series and are external, i.e. they do not depend on the time series to be forecast. In our case, an exogenous variable could be the external temperature. In this post, we do not consider them.

We use a standard pipeline for training and testing the XGBoost model with 80/20 train test split, cross-validation with rolling time window, and hyperparameters optimization using the hyperopt library. The best model yields a mean average percentage error (MAPE) of 14.302 from cross-validation and 8.378 when predicting on the much smaller test set. The predicted values on the test set and corresponding importance coefficients for the top features are shown below.

XGBoost model predictions on the test with corresponding feature importance histogram.

Multi-step ahead forecasting with ML models

The ML model results are acceptable and our job is done. Not quite! We mentioned in the introduction that getting satisfactory results from the model is not enough to obtain an actionable outcome. Similarly to standard supervised ML problems, one wants to use the model on unknown data points which in our case are future time periods: ultimately we want to predict the future. The number of time periods to forecast into the future is usually referred to as the forecasting horizon.

At variance with standard ML where the model can be directly applied to new data, the temporal structure of time series problems makes forecasting more complicated. There are several strategies to achieve multi-step ahead forecasting, a good introduction to these methods can be found here. In this post, we compare two of the most widely used strategies showing also their implementation for the ML model discussed above. We choose a forecasting horizon of 48 time periods which corresponds to 2 days ahead forecasting.

Recursive strategy

The recursive forecasting strategy uses only a single model pre-trained for one-step-ahead forecasting. At each forecasting step, this model is used to predict one-step ahead and the value obtained from the forecasting is then fed into the same model to predict the following step (similarly to a recursive function, hence the name) and so on and so forth until the desired forecasting horizon is reached. The recursive strategy is the least expensive method for forecasting since it uses the same model. However, it is not the best strategy for long time horizons since it accumulates forecasting errors at each step. This problem arises because previous forecast values are used to build the feature space for the current step.

The following snippet implements the recursive strategy for our ML model.

Direct strategy

The direct forecasting strategy uses a different ML model for each forecasting step. More specifically, each model is trained using as target the time series shifted of the desired number of time periods into the future. Imagine for example that one wants to train a 4 steps ahead model. In this case, each timestamp in the target time series is chosen 4 steps ahead with respect to the corresponding timestamp in the feature set. In this way, we create a model trained to predict 4 steps ahead into the future. The same procedure is repeated for all forecasting steps. The size of the training/cross-validation set slightly decreases for each additional step ahead to perform.

The direct method outlined above does not generate error accumulation at each forecast, but it has larger computational cost making it not suitable for large forecasting horizons. Moreover, it cannot model statistical relationships among predictions since the models used for each time step are independent. As a side remark, the latter issue can be overcome by using a neural network trained to output multiple forecasts at once — one for each future time period — which also learns the relationships in the process.

Here the implementation of a simple direct forecast for our ML model.

Forecasting Results

And finally, let us look at how the two strategies above compare in forecasting the future. We consider the last 48 hours of our dataset as the future so that we have real data to compare the forecasting with. Beware that in real-world problems the future is not known in advance!

By looking at the MAPE score for the two methods, the direct one performs better than the recursive strategy as expected, with an error of around 15% lower.

Just looking at these values is not enough to assess forecasting quality. Let us plot the two forecasts.

For this relatively short horizon, both strategies perform similarly for the initial forecasting steps. After around 24 hours, however, the recursive strategy starts to diverge from the direct one. This is most likely due to the error accumulation starting to affect recursive method results. Error accumulation likely generates also the stronger fluctuations displayed by the recursive forecast. The improvement yielded by the direct strategy is however not so significant to always justify its use. In production deployments of time series models, it is often better to use a less accurate method requiring much lower computational effort such as the recursive strategy.

In conclusion, remember that the present methods are just basic forecasting strategies. Extensive research effort has been put into devising more precise and controlled forecasting techniques, one example being the Rectify strategy described here.

If you enjoyed this post and you have any comments, suggestions, criticism just connect with us (Lorenzo or Mario). The full code used for this post is available on Github.