Add Prediction Intervals to your Forecasting Model
By: Marco Cerliani
Tags: Time Series Forecasting, Editors Pick, Data Science, Machine Learning, Prediction Interval

Everyone involved in a forecasting task starts to build a model to produce point forecasts. In most prediction applications, this is related to the need to provide forecasts that tend to anticipate the future most accurately. Making these kinds of predictions is very useful in every business scenario because, given the expected future value of some KPIs, it’s possible to plan an adequate strategy to maximize the targets.

Being accurate is not all. A model fitted (and properly validated) to produce future mean predictions is simply saying to us: ‘‘what value we can expect to see the next few days’’. Nevertheless, the latest can be a piece of great information, it may be not enough for the business to make a correct decision.

Prediction intervals come to our rescue to enrich our forecasting report. They simply say to us: “where we can expect to see the values for the next few days”. They provide an upper and lower limit, associated with the point forecast (which lies in the middle), where the true value may fall. The prediction intervals are always associated with a percentage of tolerance which grants the uncertainty magnitude of our estimations. In other words, we expect our prediction intervals to capture the true values X% of the time in the future.

In this post, we introduce a simple yet effective methodology to make our model produce prediction intervals. The main advantage is that we can retrieve prediction intervals for every regression model completely for free.

METHODOLOGY

The method we adopt to build the prediction intervals is mainly based on residual bootstrapping. Bootstrapping is a resampling technique widely used in statistics and machine learning to approximate unknown quantities. The residual bootstrapping, as the name suggests, consists of sampling with replacement from a residual distribution. The residual distribution involved is obtained as the raw difference between the targets and the predictions on our train/validation set.

For example, to obtain a 95% prediction interval for our forecasts, we have to sample with replacement from the residual distribution and extract the 0.025 and 0.975 quantiles from it. Simply adding these bootstrapped statistics to the point forecasts, we can obtain our prediction intervals.

As introduced above, the procedure is extremely simple but for this reason, it must not be undervalued. There are some hidden pitfalls that, if not properly considered, can vanish our work.

First of all, the residual distributions must be build on unseen data. Only in this case, we have the guaranty to approximate unknown behaviors and grant a reliable uncertainty interpretation for our forecasting intervals. The best practice is to make a fit through cross-validation and estimate the residual distributions on the validation folds.

Secondly, we have to choose an adequate model according to the hidden dynamics of the system. The development of a wrong model will impact the residual bootstrapped statistics, i.e. misleading prediction intervals. This latest aspect is analyzed deeper in the artificial examples below, where we try to build forecasting intervals in the presence of two different regimes.

STATIONARY DATA

Given a stationary system composed of 3 variables Y, X1, and X2; we try to predict Y one step ahead in the future together with the corresponding prediction intervals.

Simulated stationary data (image by the author)

We start making a standard feature engineering, creating some rolling features. Then we find the best parameters for a Ridge and a RandomForest with a simple grid-search. The same cross-validation strategy is used to obtain the out-of-fold residuals for the best parameter combinations on both models.

Residual statistics for RandomForest (image by the author)

Residual statistics for Ridge (image by the author)

The residual distributions follow a pseudo-normal distribution and the residual autocorrelation doesn’t show a significative pattern. These are indicators of the goodness of fit of our models.

If we want to build 95% prediction intervals for our models, we only have to compute the bootstrapped quantiles on the residual distributions. The results, on the unseen test data, confirm what we expect. Nearly 5% of the test observations fall outside the predictions bands for both models.