Forecasting at scale with Facebook Prophet — a case study on its merits & limitations
By: Konrad Hoppe
Tags: Fbprophet, Machine Learning, Time Series Forecasting, Statistics

Forecasting at scale with Facebook Prophet — a case study on its merits & limitations

Background

Time series forecasting is usually a complex task because the structure of already univariate data often contains many unobserved factors. Standard models such as ARIMA, or filters, e.g. Kalman Filter are complex models that often need tweaking which requires a rigorous understanding of the underlying theory.

Practitioners with good domain knowledge but little statistics know-how want to make use of machine learning and forecasting methodologies to inform their business decisions. So a number of software packages and libraries attempt to bridge this gap by offering automated solutions.

The aim of this article is to investigate a promising library by Facebook, called Prophet. The promise of Prophet is to generate forecasts automatically at scale.

Results

Unfortunately, the hopes that an engine can do reliable forecasts at scale without any interaction were too high, even under laboratory circumstances. After some tweaking, however, the library performs well and recognizes all the elements that were put into the test data. So with having a little knowledge on the domain, the library produces useful results

The case

To set things up, load the forecasting library, ggplot for pretty graphics and zoo for some time series magic.

Now generate the data. We will use 5 years of data with three different seasonal patterns and a changing trend after the first half of the time horizon.

Trigonometric functions can be used to generate seasonal patterns by recognizing that sin(2π/b) has a cycle-length of b. In other words, every b periods, the pattern repeats itself. We will use a yearly pattern (b=360), a monthly cycle (b=30), and a weekly cycle (b=7).

The trend will be lifted after half of the observation horizon to test whether the FB Prophet can figure this change of structure out. We will see, that it indeed does well on this challenge.

Other features that the prophet can capture are trend saturation and special events, such as holidays. But these will not be addressed here.

The model will be trained on the first four years of data. The fifth year of data is used to assess the performance of the model.

At first, we run Prophet without further information:

To create a forecast from it, a data frame that will hold the forecast needs to be invoked:

Fit of data and forecast (Image by author)

From the plot, it’s easy to grasp that the fit is improvable. The peaks of the seasons are not well captured. To get a better understanding of what is going on, let us look at the residuals:

Residual plot (Image by author)

The range of residuals is rather wide, and a clear structure is visible. So there is a component missing in the model. Let us try to specify the yearly seasonality explicitly:

Fit of data and forecast (Image by author)

Looks better, but still not perfect. Keep in mind,that the model data is produced without noise, so the model should be able to fit the data with almost 100% accuracy.

However, the residuals show a different issue this time:

Residual plot (Image by author)

This is a prime example why domain knowledge is important. Since we know the frequencies of the underlying data, we can specify it with the parameter period.

To do that, we set up the model without a seasonal component and add them manually:

Residual plot (Image by author)

This looks better, apart from the break that occurs around the trend change. However, there is still a structure in the residuals. The problem here is overfitting, which is induced by the parameter fourier.order. The standard value of this parameter is 10.

Prophet estimates the seasonal component using a Fourier transform. The underlying idea is that any arbitrary signal can be approximated with a sufficient number of sin curves. The Fourier order parameter specifies how many of them should be used here. Since we have created each component only with a single sin curve, the true order is 1.

So let us see what happens when we inform the Prophet about this aspect of the model:

Residual plot

The residuals look much better now, apart from the disturbance when the trend changes, but this is not surprising as it poses a sudden break for the nice regular pattern of the data.

Let us now look at the out-of-sample forecast to see whether the good fit of the training data set translates into a good forecast:

Real data (grey) and fit (blue) // (Image by author)

The forecast is now perfectly in line with the real data!

Take aways

Automated forecasting is a convenient feature for large scale business applications. However, these methods do not always work out of the box. We had to weave in a great amount of knowledge from the data generation process to create a good model fit.

Without these insights, the model was still useful, but nevertheless not exact, although the data generation process follows exactly the same logic of the data exploration method.