On the Automation of Time Series Forecasting Models: Technical and Organizational Considerations.
By: Skander Hannachi
Tags: Demand Forecasting, Data Science, Deep Learning, Machine Learning, Time Series Forecasting

In this post I will go over the technical aspects of automatic forecast generation, as well as some of the organizational considerations that will arise when deciding to go with an automatic forecast generating system.

Automated times series forecasting as an exercise in model selection:

As I said earlier, in many fields, including my field of retail demand forecasting, most commercial forecasting packages do perform automatic forecast generation. Several open source packages do so as well, most notably Rob Hyndman’s auto.arima() (ARIMA) and ETS() (exponential smoothing) functions from the open source Forecast package in R. There’s also a Python implementation of auto.arima called Pyramid, and several other python packages are in the works.

Both the commercial products and the open source packages that I mentioned work based on the idea of using information criteria to choose the best forecasting model: You fit a bunch of models, and then select the model with the lowest AIC, BIC, AICc, etc….(typically this is done in lieu of out of sample validation — see this presentation for details).

An example of the AIC and BIC selecting the “true” model of our data: We generate a data set using a 6th order polynomial with Gaussian noise, we then notice that both the AIC and BIC are at there lowest for the polynomial regression that is of order 6, i.e. they allowed to select the true model, even if polynomials of order 8 and 9 gives us lower RMSE.

There is however a major caveat: all of these methods work within a single family of models. They choose the best possible model amongst a set of ARIMA models, or the best possible model amongst a set of exponential smoothing models, etc…

It is much more challenging to do so if you want to choose from different families of models, for example if you want to choose the best model from ARIMA, Exponential smoothing and the Theta method. In theory, you can do so in the same way that you do within a single family of models, i.e. by using information criteria. However in practice, you need to calculate the AIC or BIC in exactly the same way for all models considered, and that is a significant challenge. It might be better to use time series cross-validation, or out of sample validation, instead of information criteria, but that will be much more computationally intensive (and tedious to code), not to mention the question of which suitable forecast horizon to cross-validate against.

The “One model to rule them all” approach:

Facebook’s Prophet package also makes it possible to automate forecast generation based on General Additive Models (See here for details). However Prophet fits only one single model, albeit a very flexible model with many parameters. Prophet’s implicit assumption is that a GAM is the “One model to rule them all”, which might not be theoretically justified but is very pragmatic and useful for real world scenarios. More specifically, the basic assumption underlying the prophet model is that most useful real world time series do not contain any structure beyond trend, seasonality, and causal/holiday effects, and we don’t gain much by trying to mine them for complex auto-correlations the way a non-trivial ARIMA model does. Besides making it easy to strike a balance between automation and flexibility, the trend + seasonality + causal effects approach to Prophet makes it very convenient for communicating results to non-technical or scientific people (If you are a student or a beginner in the field, brace yourself: Haggling with business over the validity of your forecasts will consume a significant portion of your day. Go get your soft skills brushed up while you are at it).

The Facebook Prophet API makes it very easy to communicate the results of a forecast to business stake holders, compared to say, explaining what and ARIMA(2,1,2) model does.

One might ask: But isn’t that what Triple Exponential Smoothing (Holt-Winters) does? Fit a trend component and a seasonal component and leave it at that? Are we just trying to use a Facebook API for hype purposes? Not quite. Prophet is somewhat more sophisticated than Holt-Winters in that regard: It can model break points in the trend, and multiple seasonalities (e.g. for example daily and monthly peaks in a cusotmer time series). It can also handle causal events very well, which HW can’t (You would have to add the causal effects as some sort or post-processing step). Part of Prophets felxibility comes from the fact that it is not “just” a GAM model. Under the hood, there is a lot of Bayesian heavy lifting going on, including in the causal modeling part.

The No Free Lunch Theorem, the time series forecasting edition (sort of):

Presumably, one of the main reasons you want to do automated time series forecasting, is because you want to forecast multiple time series, too many to analyze manually (The tool my team uses generates millions of forecasts daily — one for each of our product/location combinations). Therefore your automated forecasting procedure must be able to fit to different types of time series with different business scenarios. In the general ML case, there is a theoretical result called the No Free Lunch Theorem: There is no such thing as a “supervised ML model to rule them all”, one that can generate the best out of sample predictions without having to make any assumptions about the structure of the data set. So if you’re going to throw a generic ML model, say a feed forward neural network, at your problem space, and hope that it will work for every possible data configuration that can occur, you will also have to accept that sometimes there will be a model that performs better than the one you ended up with. It is the price you pay for having one generic model for everything, hence “no free lunch”.

Something similar happens with time series models: You need to keep in mind that an automated forecasting approach is never going to find the absolute best model for each and every single time series — it is going to give a reasonably good model on average over all the time series, but it is still possible that some of those time series could have better models than the ones selected by the automated method.

See this post for an example of this. To put it simply, if you are going to go with automated forecasting — you will have to occasionally tolerate “good enough” forecasts instead of the best possible forecasts for each time series. That is the price you pay for flexibility and robustness.

In this case, manually selecting higher order ARIMA parameters — i.e with higher AIC and BIC — gives better out of sample forecasts than the one automatically by auto.arima(), but the one selected using the lowest AIC and BIC is still pretty close to the ground truth, compared to, say, a non-seasonal model (seasonality and trend were not passed in any way to auto.arima(), it figured that out on its own).

ML Based forecasting - things get a lot more complicated, yet paradoxically simplified:

As long as you stick to statistical methods for time series, automatic forecasting remains, if not an easy problem, at least an approachable one.The problem amounts to a statistical model selection question (Or a curve fitting exercise in the case of FB Prophet), which has some solid theoretical foundations and is discussed and explored in several graduate level text books.

If you are planning on using ML based forecasting models, then the issue becomes a case of the more general problem of auto-ML (automated machine learning). Both the academic literature and the technology are not as mature when it comes to the question of hyper-parameter tuning and auto-ML as it applies to time series forecasting (as always, NLP and Computer vision get all the spotlight and attention first…). There are some interesting auto-ML and Bayesian Optimization tools which can be used for the task, as well as a couple of commercial products (e.g. Google Vizier), but there are still many open questions regarding Bayesian Optimization and Transfer Learning in the time series case.