3 facts about time series forecasting that surprise experienced machine learning practitioners.
By: Skander Hannachi
Tags: Machine Learning, Data Science, Timeseries, Forecasting, Statistics

Time series forecasting is something of a dark horse in the field of data science: It is one of the most applied data science techniques in business, used extensively in finance, in supply chain management and in production and inventory planning, and it has a well established theoretical grounding in statistics and dynamic systems theory. Yet it retains something of an outsider status compared to more recent and popular machine learning topics such as image recognition and natural language processing, and it gets little or no treatment at all in introductory courses to data science and machine learning.

My original training is in neural networks and other machine learning methods, but I gravitated towards time series methods after my career led me to the role of demand forecasting specialist.

In recent weeks, as part of my team’s effort to expand beyond traditional time series forecasting capabilities and into a borader ML based approach to our business, I found myself having several discussions with experienced ML engineers, who were very good at ML in general, but didn’t have much experience with times series methods.

I realized from those discussions that there were several things specific to time series forecasting that the forecasting community takes for granted but are very surprising to other ML practioners and data scientists, especially when compared to the way standard ML problems are approached.

At the crux of this disconnect is that time series forecasting can be cast as a supervised learning problem, and hence the entire arsenal of ML methods — Regression, Neural Networks, Support Vector Machines, Random Forests, XGBoost, etc…. — can be thrown at it. But at the same time, time series forecasting problems have several unique quirks and idiosyncrasies that set them apart from typical approaches to supervised learning problems, which require ML engineers to rethink their approaches to building and evaluating models.

Based on my recent discussions, here are the 3 biggest surprises ML practioners come across when dealing with forecasting challenges:

You need to retrain your model every time you want to generate a new prediction:

For most ML models, you train a model, test it, retrain it if necessary until you’ve gotten satisfactory results, and then evaluate it on a hold out data set. After you’re satisfied with the model’s perfromance, you then deploy it to production. Once in production, you score new data as it comes in. Eventually after a few months, you might want to update your model if a significant amount of new training data comes in. Model training is a one time activity, or done at most at periodic intervals to maintain the model’s performance to take int account new information.

For time series models, this is not the case. Instead we have to retrain our model everytime we want to generate a new forecast. To understand why this happens condsider the following example:

We will use ARIMA models to forecast Australian quarterly beer sales (the data set is taken from Hyndman’s Forecast package in R). First we will train a model on data from 1956 to 1970 and then test it on data from 1970 to 1973. Using a seasonal ARIMA(1,1,1)(0,1,1) model we can get a reasonably good forecast (MAPE =1.94% )(Fig. 1).

Next, we will use the same model to forecast sales all the way until 1993. You can see in Fig.2 that the forecast is no longer as good: The forecast continues to follow the same pattern that it had in 1970~1973, but the pattern of the actuals has changed — the constant trend that we can see from 1956 to 1974 starts to taper off after 1974, and the seasonal variations start to vary in amplitude as well. If you use the same ARIMA(1,1,1)(0,1,1) to forecast the sales for 1990 to 1993, you will get worse accuracy (MAPE = 44.92%).

Instead we would have to refit a second model that takes into account the new data and the changes in the pattern of the sales. This time, we will train an ARIMA(1,1,1)(1,1,2) model on the data from 1956 to 1990 and use it to forecast the values for 1990~1993— then we get a more reasonable MAPE value closer to that obtained in our first experiment (MAPE=5.22%).

To get an intuitive understanding of why this happens, first consider a classic ML task: Classifying cat images. The visual properties of cats are stable over time (unless we start looking at evolutionary time scales), so when we train a neural network to recognize pictures of cats, an implicit assumption is that the features that define cats are going to remain the same for the foreseeable future. We don’t expect cats to look different next week, or next year, or even ten years from now. Given enough data, the model we trained this week is good enough for the foreseeable future as well.

In statistical parlance, we say that the distribution of cat picture features is a stationary distribution, meaning that its properties such as its mean and standard deviation remain the same over time.

Now recall that a common pitfall in ML projects happens when the distribution of the development data set and the distribution of the production data set are not the same, causing the model to fail in production. Well for time series, it is almost always the case that the development data set and the production data set are not from the same distribution, because real world business time series (such as the Australian beer sales) data are not stationary, and the statistical properties of your distribution will keep shifting as new actuals come in.

The only way around this is to retrain your model every time you get new data. Note that this is not the same as continuous learning, where an already trained model is updated as new data comes in. You are actually retraining a new model from scratch everytime you want to generate a new forecast (although it would be an interesting research topic to see if continuous learning can be applied to time series forecasting).

From a practical point of view, this means that deploying forecasting algorithms to production is very different from deploying other ML models. You can’t just deploy a static model and score it, the concept of model serving doesn’t make sense for time series forecasting. Instead, you need to ensure that training and model selection can be done on the fly in production, and you have to insure that your entire training set can be stored and processed in production. Which brings me to our second point of the blog post:

Sometimes, you have to do away with train/test splits:

Let’s circle back to the basic approach to finding an ML model: usually you build a model using a train set and then evaluate it on a test set. This requires that you have enough data to set aside a test set and still have data to build a model with. But time series data is often very small compared to the data sets used in image processing or NLP. Two years of weekly sales data for a product at a given location is only 104 data points (barely enough to capture any seasonality). A Quarterly economic indicator data, taken over 10 years is only 40 data points. With data sets as small as this, we don’t have the luxury of setting aside 20% or 30% of the data for testing purposes. Nor is cross validation much help, since, depending on the algorithm you want to use, CV is at best tricky to set up for time series models, and at worst not applicable at all (Think about it: It doesn’t make sense to try to forecast values for February using training data from January and March — that would amount to leakage).

So instead, we resort to using information criteria such as the AIC, the AICc or the BIC. These are model selection metrics, which essentially try to approximate the test step analytically. The idea is that we don’t have an empirical way of determining the generalization error of a model, but we can estimate this error by using information theoretical considerations.

In practice, we train a predetermined set of models and then select the one that has the lowest AIC or BIC. Besides allowing for training models with limited data, using such criteria for model selection is very convenient when we want to automate forecast generation. For example in retail it is not unusual that we have to generate forecasts for millions of individual time series — a large retrailer will carry 20K ~ 30K products in several hundered locations, resulting in millions individual times series (one for each prodcut/location combination).

In a situation like this, each individual data set is small, but you have to deal with millions of data sets, so it is impossible for an analyst or an engineer to perform evalutions on a test set for each series, and automatic model selection becomes essential.

Many forecasting tools use this approach: Rob Hyndman’s popular Forecast package in R (1) uses the AIC for model selection in its auto.arima() and ets() functions, and many commercial demand forecasting applications, such as Oracle’s RDF, use the BIC for model selection. See Hastie and Tibshirani (2) for a introduction on the use of the AIC and the BIC for model selection.

The uncertainty of the forecast is just as important as, or even more so, than the forecast itself:

One more thing that distinguishes forecasting from other supervised learning tasks is that your forecasts are almost always going to be wrong. Somebody working with an image classification problem or an NLP problem can reasonably expect to eventually classify all new incoming examples accurately — given enough training data. All you have to do is make sure that your training data and the real world data are sampled from the same distribution. As mentioned in my first and second points, this is usually not the case in business forecasting applications, and your forecasts are almost always going to be wrong. What are the chances that you are going to predict exactly how many size M red Adidas shirts you are going to sell next week? So you always need not just a point forecast, but also a measure of the uncertainty of your forecast.

In demand forecasting and inventory applications, the uncertainty of your forecast is crucial for the applications that consume the forecast. The uncertainty of your forecast (represented by forecast intervals or by forecast quantiles) is what you will use to calculate your safety stock, that is the additional amount of inventory you want to carry to make sure you don’t lose any customers.

They take this idea a step further in Financial Time series modelling, where they actually have classes of models built explicitly for modeling the uncertainty of a time series, as opposed to the time series itself, such as ARCH and GARCH models.

Maridakis et al. (3) speak of subway uncertainty and coconut uncertainty. Coconut uncertainty, an allusion to a coconut unexpectedly falling on someone’s head while they’re on the beach, is the uncetainty of “unknown unknowns”, events that could never have been predicted no matter how hard we tried.

More generally, some time series exhibit ill-behaved uncertainty that is unpredictable. The distributions of forecast errors doesn’t follow any known distribution. Such information is useful for making judgmental decisions, but cannot be modeled and used for forecasting.

On the other hand, subway uncertainty (an allusion to the uncertainty of the how much time it will take to get from A to B using the subway) follows a known distribution such as the normal distribution or the Poisson distribution, and can be modeled and used to bound predictions within a certain range, even if exact values can never be predicted.

Those used to the very high accuracies achievable in other ML domains, and more importantly decision makers and business leaders caught up in the current machine learning hype, need to understand that often times the best we can achieve with a time series model is subway uncertainty. The persistent belief that if we could just find the right algorithm and the right data, a deep enough neural network or a rich enough unstructured data set, we will be able to achieve better forecasts is dangerously misleading. Sometimes the best we can get is manageable uncertainty — and we should build our data pipelines and decision support systems accordingly.

References:

(1) https://cran.r-project.org/web/packages/forecast/forecast.pdf

(2) Hastie, Tibshirani, Friedman, Elements of Statistical Learning, Chapter 7 (pages 228–229)

(3) Makridakis, S., Hogarth, R.M. & Gaba, A. (2009) Forecasting and uncertainty in the economic and business world. International Journal of Forecasting