A Gentle Introduction to Backtesting for Evaluating the Prophet Forecasting Models
By: Kan Nishida
Tags: Machine Learning, Time Series Analysis, Statistics, Forecasting, Data Science

The Prophet time series forecasting algorithm is amazing, it has definitely democratized the time series forecasting. It’s easier to start with and easier to improve the model quality if you know more about your subject.

If you’re not familiar with the Prophet, take a look at this introduction post.

But wait, what do you mean by ‘model quality?’

How can we know if a given forecasting model is good or bad?

How should we go about evaluating the model?

There are a few ways to test the forecasting models, but in this post, I want to introduce one of the most commonly used methods called Backtesting as a way to evaluate the forecasting model with metrics like RMSE, MAE, MAPE, etc.

I’m going to use the Prophet as an example in this post, but this is not specific to the Prophet, it can be used for other time series forecasting models.

Anyway, let’s start.

What is Backtesting?

The concept of the Backtesting is actually pretty simple.

It is to keep the later part of the data as Test data so that we can evaluate the model against this Test data.

Let’s take a look at how it works step by step.

Split the data into Training data and Test data

Before building the model, we separate the data into two parts, one is Training data and another is Test data.

2. Build a model based on the training data

Then, we can build the forecasting model based on the training data with the Prophet.

3. Use the model to forecast for the Test data period

Once the model is built, then we can use the model to forecast for the test data period.

4. Compare between Forecasted and Actual

For this Test period, we have answers, meaning that we have the actual data. Therefore, we can compare the forecasted values against this actual data and see how big or small the differences between the forecasted and the actual.

Example Use Case: Forecasting Store Orders

Now, let’s take a look at by running the Prophet in Exploratory.

Here, I have sales data. Each row represents an order at this e-commerce store.

Here is a line chart that visualizes the trend of the orders by week.

Now, let’s say we want to forecast the orders for the next 52 weeks (about 1 year).

We can go to the Analytics view in Exploratory and select Time Series Forecasting (Prophet).

Second, select Date column and set the time scale to Weekly.

Then, set the forecasting period to 52 weeks in the Property.

Finally, run the Prophet. We’ll get the result like below.

The orange line is for the forecasted values and the blue line is for the actual values.

After Jan 2016, there is only the orange line, this is the period that we don’t have the actual data and the model, which was built based on the blue line data, is forecasting the values.

Take a look at the below introductory post for more details on the Prophet and how to use it in Exploratory.

Evaluate Prophet Forecasting Model

Now, let’s evaluate the quality of this forecasting model.

We can open the property dialog and set ‘Test Mode’ to TRUE.

It will use the forecasting period as the length of the test period. We have set to 52 weeks above, which means that it will use the last 52 weeks of this data as the Test period.

When we re-run it with Test mode enabled, we get a slightly different chart like below.

You would notice that the actual data is split into two, Training period as dark blue and Test period as light blue.

Again, the orange line is showing the forecasted values.

But, the actual values are split into two lines. The dark blue is the actual values in the Training period, and the light blue is the actual values in the Test period.

The forecasted values in the Training period are very close to the actual data (dark blue). This is because the forecasting model was built based on this actual data! ;)

On the other hand, the forecasted values in the Test period seem to be farther from the actual values (light blue). Note that this is the period that the forecasting model doesn’t know the answers.

In the Test period, we can see that there was a big drop in February and big increases compared to the forecasted line.

The forecasted values tend to be further away from the actual values in the Test period. This is because the forecasting model assumes that whatever the trend and seasonality found in the Training data will be repeated in the future.

But hey, the history doesn’t repeat itself, but it rhymes, right? ;)

Anyway, the real question here is, how big or small the differences are.

There are several metrics that can be used to evaluate how different between the actual values and the forecasted values, or how different between the orange line and the light blue line above.

Here are the three metrics that are used widely when evaluating the time series forecasting models.

RMSE (Root Mean Square Error)

MAE (Mean Absolute Error)

MAPE (Mean Absolute Percentage Error)

Let’s take a look at one by one.

RMSE (Root Mean Square Error)

This RMSE is the same RMSE that is used for evaluating prediction models that predict numeric values such as Linear Regression, Random Forest, etc.

It calculates the difference between the forecasted and the actual (Error), squares them (Square), takes the average (Mean), then roots it back (Root).

Wow, that’s a lot of not-so-straightforward lingos right there!

No worry, here is how it works step by step.

Here’s an example of the forecasting data. The blue line is the actual and the orange is the forecasted.

And we are interested in how different between the two lines.

Calculate the differences

First, we want to calculate the differences.

2. Square them

If we just calculate the differences and take an average it might end up becoming closer to zero when we have positive and negative differences between the actual and the forecasted.

So, we square them to guarantee that they are all positive first.

3. Calculate the mean (average)

Then, we can calculate the mean (average).

4. Root it

Now this average number is pretty big and it’s out of context because we squared the original difference values above.

So we want to root the number to bring it back into the original context (or unit of orders).

And this will become 2.65. If the original unit is a number of orders, then it can be considered as 2.65 orders.

And what we have done here is to calculate a sort of an average difference between the forecasted and the actual. Yes, RMSE is basically trying to tell us this, and in this case, it is saying that the forecasted values tend to be 2.65 orders different on average compared to what the real values would be.

MAE (Mean Absolute Error)

MAE is very similar to RMSE.

For RMSE, we square the original difference values, then later root the average number. The reason we did all such extra works was that we were concerned that we might have ended up getting a value close to zero when we have positive and negative numbers in differences and add them all up.

But that is not the only way to solve the problem.

Instead of squaring/rooting, we can simply absolute the difference numbers, meaning that we make any negative numbers positive.

And that is MAE (Mean Absolute Error).

For MAE, we simply absolute the difference numbers, in this example all the numbers are positive so no extra action here, then we take the average.

This is saying that the forecasted values tend to be 2.5 orders different on average compared to what the real values would be.

RMSE vs. MAE

Both trying to do the same thing and trying to address the same problem.

But which one should we use?

The calculation of RMSE involves squaring the difference numbers, this means that this metric can be signified by large differences compared to MAE.

So when you are comparing two models (or three or more) and are interested in improving the larger differences, then RMSE would give you a more reliable insight.

Problems with RMSE & MAE

Remember, these are telling us the average difference between the actual and the forecasted values in the original unit, such as sales dollars, number of orders, number of users, temperatures, etc.

This creates a bit of a problem sometimes.

For example, when you are dealing with big numbers in your data then these RMSE and MAE tend to become bigger. On the other hand, when you are dealing with small numbers then they become small.

So depending on which part of the data you’re testing against, the metrics values can be quite different.

And also, when you have two sales forecasting models created for two countries. One country has data ranging between 10 to 50 and another country is ranging between 1,000 and 5,000.

Their RMSEs are quite different from one another. And this makes it harder for us to say which country’s model fits better for each country’s data.

This is when other metrics like MAPE come in handy

MAPE (Mean Absolute Percentage Error)

This metric is calculated as an average of absolute percentage differences between the actual and the forecasted values.

It emphasizes the percentage (or ratio) difference, not the actual difference, hence it normalizes and would be much less impacted by whether the actual data is big or small.

Here’s how you calculate step by step.

First, you take the actual values.

Second, you take the differences (or errors) between the actual and the forecasted.

Third, calculate the ratio of the differences (or errors) against the actual values. Note that we make the numbers always positive, just like we did for MAE. This part is the Absolute part of ‘Mean Absolute Percentage Error’.

Lastly, we take the mean of these numbers.

That’s it!

Now that we understand those metrics to evaluate the forecasting models, let’s go back to our original example and see how they are.

Metrics in Exploratory

We can go to the Summary tab and take a look at those metrics.

The RMSE is 49.55, which means that there is about 50 orders difference on average between the forecasted and the actual in the Test period.

Metrics alone don’t help us to see if this model is good or bad. We always want to compare to see if it is better or worse.

There is a variable (column) in this sales data called Market. Let’s build multiple forecasting models each of which is for each market such as North America, Europe, Asia Pacific, etc.

We can check the quality of each model and compare them against each other under the Summary tab.

When we look at RMSE, we can see that Africa has the best forecasting model and North America comes in the 2nd. On the other hand, Europe has the worst performing model. Remember, for RMSE, the smaller is better, and the bigger is worse.

But not so fast.

If we take a look at MAPE, Africa is the 2nd worst and Europe is the 2nd best model. MAPE is also, the bigger is worse, and the smaller is better.

We are getting contradicting insights between RMSE and MAPE!

This is because RMSE is depending on the scale of original values.

The number of orders in Europe tend to be much bigger than the one in Africa. This makes the RMSE in Europe tend to be much bigger than the one in Africa.

On the other hand, MAPE is based on the percentage ratio. So it doesn’t really matter what the original values scales are. This is why we are seeing that Europe’s model comes out better than Africa’s.

To show what I’m really trying to say, I’ve set the Chart Setting to ‘Not sync the scale among the charts’ from the chart property.

This makes each chart has its own scale.

Now we can see the differences between forecasted and the actual for Africa wider!