Moving to Tidy Forecasting in R: Benchmark Methods and Residual Diagnostics
By: Joon Im
Tags: R, Time Series Forecasting, Tidyverse, Time Series Analysis, Forecasting

1. Set Up

1.1 Introduction

This guide is the second in a series about forecasting in R with the fable forecasting framework. The following is a code comparison of benchmark forecasting methods and residual diagnostics between the old method (fpp2) and the new tidy method (fpp3).

1.2 Load Libraries

# Load libraries

library(fpp2) # An old forecasting framework

library(fpp3) # A new forecasting framework

library(tidyverse) # Collection of data manipulation tools

library(tidyquant) # Business Science ggplot theme

library(cowplot) # A ggplot add-on for arranging plots

2. Benchmark Forecasting Methods

2.1 Naïve Forecasting

The simplest method of all where future values are simply the last known value of a series

Generally not effective but useful for benchmarking other methods

2.1.1 fpp2 Method: Naïve Forecast

# Fit and forecast Google stock price with a naïve model

fcgoog <- naive(goog, h = 20) # h = 20 days # Plot forecasts

autoplot(fcgoog)

Naïve forecasting with fpp2

2.1.2 fpp3 Method: Naïve Forecast

# Convert Google stock data to tsibble format

g <- goog %>% as_tsibble() # Fit a naïve forecast model

fit <- g %>%

model(

naive = NAIVE(value)

) # Naïve forecast using fitted data

fc <- fit %>%

forecast(h = 20) # h = 20 days # Plot forecasts

fc %>%

autoplot(g)

Naïve forecasting with fpp3

2.2 Seasonal Naïve Forecasting

A method for seasonal data where future values are the prior season’s value

season’s value Generally not effective unless the data is stationary but useful for benchmarking other methods

2.2.1 fpp2 Method: Seasonal Naïve Forecast

# Fit and forecast with a seasonal naïve model

fcbeer <- snaive(ausbeer, h = 16) # h = 16 quarters or 4 years # Plot forecasts

autoplot(fcbeer)

Seasonal naïve forecasting with fpp2

2.2.2 fpp3 Method: Seasonal Naïve Forecast

# Convert Australian beer production data to tsibble format

a <- fpp2::ausbeer %>% as_tsibble()



# Fit a seasonal naïve model

fit <- a %>%

model(

snaive = SNAIVE(value)

) # Naïve forecast using fitted data

fc <- fit %>%

forecast(h = 16) # h = 16 quarters or 4 years # Plot forecasts with original data

fc %>%

autoplot(a)

3. Residual Diagnostics

The residuals of a time series are the values that are left after fitting a model. They are essentially the differences between the observations and their fitted values, otherwise known as one-step forecast errors.

If the residuals show no outliers or patterns, then this means the forecasting method that was utilized captured all of the available information i.e. the series is white noise.

In order for a time series to be considered white noise:

Essential: The residuals must be uncorrelated. If not, then the residuals still have information left that can be used. Essential: The residuals must have zero mean. If not, then the forecasts are considered biased and require adjustment. Non-Essential: The residuals have constant variance. If not, then the prediction intervals will be larger and less accurate. Non-Essential: The residuals are normally distributed. If not, then the prediction intervals will be larger and less accurate.

3.1 fpp2 Method: Check Non-Seasonal Residuals

The Ljung-Box test is a simpler way of testing groups of autocorrelations together rather than considering each one separately using the ACF.

# Residuals from the naïve forecasts applied to Google stock price

goog %>%

naive() %>%

checkresiduals()

Since p > 0.05, the residuals are considered to be white noise.

Check non-seasonal residuals with fpp2 method

3.2 fpp3 Method: Check Non-Seasonal Residuals

# Google stock price: fit a naïve forecast model

fc <- fpp2::goog %>%

as_tsibble() %>%

model(naive = NAIVE(value)) # Ljung-Box test: If p > 0.05, then no problems with autocorrelations

fc_lj <- fc %>%

augment() %>%

select(index, .resid) %>%

features(.resid, ljung_box, lag=10, dof=0) # Print Ljung-Box results

fc_lj # Check the residuals

fc %>% gg_tsresiduals()

Since p > 0.05, the residuals are considered to be white noise.

Check non-seasonal residuals with fpp3 method

Correlation: High p-value. ACF shows <5% of spikes are significant. Lags are mostly uncorrelated as we would expect of non-seasonal data.

Mean: The mean of the residuals is zero.

Variance: The variance is generally constant except for a large jump around the day 600.

Distribution: Non-normal. Since both tails are heavy, prediction interval coverage will be larger than it should be (e.g. 95% interval is actually 97% interval)

Conclusion: The residuals are white noise.

3.3 fpp2 Method: Seasonal Residuals

# Residuals from the seasonal naïve forecasts applied to Australian beer production

ausbeer %>%

snaive() %>%

checkresiduals()

Since p < 0.05, the residuals are not considered to be white noise.

Check seasonal residuals with fpp2 method

3.4 fpp3 Method: Seasonal Residuals

# Beer production: fit a seasonal naïve forecast model

fc <- ausbeer %>%

as_tsibble() %>%

model(snaive = SNAIVE(value)) # Ljung-Box test: If p > 0.05, then no problems with autocorrelations

fc_lj <- fc %>%

augment() %>%

select(index, .resid) %>% # Fixed in new version of feasts

features(.resid, ljung_box, lag=8, dof=0) # Print Ljung-Box results

fc_lj # Plot seasonal residuals

fc %>% gg_tsresiduals()

Since p < 0.05, the residuals are not considered to be white noise.

Check seasonal residuals with fpp3 method

Correlation: Low p-value. ACF shows >5% of spikes are significant. Multiple lags are correlated as we would expect with seasonal data.

Mean: The mean of the residuals is not zero.

Variance: The variance is not constant.

Distribution: Normal. The prediction interval coverage should be accurate.

Conclusion: The residuals are not white noise. There is still information left that could be utilized with some adjustments.

3.5 A Word About Residuals

It is important to remember that residual diagnostics only determine whether a certain forecasting method is using all of the available information in the time series. It does not tell you what the best forecasting method is.

To be continued…