Searching for the Best Forecasting Model: A Comparison of Different Univariate Forecasting Models | by Justin Eloriaga | Towards Data Science

Forecasting may be a daunting challenge, especially to those with only a few or no backrgound in statistics. But I think it is much easier that people think it is. In this article, I will go through the basic univariate forecasting models and forecast a variable with relative ease.
There are two basic models in univariate forecasting. The first is the autoregressive model which makes use of past values of the forecast variable and the moving average model which uses past values of a white noise error term.
The autoregressive model looks like this
Notice that the model just uses the past (lagged) values of the forecast variable (y). We use p number of lags, p being determined by some tests.
The moving average model looks like this
Like the AR, the MA uses past values but instead of the forecast variable, it uses error terms. We use q number of lags, q being determined by some tests.
We can combine both the autoregressive and moving average models in a model called the ARMA. However, it is often the case that our forecast variable is non-stationary. This means that the series often contains and upward or downward trend. To alleviate this, we can difference the series in order to eliminate those upward or downward movements to stationarize the series. In doing so, we generate the ARIMA model or Autoregressive Integrated Moving Average. Some time series variables may have seasonality like sales or revenue which often spike up during holiday months. To accommodate this, we have a generalized variant of the ARIMA called the SARIMA which incorporates seasonal lags as well.
The ARIMA Model looks like this
In this model, we difference the series d number of times until it becomes stationary. The phi and theta components represent the AR and MA models, respectively.
The SARIMA Model looks like this
This is merely a generalization of the ARIMA but adds seasonal differences, seasonal moving average lags, and seasonal autoregressive lags.
Our goal is to forecast the next month’s inflation rate. What you should have is the comma-separated values file (.csv) of the Monthly Inflation Rate of the Philippines from January 2000 until April 2020. The file name is InflationRateLecture.csv. For the full series, you can download it from the Bangko Sentral ng Pilipinas Key Statistical indicators page. You should also have a copy of R and RStudio installed on your Mac or PC. If this is your first time using R, fear not, just merely follow the codes below and you should be fine.
The first step is to install the necessary packages. For this example, we would need to install the required packages. These are the following: tidyverse for general data manipulation and urca, forecast, TSstudio, and tseries for the necessary forecasting commands. Feel free to explore the various functions you can get out of these.
Generate a new script so that you can keep track of things. The first order of business is installing the required packages. We can do that in the tools tab and finding the packages manually, or you can just run the commands that follow. Note that this may take some time to install depending on your internet connection.
install.packages("tidyverse")install.packages("urca") install.packages("forecast") install.packages("tseries") install.packages("TSstudio")
After installation, you must now tell R that you are about to use these packages. We do this by invoking the library() function. Consider this like taking a book from a library you want to borrow and use for a specific study session.
library(tidyverse)library(urca)library(forecast)library(tseries) library(TSstudio)
After you run all of these commands, you should be ready to run all necessary commands afterward. If in case you close your R session in the middle of forecasting, merely use the library commands once again to reload the packages. No need to reinstall (unless an update to a package is available).
A simple way to load the dataset is using the file.choose() function which opens up a dialogue box similar to what we are accustomed to when opening files. It is important that we store the dataset in an object so we can refer to it later. First, we load the dataset and store it in an object. For this tutorial, I’ll name the object as “inflation” but you can name it anything you want. The codes and dataset can be found here
https://drive.google.com/drive/folders/11RZI8p-qCvwLAmdro93FnslQLxCO39Ef?usp=sharing
inflation <- read_csv(file.choose())
The read_csv is part of the readr package in R inside tidyverse. This is used to read a csv file and to be able to load it in R. You should notice that a dialogue box should appear prompting you to choose the file. Be sure to pick InflationRateLecture.csv.
head(inflation)nrow(inflation)
Running the head(inflation) command lists the first 6 rows of the dataset. You should see that the inflation rate in January 2000 is 5.5, February 2000 is 5.6, and so on. The nrow(inflation) gives the total number of rows in the dataset. This suggests that there are 244 time periods (months) in the dataset. You can also view the dataset by looking at the Environment Tab and clicking the grid beside inflation.
As it stands, the dataset we loaded is a frame. To be able to forecast, we need a series. If you look at the dataset, we are only interested in forecasting the “Rate” variable that is in there. As such, we need to transform that “Rate” variable into a series.
inf <- ts(inflation$Rate, start = c(2000,1,5), frequency = 12)
We are creating a new object called “inf” which is the series we will use. The ts command creates that object. The first argument in the command is the variable. The $ tells R that we want to get the Rate variable from the dataset inflation. Afterward, we specify the start date which is January 5, 2000. R can automatically detect the end after this. Lastly, we specify the frequency of the series. Since we are dealing with monthly data, the frequency would be 12. If it were quarterly, frequency is 4, weekly would be 52, and bi-annual would be 2.
It is important to visualize our series. We can use the autoplot command to do this.
autoplot(inf) + ggtitle("Inflation Rate (Philippines), January 2000 to April 2020") + labs(x = "Time", y = "Inflation Rate")
The command plots the series “inf” with the corresponding title “Inflation Rate (Philippines), January 2000 to April 2020” where the x-axis is labeled Time and the y-axis is labeled Inflation Rate. We use the ggtitle and the labs option for those specifications. If you did it correctly, you should have a graph that looks something like the figure above.We can see the inflationary spike during the Global Financial Crisis, the Taper Tantrum, and the recent TRAIN 1 package that was passed.
We now move on to generating the ACF and PACF graph and determining whether the series is non-stationary.
Let us first generate the ACF and the PACF of the inflation rate. Again, the ACF and the ACF can tell us a lot about the properties of the series. Usually, it gives some indication as to the underlying process of the series, whether it is an AR, an MA or an ARMA. Furthermore, we will see what the ACF and PACF of the differenced value of inflation should we deem the series non-stationary.
ggAcf(inf) + ggtitle("ACF of Inflation")ggPacf(inf) + ggtitle("PACF of Inflation")
The ACF in the figure is geometrically decaying while the PACF registers an immediate cutoff. If we recall our last lecture on the ACFand PACF, this is consistent with an AR process. While it is not definitive, this gives us a clue of what the underlying process might be. In the ACF, notice that the first until eleventh lag is statistically significant (beyond the blue confidence band). We can also see that some partial autocorrelations are significant such as the first, second, and fourteenth.
One way to alleviate non-stationarity is by differencing the series. To do this, we use the diff() command.
dinf <- diff(inf)ggAcf(dinf) + ggtitle("ACF of Inflation (Differenced)") ggPacf(dinf) + ggtitle("PACF of Inflation (Differenced)")
The first command utilizes the diff() function which differences the series and creates an object “dinf” which is the differenced value of inflation. By default, it differences once but we can set it to difference twice or whatever number of times specified.
Similar to the levels, we can see that the ACF is geometrically decaying while an immediate cutoff is seen in the PACF. This is more evidence in support of an AR process. However, notice that not many lags are significant. This is some signal that the series has been “stationary” so to speak which forces it to display properties of covariance stationarity. We will explore more into this as we go along.
Comparing the level values and the differenced values of inflation yield some interesting results. First, there is evidence of non-stationarity using a simple graphical look. However, we can’t be too sure until we conduct formal statistical testing. Secondly, notice that there are still wild spikes (outliers) in the differenced series. These may be indicators for structural breaks which we would need to account for should they be present.
It is also important to see a proper decomposition of the time series we have. These components are the trend, seasonality, and random components in the series. Fortunately, there are commands in R that graph this automatically for us.
ts_decompose(inf, type = "additive", showline = TRUE)
The ts_decompose command in the library TSstudio gives us a good visualization of the components of the series. This command will segment the series into the observed, trend, seasonal, and random components of the series. Notice that there is some observed seasonality present, likely every August or September. The reason for which I will leave for you to figure out. The decompose command is in TSstudio is far more interactive allowing you to hover your cursor around the graph (generated by plot.ly, a fun graphics library) and dig into the series. We will be using this more as we go along.
We will use three most popular tests in testing for non-stationarity. These are the Augmented Dickey-Fuller, the Phillips Perron, and the KPSS test. Bear in mind that the ADF and PP tests are unit root tests. As such, their null hypothesis is non-stationarity while their alternative is stationary. Conversely, the KPSS test is a stationarity test in which its null hypothesis is stationarity while its alternative hypothesis is non-stationarity.
The command to do the ADF test is adf.test() in R. We will use this test on the series using both the alternative hypothesis of stationarity. We will run this on both the raw and differenced series.
adf.test(inf) adf.test(inf, k = 1)adf.test(inf, k = 2)adf.test(dinf)
Notice that when we ran the first line, there was a prompt suggesting that the series was stationary. The same is true for when we tested the fourth line. While inflation’s differenced variable is certainly stationary, the value of the level may be very deceiving. As we had said earlier, the limitation of the ADF test is that you have to specify the lag order. These tests (the first and third) were conducted in the sixth lag. If we use a more realistic lag say 1 or 2, the series was determined to be non-stationary.
For the Phillips Perron test, we use the pp.test() command. Unlike the ADF test, we no longer need to specify the lag order and is more general.
pp.test(inf)pp.test(dinf)
The results of the PP test are fairly conclusive. We see that at levels, the series is non-stationary as we fail to reject the null hypothesis. Once it is differenced, the series is stationary. As such, when we start building a forecasting model, we would need to difference inflation in order to stationarize it.
Lastly, we turn to the KPSS Test. Again, bear in mind that the null hypothesis of this test is stationarity and the alternative is non-stationarity. The command for the KPSS test is kpss.test(). We can specify an option called null to be “trend” or “level” or both. For this case, let’s just leave it at the default which considers both.
kpss.test(inf)kpss.test(dinf)
As expected, we find that the level series is not stationary. Curiously, the test also finds the differenced series is non-stationary. Still, we can refer as well to the results of the ADF and the PP tests as a backup for our claim that the differenced series is stationary.
Let us now move on to the forecasting proper. First, let us forecast in-sample before we do an out of sample forecast. As we mentioned in the first section, the reason for forecasting in-sample is to see the quality of the generated model and for us to compare different forecasting models and their estimates against actual realized values. We will use the forecast quality indicators discussed in the last chapter. Finally, towards the end, we will forecast inflation for the next year using the best forecasting model we determined.
Firstly, let’s divide the dataset into two samples, a training sample, and a testing sample. The training sample is the available data we have to forecast. The testing sample is data that is already realized (i.e. we already know the series values) but we holdout to test the quality of forecasting models. While before, this step used to be a pain, with R, it’s a breeze. We use the ts_split command to do this.
split_inf <- ts_split(inf, sample.out = 12)training <- split_inf$train testing <- split_inf$testlength(training)length(testing)
We create an object called split_inf using the ts_split function which is essentially inf but partitioned into two. The object inf is divided into two parts where the testing set is given by the command sample.out. We set the sample.out equal to 12 which translates to 12 months or a year. Hence, we want our testing (validation) period to be equal to a year. After this, we create two objects (training and testing) which creates two separate series. Using the length command, we can determine how many periods there are in each of the generated series. You will notice that the training dataset has 232 periods while the testing dataset has 12 periods (the same as the number specified by the sample.out option).
It is important to have at least some idea of what model to implement. To do this, we need to try and diagnose the training dataset we have to see if we can come up with suggestions as to which model may fit the series. To do this, we implore the use of the arima_diag() command.
arima_diag(training)
Running this yields us the figure to follow. Since this was created using plot.ly, it is quite interactive. We can see that it shows us the values of the training dataset as well as the associated ACF and PACF. It also shows us a graph of the differenced training series. What we can see is that the ACF is geometrically declining and an immediate cutoff is seen in the PACF. This is similar to the pattern we expect from an AR model. To determine a guess of the number of lags to use, notice that in the PACF, the first two lags are still significant. After the second lag, the majority of the remaining lags are insignificant. For now, let us peg our guess at two autoregressive lags. Notice also that in the PACF and ACF, there are red data points. These red data points indicate the potential for a seasonal lag.
Let us try to forecast three models with three varying specifications. The three models we will try to use are
· An ARIMA(2,1,1)
· A SARIMA(2,1,1)(1,0,0)
· The model determined most fit using the auto.arima() function
For the first model, we will try to use 2 autoregressive lags since this is what the PACF is indicating. We will difference the series once because we know that at levels, the series is non-stationary as seen in the results of the ADF, PP, and KPSS tests. We guess that we have 1 moving average lag. For the second model, we will use the same specification as the first except that we add a seasonal autoregressive lag. Lastly, for the third model, we use the built-in auto.arima() function in R to take the guesswork out of choosing which lag specification is best.
Let us move to Model 1. We create an object called arima211 and use the arima() command to model. The first option in the arima() command is training which is the series we will be using. The order option specifies the order of the ARIMA, in this case, we specify an ARIMA with two autoregressive lags, 1 difference to stationarize (i.e. the series is integrated of order 1), and 1 moving average lag. The autoplot() command checks whether the stability conditions have been met. This is the stationarity condition laid out in section two. Since we are dealing with the inverse roots, all inverse roots must be inside the unit circle. Lastly, the check_res commands provide diagnostics on the residuals, which we want to be white noise. Model 2 is structured the same way except that we add a seasonal order when we invoke the arima() command. In that case, we add 1 seasonal autoregressive lag. Model 3 is structured the same way except we let R decide the order using the auto.arima() command. The option seasonal = TRUE just ensures that it can choose a SARIMAmodel if it deems it the most optimal.
#For Model 1arima211 <- arima(training, order = c(2,1,1))autoplot(arima211)check_res(arima211)#For Model 2 sarima2111 <- arima(training, order = c(2,1,1), seasonal = list(order = c(1,0,0)))autoplot(sarima2111)check_res(sarima2111)#For Model 3auto <- auto.arima(training, seasonal = TRUE)auto #We obtained a SARIMA(2,1,0)(2,0,1) based on auto.arima()autoplot(auto)check_res(auto)
If you did it correctly, you should be greeted with the figures to follow. Notice that for all three models, the roots are inside the unit circle, as such, the models have passed the stationarity criterion suggesting that there are no more unit roots. Looking at the residual plots, we see that the residuals are generally white noise for all the three models. We do note however that there are some lags which are marginally outside the confidence band, but certainly, model 2 and model 3 produce more white noise errors vis-a-vis model 1 which has an ACF which is quite significant at the 12th lag. This red lag is a seasonal lag which is an indication that a seasonal model is more adept.
Now we get to the moment of truth. Let us now generate the forecasts using each model and evaluate them against the baseline. The commands for each model are structured similarly. First, we create three objects, namely, fcast1, fcast2, and fcasta, which represent the forecasts for each model. Each object is made using the forecast() function which has two options. The first is we key in what model it will use to forecast. The second is the number of periods to forecast. Since the test dataset is 12 periods long, we set the horizon h to 12. Next, we use the test_forecast() command to compare the forecast against the actual value. The command generates an interactive graph and compares each forecast against the actual value while also revealing key forecast quality indicators at every data point. Lastly, thee accuracy() command generates the full average of the forecast indicators which we discussed in the last section. The first option in the accuracy() command is the forecast generated by each model is compared to the test or validation series.
We can see the graphs of the actual vs forecasting values in the figure to follow. What we notice is that it seems that Model 3 which uses the specification generated by the auto.arima() command seems best based on fit. Notice that when we generated these graphs, we can hover on each data point and it will give us.
#For Model 1fcast1 <- forecast(arima211, h = 12)test_forecast(actual = inf, forecast.obj = fcast1, test = testing)accuracy(fcast1,testing)#For Model 2fcast2 <- forecast(sarima2111, h = 12)test_forecast(actual = inf, forecast.obj = fcast2, test = testing)accuracy(fcast2,testing)#For Model 3fcasta <- forecast(auto, h = 12)test_forecast(actual = inf, forecast.obj = fcasta, test = testing)accuracy(fcasta,testing)
Further using the accuracy() command, we can see the forecast statistics comparison of each model. The forecast indicators are seen in the table to follow produced by the accuracy() command. We can see an improvement from model 1 to 2 and from model 2 to 3. This suggests that adding a season lag was vital and that the model generated by the auto.arima() seems to be the most optimal based on these indicators.
Now that we know what the best model is for forecasting, we shall now generate an out-of-sample forecast. In this, we don’t know the actual values of the series. We can only use whatever information we have to forecast the future. This is, essentially, the forecasting you have been looking forward to.
Similar to generating the model, we create an object wherein the model of choice is stored. Instead of using the training data, we use the whole series of inflation which is inf. We then test for the inverse AR and MA roots as well as the residuals.
finalfit <- auto.arima(inf, seasonal = TRUE)autoplot(finalfit)check_res(finalfit)
We find that all inverse roots are within the AR and MA unit circle. As such, we have satisfied the stability condition, there are no longer any unit roots. On the residual diagnostics, while there are still a few significant lags, it is generally white noise already.
We now forecast using our chosen model. The commands below will generate the forecast. As inflation is usually re-forecasted, let us set a short horizon of 4 months ahead. Then, let us see its forecast for the May to August 2020 inflation rates. The summary() command lists the point forecasts and the confidence interval for the next four months.
fcastf <- forecast(inf, model = finalfit, h = 4)plot_forecast(fcastf)summary(fcastf)
We can see that the forecasted values for May (2.0 percent), June (2.1 percent), July (2.1 percent), and August (2.1 percent) are fairly consistent with the BSP’s inflation target and other forecast estimates. Notice too that as the horizon gets larger and larger, the confidence band expands along with it. Good job, you just forecasted your first key economic variable.
We have now forecasted out-of-sample and have obtained values for the next four months. We shall see if our forecasts are indeed reliable through time. Suffice to say, forecasting is a trial and error process but I hope you saw that the indicators and built-in functions were able to take the guesswork out of the process. Still, forecasting is also an art, and specifying lags and conditions based on economic structure and intuition may play a pivotal role in improving forecasts. We shall see more of this as we discuss multivariate forecasting.
For now, it is recommended that you experiment with the many tools you now have learned and see if you can come up with a model that provides even better forecast quality indicators than even the model selected using the auto.arima() function. That is certainly possible and it just goes to show that while machines have advanced so much, there is still room for art in a place riddled full of math and science. One just needs to look for the motivation to do so.
For a more hands on approach, I made videos on this very article which can be found on my YouTube Channel.
[1] Brooks, C. Introductory econometrics for finance.(2019) Cambridge university press.
[2] Hamilton, J. Time series econometrics. (1994) Princeton U. Press, Princeton.