Understanding ARIMA (Time Series Modeling) | by Tony Yiu | Towards Data Science

What’s that old Mark Twain quote again? “History doesn’t repeat itself but it often rhymes.”
I love analyzing time series. So I’m probably biased when I say that I think Mark Twain is right. We definitely should treat all forecasts skeptically — the future is inherently uncertain and no amount of computing or data will ever change that fact. But by observing and analyzing the trends of history, we can at least unravel a small portion of that uncertainty.
ARIMA models are a subset of linear regression models that attempt to use the past observations of the target variable to forecast its future values. A key aspect of ARIMA models is that in their basic form, they do not consider exogenous variables. Rather, the forecast is made purely with past values of the target variable (or features crafted from those past values).
ARIMA stands for Autoregressive Integrated Moving Average. Let’s walk through each piece of the ARIMA model so that we fully understand it.
This is the easiest part. Autoregressive means that we regress the target variable on its own past values. That is, we use lagged values of the target variable as our X variables:
Y = B0 + B1*Y_lag1 + B2*Y_lag2 + ... + Bn*Y_lagn
That’s pretty straightforward. All this equation is saying is that the currently observed value of Y is some linear function of its past n values (where n is a parameter we choose; and B0, B1, etc. are the regression betas that we fit when we train our model). The previous equation is commonly called an AR(n) model where n denotes the number of lags. We can easily make this a forecast of the future by changing around the notation a little bit:
Y_forward1 = B0 + B1*Y + B2*Y_lag1 + B3*Y_lag3 + ... Bn*Y_lag(n-1)
Now we are predicting the future value (1 time step ahead) using the current value and its past lags.
Integrated denotes that we apply a differencing step to the data. That is, instead of running a regression like the following:
Y_forward1 = B0 + B1*Y + B2*Y_lag1 + ...
We do this:
Y_forward1 - Y = B0 + B1*(Y - Y_lag1) + B2*(Y_lag1 - Y_lag2) + ...
What the second equation is saying is that the future change in Y is a linear function of the past changes in Y. Why bother with differencing? The reason is that differences are generally much more stationary than the raw undifferenced values. When we do time series modeling, we like our Y variables to be mean variance stationary. This means that the main statistical properties of a model do not vary depending on when the sample was taken. Models built on stationary data are generally more robust.
Take real GDP (real means that it’s been adjusted for inflation) for instance. It’s obvious from the plot that the raw GDP data is not stationary. It’s rising making the mean GDP in the first half of the plot much lower than the mean in the second half.
If we difference the data, we get the following plot. Notice that it is now significantly more stationary (the mean and variance are roughly consistent over the years).
When I first studied time series, I assumed the moving average was simply just the trailing moving average of the Y variable (e.g. a 200 day moving average). But while they are somewhat similar in spirit, they are distinct mathematical entities.
A moving average model is summarized by the following equation:
Y = B0 + B1*E_lag1 + B2*E_lag2 + ... + Bn*E_lagn
Similarly to the AR part, we are doing something with historical values here, hence all the lags. But what is this E? E is commonly called error in most explanations of MA models, and it represents the random residual deviations between the model and the target variable (if you’re asking how it’s possible that we can have errors before we’ve even fit the model, hold that thought for just a second).
The full equation of a basic regression model is:
Y = B0 + B1*X + E
We need the E in the equation to show that the regression output, B0 + B1*X, is merely an approximation of Y. The plot to the left shows what I mean. The black dots are what we are trying to predict and the blue line is our prediction. And while we have successfully captured the general trend, there will always be some idiosyncratic variance that is not capture-able. The E term accounts for this un-captured part — in other words, E represents the difference between the exact answer and the approximately correct answer delivered by our model.
So if you’ve been reading carefully, you might ask, “Don’t we need to have a model first before there can be model errors (Es)?” You are exactly right. E is what we call an unobservable parameter. Unlike lagged values of Y (our target variable) or exogenous X variables, E is not directly observable. On a side note, this also means that we can’t fit an ARIMA model using OLS (because we have unobservable parameters). Rather, we need an iterative estimation method like MLE (Maximum Likelihood Estimation) that can simultaneously estimate both the beta parameters and residuals (and the betas on the residual terms).
So an MA model forecasts Y using the model’s past errors (similarly to the AR model, we tell it how many past errors we want it to consider).
Let’s take a second to think about why this works. Consider the following simplified MA(1) model:
Y = u + B1*E_lag1where u = the mean of YE = Y - predicted
The first term, u, means that our model centers its forecast around the mean of Y (it’s like saying knowing nothing else, I will guess the mean). The second term B1*E_lag1 is where the error comes in. Let’s assume that for now we estimate B1 to be 0.2. And error (E) is defined as the actual values of Y less the model’s prediction.
This means that if the most recent error (E_lag1) was positive (meaning that the actual was greater than our prediction), we will shift our forecast up by 1/5 of the error amount. This has the effect of making the model slightly less wrong (because the second term, B1*E_lag1 pushes the model’s forecasts slightly towards the correct answer).
So if the model knows its own errors, then why does it not constantly overfit (in other words, why are we not biasing our model by unfairly giving it some of the answers)? Barring overfitting, MA models are not inherently biased because model errors are independent and approximately normally distributed (error is a random variable). And because they’re random variables, the errors (E_lag1) can take on widely divergent values depending on at which time steps they’re observed. This inherent noisiness in the errors means that with just one beta (B1) to fit, it would be more or less impossible to find a B1 that perfectly adjusts every error so that their product perfectly plugs each and every hole between the actual and predicted Ys. In other words, the MA(1) model can only use its knowledge of the errors to very approximately nudge itself back in the right direction.
Of course, like with any other model, we can overfit an MA model by allowing more features — in this case, it would be more and more lagged errors (which means fitting a lot more betas). But such an overfit model would perform terribly out of sample.
Let’s put it all together by building an ARIMA model to forecast real GDP. We will use the statsmodel implementation of ARIMA. I pulled the data for real GDP from FRED (the Federal Reserve Bank of St. Louis’ data repository).
The ARIMA function from statsmodel requires at least two arguments:
The data — in this case, we give it a Pandas series of raw real GDP values (we don’t need to difference it in advance as the ARIMA algorithm will do it for us).The second argument, order, tells the ARIMA function how many components of each model type to consider in the following sequence — (AR lags, time steps between difference, MA lags).
The data — in this case, we give it a Pandas series of raw real GDP values (we don’t need to difference it in advance as the ARIMA algorithm will do it for us).
The second argument, order, tells the ARIMA function how many components of each model type to consider in the following sequence — (AR lags, time steps between difference, MA lags).
Let’s check out what an AR(4) model looks like (with no MA). We take a 1 time step difference to make our data stationary:
from statsmodels.tsa.arima_model import ARIMAmod = ARIMA(master_df['GDP'], order=(4,1,0))mod.fit()predictions = mod.fit().predict()
Our forecast is plotted in the graph below. If you look carefully, the orange line (our prediction) lags the blue line (actual). This is bad. It means our forecast is always behind reality; so we would always be a few steps behind if we follow this model. But that’s to be expected from an AR model, which attempts to predict the future by extrapolating the recent past (if only it were that easy).
It’s also important to note that in this demonstration, I am not splitting out a train and test set; and I’m using the entire dataset to fit my parameters (as opposed to an expanding window). In practice, we would want a much more robust test of the model’s ability to predict out of sample. So we would want to use point in time data (as opposed to revised GDP data) and an expanding window regression so that at each time step, we estimate parameters with only the data that would have been available to us at that point in history.
Now let’s add the 2 MA components to our model:
mod = ARIMA(master_df['GDP'], order=(4,1,2))mod.fit()predictions = mod.fit().predict()
We’ve given our model the ability to course correct a bit by allowing it to consider the magnitudes and directions of its errors. This comes at the expense of a longer model estimation process — MLE takes some time to converge as opposed to OLS, which is very fast. Let’s see how our new forecast looks:
Not much better. But we shouldn’t expect massive improvement merely from adding a few MA components. AR and MA components are both derived from the target variable’s past values — so they are both attempts to forecast the future by extrapolating the past. You would expect this to work O.K. during periods of calm, but to fail miserably when attempting to predict turning points.
So after all this, we ended up with a lackluster model. But don’t despair. An ARIMA model is not meant to be a perfect forecasting tool. Rather it’s a first step. Features derived from the past values of our target variable are meant to be complements rather than substitutes for exogenous variables. So in reality, our GDP model would consist of not just AR and MA components, but exogenous ones that correlate well with GDP such as inflation, stock returns, interest rates, etc.
Also, the fitted betas themselves are often of interest. For example, if we are building a simulation of real GDP, then we need to measure GDP’s autocorrelation (autocorrelation means the correlation between the current change in GDP and its past values). Because if there is autocorrelation, then we definitely don’t want to build a GDP model where we simulate each quarter over quarter change in GDP as independent from all others. That would be wrong and our model would produce results that detach from reality. So analyzing the betas of our ARIMA model help us better understand the statistical properties of the target variable of interest.
If you liked this article and my writing in general, please consider supporting my writing by signing up for Medium via my referral link here. Thanks!