ARIMA Forecasting in Python. Manual and automatic ARIMA quickly up... | by Daniel Holmberg | Towards Data Science

I will use the weekly Spotify global top 200 list as a timeseries for experimenting with ARIMA models. The data ranges from 2017 to 2019 and the whole jupyter notebook is available here.
Here is a subset of the data we are doing the forecasting on:
feature_mean.head()
ARIMA stands for Autoregressive Integrated Moving Average and it depends on three key variables p, d, q to be successful. Those are briefly as follows:
p = number of lags / order of AR terms
d = order of differencing
q = number of lagged forecast errors / order of MA terms
Mishra1 has written more in depth on the inner workings of the ARIMA model including the parameters. My goal here is to explain how to get ARIMA quickly up and running in Python both manually and automatically. I will do the forecasting on the acousticness feature:
timeseries = feature_mean["acousticness"]
Let’s use the Augmented Dickey Fuller (ADF) test to see if the timeseries is stationary:
from statsmodels.tsa.stattools import adfullerprint("p-value:", adfuller(timeseries.dropna())[1])
p-value: 0.43
The p-value is greater than the significance level 0.05 so it is not stationary and differencing is as such needed, ie. d > 0.
We start by finding out the order of differencing, d, using auto correlation:
from statsmodels.graphics.tsaplots import plot_acf, plot_pacffig = plt.figure(figsize=(10, 10))ax1 = fig.add_subplot(311)fig = plot_acf(timeseries, ax=ax1,               title="Autocorrelation on Original Series") ax2 = fig.add_subplot(312)fig = plot_acf(timeseries.diff().dropna(), ax=ax2,                title="1st Order Differencing")ax3 = fig.add_subplot(313)fig = plot_acf(timeseries.diff().diff().dropna(), ax=ax3,                title="2nd Order Differencing")
The timeseries is stationary at d = 1 where only the first lag is above the significance level. If your series is slightly under differenced, try adding an additional AR term and if it is slightly over-differenced, maybe add an additional MA term.
Knowing we should difference once, we go on to find out the order of AR, p. We get it by counting the number of lags above the level of significance in partial autocorrelation:
plot_pacf(timeseries.diff().dropna(), lags=40)
The first lag is the only one vastly above the signicance level and so p = 1.
The autocorrelation function can tell the order of MA terms, q, needed to remove autocorrelation in the stationary series.
plot_acf(timeseries.diff().dropna())
One lag can be found above the significance level and thus q = 1.
from statsmodels.tsa.arima_model import ARIMAmodel = ARIMA(timeseries, order=(1, 1, 1))results = model.fit()results.plot_predict(1, 210)
Akaike information criterion (AIC) estimates the relative amount of information lost by a given model. The less the better.
results.summary()
Model: ARIMA(1, 1, 1), ..., AIC: -806.848 ...
We keep that in the back of our head and go on to test auto_arima:
import pmdarima as pm
Creating the model: automatic ARIMA using ADF to test stationarity, start values for p and q are set to 1, and the Spotify data is not assumed to be seasonal:
def arimamodel(timeseries):    automodel = pm.auto_arima(timeseries,                               start_p=1,                               start_q=1,                              test="adf",                              seasonal=False,                              trace=True)    return automodel
We want to plot it as neatly as was done using statsmodel’s plot_predict, therefore the area between the upper and lower prediction bounds has to be filled.
def plotarima(n_periods, timeseries, automodel):    # Forecast    fc, confint = automodel.predict(n_periods=n_periods,                                     return_conf_int=True)    # Weekly index    fc_ind = pd.date_range(timeseries.index[timeseries.shape[0]-1],                            periods=n_periods, freq="W")    # Forecast series    fc_series = pd.Series(fc, index=fc_ind)    # Upper and lower confidence bounds    lower_series = pd.Series(confint[:, 0], index=fc_ind)    upper_series = pd.Series(confint[:, 1], index=fc_ind)    # Create plot    plt.figure(figsize=(10, 6))    plt.plot(timeseries)    plt.plot(fc_series, color="red")    plt.xlabel("date")    plt.ylabel(timeseries.name)    plt.fill_between(lower_series.index,                      lower_series,                      upper_series,                      color="k",                      alpha=0.25)    plt.legend(("past", "forecast", "95% confidence interval"),                 loc="upper left")    plt.show()
Then we have all we need to fit and plot the model:
automodel = arimamodel(feature_mean["danceability"])plotarima(70, feature_mean["acousticness"], automodel)
automodel.summary()
Model: ARIMA(1, 1, 1), ..., AIC: -806.848 ...
This turned out to be the exact same parametrization we got manually. Wow that worked out well! It was far easier and faster to get the parameters right using auto_arima, the only slight downside is that the plotting has to be done from scratch to look as nice as the one statsmodels has built in.
Huge credits to the exhausting guide2 on the ARIMA model over at MachineLearning+. Oh, and you can also try changing the feature from acousticness to something else eg. danceability using the aforementioned notebook.
[1]: M. Mishra, Unboxing ARIMA Models (June 11 2018), https://towardsdatascience.com/unboxing-arima-models-1dc09d2746f8
[2]: ARIMA Model — Complete Guide to Time Series Forecasting in Python, https://www.machinelearningplus.com/time-series/arima-model-time-series-forecasting-python/