Do you remember those magical days when your Dad used to bring home happiness just for you, no…
By: Koyel Guha
Tags: Time Series Forecasting, Time Series Analysis, Data Science, Sarima, Machine Learning

Do you remember those magical days when your Dad used to bring home happiness just for you, no matter how tiring was his day at work! I had my own magical wand ( ‘my super Dada’ ) who used to bring candies for me every day on his way back home from work.

Don’t you have such ‘candid’ memories from your childhood days!!?! If you don’t, never-mind because not all kids are as lucky as a few to have a whole candy floss factory of their own! (* wink wink * )

By the way, I always had an interesting desire to be able to predict the number of candies I would get for the next day so that I can have my own plan of distributing it among my amigos and never miss a chance of being called ‘The Candy Friend’ !! She is grown up now, nevertheless, let’s put wings to her desire via ‘Time Series Forecasting’… lol. :)

Time Series forecasting is nothing but a machine learning algorithm which gives you the power of time travel. (Interesting?? But please do not rattle this as a definition in any interview! *Screw-up-yourself-alert* )

To begin with our time travel, let’s import all the important libraries to keep them handy. I am using Jupyter notebook and Python to cover this write-up. You can use the same or any other IDE you are comfortable with.

Let’s begin with importing the frequently used libraries for our ease:

Next step, should be reading the dataset. The csv file contains only two columns Date and Candy_Count. We import the dataset using pandas function and make Date as the index of the dataframe.

Before we delve deeper into our analysis, let’s recall what is SARIMA. Seasonal Autoregressive Integrated Moving Average, SARIMA or Seasonal ARIMA, is an extension of ARIMA that explicitly supports univariate time series data with a seasonal component.

It adds three new hyperparameters to specify the autoregression (AR), differencing (I) and moving average (MA) for the seasonal component of the series, as well as an additional parameter for the period of the seasonality. In addition to these, when exogenous variables are added which aid in the better explanation of the data, it is nothing but SARIMA-X model.

A clear understanding of the above discussed model can be depicted in this article. Sometimes, we might have noticed that a ‘Weekend’ becomes a happy cause for Dad to bring in more candies than a dreaded Monday! Makes sense??

So, why not consider the week days as well to better forecast our candy count! Well done guys, we are on the right track :) Let’s add the days in the dataframe and use get_dummies() from pandas library to encode them in order to pass in as parameters later in the model.

The next most important step is to set the frequency of the index of our dataframe. By default, it is set to None. Since you can see we are overjoyed to get candies ‘every-day’, so naturally the frequency of the index of the dataset should be set to ‘Daily’ (D) in this scenario:

Now, let’s take a sneak peak at the whole data by plotting it using matplotlib library. We will also draw a black vertical line on the plot to represent the ‘Saturdays & Sundays’ and notice if there is any change in the pattern of candy count at all…

Weekends do have a candy surge, you see… ;) Besides, the data looks quite stationary or non-seasonal in nature.You can always perform an Augmented Dickey Fuller test to confirm its stationarity with statistical evidence!

Now let’s go ahead and perform ETS ( Error-Trend-Seasonality) Decomposition to view the components clearly. It can be either an additive or a multiplicative model :

+

Taking a closer look at the seasonal plot will reveal the periodicity as weekly or 7 days!

Now, split up the dataset into training and test respectively. Quick Tip: The test set should ideally be at least as large as the maximum forecast horizon required.

|Drums roll..Trumpets flair..Shout out for the showstopper — AUTO-ARIMA|

The most difficult part of the analysis wherein we need to decide on the optimal order of p, d and q to be used as input parameters for the model should technically be done by analyzing ACF ( auto-correlation ) and PACF (partial auto-correlation ) plots. The ‘p’ component corresponds to the number of lags wherein a sharp drop is seen in the plot and ‘d’ component can rightly be judged by differencing and shifting the data to find out how many times it needs to be done so that the non-stationary data changes into a stationary one. Phew… too many stuffs to look for!

Wait a sec, AUTO-ARIMA comes to our rescue which uses a Grid-Search technique to automatically run the model across a range of parameters to find the best one among them!

It takes in the candy_count column as the data, the seasonality as 7 (weekly as seen from seasonal plot in ETS ) and the list of exogenous variables which in this scenario is the list of days of the week which we had encoded before.

Auto-arima rightly suggests to use SARIMAX model with order (1,0,1) which is equivalent to (p,d,q) parameters. Also notice, the d-differencing component is rightly given as 0 since the data is stationary in our scenario. Now, let’s fit in this model directly on the training data and use it to predict the data for our test data.

For using the model to predict the test results, the start and end parameters should run from length of the train dataframe to one less from the combined length of train and test dataframe. Since, we are using SARIMAX model, the exogenous variables list should also be passed for test dataframe using ‘exog’ parameter.

Now, let’s plot them up to check out how well it is in predicting the test data:

The green shows ‘original test data’ and the red shows ‘predicted test data’.

Not bad ! :) There seems to be a fine-tune required for predicting the lower stream data but overall the model is pretty good in predicting the test data. You are always welcome to try this on your own and have your own suggestions!

For now, let’s go ahead and evaluate the measures using Root mean Squared Error and Mean Squared Error against test data and predicted test data:

So far so good! Let’s take a step ahead and train the model on the entire dataset and predict the future data:

Before predicting the future data, we need to keep in mind that we are using SARIMAX model which uses the exogenous variables to predict the future. Hence, before we predict the data for the future days, we need to provide the exogenous variables i.e. the day of the week for the same in advance.

I have manually created the dataframe called exog_forecast which contains the week days’ name in advance corresponding to the future dates for which candy count needs to be predicted. You can create your own using date range function. Mine looks like below:

Let’s forecast the data using the model for the next 17 days!!

Don’t worry about such an odd number of days used in predicting, you can go ahead and choose your desired number of days, but never choose a figure more than the horizon of your test dataset! :) Remember to use end param as number of days needed to forecast -1 for limit issues restraint.

Voila!! Now let’s plot the forecasted data with the original data on a same plot using matplotlib library:

Woohoo!! we just finished predicting the number of candies to receive in future and Guess What… you can even verify your predictions asking your Dad to start bringing candies for you if he hasn’t already started!! :D

Keep exploring and have fun! :)

P.S.- If you like my attempt, please lend me a clap. :)