Time-Series Forecasting with Google BigQuery ML | by Tobi Sam | Towards Data Science

If you have worked with any kind of forecasting models, you will know how laborious it can be at times especially when trying to predict multiple variables. From identifying if a time-series is stationary or not, to estimation and finally model evaluation, this process is nothing short of time-consuming. For example, before you can start time-series forecasting with the ARIMA method in Python, you will have to follow these steps:
Identification/Data-Prep: This tests if a time-series is stationary by visualising the data and/or applying one of the statistical stationarity testing methods such as; the Dickey-Fuller test or the KPSS test. You can also use an ACF plot to identify non-stationary time-series data.
Differencing: Differencing can help stabilise the mean of a time series by removing changes in the level of a time series, and therefore eliminating (or reducing) trend and seasonality. Based on the results of the tests from the previous step, this will determine what kind of differencing method you will use. Example; Lag differencing, Difference Transform, Log Transform and others (see reference links at the bottom for more information).
Parameter Estimation: To create your ARIMA model, you will find the order of each ARIMA coefficient, “p”,”q” and “d” by looking at the ACF and PACF plots in order to select the right model for prediction. It basically means finding the values of the model coefficients which provide the best fit to the data. You can use AIC — Akaike information criterion or BIC — Bayesian information criterion as the selection criterion.
Diagnostic: Involves testing the assumptions of the model to identify any areas where the model is inadequate. If the model is found to be inadequate, it is necessary to go back to Step 2 and try to identify a better model. If your model is adequate, you can proceed to the final step...
Forecasting/Production: You can now use your model to make forecasts.
Following this necessary process, known as the Box-Jenkins method, can take a while before arriving at a suitable model. Thankfully, Google has incorporated a number of Machine Learning models right into BigQuery, and time-series forecasting with ARIMA is one of them. What this means is that you can now use plain old SQL to design complex ML models and at a fraction of the time while Google takes care of model selection and the forecasting process under the hood!
In the article, I will show you how to make forecasts 90 days into the future, using BigQuery ML’s time series model.
Forecast a single variable
Forecast multiple variables
Evaluating the model
As with any Machine Learning problem, gathering the right training data is the key to getting good predictions. So in my experience, the more extensive your time-series data is, the more likely you are to get good predictions. I try to capture as many seasonal patterns as possible in my training data.
For an e-commerce company, it is important to capture at least one holiday period in your historical data so your model can learn the spikes and dips during those periods. In our case, we are looking at an 8-month historical view. I prefer looking at even longer historical windows but we will stick with this for now.
#standardSQLSELECT FORMAT_DATE(“%Y-%m-%d”, DATE(orderDate)) AS day, SUM(quantity) AS salesFROM   `myproject.mydataset.mytable`WHERE orderDate > CAST(date_add(DATE_TRUNC((DATE_SUB(CURRENT_DATE, INTERVAL 8 MONTH)), MONTH), INTERVAL 1 day) AS TIMESTAMP)GROUP BY dayORDER BY day, sales DESC
From this chart, we can see recurring patterns throughout the year and a sales spike around December.
In this step, we will create a time-series ARIMA model called
mydataset.sales_forecast
The CREATE MODEL clause creates and trains the model for a single time-series.
CREATE OR REPLACE MODEL mydataset.sales_forecastOPTIONS(model_type=’ARIMA’,time_series_data_col=’sales’,time_series_timestamp_col=’day’,data_frequency=’DAILY’holiday_region=’US’) ASSELECT FORMAT_DATE(“%Y-%m-%d”, DATE(orderDate) as day, SUM(quantity) AS salesFROM   `myproject.mydataset.mytable`WHERE orderDate > cast(date_add(DATE_TRUNC((DATE_SUB(CURRENT_DATE, INTERVAL 8 MONTH)), MONTH), INTERVAL 1 day) as TIMESTAMP)GROUP BY dayORDER BY day, sales DESC
BigQuery ML automatically divides your training data into train/test during model creation.
The OPTIONS(model_type='ARIMA', time_series_timestamp_col='date', ...) clause is where most of the magic happens.
model_type: indicates that you are creating a ARIMA-based time series model.
time_series_data_col: indicates what column we will be forecasting
time_series_timestamp_col: indicates the time-series timestamp column
data_frequency: tells the model what format your input data is in. This is usually inferred automatically but explicitly defining it comes in handy sometimes
holiday_region (optional): this indicates the geographical region based on which the holiday effects are applied in modelling. By default, holiday effects modelling is disabled but enabling it allows a more accurate modelling for holiday points in your time-series.
After creating the model, you can use the ML.EVALUATE function to see the evaluation metrics of all the candidate models evaluated during the process, as many ARIMA models are run in parallel.
BigQuery selects the best performing model and uses it when you call the ML.FORECAST method (in the next step). If you aren’t happy with the results, you can play around with the time-series training data. For example, looking at weekly data as opposed to daily.
#standardSQLSELECT *FROM ML.EVALUATE(MODEL mydataset.sales_forecast)
The key step is in the query below. By calling the ML.FORECAST function, the model below forecasts 90 days into the future with a prediction interval of 80%.
#standardSQLSELECT  * FROM ML.FORECAST(MODEL `mydataset.sales_forecast`, STRUCT(90 AS horizon, 0.8 AS confidence_level))
And like that, we have sales forecasts for the next 90 days!
Now we can merge the forecast to our actuals to see how well our forecasts performs over time. We group the results by week to get a more holistic picture.
WITH actuals AS (SELECT  DATE_TRUNC(DATE(orderDate), WEEK) AS week, SUM(quantity) AS sales FROM `myproject.mydataset.mytable`WHERE orderDate > "2020-12-25"GROUP BY week)  SELECT  DATE_TRUNC(date(forecast_timestamp), WEEK) AS week, SUM(sales) AS actuals, SUM(prediction_interval_lower_bound) AS lower_value, SUM(forecast_value) AS middle_value, SUM(prediction_interval_upper_bound) AS upper_value FROM ML.FORECAST(MODEL `mydataset.sales_forecast`, STRUCT(90 AS horizon, 0.9 AS confidence_level)) as ML LEFT JOIN actuals AS ac  ON DATE_TRUNC(date(forecast_timestamp), WEEK) = ac.week GROUP BY 1
As you can see, our forecasts were pretty good and our actuals lie well within the forecasted value.
To forecast multiple time-series, all you have to do is include a new option called time_series_id_col. This specifies what column will be used to identify the different time-series. So looking at our example above, if we wanted to predict 90 day sales for different store locations, we will just need to add the time_series_id_col in the model creation and include a new column in the training data.
CREATE OR REPLACE MODEL mydataset.sales_forecastOPTIONS(model_type=’ARIMA’,time_series_data_col=’sales’,time_series_timestamp_col=’day’,time_series_id_col=’location’data_frequency=’DAILY’holiday_region=’US’) ASSELECT FORMAT_DATE(“%Y-%m-%d”, DATE(orderDate) as day,location,SUM(quantity) AS salesFROM   `myproject.mydataset.mytable`WHERE orderDate > cast(date_add(DATE_TRUNC((DATE_SUB(CURRENT_DATE, INTERVAL 8 MONTH)), MONTH), INTERVAL 1 day) as TIMESTAMP)GROUP BY day, locationORDER BY day, sales DESC
Now when you run ML.FORECAST like before, you get a 90-day sales prediction for each individual location in the store.
#standardSQLSELECT  * FROM ML.FORECAST(MODEL `mydataset.sales_forecast`, STRUCT(90 AS horizon, 0.8 AS confidence_level))
Make sure to give this a try in your next time-series project.
Feel free to share any suggestions you have on how the methods here could be improved.
You can become a Medium member and enjoy more stories like this.
References
8.1 Stationarity and differencing | Forecasting: Principles and Practice (otexts.com)
Detecting stationarity in time series data — KDnuggets
How to Remove Trends and Seasonality with a Difference Transform in Python (machinelearningmastery.com)