Prediction task with Multivariate Time Series and VAR model. | by Alena Nazarava | Towards Data Science

Time Series data can be confusing, but very interesting to explore. The reason this sort of data grabbed my attention is that it can be found in almost every business (sales, deliveries, weather conditions etc.). For instance: using Google BigQuery how to explore weather effects on NYC link.
The main steps in the task:
Problem definitionGathering informationExploratory AnalysisChoosing and fitting modelsUsing and evaluating a forecast model
Problem definition
Gathering information
Exploratory Analysis
Choosing and fitting models
Using and evaluating a forecast model
We want to predict the future values of the series using current information from the dataset. This information contains current and past values of the series. There are lots of projects with univariate dataset, to make it a bit more complicated and closer to a real life problem, I chose a multivariate dataset. Multivariate time series analysis considers simultaneous multiple time series that deals with dependent data. (The dataset contains more than one time-dependent variable.)
I want to make a weather forecast. The task of predicting the state of the atmosphere at a future time and a specified location using a statistical model.
I will be using data from the following sources:
NOAA. Climate Data Online (CDO) offers web services that provide access to available data through API. linkGoogleCloud BigQuery public data. linkKaggle. link You can download from this page and run in your env or use kaggle account and use it in your notebook, that kaggle provide.
NOAA. Climate Data Online (CDO) offers web services that provide access to available data through API. link
GoogleCloud BigQuery public data. link
Kaggle. link You can download from this page and run in your env or use kaggle account and use it in your notebook, that kaggle provide.
Getting Data from NOAA
In order to access the CDO web services a token must be first obtained from the token request page. Documentation is straight-forward and should not take a lot time to understand.
Example: To find information about weather in Portland, Maine you need to know: State_id and coordinates of Portland. At first I will get the list of all states.
def get_states_list(self):    url = conf.noaa['host']+'locations?locationcategoryid=ST&limit=50'    headers = dict(token=conf.noaa['token'])    r = requests.get(url, headers=headers)    data = json.loads(r.content)    states_list = json_normalize(data['results'])    return states_list
Then, using parameter ‘name’: ‘Maine’ find information about the particular state:
def state_info(self, state):    dataset = NoaaApi()    states_info = dataset.get_states_list()    df_states = pd.DataFrame(states_info)    id = df_states[(df_states.name == state)].id    return id
Results: Where id of state Maine is: ‘FIPS:23’
dataset.state_info('Maine')State id: FIPS:23
The next step is to get Stations near Portland, Portland’s coordinates could be found easily in google map ( 43.6591 N, 70.2568 W Portland).
The list of Stations in the Maine state could be found with the following GET request.
def get_stations_list(self):    url = conf.noaa['host']+'stations?locationid=FIPS:23&limit=687'    headers = dict(token=conf.noaa['token'])    r = requests.get(url, headers=headers)    data = json.loads(r.content)    stations_list = json_normalize(data['results'])    return stations_list
I do not need all stations in the state, I would like to find Stations just near Portland. To do this I will specify coordinates.
def station_info(self):    dataset = NoaaApi()    stations_list = dataset.get_stations_list()    df_stations = pd.DataFrame(stations_list)    portland_st = df_stations[(df_stations.latitude > 43.5)                           & (df_stations.latitude < 43.7)                           & (df_stations.longitude > -70.3)                           & (df_stations.longitude < -70.1)]    return portland_st
Conclusion: We got list of Stations near Portland, that can be used in the feature to get weather data in that region.
Getting Data from Kaggle
There are bunch of datasets that provide opportunities to learn and improve your skills as well as participate in competitions where you can earn money and show off your skills. I found Daily Global Historical Climatology Network (BigQuery Dataset) The whole description could be found in the overview page: link . A perfect chance to practice your BigQuery skills.
import numpy as np import pandas as pd import osimport bq_helperfrom bq_helper import BigQueryHelpernoaa = bq_helper.BigQueryHelper(active_project="bigquery-public-data", dataset_name="ghcn_d")
Create a query and estimate size:
#43.6591° N, 70.2568° W  Portland/Coordinates Maine 45.2538° N, 69.4455° WqueryStations = """SELECT  id,  name,  state,  latitude,  longitudeFROM  `bigquery-public-data.ghcn_d.ghcnd_stations`WHERE  latitude > 43.5  AND latitude < 44  AND longitude > -70.4  AND longitude < -69.4;   """  noaa.estimate_query_size(queryStations)
Kaggle BigQuery.helper provides estimation for query to avoid exceed limit.
0.0065232375636696815
Now run the query:
stations_portland = noaa.query_to_pandas_safe(queryStations)stations_portland.head(20)
As we can see, we got almost the same list of Stations.
Getting Data from GoogleCloud
Google provides 30GB of public weather data. There is the possibility to have a free account and practice to improve your skills in Bigquery. Also there is the possibility to select data that you need and download the dataset for further investigation. Super useful for private use as well as for companies. I will not try to find list of Stations, but will request daily temperature from the Station that was found.
SELECT  TIMESTAMP(CONCAT(year,"-",mo,"-",da)) AS date,  AVG(IF (temp=9999.9, 0, temp)) AS temperature,  MIN(IF (temp=9999.9, 0, min)) AS temperature_min,  MAX(IF (temp=9999.9, 0, max)) AS temperature_max,  AVG(IF (visib=999.9, 0, visib)) AS visibility,  AVG(IF (gust=999.9, 0, gust)) AS wind_gust,  AVG(IF (wdsp="999.9", 0, CAST(wdsp AS Float64))) AS wind_speed,  AVG(IF (prcp=99.99, 0, prcp)) AS precipitation,  AVG(IF (sndp=999.9, 0, sndp)) AS snow_depthFROM  `bigquery-public-data.noaa_gsod.gsod20*`WHERE  CAST(YEAR AS INT64) > 2008  AND (stn="726060") GROUP BY  dateORDER BY  date
This query will process 5.19 GB when run.
This query will return daily weather from one station in city Portland, Maine for the last 10 years.
At first I would like to visualise important information, to make it clear what kind of data we have.
I will use data from the city of Portland, Maine to plot the locations of weather Stations, using GeoPandas library. The first step is to find and download a shape-file(.shp file). I already showed how to get Stations near Portland using NOAA API(portland_st)
highway_map = gpd.read_file('./maine_highway/maine_highway.shp')crs = {'init': 'epsg:4326'}point = [Point(xy) for xy in zip(portland_st['longitude'], portland_st['latitude'])]geo_stations = gpd.GeoDataFrame(portland_st, crs=crs, geometry=point)fig, ax = plt.subplots(figsize=(5, 5))highway_map.plot(ax=ax, alpha=0.4, color='grey')geo_stations.plot(ax=ax, color='red', marker='o')plt.legend(prop={'size': 5})plt.show()
I found the shape-file (‘maine_highway.shp’) that shows main roads in Portland and combine it with the Stations location. Red dots are the locations of the weather stations. Now it’s obvious which station is needed for the feature model. It is an easy way to visualise geospatial data.
The goal of time series analysis is to find patterns in the data and use the data for predictions.
It is also useful to explore relationships between time series. Let’s look at subplots with all features for the year 2017:
For sure we want to look at the series that is going to be forecast. I did it using iplot library.
df[['temperature', 'temperature_min']].iplot(    mode='lines+markers',    opacity=0.8,    size=8,    symbol=1,    xTitle='Time',    yTitle='Temperature',    title='Temperature over last 10 years')
Just for curiosity’s sake I can illustrate line plot that shows avg/min/max temperature at New Year’s for the last 10 years:
sns.set(rc={'figure.figsize': (14.7, 8.27)})g = sns.lineplot(data=newyear.temperature_max, palette="tab10", linewidth=2.5, label="max")g = sns.lineplot(data=newyear.temperature_min, palette="tab10", linewidth=2.5, label="min")g = sns.lineplot(data=newyear.temperature, palette="tab10", linewidth=2.5, label="avg")g.tick_params(labelsize=10)plt.xticks(rotation=45)plt.show()
Temperature during the period from 27.12 till 03.01 for 10 years. In case someone would need historical statistics from the past years only for a defined period, let say New Year’s week.
If the main purpose is only to prepare data and select a model. The right solution would be to plot four graphs when looking at particular series.
As well as looking at the time plot of the data, the ACF plot is also useful for identifying non-stationary time series. For a stationary time series, the ACF will drop to zero relatively quickly, while the ACF of non-stationary data decreases slowly.
def tsplot(y, title, lags=None, figsize=(10, 6)):    fig = plt.figure(figsize=figsize)    layout = (2, 2)    ts_ax = plt.subplot2grid(layout, (0, 0))    hist_ax = plt.subplot2grid(layout, (0, 1))    acf_ax = plt.subplot2grid(layout, (1, 0))    pacf_ax = plt.subplot2grid(layout, (1, 1))    y.plot(ax=ts_ax)    ts_ax.set_title(title, fontsize=12, fontweight='bold')    y.plot(ax=hist_ax, kind='hist', bins=25)    hist_ax.set_title('Histogram')    sm.tsa.graphics.plot_acf(y, lags=lags, ax=acf_ax)    sm.tsa.graphics.plot_pacf(y, lags=lags, ax=pacf_ax)    sns.despine()    plt.tight_layout()    plt.show()    return ts_ax, acf_ax, pacf_ax
In a series of daily temperatures, the mean temperature is not constant, since it varies by day, but for the same month in different years we can expect a constant average value.
The seasonal period, s, defines the number of observations that make up a seasonal cycle. For example, s = 12 for monthly series, s = 4 for quarterly series, s = 365 for daily series.
However, this may not be precisely true: for example, if we have daily data and the seasonal period is the length of the month, s will be approximately 30, but it will vary from month to month. And the trickiest thing is, there may be more than one type of seasonality. For example, with daily data we can have monthly seasonality, with s = 30 and yearly, with s = 365.
Daily data are challenging for a different reason — they often involve multiple seasonal patterns, and so we need to use a method that handles such complex seasonality.
Time Series data might also have other patterns, such as trends or cyclic. And all of this shows that the data is not stationary.
To use statistical model such as the VAR model we should take care of these patterns before the data can be fit into the model. For the VAR model you need stationary condition to be performed. Put enough structure into the model to identify some of the parameters in a model.
Applying seasonal differencing to the log of the series should make the above series stationary. It is quite challenging to define parameters for climate data, that are a lot of publications about this topic. I will not dig deep inside and choose just simple parameters lag=365 and diff=1. More information about parameters you can find in Trend analysis of climate time series publication.
A stationary time series is one whose properties do not depend on the time at which the series is observed. Time series with seasonality, are not stationary, because seasonality will affect the value of the time series at different times.
transformed = df.copy()transformed['temperature'] = np.log(df.iloc[:, 0]).diff(1)transformed['wind_speed'] = np.log(df.iloc[:, 1]).diff(1)
The purpose of transformation is make the series stationary.
The next step is to split the data.
Split the samples into training/testing and validation set. It needs to be split carefully regarding the dependency structure. The approach that can mostly fit to this case is a rolling window. But in this particular example I will just leave a few observations at the end to validate my model. I want to forecast 7 days. Later to check the performance of the model I will use these 7 true values.
This kind of model required the series to be stationary, if the series are not stationary, the individual series have to be transformed to be stationary.
A VAR(p) in two variables can be written in matrix form (where p=1) as
model = sm.tsa.VARMAX(y_train, order=(5, 0), trend='c')model_result = model.fit(maxiter=1000, disp=False)model_result.summary()
The way to select an order is to use information criteria AIC, BIC, HQIC. Estimate a range of models and choose the best one. The most common approach for model order selection involves minimizing one or more information criteria evaluated over a range of model orders.
for i in range(5):    i += 1    model = sm.tsa.VARMAX(y_train, order=(i,0))    model_result = model.fit(maxiter=1000, disp=False)    print('Order = ', i)    print('AIC: ', model_result.aic)    print('BIC: ', model_result.bic)    print('HQIC: ', model_result.hqic)
more details about how to choose right order you can find in the book New Introduction to Multiple Time Series Analysis by Lütkepohl, Helmut (chapter 4). In this particular example I will use p = 5, where AIC criteria reach their minimum.
The next step is the model diagnostic and checking that underlying assumption is true. The VARMAX class in Statsmodels allows estimation of VAR, VMA, and VARMA models (through the order argument) The VARMAX procedure is a well-established and powerful tool for analysing multivariate time series.
model = sm.tsa.VARMAX(y_train, order=(5, 0))model_result = model.fit(maxiter=1000, disp=False)model_result.plot_diagnostics()plt.show()
A common overall diagnostic is the plot of the residuals and to see whether the residuals obey the models assumptions.
It can be possible to apply tests:
tests to confirm normalityabsence of serial correlationabsence of heteroscedasticity
tests to confirm normality
absence of serial correlation
absence of heteroscedasticity
The two series do not exhibit trends or regular seasonal patterns. Therefore we assume that they are the realisation of a stationary VAR(p) process.
To recap: We transformed temperature and wind speed with the following equation, where yt is an original series:
This means that it should be transformed back to define and estimate real results with predicted. For this purpose we calculate the equation and apply to predicted values:
z = model_result.forecast(steps=7)
Transform values and print a result:
pd.concat([df['temperature'][-7:], forecast], axis=1)
Calculate Mean Squared Error:
rmse = sqrt(mean_squared_error(df['temperature'][-7:], forecast))RMSE: 4.200
Conclusion: Time Series data can be very tricky to deal with correctly. You need to properly explore your data to understand it, be aware of any seasonality or other patterns. It is not always possible to use the same approach. In general is best to start with simple models and build your analysis from there.
AM207: Pavlos Protopapas, Harvard UniversityForecasting: Principles and Practice, Rob J Hyndman and George AthanasopoulosJeffrey Yau, PyData Conference 2017New Introduction to Multiple Time Series Analysis by Lütkepohl, HelmutGoogle BigQuery to explore weather effects
AM207: Pavlos Protopapas, Harvard University
Forecasting: Principles and Practice, Rob J Hyndman and George Athanasopoulos
Jeffrey Yau, PyData Conference 2017
New Introduction to Multiple Time Series Analysis by Lütkepohl, Helmut
Google BigQuery to explore weather effects