Multi-variate LSTM Time Series Forecasting.
By: Bob Rupak Roy - Ii
Tags: Multivariate Analysis, Time Series Analysis, Sales, Forecasting, Machine Learning

Hi there how’s everything going? I hope it’s good.

Well in our last article we have seen how we can apply LSTM for time series forecasting when we have a single series of sequential data. Search for this 2 articles Times Series for Regression and Univariate LSTM Time Series Forecasting from my home page. Moving ahead to our next experiment using multiple series of input to understand and forecast the next time series sequential data. We will call it as Multi-variate Time Series Forecasting.

The article was originally found in ‘machine learning mastery’ by Jason. What we will try to achieve here is to simply the complex steps with different settings and understanding to get a clear picture about the concepts to quickly solve our day to day time series problem.

But before that let me repeat in brief what is LSTM

Long-Strong-Term Memory (LSTM) is the next generation of Recurrent Neural Network (RNN) used in deep learning for its optimized architecture to easily capture the pattern in sequential data. The benefit of this type of network is that it can learn and remember over long sequences and doesnot rely on pre-specified window lagged observation as input.

In keras this is referred as stateful and involves settings the “stateful” argument to “True” in LSTM layer

What is LSTM in brief?

It is a recurrent neural network that is trained by using Back-propagation through time and overcomes the vanishing gradient problem.

Now instead of having Neurons, LSTM networks have memory blocks that are connected through layers. The blocks of LSTM contains 3 non-linear gates that makes it smarter than a classical neuron and a memory for sequences. The 3 types of non-linear gates include

a.) Input Gate: decides which values from the input to update the memory state.

b.) Forget Gate: handles what information to throw away from the block

c.) Output Gate: finally handles what to be in output based on input and the memory gate.

Each LSTM unit is like a mini-state machine that utilizes a ”memory” cell that may maintain its state value over a longer time, where the gates of the units have weights that are learned during the training procedure.

There are tons of articles available in the internet about the workings of LSTM even the math behind LSTM. So here I will concentrate more for a quicker practical implementation of LSTM for our day to day problems.

Let’s get started.

First is data pre-processing step where we have to give structure the data into supervised learning that is X and Y format.

In simple words it identifies the strength and values of the relationship (positive/negative impact and the values derived is call quantification of impact) between one dependent variable(Y) and series of other independent variables X

For this example we have a retail sales time series data recorded over a period of time.

Now as u know supervised learning requires X & Y independent and dependent variable for the algorithm to learn /train, so we will first convert our data into such format

Assume we have sales data. What we will do we will first take the sales data(t) in our first column than the second column will have the next months(t+1)sales data that we will use to predict. Remember X & Y independent and dependent variable format where we use Y to predict the data.

Now we will replicate the same if we have more columns.

Let’s take an example of weather conditions data set where we will have attributes/ columns/ like pollution, dew, temp, wind speed, snow, rain. Now we will use Multivariate LSTM time series forecasting technique to predict the pollution for the next hours based on pollution, dew, temp, wind speed, snow, rain conditions.

We will use series to supervised function to frame the our dataset / variables of the dataset into t+1 and t format

#transform series to supervised learning

def series_to_supervised(data, n_in=1, n_out=1, dropnan=True):

n_vars = 1 if type(data) is list else data.shape[1]

df = DataFrame(data)

cols, names = list(), list() #input sequence (t-n, ... t-1)

for i in range(n_in, 0, -1):

cols.append(df.shift(i))

names += [('var%d(t-%d)' % (j+1, i)) for j in range(n_vars)] #forecast sequence (t, t+1, ... t+n)

for i in range(0, n_out):

cols.append(df.shift(-i))

if i == 0:

names += [('var%d(t)' % (j+1)) for j in range(n_vars)]

else:

names += [('var%d(t+%d)' % (j+1, i)) for j in range(n_vars)] #put it all together

agg = concat(cols, axis=1)

agg.columns = names #drop rows with NaN values

if dropnan:

agg.dropna(inplace=True)

return agg

Here is it after converting where we have all the variables in var(t-1) and var (t) is the same meaning as (t) (t+1) where we are using 1 step back to format it into X and Y variable.