How to reshape data and do regression for time series using LSTM | by Timur Bikmukhametov | Towards Data Science

Over the last years, recurrent neural networks in general and LSTM in particular, have become very popular among many data science practitioners. Despite the fact that the major application of LSTMs is Natural Language Processing (NLP), for computing scientists and engineers LSTMs offer possibilities for creating powerful regression models for problems where there is a time dependency in the data.
What you will learn
In this article, you will learn how to:
Reshape your input data for LSTM training
Fit LSTM to time series data with Keras
Handle univariant and multivariant datasets with LSTM
Confusions with input data shapes
One of the main reasons why this article is written is the fact that in neural network frameworks such as Keras, you need to provide the LSTM input data in a 3D format, which can be very confusing for many people. What makes it even more confusing is the way the input dimensions are named in Keras. According to the documentation and the source code, the Keras LSTM input data must be in the form: [batch_size, timesteps, input_dim]. Moreover, the batch_size may not need to be specified. Confusing. I know. Let’s check out how we can reshape our 1D and 2D data to 3D data shape such that LSTM works!
LSTM architectures and relation to input data shapes
Before starting with data reshaping, let us recap which LSTM architectures exist. Let’s look at the picture below:
We will not go through each of them. In my opinion, for time series problems the most useful ones are many-to-one and many-to-many (the last one in Fig.1), so we will cover them in more detail.
In Keras, the number of time steps is equal to the number of LSTM cells. This is what the word “time steps” means in the 3D tensor of the shape [batch_size, timesteps, input_dim].
To better understand this, let’s consider a toy (totally random) dataset, shown in Fig.2. In the data set, we have 2 features and 6 timesteps (since we use Python, we start counting from 0).
First, consider the many-to-one example.
In this case, what Keras wants from us is to reshape the data such that the prediction at time step t is computed based on the specified number of previous and the current timestep features. In the figure to the right which shows the LSTM architecture, the number of timesteps is set to 3.
The number of features is equal to the number of features in the 2D dataset. So, the word “ input_dim” in the 3D tensor of the shape [batch_size, timesteps, input_dim] means the number of the features in the original dataset. In our example, the “input_dim”=2
In order to reshape our original 2D data into 3D “sliding window” shape, as shown in Fig.2, we will create the following function:
def lstm_data_transform(x_data, y_data, num_steps=5):    """ Changes data to the format for LSTM training for sliding window approach """    # Prepare the list for the transformed data    X, y = list(), list()    # Loop of the entire data set    for i in range(x_data.shape[0]):        # compute a new (sliding window) index        end_ix = i + num_steps        # if index is larger than the size of the dataset, we stop        if end_ix >= x_data.shape[0]:            break        # Get a sequence of data for x        seq_X = x_data[i:end_ix]        # Get only the last element of the sequency for y        seq_y = y_data[end_ix]        # Append the list with sequencies        X.append(seq_X)        y.append(seq_y)    # Make final arrays    x_array = np.array(X)    y_array = np.array(y)    return x_array, y_array
The batch_size in the 3D tensor of the shape [batch_size, timesteps, input_dim] may not need to be specified and can be specified just to speed up the training process.
Now, let us see how we need to reshape the data for a many-to-many LSTM model. To do that, look at Fig. 3. Here, we have 2 features, 6 timesteps in the dataset and 3 timesteps in the LSTM neural network. As such, in order to transform the 2D data into the required 3D tensor, we may use the Numpy functionality as the following:
num_steps = 3num_features = 2x_shaped = np.reshape(x, newshape=(-1, num_steps, num_features))
Now, let us consider 2 examples, so we can fully understand how to reshape the data and use many-to-one and many-to-many LSTM models. In the first example, we will have a1D autoregression model of the sin wave, while in the second example we will have a 2D dataset with 3 features.
In this case, what we want to do is to predict the value of the function at time step t using the specified number of previous time steps. First, let’s generate the data.
x = np.arange(0, 200, 0.5).reshape(-1, 1)y = np.sin(x).reshape(-1, 1)
Imagine we want to use 10 previous time steps of the sin wave in order to predict the next value of the sin function. Then, to use LSTM for this task, we transform the data using the already defined function:
num_steps = 10x_new, y_new = lstm_data_transform(y, y, num_steps=num_steps)print ("The new shape of x is", x_new.shape)The new shape of x is (390, 10, 1)
Please note, that we pass y both for x and y argument of the function because we want to create an autoregressive model.
Great! Now, we have got the LSTM shape which should work for us. Next, we split our dataset such that 80% is taken for the model training and 20% is taken for the model testing.
train_ind = int(0.8 * x.shape[0])x_train = x_new[:train_ind]y_train = y_new[:train_ind]x_test = x_new[train_ind:]y_test = y_new[train_ind:]
Finally, we define a simple Keras model and train it!
model = Sequential()model.add(LSTM(100, activation='tanh', input_shape=(num_steps, 1),                return_sequences=False))model.add(Dense(units=50, activation='relu'))model.add(Dense(units=1, activation='linear'))adam = optimizers.Adam(lr=0.0001)model.compile(optimizer=adam, loss='mse')model.fit(x_train, y_train, epochs=20)
And then make predictions on the test set:
test_predict = model.predict(x_test)plt.style.use('ggplot')plt.figure(figsize=(20, 7))plt.plot(y_test, label="True value")plt.plot(test_predict.ravel(), label="Predicted value")plt.legend()
We have got quite a nice fit for the sin function! This provides you an example of how you can reshape your 1D data to 3D data to create an LSTM model with many-to-one architecture. Note, that here we made an autoregressive model and put y to both x_data and y_data inputs of the “lstm_data_transform”. Feel free to put you feature data to x_data, such that it transforms x_data to the required form.
In this example, we will consider the case where we have 3 features and 1 target variable and will use both many-to-many and many-to-one architectures. The target variable is the flow through a choke valve. If you do not know anything about this process, do not worry, this is just an illustrative example and the concept will entirely work for any problem at hand.
Let us look at the head of the data frame. Here in features, we have pressure before the choke (P_WHCU), after the choke (P_WHCD) and the choke opening (a value between 0 and 1). As such, we have 3 features and the 1 target variable which is the flow through the choke.
In the dataset, we have 6570 points. 80% is used for training and 20% for testing as before. Initially, I also rescale the data with the standard scaler applied to the train data and transforming both training and test data.
Splitting:
train_ind = int(0.8 * x_data.shape[0])x_train = x_data[:train_ind]x_test = x_data[train_ind:]y_train = y_data[:train_ind]y_test = y_data[train_ind:]
Scaling:
# scalersscaler_x = StandardScaler()scaler_y = StandardScaler()# scalingx_train_sc = scaler_x.fit_transform(x_train)x_test_sc = scaler_x.transform(x_test)y_train_sc = scaler_y.fit_transform(y_train)y_test_sc = scaler_y.transform(y_test)
Many-to-many training
Let us assume we want to use 3 previous time step features in order to predict the current value of the flow through the choke. First, we reshape the data as the following:
num_steps = 3# training setx_train_shaped = np.reshape(x_train_sc, newshape=(-1, num_steps, 3))y_train_shaped = np.reshape(y_train_sc, newshape=(-1, num_steps, 3))assert x_train_shaped.shape[0] == y_train_shaped.shape[0]# test setx_test_shaped = np.reshape(x_test_sc, newshape=(-1, num_steps, 3))y_test_shaped = np.reshape(y_test_sc, newshape=(-1, num_steps, 3))assert x_test_shaped.shape[0] == y_test_shaped.shape[0]
The last argument of the “newshape” argument is equal to 3 because we have 3 features.
Now, we compile the model. Please, note that we need to use return_sequences=True for the many-to-many model:
model = Sequential()model.add(LSTM(20, activation='tanh', input_shape=(num_steps, 3), return_sequences=True))model.add(Dense(units=20, activation='relu'))model.add(Dense(units=1, activation='linear'))adam = optimizers.Adam(lr=0.001)model.compile(optimizer=adam, loss='mse')
Fit the model and predict on the test data:
model.fit(x_train_shaped, y_train_shaped, epochs=10)test_predict = model.predict(x_test_shaped)
We see that the prediction is good except for some fluctuating behavior, which can be caused by overfitting and reduced by better model training. Now, let’s consider the same dataset, but with many-to-one architecture.
Many-to-one training
As before, we need to reshape our data, but now we need to use the pre-built function:
num_steps = 3# training set(x_train_transformed, y_train_transformed) = lstm_data_transform(x_train_sc, y_train_sc, num_steps=num_steps)assert x_train_transformed.shape[0] == y_train_transformed.shape[0]# test set(x_test_transformed, y_test_transformed) = lstm_data_transform(x_test_sc, y_test_sc, num_steps=num_steps)assert x_test_transformed.shape[0] == y_test_transformed.shape[0]
After this, the dataset is ready for training. We compile the same model as for many-to-many. Please, note that we need to use return_sequences=False for the many-to-one model:
model = Sequential()model.add(LSTM(20, activation='tanh', input_shape=(num_steps, 3), return_sequences=False))model.add(Dense(units=20, activation='relu'))model.add(Dense(units=1, activation='linear'))adam = optimizers.Adam(lr=0.001)model.compile(optimizer=adam, loss='mse')
Then we fit the model and make predictions on the test set:
model.fit(x_train_transformed, y_train_transformed, epochs=10)test_predict = model.predict(x_test_shaped)
As we see, we have got a similar fit as with the many-to-many model. However, this is a simple dataset, and for many problems, the results can be different.
In this article, we considered how to use Keras LSTM models for time series regression. We showed how we need to transform 1D and 2D datasets into 3D tensors such that LSTM works for both many-to-many and many-to-one architectures. In the many-to-many case, we can use Numpy functionality, while for many-to-one we need to use a transforming function. Also, for many-to-many, the Keras parameter return_sequence must be equal to True, while for the many-to-one model it must be equal to False. I hope you find this article useful and it will allow using LSTM models more often in your research and day-to-day work since now!