LSTM and Bidirectional LSTM for Regression | by Mohammed Alhamid | Towards Data Science

LSTM stands for Long Short-Term Memory, a model initially proposed in 1997 [1]. LSTM is a Gated Recurrent Neural Network, and bidirectional LSTM is just an extension to that model. The key feature is that those networks can store information that can be used for future cell processing. We can think of LSTM as an RNN with some memory pool that has two key vectors:
(1) Short-term state: keeps the output at the current time step.
(2) Long-term state: stores, reads, and rejects items meant for the long-term while passing through the network.
The decision of reading, storing, and writing is based on some activation functions as in Figure 1. The output from those activate functions is a value between (0, 1).
The forget and output gates decide whether to keep the incoming new information or throw them away. The memory of the LSTM block and the condition at the output gate produces the model decision. The output then is passed to the network again as an input making a recurrent sequence.
In bidirectional LSTM, instead of training a single model, we introduce two. The first model learns the sequence of the input provided, and the second model learns the reverse of that sequence.
Since we do have two models trained, we need to build a mechanism to combine both. It is usually referred to as the Merge step. Merging can be one of the following functions:
Sum
Multiplication
Averaging
Concatenation (default)
There are many problems that LSTM can be helpful, and they are in a variety of domains. LSTM models can be used to detect a cyber breach or unexpected system behavior, or fraud in credit card transactions. In these contexts, LSTM has one goal: predicting events that do not conform to expected patterns. To demonstrate a use-case where LSTM and Bidirectional LSTM can be applied in a real example, we will solve a regression problem predicting the number of passengers using the taxi cars in New York City. We can predict the number of passengers to expect next week or next month and manage the taxi availability accordingly.
LSTM for regression in Machine Learning is typically a time series problem. The critical difference in time series compared to other machine learning problems is that the data samples come in a sequence. The sequence represents a time dimension explicitly or implicitly. The implicit part is the timesteps of the input sequence.
This example will use an LSTM and Bidirectional LSTM to predict future events and predict the events that might stand out from the rest. The dataset used in this example can be found on Kaggle.
(1) Exploring the dataset
We load the dataset using Pandas to get the dataframe shown in Figure 2. The dataset has 10320 entries representing the passenger demand from July 2014 to January 2015. Plotting the demand values for the last six months of 2014 is shown in Figure 3.
As appears in Figure 3, the dataset has a couple of outliers that stand out from the regular pattern. Those high up-normal peaks or reduction in demand hint us to Look deeply at the context of the days. For instance, a snapshot of the demand on the holidays (December 24, 25) in Figure 4 holds unique data points that are not comparable to other days.
(2) Data Sequence and Feature Engineering
Looking into the dataset, we can quickly notice some apparent patterns. For instance, there are daily patterns (weekdays vs. weekends), weekly patterns (beginning vs. end of the week), and some other factors such as public holidays vs. working days. We consider building the following additional features that help us to make the model:
Adding day of a week in addition to the day of a month
The average of rides per hour for the same day of the week.
The number of rides during the day and the night.
# (1) Adding day of the week.df['Weekday'] = df.timestamp.dt.weekday# (2) Adding the average of rides grouped by the weekday and hour# 7 Days, 24 Hours = 168 valuesdf.avg_hour_day = df.avg_hour_day.replace(df[:7344].groupby(df.Weekday.astype(str) + ' ' + df.Hour.astype(str))['value'].mean().to_dict())# (3) Featuring the number of rides during the day and during the night.# We define the day time to be any hours between 6 AM and 10 PM while Night time where usually there is less # demand is any time between 10:00 PM and 6:00 AMdf['day_time'] = ((df['Hour'] >= 6) & (df['Hour'] <= 22)).astype(int)
Another look of the dataset after adding those features is shown in Figure 5.
Now, we would see the patterns of demand during the day hours compared to the night hours.
Like most ML models, LSTM is very sensitive to the input scale. We need to rescale the dataset. We will use the standard scaler from Sklearn.
(3) Understanding the Future
The model we are about to build will need to receive some observations about the past to predict the future. The past observations will not explicitly indicate the timestamp but will receive what we call a window of data points. The window has 48 data points: two records per hour for 24 hours per day, as in Figure 7.
X_train = create_sequence(X_train, 48)X_test  = create_sequence(X_test, 48)y_train = y_train[-X_train.shape[0]:]y_test  = y_test[-X_test.shape[0]:]
Each learning example consists of a window of past observations that can have one or more features. The target variable can be a single or a sequence of targets. In this example, the model learns to predict a single-step value, as shown in Figure 8. The input structure must be in the following format [training examples, time steps, features]. Our design has three features with a window of 48 timesteps, making the input structure be [9240, 48, 3].
Building a bidirectional LSTM using Keras is very simple. Keras provides a Bidirectional layer wrapping a recurrent layer. In our code, we use two bidirectional layers wrapping two LSTM layers supplied as an argument. We will show how to build an LSTM followed by an Bidirectional LSTM:
model = Sequential()model.add(LSTM(64,return_sequences=True, input_shape=(X_train.shape[1], X_train.shape[-1])))model.add(Dropout(0.5))model.add(LSTM(20,return_sequences=False))model.add(Dropout(0.5))model.add(Dense(1))model.compile(loss='mse', optimizer='rmsprop')
The return sequences parameter is set to True to get all the hidden states. For the Bidirectional LSTM, the output is generated by a forward and backward layer. The first bidirectional layer has an input size of (48, 3), which means each sample has 48 timesteps with three features each. The corresponding code is as follows:
# Building the modelmodel = Sequential()# Adding a Bidirectional LSTM layermodel.add(Bidirectional(LSTM(64,return_sequences=True, dropout=0.5, input_shape=(X_train.shape[1], X_train.shape[-1]))))model.add(Bidirectional(LSTM(20, dropout=0.5)))model.add(Dense(1))model.compile(loss='mse', optimizer='rmsprop')
(4) Results
Once we run the fit function, we can compare the model’s performance on the testing dataset. Figure 9 demonstrates the obtained results. The model achieved a great futuristic prediction. Although the model we built is simplified to focus on building the understanding of LSTM and the bidirectional LSTM, it can predict future trends accurately.
In this article, we learned what LSTM networks are and how to build a bidirectional network. Note that we mentioned LSTM as an extension to RNN, but keep in mind that it is not the only extension. For instance, Attention models, Sequence-to-Sequence RNN are examples of other extensions.
LSTM is helpful for pattern recognition, especially where the order of input is the main factor. We have seen in the provided an example how to use Keras [2] to build up an LSTM to solve a regression problem. Only part of the code was demonstrated in this article. You can find a complete example of the code with the full preprocessing steps on my Github. Thank you!
[1] Sepp Hochreiter, Jürgen Schmidhuber; Long Short-Term Memory. Neural Comput 1997; 9 (8): 1735–1780. doi: https://doi.org/10.1162/neco.1997.9.8.1735
[2] Keras, LSTM Layer, available on https://keras.io/api/layers/recurrent_layers/lstm/