Title: Long Short-Term Memory Networks - PyImageSearch

URL Source: https://pyimagesearch.com/2022/08/01/long-short-term-memory-networks/

Published Time: 2022-08-01T13:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [Long Short-Term Memory Networks](#h2BPTitle)

*   [Introduction](#h3Introduction)
*   [Configuring Your Development Environment](#h3Configuring)
*   [Having Problems Configuring Your Development Environment?](#h3Having)
*   [Project Structure](#h3Project)
*   [Modeling Sequential Data with RNNs](#h3Modeling)
*   [The Problem with RNNs](#h3RNNs)
*   [Long Short-Term Memory Networks](#h3Networks)
*   [Vanishing and Exploding Gradients](#h3Vanishing)
*   [Training and Visualizations](#h3Training)
*   [Loading and Inference](#h3Loading)

*   [Summary](#h2Summary)

*   [Citation Information](#h3Citation)

* * *

In this tutorial, we will learn how to train a movie review sentiment classification model using [Long Short-Term Memory](http://www.bioinf.jku.at/publications/older/2604.pdf) (LSTM) Networks introduced by Hochreiter and Schmidhuber. In our previous tutorial, [Introduction to Recurrent Neural Networks with Keras and TensorFlow](https://pyimg.co/a3dwm), we were introduced to the task of movie review sentiment classification.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/08/long-short-term-memory-networks-featured.png)

As before, we will use the [imdb\_reviews](https://www.tensorflow.org/datasets/catalog/imdb_reviews), a dataset of 25,000 highly polar movie reviews.

We will also understand some of the shortcomings of Recurrent Neural Networks and how to circumvent them using Long Short-Term Memory Networks.

This lesson is the 2nd in a 3-part series on **NLP 102**:

1.  [_Introduction to Recurrent Neural Networks with Keras and TensorFlow_](https://pyimg.co/a3dwm)
2.  [**_Long Short-Term Memory Networks_**](https://pyimg.co/wtby9) **(today’s tutorial)**
3.  _Neural Machine Translation_

****To learn how to build a Long Short-Term Memory Network for sentiment classification using TensorFlow and Keras, _just keep reading._****

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

* * *

[**Long Short-Term Memory Networks**](#TOC)
-------------------------------------------

* * *

### [**Introduction**](#TOC)

At its core, LSTMs are an extension of Recurrent Neural Networks. RNNs are great for modeling sequential data, but they strictly do not have much efficiency in retaining information for a long period.

This means a word at the beginning of a long sentence will lose its context by the end. Not a very useful feature for a language or sequence model.

LSTMs are aimed at solving this problem. Typically as the name suggests, LSTMs aim to retain long-term information from the sequence.

Before we begin this tutorial, it is advisable to go through the [previous tutorial](https://pyimg.co/a3dwm) in this series to familiarize ourselves with the code and project structure. A number of modules have been reused from the RNN code.

* * *

### [**Configuring Your Development Environment**](#TOC)

To follow this guide, you need to have TensorFlow, TensorFlow Datasets, and the Matplotlib library installed on your system.

Luckily, all are pip-installable:

$ pip install tensorflow
$ pip install tensorflow\_datasets
$ pip install matplotlib

* * *

### [**Having Problems Configuring Your Development Environment?**](#TOC)

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-university/)

**Figure 1:** Having trouble configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) — you’ll be up and running with this tutorial in a matter of minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code** **_right now_** **on your Windows, macOS, or Linux system?**

Then join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are** **_pre-configured_** **to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

* * *

### [**Project Structure**](#TOC)

We first need to review our project directory structure.

Start by accessing the **_“Downloads”_** section of this tutorial to retrieve the source code and example images.

From there, take a look at the directory structure:

$ tree -dirsfirst
.
|\_\_\_\_ output
| |\_\_\_\_ lstm\_plot.png
| |\_\_\_\_ rnn\_plot.png
|\_\_\_\_ pyimagesearch
| |\_\_\_\_ plot.py
| |\_\_\_\_ save\_load.py
| |\_\_\_\_ config.py
| |\_\_\_\_ standardization.py
| |\_\_\_\_ \_\_init\_\_.py
| |\_\_\_\_ model.py
| |\_\_\_\_ dataset.py
|\_\_\_\_ train.py
|\_\_\_\_ inference.py
|\_\_\_\_ terminal\_output.txt

In the `pyimagesearch` directory, we have:

*   `plot.py`: Script to help us visualize outputs.
*   `save_load.py`: Script to save and load the trained model weights.
*   `config.py`: Script containing the entire configuration pipeline.
*   `standardization.py`: Script containing utilities to help us prepare the data pipeline.
*   `__init__.py`: Script that turns the directory into a python package.
*   `model.py`: Script housing the model.
*   `dataset.py`: Script to help us load the data to our project.

In the core directory, we have two scripts:

*   `train.py`: Script to train the LSTM model.
*   `inference.py`: Script to draw inference from our trained LSTM model.

**_Note:_** The code download for this blog post also contains code snippets for Recurrent Neural Networks (RNNs). These are covered in a previous blog post: [Introduction to Recurrent Neural Networks with Keras and TensorFlow](https://pyimg.co/a3dwm).

* * *

### [**Modeling Sequential Data with RNNs**](#TOC)

In the [previous tutorial](https://pyimg.co/a3dwm), we modeled sequential data with Recurrent Neural Networks. We learned how the RNN learns from the current input and all the previous hidden states. This is demonstrated in **Figure 2**.

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20293'%3E%3C/svg%3E)

**Figure 2:** Diagram of an unfolded RNN cell.

However, RNNs are not perfect when it comes to modeling sequential data. Certain shortcomings need to be addressed to improve its architecture.

* * *

### [**The Problem with RNNs**](#TOC)

Recurrent Neural Networks have a _serious_ problem, and the problem stems from their strength: **the recurrence**. To fully appreciate the situation, one needs to look at the gradients and the backpropagation formulae for RNNs.

We begin with the recurrence formula.

![Image 5: {(1)} \qquad h_{t} \ =\ \tanh\left( W_{hh} h_{t-1} +W_{xh} x_{t}\right)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20268%2018'%3E%3C/svg%3E)

Now we take the derivative of the current hidden state with the previous one. Hold on to this equation, as we will need it later.

![Image 6: (2) \qquad \displaystyle\frac{\partial h_{t}}{\partial h_{t-1}} =\tanh^{\prime}( W_{hh} h_{t-1} +W_{xh} x_{t}) W_{hh}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20317%2038'%3E%3C/svg%3E)

So to backpropagate in an RNN, we calculate the gradient loss with respect to the weights. Unfortunately, due to recurrence, the chain rule looks a little intimidating.

To condense the terms, we can represent them with a product notation.

![Image 7: \text{(3a)} \qquad \left.\displaystyle\frac{\partial L}{\partial W_{hh}} \ = \ \left(\displaystyle\frac{\partial L}{\partial h_{t}}\right)   \left(\displaystyle\frac{\partial h_{t}}{\partial h_{t-1}} \right)   \left(\displaystyle\frac{\partial h_{t-1}}{\partial h_{t-2}}\right)   \left(\displaystyle\frac{\partial h_{t-2}}{\partial h_{t-3}}\right) \dotsc \left(\displaystyle\frac{\partial h_{1}}{\partial W_{hh}}\right)\right.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20488%2041'%3E%3C/svg%3E)

![Image 8: \text{(3b)} \qquad \left.\displaystyle\frac{\partial L}{\partial W_{hh}} \ = \ \left(\displaystyle\frac{\partial L}{\partial h_{t}}\right) \left(\prod _{t=2}^{T}\displaystyle\frac{\partial h_{t}}{\partial h_{t-1}}\right) \left(\displaystyle\frac{\partial h_{1}}{\partial W_{hh}}\right)\right.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20354%2051'%3E%3C/svg%3E)

Let’s expand equation (3b) using equation (2). Finally, the gradient formula looks like this.

![Image 9: \text{(4)} \qquad \displaystyle\frac{\partial L}{\partial W_{hh}} \ = \  \displaystyle\frac{\partial L}{\partial h_{t}}\left(\sum _{t=2}^{T}\tanh^{\prime}( W_{hh} h_{t-1} +W_{xh} x_{t}) W_{hh}^{T-1}\right) \displaystyle\frac{\partial h_{1}}{\partial W_{hh}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20468%2051'%3E%3C/svg%3E)

Now that we are done with the equations, let’s drive the point home. In the interactive demo, we plotted three graphs:

*   tanh
*   derivative of tanh
*   product of the derivative term

Do you notice something?

The graph goes closer to zero as the number of product terms increases. Likewise, as the recurrence increases, the derivative term will get closer to zero. In other words, recurrence can lead to the “vanishing gradient.”

Vanishing gradient is notoriously difficult to manage. It is proven to be a bottleneck for RNNs. With gradients vanishing, the weights are seldom optimized, and the network learns poorly.

* * *

### [**Long Short-Term Memory Networks**](#TOC)

In a vanilla RNN, the gradient flow was faulty. As a consequence, the RNN did not learn a longer context. With LSTMs, we want to counter the problem. The architecture of Long Short-Term Memory Networks provides a better way for the gradients to backpropagate.

In this section, we see how to attain a better gradient flow with LSTMs. Let’s consider the following **Figure 3** as the diagram of LSTM.

![Image 10](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20384'%3E%3C/svg%3E)

**Figure 3:** Diagram of an LSTM cell.

![Image 11: \begin{pmatrix} f\\ i\\ g\\ o\end{pmatrix} =\begin{pmatrix} \sigma \\ \sigma \\ \tanh\\ \sigma \end{pmatrix} W^{l}\begin{pmatrix} h_{t-1}\\ x_{t} \end{pmatrix}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20196%2080'%3E%3C/svg%3E)

![Image 12: c_{t} =f\odot c_{t-1} +i\odot g](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20145%2016'%3E%3C/svg%3E)

![Image 13: h_{t} =o\odot \tanh c_{t}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20111%2016'%3E%3C/svg%3E)

When we started with LSTMs, the image and formulae were not intuitive to us at all. If you feel lost here, we can assure you things will ease up as we go a little further.

The LSTM cell has two recurrence states, ![Image 14: c_{t}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2011%2010'%3E%3C/svg%3E), the memory highway, and the ![Image 15: h_{t}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2014'%3E%3C/svg%3E), the hidden state representation. During feedforward, the inputs ![Image 16: (h_{t-1})](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2040%2018'%3E%3C/svg%3E) and ![Image 17: x_{t}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2010'%3E%3C/svg%3E) are concatenated as a single vector ![Image 18: z_{t}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2012%2010'%3E%3C/svg%3E) for the simplicity and efficiency of calculations.

This input ![Image 19: z_{t}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2012%2010'%3E%3C/svg%3E) is then passed through multiple gates. The functions ![Image 20: f](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%2014'%3E%3C/svg%3E), ![Image 21: i](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%205%2012'%3E%3C/svg%3E), ![Image 22: g](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%2010'%3E%3C/svg%3E), ![Image 23: o](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%207'%3E%3C/svg%3E) are termed as gates of the LSTM architecture. They provide the intuition of how much a particular data needs to travel to represent it better.

A gate in the LSTM architecture is a multilayer perceptron with a non-linearity function. The choice of the non-linearity function will make sense as we study in-depth the different gates.

The Gates:

*   **Forget Gate** ![Image 24: f](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%2014'%3E%3C/svg%3E): This gate is concerned with how much to _forget_. It takes in the input ![Image 25: z_{t}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2012%2010'%3E%3C/svg%3E) and then decides how much of the previous memory state ![Image 26: c_{t-1}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2025%2011'%3E%3C/svg%3E) should be forgotten. The activation non-linearity is ![Image 27: \sigma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%207'%3E%3C/svg%3E). The gate’s output is 0 to 1. The 0 means to forget everything, while the 1 means to remember everything. This gate acts as a switch for the memory state circuit.![Image 28: f \ = \ \sigma W_{f}\begin{pmatrix} h_{t-1}\\ x_{t} \end{pmatrix}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20129%2041'%3E%3C/svg%3E)
    
    After forgetting, we have the amount of memory state that we need from the previous step:
    
    ![Image 29: c_{t\_f} \ = \ f\odot c_{t-1}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20111%2016'%3E%3C/svg%3E)
    

*   **Input gate** ![Image 30: i](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%205%2012'%3E%3C/svg%3E): This gate decides how much of the present input needs to flow. This acts as a switch for the present input circuit. This gate also uses a ![Image 31: \sigma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%207'%3E%3C/svg%3E) non-linearity function.![Image 32: i \ = \ \sigma W_{i}\begin{pmatrix} h_{t-1}\\ x_{t} \end{pmatrix}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20121%2041'%3E%3C/svg%3E)
    
*   **Gate gate** ![Image 33: g](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%2010'%3E%3C/svg%3E): This gate closely resembles the recurrence formula of a vanilla RNN. We can say that this gate is the hidden state of the RNN in an LSTM. The resemblance to the RNN formula intensifies upon noticing the non-linearity function. This gate is the only one that uses a ![Image 34: \tanh](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2032%2014'%3E%3C/svg%3E) function.![Image 35: g \ = \ \tanh W_{g}\begin{pmatrix} h_{t-1}\\ x_{t} \end{pmatrix}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20152%2041'%3E%3C/svg%3E)
    
    The Input gate and the Gate gate usage will make sense now. This is because the input gate behaves like a switch to the output of the Gate gate.
    
    ![Image 36: h_{t\_i} \ = \ i \odot g](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2089%2015'%3E%3C/svg%3E)
    
    Upon pointwise addition of ![Image 37: c_{t\_f}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2022%2012'%3E%3C/svg%3E) and ![Image 38: h_{t\_i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2022%2014'%3E%3C/svg%3E) we get the present memory state. The memory state not only holds the past and present information but also holds a definite amount of both to make a better representation possible.
    
    ![Image 39: c_{t}^{l} \ = \ c_{t\_f}+h_{t\_i}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20110%2019'%3E%3C/svg%3E)
    

*   **Output gate** ![Image 40: o](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%207'%3E%3C/svg%3E): This gate is responsible for deciding how much output will flow into making the present hidden state.![Image 41: o \ = \ \sigma W_{o}\begin{pmatrix} h_{t-1}\\ x_{t} \end{pmatrix}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20125%2041'%3E%3C/svg%3E)
    

*   Let us pass the memory state from a ![Image 42: \tanh](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2032%2014'%3E%3C/svg%3E) first.![Image 43: c_{t\_o} \ = \ \tanh c_{t}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20101%2016'%3E%3C/svg%3E)
    

And now, we begin to build our LSTM model. But, first, let us go through the `model.py` file.

\# import the necessary packages
from tensorflow.keras import layers
from tensorflow import keras

We begin with the necessary imports on **Lines 1-3**.

def get\_lstm\_model(vocabSize):
	# input for variable-length sequences of integers
	inputs = keras.Input(shape=(None,), dtype="int32")

	# embed the tokens in a 128-dimensional vector with masking
	# applied and apply dropout
	x = layers.Embedding(vocabSize, 128, mask\_zero=True)(inputs)
	x = layers.Dropout(0.2)(x)

	# add 3 LSTMs
	x = layers.LSTM(64, return\_sequences=True)(x)
	x = layers.LSTM(64, return\_sequences=True)(x)
	x = layers.LSTM(64)(x)

	# add a classifier head
	x = layers.Dense(units=64, activation="relu")(x)
	x = layers.Dense(units=32, activation="relu")(x)
	x = layers.Dropout(0.2)(x)
	outputs = layers.Dense(1, activation="sigmoid")(x)
	
	# build the LSTM model
	model = keras.Model(inputs, outputs, name="LSTM")
	
	# return the LSTM model
	return model

On **Line 31**, we define the `get_lstm_model` that we use to create the model. Next, on **Line 37**, we pad and mask the variable length sequence and embed them on a 128-dimensional vector. For those who are a bit unclear about this, consider this 128-dimensional matrix as a way the computer views and assigns meaning to text.

On **Line 38**, we apply dropout to the inputs. Then, we add 3 simple LSTM cells on **Lines 41-43** and a classifier head with a sigmoid activation on **Lines 46-49**.

Finally, the model is built using keras.Model API on **Line 52** and returned on **Line 55**.

* * *

### [**Vanishing and Exploding Gradients**](#TOC)

Here, we see why LSTMs are preferred over RNNs.

Upon looking at **Figure 4**, we notice that the gradients of the memory state ![Image 44: \partial{c_{t}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2020%2015'%3E%3C/svg%3E) flow without much perturbation. Apart from the element-wise multiplication with the forget gate, the gradient ![Image 45: \partial{c_{t}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2020%2015'%3E%3C/svg%3E) flows freely in the circuit provided for the memory state.

![Image 46](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20408'%3E%3C/svg%3E)

**Figure 4:** Diagram of gradients backpropagating through all the gates of an LSTM node.

LSTMs provide a **gradient highway** for the gradients to backpropagate efficiently. This makes the architecture robust to vanishing or exploding gradients.

* * *

### [**Training and Visualizations**](#TOC)

With the model created and ready, we can finally begin our training procedure. But before that, we must first look at some helper functions that will assist with visualizations and saving.

We begin with a function inside `plot.py` that will help us plot the loss and accuracy.

\# import the necessary packages
import matplotlib.pyplot as plt

def plot\_loss\_accuracy(history, filepath):
	# plot the training and validation loss
	plt.style.use("ggplot")
	(fig, axs) = plt.subplots(2, 1)
	axs\[0\].plot(history\["loss"\], label="train\_loss")
	axs\[0\].plot(history\["val\_loss"\], label="val\_loss")
	axs\[0\].set\_xlabel("Epoch #")
	axs\[0\].set\_ylabel("Loss")
	axs\[0\].legend()
	axs\[1\].plot(history\["accuracy"\], label="train\_accuracy")
	axs\[1\].plot(history\["val\_accuracy"\], label="val\_accuracy")
	axs\[1\].set\_xlabel("Epoch #")
	axs\[1\].set\_ylabel("Accuracy")
	axs\[1\].legend()
	fig.savefig(filepath)

On **Line 2**, we begin by importing matplotlib. Then, on **Line 4**, we define the `plot_loss_accuracy` function that takes the model `history` and `filepath` as input.

Next, on **Lines 6-17**, we plot the loss and accuracy for training and validation and save the figure in the specified `filepath` on **Line 18**.

Next, we define another helper function called `save_load.py` to save the adapted vectorization layer for later use.

from tensorflow.keras.layers import TextVectorization
import tensorflow as tf
import pickle

def save\_vectorizer(vectorizer, name):
	# pickle the weights of the vectorization layer
	pickle.dump({"weights": vectorizer.get\_weights()},
		open(f"{name}.pkl", "wb"))

We begin with the necessary imports on **Lines 1-3**. Next, we define a function called `save_vectorizer` on **Lines 5-8** that pickles and saves the weight of the vectorization layer.

And finally, with all the necessary functions defined, we can start with `train.py`, where we actually train our LSTM model.

\# set the seed for reproducibility
import tensorflow as tf
tf.keras.utils.set\_random\_seed(42)

# import the necessary packages
from pyimagesearch.standardization import custom\_standardization
from pyimagesearch.plot import plot\_loss\_accuracy
from pyimagesearch.save\_load import save\_vectorizer
from pyimagesearch.dataset import get\_imdb\_dataset
from pyimagesearch.model import get\_rnn\_model
from pyimagesearch.model import get\_lstm\_model
from pyimagesearch import config
from tensorflow.keras import layers
from tensorflow import keras
import os

We begin with all the necessary imports on **Lines** **5-****18**.

\# get the IMDB dataset
print("\[INFO\] getting the IMDB dataset...")
(trainDs, valDs) = get\_imdb\_dataset(folderName=config.DATASET\_PATH,
	batchSize=config.BATCH\_SIZE, bufferSize=config.BUFFER\_SIZE,
	autotune=tf.data.AUTOTUNE, test=False)

# initialize the text vectorization layer
vectorizeLayer = layers.TextVectorization(
	max\_tokens=config.VOCAB\_SIZE,
	output\_mode="int",
	output\_sequence\_length=config.MAX\_SEQUENCE\_LENGTH,
	standardize=custom\_standardization,
)

# grab the text from the training dataset and adapt the text
# vectorization layer on it
trainText = trainDs.map(lambda text, label: text)
vectorizeLayer.adapt(trainText)

# vectorize the training and the validation dataset
trainDs = trainDs.map(lambda text, label: (vectorizeLayer(text), label))
valDs = valDs.map(lambda text, label: (vectorizeLayer(text), label))

Next, on **Lines 22-24**, we get the IMDb dataset for movie reviews.

On **Lines 27-32**, we initialize the text vectorization layer with:

*   `max_tokens`: The maximum number of tokens inside the vocabulary.
*   `output_mode`: The data type for the output.
*   `output_sequence_length`: The maximum sequence length needed for padding and masking.
*   `standardize`: The custom standardization function that we defined previously.

On **Lines 36 and 37**, we adapt the vectorization layer on the training dataset.

When a text vectorization layer is initialized, it does not hold any information about the vocabulary of the training corpus. Therefore, to build a vocabulary, we need to pass the entire training dataset through the vectorization layer. This is known as adapting to the text.

Finally, on **Lines 40-41**, we vectorize the text of the training and validation dataset using the adapted vectorization layer.

\# get the LSTM model and compile it
print("\[INFO\] building the LSTM model...")
modelLSTM = get\_lstm\_model(vocabSize=config.VOCAB\_SIZE)
modelLSTM.compile(metrics=\["accuracy"\],
	optimizer=keras.optimizers.Adam(learning\_rate=config.LR),
	loss=keras.losses.BinaryCrossentropy(from\_logits=False),
)

# train the LSTM model
print("\[INFO\] training the LSTM model...")
historyLSTM = modelLSTM.fit(trainDs, epochs=config.EPOCHS,
	validation\_data=valDs,
)

On **Lines 59-63**, we initialize the predefined LSTM model and compile it with Adam optimizer and Binary Cross Entropy loss.

On **Lines 66-69**, we fit the model on the training data and save its history onto the `historyLSTM` variable.

\# check whether the output folder exists, if not build the output folder
if not os.path.exists(config.OUTPUT\_PATH):
	os.makedirs(config.OUTPUT\_PATH)

# save the loss and accuracy plots of RNN and LSTM models
plot\_loss\_accuracy(history=historyRNN.history, filepath=config.RNN\_PLOT)
plot\_loss\_accuracy(history=historyLSTM.history, filepath=config.LSTM\_PLOT)

On **Line 72**, we check whether the output path exists or create it otherwise. Finally, the `plot_loss_accuracy` function is called on **Line 77** to visualize the loss and accuracy of the model.

\# save the trained RNN and LSTM models to disk
print(f"\[INFO\] saving the RNN model to {config.RNN\_MODEL\_PATH}...")
keras.models.save\_model(model=modelRNN, filepath=config.RNN\_MODEL\_PATH,
	include\_optimizer=False)
print(f"\[INFO\] saving the LSTM model to {config.LSTM\_MODEL\_PATH}...")
keras.models.save\_model(model=modelLSTM, filepath=config.LSTM\_MODEL\_PATH,
	include\_optimizer=False)

On **Lines 84 and 85,** we then save the trained LSTM model on our disk.

\# save the text vectorization layer to disk
save\_vectorizer(vectorizer=vectorizeLayer, name=config.TEXT\_VEC\_PATH)

Finally, on **Line 88**, we save the text vectorization layer we used to vectorize the training and validation data.

We can verify the training and validation loss and accuracy from the model output. We have a 76% validation accuracy in just 10 epochs!

$ python train.py
\[INFO\] getting the IMDB dataset...
\[INFO\] building the LSTM model...
\[INFO\] training the LSTM model...
Epoch 1/10
22/22 \[==============================\] - 17s 280ms/step - loss: 0.6930 - accuracy: 0.5186 - val\_loss: 0.6928 - val\_accuracy: 0.5404
Epoch 2/10
22/22 \[==============================\] - 3s 116ms/step - loss: 0.6921 - accuracy: 0.5573 - val\_loss: 0.6908 - val\_accuracy: 0.5964
Epoch 3/10
22/22 \[==============================\] - 3s 121ms/step - loss: 0.6862 - accuracy: 0.6190 - val\_loss: 0.6782 - val\_accuracy: 0.6412
Epoch 4/10
22/22 \[==============================\] - 3s 123ms/step - loss: 0.6477 - accuracy: 0.6833 - val\_loss: 0.6103 - val\_accuracy: 0.6968
Epoch 5/10
22/22 \[==============================\] - 3s 119ms/step - loss: 0.5502 - accuracy: 0.7417 - val\_loss: 0.5463 - val\_accuracy: 0.7116
Epoch 6/10
22/22 \[==============================\] - 3s 123ms/step - loss: 0.4768 - accuracy: 0.7838 - val\_loss: 0.5241 - val\_accuracy: 0.7440
Epoch 7/10
22/22 \[==============================\] - 3s 117ms/step - loss: 0.4233 - accuracy: 0.8175 - val\_loss: 0.5087 - val\_accuracy: 0.7508
Epoch 8/10
22/22 \[==============================\] - 3s 118ms/step - loss: 0.3846 - accuracy: 0.8417 - val\_loss: 0.5108 - val\_accuracy: 0.7532
Epoch 9/10
22/22 \[==============================\] - 3s 119ms/step - loss: 0.3502 - accuracy: 0.8606 - val\_loss: 0.5175 - val\_accuracy: 0.7720
Epoch 10/10
22/22 \[==============================\] - 3s 121ms/step - loss: 0.3237 - accuracy: 0.8730 - val\_loss: 0.5280 - val\_accuracy: 0.7612
\[INFO\] saving the RNN model to output/rnn\_model...
\[INFO\] saving the LSTM model to output/lstm\_model...

* * *

### [**Loading and Inference**](#TOC)

Now that the model has been trained and saved to disk, we need to perform inference to understand the model performance. However, before starting with inference, let us first open `save_load.py` again and look at the `load_vectorizer` function.

def load\_vectorizer(name, maxTokens, outputLength, standardize=None):
	# load the pickles data
	fromDisk = pickle.load(open(f"{name}.pkl", "rb"))

	# build a new vectorization layer
	newVectorizer = TextVectorization(max\_tokens=maxTokens,
		output\_mode="int", output\_sequence\_length=outputLength,
		standardize=standardize)

	# call the adapt method with some dummy data for the vectorization
	# layer to initialize properly
	newVectorizer.adapt(tf.data.Dataset.from\_tensor\_slices(\["xyz"\]))
	newVectorizer.set\_weights(fromDisk\["weights"\])

	# return the vectorization layer
	return newVectorizer 

We define the `load_vectorizer` function on **Line 10**, which takes the following input:

*   `name`: the filepath of the saved `TextVectorization` layer weights
*   `maxTokens`: the maximum number of tokens in the vocabulary
*   `outputLength`: the length of the output sequence
*   `standardize`: we do not need any standardization function here

In the training pipeline, we build a `TextVectorization` layer to tokenize and vectorize our training data. In the inference pipeline, we need the same `TextVectorization` layer as the training pipeline.

We save the weights of the adapted `TextVectorization` layer and load the weights on top of a newly initialized layer. On **Line 12**, we load the weights of the saved `TextVectorization` layer. On **Line 25**, we return the loaded vectorizer.

\# USAGE
# python inference.py

# import the necessary packages
from pyimagesearch.standardization import custom\_standardization
from pyimagesearch.save\_load import load\_vectorizer
from pyimagesearch.dataset import get\_imdb\_dataset
from pyimagesearch import config
from tensorflow import keras
import tensorflow as tf

On **Lines 1-10**, we import the necessary packages.

print("\[INFO\] loading the pre-trained LSTM model...")
modelLstm = keras.models.load\_model(filepath=config.LSTM\_MODEL\_PATH)
modelLstm.compile(optimizer="adam", metrics=\["accuracy"\],
	loss=keras.losses.BinaryCrossentropy(from\_logits=False),
)

# get the IMDB dataset
print("\[INFO\] getting the IMDB test dataset...")
testDs = get\_imdb\_dataset(folderName=config.DATASET\_PATH,
	batchSize=config.BATCH\_SIZE, bufferSize=config.BUFFER\_SIZE,
	autotune=tf.data.AUTOTUNE, test=True)

# load the pre-trained text vectorization layer
vectorizeLayer = load\_vectorizer(name=config.TEXT\_VEC\_PATH,
	maxTokens=config.VOCAB\_SIZE,
	outputLength=config.MAX\_SEQUENCE\_LENGTH,
	standardize=custom\_standardization)

# vectorize the test dataset
testDs = testDs.map(lambda text, label: (vectorizeLayer(text), label))

# evaluate the trained RNN and LSTM model on the test dataset
for model in \[modelRnn, modelLstm\]:
	print(f"\[INFO\] test evaluation for {model.name}:")
	(testLoss, testAccuracy) = model.evaluate(testDs)
	print(f"\\t\[INFO\] test loss: {testLoss:0.2f}")
	print(f"\\t\[INFO\] test accuracy: {testAccuracy \* 100:0.2f}%")

On **Line 19**, the saved LSTM model is loaded back to disk. The loaded model is then compiled with the suitable metrics, loss, and optimizer on **Line 20.**

On **Lines 26-28**, we get the testing dataset and pre-process it. Finally, **Line 37** will map the dataset to get the vectorized tokens and labels.

On **Lines 40-44**, we evaluate the testing accuracy and loss of the LSTM model. Our model achieves 74.5% accuracy at inference!

$ python inference.py
\[INFO\] loading the pre-trained LSTM model...
\[INFO\] getting the IMDB test dataset...
\[INFO\] test evaluation for LSTM:
2022-04-04 13:04:26.163129: I tensorflow/stream\_executor/cuda/cuda\_dnn.cc:368\] Loaded cuDNN version 8202
25/25 \[==============================\] - 8s 71ms/step - loss: 0.5660 - accuracy: 0.7450
        \[INFO\] test loss: 0.57
        \[INFO\] test accuracy: 74.50%

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC)
-------------------

This brings us to the end of this tutorial on Long Short-Term Memory Networks. We started with RNNs and examined the reason why they failed. We also learned about an efficient solution in LSTMs. But this is not the end of the journey of sequence modeling. In fact, we are just getting started.

Imagine, for a moment, that you are not only able to predict which word comes where in the sentence but also how much that word contributes to the overall meaning of the sentence.

This sounds like a game changer, and it is. This concept is called _attention_, and we will discuss it in the next tutorial.

* * *

### [**Citation Information**](#TOC)

**A. R. Gosthipaty, D. Chakraborty, and R. Raha.** “Long Short-Term Memory Networks,” _PyImageSearch_, P. Chugh, S. Huot, K. Kidriavsteva, and A. Thanki, eds., 2022, [https://pyimg.co/wtby9](https://pyimg.co/wtby9)

@incollection{ADR\_2022\_LSTM,
  author = {Aritra Roy Gosthipaty and Devjyoti Chakraborty and Ritwik Raha},
  title = {Long Short-Term Memory Networks},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Susan Huot and Kseniia Kidriavsteva and Abhishek Thanki},
  year = {2022},
  note = {https://pyimg.co/wtby9},
}

* * *

![Image 47: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 48](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!