Word and Character Based LSTM Models
By: Ruthu S Sanketh
Tags: NLP, Text Preprocessing, Lstm, Keras, Language Model

Word and Character Based LSTM Models

If you’re not familiar with the NLTK library and data preprocessing, take a look at this article. If you’re interested in language models and how to build them, read this article. If you’re familiar with NLP and language models, continue reading!

What are LSTMs?

Long-short-term memory models or LSTMs are used to solve the problem of short term memory by using gates that regulate the flow of information. These models have mechanisms that decide whether or not to keep information, thereby being able to retain important information over a long time. Due to their ability to learn long-term dependencies, they are extensively used in machine translation, speech recognition, handwriting recognition and generation, language modeling and translation, speech synthesis, and many other deep learning tasks.

LSTM models for text prediction — Photo by Markus Spiske on Unsplash

Sequence classification is a predictive modeling problem where there is some sequence of inputs over space or time and we wish to predict a category for the sequence. The sequences can vary in length, be comprised of a very large vocabulary of input symbols and may require the model to learn long-term dependencies between symbols in the input sequence, making this a challenging problem for RNNs but easily solvable by LSTMs.

In this article, we will look at building word based as well as character based LSTM models, and compare the next word predictions of the two. We will also look at different parameters that can be changed while training the models and analyze which combination gives us better results.

Initial Steps

First we import the required toolkits and libraries.

import numpy

import re

import pandas as pd

import numpy as np

import keras

import string

import nltk



from keras.models import Sequential

from keras.layers import Dense

from keras.layers import Dropout

from keras.layers import LSTM

from keras.layers import Embedding

from keras.callbacks import ModelCheckpoint

from keras.utils import np_utils

from sklearn.model_selection import train_test_split

from keras.preprocessing.text import Tokenizer

from keras.preprocessing.sequence import pad_sequences



string.punctuation = string.punctuation +'“'+'”'+'-'+'’'+'‘'+'—'

string.punctuation = string.punctuation.replace('.', '')

Now we perform basic preprocessing, such as lowering, etc., on the dataset. We also extract some statistics about the dataset such as the number of unique tokens, which can later be passed as a parameter while training our model. The dataset used in this article can be found here.

# Loads the data and preprocesses data and stores corpus in raw_text

raw_text = open('/content/sample_data/corpus.txt', encoding = 'utf8').read()



file_nl_removed = ""

for line in raw_text:

line_nl_removed = line.replace("

", " ")

#removes newlines

file_nl_removed += line_nl_removed



file_p = "".join([char for char in file_nl_removed if char not in string.punctuation])

#removes all special characters

sents = nltk.sent_tokenize(file_p)

print("The number of sentences is", len(sents))

#prints the number of sentences



string.punctuation = string.punctuation + '.'

file_q = "".join([char for char in file_p if char not in string.punctuation]) #removes even periods.

words = nltk.word_tokenize(file_q)

print("The number of tokens is", len(words))

#prints the number of tokens



average_tokens = round(len(words)/len(sents))

print("The average number of tokens per sentence is", average_tokens)

#prints the average number of tokens per sentence



unique_tokens = set(words)

print("The number of unique tokens are", len(unique_tokens))

#prints the number of unique tokens



preprocessed_text = file_p.lower()

#converts corpus into lowercase

Statistics about the data — Image by author

Training the Word Based LSTM Model

Now that we know what our data looks like after preprocessing, we can set the hyperparameters of our word based LSTM model. We also tokenize our dataset into sentences and obtain sequences of length 50 tokens each from the sentences.

# Hyperparameters of the model

vocab_size = 2750 #chosen based on statistics of the model

oov_tok = '<OOV>'

embedding_dim = 100

padding_type='post'

trunc_type='post' # tokenizes sentences

tokenizer = Tokenizer(num_words = vocab_size, oov_token=oov_tok)

tokenizer.fit_on_texts([preprocessed_text])

word_index = tokenizer.word_index seq_length = 50

tokens = tokenizer.texts_to_sequences([preprocessed_text])[0]

We have to prepare our data into X and Y columns. Since our sequence length is 50, we take every 50 tokens as the input and the 51st token as the output. We also one-hot encode the output variable Y.

dataX = []

dataY = []



for i in range(0, len(tokens) - seq_length-1 , 1):

seq_in = tokens[i:i + seq_length]

seq_out = tokens[i + seq_length]



if seq_out==1: #Skip samples where target word is OOV

continue



dataX.append(seq_in)

dataY.append(seq_out)



N = len(dataX)

print ("Total training data size is -", N) X = numpy.array(dataX)



# one hot encodes the output variable

y = numpy.array(dataY)

y = np_utils.to_categorical(dataY)

Training data size — Image by author

Let us now pass the required parameters to our model and compile it. We use word embeddings, which is a technique where words are encoded as real-valued vectors in a high dimensional space, such that the similarity between words in terms of meaning translates to closeness in the vector space, using an Embedded Layer provided by Keras.

We will map each word onto a 100 length real valued vector, using an embedding dimension of 100 as defined by our hyperparameters. Bidirectional LSTMs, which we are using, are an extension of traditional LSTMs that improve model performance by training two instead of one LSTM on the input sequence.

# with embedding

model = keras.Sequential([

keras.layers.Embedding(vocab_size, embedding_dim, input_length=seq_length),

keras.layers.Bidirectional(keras.layers.LSTM(64)),

keras.layers.Dense(vocab_size, activation='softmax')

])



# compiles model

model.compile(loss='categorical_crossentropy',

optimizer='adam',

metrics=['accuracy'])



# model summary

model.summary()

Model summary — Image by author

We train the model with 5 epochs and a train-validation split of 80–20%. Increasing the number of epochs might improve the model performance up to a certain number of epochs.

# Uses validation split of 0.2 while training

num_epochs = 5

history = model.fit(X, y, epochs=num_epochs, batch_size = 128, verbose=1, validation_split=0.2)

Training the model — Image by author

Next Word Predictions

Now that we have a trained model, we can use it to get the next word predictions. To do this, first we have to create a word map from the predictions to the actual word using which we can find the word corresponding to a certain prediction. We then write a function to return the next n words greedily.

#Creates word to idx map using tokenizer.word_index

reverse_word_map = dict(map(reversed, tokenizer.word_index.items())) # Returns the next n words greedily

def next_tokens(input_str, n):

print ("Seed -", input_str, sep = '



')

final_string = ''

for i in range(n):

token = tokenizer.texts_to_sequences([input_str])[0]

prediction = model.predict(token, verbose=0)

final_string = final_string + reverse_word_map[numpy.argmax(prediction[0])] + ' '

input_str = input_str + ' ' + reverse_word_map[numpy.argmax(prediction[0])]

input_str = ' '.join(input_str.split(' ')[1:])

return final_string

Let us look at the next word predictions by picking a random seed as the input and predicting the next 10 words greedily, and also by specifying an unseen input string.

# picks a random seed

start = numpy.random.randint(0, len(dataX)-1)

pattern = dataX[start]

input_str = ' '.join([reverse_word_map[value] for value in pattern])



output = next_tokens(input_str, 10)

print("

Generated string -



", output) #speficies an unseen input string input_str = "The boy laughed at the fright he had caused. This time, the villagers left angrily. The third day, as the boy went up\

the small hill, he suddenly saw a wolf attacking his sheep. He cried as hard as he could, “Wolf! Wolf! Wolf!”, but not \

a single villager came to help him. The villagers thought that he was trying to fool them again and did not come to rescue \

him or his sheep."



# Uses first 50 tokens from given input_str as input. Since the seq_length is 50, only 50 tokens are taken using the tokenizer.

output = next_tokens(input_str, 10)

print("

Generated string -



", output)

Generated string on a random seed — Image by author

Generated string on an unseen input string — Image by author

We see that word based LSTM model in general has a very poor prediction of next words, regardless of whether the data is seen or unseen. Predictions may be made better by training the model for a larger number of epochs.

Training the Character Based LSTM Model

Let us train 2 character based models with different hyperparameters and then compare the outputs to analyze which hyperparameters are best suited to next character prediction.

Character Based LSTM Model 1

First we use the preprocessed data to create the raw data for the model. Periods have not been removed for better results. Sometimes, not removing other special characters such as apostrophes, commas, etc., can also give better results. Feel free to try it out! We then create a mapping of unique characters to integers and vice versa.

# Uses the preprocessed data and create raw_text

raw_text = preprocessed_text #periods have not been removed for better results



# creates mapping of unique characters to integers

chars = sorted(list(set(raw_text)))

char_to_int = dict((c, i) for i, c in enumerate(chars))

int_to_char = dict((i, c) for i, c in enumerate(chars)) # Prints the total characters and character vocab size

n_chars = len(raw_text)

n_vocab = len(chars)



print("The number of total characters are", n_chars)

print("

The character vocab size is", n_vocab)

Data statistics— Image by author

Now we prepare the dataset where the input is a sequence of 100 characters and the target is next character. We also one-hot encode the output variable Y.

#Prepares dataset where the input is sequence of 100 characters and target is next character.

seq_length = 100



dataX = []

dataY = []



for i in range(0, n_chars - seq_length, 1):

seq_in = raw_text[i:i + seq_length]

seq_out = raw_text[i + seq_length]



dataX.append([char_to_int[char] for char in seq_in])

dataY.append(char_to_int[seq_out])



n_patterns = len(dataX)

print ("Total Patterns: ", n_patterns) # reshapes X to be [samples, time steps, features]

X = numpy.reshape(dataX, (n_patterns, seq_length, 1))



# one hot encodes the output variable

y = np_utils.to_categorical(dataY)

Total number of patterns — Image by author

Let us now pass the required parameters to our model and compile it. We use word embeddings again. We will map each word onto a 100 length real valued vector, using an embedding dimension of 100 as defined by our hyperparameters. We also use a dropout layer which is a regularization method to prevent overfitting.

embedding_dim =100

max_length =100 model = Sequential()

model.add(Embedding(n_vocab, embedding_dim, input_length=max_length))

model.add(LSTM(256))

model.add(Dropout(0.2))

model.add(Dense(y.shape[1], activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam')

model.summary()

Model summary — Image by author

We now train the model using 20 epochs, and a batch size of 128. Using a local machine usually makes this process long and requires a lot of runtime, so I would suggest using Google Colab with a GPU runtime.

model.fit(X, y, epochs = 20, batch_size=128)

Training the model — Image by author

We need to write a function to be able to get the actual character from the prediction.

#Gets the generated string using the model.

def predict_next_n_chars(pattern, n):

for i in range(n):

x = numpy.reshape(pattern, (1, len(pattern), 1))

prediction = model.predict(x, verbose=0)

print (int_to_char[numpy.argmax(prediction)], end = '') #get next char index.

seq_in = [int_to_char[value] for value in pattern]

pattern.append(numpy.argmax(prediction))

pattern = pattern[1:len(pattern)]

Let us now look at the next character predictions by picking a random seed as the input and predicting the next 200 characters greedily, and also by specifying an unseen input string.

#picks a random seed

start = numpy.random.randint(0, len(dataX)-1)

pattern = dataX[start]

input_str = ''.join([int_to_char[value] for value in pattern])

print ("Seed -", input_str, sep = '



')

print ("

Generated string -

")



predict_next_n_chars(pattern, 200) input_str = "The boy laughed at the fright he had caused. This time, the villagers left angrily. The third day, as the boy went up\

the small hill, he suddenly saw a wolf attacking his sheep. He cried as hard as he could, “Wolf! Wolf! Wolf!”, but not \

a single villager came to help him. The villagers thought that he was trying to fool them again and did not come to rescue \

him or his sheep." #Uses the first 100 characters from given input_str as input to generate next 200 characters.

input_str = input_str.lower()

input_string = ''

for each in input_str:

if each in chars:

if (len (input_string)<100):

input_string += each



pattern = []

pattern.append([char_to_int[char] for char in input_string])



print ("Seed -", input_str, sep = '



')

print ("

Generated string -

")

predict_next_n_chars(pattern[0], 200)

Generated characters on a random seed — Image by author

Generated characters on an unseen input string — Image by author

We see that the character generation is better, and has more overall word dependency captured in the seen data than the unseen data. In both the cases, the generated text has nearly the same degree of syntactical accuracy, but the semantic meaning of the overall text is higher in the seen data generation.

Character Based LSTM Model 2

We use the same data that was used in the first model. Let us pass the required parameters to our model and compile it. We use word embeddings again. We will map each word onto a 100 length real valued vector, using an embedding dimension of 100 as defined by our hyperparameters. We also use 2 dropout layers.

model1 = Sequential()

model1.add(Embedding(n_vocab, embedding_dim, input_length=max_length))

model1.add(LSTM(256, input_shape=(X.shape[1], embedding_dim),return_sequences=True))

model1.add(Dropout(0.2))

model1.add(LSTM(256))

model1.add(Dropout(0.2))

model1.add(Dense(y.shape[1], activation='softmax'))

model1.compile(loss='categorical_crossentropy', optimizer='adam')

model1.summary()

Model summary — Image by author

We now train the model using 20 epochs, and a batch size of 64.

model1.fit(X, y, epochs = 20, batch_size=64)

Training the model — Image by author

We now write a function to get the actual character from the prediction.

# Generates the sequence similar to above methods. Gets the generated string using the model.

def predict_next_n_chars(pattern, n):

for i in range(n):

x = numpy.reshape(pattern, (1, len(pattern), 1))

prediction = model1.predict(x, verbose=0)

print (int_to_char[numpy.argmax(prediction)], end = '') #get next char index.

seq_in = [int_to_char[value] for value in pattern]

pattern.append(numpy.argmax(prediction))

pattern = pattern[1:len(pattern)]

Let us look at the next character predictions by picking a random seed as the input and predicting the next 200 characters greedily, and also by specifying an unseen input string.

#picks a random seed

start = numpy.random.randint(0, len(dataX)-1)

pattern = dataX[start]

input_str = ''.join([int_to_char[value] for value in pattern])

print ("Seed -", input_str, sep = '



')

print ("

Generated string -

")



predict_next_n_chars(pattern, 200) # specifies an unseen input string input_str = "The boy laughed at the fright he had caused. This time, the villagers left angrily. The third day, as the boy went up\

the small hill, he suddenly saw a wolf attacking his sheep. He cried as hard as he could, “Wolf! Wolf! Wolf!”, but not \

a single villager came to help him. The villagers thought that he was trying to fool them again and did not come to rescue \

him or his sheep."



#Uses the first 100 characters from given input_str as input to generate next 200 characters.

input_str = input_str.lower()

input_string = ''

for each in input_str:

if each in chars:

if (len (input_string)<100):

input_string += each



pattern = []

pattern.append([char_to_int[char] for char in input_string])



print ("Seed -", input_str, sep = '



')

print ("

Generated string -

")

predict_next_n_chars(pattern[0], 200)

Generated characters on a random seed — Image by author

Generated characters on an unseen input string — Image by author

We see that the character generation is better, has higher overall word dependency captured in the seen data than the unseen data, as well as higher semantic meaning, just like Model 1. Model 2 is comparatively less overfitted than Model 1, in terms of unseen as well as seen data prediction. The unseen data prediction of this model is thus slightly better than that of Model 1.

Comparison between LSTM Character Based Model 1 and 2

Model 2 has a higher accuracy, as well as semantic meaning and captures word dependencies better than the Model 1 for unseen data, whereas Model 1 makes slightly better predictions on the seen data.

Some differences between Model 1 and Model 2 are -

Model 2 has 2 dropout layers while Model 1 has just 1. Dropout is a regularization method where input and recurrent connections to LSTM units are probabilistically excluded from activation and weight updates while training a network. This has the effect of reducing overfitting and improving model performance. Hence we see that unseen data generation is done better by Model 2 than Model 1.

Model 1 has just one layer, while Model 2 has an added layer with 256 units, and Model 2 returns sequences while Model 1 does not. Hence the overall performance of Model 2 is higher than that of model one.

The batch size of Model 1 is 128, while that of Model 2 is 64, giving the network more time to update and learn, hence we get better outputs from Model 2.

Conclusion

I hope this article was a good introduction to word and character based LSTM models, how to build as well as analyze them without needing a lot of mathematical knowledge of their working. Varying hyperparameters, the number of layers, as well the number of training epochs can help improve model performance. The entire code and dataset used in this article can be found here.

Further Reading