NLP: Preparing text for deep learning model using TensorFlow2 | by Sudip Shrestha, PhD | Towards Data Science

Natural Language Processing (NLP) is commonly used in text classification tasks such as spam detection and sentiment analysis, text generation, language translations and document classification. Text data can be considered either in sequence of character, sequence of words or sequence of sentences. Most commonly, text data are considered as sequence of words for most problems. In this article we will delve into, pre-processing using simple example text data. However, the steps discussed here are applicable to any NLP tasks. Particularly, we’ll use TensorFlow2 Keras for text pre-processing which include:
Tokenization
Sequencing
Padding
The figure below depicts the process of text pre-processing along with example outputs.
First, let’s import the required libraries. (A complete Jupyter notebook is available in my GitHub page).
import tensorflow as tffrom tensorflow.keras.preprocessing.text import Tokenizerfrom tensorflow.keras.preprocessing.sequence import pad_sequences
Tokenizer is an API available in TensorFlow Keras which is used to tokenize sentences. We have defined our text data as sentences (each separated by a comma) and with an array of strings. There are 4 sentences including 1 with a maximum length of 5. Our text data also includes punctuations as shown below.
sentences = ["I want to go out.",             " I like to play.",             " No eating - ",             "No play!",            ]sentences['I want to go out.', ' I like to play.', ' No eating - ', 'No play!']
As deep learning models do not understand text, we need to convert text into numerical representation. For this purpose, a first step is Tokenization. The Tokenizer API from TensorFlow Keras splits sentences into words and encodes these into integers. Below are hyperparameters used within Tokenizer API:
num_words: Limits maximum number of most popular words to keep while training.
filters: If not provided, by default filters out all punctuation terms (!”#$%&()*+,-./:;<=>?@[\]^_’{|}~\t\n).
lower=1. This is a default setting which converts all words to lower case
oov_tok : When its used, out of vocabulary token will be added to word index in the corpus which is used to build the model. This is used to replace out of vocabulary words (words that are not in our corpus) during text_to_sequence calls (see below).
word_index: Convert all words to integer index. Full list of words are available as key value property: key = word and value = token for that word
Let’s use the Tokenizer and print out word index. We have used num_words= 100 which is a lot for this data as there are only 9 distinct words and <OOV> string for out of vocabulary token.
tokenizer = Tokenizer(num_words=100, lower= 1, oov_token="<OOV>")tokenizer.fit_on_texts(sentences)word_index = tokenizer.word_indexprint(word_index){'<OOV>': 1, 'i': 2, 'to': 3, 'play': 4, 'no': 5, 'want': 6, 'go': 7, 'out': 8, 'like': 9, 'eating': 10}
As seen above, each word in our sentences has been converted to numerical tokens. For instance, “i” has a value of 2. The tokenizer also ignored the exclamation mark after the word. For example, there is only one token for the word “play” or “play!” i.e. 4.
Next, let’s represent each sentence by sequences of numbers using texts_to_sequences from tokenizer object. Below, we printed out raw sentences, word index and sequences.
sequences = tokenizer.texts_to_sequences(sentences)print(sentences)print(word_index)print(sequences)['I want to go out', ' I like to play', ' No eating - ', 'No play!']{'<OOV>': 1, 'i': 2, 'to': 3, 'play': 4, 'no': 5, 'want': 6, 'go': 7, 'out': 8, 'like': 9, 'eating': 10}[[2, 6, 3, 7, 8], [2, 9, 3, 4], [5, 10], [5, 4]]
As shown above, texts are represented by sequences. For example,
“I want to go out” — -> [2, 6, 3, 7, 8]
“I like to play” — -> [2, 9, 3, 4]
“No eating” — -> [5, 10]
“No play!” — -> [5, 4]
In any raw text data, naturally there will be sentences of different lengths. However, all neural networks require to have inputs with the same size. For this purpose, padding is done. Below, let’s use pad_sequences for padding. pad_sequences uses arguments such as sequences, padding, maxlen, truncating, value and dtype.
sequences: list of sequences that we created earlier
padding = ‘pre’ or ‘post (default pre). By using pre, we’ll pad (add 0) before each sequence and post will pad after each sequence.
maxlen = maximum length of all sequences. If not provided, by default it will use the maximum length of the longest sentence.
truncating = ‘pre’ or ‘post’ (default ‘pre’). If a sequence length is larger than the provided maxlen value then, these values will be truncated to the maxlen. ‘pre’ option will truncate at the beginning whereas ‘post’ will truncate at the end of the sequences.
value: padding value (default is 0)
dtype: output sequence type (default is int32)
Let’s focus on important arguments used in pad_sequences : padding, maxlen and truncating.
pre and post padding
Use of ‘pre’ or ‘post’ padding depends upon the analysis. In some cases, padding at the beginning is appropriate while not in others. For instance, if we use Recurrent Neural Network (RNN) for spam detection, then padding at the beginning would be appropriate as RNN can not learn long distance patterns. Padding at the beginning allows us to keep the sequences in the end hence RNN can make use of these sequences for prediction of next. However, in any case padding should be done after careful consideration and business knowledge.
Below, the outputs for ‘pre’ followed by ‘post’ padding are shown with default maxlen value of maximum length of sequence.
# pre paddingpre_pad = pad_sequences(sequences, padding='pre')print("\nword_index = ", word_index)print("\nsequences = ", sequences)print("\npadded_seq = " )print(pre_pad)word_index =  {'<OOV>': 1, 'i': 2, 'to': 3, 'play': 4, 'no': 5, 'want': 6, 'go': 7, 'out': 8, 'like': 9, 'eating': 10}sequences =  [[2, 6, 3, 7, 8], [2, 9, 3, 4], [5, 10], [5, 4]]padded_seq = [[ 2  6  3  7  8] [ 0  2  9  3  4] <---------- 0 Padded at the beginning [ 0  0  0  5 10]  [ 0  0  0  5  4]]
In our example above, the sequence with maximum length is [ 2, 6, 3, 7, 8] which corresponds to “I want to go out”. When padding =’pre’ is used, padded value of 0 is added at the beginning of all other sequences. Because other sequences have shorter sequence than [ 2, 6, 3, 7, 8], padding actually made all other sequences to be of same size with this sequence.
Whereas, when padding = ‘post’ is used , padded value i.e. 0 is added at the end of the sequences.
# post paddingpost_pad = pad_sequences(sequences, padding='post')print("\nword_index = ", word_index)print("\nsequences = ", sequences)print("\npadded_seq = " )print(post_pad)word_index =  {'<OOV>': 1, 'i': 2, 'to': 3, 'play': 4, 'no': 5, 'want': 6, 'go': 7, 'out': 8, 'like': 9, 'eating': 10}sequences =  [[2, 6, 3, 7, 8], [2, 9, 3, 4], [5, 10], [5, 4]]padded_seq = [[ 2  6  3  7  8] [ 2  9  3  4  0]<---------- 0 Padded at the end [ 5 10  0  0  0] [ 5  4  0  0  0]]
pre and post padding with maxlen and truncating option
We can use both padding and truncating argument together if needed. Below we have shown two-scenarios, 1) pre padding with pre truncation and 2) pre padding with post truncation
The truncating with ‘pre’ option allows us to truncate the sequence at the beginning. Whereas, truncating with ‘post’ will truncate the sequence at the end.
Let’s look at the example of pre padding with pre truncation.
# pre padding, maxlen and pre truncationprepad_maxlen_pretrunc = pad_sequences(sequences, padding = ‘pre’, maxlen =4, truncating = ‘pre’)print(prepad_maxlen_pretrunc)[[ 6  3  7  8]<-----Truncated from length 5 to 4, at the beginning [ 2  9  3  4] [ 0  0  5 10]<---------- Padded at the beginning [ 0  0  5  4]]
By use of maxlen =4, we are truncating the length of padded sequences to 4. As shown, above, the use of maxlen=4 impacted the first sequence [2, 6, 3, 7, 8]. This sequence had length of 5 and is truncated to 4. The truncation happened at the beginning as we used truncating = ‘pre’ option.
Let’s look at the truncation = ‘post’ option.
# pre padding, maxlen and post truncationprepad_maxlen_posttrunc = pad_sequences(sequences, padding = 'pre', maxlen =4, truncating = 'post')print(prepad_maxlen_posttrunc)[[ 2  6  3  7]<-----Truncated from length 5 to 4, at the end [ 2  9  3  4] [ 0  0  5 10]<---------- Padded at the beginning [ 0  0  5  4]]
The truncation happened at the end as we used truncating = ‘post’ option. When the post truncation was applied, it impacted the first sequence [ 2, 6, 3, 7, 8] and truncated to length 4 resulting in the sequence [ 2, 6, 3, 7].
In this article, we focused on pre-processing raw text data and preparing it for deep learning models. Specifically, we covered tokenizing sentences, representing it as sequences and padding it to make all the sequences of same length. This padded sequences are now ready for train/test split that can be used for neural network. Please refer to video by Laurence Moroney’s NLP zero to hero for further reading.
In future articles, I will explain how we can use pre-processing in real-world data and use such padded sequence data with embedding in deep learning models.
Thanks for reading this and please leave if any comments or suggestions.