NLP: Detecting Spam Messages with TensorFlow (Part I) | by Michael Grogan | Towards Data Science

Here is an example of how a recurrent neural network can be used to detect spam messages. The dataset used in this example is sourced from Kaggle (original authors Almeida and Hidalgo, 2011).
In the training set, certain messages are marked as “spam” (this has been replaced with a 1 for this purpose). Non-spam messages are marked as “ham” (replaced with a 0 for this purpose).
The recurrent neural network is built using the original Word Embeddings and Sentiment notebook from the TensorFlow Authors — the original notebook is available here.
The analysis is conducted through the following steps:
The data is loaded, with the sentences split into training and test sets.
The data is loaded, with the sentences split into training and test sets.
dataset = pd.read_csv('spam.csv')datasetsentences = dataset['Message'].tolist()labels = dataset['Category'].tolist()# Separate out the sentences and labels into training and test setstraining_size = int(len(sentences) * 0.8)training_sentences = sentences[0:training_size]testing_sentences = sentences[training_size:]training_labels = labels[0:training_size]testing_labels = labels[training_size:]# Make labels into numpy arrays for use with the network latertraining_labels_final = np.array(training_labels)testing_labels_final = np.array(testing_labels)
2. The dataset is tokenized. In other words, a unique number is assigned to each word — which is necessary for the neural network to interpret the input.
vocab_size = 1000embedding_dim = 16max_length = 100trunc_type='post'padding_type='post'oov_tok = "<OOV>"from tensorflow.keras.preprocessing.text import Tokenizerfrom tensorflow.keras.preprocessing.sequence import pad_sequencestokenizer = Tokenizer(num_words = vocab_size, oov_token=oov_tok)tokenizer.fit_on_texts(training_sentences)word_index = tokenizer.word_index
3. These tokens are then sorted into sequences, to ensure that the tokens for each word follow the correct order as dictated by each sentence.
sequences = tokenizer.texts_to_sequences(training_sentences)
4. Padding is then introduced — which introduces 0s at the end of each sentence. This is necessary when one sentence is longer than another, as each sentence must have the same length for the purposes of analysis by the RNN.
padded = pad_sequences(sequences,maxlen=max_length, padding=padding_type,                        truncating=trunc_type)testing_sequences = tokenizer.texts_to_sequences(testing_sentences)testing_padded = pad_sequences(testing_sequences,maxlen=max_length,                                padding=padding_type, truncating=trunc_type)
5. The recurrent neural network is built and trained across 20 epochs — with the input layer comprised of an embedding layer which represents the sentences with dense vector representation.
Here is the recurrent neural network configuration:
model = tf.keras.Sequential([    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),    tf.keras.layers.Flatten(),    tf.keras.layers.Dense(6, activation='relu'),    tf.keras.layers.Dense(1, activation='sigmoid')])model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])model.summary()
Here is a closer look at the model parameters:
Model: "sequential"_________________________________________________________________Layer (type)                 Output Shape              Param #   =================================================================embedding (Embedding)        (None, 100, 16)           16000     _________________________________________________________________flatten (Flatten)            (None, 1600)              0         _________________________________________________________________dense (Dense)                (None, 6)                 9606      _________________________________________________________________dense_1 (Dense)              (None, 1)                 7         =================================================================Total params: 25,613Trainable params: 25,613Non-trainable params: 0
The model produces the following training and validation loss:
num_epochs = 20history=model.fit(padded, training_labels_final, epochs=num_epochs, validation_data=(testing_padded, testing_labels_final))
In this instance, we see that the validation loss bottoms out after 5 epochs. In this regard, the model is run again with 5 epochs chosen.
Now that the model has been built, let’s see how the classifier does in identifying spam on the following messages (which I invented randomly):
‘Greg, can you call me back once you get this?’ (intended as genuine)
‘Congrats on your new iPhone! Click here to claim your prize...’ (intended as spam)
‘Really like that new photo of you’ (intended as genuine)
‘Did you hear the news today? Terrible what has happened...’ (intended as genuine)
‘Attend this free COVID webinar today: Book your session now...’ (intended as spam)
Here are the scores generated (the closer the score is to 1, the higher the probability that the sentence is spam:
['Greg, can you call me back once you get this?', 'Congrats on your new iPhone! Click here to claim your prize...', 'Really like that new photo of you', 'Did you hear the news today? Terrible what has happened...', 'Attend this free COVID webinar today: Book your session now...']Greg, can you call me back once you get this?[0.0735679]Congrats on your new iPhone! Click here to claim your prize...[0.91035014]Really like that new photo of you[0.01672107]Did you hear the news today? Terrible what has happened...[0.02904579]Attend this free COVID webinar today: Book your session now...[0.54472804]
We see that for the two messages intended as spam — the classifier shows a significant probability for both. In the case of the sentence, “Attend this free COVID webinar today: Book your session now...” — the classifier does reasonably well at marking a higher than 50% probability of being spam — even though COVID was not a term at the time the training set was constructed.
In this regard, it is evident that the classifier relies on the context in which the words are being used — as opposed to individual words simply being marked as spam in their own right.
Given that the recurrent neural network is effective at modelling sequential data and identifying patterns between words — this simple spam detector built using TensorFlow has been shown to be quite effective on the limited data we have used for testing purposes.
In this example, we have seen:
How recurrent neural networks can be used for text classification
Preparation of text data for analysis using tokenization, sequences and padding
Configuration of a neural network model to analyse text data
In addition, we saw how the model can then be used to predict unseen data (or messages in this case) to determine how the model would potentially work in real-world scenarios.
Many thanks for reading, and the associated GitHub repository for this example can be found here.
Update: You can also find updates made to this model under Part II of this article, available here.
Disclaimer: This article is written on an “as is” basis and without warranty. It was written with the intention of providing an overview of data science concepts, and should not be interpreted as any sort of professional advice. The author has no relationships with any parties mentioned in this article, nor is this article or its findings endorsed by the same.