Detect Spam Messages with C# And A CNTK Deep Neural Network
By: Mark Farragher
Tags: Programming, Data Science, Artificial Intelligence, Csharp, Machine Learning

It’s a TSV file with only 2 columns of information:

Label: ‘spam’ for a spam message and ‘ham’ for a normal message.

Message: the full text of the SMS message.

I will build a binary classification network that reads in all messages and then makes a prediction for each message if it is spam or ham.

Let’s get started. Here’s how to set up a new console project in NET Core:

$ dotnet new console -o SpamDetection

$ cd SpamDetection

Next, I need to install required packages:

$ dotnet add package Microsoft.ML

$ dotnet add package CNTK.GPU

$ dotnet add package XPlot.Plotly

$ dotnet add package Fsharp.Core

Microsoft.ML is the Microsoft machine learning package. We will use to load and process the data from the dataset.

The CNTK.GPU library is Microsoft’s Cognitive Toolkit that can train and run deep neural networks.

And Xplot.Plotly is an awesome plotting library based on Plotly. The library is designed for F# so we also need to pull in the Fsharp.Core library.

The CNTK.GPU package will train and run deep neural networks using your GPU. You’ll need an NVidia GPU and Cuda graphics drivers for this to work.

If you don’t have an NVidia GPU or suitable drivers, the library will fall back and use the CPU instead. This will work but training neural networks will take significantly longer.

CNTK is a low-level tensor library for building, training, and running deep neural networks. The code to build deep neural network can get a bit verbose, so I’ve developed a little wrapper called CNTKUtil that will help you write code faster.

You can download the CNTKUtil files and save them in a new CNTKUtil folder at the same level as your project folder.

Then make sure you’re in the console project folder and crearte a project reference like this:

$ dotnet add reference ..\CNTKUtil\CNTKUtil.csproj

Now I’m ready to start writing code. I will edit the Program.cs file with Visual Studio Code and add the following code:

The SpamData class holds all the data for one single spam message. Note how each field is tagged with a LoadColumn attribute that will tell the TSV data loading code from which column to import the data.

Unfortunately I can’t train a deep neural network on text data directly. I first need to convert the text to numbers.

I will get to that conversion later. For now I’ll add a class here that will contain the converted text:

There’s the Label again, but notice how the message has now been converted to a VBuffer and stored in the Features field.

The VBuffer type is a sparse vector. It’s going to store a very large vector with mostly zeroes and only a few nonzero values. The nice thing about the VBuffer type is that it only stores the nonzero values. The zeroes are not stored and do not occupy any space in memory.

The GetFeatures method calls DenseValues to return the complete vector and returns it as a float[] that our neural network understands.

And there’s a GetLabel method that returns 1 if the message is spam (indicated by the Label field containing the word ‘spam’) and 0 if the message is not spam.

The features represent the text converted to a sparse vector that we will use to train the neural network on, and the label is the output variable that we’re trying to predict. So here we’re training on encoded text to predict if that text is spam or not.

Now it’s time to start writing the main program method:

When working with the ML.NET library we always need to set up a machine learning context represented by the MLContext class.

The code calls the LoadFromTextFile method to load the CSV data in memory. Note the SpamData type argument that tells the method which class to use to load the data.

I then use TrainTestSplit to split the data in a training partition containing 70% of the data and a testing partition containing 30% of the data.

Note that I’m deviating from the usual 80–20 split here. This is because the data file is quite small, and so 20% of the data is simply not enough to test the neural network on.

Now it’s time to build a pipeline to convert the text to sparse vector-encoded data. I will use the FeaturizeText component in the ML.NET machine learning library:

Machine learning pipelines in ML.NET are built by stacking transformation components. Here I am using a single component, FeaturizeText, that converts the text messages in SpamData.Message into sparse vector-encoded data in a new column called ‘Features’.

The FeaturizeText component is a very nice solution for handling text input data. The component performs a number of transformations on the text to prepare it for model training:

Normalize the text (=remove punctuation, diacritics, switching to lowercase etc.)

Tokenize each word.

Remove all stopwords

Extract Ngrams and skip-grams

TF-IDF rescaling

Bag of words conversion

The result is that each message is converted to a vector of numeric values that can easily be processed by a deep neural network.

I call the Fit method to initialize the pipeline, and then call Transform twice to transform the text in the training and testing partitions.

Finally I call CreateEnumerable to convert the training and testing data to an enumeration of ProcessedData instances. So now I have the training data in training and the testing data in testing. Both are enumerations of ProcessedData instances.

But CNTK can’t train on an enumeration of class instances. It requires a float[][] for features and float[] for labels.

So I need to set up four float arrays:

These LINQ expressions set up four arrays containing the feature and label data for the training and testing partitions.

Now I need to tell CNTK what shape the input data has that I’ll train the neural network on, and what shape the output data of the neural network will have:

I don’t know in advance how many dimensions the FeaturizeText component will create, so I simply check the width of the training_data array.

The first Var method tells CNTK that my neural network will use a 1-dimensional tensor of nodeCount float values as input. This shape matches the array returned by the ProcessedData.GetFeatures method.

And the second Var method tells CNTK that I want my neural network to output a single float value. This shape matches the single value returned by the ProcessedData.GetLabel method.

My next step is to design the neural network.

I will use a deep neural network with a 16-node input layer, a 16-node hidden layer, and a single-node output layer. I’ll use the ReLU activation function for the input and hidden layers, and Sigmoid activation for the output layer.

The sigmoid function forces the output of a regression network to a range of 0..1 which means I can treat the number as a binary classification probability. So we can turn any regression network into a binary classification network by simply adding the sigmoid activation function to the output layer.

Here’s how to build this neural network:

Each Dense call adds a new dense feedforward layer to the network. I am stacking two layers, both using ReLU activation, and then add a final layer with a single node using Sigmoid activation.

Then I use the ToSummary method to output a description of the architecture of the neural network to the console.

Now I need to decide which loss function to use to train the neural network, and how I am going to track the prediction error of the network during each training epoch.

I will use BinaryCrossEntropy as the loss function because it’s the standard metric for measuring binary classification loss.

And I’ll track the error with the BinaryClassificationError metric. This is the number of times (expressed as a percentage) that the model predictions are wrong. An error of 0 means the predictions are correct all the time, and an error of 1 means the predictions are wrong all the time.

Next I need to decide which algorithm to use to train the neural network. There are many possible algorithms derived from Gradient Descent that we can use here.

I am going to use the AdamLearner. You can learn more about the Adam algorithm here: https://machinelearningmastery.com/adam...

These configuration values are a good starting point for many machine learning scenarios, but you can tweak them if you like to try and improve the quality of the predictions.

We’re almost ready to train. My final step is to set up a trainer and an evaluator for calculating the loss and the error during each training epoch:

The GetTrainer method sets up a trainer which will track the loss and the error for the training partition. And GetEvaluator will set up an evaluator that tracks the error in the test partition.

Now I am finally ready to start training the neural network!

I need to add the following code:

I am training the network for 10 epochs using a batch size of 64. During training I’ll track the loss and errors in the loss, trainingError and testingError arrays.

Once training is done, I show the final testing error on the console. This is the percentage of mistakes the network makes when predicting spam messages.

Note that the error and the accuracy are related: accuracy = 1 — error. So I also report the final accuracy of the neural network.

Here’s the code to train the neural network. This should go inside the for loop:

The Index().Shuffle().Batch() sequence randomizes the data and splits it up in a collection of 64-record batches. The second argument to Batch() is a function that will be called for every batch.

Inside the batch function I call GetBatch twice to get a feature batch and a corresponding label batch. Then I call TrainBatch to train the neural network on these two batches of training data.

The TrainBatch method returns the loss and error, but only for training on the 64-record batch. So I simply add up all these values and divide them by the number of batches in the dataset. That gives me the average loss and error for the predictions on the training partition during the current epoch, and I report this to the console.

So now I know the training loss and error for one single training epoch. The next step is to test the network by making predictions about the data in the testing partition and calculate the testing error.

This code goes inside the epoch loop and right below the training code:

I don’t need to shuffle the data for testing, so now I can call Batch directly. Again I’m calling GetBatch to get feature and label batches, but note that I am now providing the testing_data and testing_labels arrays.

I call TestBatch to test the neural network on the 64-record test batch. The method returns the error for the batch, and I again add up the errors for each batch and divide by the number of batches.

That gives me the average error in the neural network predictions on the test partition for this epoch.

After training completes, the training and testing errors for each epoch will be available in the trainingError and testingError arrays. Let’s use XPlot to create a nice plot of the two error curves so we can check for overfitting:

This code creates a Plot with two Scatter graphs. The first one plots the trainingError values and the second one plots the testingError values.

Finally I use File.WriteAllText to write the plot to disk as a HTML file.

I am now ready to build and run the app!

First I need to build the CNTKUtil library by running this command in the CNTKUtil folder:

$ dotnet build -o bin/Debug/netcoreapp3.0 -p:Platform=x64

This will build the CNKTUtil project. Note how I’m specifying the x64 platform because the CNTK library requires a 64-bit build.

Now I need to run this command in the SpamDetection folder:

$ dotnet build -o bin/Debug/netcoreapp3.0 -p:Platform=x64

This will build the app. Note how I’m again specifying the x64 platform.

Now I can run the app:

$ dotnet run

The app will create the neural network, load the dataset, train the network on the data, and create a plot of the training and testing errors for each epoch.

Here’s the neural network being trained on my laptop:

And here are the results:

The final classification error is 0 on training and 0.010 on testing. That corresponds to a final accuracy on testing of 0.99. This means the neural network makes 99 correct predictions for every 100 messages.

These seem like amazing results, notice how the training and testing curves start to diverge at epoch 2? The training error continues to converge towards zero while the testing error flatlines at 0.01. This is classic overfitting.

Overfitting means that the messages are too complex for the model to process. The mode is not sophisticated enough to capture the complexities of the patterns in the data.

And this is to be expected. Processing English text is a formidable problem and an area of active research, even today. A simple 32-node neural network is not going to be able to generate accurate spam predictions.

What if we increased the complexity of the neural network? Let’s double the number of nodes in the input and hidden layers:

The neural network now has 766,881 configurable parameters to train during each epoch. This is a massive network, but what will the results look like?

Well, check it out:

Nothing has changed. I’m still getting a training error of zero and a testing error of 0.01. And the curves still diverge, now at epoch 1.

Let’s go all out and crank up the number of nodes to 512:

The neural network now has an astounding 12,515,841 trainable parameters.

And here are the results:

Again no change. What’s going on here?

The reason this isn’t working is because the original neural network was big enough already. The 16 input nodes can check for the presence of 16 different words in a message to determine if the message is spam or not. That’s actually more than enough to do the job.

The reason I’m getting poor results is because the meaning of an English sentence is determined by the precise sequence of words in the sentence.

For example, in the text fragment “not very good”, the meaning of “good” is inverted by the presence of “not very”. If I simply check for the presence of the word “good”, I get a totally incorrect picture of the meaning of the sentence.

My neural network is looking at all the words in a message at once, ignores their order, and simply tests for spam by checking if certain words appear anywhere or not. This approach is not good enough for language processing.

So what do you think?

Are you ready to start writing C# machine learning apps with CNTK?