Creating a spam filter using Naive Bayes with TextAnalysis.jl | by Kim Fung | Towards Data Science

Hello! Today I’m gonna tell you more about what I did to make a spam filter using Naive Bayes to detect spam data from this dataset on kaggle by UCI Machine Learning, along with the use of TextAnalysis.jl on Julia.
I started off by looking at the docs off TextAnalysis.jl to understand more about how exactly the NaiveBayes Classifier works.
using TextAnalysis: NaiveBayesClassifier, fit!, predictm = NaiveBayesClassifier([:legal, :financial])fit!(m, "this is financial doc", :financial)fit!(m, "this is legal doc", :legal)predict(m, "this should be predicted as a legal document")
I ran the example from the docs and I learned that the function NaiveBayesClassifier takes in the argument of an array of possible classes that the concerned data could perhaps belong to.
In this case, it was :legal and :financial. I also learned that we have train the model by fitting the concerned data with the fit! function where it takes in the arguments of the model itself, the string of data we are trying to train, and the class that data belongs to. The data here is a string, for example “this is financial doc” and the class it belongs to is, in this case, :financial.
Finally, I learned that the predict function allows us to enter a string of data and uses the NaiveBayesClassifier algorithm to predict what class the string belongs to, based on the strings of data trained before using the fit! function. The predict function takes in the arguments of the model itself as well as the string of data we are trying to predict.
My first approach to the problem was that I thought it would be a good idea to first import all of the data. As I have experience with the packages CSV.jl and DataFrames.jl, I am familiar with the importing of the data.
using CSV, DataFramesspamdata = DataFrame(CSV.read("spam.csv"; allowmissing=:none))---------------------------------------------------------------julia> showall(spamdata)5572×5 DataFrame│ Row  │ v1     ││      │ String │├──────┼────────┤│ 1    │ ham    ││ 2    │ ham    ││ 3    │ spam   ││ 4    │ ham    ││ 5    │ ham    ││ 6    │ spam   ││ 7    │ ham    ││ 8    │ ham    ││ 9    │ spam   ││ 10   │ spam   │
This image below shows the structure of the original .csv file with the data.
The csv file has two columns. v2 is the string of the data we want to use to train and v1 is the class of the particular string of data, corresponding to v2.
I want a way to loop through every single row of the file and split the ham data in one condition and spam data in another, such that when it comes to training, I will be able to train the ham data using the fit! function, which requires me to specify the class of that data.
for row in eachrow(spamdata)    if row.v1 == "ham"        println("ham")    elseif row.v1 == "spam"        println("spam")    endend
This was a success and I ended up having hams and spams being printed!
hamspamhamspamhamhamspamspamhamspam⋮
Now that it is out of the way, I can define my model using the NaiveBayesClassifier function. I want to define 2 classes, :ham and :spam.
using TextAnalysis: NaiveBayesClassifier, fit!, predictm = NaiveBayesClassifier([:ham, :spam])
Next, I want to start training my model. As we saw from the original .csv file’s structure, v2 is the string we are trying to train. Combining the for loop with the fit! function, I did the following to try and train all the data we have available.
using CSV, DataFramesusing TextAnalysis: NaiveBayesClassifier, fit!, predictspamdata = DataFrame(CSV.read("spam.csv"; allowmissing=:none))global m = NaiveBayesClassifier([:ham, :spam])for row in eachrow(spamdata)    if row.v1 == "ham"        fit!(m, row.v2, :ham)    elseif row.v1 == "spam"        fit!(m, row.v2, :spam)    endend
But when I ran that, I got the following error: LoadError: Base.InvalidCharError{Char}('\xe5\xa3')
It was then when I realized that the dataset had invalid characters in certain strings in the v2 column. To eliminate this error, we need to filter out the unsupported characters using the following function: filter(isvalid, <string>)
for row in eachrow(spamdata)    if row.v1 == "ham"        fit!(m, filter(isvalid, row.v2), :ham)    elseif row.v1 == "spam"        fit!(m, filter(isvalid, row.v2), :spam)    endend
When I replaced the string row.v2 with filter(isvalid, row.v2) and ran the program once again, no errors came up. Therefore the model was successfully trained and so far so good!
Nothing was actually printed out in the REPL because there was no error and nothing in the program had anything printed out. To fully test that the model is working, we can try and create a prediction and print out the results of the prediction to see whether the training was really done or not.
prediction1 = predict(m, "hello my name is kfung")prediction2 = predict(m, "text 31845 to get a free phone")println(prediction1)println(prediction2)
Here I created two predictions, where the first one looks like a ham message (as it doesn’t look suspicious), and a second one that looks like a spam message to test out the model.
The results I got are below:
Dict(:spam => 0.013170434049325023, :ham => 0.986829565950675)Dict(:spam => 0.9892304346396908, :ham => 0.010769565360309069)
As we can tell, the predictions were pretty accurate as the first prediction had a :ham value close to 1 meaning it is most probably a ham message, and the second prediction had a :spam value close to 1 meaning it is most likely a spam message. Exactly what we expected.
But for users who are not familiar with dictionaries or Julia syntax, they may be confused as to what the dictionaries above mean. I modified the code so that it checks for the :spam and the :ham values in the dictionary and prints out the class of the bigger value of those two.
prediction = predict(m, "hello my name is kfung")if prediction[:spam] > prediction[:ham]    println("spam")else    println("ham")end
As we expected, the results of this program was the string “ham” because the :ham value it predicted was greater than the :spam value, therefore it is more likely to be a ham message.
In the end, I wrapped everything in a function that takes a string as an argument so that when you call the function with a string, it will print out either “spam” if the model predicts it as a spam message, or “ham” if the model predicts that it is not a spam message.
using CSV, DataFramesusing TextAnalysis: NaiveBayesClassifier, fit!, predictfunction checkspam(msg::String)    spamdata = DataFrame(CSV.read("spam.csv"; allowmissing=:none))    m = NaiveBayesClassifier([:ham, :spam])    for row in eachrow(spamdata)        if row.v1 == "ham"            fit!(m, filter(isvalid, row.v2), :ham)        elseif row.v1 == "spam"            fit!(m, filter(isvalid, row.v2), :spam)        end    end    prediction = predict(m, msg)    if prediction[:spam] > prediction[:ham]        println("spam")    else        println("ham (not spam)")    endend
In the end I realized how I was training the model every time a classify a message. The makes the program not very efficient at run time because every time we are trying to predict using the model, it has gone through through the 5600 lines of data all over again. Instead, we can bring the modal outside of the function (so that it is run at the start only once) and store the model in a global variable so that afterwards it can just used the pre-trained model stored in the global variable to classify any other messages.
using CSV, DataFramesusing TextAnalysis: NaiveBayesClassifier, fit!, predictspamdata = DataFrame(CSV.read("spam.csv"; allowmissing=:none))global m = NaiveBayesClassifier([:ham, :spam])for row in eachrow(spamdata)    if row.v1 == "ham"        fit!(m, filter(isvalid, row.v2), :ham)    elseif row.v1 == "spam"        fit!(m, filter(isvalid, row.v2), :spam)    endendfunction checkspam(msg::String)    prediction = predict(m, msg)    if prediction[:spam] > prediction[:ham]        println("spam")    else        println("ham (not spam)")    endend
Overall, I think this was a very successful attempt at creating a new spam filter. I learnt more about how I could apply Naive Bayes to model independent assumptions for probabilistic classification. Naive Bayes continues to be at the moment a rather popular option for text categorisation, of which this is an example, and can be contrasted to other methods of model training such as logistic regression where the data is conditionally trained. This model has other applications such as sentiment analysis which is what I’ll be focusing on soon in another article, so stay tuned!
Thanks so much for reading!