Spam Email Classifier with KNN — From Scratch (Python) | by Abdelrahman Ragab | Towards Data Science

KNN (K-Nearest Neighbours) is one of the very straightforward supervised learning algorithms. However, unlike the traditional supervised learning algorithms, such as Multinomial Naive Bayes algorithm, KNN doesn’t have an independent training stage, and then a stage where the labels for the test data are predicted based on the trained model. Rather, the features of every test data item are compared with the features of every training data item in real time, and then the K nearest training data items are selected, and the most frequent class among them is given to the test data item.
In the context of email classification (spam or ham), the features to be compared are the frequencies of words in each email. The euclidean distance is used to determine the similarity between two emails; the smaller the distance, the more similar. The Euclidean Distance formula used in the algorithm is as follows :
Once the Euclidean Distance between a test email and each training email is calculated, the distances are sorted in ascending order (nearest to farthest), and the K-nearest neighbouring emails are selected. If the majority is spam, then the test email is labelled as spam, else, it is labelled as ham.
In the example shown above, K = 5; we are comparing the email we want to classify to the nearest 5 neighbours. In this case, 3 out of 5 emails are classified as ham (non-spam), and 2 are classified as spam. Therefore, the unknown email will be given the class of the majority: ham. Now that we have seen how KNN works, let’s move on to implementing the classifier using code!
To have a quick idea of what we’ll be coding in Python, it’s always a good practice to write pseudo code:
1. Load the spam and ham emails2. Remove common punctuation and symbols3. Lowercase all letters4. Remove stopwords (very common words like pronouns, articles, etc.)5. Split emails into training email and testing emails6. For each test email, calculate the similarity between it and all training emails    6.1. For each word that exists in either test email or training email, count its frequency in both emails    6.2. calculate the euclidean distance between both emails to determine similarity7. Sort the emails in ascending order of euclidean distance8. Select the k nearest neighbors (shortest distance)9. Assign the class which is most frequent in the selected k nearest neighbours to the new email
The email data set for spam and ham (normal email) is obtained from “The Enron-Spam datasets”. It can be found at http://nlp.cs.aueb.gr/software_and_datasets/Enron-Spam/index.html under Enron2. The data set we’re using contains 5857 emails. Every email is stored in a text file, and the text files are divided and stored into two folders, ham folder, and spam folder. This means that the emails are already labelled. Every text file will be loaded by the program, and each email will be read and stored as a string variable. Every distinct word inside the string will be counted as a feature.
import osimport stringfrom nltk.corpus import stopwordsfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import accuracy_scoreimport numpy as np
os library to open and read files.string library for list of punctuationstopwords contains the list of stopwords.train_test_split to split the data into training and test data.accuracy_score to calculate accuracy of algorithms.numpy to allow advanced array manipulation.
def load_data():    print("Loading data...")        ham_files_location = os.listdir("dataset/ham")    spam_files_location = os.listdir("dataset/spam")    data = []
os.listdir() returns a list of all filenames inside a folder. This is used to retrieve all filenames of the text files in each of the ham and spam folder and store them in ham_files_location and spam_files_location respectively. data is a list to store each email text and its corresponding label.
    # Load ham email    for file_path in ham_files_location:        f = open("dataset/ham/" + file_path, "r")        text = str(f.read())        data.append([text, "ham"])        # Load spam email    for file_path in spam_files_location:        f = open("dataset/spam/" + file_path, "r")        text = str(f.read())        data.append([text, "spam"])
We iterate over the list of ham text filenames, use open() to open a file, then use str(f.read()) to read the email text as a string and store it in variable text. A list made of text and the corresponding label “ham” are appended to the list data.
    data = np.array(data)        print("flag 1: loaded data")    return data
The list data is transformed into a numpy array, to allow better manipulation of the array later. data is then returned.
# Preprocessing data: noise removaldef preprocess_data(data):    print("Preprocessing data...")        punc = string.punctuation           # Punctuation list    sw = stopwords.words('english')     # Stopwords list
punc holds a list of punctuation and symbolssw holds a list of stopwords from nltk.corpus library
    for record in data:        # Remove common punctuation and symbols        for item in punc:            record[0] = record[0].replace(item, "")
For every record in data, for every item (symbol or punctuation) in punc, replace the item with an empty string, to delete the item from record[0] (email text string).
        # Lowercase all letters and remove stopwords         splittedWords = record[0].split()        newText = ""        for word in splittedWords:            if word not in sw:                word = word.lower()                newText = newText + " " + word          record[0] = newText            print("flag 2: preprocessed data")            return data
Use split() method on email text record[0] to return a list of all words in the email. Iterate over that list of words, and if the word is not in stopwords list, set it to lowercase, and add the word to newText. newText will contain the email but empty of stopwords. newText is assigned back to record[0]. After every record[0] is preprocessed, the clean data is returned.
The data set is split into a training set (73%) and a testing set (27%).
# Splitting original dataset into training dataset and test datasetdef split_data(data):    print("Splitting data...")        features = data[:, 0]   # array containing all email text bodies    labels = data[:, 1]     # array containing corresponding labels    print(labels)    training_data, test_data, training_labels, test_labels =\        train_test_split(features, labels, test_size = 0.27, random_state = 42)        print("flag 3: splitted data")    return training_data, test_data, training_labels, test_labels
First, it was necessary to have the email texts in an array of their own, and labels in another array of their own. So, the email texts were stored in features and the labels were stored in labels. train_test_split method was then used to split the data into training_data, test_data, training_labels, and test_labels. The random state was set to 42 to ensure the same output of random shuffling will be obtained for testing purposes. After splitting, training_data, test_data, training_labels, and test_labels are returned.
get_count() function
def get_count(text):    wordCounts = dict()    for word in text.split():        if word in wordCounts:            wordCounts[word] += 1        else:            wordCounts[word] = 1        return wordCounts
This function takes a single email text, and splits it using split(). The frequency of occurrence of each word in the email is counted and saved in wordCounts, which is of dictionary data type. The dictionary wordCounts is then returned.
euclidean_difference() function
def euclidean_difference(test_WordCounts, training_WordCounts):    total = 0
This function takes in a dictionary of word counts of a test email test_WordCounts, and another dictionary of a training email, training_wordCounts. total stores the sum of the squared difference of the frequency of a word in the test and training email.
    for word in test_WordCounts:        if word in test_WordCounts and word in training_WordCounts:            total += (test_WordCounts[word] - training_WordCounts[word])**2
First, we iterate over the words in the test email dictionary. For each word, there are three cases. First case is that it exists in both, test email and training email. In that case, total is incremented with the squared difference of the word’s frequency in the test email and training email.
            del training_WordCounts[word]
Then the common word is removed from the training email dictionary, to speed up the next for-loop
        else:            total += test_WordCounts[word]**2
Second case is that the word is in test email only. In that case there is no need to find the difference (since its frequency is 0 in training email), so we just add the word’s squared frequency to total.
    for word in training_WordCounts:            total += training_WordCounts[word]**2
The last case is that the word is only in the training email. Since we deleted all common words in the previous for loop, we just loop through the training email dictionary and add every word’s squared frequency to total.
    return total**0.5
Finally, the square root of total (the square root of the sum of the squared difference of frequencies of each word) is returned as a double. This is the end of the Euclidean Distance calculation function.
get_class() function
def get_class(selected_Kvalues):    spam_count = 0    ham_count = 0
This function takes in the list of the selected K nearest neighbours to determine the class of the current test email. spam_count and ham_count store the frequency of occurrence of each “spam” label and “ham” label respectively within the K selected nearest neighbours.
    for value in selected_Kvalues:        if value[0] == "spam":            spam_count += 1        else:            ham_count += 1
Using a for loop, for each value in the K selected values, if the label value[0] is equal to “spam”, then spam_count is incremented by 1. Else, ham_count is incremented by 1.
    if spam_count > ham_count:        return "spam"    else:        return "ham"
After the for loop, if spam_count is greater than ham_count, it means that the current test email has a greater tendency to be spam, so a string “spam” is returned as the predicted label. Else, the string “ham” is returned as the predicted label.
knn_classifier() function
def knn_classifier(training_data, training_labels, test_data, K, tsize):    print("Running KNN Classifier...")        result = []    counter = 1
This is the KNN classifier function. It takes in the training email, training labels, test data, the K value, and the number of test emails to be tested out of the original 27% test emails. result is the list which would contain the predicted labels. counter will be just used for display purposes to indicate progress when the program is run.
    # word counts for training email    training_WordCounts = []     for training_text in training_data:            training_WordCounts.append(get_count(training_text))
Since the training set is constant, we can count the word frequency in each training email once and for all. So, for each email text in training data, its word frequency dictionary is obtained using get_count(). The dictionary is then appended to the training_WordCounts list to be stored.
    for test_text in test_data:        similarity = [] # List of euclidean distances        test_WordCounts = get_count(test_text)  # word counts for test email
Now, for every test email in the test data, the following is done. An empty list similarity is declared. It will store the Euclidean distances between the current test email and each training email. Then, the test email’s word frequency dictionary is obtained using get_count().
        # Getting euclidean difference         for index in range(len(training_data)):            euclidean_diff =\                euclidean_difference(test_WordCounts, training_WordCounts[index])            similarity.append([training_labels[index], euclidean_diff])
Since we already have the word frequency dictionaries of all training emails, and the current test email. We can go ahead calculate the Euclidean distance between the current test email, and each training email using a for-loop which iterates x times, where x is equal to the size of the training data set. After every iteration, the Euclidean distance calculated is appended to the similarity list, along with the corresponding label of the training email.
        # Sort list in ascending order based on euclidean difference        similarity = sorted(similarity, key = lambda i:i[1])
After all Euclidean distances have been stored. We sort the similarity list in ascending order based on the second column, i.e. based on the Euclidean distance (nearest to farthest).
        # Select K nearest neighbours        selected_Kvalues = []         for i in range(K):            selected_Kvalues.append(similarity[i])
Now, since the similarity list is already sorted, we can easily append the nearest K neighbours to the selected_Kvalues list using a simple for loop.
        # Predicting the class of email        result.append(get_class(selected_Kvalues))
Lastly, and before moving on to the next test email. We determine the class of the current test email using get_class(). Now, we have reached the end of one iteration, and the next iteration can begin to classify the next test email.
    return result
Once all test emails have been classified, and the for-loop has reached its end, the result list, which contains the list of predicted labels, is returned.
main() function
def main(K):    data = load_data()    data = preprocess_data(data)    training_data, test_data, training_labels, test_labels = split_data(data)
This is the main function where the program begins to run. This is where everything is put together. The main function takes in the K value. First, all emails are loaded using load_data() and then stored in data. The emails are then preprocessed using preprocess_data() and stored again in data. data is then split into training_data, test_data, training_labels, and test_labels using split_data().
    tsize = len(test_data)
tsize specifies the number of test emails (out of the original 27% test data) to predict their labels. Currently tsize is set to be equal to the entire set of test emails.
    result = knn_classifier(training_data, training_labels, test_data[:tsize], K, tsize)     accuracy = accuracy_score(test_labels[:tsize], result)
Now, the knn_classifier() function is called to predict the labels for the test emails. The returned list of predicted labels is stored in result. After that, accuracy is calculated using the accuracy_score() method from sklearn library. This method compares the actual labels list test_labels with the predicted labels list result.
    print("training data size\t: " + str(len(training_data)))    print("test data size\t\t: " + str(len(test_data)))    print("K value\t\t\t\t: " + str(K))    print("Samples tested\t\t: " + str(tsize))    print("% accuracy\t\t\t: " + str(accuracy * 100))    print("Number correct\t\t: " + str(int(accuracy * tsize)))    print("Number wrong\t\t: " + str(int((1 - accuracy) * tsize)))
These lines display the details of the run such as the training data size, test data size, K value, number of samples tested, percentage accuracy, number of emails correctly identified, and number of emails falsely identified.
main(11)
Finally, this is the line which initiates the program by calling the main function, and gives it the K value (which is 11 in this case).
This is the final output of all the code that has been explained above. It can be seen that using KNN algorithm to classify email into spam and ham, with a K value of 11, and test data size 1582, it gives a 76.7% accuracy rate. Though not the best, it is satisfactory. One downside to note is that it takes a long time to classify the 1582 emails. This is primarily due to the high time complexity, which is the result of three nested for-loops when calculating the euclidean difference between test emails and training emails.
You can find the full source-code on my Github repository here.