Building a Naive Bayes Machine Learning Model to Classify Text | by Aden Haussmann | Towards Data Science

Natural Language Processing (NLP) is an extremely exciting field. It lies at the confluence of computer science, linguistics and artificial intelligence, and is concerned with the interaction between human language and computers. More specifically: its goal is to understand how to program computers to understand and interpret our natural language.
It’s a very hot area of research right now, and I’m fortunate enough to be attending one of the universities at the absolute forefront of this research. Although as a lowly undergrad, I’m not exposed to this cutting-edge work often — as if I would understand it if I was!
Nonetheless, building models to classify natural language is relatively straightforward. It’s a cool exercise because it’s relevant. This is a very real application of ML that you could use in your own projects.
One of the simpler supervised Bayesian Network models, the Naive Bayes algorithm is a probabilistic classifier based on Bayes’ Theorem (which you might remember from high school statistics). But its simplicity doesn’t make it a poor choice, it can produce highly accurate predictions even when the dataset is not very large (just a few thousand samples).
If you’re really new to Machine Learning, I suggest reading some of my articles on more basic algorithms, and then coming back to this one, as I will be building upon concepts that are explained in detail in those.
towardsdatascience.com
Essentially, the conditional probabilities of the occurrence of two events are computed, based on the conditional probabilities of each individual event. So, the probability of each tag for a given piece of text is calculated, and the tag with the highest probability is outputted.
There are other suitable options as well, such as Support Vector Machine (SVM), which takes more time and computational resources to work, but will produce more accurate predictions than Naive Bayes. Alternatively, you might consider a Deep Learning approach based on neural networks, but this would require far more training data.
This is the approach:
Import and setup the data.Do some analysis to learn more about the data landscape.Create our dependent and independent variable lists for training and validation.Encode the labels.Extract features from the descriptions.Fit the data to the model.Check the model’s accuracy.
Import and setup the data.
Do some analysis to learn more about the data landscape.
Create our dependent and independent variable lists for training and validation.
Encode the labels.
Extract features from the descriptions.
Fit the data to the model.
Check the model’s accuracy.
What this project will demonstrate is classifying bank transactions into categories based on their descriptions. My dataset, which contains 12500 samples, includes other features like the transaction amount and transaction type, and you can use as many features as you want in your model — even do some feature selection to see which ones have the highest impact on the predictions — but for the sake of simplicity, we’ll just use the descriptions.
In my dataset, the descriptions are things like:
citylink1Jul19 OYSTERtravelodge6Jul19 RINGGOSUNDRY DEBIT CONTACTLESS CAMDEN PARKINGstgcoachtrainlineFin: CMT UK LTD  Cash at Transact
Of course, start with the imports.
Pandas for creating dataframes.Scikit-Learn for label encoding, feature extraction, modelling and measuring success metrics
Pandas for creating dataframes.
Scikit-Learn for label encoding, feature extraction, modelling and measuring success metrics
import pandas as pdfrom sklearn.preprocessing import LabelEncoderfrom sklearn.feature_extraction.text import CountVectorizerfrom sklearn import naive_bayes, metrics
Next, let’s process the csv files and get them ready for the model. (Of curse, you may not be working with csv files, just import whatever data you have into a Pandas dataframe):
features = pd.read_csv("bank_transaction_features.csv")labels = pd.read_csv("bank_transaction_labels.csv")
I have two datasets: one with features, one with labels (categories). These are their respective columns:
bank_transaction_features:bank_transaction_id, bank_transaction_amount, bank_transaction_typebank_transaction_labels:bank_transaction_id, bank_transaction_category, bank_transaction_dataset
They have in common, an ID column. So I’ll merge them on this column and delete any rows with null entries (in my data there are very few — you should calculate how many — so removing them will have negligible effect on the predictions):
combined_df = pd.merge(left=features, right=labels)combined_df = combined_df.dropna()
Always begin with some exploratory data analysis. My dataset has thousands of samples, I can’t see what all the categories are just by scanning it. We need to do some simple things to gain a better understanding of the data.
This dataset has a column that designates whether a sample is for training or validating. This will come in handy later as we won’t need to create our own train/validate split. But for exploratory analysis, we can drop this column (taking to only drop the column from a new dataframe variable which we will create, retaining the one with this column).
explore_df = combined_df.drop(labels=['bank_transaction_dataset'], axis=1)
We can easily see what categories are present in the data as well as how many samples there are in each category:
print(explore_df['bank_transaction_category'].value_counts())
The result:
ACCOMMODATION_AND_MEALS    3765TRAVEL                     3166BANK_OR_FINANCE_CHARGES    2659MOTOR_EXPENSES             1609INSURANCE                  1170Name: bank_transaction_category, dtype: int64
Now let’s just check what the train/validate split is (using the dataframe which still has the relevant column):
train_set = combined_df.loc[combined_df["bank_transaction_dataset"] == "TRAIN"]val_set = combined_df.loc[combined_df["bank_transaction_dataset"] == "VAL"]len_train = len(train_set)len_val = len(val_set)len_whole = len(explore_df)print('Amount of training data: ', len_train)print('Amount of validation data: ', len_val)
Output:
Amount of training data:  9891Amount of validation data:  2478
Which makes the train/validate split 80/20, a good ratio, so no need to adjust that at all.
Finally, the fun bit. First, create the x and y training and validating subsets. We can accomplish this by creating lists containing only the data in the relevant columns:
y_train = train_set['bank_transaction_category'].valuesx_train = train_set['bank_transaction_description'].valuesy_val = val_set['bank_transaction_category'].valuesx_val = val_set['bank_transaction_description'].values
Now it gets really interesting. The thing to keep in mind is that ML models don’t “understand” text and words. They understand numbers. Therefore, the first thing that needs to be done to prepare the data to be fitted to the model, is to encode the labels. All that means, is assigning a number to each label. For example:
ACCOMMODATION_AND_MEALS    => 0TRAVEL                     => 1BANK_OR_FINANCE_CHARGES    => 2MOTOR_EXPENSES             => 3INSURANCE                  => 4
This is accomplished by creating a LabelEncoder object and using its fit_transform function on the dependent variable data (y):
label_encoder = LabelEncoder()y_train = label_encoder.fit_transform(y_train)y_test = label_encoder.fit_transform(y_val)
The same goes for the description data, except this is a little more complicated. In some models, where you have a set number of uniform descriptions and you know what they are, you can just encode each one to an integer and they’re ready to go. But here, each transaction description is potentially unique.
The solution is to extract features from the text and turn those into vectors that can be understood by the model.
Although technically complicated, this can be implemented simply, by using the CountVectorizer to convert the texts into a matrix of tokens and transforming the training and validation independent variable (x):
count_vector = CountVectorizer(analyzer='word', token_pattern=r'\w{1,}')count_vector.fit(combined_df['bank_transaction_description'])x_train_count = count_vector.transform(x_train)x_valid_count = count_vector.transform(x_val)
And that’s it! We can fit the data, train the model and make predictions:
classifier = naive_bayes.MultinomialNB()classifier.fit(x_train_count, y_train)predictions = classifier.predict(x_valid_count)
There are several metrics you can use to determine how well the model is working. We’ll use accuracy. this will tell us how often the model is correctly predicting the category of the bank transactions:
print(metrics.accuracy_score(predictions, y_test))
The accuracy for this model was:
0.91
Not bad at all.
This was a fun example because it was tangible. Classifying bank transactions into categories in, for example, a mobile finance tracking app, is a very real use case for this algorithm. I hope you gained a good high level understanding of how Naive Bayes works, and how to implement it for classifying text, specifically.
If you did learn something, have any more questions, think I missed anything important, or are planning to use this algorithm in a project of your own, let me know by leaving a response and we can discuss it.
Happy coding!
Scikit Learn sklearn.preprocessing.LabelEncoder https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html
Scikit Learn sklearn.feature_extraction.text.CountVectorizer https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html
Wikipedia Natural Language Processing https://en.wikipedia.org/wiki/Natural_language_processing
MonkeyLearn Text Classification https://monkeylearn.com/text-classification/