Natural Language Processing in Python with Code (Part II)
By: Meet Nandu
Tags: Artificial Intelligence, Machine Learning, Technology, AI, Data Science

In the last part, I explained what Natural Language Processing exactly is. Then, we saw the data set with which we are trying to understand NLP. A bit of text pre-processing was done on the data with the help of bag-of-words technique to normalize the data. If you want to check out the first part, click here.

Continuing Normalizing

There are a lot of ways to continue normalizing this text. Such as Stemming or distinguishing by part of speech.

NLTK has lots of built-in tools and great documentation on a lot of these methods. Sometimes they don’t work well for text-messages due to the way a lot of people tend to use abbreviations or shorthand, For example:

'Nah dawg, IDK! Wut time u headin to da club?'

versus

'No dog, I don't know! What time are you heading to the club?'

Some text normalization methods will have trouble with this type of shorthand and so I’ll leave you to explore those more advanced methods through the NLTK book online.

For now we will just focus on using what we have to convert our list of words to an actual vector that SciKit-Learn can use.

Vectorization

Currently, we have the messages as lists of tokens (also known as lemmas) and now we need to convert each of those messages into a vector the SciKit Learn’s algorithm models can work with.

Now we’ll convert each message, represented as a list of tokens (lemmas) above, into a vector that machine learning models can understand.

We’ll do that in three steps using the bag-of-words model:

Count how many times does a word occur in each message (Known as term frequency) Weigh the counts, so that frequent tokens get lower weight (inverse document frequency) Normalize the vectors to unit length, to abstract from the original text length (L2 norm)

Let’s begin the first step:

Each vector will have as many dimensions as there are unique words in the SMS corpus. We will first use SciKit Learn’s CountVectorizer. This model will convert a collection of text documents to a matrix of token counts.

We can imagine this as a 2-Dimensional matrix. Where the 1-dimension is the entire vocabulary (1 row per word) and the other dimension are the actual documents, in this case a column per text message.

Since there are so many messages, we can expect a lot of zero counts for the presence of that word in that document. Because of this, SciKit Learn will output a Sparse Matrix.

from sklearn.feature_extraction.text import CountVectorizer

There are a lot of arguments and parameters that can be passed to the CountVectorizer. In this case we will just specify the analyzer to be our own previously defined function:

# Might take awhile...

bow_transformer = CountVectorizer(analyzer=text_process).fit(messages['message'])



# Print total number of vocab words

print(len(bow_transformer.vocabulary_)) 11444

Let's take one text message and get its bag-of-words counts as a vector, putting to use our new bow_transformer :

message4 = messages['message'][3]

print(message4)

U dun say so early hor... U c already then say...

Now let's see its vector representation:

bow4 = bow_transformer.transform([message4])

print(bow4)

print(bow4.shape) (0, 4073) 2

(0, 4638) 1

(0, 5270) 1

(0, 6214) 1

(0, 6232) 1

(0, 7197) 1

(0, 9570) 2

(1, 11444)

This means that there are seven unique words in message number 4 (after removing common stop words). Two of them appear twice, the rest only once. Let's go ahead and check and confirm which ones appear twice:

print(bow_transformer.get_feature_names()[4073])

print(bow_transformer.get_feature_names()[9570]) U

say

Now we can use .transform on our Bag-of-Words (bow) transformed object and transform the entire DataFrame of messages. Let's go ahead and check out how the bag-of-words counts for the entire SMS corpus is a large, sparse matrix:

messages_bow = bow_transformer.transform(messages['message'])

print('Shape of Sparse Matrix: ', messages_bow.shape)

print('Amount of Non-Zero occurences: ', messages_bow.nnz) Shape of Sparse Matrix: (5572, 11444)

Amount of Non-Zero occurences: 50795

sparsity = (100.0 * messages_bow.nnz / (messages_bow.shape[0] * messages_bow.shape[1]))

print('sparsity: {}'.format(round(sparsity))) sparsity: 0

After the counting, the term weighting and normalization can be done with TF-IDF, using scikit-learn's TfidfTransformer .

So what is TF-IDF?

TF-IDF stands for term frequency-inverse document frequency, and the tf-idf weight is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus. Variations of the tf-idf weighting scheme are often used by search engines as a central tool in scoring and ranking a document’s relevance given a user query.

One of the simplest ranking functions is computed by summing the tf-idf for each query term; many more sophisticated ranking functions are variants of this simple model.

Typically, the tf-idf weight is composed by two terms: the first computes the normalized Term Frequency (TF), aka. the number of times a word appears in a document, divided by the total number of words in that document; the second term is the Inverse Document Frequency (IDF), computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears.

TF: Term Frequency, which measures how frequently a term occurs in a document. Since every document is different in length, it is possible that a term would appear much more times in long documents than shorter ones. Thus, the term frequency is often divided by the document length (aka. the total number of terms in the document) as a way of normalization:

TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document).

IDF: Inverse Document Frequency, which measures how important a term is. While computing TF, all terms are considered equally important. However it is known that certain terms, such as “is”, “of”, and “that”, may appear a lot of times but have little importance. Thus we need to weigh down the frequent terms while scale up the rare ones, by computing the following:

IDF(t) = log_e(Total number of documents / Number of documents with term t in it).

See below for a simple example.

Example:

Consider a document containing 100 words wherein the word cat appears 3 times.

The term frequency (i.e., tf) for cat is then (3 / 100) = 0.03. Now, assume we have 10 million documents and the word cat appears in one thousand of these. Then, the inverse document frequency (i.e., idf) is calculated as log(10,000,000 / 1,000) = 4. Thus, the Tf-idf weight is the product of these quantities: 0.03 * 4 = 0.12.

Let’s go ahead and see how we can do this in SciKit Learn:

from sklearn.feature_extraction.text import TfidfTransformer



tfidf_transformer = TfidfTransformer().fit(messages_bow)

tfidf4 = tfidf_transformer.transform(bow4)

print(tfidf4) (0, 9570) 0.538562626293

(0, 7197) 0.438936565338

(0, 6232) 0.318721689295

(0, 6214) 0.299537997237

(0, 5270) 0.297299574059

(0, 4638) 0.266198019061

(0, 4073) 0.408325899334

We'll go ahead and check what is the IDF (inverse document frequency) of the word "u" and of word "university" ?

print(tfidf_transformer.idf_[bow_transformer.vocabulary_['u']])

print(tfidf_transformer.idf_[bow_transformer.vocabulary_['university']]) 3.28005242674

8.5270764989

To transform the entire bag-of-words corpus into TF-IDF corpus at once:

messages_tfidf = tfidf_transformer.transform(messages_bow)

print(messages_tfidf.shape) (5572, 11444)

There are many ways the data can be preprocessed and vectorized. These steps involve feature engineering and building a "pipeline". I encourage you to check out SciKit Learn's documentation on dealing with text data as well as the expansive collection of available papers and books on the general topic of NLP.

Training a model

With messages represented as vectors, we can finally train our spam/ham classifier. Now we can actually use almost any sort of classification algorithms. For a variety of reasons, the Naive Bayes classifier algorithm is a good choice.

We’ll be using scikit-learn here, choosing the Naive Bayes classifier to start with:

from sklearn.naive_bayes import MultinomialNB

spam_detect_model = MultinomialNB().fit(messages_tfidf, messages['label'])

Let's try classifying our single random message and checking how we do:

print('predicted:', spam_detect_model.predict(tfidf4)[0])

print('expected:', messages.label[3]) predicted: ham

expected: ham

Fantastic! We've developed a model that can attempt to predict spam vs ham classification!

Part 6: Model Evaluation

Now we want to determine how well our model will do overall on the entire dataset. Let’s begin by getting all the predictions:

all_predictions = spam_detect_model.predict(messages_tfidf)

print(all_predictions) ['ham' 'ham' 'spam' ..., 'ham' 'ham' 'ham']

We can use SciKit Learn’s built-in classification report, which returns precision, recall, f1-score, and a column for support (meaning how many cases supported that classification). Check out the links for more detailed info on each of these metrics and the figure below:

from sklearn.metrics import classification_report

print (classification_report(messages['label'], all_predictions)) precision recall f1-score support



ham 0.98 1.00 0.99 4825

spam 1.00 0.85 0.92 747



avg / total 0.98 0.98 0.98 5572

There are quite a few possible metrics for evaluating model performance. Which one is the most important depends on the task and the business effects of decisions based off of the model. For example, the cost of mis-predicting "spam" as "ham" is probably much lower than mis-predicting "ham" as "spam".

In the above "evaluation", we evaluated accuracy on the same data we used for training. You should never actually evaluate on the same dataset you train on!

Such evaluation tells us nothing about the true predictive power of our model. If we simply remembered each example during training, the accuracy on training data would trivially be 100%, even though we wouldn’t be able to classify any new messages.

A proper way is to split the data into a training/test set, where the model only ever sees the training data during its model fitting and parameter tuning. The test data is never used in any way. This is then our final evaluation on test data is representative of true predictive performance.

There is a lot more to Natural Language Processing than what we’ve covered here. We have always heard “This call may be recorded for quality and training purposes” when we call the company’s call centres for required services. Although some calls are used for training purposes, but these are even used to improve natural language processing algorithms.

In the end, I would like to thank my professor — Jose Marcial Portilla, whose course ‘Python for Data Science and Machine Learning Bootcamp’ on Udemy made it all easy for me to understand all these concepts.