Natural Language Processing (NLP) for all Levels in Python
By: Sheenal Srivastava
Tags: Python Programming, Python, Naturallanguageprocessing, Text Analytics, NLP

Natural Language Processing (NLP) for all Levels in Python

Background

Natural Language Processing or NLP is all the rage currently and it can sound quite confronting to someone who is new to Data Science or new to the Python coding language. Having always written code in R, people often tend to dismiss me for not being a ‘real Data Scientist’ or probably not knowing anything about NLP. However, this is very far from reality.

NLP is literally the processing of textual data to gain insights from it. The beauty of programming languages like R and Python is that they have a variety of packages that can derive meaning from human languages. This includes, but is not limited to, grammar, translation, and sentiment analysis.

This article goes through the basics of NLP in Python with code examples to complete common NLP tasks.

Packages required

The most common packages used for NLP in Python are SPACY and NLTK. To import these packages into your script, refer to the below code block.

NLTK is a large package and as such you need to download various components of it (such as stopwords) for text processing.

Figure 1: Python packages for NLP — Image by Author

Now, let’s look at common tasks that you may need to carry out when dealing with text.

Text Pre-Processing

It is always a good idea to pre-process text to get rid of special characters, punctuation, replace empty strings, remove numbers, and convert everything to the same case (lowercase for example) so that identical words are not seen as distinct words because of their case. The following code block can be useful.

Figure 2: Text pre-processing — Image by Author

Summarizing text

Often, we may want to summarize the text by determining the average number of words and average number of characters for all strings in a given column.

Figure 3: Summarizing text — Image by Author

Finding particular words or phrases in a string

Regular expression matching or looking for words or phrases within a string is a common function that is performed. We can do this by simply checking to see if one string is within another. I have done this within a function below where my score allocation is based on the number of substrings that are found within “my_string”.

Figure 4: Checking for substrings within a string — Image by Author

We can also look for exact matches of strings using the “findall” function. In the code below, I am looking for bi-grams (phrases consisting of two words) within my column of interest. I use “set” to only retain unique matches. The function “len” is used to count these matches.

Figure 5: Checking for exact matches within a string — Image by Author

Measuring the similarity between text fields

There are smart ways to check whether two strings are similar and Python has an entire package devoted to it called textdistance. The package contains more than 30 algorithms that allows one to compare strings. In the example below, I have used cosine similarity to measure the similarity between the summary text between two documents. This metric calculates the dot product between the vectors of two strings in order to find the cosine angle between them. It judges similarity based on the orientation between the strings rather than magnitude/length of the strings.

Cosine similarity requires a string to be split into tokens or words in a sentence for comparison.

Figure 6: Calculating cosine similarity between strings — Image by Author

Counting word frequencies

I often get asked to determine word frequencies. Yes, word clouds are good for visualisation, but what if you wanted to list this information and store it in .csv file.

In the code below, I take a column called “Description” and join all the text into one long list. I then remove stop words (commonly used words that do not carry meaning such as “a”, “the”, “then” etc). I also perform lemmatization. This is to remove words that are identical in their purest form such as typing and type, organizations and organization.

I then use the “Collections” package to find the top 200 words and bi-grams by their frequency, which I then output into a csv file. Python is nifty in that it has functions that allow you to measure the frequency of n-grams (unigrams, tri-grams, etc).

Figure 7: Counting word and bi-gram frequencies — Image by Author

Summary

Hopefully, you can now successfully do the following:

Process text to get a clean version Summarize text Find substrings within long strings Match text strings Measure the similarity between text Calculate the frequency of words or bi-grams within a string

Let me know what you think of this blog post and other common NLP functions that you have come across. Please note that I did not cover word clouds and sentiment analysis because I go over that in my article here: https://towardsdatascience.com/social-listening-via-twitter-feed-analysis-23fb4fbe7531