Introducing Natural Language Processing
By: Ai, Port Harcourt
Tags: Text Mining, Data Science, Naturallanguageprocessing, Beginners Guide, NLP

A beginner’s guide to NLP and extracting insight from text data

Photo by Taylor Wilcox on Unsplash

Before we had computers, papers, ink or Lead to make pencils, human communication was oral. Languages evolved and with it, structure and semantics. Languages went from strictly oral form to oral and written form. NLP is the process of making machines understand human language — text or speech.

Natural Language Processing (NLP) has gained more popularity and advancement over the years. The ability to draw insight from text data has opened a world of opportunities. Text data is abundant, from books to tweets, movie or product review, transcripts, meeting proceedings and health records.

With Natural language Processing techniques, Machine Learning and Deep learning, we can turn this jumble of data into valuable insight.

While these text data are numerous, they pose a problem. They are largely unstructured and have to be converted into a format which algorithms can understand. Here we will go through the applications of NLP, data preprocessing for text data and extracting features from text data.

Applications of NLP

Understanding consumer’s reaction to product or service also known as Sentiment Analysis

Extract topics from text or document: Topic Modelling

Text classification, product classification

Chatbox and AI assistants

Search Engines

Text to Speech, Speech to Text

Language detection and translation

Text summarization

Predicting next sequence of words: Text generation

Speech recognition

Python has become a leading programming language for NLP because of its simplicity and text processing ability. Python has an inbuilt text data structure known as the string. The string manipulations available in Python as well as its support for NLP libraries makes it a good start for processing text data.

Some important libraries when dealing with text data include:

Regular Expressions or RegEx

Pandas

NLTK

Numpy

SpaCy

TextBlob

CoreNLP

Wordcloud

Python Strings

The string data structure in python is a collection of characters enclosed in single or double quotes. A string is a body of text. Though it may contain numbers or other characters, once enclosed in quotes, those characters are classified as text data.

The python string has common operations that are handy when preprocessing text data. These operations include lower() which is used to standardized the format of all text by making all lowercase.

string.punctuation is an inbuilt string constant that contains all punctuations. It is useful in removal of punctuations from text which is an important text preprocessing step.

Common string operations for text preprocessing are shown below

Read the Python string methods documentation for more information.

Regular Expressions

Depending on the source of text data, our text may contain html tags, URLs, punctuations, emoticons, numbers or unwanted symbols.

Regular Expression is a sequence of characters used to matching or search for a pattern within a text. RegEx is used in text data preprocessing much like the python string. RegEx are special characters which can be used to match a pattern quickly, making it a useful tool for text preprocessing.

We will use some regular expressions later on. Read the Python regular expression guide to learn some of these expressions.

Steps in Text Preprocessing

Text preprocessing are the steps taken in order to transform our text data into a form understandable by algorithms.

These include:

Making all text lowercase

This is the most basic step in processing text data. Because Python like many programming languages is case sensitive, one word can be counted as different words if there is a difference in capitalization or casing. The word ‘Explore’ differs from the word ‘explore’ and ‘EXPLORE’.

During Exploratory Data Analysis of text data, one important feature of text is the number of unique words which form the vocabulary of the text. Therefore, it is important that all text are in the same format so a word does not appear more than once.

To be fair, capitalizing all text does the same but lowercasing is much more appealing to the eyes and has become the convention over time. This can be simply achieved by applying the Python string lower() method.

s = 'Hello, I am new here and i am happy to meet you'

In the above string, the words I and i will be counted as two separate words.

s.lower()

>> 'hello, i am new here and i am happy to meet you'

We can also apply the lower() method across data frames.

#This applies the lower method across a specified column

df['text'] = df['text'].apply(lambda x: "".join(x.lower() for x in x.split()))

Remove all punctuations

Text data processing is quite unique. We have to remove characters and symbols that do not add any value or information including punctuations, URLs, emoticons etc. Regex offers us a simple way to achieve this in python. As shown above, the built-in python string.punctuation can also be used to remove punctuations.

#Using Regex

import re

string.sub(r'[^\w\s]',' ',string)

Remove Stop Words

Stop words are common words that have little or no meaning in a text. By removing stop words, we are able to focus our attention on words that add meaning or keywords.

During human speech or writing, there are words that occur frequently but do not in fact add to the general meaning or understanding of a sentence.

The NLTK library has a stop words corpus which contains frequent stop words. We can use this to remove stop words in text. You need to have the NLTK library installed and download NLTK corpus.

#Let's take a look at nltk stop words import nltk

nltk.download()

from nltk.corpus import stopwords #remove stop words

stop = stopwords.words('english')

print(stop)

This will print out the NLTK English stop words

['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't"]

To remove stop words from text data:

import nltk

#nltk.download()

from nltk.corpus import stopwords

text = '''But soft what light through yonder window breaks

It is the east and Juliet is the sun

Arise fair sun and kill the envious moon

Who is already sick and pale with grief''' #remove stop words

stop = stopwords.words('english') new_text = " ".join(x for x in text.split() if x.lower() not in stop)

new_text

Output:

'soft light yonder window breaks east Juliet sun Arise fair sun kill envious moon already sick pale grief'

Tokenizing Text

Tokens are small units of text, they may be sentence tokens but are commonly word tokens. You can call a token the basic unit of a text.

Tokenization is the process of converting text into these tokens. It is a basic process in NLP, be it traditional NLP or Deep Learning methods.

The Python spilt method is a basic way to split text into tokens. Libraries such as NLTK, SpaCy and TextBlob can also be used to generate tokens

Stemming

In linguistics, Stemming is the process of reducing words to their stem or root form. This also holds true in NLP. In stemming, we aim to reduce different forms of a word into a basic form and therefore, have them counted as one word rather than multiple words. Stemming reduces our text vocabulary making search and retrieval easy to carry out.

Stemming works by cutting off unnecessary prefixes or suffixes leaving a root word.

Lemmatization

Similar to stemming, lemmatization reduces words into their root word or lemma based on vocabulary. In lemmatization, several forms of a word are mapped into one word.

One difference between stemming and lemmatization is that stemming looks crude. Stemming looks like all variations of a word were placed on top one another and then the excesses were hacked off with a saw leaving the root word behind.

Exploring Text Data

The Exploration of text data is similar to that of numeric data. Some common features to look out for in text data include:

The number of words in the text

Frequency of all words, words that occur very frequently without adding any meaning to the text can also be added to Stop words

Mean word length

Number of unique words

Plotting a word cloud — Word clouds are plots used to visualize text data

Word Cloud showing words associated with Disaster and Non-disaster tweets from Kaggle Dataset

Extracting Features from Text Data

As earlier discussed, most algorithms cannot process text data as they are. Just as we convert categorical data into numerical features before fitting to a model, we have to convert text data into numerical features. One Hot Encoding, can be applied to text data. However, if you have a large amount of text data, you run the risk of having numerous columns created causing high dimensionality.

For text data, we will discuss two common text engineering practices: Count Vectorizer and Tf-Idf.

Bag of Words

In simple terms, Bag of words is a vector of all unique words in a text. This collection of unique words form the vocabulary of the text. It is referred to as Bag of Words because it is a collection of text disregarding word order and grammar.

CountVectorizer

The Count Vectorizer from sklearn is an easy way to create a Bag of Words vectors. The count vectorizer returns an array that consisting of the number of times each word from the vocabulary appears in each text. It can also be made into a data frame with feature names as columns.

Imagine looking through a dataset containing various random tweets. Fitting the data to count vectorizer extracts all unique words to form a vocabulary. An array is created with count values corresponding to the number of times a particular word appears in a single tweet. It does not matter what position this word occurs, only that it occurs. All words have the same unit weight and therefore, the same importance. If a word appears three times in a text, its count value is 3, likewise if it appears a thousand times in a tweet, it has a count value of 1000.

Read the CountVectorizer documentation for more details.

Tf-idf

A good way to deal with frequently occurring words is using Term Frequency Inverse Document Frequency (Tf-idf).

The Tf-Idf vectorizer assigns weights to each of the words. Using our tweets example, words with higher frequency, appearing in all the tweets have lower weights and therefore, less importance. Words that appear less frequently are given high weight values.

Conclusion

We have looked at some of the most basic processes in Natural Language Processing with practical examples. To truly master NLP, make sure to practice some projects of your own. You can find datasets for NLP on Kaggle, Zindi or any public data source.

I have attached above a full repository on NLP tweets disaster classification so you can take a look. I also found Natural Language Processing Recipes: Unlocking Text Data with Machine Learning and Deep Learning using Python an interesting help while learning NLP.

Other helpful resources for NLP include videos on YouTube of which I found a particular playlist and video helpful.

Good luck in your NLP journey. May the force be with you.