7 NLP Techniques You Can Easily Implement with Python | by Frank Andrade | Towards Data Science

Natural Language Processing (NLP) is focused on enabling computers to understand and process human languages. Computers are great at working with structured data like spreadsheets; however, much information we write or speak is unstructured.
The goal of NLP is to make computers understand unstructured texts and retrieve meaningful pieces of information from it. We can implement many NLP techniques with just a few lines of code of Python thanks to open-source libraries such as spaCy and NLTK.
In this article, we’ll learn the core concepts of 7 NLP techniques and how to easily implement them in Python.
Table of Contents1. Sentiment Analysis2. Named Entity Recognition (NER)3. Stemming4. Lemmatization5. Bag of Words (BoW)6. Term Frequency–Inverse Document Frequency (TF-IDF)7. Wordcloud
Sentiment Analysis is one of the most popular NLP techniques that involves taking a piece of text (e.g., a comment, review, or a document) and determines whether data is positive, negative, or neutral. It has many applications in healthcare, customer service, banking, etc.
For simple cases, in Python, we can use VADER (Valence Aware Dictionary for Sentiment Reasoning) that is available in the NLTK package and can be applied directly to unlabeled text data. As an example, let’s get all sentiment scores of the lines spoken by characters in a TV show.
First, we wrangle a dataset available on Kaggle or my Github named ‘avatar.csv’, then with VADER we calculate the score of each line spoken. All of this is stored in the df_character_sentiment dataframe.
In the df_character_sentiment below, we can see that every sentence receives a negative, neutral and positive score.
We could even group the scores by character and calculate the mean to obtain the sentiment score for a character and then represent it with horizontal bar plots by using the matplotlib library (the result of this is displayed in this article)
Note: VADER is optimized for social media text, so we should take the results with a grain of salt. You can use a more complete algorithm or develop your own with machine learning libraries. In the link below, there’s a complete guide on how to create one from scratch with Python using the sklearn library.
towardsdatascience.com
Named Entity Recognition is a technique used to locate and classify named entities in text into categories such as persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. It‘s used for optimizing search engine algorithms, recommendation systems, customer support, content classification, etc.
In Python, we can use SpaCy’s named entity recognition that supports the following entity types.
To see it in action, we first import spacy, and then create a nlp variable that will store theen_core_web_sm pipeline. This is a small English pipeline trained on written web text (blogs, news, comments), that includes vocabulary, vectors, syntax, and entities. To find the entities, we apply nlp to a sentence.
import spacynlp = spacy.load("en_core_web_sm")doc = nlp("Biden invites Ukrainian president to White House this summer")print([(X.text, X.label_) for X in doc.ents])
This will print the following values
[('Biden', 'PERSON'), ('Ukrainian', 'GPE'), ('White House', 'ORG'), ('this summer', 'DATE')]
Spacy found that “Biden” is a person, “Ukranian” is GPE (countries, cities, states, “White House” is an organization, and “this summer” is a date.
Stemming and lemmatization are 2 popular techniques in NLP. Both normalizes a word but in different ways.
Stemming: It truncates a word to its stem word. For example, the words “friends,” “friendship,” “friendships” will be reduced to “friend.” Stemming may not give us a dictionary, grammatical word for a particular set of words.
Lemmatization: Unlike the stemming technique, lemmatization finds the dictionary word instead of truncating the original word. Lemmatization algorithms extract the correct lemma of each word, so they often require a dictionary of the language to be able to categorize each word correctly.
Both techniques are widely used and you should choose them wisely based on your project’s goals. Lemmatization has a lower processing speed, compared to stemming so if accuracy is not the project’s goal but speed, then stemming is an appropriate approach; however. if accuracy is crucial, then consider using lemmatization.
Python’s library NLTK makes it easy to work with both techniques. Let’s see it in action.
For the English language, there are two popular libraries available in nltk — Porter Stemmer and LancasterStemmer.
from nltk.stem import PorterStemmerfrom nltk.stem import LancasterStemmer# PorterStemmerporter = PorterStemmer()# LancasterStemmerlancaster = LancasterStemmer()print(porter.stem("friendship"))print(lancaster.stem("friendship"))
PorterStemmer algorithm doesn’t follow linguistics, but a set of 5 rules for different cases that are applied in phases to generate stems. The print(porter.stem(“friendship”)) code will print the word friendship
LancasterStemmer is simple, but heavy stemming due to iterations and over-stemming may occur. This causes the stems to be not linguistic, or they may have no meaning. The print(lancaster.stem(“friendship”)) code will print the word friend.
You can try any other word to see how both algorithms differ. In the case of other languages, you can import SnowballStemme from nltk.stem
We’ll use NLTK again, but this time we import WordNetLemmatizer as shown in the code below.
from nltk import WordNetLemmatizerlemmatizer = WordNetLemmatizer()words = ['articles', 'friendship', 'studies', 'phones']for word in words:    print(lemmatizer.lemmatize(word))
Lemmatization generates different outputs for different Part Of Speech (POS) values. Some of the most common POS values are verb (v), noun (n), adjective (a), and adverb (r). The default POS value in lemmatization is a noun, so the printed values for the previous example will be article, friendship, study and phone.
Let’s change the POS value to verb (v).
from nltk import WordNetLemmatizerlemmatizer = WordNetLemmatizer()words = ['be', 'is', 'are', 'were', 'was']for word in words:    print(lemmatizer.lemmatize(word, pos='v'))
In this case, Python will print the word be for all the values in the list.
The Bag of Words (BoW) model is a representation that turns text into fixed-length vectors. This helps us represent text into numbers so we can use it for machine learning models. The model doesn’t care about the word order, but it’s only concerned with the frequency of words in the text. It has application in NLP, information retrieval from documents, and classifications of documents.
The typical BoW workflow involves cleaning raw text, tokenization, building a vocabulary, and generating vectors.
Python’s library sklearn contains a tool called CountVectorizer that takes care of most of the BoW workflow.
Let’s use the following 2 sentences as examples.
Sentence 1: “I love writing code in Python. I love Python code”
Sentence 2: “I hate writing code in Java. I hate Java code”
Both sentences will be stored in a list named text. Then we’re going to create a dataframe df to store this text list. After this, we‘ll initiate an instance of CountVectorizer(cv), and then we’ll fit and transform the text data to obtain the numeric representation. This will be stored in a document-term matrix df_dtm.
import pandas as pdfrom sklearn.feature_extraction.text import CountVectorizertext = ["I love writing code in Python. I love Python code",        "I hate writing code in Java. I hate Java code"]df = pd.DataFrame({'review': ['review1', 'review2'], 'text':text})cv = CountVectorizer(stop_words='english')cv_matrix = cv.fit_transform(df['text'])df_dtm = pd.DataFrame(cv_matrix.toarray(),                      index=df['review'].values,                      columns=cv.get_feature_names())df_dtm
The BoW representation made with CountVectorizer stored in df_dtm looks like the picture below. Keep in mind that words with 2 letters or fewer are not taking into account by the CountVectorizer.
As you can see the numbers inside the matrix represent the number of times each word was mentioned in each review. Words like “love,” “hate,” and “code” have the same frequency (2) in this example.
Overall, we can say that CountVectorizer does a good job tokenizing text, building a vocabulary, and generating vectors; however, it won’t clean raw data for you. I made a guide on how to clean and prepare data in Python, check it out in case you want to learn the best practices.
towardsdatascience.com
Unlike the CountVectorizer, the TF-IDF computes “weights” that represent how relevant a word is to a document in a collection of documents (aka corpus). The TF-IDF value increases proportionally to the number of times a word appears in the document and is offset by the number of documents in the corpus that contain the word. Simply put, the higher the TF-IDF score, the rarer or unique or valuable the term and vice versa. It has applications in information retrieval like search engines that aim to deliver results that are most relevant to what you’re searching for.
Before we see the Python implementation, let’s see an example so you have an idea of how the TF and IDF are calculated. For the following example, we’ll use the same sentences used for the CountVectorizer example.
Sentence 1: “I love writing code in Python. I love Python code”
Sentence 2: “I hate writing code in Java. I hate Java code”
There are different ways to define term frequency. One suggests the raw count itself (i.e., what the Count Vectorizer does), but others suggest it’s the frequency of the word in the sentence divided by the total number of words in the sentence. For this simple example, we’ll use the first criteria so the term frequency is shown in the following table.
As you can see, the values are the same as the calculated for the CountVectorizer before. Also, words with 2 letters or fewer are not taking into account.
The IDF is also calculated in different ways. Although standard textbook notation defines the IDF as idf(t) = log [ n / (df(t) + 1), the sklearn library we’ll use later in Python calculates the formula by default as follows.
Also, sklearn assumes natural logarithm ln instead of log and smoothing (smooth_idf=True). Let’s calculate the IDF values for each word as sklearn will do it.
Once we have the TF and IDF values, we can obtain the TF-IDF by multiplying both values (TF-IDF = TF * IDF). The values are shown in the table below.
Calculating the TF-IDF shown in the table above in Python requires a few lines of code thanks to the sklearn library.
import pandas as pdfrom sklearn.feature_extraction.text import TfidfVectorizertext = ["I love writing code in Python. I love Python code",        "I hate writing code in Java. I hate Java code"]df = pd.DataFrame({'review': ['review1', 'review2'], 'text':text})tfidf = TfidfVectorizer(stop_words='english', norm=None)tfidf_matrix = tfidf.fit_transform(df['text'])df_dtm = pd.DataFrame(tfidf_matrix.toarray(),                      index=df['review'].values,                      columns=tfidf.get_feature_names())df_dtm
The TF-IDF representation made stored in df_dtm looks like the picture below. Again, words with 2 letters or fewer are not taking into account by the TF-IDF.
Note: By default TfidfVectorizer() uses l2 normalization, but to use the same formulas shown above we set norm=None as a parameter. For more details of the formulas used by default in sklearn and how you can customize it check its documentation.
Wordcloud is a popular technique that helps us identify the keywords in a text. In a wordcloud, more frequent words have a larger and bolder font, while less frequent words have smaller or thinner fonts. In Python, you can make simple wordclouds with the wordcloud library and nice-looking wordclouds with the stylecloudlibrary.
Below you can find the code to make a wordcloud in Python. I’m using a text file of a Steve Jobs speech that you can find on my Github.
import stylecloudstylecloud.gen_stylecloud(file_path='SJ-Speech.txt',                          icon_name= "fas fa-apple-alt")
This is the result of the code above.
Wordclouds are so popular because they’re engaging, easy to understand, and easy to create.
You can take customization even further by changing the colors, removing stopwords, choosing your image, or even adding your own image to use it as a mask of the wordcloud. For more details, check the guide below.
towardsdatascience.com
That’s it! You’ve just learned the core concept of 7 NLP techniques and how to implement them in Python. All the code written in this article can be found on my Github.
Join my email list with 3k+ people to get my Python for Data Science Cheat Sheet I use in all my tutorials (Free PDF)