4 Top Python Packages to learn NLP | by Cornellius Yudha Wijaya | Towards Data Science

Natural Language Processing or NLP is a field that concerns the interaction between computers and the human language. It is a hot field that is still in many demands from industry and academia, which I don't think would die down anytime soon.
Why is it in demand? NLP is in demand because the industry application would automate the thing we could not do in the past. For example, sentiment analysis, chatbot, text identification, and many more. The main point is that NLP applications are useful and important for data scientists to learn.
Because learning NLP would be important for the data scientist, I want to outline my top Python package to learn NLP. Let's get into it.
Natural Language Toolkit or NLTK is an open-source Python package developed specifically for working with the human language. It is arguably the most-used package for beginners and professionals in the NLP field as NLTK offers many useful APIs in NLP research. According to the homepage, NLTK is suitable for any profession — linguist, data scientist, researcher, student, and many more.
NLTK contains all the common API we use in our everyday NLP everyday activities work. If we explore the homepage, we will find that NLTK provides various tools for parsing, stemming, tokenization, and many more. They also include API to read data from sources such as Twitter. You can check the complete API list here.
To better understand the NLP analysis and how to do it with Python, the NLTK creator has written an introductory practical programming book for programmers called Natural Language Processing with Python. The material is free yet complete for the beginner in NLP. I suggest you take your time reading the book if you are never exploring the NLP field.
For the NLTK installation, you could use the pip distribution in your Python by executing the following code.
pip install --user -U nltk
NLTK offers a complete list of corpora for you to practice and explore that you could visit here. We could access the data using an in-build downloader from the NLTK package. Let's try to download one of the corpora.
#Download the Brown Corpusimport nltknltk.download('brown')#Preview the brown wordsfrom nltk.corpus import brownbrown.words()
We can see in the image above that we have a list of words from the brown corpus. Let's try using NLTK to calculate the word frequency.
#Import the classfrom nltk import FreqDist#Showing the top 10 wordsword_count = FreqDist(data)word_count.most_common(10)
The most common words from the Brown corpora are 'the' and followed by ',' and '.'. In a real NLP use case, we probably would remove all the words above because they didn't give much information to the model (These words were called Stopwords), and we would pre-process the text before calculating the frequency.
Pattern package is an open-source Python package developed for text processing and web-mining data. The API provides many functions, such as:
Data Mining API from various sources (Google, Twitter, and Wikipedia)
NLP Processing
Machine Learning Modelling
Network Analysis
If we compared the Pattern package to the NLTK package, the functions for the text processing within the Pattern are incomplete than NLTK. However, Pattern contains the web-mining data, which NLTK doesn't have. This is because Pattern packages were developed with a focus on data mining.
The Pattern is a great package for you who already understand the NLP subject and want to explore more into the real-life data. What I like from this package for the learner is the documentation which you could check out here. The documentation contains all the basic tutorials for data mining and network analysis, and case studies.
Let's try the Pattern package capability for data mining because this package is mainly used for web data mining. For this example, I would try to mining the recent data from Twitter.
For starters, if you do not have Pattern installed, you could do it by using the following code.
pip install pattern
Then we would try to run the following code to mine recent tweets that mention 'data science.'
from pattern.web import Twitter, plaintexttwitter = Twitter(language='en') for tweet in twitter.search('"data science"', cached=False):    print(plaintext(tweet.text))
As we can see from the above image, we successfully mined the tweet from Twitter, and we could analyze it completely. If we take a look at the single tweet typed, this is the data sample.
The tweet data is in the JSON format, with all the information you need to analyze present(text, profile, time, etc.). Pattern packages could mine data in other web sources, for example, in the search engine Google.
from pattern.web import Google, plaintextfor result in Google().search('"Data Science"', cached  = False):    print(plaintext(result.text))
There is much more data you could mine using the Pattern packages. If you are interested, you should refer to their documentation for more information.
TextBlob is a Python text processing package that provides many APIs to easing the NLP project tasks. TextBlob was built on top of the NLTK and Pattern packages, which means you would find many of the familiar APIs.
TextBlob stands out for how beginner-friendly this package is to do NLP activity with their simple API. TextBlob package was developed specifically for NLP tasks such as tagging, translation, sentiment analysis, and more in a simple way.
The documentation is a good start to learn the TextBlob package and the NLP field. They write the tutorial in a simple way where any beginner could follow and experiment with the code. The tutorial to create a text classifier is located here if you want to try it out. For example, let's try the TextBlob package capability on our own.
First, we need to install the TextBlob package by using the following code provided in the documentation.
pip install -U textblobpython -m textblob.download_corpora
Let's try to analyze the sentiment of the google search I have done previously. In this part, I would mine the data using Pattern, predict the google search sentiment, and store it into the Pandas DataFrame object.
from pattern.web import Google, plaintextfrom textblob import TextBlobimport pandas as pdres = []for result in Google().search('"Data Science"', cached  = False):    search = plaintext(result.text)    res.append([search, TextBlob(search).polarity, TextBlob(search).subjectivity])    data = pd.DataFrame(res, columns = ['Google Search', 'Polarity', 'Subjectivity'])data
The TextBlob sentiment analysis output has two kinds of measurements — polarity and Subjectivity. The polarity measures whether the sentiment is Neutral, Positive, or Negative. The closer the value to the zero (0) means the sentiment is close to neutral, and vice versa — if the values are closer to 1/-1, then it means the sentiment is close to Positive/Negative.
Subjectivity values are within the range of 0 to 1, where 0 means the text is not subjective at all (written objectively) and 1 is fully subjective (written like an opinion).
If you want to know more about the Sentiment Analysis using TextBlob, you can visit the documentation here.
SpaCy is an Open-source Python package with a tagline for Industrial-Strength Natural Language Processing. It means that SpaCy is developed for production environment and industrial activity than for academic purposes.
Although it is developed for Industrial purposes, the tutorial and documentation they have are quite complete. The pages offer you the guide, lesson, and online video to learn NLP from the beginning and use SpaCy. For example, the SpaCy 101 Guide would let you know many subjects such as Linguistic annotations, Tokenization, POS tags and dependencies, Vocab, hashes, and lexemes, and many more.
If you want to dive deep into the packages, you could check the following materials:
101 Guide
Interactive Online Courses
Video Courses
All these materials from SpaCy would help you understand the NLP concepts better as it is developed for a beginner. Let's try to analyze a text using SpaCy. We would use the previous data we have mined as a sample.
First, we need to install the SpaCy package. You could do that using the following code.
pip install -U pip setuptools wheelpip install -U spacypython -m spacy download en_core_web_sm
After installation, we are ready to process the text data using SpaCy. For this example, I would try to analyze the text part-of-speech and their characteristic using SpaCy.
import spacynlp = spacy.load("en_core_web_sm")#I would use the first text data from the Google Searchdoc = nlp(data['Google Search'].loc[0])pos = []for token in doc:        pos.append([token.text, token.lemma_, token.pos_, token.tag_, token.dep_, token.shape_, token.is_alpha, token.is_stop])pd.DataFrame(pos, columns = ['TEXT','LEMMA','POS','TAG','DEP','SHAPE','ALPHA','STOP'])
Using SpaCy, we are easily tokenized and identify the text data we pass on. With the SpaCy object, we could know the Lemma, Part-of-Speech, Tag, and many more. If you want to know more about what are these terms refer to, you could check out the SpaCy Linguistic Features guide.
Natural Language Processing or NLP is a field that is still growing in the industry, and many are trying to applied NLP capability in their business. However, learning NLP is not an easy process because so many concepts are different from the usual Machine Learning modeling.
In this article, there are 4 top Python packages I find good as an NLP learning material; they are:
NLTKPatternTextBlobSpaCy
NLTK
Pattern
TextBlob
SpaCy
I hope it helps!
If you enjoy my content and want to get more in-depth knowledge regarding data or just daily life as a Data Scientist, please consider subscribing to my newsletter here.
If you are not subscribed as a Medium Member, please consider subscribing through my referral.
Visit me on my LinkedIn or Twitter.