spaCy Basics: NLP in Python | Towards Data Science

A major challenge of text data is extracting meaningful patterns and using those patterns to find actionable insights.
NLP can be thought of as a two part problem:
Processing. Converting the text data from its original form into a form the computer can understand. This includes data cleaning and feature extraction.Analysis. Using the processed data to extract insights and make predictions.
Processing. Converting the text data from its original form into a form the computer can understand. This includes data cleaning and feature extraction.
Analysis. Using the processed data to extract insights and make predictions.
Here we will focus on the processing step.
We’ll use a dataset created by scraping Twitter for Donald Trump tweets. A tutorial for creating your own Twitter dataset can be found here:
towardsdatascience.com
And we will focus exclusively on spaCy “a free, open-source library for advanced Natural Language Processing (NLP) in Python.”
spaCy is a great choice for NLP tasks, especially for the processing text and has a ton of features and capabilities, many of which we’ll discuss below.
The full notebook can be found here.
A good definition of a token, according to researchgate.net, is “a string of contiguous characters between two spaces, or between a space and punctuation marks.” In English, tokens are usually individual words, but can also be punctuation marks and numbers.
First we use the spacy.load() method to load a model package by and return the nlp object. Next we call nlp() on a string and spaCy tokenizes the text and creates a document object:
# Load model to return language objectnlp = spacy.load('en')# Calling nlp on our tweet texts to return a processed Doc for eachdf['doc'] = [nlp(text) for text in df.text]df.sample(3)
After tokenizing, we can do all sorts of things.
We could do something basic like count the number of tokens per document and visualize it with a histogram:
# Sum the number of tokens in each Docdf['num_tokens'] = [len(token) for token in df.doc]# Visualize histogram of tokens per tweetg = sns.distplot(df.num_tokens)
Or we can utilize some of the many available token attributes spaCy has to offer.
Spacy makes it easy to get part-of-speech tags using token attributes:
# Print sample of part-of-speech tagsfor token in sample_doc[0:10]:    print (token.text, token.pos_)
In addition to the part-of-speech tags, we can also predict named entities that appear in our documents.
According to Wikipedia, a named entity “is a real-world object, such as persons, locations, organizations, products, etc., that can be denoted with a proper name.”
for ent in sample_doc.ents:    print(ent.text, ent.label_)
And, if we don’t know what the output means, spaCy has print(spacy.explain('NORP')) to learn that NORP stands for Nationalities or religious or political groups.
In this case, the model is correctly predicts that “Democrats” is a NORP, but is incorrect in classifying “The Oval Office” as an organization.
We can visualize named entities using spaCy’s visualizer displaCy:
# Visualize named entities with displacyspacy.displacy.render(sample_doc, style='ent', jupyter=True)
Lemmatization is the reduction of each word to its root, or lemma.
For example, talk, talks, and talking are forms of the same word. They all share the same root word: talk.
In spaCy we call token.lemma_ to get the lemmas for each word.
Here we explored some of spaCy’s basic functionality, including tokenization, part-of-speech tagging, and lemmatization. These are just a few of the many tools the library has to offer.
In future articles we will explore using some of the above methods to gain insights from text data.