Structured natural language processing with Pandas and spaCy | by Conor Mc. | Towards Data Science

Working with natural language data can often be challenging due to its lack of structure. Most data scientists, analysts and product managers are familiar with structured tables, consisting of rows and columns, but less familiar with unstructured documents, consisting of sentences and words. For this reason, knowing how to approach a natural language dataset can be quite challenging. In this post I want to demonstrate how you can use the awesome Python packages, spaCy and Pandas, to structure natural language and extract interesting insights quickly.
spaCy is a very popular Python package for advanced NLP — I have a beginner friendly introduction to NLP with SpaCy here. spaCy is the perfect toolkit for applied data scientists when working on NLP projects. The api is very intuitive, the package is blazing fast and it is very well documented. It’s probably fair to say that it is the best general purpose package for NLP available. Before diving into structuring NLP data, it is useful to get familiar with the basics of the spaCy library and api.
After installing the package, you can load a model (in this case I am loading the simple Engilsh model, which is optimized for efficiency rather than accuracy) — i.e. the underlying neural network has fewer parameters.
import spacynlp = spacy.load("en_core_web_sm")
We instantiate this model as nlp by convention. Throughout this post I’ll work with this dataset of famous motivational quotes. Let’s apply the nlp model to a single quote from the data and store it in a variable.
doc = nlp(df.quote[819]) print(doc)
If we print the document, you can see that it just returns the original quote, but under the hood a lot of NLP magic has happened — we have created a spaCy doc object. So what can we do with it?
Well first of all, there are two very cool and interesting visualizations that we can make. The visualization below shows the dependency structure of the document.
These dependencies describe the linguistic structure of the sentence by showing how each word relates to other words in the document. It’s worth stopping for a second to marvel at this. Under the hood spaCy has applied a large neural language model to the document, parsed the dependency structure and made it possible for us to easily view this — all with a single line of code. If you want to better understand how to interpret this dependency structure you can use the spacy.explain function to get a definition for each dependency type.
The next visualization shows the entities that have been recognized by the language model. In this cases the language model has recognised several words that are date types.
We can access these properties of the doc programmatically, as follows.
[(i, i.label_) for i in doc.ents]
The doc.ents method enables access to the entities that have been predicted by the spaCy language model. As with the dependency tags you can use spacy.explain to understand the meaning of each entity tag.
Other useful properties of the doc object are the the sents and noun_chunks methods, which enable you to split the doc into sentences or noun chunks (i.e. nouns plus their descriptors).
doc = nlp(df.quote[0])spacy.displacy.render(doc, style="ent")doc_nouns = list(doc.noun_chunks)print(doc_nouns)
If we access a single word from the doc we are actually accessing a spaCy token object. Like the docobject, the token object contains many useful properties. For example, we can iterate through the doc to extract each token and it’s various attributes.
[(i, i.ent_type_, i.is_stop) for i in doc]
Now that we have covered some of the basics of spaCy and the api, let’s take a more structured approach to the data.
The goal here is to transform the unstructured documents into a structured table of data. Essentially, we want to move from an unstructured corpus of documents, tokens and their metadata to a structured dataset of rows and columns.
Step 1: We first need to apply the spaCy language model to the entire collection of quotes. The easiest and most computationally efficient way to do this is to use the nlp.pipe function. This will iterate over each document and will apply the language model.
docs = list(nlp.pipe(df.quote))
Step 2: Define a function to extract all the properties of each word that you want to include in the table. This function will iterate over the tokens in a single doc and will extract various attributes such as the lemma, pos, entity and tag. Use this function to define all the attributes you want to pull out of the token objects. In this case I am extracting common attributes that I use frequently and that will help when analysing the data.
Step 3: Define a function to apply the function above to all documents and store the outputs in a Pandas dataframe.
If we run these steps we end up with a dataset that looks like this:
Each row represents a single token and the columns capture various metadata on that token. Importantly we can also link each individual word back to its doc using the doc_id column.
Now that we have the data in this format we can analyse it in the same way we would any other structured dataset. For example, we can count the number of words per doc and visualise this with a histogram.
tidy_docs.groupby("doc_id").size().hist(figsize=(14, 7), color="red", alpha=.4, bins=50);
Alternatively, we might be interested in understanding the most common entities that have been identified across the corpus:
tidy_docs.query("ent_type != ''").ent_type.value_counts()
We can do the same thing for words after filtering out stop words and puncutation:
tidy_docs.query("is_stop == False & is_punct == False").lemma.value_counts().head(10).plot(kind="barh", figsize=(24, 14), alpha=.7)plt.yticks(fontsize=20)plt.xticks(fontsize=20);
Another interesting analysis we can do with the structured data is to look at possessives to understand ownership within the corpus. The code below groups by each document then extracts the preceding and following token, then filters for tokens that are flagged with the POS tag, which indicates a possessive relationship i.e. the object of the previous token owns the subject of the following token.
This enables us to understand possessive relationships across the corpus. For example, you can see phrases such as, “One’s courage”, “someone’s prayers” and “today’s games”. Deep!
As you can see working with the data becomes quite trivial now that it is in a structured format.
This post shows how you can go from unstructured text data to a structured dataset that can be easily analysed. I often find this to be a useful first step when getting a feel for the data on NLP projects and can often reveal interesting and useful insights.
Thanks for reading!
P.S. All code for this post can be found here.