How to Start An NLP Project
By: Lily Hughes-Robinson
Tags: Programming, AI, NLP, Business, Data Science

First, one needs to decide what text they want to analyze. This may seem obvious at first, but there are a number of finer technical considerations that need to be made. For some, the answers to these questions will be necessitated by their desired outcome. For example, when examining free-text survey input, it is obvious that the survey responses are going to make up your corpus. In other cases, this may be less well defined.

There are a few useful terms to explain here. Corpus refers to a collection of text examined in an analysis. This term can scale from a single sentence to every word ever know in every language ever spoken. Sometimes a corpus can be subdivided into documents, which are smaller corpora that usually encapsulate a single idea or topic. A document can be as small as a tweet or as large as a book. Dividing a corpus into documents isn’t strictly necessary for every NLP task, but it is useful for comparing different documents against each other. The smallest unit of measure in NLP is a token. A token is an individual word within a document. A vocabulary is the number of distinct tokens present within a corpus. Some techniques, such as GloVe word embeddings, grow in time complexity as the vocabulary grows while others increase in complexity with a larger corpus. One’s use case will drive which analysis and corpus size make the most sense.

Unless you have a large compute cluster and infinite amounts of time, you cannot look at every tweet ever made. What makes sense is to choose a time period to base your analysis on. One’s choice in time period is going to be based around a few factors: the size of your time window and when it starts. Both of these parameters affect the number of words/documents included in your analysis. Most likely you won’t know the distribution of word sizes over time without doing some exploratory data analysis that can reveal how the corpus size grows as you add more time to the analysis. Once you know how the corpus size grows you will be able to make an educated decision based on your available compute resources.

I performed a sample analysis to measure corpus and vocabulary growth. I dug into my favorite data set for pet projects: Reddit. I used library called praw to grab comments from the Reddit API. I then plotted the growth as a series. Below is a gist I wrote to track the number of words as explore more posts. I also include the outputted graph when I ran the code around the time of publishing.

Example analysis measuring corpus and vocabulary growth as more posts are added

We see linear growth of the corpus while the vocab grows logarithmically

Looking at the graph above, we can see that the corpus grows in a roughly linear manner while the vocabulary is much more logarithmic. Using this knowledge, one can now make an educated decision about how many Reddit posts to include in one’s analysis. One way this analysis could be taken further is to calculate these values across several different subreddits, to see which community has the richest vocabulary and which is the most verbose.

Once one has solidified the parameters to create their corpus, there are preprocessing steps that need to be taken before analysis can begin. Raw natural language has a number of properties that can muddy analysis down the line. A common first step is to remove stop words; common words that do not offer rich semantic meaning and instead add more noise than signal. Most NLP libraries come with a list of stop words you can utilize to clean your corpus. Below is an example of removing stop words using the spaCy library.

Remove stop words with spaCy

This example has spaCy ingest each comment, which then get turned into a pared document containing tokens. Iterating over the document yields individual tokens, which we can examine to see if they should be considered in the analysis. The gist above removes stop words, but one could just as easily remove numbers or do basic text transformations such as lemmatisation or stemming. SpaCy performs much of this analysis when it first parses each document. Here is a list of token attributes that spaCy provides.

Another common step involves dealing with numerical data within your text. In general, it is common to remove numbers as they usually do not provide any semantic meaning on their own. To remove numbers as well as stop words, you can use regular expressions (regexes) or built in functions most NLP libraries. For example, in spaCy, tokens have a property “is_alpha” one can filter on, similarly to stop words in the above gist.

However, it is possible that numbers have a specific context you want to preserve. For example, some numerical sequences have a specific meaning, such as IP addresses or MAC addresses. If the data set contains such information, one can use masking to replace all instances of numerical entities with a predetermined token, such as “IP_ADDRESS.” This will preserve the semantic meaning of the numerical entity while removing the noise numbers. Often a simple regex replace is the best tool of the job here.

Masking is useful for more than just numerical entities. It can be used for any entity you want to preserve syntactic meaning while removing specific detail. This can be done to remove personally identifying information (PII) from your data, which may be a legal or regulatory requirement depending on your data set. Looking back at Reddit, I am going to demonstrate masking usernames with the token “USERNAME”. Luckily, a post on r/help explains that usernames can consist of any alphanumeric (plus dash and underscore) 3 to 20 characters long and are usually prefixed with “u/.” In the gist below, I use python’s built in regular expression library to find and replace all Reddit usernames with our token.

Example of masking using regular expressions

These steps will set the stage for a successful NLP project. There are a myriad of tools and techniques one could use to summarize, extract meaning, or any number of other NLP tasks. I have provided a notebook with all the code listed in the gists in a jupyter notebook on azure notebooks.