NLP Text Preprocessing and Cleaning Pipeline in Python
By: Emily Elia
Tags: Naturallanguageprocessing, Programming, Computer Science, Machine Learning, Data Science

Through language we express the human experience. Language is how we communicate, express sentiment, listen, think and converse. Over the past decade, tremendous progress has been made in natural language processing(NLP) where computers can classify, generate, and respond to language like a human can. These models and algorithms have started to give computers the tools to understand the human experience. The most important reason that these areas have grown is because of the exponential increase in data that can be feature engineered and then fed into models.

Feature Engineering:

The data scientists, PhDs, machine leaning engineers, and data engineers who have been on the front lines of the advancements in natural language processing spend most of their time cleaning and exploring datasets.

Garbage in Garbage out

Models will produce state of the art results if they are fed state of the art data; models will produce garbage if they are fed garbage, therefore data cleaning is one of the most important parts of the entire machine learning process. It does not matter if you are training a state of the art transformer like BERT or old-school word2vec. The quality of data that you feed the model will determine the quality of the results that you get.

garbage in garbage out; data on fire in data on fire out

Feature engineering and data cleaning are not tedious and pointless tasks — they are important filtering mechanisms that work the same way the human brain does when it processes a language. When humans are young they learn how to separate noises from a language and then find meaning in that language through the most important words and phrases in that language. This process is similar to text preprocessing. Text preprocessing breaks down a corpus into smaller parts then extracts the most important information from those parts which a model will then derive meaning from.

#tldr show me the code!

Steps in the Pipeline

📜 ,👀, 📊 (Load, Look at, and Analyze the Data):

The first step of any machine learning pipeline is to load the data! For good measure we will also take a peek at the data to make sure everything loaded correctly and collect a few basic stats like the number of words, the number of lines, and the number of characters. This will give us a good idea of what we are working with for the next steps in the pipeline. I am using the Reuters dataset that can be found here.

data_folder = Path("/Users/emilyelia/Downloads/reuters/reuters/reuters/training")

file_to_open = data_folder / "104"

f = open(file_to_open) print(f.read())

num_lines =0

num_words =0

num_chars =0

with open(file_to_open, 'r') as f:

for line in f:

words = line.split() num_lines += 1

num_words += len(words)

num_chars += len(line) print ("numbers of words", num_words)

print("number of lines", num_lines)

print("number of chars", num_chars)

Tokenization:

Tokenization is the next preprocessing step. It takes the text corpus and it splits it into “tokens” (words, sentences, etc.).

This process is not as simple as using some kind of separator. There are a lot of different situations where separators don’t work such as abbreviations with dots like “Dr.” or periods “.” at the end of sentences. A more complex model is needed to properly tokenize, but don’t worry because tokenization is a built-in feature in commonly used NLP libraries such as nltk.

from nltk.tokenize import sent_tokenize, word_tokenize

f = open(file_to_open)

#use built in tokenize to seperate each indiviual word

nltk_words = word_tokenize(f.read())

print(f"Tokenized words: {nltk_words}")

Cleaning:

Cleaning is the process of removing all unnecessary content from the corpus. The unnecessary content includes stop words and punctuation since they do not add any value or meaning to the overall corpus.

Punctuation

Punctuation removal is an important step since punctuation does not provide any additional value or insight into the overall corpus and the vectorization of the corpus. Removing punctuation is best done after the tokenization step because doing so before might cause some unforeseen results.

Stop Words

Stop words are the most common words in the language that you are using. There are between 50–100 stop words depending on the library that you are using and they are words that don’t add meaning like “the”, “an”, and “it”. Removing these words will not change the meaning of the corpus that you are working with and it will lead to better results because the remaining words will be the most important to determining the meaning of the corpus.

print(nltk_words)

punctuation = list(string.punctuation)

stopWords = set(stopwords.words('english'))

filter = []

for w in nltk_words:

if w.lower()not in stopWords and w not in punctuation:

filter.append(w)

Normalization

Normalization is the process of returning to a standard form or state. In terms of text preprocessing, it means taking numbers, abbreviations, and special characters, and converting them to text. This process uses the same associations that our brains do when processing special characters, misspellings, and abbreviations by taking them and assigning them to the words that we say, read, or think in the language of our choice. I am using the library normalise that can be found here.

The most common features to normalize are dates, numbers, abbreviations, currency, percents, and misspellings. In normalise you have to list the abbreviations that you want to spell out and then call the normalise function to perform the rest of the normalizations.

from normalise import normalise

abbr = {

"lb": "pounds",

"USDA": "United States Department of Agriculture",

"cts": "cents",

"U.S.": "United States"

}

normalise(text, user_abbrevs=abbr)

nt =normalise(filter, user_abbrevs=abbr)

display( ' '.join(nt))

These normalizations ensure that the model will be able to understand all of the text that is has to process because the numbers, special characters, misspellings, and abbreviations will have the same representation as everything else in the corpus.

Check out the entire project here!