NLP Text Preprocessing and Cleaning Pipeline in Python | by Emily Elia | Towards Data Science

Through language we express the human experience. Language is how we communicate, express sentiment, listen, think and converse. Over the past decade, tremendous progress has been made in natural language processing(NLP) where computers can classify, generate, and respond to language like a human can. These models and algorithms have started to give computers the tools to understand the human experience. The most important reason that these areas have grown is because of the exponential increase in data that can be feature engineered and then fed into models.
The data scientists, PhDs, machine leaning engineers, and data engineers who have been on the front lines of the advancements in natural language processing spend most of their time cleaning and exploring datasets.
Models will produce state of the art results if they are fed state of the art data; models will produce garbage if they are fed garbage, therefore data cleaning is one of the most important parts of the entire machine learning process. It does not matter if you are training a state of the art transformer like BERT or old-school word2vec. The quality of data that you feed the model will determine the quality of the results that you get.
Feature engineering and data cleaning are not tedious and pointless tasks — they are important filtering mechanisms that work the same way the human brain does when it processes a language. When humans are young they learn how to separate noises from a language and then find meaning in that language through the most important words and phrases in that language. This process is similar to text preprocessing. Text preprocessing breaks down a corpus into smaller parts then extracts the most important information from those parts which a model will then derive meaning from.
#tldr show me the code!
The first step of any machine learning pipeline is to load the data! For good measure we will also take a peek at the data to make sure everything loaded correctly and collect a few basic stats like the number of words, the number of lines, and the number of characters. This will give us a good idea of what we are working with for the next steps in the pipeline. I am using the Reuters dataset that can be found here.
data_folder = Path("/Users/emilyelia/Downloads/reuters/reuters/reuters/training")file_to_open = data_folder / "104"f = open(file_to_open)print(f.read())num_lines =0num_words =0num_chars =0with open(file_to_open, 'r') as f:    for line in f:        words = line.split()num_lines += 1        num_words += len(words)        num_chars += len(line)print ("numbers of words", num_words)print("number of lines", num_lines)print("number of chars", num_chars)
Tokenization is the next preprocessing step. It takes the text corpus and it splits it into “tokens” (words, sentences, etc.).
This process is not as simple as using some kind of separator. There are a lot of different situations where separators don’t work such as abbreviations with dots like “Dr.” or periods “.” at the end of sentences. A more complex model is needed to properly tokenize, but don’t worry because tokenization is a built-in feature in commonly used NLP libraries such as nltk.
from nltk.tokenize import sent_tokenize, word_tokenizef = open(file_to_open)#use built in tokenize to seperate each indiviual wordnltk_words = word_tokenize(f.read())print(f"Tokenized words: {nltk_words}")
Cleaning is the process of removing all unnecessary content from the corpus. The unnecessary content includes stop words and punctuation since they do not add any value or meaning to the overall corpus.
Punctuation removal is an important step since punctuation does not provide any additional value or insight into the overall corpus and the vectorization of the corpus. Removing punctuation is best done after the tokenization step because doing so before might cause some unforeseen results.
Stop words are the most common words in the language that you are using. There are between 50–100 stop words depending on the library that you are using and they are words that don’t add meaning like “the”, “an”, and “it”. Removing these words will not change the meaning of the corpus that you are working with and it will lead to better results because the remaining words will be the most important to determining the meaning of the corpus.
print(nltk_words)punctuation = list(string.punctuation)stopWords = set(stopwords.words('english'))filter = []for w in nltk_words:    if w.lower()not in stopWords and w not in punctuation:        filter.append(w)
Normalization is the process of returning to a standard form or state. In terms of text preprocessing, it means taking numbers, abbreviations, and special characters, and converting them to text. This process uses the same associations that our brains do when processing special characters, misspellings, and abbreviations by taking them and assigning them to the words that we say, read, or think in the language of our choice. I am using the library normalise that can be found here.
The most common features to normalize are dates, numbers, abbreviations, currency, percents, and misspellings. In normalise you have to list the abbreviations that you want to spell out and then call the normalise function to perform the rest of the normalizations.
from normalise import normaliseabbr = {    "lb": "pounds",    "USDA": "United States Department of Agriculture",    "cts": "cents",    "U.S.": "United States"}normalise(text, user_abbrevs=abbr)nt =normalise(filter, user_abbrevs=abbr)display( ' '.join(nt))
These normalizations ensure that the model will be able to understand all of the text that is has to process because the numbers, special characters, misspellings, and abbreviations will have the same representation as everything else in the corpus.
Check out the entire project here!