Preprocessing: Customisation over the usual drill
By: Eram Munawwar
Tags: Getting Started, Editors Pick, Machine Learning, NLP, Text Preprocessing

Preprocessing: Customisation over the usual drill

Photo by Dollar Gill on Unsplash

Introduction

It is often said that data scientists spend almost 70% of their time in feature engineering. With NLP, the same is true for cleaning the text. Almost all real life NLP datasets are extremely unstructured. Sources like social media, customer reviews, even news articles or wikipedia data are exploding with information, all of which might not be needed for our use case. They all need to be processed for feature extraction.

When it comes to text preprocessing, there is a tendency to follow a usual drill. I will be talking more about the drill in the coming section. On a high level, it is a 5–6 step standard preprocessing which includes lowercasing, removing stop words, numbers, punctuations and stemming or lemmatising the data. And then finally tokenise the text to words. You’ll see a ton of blogs and articles revolving around this drill.

However, working with specific NLP problems like Question Answering, Named Entity extraction, topic modelling etc., I have learnt that giving more time to customised preprocessing can give a pretty good boost to the model’s overall performance. Moreover, the drill usually fails when working with any language other than english. Working with Bahasa Indonesia on a daily basis, I realised that customisation is key to preprocessing.

The Preprocessing Drill

Lowercasing : Convert all the text to lowercase. You wouldn’t want a separate vector for `hello` and `Hello`, right? Lowercasing is usually a default for most use cases, however, it might not be useful, or can have a negative impact for cases like abstractive summarisation, machine translation etc.

: Convert all the text to lowercase. You wouldn’t want a separate vector for `hello` and `Hello`, right? Lowercasing is usually a default for most use cases, however, it might not be useful, or can have a negative impact for cases like abstractive summarisation, machine translation etc. Removing punctuations : We generally overlook these nosy characters and use one regex to swipe them all out. Punctuation symbols might be extremely important, eg. a question mark is essential in detecting a question sentence. Emoticons are key to showcasing a user’s sentiment. If we remove “+” from Galaxy S20+ , it would change to another series altogether.

: We generally overlook these nosy characters and use one regex to swipe them all out. Punctuation symbols might be extremely important, eg. a question mark is essential in detecting a question sentence. Emoticons are key to showcasing a user’s sentiment. If we remove “+” from Galaxy S20+ , it would change to another series altogether. Removing numbers : If the final vocabulary does not contain numbers, it becomes an out of vocabulary token (OOV). With an OOV token, the numbers 200 and 300 have the same weightage which might not be ideal if these numbers correspond to prices of two products run through a siamese network.

: If the final vocabulary does not contain numbers, it becomes an out of vocabulary token (OOV). With an OOV token, the numbers 200 and 300 have the same weightage which might not be ideal if these numbers correspond to prices of two products run through a siamese network. Removing stop words : Stop words are a list of words containing pronouns (she, he, her, him etc.), question words (what, where, who etc.), conjunctions (and, but, for etc.), articles (a, an, the), and other words like ok, that, there etc. Stop words can generally be removed, however, depending on the use case, we might choose to customise the list. For example, the nltk stop word list for Bahasa Indonesia contains more than 700 words. Many of these words are meaningful to a variety of use cases. E.g. words like send, deliver etc. (translated to english) are present in the stop word list, but they might be important words if we want to know how good a particular delivery service is.

: Stop words are a list of words containing pronouns (she, he, her, him etc.), question words (what, where, who etc.), conjunctions (and, but, for etc.), articles (a, an, the), and other words like ok, that, there etc. Stop words can generally be removed, however, depending on the use case, we might choose to customise the list. For example, the nltk stop word list for Bahasa Indonesia contains more than 700 words. Many of these words are meaningful to a variety of use cases. E.g. words like send, deliver etc. (translated to english) are present in the stop word list, but they might be important words if we want to know how good a particular delivery service is. Stemming : Stemming converts the words to their “root” form. Although this root word might not be an actual word. Eg. words like “frustrating” and “frustrated” will change to “frustrat”. The most popular stemmer for english language is the Porter Stemmer, also available in nltk.

: Stemming converts the words to their “root” form. Although this root word might not be an actual word. Eg. words like “frustrating” and “frustrated” will change to “frustrat”. The most popular stemmer for english language is the Porter Stemmer, also available in nltk. Lemmatisation: It is similar to stemming with the difference that the “root” word is an actual word from a lexicon. Taking the previous example, the root word would be “frustrate”.

Stemming and lemmatisation might not be very beneficial when working with the english language. However, if we’re working with another language, e.g. Bahasa Indonesia, each word comes with almost 5–6 variants by adding prefixes and suffixes to it. Therefore stemming becomes very essential. Or else the vocabulary size would be too high! For the Indonesian language, Sastrawi Stemmer is the best one I have encountered so far. Do check it out if you’re also working with Bahasa!

Now that we understand what the preprocessing drill looks like, along with the dangers of following it blindly, let us see a rule of thumb for custom preprocessing.

Rule of Thumb

Below are tricks that I use for picking up the best preprocessing for my use case.

Lowercasing:

Image by author

Punctuations:

Image by author

Numbers:

Image by author

Stopwords:

Image by author

Stemming/Lemmatise:Lemmatisation is generally slower than stemming, therefore, I personally prefer stemming over it.

Image by author

Extras

Some more steps in preprocessing can be:

Removing non-ASCII characters: Sometimes, text can contain non ASCII unicode characters like \xe5 \u177 etc. These should be removed before starting any preprocessing.

Removing extra lines and spaces: We can use regular expressions to remove any extra spaces, tabs, and newlines.

Removing URLs and links

Spell correction: Use pyspellchecker, or a defined set of spelling errors with the corrections to perform this step.

Conclusion

Text preprocessing can be tedious but its advantages cannot be denied. Instead of doing it all, we must try to follow the rule of thumb to get a pretty clean and usable text, ready for feature extraction.

Many times, I have used pre-trained models on a custom dataset for transfer learning and seen that a few additional customisations in preprocessing has yielded a higher score than without it.

I hope the general knowledge about text cleaning and my tricks of customisation would be helpful for your NLP tasks. Let’s keep in mind that sometimes less can be enough! It is not required to do all the steps, instead, a few might be better.

Cheers!

Eram