NLP: Building Text Cleanup and PreProcessing Pipeline | by Dinesh Yadav | Towards Data Science

Natural Language Processing, in short NLP, is subfield of Machine learning / AI which deals with linguistics and human languages. NLP deals with interactions between computers and human languages. In other words, it enables and programs computers to understand human languages and process & analyse large amount of natural language data.
But before a computer program can understand and interpret human language, a lot of pre-processing goes behind the scenes. Have you ever wondered what happens behind the scenes for sentiment analysis, text classification, tweet analysis etc.? Thanks to pre-processing, programs can easily transform language or text into a more assimilable form so that machine learning algorithms can perform better.
These are used most often in sentiment analysis, feedback classification, translation, summarization etc. In many cases, pre-processing steps improves the accuracy of algorithms. Usually, input data is presented in natural form, which is in the format of text, sentences, comments, paragraphs, tweets etc. Before such input can be passed to machine learning algorithms, it needs some clean-up or pre-processing so algorithms can focus on main/important words instead of words which adds minimal to no value.Enough said, lets deep dive into NLP pre-processing techniques. In this example, I am using python as the programming language.
Often, unstructured text contains a lot of noise, especially if you use techniques like web or screen scraping. HTML tags are typically one of these components which don’t add much value towards understanding and analysing text so they should be removed. We will use BeautifulSoup library for HTML tag clean-up.
# importsfrom bs4 import BeautifulSoup# function to remove HTML tagsdef remove_html_tags(text):    return BeautifulSoup(text, 'html.parser').get_text()# call functionremove_html_tags( ‘<html> \ <h1>Article Heading</h1> \ <p>First sentence of some important article. And another one. And then the last one</p></html>’)
Output:
' Article Heading First sentence of some important article. And another one. And then the last one'
Accented characters are important elements which are used to signify emphasis on a particular word during pronunciation or understanding. In some instances, the accent mark also clarifies the meaning of a word, which might be different without the accent. While their use in English is largely limited but there are very good chances that you will come across accented characters/letters in a free text corpus. Words such as résumé, café, prótest, divorcé, coördinate, exposé, latté etc.
Accents might get induced because of someone keyboard default setting or typing style. Handling accented characters gets more important, especially if you only want to analyse the English language. Hence, we need to make sure that these characters are converted and standardized into ASCII characters. A simple example — converting é to e.
# importsimport unicodedata# function to remove accented charactersdef remove_accented_chars(text):    new_text = unicodedata.normalize('NFKD', text).encode('ascii', 'ignore').decode('utf-8', 'ignore')    return new_text# call functionremove_accented_chars('Sómě Áccěntěd těxt. Some words such as résumé, café, prótest, divorcé, coördinate, exposé, latté.')
Output:
'Some Accented text. Some words such as resume, cafe, protest, divorce, coordinate, expose, latte.'
Contractions are shortened versions of words or syllables. They are created by removing specific, one or more letters from words. Often more than word are combined to create a contraction. In writing, an apostrophe is used to indicate the place of missing letters. In English language/text, contractions often exist in either written or spoken forms.
Nowadays, many editors will induce contractions by default. For examples do not to don’t, I would to I’d, you are to you’re. Converting each contraction to its expanded, original form helps with text standardization.
For removing contractions, I am leveraging a standard set of contractions available in the contractions library.
# importsfrom contractions import CONTRACTION_MAP # from contractions.pyimport re # function to expand contractionsdef expand_contractions(text, map=CONTRACTION_MAP):    pattern = re.compile('({})'.format('|'.join(map.keys())), flags=re.IGNORECASE|re.DOTALL)    def get_match(contraction):        match = contraction.group(0)        first_char = match[0]        expanded = map.get(match) if map.get(match) else map.get(match.lower())        expanded = first_char+expanded[1:]        return expanded     new_text = pattern.sub(get_match, text)    new_text = re.sub("'", "", new_text)    return new_text# call function expand_contractions(“Y’all i’d contractions you’re expanded don’t think.”)
Output:
'You all i would contractions you are expanded do not think.'
Another way can be:
# importsfrom pycontractions import Contractionscont = Contractions(kv_model=model)cont.load_models()# function to expand contractionsdef expand_contractions(text):    text = list(cont.expand_texts([text], precise=True))[0]    return text
Special characters, as you know, are non-alphanumeric characters. These characters are most often found in comments, references, currency numbers etc. These characters add no value to text-understanding and induce noise into algorithms. Thankfully, regular-expressions (regex) can be used to get rid of these characters and numbers.
# importsimport re# function to remove special charactersdef remove_special_characters(text):    # define the pattern to keep    pat = r'[^a-zA-z0-9.,!?/:;\"\'\s]'     return re.sub(pat, '', text) # call functionremove_special_characters(“007 Not sure@ if this % was #fun! 558923 What do# you think** of it.? $500USD!”)
Output:
'007 Not sure if this  was fun! 558923 What do you think of it.? 500USD!'
As you saw above, the text is retained. But sometimes these might not be required. Since we are dealing with text, so the number might not add much information to text processing. So, numbers can be removed from text. We can use regular-expressions (regex) to get rid of numbers. This step can be combined with above one to achieve in single step.
# importsimport re# function to remove numbersdef remove_numbers(text):    # define the pattern to keep    pattern = r'[^a-zA-z.,!?/:;\"\'\s]'     return re.sub(pattern, '', text) # call functionremove_numbers(“007 Not sure@ if this % was #fun! 558923 What do# you think** of it.? $500USD!”)
Output:
' Not sure if this  was fun!  What do you think of it.? USD!'
This can be clubbed with step of removing special characters. Removing punctuation is fairly easy. It can be achieved by using string.punctuation and keeping everything which is not in this list.
# importsimport string# function to remove punctuationdef remove_punctuation(text):    text = ''.join([c for c in text if c not in string.punctuation])    return text# call functionremove_punctuation('Article: @First sentence of some, {important} article having lot of ~ punctuations. And another one;!')
Output:
'Article First sentence of some important article having lot of  punctuations And another one'
Stemming is the process of reducing inflected/derived words to their word stem, base or root form. The stem need not be identical to original word. There are many ways to perform stemming such as lookup table, suffix-stripping algorithms etc. These mainly rely on chopping-off ‘s’, ‘es’, ‘ed’, ‘ing’, ‘ly’ etc from the end of the words and sometimes the conversion is not desirable. But nonetheless, stemming helps us in standardizing text.
# importsimport nltk# function for stemmingdef get_stem(text):    stemmer = nltk.porter.PorterStemmer()    text = ' '.join([stemmer.stem(word) for word in text.split()])    return text# call functionget_stem("we are eating and swimming ; we have been eating and swimming ; he eats and swims ; he ate and swam ")
Output:
'we are eat and swim ; we have been eat and swim ; he eat and swim ; he ate and swam'
Though stemming and lemmatization both generate the root form of inflected/desired words, but lemmatization is an advanced form of stemming. Stemming might not result in actual word, whereas lemmatization does conversion properly with the use of vocabulary, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma.
Before using lemmatization, we should be aware that it is considerably slower than stemming, so performance should be kept in mind before choosing stemming or lemmatization.
# importsimport spacynlp = spacy.load('en',parse=True,tag=True, entity=True)# function to remove special charactersdef get_lem(text):    text = nlp(text)    text = ' '.join([word.lemma_ if word.lemma_ != '-PRON-' else word.text for word in text])    return text# call functionget_lem("we are eating and swimming ; we have been eating and swimming ; he eats and swims ; he ate and swam ")
Output:
'we be eat and swim ; we have be eat and swim ; he eat and swim ; he eat and swam'
Stopwords are often added to sentences to make them grammatically correct, for example, words such as a, is, an, the, and etc. These stopwords carry minimal to no importance and are available plenty on open texts, articles, comments etc. These should be removed so machine learning algorithms can better focus on words which define the meaning/idea of the text. We are using list from nltk.corpus and this list can further be enhanced by adding or removing custom words based on the situation at hand.
# importsimport nltkfrom nltk.tokenize import ToktokTokenizertokenizer = ToktokTokenizer()stopword_list = nltk.corpus.stopwords.words('english')# custom: removing words from liststopword_list.remove('not')# function to remove stopwordsdef remove_stopwords(text):    # convert sentence into token of words    tokens = tokenizer.tokenize(text)    tokens = [token.strip() for token in tokens]    # check in lowercase     t = [token for token in tokens if token.lower() not in stopword_list]    text = ' '.join(t)        return text# call functionremove_stopwords("i am myself you the stopwords list and this article is not should removed")
Output:
'stopwords list article not removed'
Extra whitespaces and tabs do not add any information to text processing. Handling these should be fairly easy.
# importsimport re# function to remove special charactersdef remove_extra_whitespace_tabs(text):    #pattern = r'^\s+$|\s+$'    pattern = r'^\s*|\s\s*'    return re.sub(pattern, ' ', text).strip()# call functionremove_extra_whitespace_tabs('  This web line  has \t some extra  \t   tabs and whitespaces  ')
Output:
'This web line has some extra tabs and whitespaces'
Changing case to lower can be achieved by using lower function.
# function to remove special charactersdef to_lowercase(text):    return text.lower()# call functionto_lowercase('ConVert THIS string to LOWER cASe.')
Output:
'convert this string to lower case.'
Some of these steps can be combined in a single step. Also, note that few of these might not be required depending upon the context and requirement. Furthermore, we can define a new function encapsulating all of the above pre-processing functions to form a text-normalizer pipeline. These steps should give you a good idea about how to build your clean-up and pre-processing strategies. After these pre-processing steps, the corpus of text is ready for NLP algorithms such as Word2Vec, GloVe etc. These pre-processing steps will definitely improve the model’s accuracy.
Hope you liked this article, and if so, then please say so in comments. If you would like to share any suggestion on any approach, then feel free to say so in comments and I will revert right away.