Basic Steps In Natural Language Processing Pipeline
By: Dr. Monica
Tags: Machine Learning, Data Science, NLP, Artificial Intelligence, Nltk

Source: Unsplash

Natural Language Processing (NLP) deals with text data. The applied research in NLP is motivated to design the technology that understands the human language more effectively. The research in NLP is more demanding and challenging as it is difficult to understand how the human brain understands the secrets of language and its communication methods.

Many laboratories, researchers from all around the world are working their best to synchronize between technology and human language with machine learning and deep learning frameworks. But before feeding the text data to different machine learning models, there are some basic steps that are needed to be implemented for any raw text data.

This blog is intended to understand the basic pipeline in NLP and convert it into the desired format, that is given as input for any Machine learning or deep learning algorithms.

Basic NLP Pipeline-

Steps in NLP

As shown in the above figure, there are basic 5 steps. Definitely there are many others like Named-Entity Recognition(NER) tagging, coreference resolution etc but at an initial stage or as a beginner the above mentioned steps are important to know and understand. These steps are easily implemented with Python either with Nltk or Spacy libraries. I will be demonstrating here using nltk library.

Tokenization:

Tokenization is a way of separating a piece of text into smaller units called tokens.This can be done at the sentence level or word level. The following example shows the word tokenize implementation.

The output will be as follows:

['[',

"'The",

'Plateau',

'coffee',

'tables',

',',

'designed',

'by',

'Büro',

'Famos',

',',

'comprises',

'just',

'two',

'simple',

'wooden',

'components',

'a',

'top'

Text Cleaning:

This phase deletes words and items from a corpus of text data to help enhance a machine-learning model ‘s efficiency. Numbers, capitalization, punctuation, stopwords, single quotes will be removed from the text data. Text cleaning process is done using regular expression. The following piece of code snippet will perform the task of text cleaning efficiently.

POS tagging:

POS tagging is a task of labelling each word in a sentence with its appropriate part of speech. Parts of speech include nouns, verb, adverbs, adjectives, pronouns, conjunction and their sub-categories. This will help as a pre-requisite to simplify a lot of different problems in NLP.

The output for POS tagging is as follows:

[('You', 'PRP'),

('just', 'RB'),

('gave', 'VBD'),

('me', 'PRP'),

('a', 'DT'),

('scare', 'NN')]

Stopwords:

These are the words that do not add much value to the meaning of the document. Stopwords are common in very language.Removing of stopwords must be done after tokenization. Nltk provides stopwords in many different languages like Danish,German,English and so on. Some of the benefits of removing stopwords are:

Data set size decreases when stopwords are removed and the time to train the model also decreases. Stopword elimination will theoretically help boost performance, as there are fewer and only relevant tokens remaining. This could thus improve the accuracy of the classification.

Lemmatization:

This reduces the inflected words with properly ensuring that the root word belongs to the language. It helps to get necessary and valid words. Nltk provides WordNet Lemmatizer that uses the WordNet Database to lookup lemmas of words.

After performing these steps, the text will be in the desired format which will be given as input for ML model.