Basics of Natural Language Processing for Beginners
By: Keerthana Durai
Tags: Bag Of Words, Word Embeddings, Beginners Guide, Neural Networks, Tokenization

Natural Language Processing (NLP) is a part of AI (artificial intelligence) that deals with understanding and processing of human language. In real time, majority of data exists in the unstructured form such us text, videos, images. Mass of data in unstructured category, will be in textual form. To process this textual data's with machine learning algorithms, NLP comes in to play.

NLP use cases are Language translation, Speech recognition, Hiring and Recruitment, Chat Bot, Sentimental analysis and so on.

Examples:

Sentimental analysis

Sentimental analysis predicts whether the review is positive or negative.

Positive Review:

“Love the phone. the phone is sleek and smooth and beautiful I highly recommend the phone you will not regret getting this phone. “

Negative Review:

“The telephone headset is of poor quality , not understood that Dijon you when you call that sounds very bad, not recommending this article. I wanted to return it but it is very difficult and expensive it back from Venezuela , since the dollars in my country are scarce and would consist more than the…”

2. Named entity recognition

It is the task of identifying and categorizing entities from the text.

Input:

Dhoni is one of the famous cricketer in India team.

Output:

Dhoni |Person

India| Country

Terms used in NLP

Corpus refers to the collection of documents or text files. The tweet data from twitter is a corpus.

refers to the collection of documents or text files. The tweet data from twitter is a corpus. A text sample defined as Documents . Each tweet is a document.

. Each tweet is a document. Document comprises of Sentences . Each tweet has one or more sentences.

. Each tweet has one or more sentences. A text get divided in to smaller unit called Tokens. Each tweet sentence has one or more tokens.

NLP Techniques For Text

Natural Language Toolkit (NLTK) is the leading platform for building Python programs to work with human language data. It provides pre-defined functions for classification, tokenization, stemming, tagging and parsing.

Explore it here https://www.nltk.org/

Tokenization

Tokenization is the process of the splitting the text in to smaller units called tokens. This tokens comprises sentences, words, symbols, numbers etc.,

There are different type of tokenizers such us White Space Tokenizer, Tweet Tokenizer, Regex Tokenizer, Word Tokenizer, Sentence Tokenizer.

White Space Tokenizer is the commonly used one but it depends on the use cases.

Examples for Tokenizer:

Let’s import nltk and different form of tokenizer. sent_tokenizer function apply sentence level tokenization, word_tokenizer function apply tokenization to word level and TweetTokenizer function apply tokenization to twitter data's as shown in below code sample.

The inputs get split as two individual sentence for Sent_tokenizer output. In Word_tokenizer output, the inputs get broken to four words. In TweetTokenizer output the inputs get split in to ten words, symbols etc.,

Normalization

Normalization is the process of removing inflection from the word.

Documents may contain grammars to give meaningful information's but these words does not influence in text processing. In order to remove it, let us use techniques such us Stemming and lemmatization to chop off inflection.

Examples:

Trouble, Troubling, Troubled => Trouble

am, are, is => be

Stemming refers to a process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes. Lemmatization refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma.

Stemming vs Lemmatization

Both stemming and lemmatization removes inflection from word. However stemming induce words that are not in dictionary because it does not use any part of speech tagging, vocabularies and grammar relationship to words whereas lemmatization use all above mentioned process step by step and produce desired word.

Below code illustrate Stemming vs Lemmatization.

Stop words

Stop words are the common words which doesn’t add much meaning to the text. These words need to be removed in the text pre-processing stage, if not it cause noise while processing text .

Examples :

Conjunctions: for, and, nor, but, or, yet, and so

Articles: a, an, the

from nltk.corpus import stopwords allows to import stopwords module. NLTK has pre-defined stop words for English, it can be imported by stopwords.words(‘english’).

Tokenizer used to get tokens from text, with those tokens stop words get removed and placed in text_without_stopwords.

Code Sample:

BOW (Bag Of Words)

As discussed earlier, NLP process textual data’s with machine learning algorithms. Text can’t be given as input to ML algorithms, Text need to convert as vectors. This process also known as feature extraction from text. For this conversion available techniques are BOW(Bag Of Words) and word embeddings.

A bag of words represent the occurrence of words within a document. By implementing Tokenization, ignoring the stop words ,applying normalization and applying further pre-processing steps we trail and collect the words. Here, the model concern, whether the word present in document and has unique collection. This approach is a simple and flexible way of extracting features from documents.

Steps involved to create BOW

Import Necessary Libraries & Load the text.

Text pre-processing & Vocabulary collection.

Implement Vectorizer.

Let us import all the necessary libraries from NLTK to perform the earlier discussed pre-processing techniques such us Tokenization, Normalization, Ignoring Stop words. Additionally, in below code while performing lemmatization the special characters are removed and text case changed to lower in order to perform identical case to all text.

Import Necessary Libraries & Load the text

Text Pre-processing & Vocabulary collection

Vocabulary collection can be done by two ways. Easy way of implementation is with Count vectorizer from Scikit-learn or with following all pre-processing steps & vectorize it.

Vocabulary collection without Count Vectorizer:

Output:

Vocabulary collection with Count Vectorizer:

Next we need to convert the text to vector. The simplest method is to note the presence of words with 1 and 0 for absence of word. For this, we need to implement Count Vectorizer.

Output:

Vectorizer

Without Count Vectorizer:

Check whether word present in document if yes append 1 or else append 0.

With Count Vectorizer:

Output:

Better Understanding:

Let’s consider 1st sentence “I like this car”. Now observe the vector given by count vectorizer, wherever the words of sentence (i.e., ‘I’, ‘like’, ‘this’ ,‘car’) seen it mark as 1 apart from that it marked as 0.

Word Embeddings

Why Word Embedding? If the new sentences contain new words, then our vocabulary size would increase and thereby, the length of the vectors would increase too and the vector representations will have a lot of zeros which are called sparse vectors. These vectors do not have a neighborhood or directional relationship between words

Word embedding techniques are TF-IDF (Term frequency — Inverse Document Frequency), Word2vec, Glove (Global Vectors).

1. TF-IDF

Term frequency is defined as the number of times a word appears in a document divided by the total number of words in the document. Generally, known as Normalization in some sense.

Term Frequency

Inverse data frequency defined as the log of the total number of documents divided by the number of documents that contain the word. In below formulae 1 may or may not included it’s up to your wish. It is just for standardization.

Inverse Document Frequency

TF-IDF defined as below

Example of TF_IDF Calculation:

Let us assume that the word “system” appears 6 times in a document of 10000 words. Calculate TF?

TF= No. of. times word appear/Total number of words

TF= 6/10000=>0.006

TF=0.006

Let us assume that the word “system” appears 60 times in a 10000 document. Calculate IDF?

IDF=log( No. of. total documents/No. of. documents that contain the word)

IDF=log(10000/6)=>3.21

IDF=3.21

TF-IDF=0.006 * 3.21

TF-IDF=0.01926

TfidfVectorizer from Scikit-learn allows to apply TF-IDF word embedding technique.

Code Sample:

Output:

From above output, let us try to impute it manually for 1st sentence ‘I like this car’. Consider the word “car” in the text for TF-IDF calculation.

Calculation:

Total number of words = 11

TF=1/4=>0.25

IDF=log(3+1/1+1)=>1.308

TF-IDF=0.25 * 1.308=>0.327

The Scikit-learn implementation applies normalization on the product between TF and IDF

Normalization of the word “car”= square root(0.327) =>0.57

2. Word2vec

Word2vec invented in 2013. It predicts the vectors only by considering the local co-occurrence of the predictive word. The word prediction is happened by looking the range of window size.

Word2vec incorporated with two architecture such us CBOW and Skip-Gram. In CBOW (Continuous bag of words) we try to predict a word by looping many words whereas in Skip-Gram we try to model the contextual words by given particular word.The below architecture illustrate the same .

In Word2Vec, the similar words are located together in the vector space and arithmetic operations on word vectors can pose semantic relationships.

For instance: “king” — “man” + “woman” => “queen”

Example for skip-gram :

Q: Walt Disney invented Disneyland. In this sentence Let us predict the word Disneyland.

Word2vec is built by neural networks. As known we can’t pass text as input to models so we need to convert the words to vectors by marking predicting word as 1 and others as 0. In order to predict the word “Disneyland” high weightage is given to it. So, the output layer ( soft max ) predict word with high weightage accurately..

Skip-Gram Example Architecture

Example for CBOW

Q: Walt Disney invented Disneyland. In this sentence Let us predict all the words except Disneyland.

Here, we need to pass the vice-versa of skip-gram model and predict the words by inducing high weightage to all words except “Disneyland”

CBOW Example Architecture

Code Sample:

Before look in to with code sample, I would recommend to explore Gensimword2vec

3. GloVe (Global Vectors)

GloVe was invented in 2014. It is a unsupervised learning algorithm for representations words by vectors. It captures word-word co-occurrences in the entire corpus better than the word2vec. In other words, it creates a global co-occurrence matrix by estimating the probability of other words in corpus.

Why Glove is better than Word2vec ? Both models are fundamentally same, they learn vectors from the co-occurrence. However Word2vec is predictive model and GloVe is count based model. This predictive model learnt the vectors in order to improve the prediction accuracy but count based model overall statistics of the co-occurrence. In Glove , it is easier to train more data’s than word2vec.

Example for GloVe:

Let’s Consider text as “I love NLP. I Love to write blog”. First, find all unique words from text including any special characters present. Next find co-occurrence for each words.

For instance, consider the word “I” and find left and right words of “I” (i.e., co-occurrence of word). Mark as 1 for all co-occurrence words and 0 for others.

Summarization of above table.

Glove Vectors Representation

Resources

I would love to hear some feedback . Thank you for your time!