De-mystifying English text for computers through Natural Language Processing (NLP) in Python
By: Astika Nehra
Tags: Nltk, Tokenization, NLP, Text Processing, Spacy

The field of machine learning and artificial intelligence is very fascinating especially when we use it to understand the emotions and feelings of a person. But do you know what is the building block of Emotional Analytics?

You guessed it right, it’s NLP…

With the help of NLP, Scientists, Researchers and Engineers are able to create machines and robots that can understand human emotions just like a normal person would do. Sophia, Alter, Nadine are some examples of such robots which are trained on tons of data and part of speech that humans use in their day to day conversations.

Relation of NLP with Artificial Intelligence, Machine Learning, Deep Learning.

1. Need of computer intelligence to understand the language.

Consider these two instances for spoken and written English:

· When we talk to a person, we can get an idea of their emotional state by listening to their tone, pitch, the way in which they are speaking and the phrases they are using while speaking. · When we read a book or watch a movie and someone asks us what was the movie about, we tend to tell a 2–3 lines as a summary and tell that it was emotional, it was scary etc.

If we want to build a product and put these kinds of processes in form of a code then NLP can help. NLP allows you to play with text and predict the genre, sentiment and emotion of a particular text.

2. NLP vs Computational linguistics

According to me NLP is a technique or tool which can be very useful to understand text for solving real world problems. It aims for building methods for solving practical problems like speech recognition, information extraction, categorization, sentiment analysis, summarization etc.

Computational linguistics or CL as most people know it is also a method that can be employed to study human language. But unlike NLP it focuses on the properties of human language (like how do we understand languages? How do we produce languages etc). CL focuses more on emotional intelligence.

Representation of sound of spoken words

3. Some common terminologies

3.1 Corpus

All machine learning models require a dataset similarly a text processing/NLP model also requires a dataset which can be further divided into training data, testing data and validation data. The dataset required for NLP model is called a corpus (plural: corpora).

If you want to study the various corpus in detail then you can go this link.

3.2 Tokens

Contiguous units of raw text (sequence of characters) are called tokens. They are generally separated by white spaces or punctuation marks. Splitting of a text into tokens is called Tokenization. When it comes to tokenization, there are two kinds of commands sent_tokenize and word_tokenize.

As you can see that sent_tokenize splits a particular data into various sentences that is it identifies the matter in between two consecutive full stops.

Whereas word_tokenize splits data into words, it identifies the matter in between two consecutive spaces.

Note: if you are using Google Colab notebook then you have to add.

nltk.download (‘punkt’)

3.3 Metadata

It is any auxiliary piece of information associated with the text of a particular corpus, like identifiers, timestamps and labels etc.

Disclaimer: The image is copyrighted to their respective owner.

3.4 Uni-grams, Bi-grams, Tri-grams, …, N-grams

N-grams are consecutive token sequences of fixed length (n) occurring with the text. A bi-gram has two tokens, a uni-gram has one.

Example:

Text = “I like to travel”

Uni-gram: [(‘I’), (‘like’), (‘to’), (‘travel’)]

Bi-gram: [(‘I’, ‘like’), (‘like’, ‘to’), (‘to’, ‘travel’)]

Tri-gram: [(‘I’, ‘like’, ‘to’), (‘like’, ‘to’, ‘travel’)]

Example of Uni-gram, Bi-gram, Tri-gram and N-gram.

4. Libraries used for Text Processing

The 2 most commonly used packages used for text processing are NLTK and SpaCy.

4.1 Natural Language Tool Kit (NLTK)

It is my personal favourite as it contains so many various kinds of corpora and different algorithms which can make text processing easier. The brown corpus of NLTK is probably the most widely used corpora for creating training and testing data for a model.

Python code for using Brown Corpus of NLTK.

4.2 SpaCy

It’s an open-source library which can help you to build various NLP related projects. It features Named Entity Recognition (NER), dependency parsing, word vectors. It is used to implements various word embedding algorithms like word2vec, one hot vector encoding and integer encoding.

Tokenization using SpaCy Python library.