Build Your First Chatbot With Python and Integrate It on Telegram
By: Shubham Sood
Tags: Machine Learning, Python, NLP, Chatbot, Programming

NLTK

NLTK (Natural Language Toolkit) is the primary platform for building Python projects to work with human language information. It gives simple-to-utilize interfaces to more than 50 corpora and lexical resources, for example, WordNet, alongside the setup of text-handling libraries for classification, tokenization, stemming, tagging, parsing, and semantic thinking, and wrappers for industrial-strength NLP libraries.

Downloading and installing NLTK

Install NLTK: run pip install nltk Test installation: run python then type import nltk

For platform-specific instructions, read the NLTK documentation.

Installing NLTK packages

In your Python interpreter, run the following commands:

import nltk and run nltk.download() .This will open the NLTK downloader from where you can choose the corpora, models, and other data packages to download. You can also download all packages at once, or you specify the package you need as an argument in nltk.download() .

Text pre-processing with NLTK

The principle issue with text information is that it is all in strings (a group of text). However, machine learning calculations need a type of numerical element vector to perform the task. So before we start with any NLP project, we have to pre-process it to make it perfect for work. Fundamental text pre-handling incorporates:

Changing over the whole content into uppercase or lowercase , with the goal that the calculation doesn’t treat the same words in different contexts.

, with the goal that the calculation doesn’t treat the same words in different contexts. Tokenization is a process in which the text of strings is converted into a list of tokens. There is a sentence tokenizer that can be used to find the list of sentences and word tokenizer that can be used to find the list of words in strings.

is a process in which the text of strings is converted into a list of tokens. There is a sentence tokenizer that can be used to find the list of sentences and word tokenizer that can be used to find the list of words in strings. Removing noise will remove everything that isn’t a standard letter or number, like a punctuation mark, extra spaces, etc.

will remove everything that isn’t a standard letter or number, like a punctuation mark, extra spaces, etc. Removing stop words: Stop words are words that are commonly used (such as the, a, an, in, etc.) that have little value in selecting the matching phrase according to a user query.

Stop words are words that are commonly used (such as the, a, an, in, etc.) that have little value in selecting the matching phrase according to a user query. Stemming: It is a process of reducing a derived form of a word to its stem, base, or root form. For example, if we stem the following words: walks, walking, walked, then the stem word would be a single word, walk.

It is a process of reducing a derived form of a word to its stem, base, or root form. For example, if we stem the following words: walks, walking, walked, then the stem word would be a single word, walk. Lemmatization: A transformed version of stemming is lemmatization. The significant difference between these is that stemming operates on a single word without the knowledge of the context and can often create a non-existing word. In contrast, after lemmatization, we will get a valid word that has meaning in the dictionary. Lemmatization is based on the part of speech of a word that should be determined to get the correct lemma of the word. An example of lemmatization is that is, am, and are are forms of the verb to be; therefore, their lemma is be.

Bag of words

After the initial cleaning and processing phase, we need to transform the text into a meaningful vector (or cluster) of numbers. According to A Gentle Introduction to the Bag-of-Words Model by Jason Brownlee,

“The bag of words is a representation of text that describes the occurrence of words inside a document. It includes two things: 1. A vocabulary of known words 2. A measure of the presence of all the known words”

In the bag-of-words model, a text (such as a sentence or a document) is represented as the bag of its words, disregarding grammar and even word order but keeping multiplicity. The main reason behind using it is to check whether the sentence is similar in content or not with the document.

Suppose the vocabulary contains the words: {knowing, must, do, enough}. And we have the following sentence: “Knowing is not enough; we must apply.” Then bag-of-words representation for this would have the resulting vector: [1, 1, 0, 1].

TF-IDF approach

The main disadvantage of using the bag-of-words model is that it leads to a high dimensional feature vector due to the large size of vocabulary, which leads to a highly sparse vector as there is non-zero value in dimensions corresponding to words that occur in the sentence.

To overcome the disadvantage of bag of words, we perform normalization on the frequency value. A new approach, TF-IDF (term frequency-inverse document frequency), was invented for document search and similarity, and information retrieval. So, words that are common in every document (such as this, what, and if) rank low even though they may appear many times since they don’t mean much to that document in particular.

TF (Term Frequency): is scoring of how frequently a word occurs in the current document.

TF = (Number of times term t appears in a document)/(Number of terms in the document)

Inverse Document Frequency: is a scoring of how frequently the word occurs across the documents.

IDF = log(N/n), where, N is the total number of documents and n is the number of documents a term t has appeared in.

Finally, by taking a multiplicative value of TF and IDF, we get the TF-IDF score.

TF-IDF = TF * IDF

Let’s take an example where we have a document of four words, and best occurs one time in it. Therefore, the TF for the word best is then(1/4) = 0.25.

Now assume, we have five documents, and the word best occurs in two of them. Then the IDF is calculated as log(5/2) = 0.39. Thus the TF-IDF is the product of these quantities: 0.25 * 0.39 = 0.0975.

Cosine similarity

We applied TF-IDF to transform the text of the document into a real-valued vector in vector space. We can obtain a cosine similarity to determine how similar the vectors are irrespective of their size. We can then get the cosine similarity of any pair of vectors by taking their dot product and dividing that by the product of their norms. That yields the cosine of the angle between the vectors. Using the following formula, we can find out the similarity between any two document d1 and d2, where d1, d2 are two non-zero vectors.