A Chatbot in Python using nltk
By: Bhargava Sai Reddy P
Tags: Nltk, Machine Learning, Data Science, Naturallanguageprocessing, Python

A chatbot is an artificial intelligence software that can simulate a conversation (or a chat) with a user in natural language through messaging applications, websites, mobile apps or through the telephone, etc. These chatbots can be used in various industries for different purposes.

We are going to build a simple chatbot using nltk library( Natural Language Toolkit). It is a leading platform for building Python programs to work with human language data.

importing necessary libraries.

import the dataset into the pandas data frame.

The above data contains 1592 data points and two columns context which can be inferred as the query and text response is the response for that query. we can see that there are null values in the dataset if you open the dataset in excel and observe we can find that in our dataset the data is in different clusters i.e, the same type of questions in one place and then followed by next similar kind of questions.

Null values are present for the same type of questions whose response can be almost similar and in that similar group of questions, the response is given to the first and the rest filled with null.so what we can do is use ffill() which returns the value of previous response in place of null values as below.

Steps involved:

So let’s get into our first step i.e, text normalization where we convert the data into lower case and then remove special characters and then perform lemmatization.

Let us create a function that converts given text to lower case and removes special characters and numbers.

We can see that our text is clean. Word tokenizing is the process of converting the normal text strings into a list of tokens.

The pos_tag function returns the parts of speech of each token so that the lemmatizer function detects the parts of speech of token and then it converts the token to its root word as below

We shall now create a function that performs all the steps mentioned above

Let’s check our function and apply it to the dataset.

We can see that our function worked well and thus we applied the same to our data. Our next step is word embedding, it is representation for text where words that have the same meaning have a similar representation. We have two models for this process bag of words (bow) and tf-idf ( Term Frequency-Inverse Document Frequency).

Using bow :

The bag-of-words is a representation of text that describes the occurrence of words within a document. Consider if our dictionary contains the words {Playing, is, love}, and we want to vectorize the text “Playing football is love”, we would have the following vector: (1, 0, 1, 1).

Consider the row 0 of the dataset the bow marks all the words present in the text as 1 and rest as 0.

Stop words are extremely common words that would appear to be of little value in matching a user’s need and hence they are excluded from the vocabulary entirely. Below are the predefined stop words.

Let us consider an example and try getting a response to the query.

From above We can see that we have taken a question ‘Will you help me and tell me about yourself more’ and then perform text normalization and then applying the bow to the question. Now to get the related response we shall find the cosine similarity between the question and the lemmatized text we have.

Cosine Similarity :

Cosine similarity is a measure of similarity between two vectors. It returns a value that is computed by taking the dot product and dividing that by the product of their norms between two vectors.

Cosine Similarity (a, b) = Dot product(a, b) / ||a|| * ||b||

We can see that at index 194 we have the highest similarity text for the query we considered. Let us print the text at that position and see whether it is related or not.

We can see that our model worked pretty well :)

tf is Term Frequency, scoring of the frequency of the word in the current document and idf is Inverse Document Frequency, scoring of how rare the word is across documents. Here document represents a single text i.e, say row 0 or row1, etc, where documents refer to all the rows in the dataset.

The above are the values obtained using tf-idf. Now using cosine similarity lets us find the response that we get with tf-idf.

At index 4 we got higher similarity text that relates to our question. Let us see the response to our question.

Using tf-idf we got a different response which is also a pretty good response to the question.

Now let’s build a function that returns the response to the query using tf-idf. It is very simple we just have to combine all the topics we saw earlier in this article.

Let’s see some output responses for a different queries.

That’s cool know, our model responded pretty well to all the queries.

Conclusion:

We can also build a bot using the bow in the same way, all we have to do is to follow three steps normalization of text, word embedding, and similarity. The model we built doesn’t have any artificial intelligence, but still, it responded pretty well. For complete code, you can check out my GitHub.