How To Build a Basic Chatbot from Scratch | by Pratheesh Shivaprasad | Towards Data Science

Be it a Whatsapp chat, Telegram group, Slack channel, or any product website, I’m sure you have encountered one of these bots popping out of nowhere. You ask some questions and it will try it’s best to resolve your queries. Today we’ll try to build a chatbot that could respond to some basic queries and respond in real-time.
Let us just assume that we’ll be building a chatbot for a restaurant. Our client has certain requirements for the chatbot.
The bot should be able to:
Greet the visitors on the website.Book seats.Show available seats.Show what’s on the menu.Show the working hours.Show contact information.Show the location of the restaurant.
Greet the visitors on the website.
Book seats.
Show available seats.
Show what’s on the menu.
Show the working hours.
Show contact information.
Show the location of the restaurant.
In the end, our chatbot will look like this:
If you have a basic understanding of Natural Language Processing (NLP), Fully Connected layers in Deep Learning, and Flask framework, this project will be a breeze for you. Even if you aren’t familiar with these terms I’ll try my best to explain everything in simple language and link helpful resources wherever possible.
With that being said, let's start building our chatbot.
Now it’s time to see what kind of data we’re dealing with here. The data required for building a chatbot is a little different than the conventional datasets we tend to see. Every intelligent machine needs data that it can see and interpret. We won’t be downloading any particular dataset for this project. We already have a small set of data. Let’s have a look at it.
This is an example of how our data looks like. It is a JSON format file that is normally used to store and transmit data. The three important terms to be noted here are “tag”, “patterns” and “responses”.
Whatever queries a user enters for the chatbot to interpret must be included in “patterns”.
After interpreting the user’s query, the chatbot will have to reply to the query and this reply will be randomly selected from the set of predefined replies in “responses”.
The “tag” groups a set of similar patterns and responses to a specific category so that it’ll be easier for the model to predict which category a particular pattern represents.
You can now copy the data given below into a file. I have named my file “intents.json”.
As you can see the data provided here meets all of the requirements given by our client. I have intentionally set the responses of the tags “menu”, “book_table” and “available_tables” as an empty list. I’ll explain the reason later in our project.
With this data, we can now train our own neural network and it will predict and try to classify it into one of the tags from the file. Once the tag is known, a random response will be selected from that tag and shown to the user. You can add any other tags you wish to this data. Just make sure that the syntax isn’t wrong. The more tags, patterns, and responses you provide, the more robust the chatbot will be.
Now that you're familiar with the data let’s load it onto the kernel using Python. The version that I’m using is Python 3.6.
Before starting with any code, it’s recommended to set up a virtual environment so that any libraries we’ll be installing won't clash with existing ones or cause any redundancy issues.
I’ll be creating a virtual environment using conda. (See here on how to install Anaconda.)
Open your command prompt and enter the command.
conda create -n simple_chatbot python=3.6
Here “simple_chatbot” is the name of the virtual environment. You can give it any name you like.
To activate this virtual environment, just enter:
conda activate simple_chatbot
Once activated, the name of your environment should show up on the left something like this:
We’ll be using pip to install the following libraries:
numpy==1.16.5
nltk==3.4.5
tensorflow==1.13.2
tflearn==0.3.2
flask==1.1.1
I have also added the version of the libraries just to be safe.
pip install packagename==version //Enter packages mentioned above
Now that we’re prepped up, let's dive into the code.
Now we’ll be importing some libraries needed to load, process, and transform our data and then feed it into a deep learning network. Just remember to keep your JSON file in the same directory as your python file. I’ll be naming my file “main.py”.
Now, we have to take the “tag” and “patterns” out of the file and store it in a list. We’ll also make a collection of unique words in the patterns to create a Bag of Words (BoW) vector.
In the code above, we have created four empty lists.
words: Holds a list of unique words.labels: Holds a list of all the unique tags in the file.docs_x: Holds a list of patterns.docs_y: Holds a list of tags corresponding to the pattern in docs_x.
words: Holds a list of unique words.
labels: Holds a list of all the unique tags in the file.
docs_x: Holds a list of patterns.
docs_y: Holds a list of tags corresponding to the pattern in docs_x.
As we loop through the data, we convert all patterns into lower case, tokenize each pattern and then add them to the respective lists. We also add the pattern’s tag into docs_y simultaneously.
Now that we’ve got the words in a list, it’s time to perform stemming on them. Stemming is basically trying to find the root origin of a word. It removes all the prefixes and suffixes of a word so that the model that we’re building will get a general idea of that word instead of getting trapped in all the intricacies of the same word with different forms. There are different types of stemmers like Porter Stemmer, Snowball Stemmer, Lancaster Stemmer, etc. We’ll be using Lancaster Stemmer in our code. (Learn more about Stemmers here.)
It’s known that Machine Learning and Deep Learning models only accept numerical inputs. So we have to convert this stemmed list of words into some kind of numerical input so that we can feed it to the neural network. This is where vectorization methods like Bag of Words, TF-IDF, Word2vec, and others come into the picture.
We’ll be using Bag of Words (BoW) in our code. It basically describes the occurrence of a word within a document. In our case, we’ll be representing each sentence with a list of the length of all unique words collected in the list “words”. Each position in that list will be a unique word from “words”. If a sentence consists of a specific word, their position will be marked as 1 and if a word is not present in that position, it’ll be marked as 0.
With this method though, the model can only understand the occurrence of a word in the sentence. The sequence of the words within the sentence will be lost, hence the name “Bag of Words”. Other methods like TF-IDF, Word2Vec tries to capture some of these lost semantics in their own way. I’d recommend you to try out other vectorization methods as well.
Similarly, for the output, we’ll create a list which will be the length of the labels/tags we have in our JSON file. A “1” in any of those positions indicates the belonging of a pattern in that particular label/tag.
We will be also saving all the processed data in a pickle file so that it can be used later for processing inputs from the user as well.
Now that we’re done with data preprocessing, it’s time to build a model and feed our preprocessed data to it. The network architecture is not too complicated. We will be using Fully Connected Layers (FC layers) with two of them being hidden layers and one giving out the target probabilities. Hence, the last layer will be having a softmax activation.
Feel free to mess around with the architecture and the numbers to get the model that suits your requirements. You could also choose to add a bit more steps into text preprocessing to get more out of the data. The more trial and error cycles you perform better will be your understanding of the architecture.
All we have to do now is feed the data to this model and begin training. We will set our epochs to 200 and batch size to 8. Again, you can experiment with these numbers and find the right one for your data. After training, we will be saving it on the disk so that we can use the trained model in our Flask application.
If everything goes right, you should have files named “model.tflearn.data”, “model.tflearn.index”, and “model.tflearn.meta” in the working directory.
Your “main.py” file should look something like this:
Now that we’ve trained our deep learning model, it’s time to integrate it into a web application. The framework we will be using here is Flask. Now explaining how to build a Flask application requires a series of articles on its own so I won’t be doing that here. However, I’ll make sure to include all the resources I referred to while building this application in the “References” section down below.
Here’s a link to my GitHub repository where you can access all the files to build the Flask application. I’d recommend you to copy all the files into your working directory so that it can access the trained model and pickle files.
We will be using AJAX for asynchronous transfer of data i.e you won’t have to reload your webpage every time you send an input to the model. The web application will respond to your inputs seamlessly. Let’s take a look at the HTML file.
Have a look at the JavaScript section where we get the input from the user, sends it to the “app.py” file where it’s fed to the trained model and then receives the output back to display it on the app.
The “app.py” file is where all the routes are mentioned, input data is processed (Stemming and Bag of Words), and fed to the model for output. Let’s have a look at it.
We first import all the required libraries and then load the pickle file in which we saved the preprocessed data. This will be required to create a BoW vector for the input data. Then we define a function “bag_of_words” where we provide user input and get a BoW vector as an output.
Remember the tags “menu”, “book_table” and “available_tables” with empty responses? We did that so we could provide custom responses to questions in those tags.
For the questions related to what’s on the menu, we would first check what day of the week it is and according to that, the chatbot will recommend special dishes for the day.
When the user asks to book a table, we decrement the counter “seat_count” by one. Now, of course, this isn’t how bookings work but the whole point of demonstrating this was to show the possibilities we can have with this chatbot. You could connect this bot to a database and conduct bookings accordingly. Or you could add any other additional tasks according to your requirements. Asking for available tables just shows the current value of “seat_count”. Whenever you reset the Flask server, the counter goes back to 50. This was done to show that you don’t have to always depend on the JSON file for a response.
Once the setup is complete, just run the “app.py” file and the Flask server will be live. It should look something like this:
The project is finally done and working like it’s supposed to. However, there are tons of stuff you can tweak and fine-tune in here. I’d love to hear what additional features you were able to add to this chatbot or any other modifications you have made to this project. You can go ahead and connect this chatbot to a database or integrate it with any website. Instead of using Bag of Words, you could also use other vectorization methods like TF-IDF, Word2Vec, etc. There is a high chance that these methods will improve the model’s accuracy in predicting the tags.
As promised I have also listed all the blogs and videos I referred to while building this application. It’ll definitely help you get a deeper understanding of the project and its working.
Contextual Chatbots with TensorflowFlask Tutorials by Corey SchaferBuilding a Chatbot in Python Using FlaskSubmit AJAX Forms with jQuery and Flask
Contextual Chatbots with Tensorflow
Flask Tutorials by Corey Schafer
Building a Chatbot in Python Using Flask
Submit AJAX Forms with jQuery and Flask