Chatbots are cool! A framework using Python | by Sundar Krishnan | Towards Data Science

Did you like what you saw on the Kelly Movie Bot video? If the answer is “Yes” and you want to implement a chatbot, you can start reading this article. This article is very detailed. I split the entire article into parts. So, sit back and relax! We will go through each one of the parts and finally at the end of the article, you should have deployed a chatbot in Slack.
Part 1: OverviewPart 2: DataPart 3: SlackPart 4: NLP using IBM WatsonPart 5: Custom NLP solutionsPart 6: LogsPart 7: Final Notes
Part 1: Overview
Part 2: Data
Part 3: Slack
Part 4: NLP using IBM Watson
Part 5: Custom NLP solutions
Part 6: Logs
Part 7: Final Notes
Audience for this article: I designed a generic chatbot framework and discussed in this article to cover a wide range of audience. Anyone who has a basic knowledge of Python, Jupyter notebooks and can perform pip installations should be able to complete this series and see the results.
The bot framework is modularized which opens up an array of opportunities for the readers to design and implement their own features. Integrations can be done easily in the framework. Also, the probability for failure is minimal since it is designed to be plug and play.
Beginner: An overall idea on how the framework is developed and used for this specific project. You should be able to download the codes from Github and complete the setup successfully. This includes package installations, slack and IBM Watson account creation and setup, run one time files to generate the links and movie recommendations. You can add extra skills in IBM Watson (like a small talk which generate static responses) and see the results in slack environment.
Intermediate: You should be able to use this framework as a template to design your own chatbot which can be deployed on a different domain. In addition, you can extend the knowledge base for the chatbot by adding new data sources which includes writing codes to connect to different databases (Elastic search, SQL databases, Excel and so on..). Also, you can add extra NLP features to the bot and see the results in slack environment.
Expert: You should be able to add/extend bot features by integrating API connections for Slack/NLP. I used IBM Watson to identify question category and to generate static responses. You can replace IBM Watson in the framework by designing your own NLP capabilities. Also, you can extend the bot integrations for different platforms (Web, Skype and so on..)
Chatbot is a tool to retrieve information and generate humanlike conversation. It is mainly a dialog system aimed to solve/serve a specific purpose. Depending upon the design, chatbots generally fall into 4 categories.
Open Domain
Open Domain bots are otherwise known as Generalist bots. Today we use Alexa, Google Home, Siri, Cortana which fall under this category(open domain/generative-based). These bots try to imitate humanlike conversation. Again, it answers questions (like FAQ’s) asked by most humans. However, they cannot answer a specific domain based question. For example: How did my company sales division performed in the last quarter? That is one of the reason, open domain/retrieval-based bots is impossible to build.
Closed Domain
Closed Domain bots are otherwise known as Specialist bots. Depending upon the type, it can be easy (retrieval-based) or hard(generative-based) to develop. The bot discussed in this article is a specialist bot and it falls under the closed domain/retrieval-based category. Other bots in this category include — order a pizza, book flights/restaurants/hotel/appointments. On the other hand, generative bots include customer service chatbots which try to imitate like a agent while answering the questions from customer. These bots are hard to build since the bots try to make the customer believe that they are talking to a actual human.
Chatbots needs to understand the following to respond to an user question.
What is the user talking about? (Intent)Did the user mention anything specific? (Entities)What should the bot ask to get further details from the user? (Dialog/Maintaining Context)How to fulfill the user request? (Response/Fulfillment)
What is the user talking about? (Intent)
Did the user mention anything specific? (Entities)
What should the bot ask to get further details from the user? (Dialog/Maintaining Context)
How to fulfill the user request? (Response/Fulfillment)
Let us use a Flight Bot example shown below to understand each of these pieces in detail. The user says “I want to book a ticket from New York to Seattle departing on September 15 and returning on September 19 for 2 people”.
Intent
The intent of the user is to book flights.
Entities
Entities are also known as keywords or slots. Here there are multiple entities.
From - New York
To - Seattle
Departure date - September 15
Return date - September 19
#people - 2
Dialog/Maintaining Context
Dialogs are back and forth communication between bot and user. A context let’s the bot know what state the bot is currently in. There are three states — Previous, Present and Future. When the user initiates the dialog, the bot reiterates the user itinerary and then checks with the user “Is this info correct?”. Here the previous state is blank, present state is “user validation” and future state is to “Provide a response based on user validation”. When the user responds “Yes”, then the bot state changes to “User validation”, “Provide a response based on user validation” and “Book a flight” for Previous, present and future state respectively. Without maintaining the context, bots cannot establish the back and forth communication. In the flight bot example, if the context is not maintained the bot would be asking “Is this info correct?” every time until the user gives up.
Response/Fulfillment
Fulfilling the user request is the final step in the bot conversation. Here the bot provides the results in the form of links “See all results”. So when the user clicks the link, they will be able to see the flights and make a reservation.
Generative-based bots use AI and Machine learning to generate user responses. So they need not have to understand the Intents and Entities to respond to a user.
I found this article which covers a lot of topics that we discussed so far and also the bot frameworks that you can use to build chatbots. It is a good read.
www.datasciencecentral.com
Okay, now it is time to deploy the Kelly movie bot. It is a simple bot that answers questions about movies. The user can ask about ratings, #people voted for the movie, genre, movie overview, similar movies, imdb and tmdb links, budget, revenue and adult content. The dataset for this exercise is taken from Kaggle — movies_metadata. The entire code for this project can be found in Github.
github.com
One important thing to note with this design is that, the data and processing is all handled in the local system. Even though we use IBM, it is used as an API service and none of the internal data is sent to IBM. This way the entire design can be implemented in your workplace without having to worry about data transfers.
Step 1 (User asks question):
Users can interact with Kelly via Slack. Once the user post a question, it is passed to the backend system for analysis.
Step 2 and 3 (NLP processing and Return the NLP results):
All the natural language processing happens in step 2. This includes IBM Watson processing, similarity search, recommendation based on collaborative filtering. After the NLP processing is completed, we have three outputs from it
Intents — What the user is trying to ask or query?Entities — What is the exact field or column they are looking for?Dialog/Interaction — Provide the appropriate request/response for the user question.
Intents — What the user is trying to ask or query?
Entities — What is the exact field or column they are looking for?
Dialog/Interaction — Provide the appropriate request/response for the user question.
Step 4 and 5(Query the data):
Currently, the data resides in a excel file. However, you can add multiple databases/excel files if needed, to access different sources. Based on the results from step 3, the appropriate database/excel file is queried and the results are returned.
Step 6 (Post the result to user):
The results obtained from the backend is posted to user via Slack
Step 7 (Log maintenance):
The interactions between the users are logged and stored in a text file. Also, if the bot is not able to identify the user questions it will add those questions to a followup file.
Package Imports
As usual, we define the program by importing the packages. Notice something, where it says “slack.slack_commands” and “nlp.nlp_commands”. This would import the python program slack_commands and nlp_commands from slack and nlp folder respectively. In addition, a couple of functions from the config file is imported.
Variables Initialization
User context is maintained in the “context” variable. Once the chat begins, the context variable generated by IBM Watson looks like below,
{'conversation_id': '76e59c57-8257-4390-ae15-ba75e7576476', 'system': {'_node_output_map': {'node_2_1541443578896': [0]},  'branch_exited': True,  'branch_exited_reason': 'completed',  'dialog_request_counter': 1,  'dialog_stack': [{'dialog_node': 'root'}],  'dialog_turn_counter': 1,  'initialized': True}}
It has a “conversation_id” which is used to track the state (previous/present and future state) of conversation flow. During the start of the conversation it is assigned to a empty dictionary value.
In addition to maintaining a conversation, we also should let the bot know when to stop a conversation. This is achieved by the “current_action” variable. An example of the ending a conversation is shown below.
Whenever the user invokes a “Goodbye” intent, the bot assigns ‘end_conversation’ to “current_action” variable. Later in the main.py program, this variable will be evaluated to reassign context, current_action and session_df variables which is shown below.
Another key variable is the “session_df” dataframe. When a user starts a conversation with the bot a session is created. This ensures that the context between a user and the bot is handled on per-user basis. When the same user talks to the bot in multiple channels, they are handled as different sessions. So, there is no overlap in conversation between bot and user.
When a user asks a random question which the bot cannot handle, it will trigger the “follow_ind” to write the question in a follow up text file. During the start of the conversation, it is assigned to 0. The final two variables “bot_id” and “RTM_READ_DELAY” corresponds to slack. When the bot is invoked, the bot_id is assigned. RTM stands for Real Time Messaging. This variable will wait for a second before it reads the next input message from the user.
Invoke the code
Finally, the rest of the program ensures that the bot is consistently running and looking for input messages from the user. Some of the functions mentioned below — parse_bot_commands, handle_command and output_command will be discussed in detail in the later section.
That’s it. We initiated our Kelly bot. Our next step is to understand the data used for this exercise.
The data for this exercise is taken from the Kaggle link below. The name of the dataset is “movies_metadata.csv”.
www.kaggle.com
The dataset contains a lot of information related to movies with less preprocessing required from users. We import the dataset using Pandas and then prepare our data.
<class 'pandas.core.frame.DataFrame'>RangeIndex: 45466 entries, 0 to 45465Data columns (total 24 columns):adult                    45466 non-null objectbelongs_to_collection    4494 non-null objectbudget                   45466 non-null objectgenres                   45466 non-null objecthomepage                 7782 non-null objectid                       45466 non-null objectimdb_id                  45449 non-null objectoriginal_language        45455 non-null objectoriginal_title           45466 non-null objectoverview                 44512 non-null objectpopularity               45461 non-null objectposter_path              45080 non-null objectproduction_companies     45463 non-null objectproduction_countries     45463 non-null objectrelease_date             45379 non-null objectrevenue                  45460 non-null float64runtime                  45203 non-null float64spoken_languages         45460 non-null objectstatus                   45379 non-null objecttagline                  20412 non-null objecttitle                    45460 non-null objectvideo                    45460 non-null objectvote_average             45460 non-null float64vote_count               45460 non-null float64dtypes: float64(4), object(20)memory usage: 8.3+ MB
adult                        0belongs_to_collection    40972budget                       0genres                       0homepage                 37684id                           0imdb_id                     17original_language           11original_title               0overview                   954popularity                   5poster_path                386production_companies         3production_countries         3release_date                87revenue                      6runtime                    263spoken_languages             6status                      87tagline                  25054title                        6video                        6vote_average                 6vote_count                   6dtype: int64
It turns out there are a lot of missing values in the data. So we remove the variables with high missing value percentage. Also, I removed a few other variables to keep the data simple. The code below is used to remove variables from the dataset.
“imdb_id” and “poster_path” variable had missing values in them. I had to remove the observations which had missing values for these variables using the code below.
Now, let us look at the genres variables in the dataset.
The genre name is stored in a list of dictionaries format. To make it a clean format, we need to apply a preparation code below.
Pretty nice format right. Now let us move to our final step of creating the IMDB, TMDB and Image url links.
Voila. We are done with the data preparation and finally export the prepared dataset using the code below.
metadata.to_csv('metadata_prep.csv')
The entire data preparation code is available in the notebook below.
github.com
That’s it. This concludes the data preparation process. Our next step is to understand slack process.
Okay, now it is time to understand the slack process and how it is handled in the framework. The first step is to create a slack bot and install it in the workspace. The link below will guide you to create the slack app. In the configuration file, you need to edit the Slack Bot token and Slack verification token. That is all the setup required for slack.
github.com
There are 6 slack functions written for Kelly bot.
parse_bot_commandsparse_direct_mentionoutput_commandfile_uploadslack_tilesmessage_buttons
parse_bot_commands
parse_direct_mention
output_command
file_upload
slack_tiles
message_buttons
Let us look into each of the functions in detail.
parse_bot_commands
When the user types a message in slack, the parse_bot_commands function is invoked. In slack, the Real Time Messaging(RTM) events (slack_events) stores all the information about a particular user conversation. This includes who is asking this question (message_user), which team and channel are they using (team, channel), what is the message or question from the user (message) and are they talking to someone else or the bot (user_id). Because, in slack a user can talk to different users by invoking @ symbol in front of the user name. For example — “@person1 show me the movie Toy Story” and “@Kelly show me the movie Toy Story” are different. The first one is directed to person1 and second one is directed to Kelly bot. Here, the bot will process a user message or question only when the “user_id” matches the “bot_id”.
parse_direct_mention
This is a sub function of parse_bot_commands. The question from the user “@Kelly show me the movie Toy Story” has two parts. The first part is the bot user name “@Kelly” and the second is the question from the user “show me the movie Toy Story”. This function takes the text from slack events (message_text) and separates the user question from the bot user name.
output_command
The output command is used to send the responses back to slack. This function only supports text output (slack_output). The “channel” variable is used to identify the slack channel from which the user is posting the question and use the same channel to direct the response.
In Kelly bot, the slack output produced by the output command function looks like below.
file_upload
File upload can be used to upload files to slack. File upload function supports multiple document types. A few among them would include PDF, Word, Excel, gifs, png etc. The complete list of supported file types can be seen in the link below.
api.slack.com
This function is not used in Kelly Bot. However it is provided here as an additional feature.
slack_tiles
Slack tiles are another way of displaying outputs to user in Slack. I used this function to show movie recommendations. The output of the function looks like below.
Pretty cool right. The images are represented using tiles so that the user attention is immediately grabbed. When you look closer, the movie names are hyperlinked. So when you click on the movie names, it redirects you to the IMDB movie page. Again, a simple and elegant way to display output in slack.
The function takes 5 inputs.
The “channel” variable as discussed before is used to post to the corresponding channel where the user is asking question.The “search_term” variable has the value “Toy Story”. Since I started my bot conversation with the question “show me the movie Toy Story”, the movie name is automatically stored in search_term and the context is maintained each time the user asks questions until the user starts over. Since the context is retained, when I asked the question “recommend me movies similar to this one” it responded in the first line saying “Recommendation for Toy Story”.The “title” variable has the movie title recommendations. Based on the highest scores obtained in the collaborative filtering method(which will be discussed later in Part5:Custom NLP solutions), it chooses the titles and stores them in a python list. This list is then processed in the function while displaying the output.The “title_url” has the corresponding IMDB links for the movie recommendationsThe “image_url” has the corresponding TMDB image links for the movie recommendations.
The “channel” variable as discussed before is used to post to the corresponding channel where the user is asking question.
The “search_term” variable has the value “Toy Story”. Since I started my bot conversation with the question “show me the movie Toy Story”, the movie name is automatically stored in search_term and the context is maintained each time the user asks questions until the user starts over. Since the context is retained, when I asked the question “recommend me movies similar to this one” it responded in the first line saying “Recommendation for Toy Story”.
The “title” variable has the movie title recommendations. Based on the highest scores obtained in the collaborative filtering method(which will be discussed later in Part5:Custom NLP solutions), it chooses the titles and stores them in a python list. This list is then processed in the function while displaying the output.
The “title_url” has the corresponding IMDB links for the movie recommendations
The “image_url” has the corresponding TMDB image links for the movie recommendations.
message_buttons
Message buttons are also a different style of displaying output in Slack. This is similar to slack tiles except that it wont have the images to display. Rather, the output contains buttons that are tied to IMDB links. This features is used in the start, when the user is trying to search a movie. I don’t remember all the movie names that I watched on the top of my head. So, I use keyword based movie search and this functionality is also implemented in the same way. The NLP program that handles the keyword based search (Similarity Search) will be discussed later in Part5:Custom NLP solutions. However, I wanted to show how this works with message buttons.
Here, when I began the movie search I did not mention “Toy Story” rather I started the conversation as “I am looking for a movie with a toy” and it gave me “Toy Story” as my 5th option. This type of search is called similarity search. Now, the user can click on button to ensure whether the movie they are looking for is the same and then proceed next. The button names are stored in the “button” variable and the corresponding IMDB links are stored in the “url” variable.
That’s it. This concludes the slack process. Our next step is to understand Natural language processing using IBM Watson.
A quick and easy way to develop chatbots is to use frameworks. There are lot of frameworks in the market, that can assist us to build bots. A few of them include — IBM Watson, Luis, Wit, Dialogflow, Rasa NLU, Botkit, Microsoft Bot Framework and so on. You can chose any one of the framework to build your bots. Again, the complexity of coding might vary depending upon the framework you choose. For this exercise, I used IBM Watson framework. The link below will assist you to create a Watson User account and getting started with IBM Watson conversation.
github.com
IBM Watson framework is used as an API service to perform Natural language processing. Kelly bot is built in such a way that none of the information is stored in IBM Watson. All the processing of information is happening in your local system. Only the question from the user is processed by IBM Watson service to provide the Intents, Entities, Dialog and Response. Other than that, none of the information is stored in IBM Watson.
These quick courses can get you started with building Intents, Entities and Dialog.
www.coursera.org
developer.ibm.com
For this exercise, you can upload the “bot.json” file that comes with the code in Github and import it to your IBM Watson conversation account to get started. Once you import the json file, the page should look like below.
Intents
Entities
I wanted to talk about a few things on how the connection is established between Watson, Slack and your python code. So, let us take a look at the entities first.
Here, look at the option entity. It can take only 5 possible values. If you can recall in the slack process, we used message buttons to get the movie name option from user.
Dialog
Let us say, if the user selects 6 instead of 5. The bot would then respond saying “Please provide a valid option — 1, 2, 3, 4 or 5”. This is established using the Dialog feature in IBM Watson. So let us look at this feature. The Dialog page should look like below.
We start the conversation by saying “hello”. The Watson would recognize this as “hello” intent, assigns the value “hello” to context variable “currentIntent”. In addition, it sends the response back to python code “Please enter a movie name to begin” and then jumps to evaluate movie name condition.
In the movie name condition evaluation, I have two child nodes as shown below.
The first child node evaluates whether a user has provided response. In IBM Watson, this is captured using “<?input_text?>” option. And then the bot responds to user to select a movie option as shown below.
Finally, based on the user option, it evaluates whether the option is a valid. If not, it would request the user to provide a valid option.
The corresponding slack interface interaction for this Watson conversation setup is shown below.
Python backend
In the python end, this is established by the handle_command function shown below.
When the IBM Watson API service is used, it returns a response which is captured in the “response” variable. Notice we pass the context variable to Watson service each time. This is used to maintain the context for the bot responses. Also, using the response variable we extract the context, entities and the bot response and assign them to “context”, “search_key” and “slack_output” respectively. The “search_term” variable contains the movie name corresponding to user selection. For now, I hope this gives a picture of how the interactions happen between the IBM Watson front end and the backend python code.
That’s it. This concludes this part — NLP using IBM Watson . Our next step is to understand the Custom NLP solutions written for Kelly bot.
Often times, we need to add extra NLP capabilities to enhance the bot skills. Well, this is your time to become more creative and start adding customized solutions. I added two custom solutions for Kelly Bot. (Located in the nlp/nlp_solutions folder).
Metadata based Collaborative filtering for movie recommendationCosine similarity search for identifying the movie titles
Metadata based Collaborative filtering for movie recommendation
Cosine similarity search for identifying the movie titles
The codes for both the function resides in “nlplearn.py” and “onetime_run_file.py”. So let us get started.
onetime_run_file.py
This function has codes that we need to run onetime before we initiate the Kelly bot. This onetime setup makes it easier in situations when the data is changing constantly and the bot needs to adapt to these data changes quickly. All we need to do is to run this file whenever we need without changing any of the internal functions in the bot. Pretty cool right. When the file run is completed, it generates a “onetime.txt” pickle file which will be used by the bot when it is initiated.
Now, let’s discuss the functions available in “nlplearn.py”. It has 5 functions,
text_preparetfidf_fit (Cosine similarity search)similarity_search (Cosine similarity search)metadata_filtering (Metadata based Collaborative filtering)get_recommendations (Metadata based Collaborative filtering)
text_prepare
tfidf_fit (Cosine similarity search)
similarity_search (Cosine similarity search)
metadata_filtering (Metadata based Collaborative filtering)
get_recommendations (Metadata based Collaborative filtering)
text_prepare
This function takes in a single document (“show me ? movie title ` similar to Toy Story######”) and prepares the text by removing spaces and bad characters. In addition, it also removes stop words from the text. The processed text looks something like this “show movie title similar toy story”.
tfidf_fit
This function takes the prepared text from step 1 and creates a tfidf fit. ‘tf’ means Term Frequency and ‘Idf’ means Inverse document frequency. You can read about tdidf in the link here. After the fit is completed, the fit and matrix gets stored in a pickle file for future processing. This is accomplished by the codes in “onetime_run_file.py”.
similarity_search
This function computes cosine similarity between the user query and the movie titles stored in the tfidf_matrix created by the tfidf_fit function. First, it performs a tfidf_fit on the user query and then it computes the cosine distance between the fit and the elements in the tfidf_matrix. The top 5 elements which has the highest cosine similarity score is chosen and sent back to slack interface. This is used in the message buttons in Slack which we discussed before.
metadata_filtering
This function performs metadata based collaborative filtering which is used for movie recommendations. You can read about this function in the link here. The final “cosine_sim” function gets stored in the pickle file for future processing. Again, this is also accomplished by the codes in “onetime_run_file.py”.
get_recommendations
This function is used to get movie recommendation based on the user search. It returns 3 similar movies and sends back to slack interface. This result is used in slack tiles which we discussed before.
Now, let us see how these custom NLP solutions looks like in Slack.
Reference: https://www.datacamp.com/community/tutorials/recommender-systems-python
That’s it. This concludes this part — Custom NLP solutions. Our next step is to understand the logs processing.
As of now, we have implemented all the features that is needed to run the bot. So why do we need to talk about logs? What importance does log files bring to bot framework?
Let us see an example to illustrate its importance. Suppose we have a “feedback” intent and the example user queries in the intent include “I am satisfied with your service”, “I don’t like your service”,”You are awesome”, “You are a disaster” and so on. We only provide a few example when we develop this intent. In future, let us say a user says “boo”. We know that this should be categorized as “feedback” intent. However, our bot is not trained on this response. So it would show it as some other intent resulting in bad user experience. Also, these type of user responses come at random and we need a way to identify and capture them.
The easiest way to collect user queries is via logs. So in the future, if a user asks a question different from the way we defined our intents, we can capture those responses and include them in the intents model training. This way we improve the overall customer experience.
In this bot framework, we have two types of log processing
Log fileFollow up file
Log file
Follow up file
Log file
The output log file looks like below.
As we see here, all the information is captured which makes it easier for future user query processing. Also, it includes the processing time in the log file (second to the last column). Using, this input we can optimize queries which takes more time to process and thus improving the bot efficiency.
Follow up file
The follow up file is also generated as a part of the log file processing. If the bot is not able to assign any intent to the user query (“Anything_else” intent) then the “follow_ind” is invoked. This will write the user query in a new file called ‘followup_file.TXT’ which looks like below.
This file can be later sent to bot developers via email using the code below.
That’s it. This concludes this part — Logs processing.
I assume if you are still reading this article, you have successfully completed this exercise. Great job! Ok, so what’s next?
As I said before, we can experiment by adding new bot features, extending the NLP capabilities and also implement the bot in other messaging platforms. The central idea around this framework is — all the work done can be reused if you need to change platforms and also for easy addition/extension of skills. That being said, I cannot cover all the possible options for you as the application of bots can vary depending upon the user requirements. So it is up to the you to explore and implement them. Good luck!
If you are looking to build some AI/Machine learning based chatbots then refer these links below.
ethancaballero.pythonanywhere.com
chatbotsmagazine.com
github.com
That’s it. This concludes the final part. If you need to discuss anything in particular or you have feedback on any of the modules, please leave a comment or reach out to me via LinkedIn or Twitter.