Beginner’s guide to creating a powerful chatbot | by Louis Teo | Towards Data Science

Are you facing too many standard requests and questions from customers and struggling to cope with them? Are you looking for a way to expand your customer service without incurring much costs?
In my previous story, I shared how I tackled data overburden issues by creating a personal text summarizer to summarize a document.
In this story, I will show you how you can easily create a powerful chatbot to handle your growing customer requests and inquiries. I will also show you how to deploy your chatbot to a web application using Flask.
The Covid-19 pandemic has hit the world hard. Many businesses have suffered major losses due to lockdown / movement controls. To ride through this tough time, many were forced to move their businesses online.
One of the main, common problems faced by online business owners is having to respond to an overwhelming number of questions and requests from customers. For those with limited manpower resources, it’s impossible to deal with all requests in time.
To solve this problem, many business owners have turned to using chatbot for their customer service.
What is a chatbot: a chatbot is AI-powered intelligent software that is capable of having conversations with human and performing human-like tasks. Chatbots are present in many smart devices (e.g. Siri (iOS), Google Assistant (Android), Cortana (Microsoft), Alexa (Amazon)), websites and applications.
Why use a chatbot: According to a research carried out by HubSpot, 71% of users were willing to use quick-to-respond chat apps to receive customer assistance, and many do so because they want their problems to be solved FAST (and efficiently, of course). A chatbot, if configured intelligently, indeed can unlock great values for businesses by allowing manpower resources to be focused on critical operations while maintaining the same level of customer satisfaction.
I will divide this into three major sections — but know that you can go directly to the section that is of your interest:
(a) Building a chatbot using Jupyter notebook
(b) Running chatbot in terminal
(c) Deploying chatbot as web app using Flask
Thanks to the ChatterBot library in Python, creating a chatbot is no longer a daunting machine learning task that it used to be.
Now, let’s get our hands dirty...
(1) Install the ChatterBot library
We will begin by installing the ChatterBot library. Installation commands for terminal are as follows:
pip install chatterbot
The ChatterBot text corpus (language resource consisting of a large and structured set of texts) is distributed in its own Python package, so you need to install it separately:
pip install chatterbot_corpus
If you have not installed spaCy (an open source library for advanced Natural Language Processing) before, then please install it now because the ChatterBot library needs spaCy library to work:
pip install spacy
Install spaCy English (‘en’) model after installing the spaCy library:
python -m spacy download en
(2) Create chatbot instance
We will develop our chatbot using Jupyter Notebook before we package the entire chatbot to Python scripts. Let’s begin by importing the module we need:
from chatterbot import ChatBot
We will create a chatbot instance, name our bot as Buddy and specify read_only parameter to True because we only want our chatbot to learn from our training data.
By creating a chatbot instance, a chatbot database named db.sqlite3 will be created for you.
bot = ChatBot('Buddy', read_only = True)
(3) Train the chatbot
It’s time to train our chatbot... (What, just like that? Yes — it’s no joke!)
The training simply means feeding conversations into the chatbot database. See the example below:
As soon as the chatbot is fed with ‘conversation 1’ and ‘conversation 2’, the chatbot will store the conversations in its ‘knowledge graph’ database in the correct order.
After that, you can initiate a conversation using any of the statements above.
For example, you can start a conversation with statement 6 (“ What payment methods do you accept?”), and the chatbot will respond with statement 7 (“We accept debit cards and major credit cards).
With this concept in mind, let’s train our chatbot with ‘conversation 1’ and ‘conversation 2’.
We do so by importing the ListTrainer module, instantiating it by passing the chatbot object (Buddy) and calling the train() method to pass a list of sentences.
from chatterbot.trainers import ListTrainertrainer = ListTrainer(bot)trainer.train([    "Hi, can I help you",    "Who are you?",    "I am your virtual assistant. Ask me any questions...",    "Where do you operate?",    "We operate from Singapore",    "What payment methods do you accept?",    "We accept debit cards and major credit cards",    "I would like to speak to your customer service agent",    "please call +65 3333 3333. Our operating hours are from 9am to 5pm, Monday to Friday"    ])trainer.train([    "What payment methods do you offer?",    "We accept debit cards and major credit cards",    "How to contact customer service agent",    "please call +65 3333 3333. Our operating hours are from 9am to 5pm, Monday to Friday"    ])
Chatbot Testing
To check if we have fed our conversations correctly to our chatbot, let’s test it out by feeding the chatbot with just a simple input statement - ‘payment method’.
response = bot.get_response ('payment method')print(response)
The chatbot responded to us with:
This response is exactly what we want.
We will create a while loop to enable our chatbot to respond to each of our queries continuously. We will put an end to the loop and stop the program when we get ‘Bye’ or ‘bye’ statement from the user.
name = input('Enter Your Name: ')print ('Welcome to Chatbot Service! Let me know how can I help you')while True:        request = input(name+':')        if request=="Bye" or request=='bye':        print('Bot: Bye')        break    else:        response=bot.get_response(request)        print('Bot: ', response)
Let’s try it out...
Congratulations! You have just created your very first, working chatbot!
Corpus data training
Of course you would want your chatbot to be able to have more conversations on top of those that we just fed in (!) — in that case, we need to train our chatbot further.
Fortunately, this task has been simplified for us. We can quickly train our chatbot to communicate in a more ‘human-like’ and smarter way with us, by using the available English corpus data in the package.
Note: If you face problems in running the corpus training, please copy this chatterbot_corpus folder to the file directory specified in the error message.
Just run the codes below, and your chatbot will be trained to have conversations in the following scope: AI, botprofile, computers, conversations, emotion, food, gossip, greetings, health, history, humor, literature, money, movies, politics, psychology, science, sports & trivia.
from chatterbot.trainers import ChatterBotCorpusTrainertrainer = ChatterBotCorpusTrainer(bot)trainer.train('chatterbot.corpus.english')
Let’s test if our chatbot has become smarter now....
Yes! It’s become smarter — it can tell you a joke now...
Preprocess input
ChatterBot comes with several built-in preprocessors that allow us to clean our input statement, before we get the statement processed by the bot’s logic adapter.
Cleaning makes our input statement more readable and analyzable by our chatbot. It removes ‘noise’ that can interfere with the text analysis from the input statement - e.g. additional space, Unicode characters and escaped html characters.
We will now include the preprocessors in our chatbot instance and rerun the chatbot instance with the codes below.
bot = ChatBot('Buddy',             read_only = True,             preprocessors=['chatterbot.preprocessors.clean_whitespace',                            'chatterbot.preprocessors.unescape_html',                          'chatterbot.preprocessors.convert_to_ascii'])
Then, rerun the chatbot, and you can see we get the same response for who are you and whó are yóu.
Low confidence response
On top of that, we can also configure our chatbot to inform users if the input is not understood, with a default response:
‘I am sorry, I do not understand. I am still learning. Please contact abc@xxx.com for further assistance.’
Let’s include the low confidence response to our chatbot instance and rerun the chatbot instance.
bot = ChatBot('Buddy',             logic_adapters = [                 {                     'import_path': 'chatterbot.logic.BestMatch',                     'default_response': 'I am sorry, I do not understand. I am still learning. Please contact abc@xxx.com for further assistance.',                     'maximum_similarity_threshold': 0.90                 }             ],             read_only = True,             preprocessors=['chatterbot.preprocessors.clean_whitespace','chatterbot.preprocessors.unescape_html','chatterbot.preprocessors.convert_to_ascii'])
Then, rerun the chatbot, and let’s try to get a response from an unexpected input — our chatbot will reply with the default_response when it doesn’t understand a statement.
Congratulations! You have successfully built a chatbot using Jupyter notebook.
Here is the link to my Github for the Jupyter Notebook on this.
Note: If you have come straight to this section (and skipped section (a) Building a chatbot using Jupyter notebook), then please ensure you have installed all the required library and packages mentioned in that section before proceeding to run the chatbot in terminal.
We will use the Jupyter notebook above to write scripts that run the steps above and perform the chatbot training based on the conversation files specified by the user, and then see our chatbot in action...!
You can download my complete scripts here which you can use right away to train and run your chatbot!
Create a training_data folder and store all the conversations you want to train in text file(s). The chatbot_training.py script will read all the text files in the training_data folder.
Create a training_data folder and store all the conversations you want to train in text file(s). The chatbot_training.py script will read all the text files in the training_data folder.
The conversation in the text file should be inputted as one sentence per line:
2. Run chatbot_training.py. You will be asked to choose if you want to train the chatbot with English corpus data — select Y or N.
Select Y — your chatbot will be trained to have conversations in the following scope: AI, botprofile, computers, conversations, emotion, food, gossip, greetings, health, history, humor, literature, money, movies, politics, psychology, science, sports & trivia.
3. Then, run the chatbot.py to launch the chatbot. Input some conversations and test if it responds properly.
Tadaa... there you go! You have completed your chatbot training and run it in the terminal.
Note: If you have come straight this section (and skipped section (a) Building a chatbot using Jupyter notebook), then please ensure you have installed all the required library and packages mentioned in that section before proceeding to deploy chatbot to web app using Flask.
We deploy our chatbot to a web application so that it can be used by customers.
To run our chatbot on a web application, we need to find a way for our application to receive incoming data and to return data. We can do this in any way we want — HTTP requests, web sockets etc.
There are a few existing examples available on the ChatterBot FAQ page that show us how to do this. I will show you how to deploy a web application using Flask.
Download the example codes from my Github, edit the index.html file in template folder and style.css file in static folder to your liking. Leave the files as they are if you just want to test the chatbot on a web application.
After that, let’s run web_app.py.
This deploys our chatbot to a local server at http://127.0.0.1:5000/
Launch your web browser and go to the URL above to launch your chatbot.
Congratulations (again)! You’ve successfully built your first business chatbot and deployed it to a web application using Flask. The chatbot, I should hope, did a pretty good job in answering some standard business questions you’ve had it trained on.
One of the things you can do to further improve the performance of your chatbot, is compile a list of FAQs that have been posted by your customers to date, provide answers to the FAQs and then train them on your chatbot.
Why some chatbots failed to live up to expectations? Some chatbots failed simply because the standard questions and requests made to a business had not been analysed sufficiently, and as a result the chatbot did not receive the training it required. Training and improving your chatbot is a continuous process in the beginning and is similar to how humans learn new skills and knowledge. And once the skills are learned, they’re built in the chatbot and the chatbot does not need to be retrained, unless and until your business grows.
Next, you can look into deploying your chatbot to a Platform-as-a-Service (PaaS) of your choice, which can host and run your web application entirely from the cloud. One of the popular free PaaS’ that you can consider is Heroku.
Thank you for reading this story! Follow me on medium for more tips on how to grow your business using DIY machine learning.