Building Yuki, A Level 3 Conversational AI Using Rasa 1.0 And Python For Beginners
Tags: Technology, Chatbots, AI, Tutorial, Bots

Building Yuki, A Level 3 Conversational AI Using Rasa 1.0 And Python For Beginners

Part 1: Setting up the base to program complex chatbots

Fascinated by artificial intelligence powered chatbots or natural language understanding and processing but not sure where to start at and how to build stuff in this domain?

You are at the right place. I have got you covered. Just read on and together let us unravel the simplicity of complex topics in our journey to build something marvelous. I would recommend you to have a quick read through the article once to get the overall idea, bookmark the link to this article, and then come back later to follow along with me in doing all the tasks I mention here. I have simplified the details to that extent that even if you are a middle school student, you can follow along well and learn all the aspects of bot design. All you need is the zeal and passion to build something and some basic understanding of how computers work and a little bit about programming!

But yes, this tutorial is not only for beginners. Those with considerable experience might also find it helpful to kick start their development process.

Before getting started I want you to know the main motivation in writing this one.

To give a different perspective to what building something means

To enable self-learners alien to this subject feel the ease at which something can be learned

and finally to highlight what the future holds, it’s gonna be bots everywhere.

Please note that I am writing this tutorial for a Windows 10 PC. However, a similar approach should work in Linux with very few modifications. If at any point in the article, any keyword, step, or technical term appears unclear, feel free to ask in the responses/comments and I will clarify it. With that being said, let us get started, keeping our end goal in mind.

“Can you create an AI as complex as Jarvis, if not better?”

Jarvis from Iron Man

✔️ Checkpoint 1: Understand What You Are Building

A level 3 conversational AI is basically a computer program that can understand human language along with the intent of the speaker, the context of the conversation, and can talk like one. We, humans, assimilate natural language through years of schooling and experience. Interestingly, that’s exactly what we are going to do here, build a bot, parallelling how a human baby learns to converse. At the end of this project, here are the objectives we want to achieve, to create a bot that-

can chitchat with you

can understand basic human phrases

can perform special actions

can autonomously complete a general conversation with a human

can store and retrieve information to and from a database (equivalent to its memory)

is deployed on Telegram, Facebook Messenger, Google Assistant, and Alexa, etc.

✔️ Checkpoint 2: Setting Up Your Development Environment

To build something incredible and quick, you will need great tools and easy to use frameworks. Here is my setup-

Download and install the latest versions of these in your system. It is preferred to install them with default paths and settings.

After installation, go to your search bar in desktop and type ‘Anaconda Prompt’ (terminal). Open it and create a new virtual environment for our project by typing the following command:

conda create -n bots python=3.6

This will create a virtual environment called ‘bots’ (you can create an env with whatever name you want) where you can install all the dependencies of your project without conflicting with the base. It is a good practice to use virtual environments for projects with a lot of dependencies.

Next, activate this virtual environment by typing (always ensure you activate it while makes any changes or running the program)-

activate bots

Now, install the rasa stack by typing the following command-

pip install rasa

It may take a few minutes to download all the packages and dependencies. There is a chance that you may encounter some error or warning. When you are stuck up somewhere, remember that Google has all the answers you want. Search your error message and you’ll find a solution in sites like StackOverflow.

In a similar way, you can install any Python package in your activated environment. We will be using a library called Spacy in the future. Install it now. ( -u is used to install only for the current user without admin privileges)

pip install -U spacy

Then run the following command in terminal to download a spacy model.

python -m spacy download en_core_web_sm

✔️ Checkpoint 3: Creating The Skeleton Of Our Bot

We are using the open-source rasa-stack which a very powerful, easy to understand, and highly customizable framework for creating conversation AIs (hence the choice). Before we finalize the functionalities of the bot, let us do some magic with a single line of command.

Create a new folder on your desktop and name it for your reference. I named mine as ‘Yuki’, the name I gave to my bot.

the name I gave to my bot. Now, go back to the anaconda prompt terminal and reach your project directory by typing the following command:

cd desktop\yuki

Note that the syntax is cd followed the path to YOUR folder .

followed . Next type the following command:

rasa init

You will see a prompt asking to enter the path to the project. Since we are already in the project folder (Yuki), you can just press ‘Enter’. Just follow along with the terminal. You will see that a core and an NLU model getting trained after which you should get a prompt to talk to the bot in the command line. Boom! you have just now built a bot! Talk to the bot, say a ‘hi’ maybe!

So what the hell happened just now? You basically created a rasa project with the above command which initialized a default bot in your project folder. We will be working on top of this to build our bot.

✔️ Checkpoint 4: Understanding The Structure Of Your Bot

Now, open the new folder you created before. You will see so many new files in it. This is your project structure. Here is a brief on each of them:

Data Folder: Here lies your training data. You will find two files, nlu.md (training data to make the bot understand the human language) and stories.md (data to help the bot understand how to reply and what actions to take in a conversation).

Here lies your training data. You will find two files, (training data to make the bot understand the human language) and (data to help the bot understand how to reply and what actions to take in a conversation). Models Folder: Your trained models are saved in this folder.

Your trained models are saved in this folder. actions.py: This is a python file where we will define all the custom functions that will help the bot achieve its tasks.

This is a python file where we will define all the custom functions that will help the bot achieve its tasks. config.yml: We mention the details of a few configuration parameters in this file. I’ll get back to this later.

We mention the details of a few configuration parameters in this file. I’ll get back to this later. credentials.yml: To deploy the bot on platforms like Telegram or Facebook Messenger , you will need some access tokens and keys. All of them are stored in this file.

To deploy the bot on platforms like or , you will need some access tokens and keys. All of them are stored in this file. domain.yml: One of the most important files, this is like an index containing details of everything the bot is capable of. We will see more about it later.

One of the most important files, this is like an index containing details of everything the bot is capable of. We will see more about it later. endpoints.yml: Here, we have the URL to the place where the actions file is running.

Once you are clear with the file structure, read ahead.

So, how exactly does it work?

Rasa has many inbuilt features that take care of a lot of miscellaneous things allowing you to focus more on the practical design of the bot from an application point of view. Don’t think too much about the inherent technical aspect at this moment. Follow along carefully, and you will find yourself building a complex AI at the end of this tutorial.

✔️ Checkpoint 5: Designing The User Experience (UX)

Before you go any further, you need to decide the purpose of your bot. What will your bot be used for? Here are some generic ideas:

Weather bot that can tell the user about various weather parameters in the required area.

A Wikipedia based bot that can answer to any kind of general knowledge-based questions.

A Twitter-based bot, that can update the user with what’s trending right now in the location of interest.

A generic search bot that can search about something (Eg: Jobs) based on user’s queries.

From a commercial standpoint, an ordering/booking bot using which the users can purchase goods like clothes, order food, or book tickets to movies, travel tickets, schedule appointments with professionals like doctors, lawyers, etc.

In short, the possibilities are limited by your own imagination. Think of JARVIS or EDITH from the MCU. It is even possible to build something like that. Too many choices? I will design my bot here, which you can try to replicate, but I hope you can be a little more creative to build upon your own ideas. Whatever idea you have, the steps will be similar.

As for Yuki, I will be demonstrating almost everything that is possible, with the end goal of creating an all-purpose conversational AI in a series of tutorials.

In my first design, I want to demonstrate these two things:

How to use APIs (this stuff is like magic, super useful and easy to use)

Playing with custom functions

Here’s what I want my bot to be capable of for now:

Fetch the latest news or articles on the internet based on users topic of interest or search query

Capability to handle the simple general conversations

Now I will list down all the functions/actions that the bot needs, to achieve its capabilities;

utter_hello

handle_out_of_scope

utter_end

get_news (and so on…)

Make a brief list like this for your bot. Next, I made a list of human intents the bot will have to detect from the messages users send it.

Intents

greet

bye

getNews

affirm

deny

thank_you

out_of_scope (Everything else for which the bot is not programmed yet for must be detected as out of scope)

All this data goes into the domain.yml file. So, here’s how I expect my ideal user to interact with Yuki:

User : Hi!

: Hi! Yuki : Hola! I am Yuki. What’s up?

: Hola! I am Yuki. What’s up? User : Get me the latest news updates

: Get me the latest news updates Yuki : Give me a topic/keyword on which you would like to know the latest updates.

: Give me a topic/keyword on which you would like to know the latest updates. User : <enters the topic name>

: <enters the topic name> Yuki: Here’s something I found.

<Links to news articles fetched>

Yuki : Hope, you found what you were looking for.

: Hope, you found what you were looking for. User : Thanks!

: Thanks! Yuki: You're welcome.

This is generally referred to as a ‘Happy Path’, the ideal expected scenario that happens when a user interacts with the bot. Whatever bot you want to design, have an idea of the basic conversational flow it is expected to handle, like this. Write down multiple flows with different possibilities for your reference. The above flow (along with any minor variations to it)is the basic expectation we want to achieve.

Apart from this, we are going to teach Yuki how to respond to chitchat questions like ‘how are you?’, ‘what’s up’, ‘who made you?’, etc.

Once you are ready, move to the next checkpoint.

✔️ Checkpoint 6: Building Your NLU Model

First, let us teach our bot some human language, to identify the intents. Start Visual Studio Code, click ‘File->Open Folder’ and choose the project folder (in my case, Yuki).

Open the data/nlu.md file in your code editor. You will already see some default intents in it. This is the place where you add data about every intent the bot is expected to understand and the text messages that correspond to that intent. Update this file with all the required intents in the same format. My finished nlu.md file will look like this:

## intent:affirm

- yes

- indeed

- of course

- that sounds good

- correct

- alright

- true



## intent:bye

- bye

- goodbye

- see you around

- see you later

- ttyl

- bye then

- let us end this



## intent:chitchat_general

- whats up?

- how you doing?

- what you doing now?

- you bored?



## intent:chitchat_identity

- what are you?

- who are you?

- tell me more about yourself

- you human?

- you are an AI?

- why do you exist?

- why are you yuki?



## intent:deny

- no

- never

- I don't think so

- don't like that

- no way

- not really

- nope

- not this

- False



## intent:getNews

- Send me latest news updates

- I want to read some news

- give me current affairs

- some current affairs pls

- Find some interesting news

- News please

- Get me latest updates in [science](topic_news)

- latest updates in [sports](topic_news)

- whats the latest news in [business](topic_news)

- send news updates

- Fetch some news

- get news

- whats happening in this world

- tell me something about whats happening around

- interesting news pls

- latest updates in [blockchain](topic_news)

- get me latest updates in [astronomy](topic_news)

- any interesting updates in [physics](topic_news)

- I want to read something interesting

- I want to read news

- latest news about [machine learning](topic_news)

- latest updates about [Taylor Swift](topic_news)



## intent:greet

- hey

- hello

- hi

- good morning

- good evening

- hey there

- hola

- hi there

- hi hi

- good afternoon

- hey

- hi



## intent:thank_you

- thanks!

- thank you

As it is directly evident here, you are basically giving your bot the data to make it understand what words imply which intent of the user. These are some fundamental human intents. You can add more if you want. Also, notice how I divided the chitchat intent into chitchat_general and chitchat_identity for more specificity. To create a robust bot, be as specific as you can with your intents.

Also, notice how I have placed some words in [] followed by () . These are the entities or the keywords with some significance in users’ text. To help the bot understand the same, this general syntax is used.

And it’s as simple as that. You are done with NLU! Rasa will take care of the training part for you. Let’s now move on to the next checkpoint.

✔️ Checkpoint 7: Updating The Domain File

The domain.yml file must contain the list of all the following stuff:

Actions: These include the names of all the custom functions we implement in actions.py as well as the ‘utter actions’

These include the names of all the custom functions we implement in as well as the Entities: These are specific keywords present in user input which the bot can extract and save it to its memory for future use. For now, my bot requires only one entity which I named ‘ topic_news ’. This basically refers to whatever topic the user seeks the news.

These are specific keywords present in user input which the bot can extract and save it to its memory for future use. For now, my bot requires only one entity which I named ‘ ’. This basically refers to whatever topic the user seeks the news. Decide upon what entities you need to extract based on your use case . For example, the Name of a person if you want to create a user profile internally, a geographical location, if you want to give the weather update, food items if your bot can order food online, email ids to subscribe the user to something, etc.

. For example, the of a person if you want to create a user profile internally, a geographical location, if you want to give the weather update, food items if your bot can order food online, email ids to subscribe the user to something, etc. Intents: We have already discussed intents. In the domain file, you’ll have to include the list of the same.

We have already discussed intents. In the domain file, you’ll have to include the list of the same. Forms: The form action is a special feature provided by rasa stack to help the bot handle situations where it needs to ask the user multiple questions to acquire some information easily. For example, if your bot has the ability to book movie tickets, it must first know details like the name of the movie, the showtime, etc. While you can program the bot to react according to user context, using forms is highly efficient in such situations. In my case, I have a getNews form action which we will explore further on how it can be implemented. The names of all the form actions you implement must be put up in the domain file.

The is a special feature provided by rasa stack to help the bot handle situations where it needs to ask the user multiple questions to acquire some information easily. For example, if your bot has the ability to book movie tickets, it must first know details like the name of the movie, the showtime, etc. While you can program the bot to react according to user context, using forms is highly efficient in such situations. In my case, I have a which we will explore further on how it can be implemented. The names of all the form actions you implement must be put up in the domain file. Slots: In short, the entities and other internal variables that the Bot needs are stored in its memory as slots.

In short, the entities and other internal variables that the Bot needs are stored in its memory as slots. Templates: These are like the blueprints of all utter actions. Utter actions need not be implemented anywhere else separately. But remember, they must be named in utter_<sample> format. You are essentially teaching the bot, the language, the text that it must use when it wants to send some message. You can have multiple text templates for each utter action.

These are like the blueprints of all utter actions. Utter actions need not be implemented anywhere else separately. But remember, they must be named in format. You are essentially teaching the bot, the language, the text that it must use when it wants to send some message. You can have multiple text templates for each utter action. Check out my domain.yml file for reference, to understand how you should update your domain file with respect to your bot design.

domain.yml

There are so many customizations and variations possible. The format you see here is given by Rasa, and includes keywords that make use of its features like ‘buttons’. As you can see, your domain file contains a summary of everything the bot can do.

Up to this point, we have hardly written a line of proper code, and we are about halfway done with the project! Notice how much ideation and brainstorming would go into building a high-level application even before you start coding. That’s the important part, the use case, and design. Anyone can write lines of code, but only a few can create brilliant products.

Remember that a good programmer isn’t always a great product developer and a great product developer isn’t always required to be a good programmer.

Highly functional AIs can have 100s of actions, intents, entities and be connected to huge databases. We are now taking the first steps towards this direction.