Building a Covid-19 Virtual Assistant with Watson Services on IBM Cloud
By: Khalil Faraj
Tags: Tutorial, Watson Translator, Watson Discovery, Watson Assistant, Watson Speech To Text

You can enable the webhook by clicking on the customize button of the node. We have the country as a parameter, so we add it as the key, and its value can be detected using the “@sys-location,” which is an entity that comes out of the box with the assistant, and it identifies any location name.

Watson Discovery

With Watson Discovery, we can feed the service documents that contain more information and connect it to Watson Assistant. This feature can help show the results of any queries that are not part of the dialog. Instead, Discovery searches for the answer in the document fed to Discovery and sends the result using Search Skill. As we can see in the below example, the assistant returned multiple responses (we can modify the number of results to return), and the answer to our question is in Q13.

Search Skill Example with Discovery

You just create a Watson Discovery service and feed it an FAQ file that has the format Question/Answer. Then you can click on configure data to start annotating the document using SDU (Smart Document Understanding). We annotate the questions with the field question and the answers with the field answer or text. Make sure to split the document using the question field.

Discovery’s SDU

Once the document is ready, you can go back to the Watson assistant and configure your search skill. Choose the question field for the title, text field or question field (based on which one you chose to annotate) for the body, and the filename for the URL.

Search Skill Configuration

Web Chat Integration

To integrate this assistant in your website, you just add the following code inside your code. You can get your integration ID and service instance ID from the assistant that you created on IBM Cloud.

Watson Assistant Web Chat Integration Snippet Code

As you can see in this code, we are saving the instance because we want to use it later on when we switch the language from English to Arabic and vice-versa. As you remember, we have two different Watson Assistant, so we want to load the right assistant correspondingly when a user clicks on the change language button.

Function to Change the Assistant

The code above is the function that changes the assistant. First, we make sure to close and destroy the first instance of the assistant, because if not then we will have a stack of open assistant windows on top of each other when we want to change the language. On the language button click, we just change the ID and set it in the Assistant Chat Options to render it. Just like before, we make sure to store the instance so we can destroy it later on when we change the language again and instantiate a new one.

Voice Agent

To provide a better experience for the users, you can integrate Watson Assistant with Voice Agent. This service offers a way to combine a set of different Watson services with a public or private telephone network by using the Session Initiation Protocol (SIP) [1]. IBM Voice Agent enables direct voice interactions over a telephone with an AI self-service agent and the ability to transcribe a phone call between a caller and an agent so that the conversation can be processed with analytics for real-time agent feedback [1].

Voice-Based Assistant Flow

Voice-Based Architecture Flow

The user interacts with his mobile phone to call a number (1), then he phones a call center phone number that is associated with a Twilio account (2). The Twilio number connects to a configured Voice Agent service through the SIP communication protocol (3). Voice Agent calls the Watson Speech to Text service to transcribe the user input (4). Then, Watson Assistant responds based on the user input (5). The results from Watson Assistant are transmitted back to the Voice Gateway service using the Watson Text to Speech service and thus back to the user who will hear the response (6). The Cloud Function and Discovery parts are the same as the one discussed in the text-based architecture flow.

How Voice Agent Works

To create a Voice Agent, we need the following services:

Watson Assistant

Speech to Text

Text to Speech

Twilio (or other SIP Trunk providers supported with IBM voice agent)

We can’t use the previously created dialog skills in the text-based version for the voice agent since they contain images, maps, and a structure that won’t correctly fit with voice agent. To solve this issue, we had to create two other dialog skills customized for the voice agent, where we removed the google map feature and all the images and customized a bit their structures so it can provide a smooth experience. But all the dialog skills have the same content, and overall follow the same structure.

Twilio connects the voice agent with a telephone network using SIP, Watson Assistant handles the conversation, Watson STT (speech-to-text) transcribes what the caller is saying, and Watson TTS (text-to-speech) converts the assistant’s response to spoken voice output. Make sure to have all these services instantiated before creating a Voice Agent service.

Once you create your account on Twilio, go to the dashboard to get your trial number. You will use this number when you want to configure and test your voice agent.

Twilio’s Generated Trial Number

Setting up Voice Agent in IBM Cloud is straight forward. Once you instantiate the service, go to the Manage tab and click create agent. Choose Voice for agent type and specify a unique name for your agent. For the phone number field, add the phone number (trial number) you have in Twilio.

Configuring a Voice Agent

Fill all the information in the other sections. Here we are using IBM services, but Voice Agent is not limited just to IBM. For example, you can configure STT and TTS using services provided by Google.

We still need to create a SIP Trunk in Twilio. Go to the Getting Started tab in the Voice Agent service in IBM Cloud, and copy the primary endpoint.

Getting Voice Agent’s Endpoint

Go back to Twilio and open all the products and services tab under the dashboard. From here, choose Elastic SIP Trunking and select the Trunks Tab, then go to Origination. In the Origination URI, add the primary endpoint that you got from the Voice Agent service in IBM Cloud.

Origination URI for SIP Trunk in Twilio

Go to the Numbers tab and add the trial number provided by Twilio (the one you have in the main dashboard). Everything is set and configured, and you can dial the number and test your voice agent.

Adding Number to the SIP Trunk in Twilio

Node-RED

Node-RED is a browser-based programming tool that allows you to connect code blocks that have specific tasks and functionalities. It uses nodes and flows to quickly write code that can connect to APIs, hardware, IoT devices, or online services [2]. A node is a predefined code block and the flow is a connection of nodes, like input, processing, and output nodes [2]. Here, we are running Node-Red on IBM Cloud, and we deployed our Node-Red app in Cloud Foundry. To know more on how to get started with Node-Red check this link https://developer.ibm.com/tutorials/how-to-create-a-node-red-starter-application/

Node-Red Flow

In our Node-Red app, we are using Watson Language Translator with our Watson assistant discussed previously. It uses the English version of the assistant but will translate the responses to Urdu. You can download this flow from the Github Repo and import it in Node-Red.

Node-Red Flow for the Assistant with Translator

In this flow we have:

Light Blue Nodes: UI elements (send button, text input, and a clear chat button).

Blue Nodes: Watson Assistant and Watson Translator language services. Click on them to customize your configurations and to set the API keys correspondingly.

Yellow Nodes: used for switch conditions, or set and change some values for the flow.

Green Nodes: Display output for debugging.

Salmon Nodes: contains some code to add some functionalities (for example modify the JSON response that we get from Watson assistant regarding a query)

Turquoise Nodes: HTML tags to add additional UI elements and to enhance existing UI elements.

The user first sends his query in Urdu. The assistant we are using is in English, so before going to Watson Assistant this query is passed to the translator node, which translates it from Urdu to English so the assistant can understand it. The translated query goes to the assistant node, which gives back the response in English. This response is passed again to the translator node, which translates the result back to Urdu and finally shows it to the user. This Node-Red is using architecture flow discussed in the first section, with language translator service.

We integrated this Node-Red app with our main website. Since this Node-Red app is deployed on IBM Cloud, we take its URL and add it to our code. Once the user clicks on “Voice & Translator” on the main page of the website, he will be directed to this Node-Red app so he can try the translator feature.

Node-Red Translator App

Summary

In this blog, we shared the journey of building an advanced COVID-19 virtual Assistant using multiple technologies to help citizens with their concerns about the current situation.

To summarize, we built the assistant in two languages English, Arabic, and supported the Urdu one with translation. We used a webhook to connect the assistant to Johns Hopkins API and Search Skill to connect to Discovery. To build the voice agent, we connected the assistant to Twilio, STT, and TTS services. Finally, we used Node-RED to build the chatbot using Watson Language Translator for Urdu and connected it to the English version chatbot Assistant.

Sign up for your IBM Cloud Account and get started with building your assistant and exploring the cloud services.

GitHub Repository

Demo