Chatbots: Creating Natural Language From Structured Data
By: Cobus Greyling
Tags: NLP, Programming, Chatbots, Artificial Intelligence, Conversational UI

For user input, the chatbot must structure the data. A large part of this structuring can include the following activities:

Sentence Boundary Detection (helpful for longer input)

Language Detection (scenarios where users speak different languages)

Intent Detection

Determining Entities

and more…

Inversely, the data output to the user must be unstructured again into natural language…

Speaking To The User

After the appropriate response to the user have been determined by the chatbot, the data which needs to be presented to the user, is in a structured format.

In the case of a weather bot, the data you want to present to the user might look something like this:

{

"id": 803,

"main": "Clouds",

"description": "broken clouds",

"icon": "http://openweathermap.org/img/wn/04d@2x.png",

"weather": "Clouds",

"temp": 80,

"high": 82,

"low": 78,

"city": "New York"

}

Under normal circumstances, to present this to a user via a mobile app or website is standard procedure. With a conversational interface, it is a whole different matter.

We need to convert the data into conversation, hence unstructured it. This brings us to this continuous process of structuring and unstructuring data.

This process of unstructuring data into conversation is referred to as Natural Language Generation, NLG.

Natural language generation Natural language generation is the natural language processing task of generating natural language from a machine representation system such as a knowledge base or a logical form. Psycholinguists prefer the term language production when such formal representations are interpreted as models for mental representations.

Basics Of Natural Language Generation (NLG)

As with everything, NLG can be performed on various levels of complexity. The most simplistic approach is to have a one-to-one match of return codes and phrases.

If the API returns 0, then the bot responds with “Thank you, your request have been logged”.

Else if the API responds with 1, the bot responds with “Sorry, something went wrong, try again later.”

You could see this as a very basic form of unstructuring data.

The Illusion of Liveness

Of course, you can take this one step further, by creating an illusion of lifeness. Some development environments, like IBM Watson Assistant allows for multiple responses to be defined per conversational node.

IBM Watson Assistant — Assistant Responses

These responses can then be set to random or sequential. In the example shown here, there is a list of goodbye messages. This list of messages can be extensive, and set to be different every time the user says goodbye to the chatbot.

Hence presenting this idea to the user of an unscripted and spontaneous agent.

Scripted Language Generation

Taking matters one step further, is creating a language generation script.

Microsoft’s Bot Framework Composer has a Bot Response option on the left, where you can define the bot responses.

Language Generation Script

In the marked example, a Language Generation script is defined called:

#DescriberWeather

The purpose of this example is to take the response from the weather API, and transform it into more natural sounding language. If the API returns “Dust”, we want our chatbot dialog to return: “There’s dust in the air” etc.

We can create multiple such scripts quick and easy for different API’s, and scenarios.