Training a Goal-Oriented Chatbot with Deep Reinforcement Learning — Part I
By: Max Brenner
Tags: Machine Learning, Reinforcement Learning, Deep Learning, Chatbots

What is a Goal-Oriented Chatbot?

A goal-oriented (GO) chatbot attempts to solve a specific problem for a user. These chatbots can help people book a ticket, find a reservation, etc. There are two main ways to train a GO chatbot: Supervised learning with an encoder-decoder that directly maps user dialogue to responses and reinforcement learning which trains a chatbot through trial-and-error conversations with either real users or a rule-based user simulator. GO chatbots trained through deep reinforcement learning is a very exciting and a ripe research field with numerous practical applications!

Dialogue System

The dialogue system for a GO chatbot using reinforcement learning is split into 3 main parts: The Dialogue Manager (DM), Natural Language Understanding (NLU) unit and Natural Language Generator (NLG) unit. Furthermore, the DM is split into the Dialogue State Tracker (DST) or just State Tracker (ST) and the policy for the agent itself, which is represented by a neural network in many cases. In addition the system loop contains a user with a user goal. A user goal represents what the user wishes to get out of the conversation which in the case of the diagram below is a restaurant reservation.

Dialogue flow for a GO chatbot system

In this loop, the user utters something which is processed by the NLU component into what’s known as a semantic frame which is a lower-level representation of a natural language utterance that can be processed by the agent. The DST processes the user dialogue act (semantic frame) and the history of the current conversation into a state representation that can be used by the agent’s policy. This state is then fed as input into the policy or neural network of the agent and outcomes an action in the form of a semantic frame. A database (DB) can also be queried to add information to the agent action such as restaurant info or movie ticket info. The agent (also known as “system” in this diagram) action is then processed by the NLG component which converts it to natural language for the user to read.

MiuLab TC-Bot

Dialogue flow for TC-Bot

This tutorial and accompanying code is based off a dialogue system by MiuLab called TC-Bot. The main contribution of their paper is that it shows how to simulate a user using basic rules so that the agent can be trained with reinforcement learning very quickly, compared to training an agent with real people. Other papers have done this as well but this paper stands out as a good example (with code!) of how to make a successful training system.

User Simulator and Error Model Controller

We will go into more detail on the user sim later in the series but for now understand that it is a deterministic rule-based simulator that attempts to model a real user. In this case it is based on user agenda modeling which means it uses an internal state that represents the constraints and needs of the user sim. This internal state keeps track of the current conversation as well as what it still needs to do to complete its current goal. The goal is picked randomly from a list of available user goals where a goal consists of a set of constraints and other information that guides the actions of the user sim as it attempts to fulfill its current goal. The error model controller (EMC) is used to add error to the user sim’s action at the level of the semantic frame which was shown to improve the results of training.

What you Should Know for this Tutorial Series

Here is a list of things that you should know that will not be covered in this series but are important to be able to understand the code:

Know how to code in python pretty well

Know how to work with dictionaries in python as we will be using them a lot!

Know how to code a simple DQN

Know how to make a very simple neural network model with Keras

We will be coding in Python ≥ 3.5, Keras (any recent version) and of course numpy.