Dialog Augmentation and Intent Detection
By: Aseem Srivastava
Tags: Deep Learning, Intent Classification, Chatbots, NLP, Conversational Ai

Intent Detection in Dialogue System is a decisive step to drive the conversation in dialogue system in the right direction. Often, Language Understanding part is paid less attention while building a dialogue system leading to ambiguous response generation. We will try to reconstruct the well-known problem statement of Intent Detection for minority intents and augment dialogues with similar intent in order to create better understanding of minority intents. Proposed architecture and mechanism shows good results on dialogues and can the particular augmentation mechanism could also be extended to other dialogue related classification problems. In this research it work will generate better understanding with even lesser number of intent classifiers using cutting edge machine learning models and state of the art deep learning techniques on goal oriented dialogue systems. In the era of deep learning, intent detection in conversational systems is no more a classical machine learning problem.

Google Colab notebooks, GitHub code and Dataset links are attached below for your reference.

INTUITION

Intent Detection(ID) in Conversational Dialogues is a very crucial phase while revolutionizing dialogue systems with Machine Learning. Leveraging the Natural Language Understanding(LU), Intent Detection helps in improvising the language generation and advancement in Dialogue Management. Previously, the task of intent detection has been done many times with different datasets. Here we will try to detect the indents precisely where the number of training examples is very less in number and encounter the phase where chat bots are unable to figure out the intent of the user.

For dialogue systems, especially for goal-oriented dialogue systems, a chat bot should be robust enough to detect the intent of the user without taking any particular accent into assumption. And that is where we are dealing even with a lesser number of intents classified in the dataset and bring them to correct identification of indents while in the test phase. For the purpose, we will be using the ATIS dataset released by Sonos in June 2017. We will see more about the data in the dataset section.

Paraphrase generation techniques to leverage dialogues with the minority intent and augmenting similar dialogues results in having better understanding of the dialogues in a dialog system.

Why is it even Important?

Dialogue Systems are build with the focus to complete the task or domain on which they are trained on. Generally, researchers pay less attention to the LU with Intents part to diagnose the intents with very less occurrence in the conversation. The ID with the help of various machine learning techniques would help us to create a better understanding of human intents and flow of conversation.

At the same time our output would also create a positive impact on the dialog management system to reproduce better language generation models.

According to the information extracted, the system can then decide on the appropriate actions to be taken, to help the users achieve their demands. LU applications are becoming increasingly significant in our everyday lives. Numerous devices, such as smartphones, have personal assistants that are built with LU technologies.

Multiple product support systems like help centers use ID to reduce the need for a large number of employees that copy-and-paste boring responses to frequently asked questions. Chat bots, automated email responders, answer recommenders from a knowledge base with questions and answers.

ATIS Dataset

Table 1: Above table shows example dialogues from user to the bot and corresponding annotated intents in the ATIS dataset.

The dataset used for ID task is ATIS — Airline Travel Information System which is a benchmark dataset originally released by Microsoft. In the dataset we are dealing with 8 different intents related to airline travel and flight scheduling protocols.

Figure 1: Intents used from ATIS dataset in the project and their corresponding frequency chart.

Also, ATIS dataset contains 4,978 training samples along with a vocabulary size of 943. The benchmark dataset is commonly used for slot filing problem statement too providing 129 slot counts. The part in which we are interested in is intent count which are 26 in total. For our experiment we are dealing with 5 different intents.

Modeling

Our Approach is a mix of two different components where at first dialogues with minority intent are leveraged by augmenting paraphrases of the dialogue utterances. Since these utterances belong to the same intent as that of the parent dialogue, ATIS dataset can be augmented to three times. In our work we augmented data for dialogues from 4 minor intent-classes. For paraphrase generation we used a pretrained T5 model. Secondly along with these dialogues, a neural network is trained from scratch on top of the BERT embedding.

In our approach we used the BERT based encoder part of transformer architecture combined with our implementation of feed forward neural network. The whole architecture is shown in Figure2.

As an input we used BERT embedding pretrained on unsupervised corpus. Those inputs were fed to a feed forward neural network (Figure2). This module consisted of a single drop layer followed by two feed-forward layer with hidden dimension of 512.