"Wait, I'm Still Talking!"Predicting the Dialogue Interaction Behavior Using Imagine-Then-Arbitrate Model

Abstract: Producing natural and accurate responses like human beings is the ultimate goal of intelligent dialogue agents. So far, most of the past works concentrate on selecting or generating one pertinent and fluent response according to current query and its context. These models work on a one-to-one environment, making one response to one utterance each round. However, in real human-human conversations, human often sequentially sends several short messages for readability instead of a long message in one turn. Thus messages will not end with an explicit ending signal, which is crucial for agents to decide when to reply. So the first step for an intelligent dialogue agent is not replying but deciding if it should reply at the moment. To address this issue, in this paper, we propose a novel Imagine-then-Arbitrate (ITA) neural dialogue model to help the agent decide whether to wait or to make a response directly. Our method has two imaginator modules and an arbitrator module. The two imaginators will learn the agent's and user's speaking style respectively, generate possible utterances as the input of the arbitrator, combining with dialogue history. And the arbitrator decides whether to wait or to make a response to the user directly. To verify the performance and effectiveness of our method, we prepared two dialogue datasets and compared our approach with several popular models. Experimental results show that our model performs well on addressing ending prediction issue and outperforms baseline models.

All species are unique, but languages make humans uniquest BIBREF0. Dialogues, especially spoken and written dialogues, are fundamental communication mechanisms for human beings. In real life, tons of businesses and entertainments are done via dialogues. This makes it significant and valuable to build an intelligent dialogue product. So far there are quite a few business applications of dialogue techniques, e.g. personal assistant, intelligent customer service and chitchat companion.
The quality of response is always the most important metric for dialogue agent, targeted by most existing work and models searching the best response. Some works incorporate knowledge BIBREF1, BIBREF2 to improve the success rate of task-oriented dialogue models, while some others BIBREF3 solve the rare words problem and make response more fluent and informative.
Despite the heated competition of models, however, the pace of interaction is also important for human-computer dialogue agent, which has drawn less or no attention. Figure FIGREF1 shows a typical dialogue fragment in an instant message program. A user is asking the service about the schedule of the theater. The user firstly says hello (U11) followed by demand description (U12), and then asks for suggested arrangement (U13), each of which is sent as a single message in one turn. The agent doesn't answer (A2) until the user finishes his description and throws his question. The user then makes a decision (U21) and asks a new question (U22). And then the agent replies with (A3). It's quite normal and natural that the user sends several messages in one turn and the agent waits until the user finished his last message, otherwise the pace of the conversation will be messed up. However, existing dialogue agents can not handle well when faced with this scenario and will reply to every utterance received immediately.
There are two issues when applying existing dialogue agents to real life conversation. Firstly, when user sends a short utterance as the start of a conversation, the agent has to make a decision to avoid generating bad responses based on semantically incomplete utterance. Secondly, dialogue agent cutting in the conversation at an unreasonable time could confuse user and mess up the pace of conversation, leading to nonsense interactions.
To address these two issues, in this paper, we propose a novel Imagine-then-Arbitrate (ITA) neural dialogue model to recognize if it is the appropriate moment for agent to reply when agent receives a message from the user. In our method, we have two imaginator modules and an arbitrator module. Imaginators will learn both of the agent's and user's speaking styles respectively. The arbitrator will use the dialogue history and the imagined future utterances generated by the two imaginators to decide whether the agent should wait user or make a response directly.
In summary, this paper makes the following contributions:
We first addressed an interaction problem, whether the dialogue model should wait for the end of the utterance or make a response directly in order to simulate real life conversation and tried several popular baseline models to solve it.
We proposed a novel Imagine-then-Arbitrate (ITA) neural dialogue model to solve the problem mentioned above, based on both of the historical conversation information and the predicted future possible utterances.
We modified two popular dialogue datasets to simulate the real human dialogue interaction behavior.
Experimental results demonstrate that our model performs well on addressing ending prediction issue and the proposed imaginator modules can significantly help arbitrator outperform baseline models.
Creating a perfect artificial human-computer dialogue system is always a ultimate goal of natural language processing. In recent years, deep learning has become a basic technique in dialogue system. Lots of work has investigated on applying neural networks to dialogue system's components or end-to-end dialogue frameworks BIBREF4, BIBREF5. The advantage of deep learning is its ability to leverage large amount of data from internet, sensors, etc. The big conversation data and deep learning techniques like SEQ2SEQ BIBREF6 and attention mechanism BIBREF7 help the model understand the utterances, retrieve background knowledge and generate responses.
Though end-to-end methods play a more and more important role in dialogue system, the text classification modules BIBREF8, BIBREF9 remains very useful in many problems like emotion recognition BIBREF10, gender recognition BIBREF11, verbal intelligence, etc. There have been several widely used text classification methods proposed, e.g. Recurrent Neural Networks (RNNs) and CNNs. Typically RNN is trained to recognize patterns across time, while CNN learns to recognize patterns across space. BIBREF12 proposed TextCNNs trained on top of pre-trained word vectors for sentence-level classification tasks, and achieved excellent results on multiple benchmarks.
Besides RNNs and CNNs, BIBREF13 proposed a new network architecture called Transformer, based solely on attention mechanism and obtained promising performance on many NLP tasks. To make the best use of unlabeled data, BIBREF14 introduced a new language representation model called BERT based on transformer and obtained state-of-the-art results.
Different from retrieval method, Natural Language Generation (NLG) tries converting a communication goal, selected by the dialogue manager, into a natural language form. It reflects the naturalness of a dialogue system, and thus the user experience. Traditional template or rule-based approach mainly contains a set of templates, rules, and hand-craft heuristics designed by domain experts. This makes it labor-intensive yet rigid, motivating researchers to find more data-driven approaches BIBREF15, BIBREF2 that aim to optimize a generation module from corpora, one of which, Semantically Controlled LSTM (SC-LSTM) BIBREF16, a variant of LSTM BIBREF17, gives a semantic control on language generation with an extra component.
In this section we will describe the task by taking a scenario and then define the task formally.
As shown in Figure FIGREF1, we have two participants in a conversation. One is the dialogue agent, and the other is a real human user. The agent's behavior is similar to most chatbots, except that it doesn't reply on every sentence received. Instead, this agent will judge to find the right time to reply.
Our problem is formulated as follows. There is a conversation history represented as a sequence of utterances: $X = \lbrace x_1, x_2, ..., x_m\rbrace $, where each utterance $x_i$ itself is a sequence of words $x_{i_1}, x_{i_2}, x_{i_3}...x_{i_n}$. Besides, each utterance has some additional tags:
turn tags $t_0, t_1, t_2 ... t_k$ to show which turn this utterance is in the whole conversation.
speakers' identification tags $agent$ or $user$ to show who sends this utterance.
subturn tags ${st}_0, {st}_1, {st}_2 ... {st}_j$ for user to indicate which subturn an utterance $t_i$is in. Note that an utterance will be labelled as ${st}_0$ even if it doesn't have one.
Now, given a dialogue history $X$ and tags $T$, the goal of the model is to predict a label $Y \in \lbrace 0,1\rbrace $, the action the agent would take, where $Y = 0$ means the agent will wait the user for next message, and $Y = 1$ means the agent will reply immediately. Formally we are going to maximize following probability:
Basically, the task can be simplified as a simple text classification problem. However, traditional classification models only use the dialogue history $X$ and predict ground truth label. The ground truth label actually ignores all context information in the next utterance. To make the best use of training data, we propose a novel Imagine-then-Arbitrate (ITA) model taking $X$, ground truth label, and the future possible $X^{\prime }$ into consideration. In this section, we will describe the architecture of our model and how it works in detail.
An imaginator is a natural language generator generating next sentence given the dialogue history. There are two imaginators in our method, agent's imaginator and user's imaginator. The goal of the two imaginators are to learn the agent’s and user’s speaking style respectively and generate possible future utterances.
As shown in Figure FIGREF7 (a), imaginator itself is a sequence generation model. We use one-hot embedding to convert all words and relative tags, e.g. turn tags and place holders, to one-hot vectors $w_n \in \textbf {R}^V$, where $V$ is the length of vocabulary list. Then we extend each word $x_{i_j}$ in utterance $x_i$ by concatenating the token itself with turn tag, identity tag and subturn tag. We adopt SEQ2SEQ as the basic architecture and LSTMs as the encoder and decoder networks. LSTMs will encode each extended word $w_t$ as a continuous vector $h_t$ at each time step $t$. The process can be formulated as following:
where $e(w_t)$ is the embedding of the extended word $w_t$, $W_f$, $U_f$, $W_i$, $U_i$, $W_o$, $U_o$, $W_g$, $U_g$ and $b$ are learnt parameters.
Though trained on the same dataset, the two imaginators learn different roles independently. So in the same piece of dialogue, we split it into different samples for different imaginators. For example, as shown in Figure FIGREF1 and FIGREF7 (a), we use utterance (A1, U11, U12) as dialogue history input and U13 as ground truth to train the user imaginator and use utterance (A1, U11, U12, U13) as dialogue history and A2 as ground truth to train the agent imaginator.
During training, the encoder runs as equation DISPLAY_FORM15, and the decoder is the same structured LSTMs but $h_t$ will be fed to a Softmax with $W_{v} \in {\textbf {R}^{h \times V}}, b_{v} \in {\textbf {R}^\textbf {V}}$, which will produce a probability distribution $p_{t}$ over all words, formally:
the decoder at time step t will select the highest word in $p_{t}$, and our imaginator's loss is the sum of the negative log likelihood of the correct word at each step as follows:
where $N$ is the length of the generated sentence. During inference, we also apply beam search to improve the generation performance.
Finally, the trained agent imaginator and user imaginator are obtained.
The arbitrator module is fundamentally a text classifier. However, in this task, we make the module maximally utilize both dialogue history and ground truth's semantic information. So we turned the problem of maximizing $Y$ from $X$ in equation (DISPLAY_FORM13) to:
where $\textbf {IG}_{agent}$ and $\textbf {IG}_{user}$ are the trained agent imaginator and user imaginator respectively, and $R^{\prime }$ is a selection indicator where $R^{\prime } = 1$ means selecting $R_{agent}$ whereas 0 means selecting $R_{user}$. And Thus we (1) introduce the generation ground truth semantic information and future possible predicted utterances (2) turn the label prediction problem into a response selection problem.
We adopt several architectures like Bi-GRUs, TextCNNs and BERT as the basis of arbitrator module. We will show how to build an arbitrator by taking TextCNNs as an example.
As is shown in Figure FIGREF7, the three CNNs with same structure take the inferred responses $R_{agent}$, $R_{user}$ and dialogue history $X$, tags $T$. For each raw word sequence $x_1,...,x_n$, we embed each word as one-hot vector $w_{i} \in \textbf {R}^V$. By looking up a word embedding matrix $E \in \textbf {R}^{V \times d}$, the input text is represented as an input matrix $Q \in \textbf {R}^{l \times d}$, where $l$ is the length of sequence of words and $d$ is the dimension of word embedding features. The matrix is then fed into a convolution layer where a filter $\textbf {w} \in \textbf {R}^{k \times d}$ is applied:
where $Q_{i:i+k-1}$ is the window of token representation and the function $f$ is $ReLU$, $W$ and $b$ are learnt parameters. Applying this filter to $m$ possible $Q_{i:i+k-1}$ obtains a feature map:
where $\textbf {c} \in \textbf {R}^{l-k+1}$ for $m$ filters. And we use $j \in \textbf {R} $ different size of filters in parallel in the same convolution layer. This means we will have $m_1, m_2, \dots , m_j$ windows at the same time, so formally:
, then we apply max-over-time pooling operation to capture the most important feature:
, and thus we get the final feature map of the input sequence.
We apply same CNNs to get the feature maps of $X$, $R_{agent}$ and $R_{user}$:
where function TextCNNs() follows as equations from DISPLAY_FORM20 to DISPLAY_FORM23. Then we will have two possible dialogue paths, $X$ with $R_{agent}$ and $X$ with $R_{user}$, representations $D_{agent}$ and $D_{user}$:
And then, the arbitrator will calculate the probability of the two possible dialogue paths:
Through learnt parameters $W_{4}$ and $b_{4}$, we will get a two-dimensional probability distribution $P$, in which the most reasonable response has the max probability. This also indicates whether the agent should wait or not.
And the total loss function of the whole attribution module will be negative log likelihood of the probability of choosing the correct action:
where $N$ is the number of samples and $Y_{i}$ is the ground truth label of i-th sample.
The arbitrator module based on Bi-GRU and BERT is implemented similar to TextCNNs.
As the proposed approach mainly concentrates on the interaction of human-computer, we select and modify two very different style datasets to test the performance of our method. One is a task-oriented dialogue dataset MultiWoz 2.0 and the other is a chitchat dataset DailyDialogue . Both datasets are collected from human-to-human conversations. We evaluate and compare the results with the baseline methods in multiple dimensions. Table TABREF28 shows the statistics of datasets.
MultiWOZ 2.0 BIBREF18. MultiDomain Wizard-of-Oz dataset (MultiWOZ) is a fully-labeled collection of human-human written conversations. Compared with previous task-oriented dialogue datasets, e.g. DSTC 2 BIBREF19 and KVR BIBREF20, it is a much larger multi-turn conversational corpus and across serveral domains and topics: It is at least one order of magnitude larger than all previous annotated task-oriented corpora, with dialogues spanning across several domains and topics.
DailyDialogue BIBREF21. DailyDialogue is a high-quality multi-turn dialogue dataset, which contains conversations about daily life. In this dataset, humans often first respond to previous context and then propose their own questions and suggestions. In this way, people show their attention others’ words and are willing to continue the conversation. Compare to the task-oriented dialogue datasets, the speaker's behavior will be more unpredictable and complex for the arbitrator.
Because the task we concentrate on is different from traditional ones, to make the datasets fit our problems and real life, we modify the datasets with the following steps:
Drop Slots and Values For task-oriented dialogue, slot labels are important for navigating the system to complete a specific task. However, those labels and accurate values from ontology files will not benefit our task essentially. So we replace all specific values with a slot placeholder in preprocessing step.
Split Utterances Existing datasets concentrate on the dialogue content, combining multiple sentences into one utterance each turn when gathering the data. In this step, we randomly split the combined utterance into multiple utterances according to the punctuation. And we set a determined probability to decide if the preprocessing program should split a certain sentence.
Add Turn Tag We add turn tags, subturn tags and role tags to each split and original sentences to (1) label the speaker role and dialogue turns (2) tag the ground truth for training and testing the supervised baselines and our model.
Finally, we have the modified datasets which imitate the real life human chatting behaviors as shown in Figure FIGREF1. Our datasets and code will be released to public for further researches in both academic and industry.
To compare with dataset baselines in multiple dimensions and test the model's performance, we use the overall Bilingual Evaluation Understudy (BLEU) BIBREF22 to evaluate the imaginators' generation performance. As for arbitrator, we use accuracy score of the classification to evaluate. Accuracy in our experiments is the correct ratio in all samples.
The hyper-parameter settings adopted in baselines and our model are the best practice settings for each training set. All models are tested with various hyper-parameter settings to get their best performance. Baseline models are Bidirectional Gated Recurrent Units (Bi-GRUs) BIBREF23, TextCNNs BIBREF12 and BERT BIBREF14.
In Table TABREF29, we show different imaginators' generation abilities and their performances on the same TextCNN based arbitrator. Firstly, we gathered the results of agent and user imaginators' generation based on LSTM, LSTM-attention and LSTM-attention with GLOVE pretrained word embedding. According to the evaluation metric BLEU, the latter two models achieve higher but similar results. Secondly, when fixed the arbitrator on the TextCNNs model, the latter two also get the similar results on accuracy and significantly outperform the others including the TextCNNs baseline.
The performances on different arbitrators with the same LSTM-attention imaginators are shown in Table TABREF30. From those results, we can directly compared with the corresponding baseline models. The imaginators with BERT based arbitrator make the best results in both datasets while all ITA models beat the baseline models.
We also present an example of how our model runs in Table TABREF37. Imaginators predict the agent and user's utterance according to the dialogue history(shown in model prediction), and then arbitrator selects the user imaginator's prediction that is more suitable with the dialogue history. It is worth noting that the arbitrator generates a high-quality sentence again if only considering the generation effect. However, referring to the dialogue history, it is not a good choice since its semantic is repeated in the last turn by the agent.
From Table TABREF30, we can see that not only our BERT based model get the best results in both datasets, the other two models also significantly beat the corresponding baselines. Even the TextCNNs based model can beat all baselines in both datasets.
Table TABREF29 figures out experiment results on MultiWOZ dataset. The LSTM based agent imaginator get the BLEU score at 11.77 on agent samples, in which the ground truth is agents' utterances, and 0.80 on user samples. Meanwhile, the user imaginator get the BLEU score at 0.3 on agent samples and 8.87 on user target samples. Similar results are shown in other imaginators' expermients. Although these comparisons seem unfair to some extends since we do not have the agent and user's real utterances at the same time and under the same dialogue history, these results show that the imaginators did learn the speaking style of agent and user respectively. So the suitable imaginator's generation will be more similar to the ground truth, such an example shown in Table TABREF37, which means this response more semantically suitable given the dialogue history.
If we fix the agent and user imaginators' model, as we take the LSTM-attention model, the arbitrators achieve different performances on different models, shown in Table TABREF30. As expected, ITA models beat their base models by nearly 2 $\sim $ 3% and ITA-BERT model beats all other ITA models.
So from the all results, we can conclude that imaginators will significantly help the arbitrator in predicting the dialogue interaction behavior using the future possible agent and user responses’ semantic information.
As shown in the DailyDialogue dataset of Table TABREF29, we can see that attention mechanism works in learning the generation task. LSTMs -Attention and LSTMs-attention-GLOVE based imaginators get more than 19 and 24 BLEU scores in corresponding target, while the LSTMs without attention gets only 4.51 and 8.70. These results also impact on the arbitrator results. The imaginator with attention mechanism get an accuracy score of 79.02 and 78.56, significantly better than the others. The evidence also exists in the results on MultiWoz. All imaginators get similar generation performance, so the arbitrators gets the similar accuracy scores.
From those results, we can conclude that there is positive correlation between the performance of imaginators and arbitrators. However, there still exists problems. It's not easy to evaluate the dialogue generation's performance. In the results of MultiWoz, we can see that LSTMs-GLOVE based ITA performs a little better than LSTMs-attention based ITA, but not the results of the arbitrator are opposite. This may indicate that (1) when the imaginators' performance is high enough, the arbitrator's performance will be stable and (2) the BLEU score will not perfectly present the contribution to the arbitrator. We leave these hypotheses in future work.
We first address an interaction problem, whether the dialogue model should wait for the end of the utterance or reply directly in order to simulate user's real life conversation behavior, and propose a novel Imagine-then-Arbitrate (ITA) neural dialogue model to deal with it. Our model introduces the imagined future possible semantic information for prediction. We modified two popular dialogue datasets to fit in the real situation. It is reasonable that additional information is helpful for arbitrator, despite its fantasy.

Figure and table captions:
Figure 1: A multi-turn dialogue fragment. In this case, user sends splited utterances in a turn, e.g. split U1 to {U11, U12 and U13}
Figure 2: Model Overview. (a) Train the agent and user imaginators using the same dialogues but different samples. (b) During training and inference step, arbitrator uses the dialogue history and two trained imaginators’ predictions.
Table 1: Datasets Statistics. Note that the statistics are based on the modified dataset described in Section 5.2
Table 2: Results of the different imaginators generation performance (in BLEU score) and accuracy score on the same TextCNNs based arbitrator. Better results between imaginators are in BOLD and best results on datasets are in RED.
Table 3: Accuracy Results on Two datasets. Better results between baselines and corresponding ITA models are in BOLD and best results on datasets are in RED. Random result is the accuracy of script that making random decisions.
Table 4: An Example of The Imaginator’s Generation and arbitrator’s Selection.

Questions about this paper:
By how much does their model outperform the baseline?

Answers:
Best accuracy result of proposed model is 82.73, 79.35 compared to best baseline result of 80.75, 78.68 on MultiWoz and DailyDialogue datasets respectively.