Finding Dominant User Utterances And System Responses in Conversations

Abstract: There are several dialog frameworks which allow manual specification of intents and rule based dialog flow. The rule based framework provides good control to dialog designers at the expense of being more time consuming and laborious. The job of a dialog designer can be reduced if we could identify pairs of user intents and corresponding responses automatically from prior conversations between users and agents. In this paper we propose an approach to find these frequent user utterances (which serve as examples for intents) and corresponding agent responses. We propose a novel SimCluster algorithm that extends standard K-means algorithm to simultaneously cluster user utterances and agent utterances by taking their adjacency information into account. The method also aligns these clusters to provide pairs of intents and response groups. We compare our results with those produced by using simple Kmeans clustering on a real dataset and observe upto 10% absolute improvement in F1-scores. Through our experiments on synthetic dataset, we show that our algorithm gains more advantage over K-means algorithm when the data has large variance.

There are several existing works that focus on modelling conversation using prior human to human conversational data BIBREF0 , BIBREF1 , BIBREF2 . BIBREF3 models the conversation from pairs of consecutive tweets. Deep learning based approaches have also been used to model the dialog in an end to end manner BIBREF4 , BIBREF5 . Memory networks have been used by Bordes et al Bor16 to model goal based dialog conversations. More recently, deep reinforcement learning models have been used for generating interactive and coherent dialogs BIBREF6 and negotiation dialogs BIBREF7 .
Industry on the other hand has focused on building frameworks that allow manual specification of dialog models such as api.ai, Watson Conversational Services, and Microsoft Bot framework. These frameworks provide ways to specify intents, and a dialog flow. The user utterances are mapped to intents that are passed to a dialog flow manager. The dialog manager generates a response and updates the dialog state. See Figure FIGREF4 for an example of some intents and a dialog flow in a technical support domain. The dialog flow shows that when a user expresses an intent of # laptop_heat, then the system should respond with an utterance “Could you let me know the serial number of your machine ”. The designer needs to specify intents (for example # laptop_heat, # email_not_opening) and also provide corresponding system responses in the dialog flow. This way of specifying a dialog model using intents and corresponding system responses manually is more popular in industry than a data driven approach as it makes dialog model easy to interpret and debug as well as provides a better control to a dialog designer. However, this is very time consuming and laborious and thus involves huge costs.
One approach to reduce the task of a dialog designer is to provide her with frequent user intents and possible corresponding system responses in a given domain. This can be done by analysing prior human to human conversations in the domain. Figure FIGREF5 (a) provides some example conversations in the technical support domain between users and agents.
In order to identify frequent user intents, one can use existing clustering algorithms to group together all the utterances from the users. Here each cluster would correspond to a new intent and each utterance in the cluster would correspond to an example for the intent. Similarly the agents utterances can be clustered to identify system responses. However, we argue that rather than treating user utterances and agents responses in an isolated manner, there is merit in jointly clustering them. There is adjacency information of these utterances that can be utilized to identify better user intents and system responses. As an example, consider agent utterances A.2 in box A and A.2 in box C in Figure FIGREF5 (a). The utterances “Which operating system do you use?" and “What OS is installed in your machine" have no syntactic similarity and therefore may not be grouped together. However the fact that these utterances are adjacent to the similar user utterances “I am unable to start notes email client" and “Unable to start my email client" provides some evidence that the agent utterances might be similar. Similarly the user utterances “My system keeps getting rebooted" and “Machine is booting time and again" ( box B and D in Figure FIGREF5 (a))- that are syntactically not similar - could be grouped together since the adjacent agent utterances, “Is your machine heating up?" and “Is the machine heating?" are similar.
Joint clustering of user utterances and agent utterances allow us to align the user utterance clusters with agent utterance clusters. Figure FIGREF5 (b) shows some examples of user utterance clusters and agent utterance clusters along with their alignments. Note that the user utterance clusters can be used by a dialog designer to specify intents, the agent utterance clusters can be used to create system responses and their alignment can be used to create part of the dialog flow.
We propose two ways to take adjacency information into account. Firstly we propose a method called SimCluster for jointly or simultaneously clustering user utterances and agent utterances. SimCluster extends the K-means clustering method by incorporating additional penalty terms in the objective function that try to align the clusters together (described in Section SECREF3 ). The algorithm creates initial user utterance clusters as well as agent utterance clusters and then use bi-partite matching to get the best alignment across these clusters. Minimizing the objective function pushes the cluster centroids to move towards the centroids of the aligned clusters. The process implicitly ensures that the similarity of adjacent agent utterances affect the grouping of user utterances and conversely similarity of adjacent user utterances affect the grouping of agent utterances. In our second approach we use the information about neighbouring utterances for creating the vector representation of an utterance. For this we train a sequence to sequence model BIBREF8 to create the vectors (described in Section SECREF5 ).
Our experiments described in section SECREF5 show that we achieve upto 10% absolute improvement in F1 scores over standard K-means using SimCluster. Also we observe that clustering of customer utterances gains significantly by using the adjacency information of agent utterances whereas the gain in clustering quality of agent utterances is moderate. This is because the agent utterances typically follow similar syntactic constructs whereas customer utterances are more varied. Considering the agent utterances into account while clustering users utterances is thus helpful. The organization of the rest of the paper is as follows. In Section SECREF2 we describe the related work. In Section SECREF3 we describe our problem formulation for clustering and the associated algorithm. Finally in sections SECREF4 and SECREF5 we discuss our experiments on synthetic and real datasets respectively.
The notion of adjacency pairs was introduced by Sacks et al SSE74 to formalize the structure of a dialog. Adjacency pairs have been used to analyze the semantics of the dialog in computational linguistics community BIBREF9 . Clustering has been used for different tasks related to conversation. BIBREF10 considers the task of discovering dialog acts by clustering the raw utterances. We aim to obtain the frequent adjacency pairs through clustering.
There have been several works regarding extensions of clustering to different scenarios such as:-
In this section we describe our approach SimCluster that performs clustering in the two domains simultaneously and ensures that the generated clusters can be aligned with each other. We will describe the model in section SECREF9 and the algorithm in Section SECREF11 .
We consider a problem setting where we are given a collection of pairs of consecutive utterances, with vector representations INLINEFORM0 where INLINEFORM1 s are in speaker 1's domain and INLINEFORM2 s are in speaker 2's domain. We need to simultaneously cluster the utterances in their respective domains to minimize the variations within each domain and also ensure that the clusters for both domains are close together.
We denote the clusters for speaker 1's domain by INLINEFORM0 with their respective means INLINEFORM1 . We denote the clusters assignments for INLINEFORM2 by INLINEFORM3 .
We denote the clusters for second speaker by INLINEFORM0 with their respective means INLINEFORM1 . We denote the clusters assignments for INLINEFORM2 by INLINEFORM3 . The usual energy function has the terms for distance of points from their corresponding cluster centroids. To be able to ensure that the clusters in each domain are similar, we also consider an alignment between the centroids of the two domains. Since the semantic representations in the two domains are not comparable we consider a notion of induced centroids.
We define the induced centroids INLINEFORM0 as the arithmetic means of the points INLINEFORM1 s such that INLINEFORM2 's have the same cluster assigned to them. Similarly, we define INLINEFORM3 as the arithmetic means of INLINEFORM4 s such that INLINEFORM5 s have the same cluster assigned to them. More formally, we define these induced centroids as:- INLINEFORM6 
and INLINEFORM0 
The alignment between these clusters given by the function INLINEFORM0 , which is a bijective mapping from the cluster indices in speaker 1's domain to those in speaker 2's domain. Though there can be several choices for this alignment function, we consider this alignment to be a matching which maximizes the sum of number of common indices in the aligned clusters. More formally we define INLINEFORM1 
Then the matching INLINEFORM0 is defined to be the bijective function which maximizes INLINEFORM1 . We consider a term in the cost function corresponding to the sum of distances between the original centroids and the matched induced centroids. Our overall cost function is now given by:- INLINEFORM2 
We explain the above definition via an example. Consider the clusters shown in Figure FIGREF10 . Here the INLINEFORM0 would match INLINEFORM1 to INLINEFORM2 , INLINEFORM3 to INLINEFORM4 and INLINEFORM5 to INLINEFORM6 , giving a match score of 6. Since INLINEFORM7 , INLINEFORM8 and INLINEFORM9 are present in the cluster INLINEFORM10 , INLINEFORM11 is given by INLINEFORM12 . Similarly INLINEFORM13 
 In a similar manner, INLINEFORM0 s can also be defined. Now the alignment terms are given by:- INLINEFORM1 
 
[] SimCluster [1] SimClusterInput: INLINEFORM0 ,k (No. of cluster) Output: A cluster assignment INLINEFORM1 for INLINEFORM2 s and a cluster assignment INLINEFORM3 for INLINEFORM4 s Initialize a set of centroids INLINEFORM5 , and INLINEFORM6 Perform simple clustering for a few iterations For each i, compute INLINEFORM7 as the index j among 1 to k which minimizes INLINEFORM8 . Similarly , compute INLINEFORM9 as the index j' among 1 to k which minimizes INLINEFORM10 . Update the centroids, INLINEFORM11 and INLINEFORM12 as:- INLINEFORM13 
and INLINEFORM0 
 Perform a Hungarian matching between the cluster indices in the two domains with weights
N(j,j') on edges from index j to index j'. convergence To minimize the above energy term we adopt an approach similar to Lloyd's clustering algorithm Llo82 . We assume that we are given a set of initial seeds for the cluster centroids INLINEFORM0 and INLINEFORM1 . We repeat the following steps iteratively:-
Minimize the energy with respect to cluster assignment keeping centroids unchanged. As in standard K-means algorithm, this is achieved by updating the cluster assignment, INLINEFORM0 for each index i to be the cluster index j which minimizes INLINEFORM1 . Correspondingly for INLINEFORM2 , we pick the cluster index j' which minimizes INLINEFORM3 .
 Minimize the energy with respect to the centroids keeping cluster assignment unchanged. To achieve this step we need to minimize the energy function with respect to the centroids INLINEFORM0 and INLINEFORM1 . This is achieved by setting INLINEFORM2 for each j and INLINEFORM3 for each j.
Setting INLINEFORM0 , we obtain INLINEFORM1 
or equivalently INLINEFORM0 
 Similarly, setting INLINEFORM0 , we obtain INLINEFORM1 
Finally we update the matching between the clusters. To do so, we need to find a bipartite matching match on the cluster indices so as to maximize INLINEFORM0 . We use Hungarian algorithm BIBREF13 to perform the same i.e. we define a bipartite graph with vertices consisting of cluster indices in the two domains. There is an edge from vertex representing cluster indices j (in domain 1) and j' in domain 2, with weight N(j,j'). We find a maximum weight bipartite matching in this graph.
Similar to Lloyd's algorithm, each step of the above algorithm decreases the cost function. This ensures that the algorithm achieves a local minima of the cost function if it converges. See Algorithm SECREF11 for a formal description of the approach. The centroid update step of the above algorithm also has an intuitive explanation i.e. we are slightly moving away the centroid towards the matched induced centroid. This is consistent with our goal of aligning the clusters together in the two domains.
The algorithm above maintains a mapping between the clusters in each speaker's domain. This mapping serves to give us the alignment between the clusters required to provide a corresponding response for a given user intent.
We performed experiments on synthetically generated dataset since it gives us a better control over the distribution of the data. Specifically we compared the gains obtained using our approach versus the variance of the distribution. We created dataset from the following generative process. [H] Generative Process [1] Generate data
Pick k points INLINEFORM0 as domain -1 means and a corresponding set of k points INLINEFORM1 as domain-2 means, and covariance matrices INLINEFORM2 
iter INLINEFORM0 upto num INLINEFORM1 samples Sample class INLINEFORM2 Sample INLINEFORM3 Sample INLINEFORM4 Add q and a so sampled to the list of q,a pairs We generated the dataset from the above sampling process with means selected on a 2 dimensional grid of size INLINEFORM5 with variance set as INLINEFORM6 in each dimension.10000 sample points were generated. The parameter INLINEFORM7 of the above algorithm was set to 0.5 and k was set to 9 (since the points could be generated from one of the 9 gaussians with centroids on a INLINEFORM8 grid).
We compared the results with simple K-means clustering with k set to 9. For each of these, the initialization of means was done using INLINEFORM0 sampling approach BIBREF14 .
To evaluate the clusters we computed the following metrics
ARI (Adjusted Rand Index): Standard Rand Index is a metric used to check the clustering quality against a given standard set of clusters by comparing the pairwise clustering decisions. It is defined as INLINEFORM0 , where a is the number of true positive pairs, b is the number of true negative pairs, c is the number of false positive pairs and d is the number of false negative pairs. Adjusted rand index corrects the standard rand index for chance and is defined as INLINEFORM1 BIBREF15 .
We compute ARI score for both the source clusters as well as the target clusters.
F1 scores: We also report F1 scores for the pairwise clustering decisions. In the above notation we considered the pair-precision as INLINEFORM0 and recall as INLINEFORM1 . The F1 measure is the Harmonic mean given as INLINEFORM2 .
We used the gaussian index from which an utterance pair was generated as the ground truth label, which served to provide ground truth clusters for computation of the above evaluation metrics. Table TABREF15 shows a comparison of the results on SimCluster versus K-means algorithm. Here our SimCluster algorithm improves the F1-scores from 0.412 and 0.417 in the two domains to 0.442 and 0.441. The ARI scores also improve from 0.176 and 0.180 to 0.203 and 0.204.
We also performed experiments to see how the performance of SimCluster is affected by the variance in the cluster (controlled by the generative process in Algorithm SECREF11 ). Intuitively we expect SimCluster to obtain an advantage over simple K-means when variance is larger. This is because at larger variance, the data points are more likely to be generated away from the centroid due to which they might be clustered incorrectly with the points from neighbouring cluster. However if the corresponding point from the other domain is generated closer to the centroid, it might help in clustering the given data point correctly. We performed these experiments with points generated from Algorithm SECREF11 at differet values of variance. We generated the points with centroids located on a grid of size INLINEFORM0 in each domain. The value of k was set to 9. The experiment was repeated for each value of variance between 0.1 to 1.0 in the intervals of 0.1. Figures FIGREF22 and FIGREF23 show the percentage improvement on ARI score and F1 score respectively achieved by SimCluster (over K-means) versus variance.
We have experimented on a dataset containing Twitter conversations between customers and Amazon help. The dataset consisted of 92130 conversations between customers and amazon help. We considered the conversations with exactly two speakers Amazon Help and a customer. Consecutive utterances by the same speaker were concatenated and considered as a single utterance. From these we extracted adjacency pairs of the form of a customer utterance followed by an agent (Amazon Help) utterance. We then selected the utterance pairs from 8 different categories, like late delivery of item, refund, unable to sign into the account, replacement of item, claim of warranty, tracking delivery information etc. A total of 1944 utterance pairs were selected.
To create the vector representation we had used two distinct approaches:-
Paragraph to vector approach (Doc2Vec) by Le and Mikolov LM14. Here we trained the vectors using distributed memory algorithm and trained for 40 iterations. A window size of 4 was used.
We also trained the vectors using sequence to sequence approach BIBREF8 , on the Twitter dataset where we considered the task of predicting the reply of Amazon Help for customer's query and vice versa.
The encoded vector from the input sequence forms the corresponding vector representation. For the task of generating the agent's response for customer utterance the encoding from the input sequence (in the trained model) forms the vector representation for the customer utterance. Similarly for the task of generating the previous customer utterance from the agent's response, the intermediate encoding forms the vector representation for the agent utterance. We used an LSTM based 3-layered sequence to sequence model with attention for this task.
We ran the K-means clustering algorithm for 5 iterations followed by our SimCluster algorithm for 30 iterations to form clusters in both the (customer and agent) domains. The hyper parameter( INLINEFORM0 ) is chosen based on a validation set. We varied the value of INLINEFORM1 from 0.5 to 1.0 at intervals of 0.025. The initialization of centroids was performed using INLINEFORM2 sampling approach BIBREF14 .
For the clusters so obtained we have computed F1 and ARI measures as before and compared with the K-means approach. We used the partitioning formed by the 8 categories (from which the utterance pairs were selected) as the ground truth clustering.
Table TABREF20 summarizes the results. We observe that for K-means algorithm, the vectors generated from sequence to sequence model perform better than the vectors generated using paragraph to vector for both the domains. This is expected as the vectors generated from sequence to sequence model encode some adjacency information as well. We further observe that the SimCluster approach performs better than the K-means approach for both the vector representations. It improves the F1-scores for Doc2Vec representation from 0.787 and 0.783 to 0.88 and 0.887 in the two domains. Also the F1-scores on Seq2Seq based representation improve from 0.83 and 0.9 to 0.86 and 0.916 using SimCluster. However the gains are much more in case of Doc2Vec representations than Seq2Seq representations since Doc2Vec did not have any information from the other domain where as some amount of this information is already captured by Seq2Seq representation. Moreover it is the clustering of customer utterances which is likely to see an improvement. This is because agent utterances tends to follow a generic pattern while customer utterances tend to be more varied. Considering agent utterances while generating clusters in the user domain thus tends to be more helpful than the other way round.
Table TABREF25 shows qualitative results on the same dataset. Column 1 and 2 consists of clusters of utterances in customer domain and agent domain respectively. The utterances with usual font are representative utterances from clusters obtained through K-means clustering. The utterances in bold face indicate the similar utterances which were incorrectly classified in different clusters using K-means but were correctly classified together with the utterances by SimCluster algorithm.
One of the first steps to automate the construction of conversational systems could be to identify the frequent user utterances and their corresponding system responses. In this paper we proposed an approach to compute these groups of utterances by clustering the utterances in both the domains using our novel SimCluster algorithm which seeks to simultaneously cluster the utterances and align the utterances in two domains. Through our experiments on synthetically generated datset we have shown that SimCluster has more advantage over K-means on datasets with larger variance. Our technique improves upon the ARI and F1 scores on a real dataset containing Twitter conversations.
We thank Dr. David Nahamoo (CTO, Speech Technology and Fellow IBM Research ) for his valuable guidance and feedback. We also acknowledge the anonymous reviewers of IJCNLP 2017 for their comments.

Figure and table captions:
Figure 1: Some intents and dialog flow
Figure 2: Some sample conversations and the obtained clusters
Figure 3: Sample clusters with matching
Table 1: Performance of SimCluster versus K-means clustering on synthetic dataset
Figure 4: Improvement in ARI figures achieved by SimCluster versus variance
Figure 5: Variation of Improvement in F1 score figures achieved by SimCluster versus variance
Table 2: Performance of SimCluster versus K-means clustering on both Doc2Vec as well as seq2seq based vectors
Table 3: Sample clusters in user and agent domains. Utterances in bold are those which were not in the given cluster using K-means, but could be correctly classified with the cluster using SimCluster

Questions about this paper:
How do they generate the synthetic dataset?

Answers:
using generative process