DialoGraph: Incorporating Interpretable Strategy-Graph Networks into Negotiation Dialogues

1 INTRODUCTION . Negotiation is ubiquitous in human interaction , from e-commerce to the multi-billion dollar sales of companies . Learning how to negotiate effectively involves deep pragmatic understanding and planning the dialogue strategically ( Thompson ; Bazerman et al. , 2000b ; Pruitt , 2013 ) . Modern dialogue systems for collaborative tasks such as restaurant or flight reservations have made considerable progress by modeling the dialogue history and structure explicitly using the semantic content , like slot-value pairs ( Larionov et al. , 2018 ; Young , 2006 ) , or implicitly with encoder-decoder architectures ( Sordoni et al. , 2015 ; Li et al. , 2016 ) . In such tasks , users communicate explicit intentions , enabling systems to map the utterances into specific intent slots ( Li et al. , 2020 ) . However , such mapping is less clear in complex non-collaborative tasks like negotiation ( He et al. , 2018 ) and persuasion ( Wang et al. , 2019 ) , where user intent and most effective strategies are hidden . Hence , along with the generated dialogue , the strategic choice of framing and the sequence of chosen strategies play a vital role , as depicted in Figure 1 . Indeed , prior work on negotiation dialogues has primarily focused on optimizing dialogue strategies—from highlevel task-specific strategies ( Lewis et al. , 2017 ) , to more specific task execution planning ( He et al. , 2018 ) , to fine-grained planning of linguistic outputs given 1Code , data and a demo system is released at https : //github.com/rishabhjoshi/ DialoGraph_ICLR21 strategic choices ( Zhou et al. , 2019 ) . These studies have confirmed that it is crucial to control for pragmatics of the dialogue to build effective negotiation systems . To model the explicit dialogue structure , prior work incorporated Hidden Markov Models ( HMMs ) ( Zhai & Williams , 2014 ; Ritter et al. , 2010 ) , Finite State Transducers ( FSTs ) ( Zhou et al. , 2020 ) and RNNs ( He et al. , 2018 ; Shi et al. , 2019 ) . While RNN-based models lack interpretability , HMMand FST-based approaches may lack expressivity . In this paper , we hypothesize that Graph Neural Networks ( GNNs ) ( Wu et al. , 2020 ) can combine the benefits of interpretability and expressivity because of their effectiveness in encoding graph-structured data through message propagation . While being sufficiently expressive to model graph structures , GNNs also provide a natural means for interpretation via intermediate states ( Xie & Lu , 2019 ; Pope et al. , 2019 ) . We propose DIALOGRAPH , an end-to-end negotiation dialogue system that leverages Graph Attention Networks ( GAT ) ( Veličković et al. , 2018 ) to model complex negotiation strategies while providing interpretability for the model via intermediate structures . DIALOGRAPH incorporates the recently proposed hierarchical graph pooling based approaches ( Ranjan et al. , 2020 ) to learn the associations between negotiation strategies , including conceptual and linguistic strategies and dialogue acts , and their relative importance in predicting the best sequence . We focus on buyer–seller negotiations in which two individuals negotiate on the price of an item through a chat interface , and we model the seller ’ s behavior on the CraigslistBargain dataset ( He et al. , 2018 ) .2 We demonstrate that DIALOGRAPH outperforms previous state-of-art methods on strategy prediction and downstream dialogue responses . This paper makes several contributions . First , we introduce a novel approach to model negotiation strategies and their dependencies as graph structures , via GNNs . Second , we incorporate these learned graphs into an end-to-end negotiation dialogue system and demonstrate that it consistently improves future-strategy prediction and downstream dialogue generation , leading to better negotiation deals ( sale prices ) . Finally , we demonstrate how to interpret intermediate structures and learned sequences of strategies , opening-up the black-box of end-to-end strategic dialogue systems . 2 DIALOGRAPH . We introduce DIALOGRAPH , a modular end-to-end dialogue system , that incorporates GATs with hierarchical pooling to learn pragmatic dialogue strategies jointly with the dialogue history . DIALOGRAPH is based on a hierarchical encoder-decoder model and consists of three main components : ( 1 ) hierarchical dialogue encoder , which learns a representation for each utterance and encodes its local context ; ( 2 ) structure encoder for encoding sequences of negotiation strategies and dialogue acts ; and ( 3 ) utterance decoder , which finally generates the output utterance . Formally , our dialogue input consists of a sequence of tuples , D = [ ( u1 , da1 , ST1 ) , ( u2 , da2 , ST2 ) , ... , ( un , dan , STn ) ] where ui is the utterance , dai is the coarse dialogue act and STi = { sti,1 , sti,2 , . . . , sti , k } is the set of k fine-grained negotiation strategies for the utterance ui.3 The dialogue context forms the input to ( 1 ) and the previous dialogue acts and negotiation strategies form the input to ( 2 ) . The overall architecture is shown in Figure 2 . In what follows , we describe DIALOGRAPH in detail . 2.1 HIERARCHICAL DIALOGUE ENCODER . A dialogue context typically comprises of multiple dialogue utterances which are sequential in nature . We use hierarchical encoders for modeling such sequential dialogue contexts ( Jiao et al. , 2019 ) . To encode the utterance ut at time t , we use the pooled representations from BERT ( Devlin et al. , 2019 ) to obtain the corresponding utterance embedding et . We then pass the utterance embeddings through a GRU to obtain the dialogue context encoding till time t , denoted by hUt . 2We focus on the seller ’ s side following Zhou et al . ( 2019 ) who devised a set of strategies specific to maximizing the seller ’ s success . Our proposed methodology , however , is general . 3For example , in an utterance Morning ! My bro destroyed my old kit and I ’ m looking for a new pair for $ 10 , the coarse dialogue act is Introduction , and the finer grained negotiation strategies include Proposing price , Being informal and Talking about family for building rapport . 2.2 STRUCTURE ENCODER . Our structure encoder is designed to model the graph representations of the strategies and dialogue acts using GATs and output their structural representations . These structural representations are used to predict the next set of strategies and dialogue acts and enrich the encoded dialogue representation . Below we describe the structure encoder for negotiation strategies . We model the sequence of negotiation strategies , ST = [ ST1 , ST2 , . . . , STt ] by creating a directed graph , where STi is the set of k fine-grained negotiation strategies for the utterance ui . Formally , we define a graph G ( V , E , X ) with |E| edges andN = |V| nodes where each node vi ∈ V represents a particular negotiation strategy for an utterance and has a d-dimensional feature representation denoted by zi . Z ∈ RN×d denotes the feature matrix of the nodes and A ∈ RN×N represents the adjacency matrix , where N is the total number of nodes ( strategies ) that have occurred in the conversation till that point . Therefore , each node represents a strategy-utterance pair . We define the set of edges as E = { ( a , b ) } ; a , b ∈ V where a and b denote strategies at utterances ua and ub , present at turns ta and tb , such that tb > ta . In other words , we make a directed edge from a particular node ( strategy in an utterance ) to all the consecutive nodes . This ensures a direct connection from all the previous strategies to the more recent ones.4 In the same way , we form the graph out of the sequence of dialogue acts . These direct edges and learned edge attention weights help us interpret the dependence and influence of strategies on each other . To get the structural representations from the strategy graphs , we pass them through a hierarchical graph pooling based encoder , which consists of l layers of GAT , each followed by the Adaptive Structure Aware Pooling ( ASAP ) layer ( Ranjan et al. , 2020 ) . As part of the ASAP layer , the model first runs GAT over the input graph representations to obtain structurally informed representations of the nodes . Then a cluster assignment step is performed which generates a cluster assignment matrix , S , which tells the model which nodes come in a similar structural context . After that , the clusters are ranked and then the graph is pooled by taking the top few clusters as new nodes and forming edges between them using the existing graph . This way the size of the graph is reduced at every step which leads to a structurally informed graph representation . We take advantage of the cluster formulation to obtain the associations between the negotiation strategies , as identified from the cluster assignment matrix , S. These association scores can later be used to interpret which strategies are associated with each other and tend to co-occur in similar contexts . Moreover , we also use the node attention scores from GAT to interpret the influence of different strategies on the 4Appendix C shows an example of the graph obtained from a sequence of strategies . representation of a particular strategy , which essentially gives the dependence information between strategies . In this way , the structure representation is learned and accumulated in a manner that preserves the structural information ( Ying et al. , 2018 ; Lee et al. , 2019 ) . After each pooling step , the graph representation is summarized using the concatenation of mean and max of the node representations . The summaries are then added and passed through fully connected layers to obtain the final structural representation of the strategies hSTt . We employ a similar structure encoder to encode the graph obtained from the sequence of dialogue acts , to obtain hdat . 2.3 UTTERANCE DECODER . The utterance decoder uses the dialogue context representation and structural representations of dialogue acts and negotiation strategies to produce the dialogue response ( next utterance ) . We enrich the dialogue representation by concatenating the structural representations before passing it to a standard greedy GRU ( Cho et al. , 2014 ) decoder . This architecture follows Zhou et al . ( 2020 ) , who introduced a dynamic negotiation system that incorporates negotiation strategies and dialogue acts via FSTs . We thus follow their utterance decoder architecture to enable direct baseline comparison . For the jth word of utterance ut+1 , w j t+1 , we condition on the previous word w j−1 t+1 to calculate the probability distribution over the vocabulary as pwjt+1 = softmax ( GRU ( ht , w j−1 t+1 ) ) where ht = [ hut ; h ST t ; h da t ] and [ ; ] represents the concatenation operator . For encoding the price , we replace all price information in the dataset with placeholders representing the percentage of the offer price . For example , we would replace $ 35 with < price− 0.875 > if the original selling price is $ 40 . The decoder generates these placeholders which are then replaced with the calculated price before generating the utterance . 2.4 MODEL TRAINING . We use hSTt to predict the next set of strategies STt+1 , a binary value vector which represents the k-hot representation of negotiation strategies for the next turn . We compute the probability of the jth strategy occurring in ut+1 as p ( stt+1 , j |hSTt ) = σ ( hSTt ) . where σ denotes the sigmoid operator . We threshold the probability by 0.5 to obtain the k-hot representation . We denote the weighted negative log likelihood of strategies LST as the loss function of the task of next strategy prediction LST = − ∑ j δj log ( p ( stt+1 , j ) ) − ∑ k log ( 1 − p ( stt+1 , k ) ) where the summation of j are over the strategies present ( st ′ t+1 , j = 1 ) and not present ( st ′ t+1 , k = 0 ) in the ground truth strategies set , ST ′ . Here δj is the positive weight associated with the particular strategy . We add this weight to the positive examples to trade off precision and recall . We put δj = # of instances not having strategy j/ # of instances having strategy j . Similarly , we use hdat to predict the dialogue act for the next utterance dat+1 . Given the target dialogue act da ′ t+1 and the class weights ρda for the dialogue acts , we denote the class-weighted cross entropy loss over the set of possible dialogue acts , LDA = −ρda log ( softmax ( hdat ) ) . We pass ht = [ hut ; h ST t ; h da t ] through a linear layer to predict the negotiation success , which is denoted by the sale-to-list ratio r = ( sale price− buyer target price ) / ( listed price− buyer target price ) ( Zhou et al. , 2019 ) . We split the ratios into 5 negotiation classes of equal sizes using the training data and use those to predict the success of negotiation . Therefore , given the predicted probabilities for target utterance u ′ t+1 from §2.3 , target ratio class y ′ r and the learnable parameters Wr and br , we use the cross entropy loss as the loss for the generation task ( LNLG ) as well as the negotiation outcome prediction task ( LR ) , thus LNLG = − ∑ wj∈u ′ t+1 log ( p wj t+1 ) and LR = − ∑ r∈ [ 1,5 ] y ′ r log ( softmax ( Wrht + br ) ) . The LR loss optimizes for encoding negotiation strategies to enable accurate prediction of negotiation outcome . We use hyperparameters α , β and γ to optimize the joint loss Ljoint , of strategy prediction , dialogue act prediction , utterance generation and outcome prediction together , using the Adam optimizer ( Kingma & Ba , 2014 ) , to get Ljoint = LNLG + αLST + βLDA + γLR .

Summary: This paper deals with the problem of natural language generation for a dialogue system involved in complex communication tasks such as negotiation or persuasion. The proposed architecture consists of two encoders: one for the utterance and the other for dialogue acts and negotiation strategies. The decoder is an RNN that converts the encoded vectors to the output utterance. Each utterance is first passed through BERT to get an utterance-level encoding. The sequence of utterance encodings is then passed through an RNN to generate a conversation level encodings. The negotiation strategies and dialogue acts in a conversation are represented using a node-edge graph, where the nodes are one of the N different strategies/acts and there exists an edge from node a to node b if an utterance with strategy A precedes any utterance with strategy B. The entire architecture is trained in a multi-task setup where the loss function accounts for both the predictions of the model and generated language. The proposed architecture is evaluated on the CraigslistBargain dataset and compared against Zhou et al. 2020.