DialoGraph: Incorporating Interpretable Strategy-Graph Networks into Negotiation Dialogues
Authors: Rishabh Joshi
Vidhisha Balachandran
Shikhar Vashishth
Alan Black
Yulia Tsvetkov
Venue: ICLR 2021 Poster (2021)
Keywords: negotiation
dialogue
graph neural networks
interpretability
structure

Abstract: To successfully negotiate a deal, it is not enough to communicate fluently: pragmatic planning of persuasive negotiation strategies is essential. While modern dialogue agents excel at generating fluent sentences, they still lack pragmatic grounding and cannot reason strategically. We present DialoGraph, a negotiation system that incorporates pragmatic strategies in a negotiation dialogue using graph neural networks. DialoGraph explicitly incorporates dependencies between sequences of strategies to enable improved and interpretable prediction of next optimal strategies, given the dialogue context. Our graph-based method outperforms prior state-of-the-art negotiation models both in the accuracy of strategy/dialogue act prediction and in the quality of downstream dialogue response generation. We qualitatively show further benefits of learned strategy-graphs in providing explicit associations between effective negotiation strategies over the course of the dialogue, leading to interpretable and strategic dialogues.

Published as a conference paper at ICLR 2021
DIALOGRAPH: INCORPORATING INTERPRETABLE STRATEGY-GRAPH NETWORKS INTO
NEGOTIATION DIALOGUES
Rishabh Joshi, Vidhisha Balachandran, Shikhar Vashishth, Alan Black, Yulia Tsvetkov
Language Technologies Institute Camegie Mellon University (rjoshi2, vbalacha, svashish, awo, ytsvetko)@cs cmu
ABSTRACT
To successfully negotiate deal, it is not enough to communicate fluently: prag- matic planning of persuasive negotiation strategies essential. While modern dialogue agents excel at generating fluent sentences, they still lack pragmatic grounding and cannot reason strategically. We present DIALOGRAPII, nego- tiation system that incorporates pragmatic strategies in a negotiation dialogue us- ing graph neural networks. DIALOGRAPH explicitly incorporates dependencies between sequences of strategies to enable improved and interpretable prediction of next optimal strategies, given the dialogue context. Our graph-based method outperforms prior state-of-the-art negotiation models both in the accuracy of strat- egy/dialogue act prediction and in the quality of downstream dialogue response generation. We qualitatively show further benefits of learned strategy-graphs providing explicit associations between effective negotiation strategies over the course of the dialogue, leading interpretable and strategic dialogues]
INTRODUCTION
Negotiation is ubiquitous in human interaction, from e-commerce to the multi-billion dollar sales of companies. Learning how to negotiate effectively involves deep pragmatic understanding and planning the dialogue strategically (Thompson] Bazerman et al. 2000b Pruitt 2013).
Modern dialogue systems for collaborative tasks such as restaurant flight reservations have made considerable progress by modeling the dialogue history and structure explicitly using the semantic content, like slot-value pairs (Larionov et al. 2018 Young 2006), or implicitly with encoder-decoder architectures (Sordoni al., 2015 Li
Buyer (Human)
youngor loves I'm looking to buy an stryl player
Family: informal
Propow
I'm zoking
much
Negalive Propune
student like me. How about 307
ett al. 2016). In such tasks, users communicate explicit intentions, enabling systems map the utterances into specific intent slots (Li et al. 2020). However, such map- ping is less clear in complex non-collaborative tasks like negotiation (He et al., (018)) and persuasion (Wang et al.) 2019), where user intent and most effective strategies are hidden. Hence, along with the generated dialogue, the strategic choice framing and the sequence chosen strategies play vital role, as depicted in Figure [ In- deed, prior work on negotiation dialogues has primarily focused optimizing dialogue strategies from high- level task-specific strategies (Lewis et al.) 2017), to more
w/o Strategy
Not poselde, can do 35 You'll havve pick
Pojuct
Strategy Fimmlly intormal,
can do 38 and throw couple of Beaties records for your bro.
Oppla
prosowe
Acccopt
Seller (Bot)
Figure 1: Both options equally plausi- ble and fluent, but a response with effective pragmatic strategies leads to better deal.
specific task execution planning (He 2018 to fine-grained planning of linguistic outputs given
'Code, data and demo system is released httpps///github.comm/t
Dia aalo@rraph_icce CI.R2

---

Published as a conference paper at ICLR 2021
strategic choices (Zhou et al. 2019). These studies have confirmed that it is crucial to control for pragmatics of the dialogue to build effective negotiation systems.
To model the explicit dialogue structure, prior work incorporated Hidden Markov Models (HMMs) Ritter et al. 2010), Finite State Transducers (FSTs) (Zhou et al. (020) and RNNS (He et al., 2018: Shi et al.) 2019). While RNN-based models lack interpretabilityy HMM- and FST-based approaches may lack expressivity. In this paper, we hypothesize that Graph Neural Networks (GNNs) (Wu et al. 2020) can combine the benefits of interpretability and expressivity be- cause of their effectiveness in encoding graph-structured data through message propagation. While being sufficiently expressive model graph structures, GNNs also provide a natural means for interpretation via intermediate states (Xie & Lu 2019 Pope et al. 2019).
(Zhai & Williams 2014
We propose DIALOGRAPH, an end-to-end negotiation dialogue system that leverages Graph At- tention Networks (GAT) (Veličković et al. 2018) to model complex negotiation strategies while providing interpretability for the model via intermediate structures. DIALOGR APH incorporates the recently proposed hierarchical graph pooling based approaches (Ranjan al., (020) to learn the as- sociations between negotiation strategies, including conceptual and linguistic strategies and dialogue acts, and their relative importance in predicting the best sequence. We focus on buyer-seller nego- tiations in which two individuals negotiate on the price of an item through a chat interface, and we model the seller's behavior the CraigslistBargain dataset (He et al. 2018 We demonstrate that DIALOGRAPPH APH outperforms previous state-of-art methods on strategy prediction and downstream dialogue responses. This paper makes several contributions. First, we introduce a novel approach to model negotiation strategies and their dependencies as graph structures, via GNNs. Second, we in- corporate these learned graphs into an end-to-end negotiation dialogue system and demonstrate that it consistently improves future-strategy prediction and downstream dialogue generation, leading to better negotiation deals (sale prices). Finally, we demonstrate how to interpret intermediate struc- tures and learned sequences of strategies, opening-up up the black-box of end-to-end strategic dialogue systems.
2 DIALOGRAPH
We introduce DIALOGRAPH, a modular end-to-end dialogue system, that incorporates GATs with hierarchical pooling to learn pragmatic dialogue strategies jointly with the dialogue history. DIALO- GRAPH is based on a hierarchical encoder-decoder model and consists of three main components: (1) hierarchical dialogue encoder, which learns a representation for each utterance and encodes its local context; (2) structure encoder for encoding sequences of negotiation strategies and dialogue acts; and (3) utterance decoder, which finally generates the output utterance. Formally, our dialogue input consists of sequence of tuples, D = [M1, dai, STT). (1/2, daz, ST2), (u,, da,, ST,)) where u, is the utterance, da; is the coarse dialogue act and ST; = {st,,, st.,2. st, is the set of & fine-grained negotiation strategies for the utterance u, M The dialogue context forms the input to (1) and the previous dialogue acts and negotiation strategies form the input to (2). The overall architecture shown in Figure 2 In what follows, we describe DIALOGRAAPH detail.
2.1 IIERARCIICAL DIALOGUE ENCODER
A dialogue context typically comprises of multiple dialogue ulterances which are sequential in na- ture. We use hierarchical encoders for modeling such sequential dialogue contexts (Jiao et al. 2019). To encode the utterance d, time f, we use the pooled representations from BERT (Devlin et al.) 2019) to obtain the corresponding utterance embedding e,, We then pass the utterance embeddings through a GRU to obtain the dialogue context encoding till time denoted by h
We focus the seller's side following Zhou al. 2019 who devised set of strategies specific maximizing the seller's success. Our proposed methodology, however, is general.
For example. in an utterance Morning! My bro destroyed my old kit and I I' looking for a new pair for $10, the coarse dialogue act is Introduction, and the finer grained negotiation strategies include Proposing price, Being informal and Talking about family for building rapport.
2

---

Published as a conference paper at ICLR 2021
Structure Encoder
Structure Encoder
Strategies
Dialogue Acts
ls
Graph
Inqulyy
Fappor
Pooling
Pooling
OCN-ASAP
GCN-ASAP
Vagus
Graph Encoding (GAT)
Graph Encoding (GAT)
Utter Utterance Encoder
A
RNM Encoder
How 530
lower
Utterance Decoder
Dialogue Context Encoder
Utterance
Utterance
p...eeeee Dil 17T010
Embedding
Generation
Figure 2: Overview of DIALOGRAPH. At time t. utterance u, is encoded using BERT and then passed to the Dialogue Context Encoder to generate the dialogue representation. This representation is enriched with the encodings of explicit strategy and dialogue act sequences using the structure encoders which is then used to condition the Utterance decoder. Please refer to for details.
2.2 STRUCTURE ENCODER
Our structure encoder is designed to model the graph representations of the strategies and dialogue acts using GATs and output their structural representations. These structural representations are used predict the next set of strategies and dialogue acts and enrich the encoded dialogue representation. Below we describe the structure encoder for negotiation strategies.
We model the sequence of negotiation strategies, ST ST,, ST.] by creating directed graph, where ST; is the set of k fine-grained negotiation strategies for the utterance u, Formally, define graph C(V, & X) with |&| edges and N | nodes where each node v, represents particular negotiation strategy for an utterance and has d-dimensional feature representation denoted by 2,. Z E N denotes the feature matrix of the nodes and A N represents the adjacency matrix, where N is the total number of nodes (strategies) that have occurred in the conversation till that point. Therefore, each node represents strategy-utterance pair.
We define the set of edges as E = {a,, b)}; a, b E V where and b denote strategies at utterances Uo and ui, present at turns ta and fg. such that 4, > f.g In other words, we make directed edge from particular node (strategy in utterance) to all the consecutive nodes. This ensures direct connection from all the previous strategies to the more recent ones 4 In the same way, we form the graph out of the sequence of dialogue acts. These direct edges and learned edge attention weights help us interpret the dependence and influence of strategies on each other.
To get the structural representations from the strategy graphs, we pass them through hierarchical graph pooling based encoder, which consists of layers of GAT, each followed by the Adaptive Structure Aware Pooling (ASAP) layer (Ranjan al. 2020). As part of the ASAP layer, the model first runs GAT over the input graph representations to obtain structurally informed representations of the nodes. Then a cluster assignment step performed which generates cluster assignment matrix, S, which tells the model which nodes come in similar structural context. After that, the clusters are ranked and then the graph is pooled by taking the top few clusters as new nodes and forming edges between them using the existing graph. This way the size of the graph is reduced at every step which leads structurally informed graph representation. We take advantage of the cluster formulation to obtain the associations between the negotiation strategies, as identified from the cluster assignment matrix. S. These association scores can later be used to interpret which strategies are associated with each other and tend to co-occur in similar contexts. Moreover, also use the node attention scores from GAT to interpret the influence of different strategies on the
AApenndx shows an example of the graph ohtained from sequence of strategies.
3

---

Published as a conference paper at ICLR 2021
representation of particular strategy, which essentially gives the dependence information between strategies.
In this way, the structure representation is learned and accumulated in manner that preserves the structural information (Ying et al., 2018 Lee et al.) 2019). After each pooling step, the graph representation is summarized using the concatenation of mean and max of the node representations. The summaries are then added and passed through fully connected layers to obtain the final structural representation of the strategies h.² We employ a similar structure encoder encode the graph obtained from the sequence of dialogue acts, to obtain h
2.3 UTTERANCE DECODER
The utterance decoder uses the dialogue context representation and structural representations of dialogue acts and negotiation strategies to produce the dialogue response (next utterance). We en- rich the dialogue representation by concatenating the structural representations before passing it to standard greedy GRU (Cho al., 2014) decoder. This architecture follows [hoou al. (2020). who introduced a dynamic negotiation system that incorporates negotiation strategies and dialogue acts via FSTs. We thus follow their utterance decoder architecture to enable direct baseline com- parison. For the jth word of utterance Ut+1, W/41, we condition on the previous word w/-1 --1 to calculate the probability distribution over the vocabulary as pi+1 softmax(GRU((t. w where h; = [h;; ; and [;] represents the concatenation operator. For encoding the price, we replace all price information in the dataset with placeholders representing the percentage of the offer price. For example, we would replace $35 with < price - 0.875 > the original selling price is $40. The decoder generates these placeholders which are then replaced with the calculated price before generating the utterance.
2.4 MODEL TRAINING
We use he to predict the next set of strategies ST i, binary value vector which represents the k-hot representation of negotiation strategies for the next turn, We compute the probability of the jth strategy occurring in u++1 as p(st(+1.)|h? (h)"") (T) = ( where a denotes the sig- moid operator. We threshold the probability by 0.5 to obtain the k-hot representation. We de- note the weighted negative log likelihood of strategies LSt as the loss function of the task of next strategy prediction LSt - -E, log(p(ste+1,)) Σx log(1 p(st++1,x)) where the summation of are over the strategies present (sf = 1) and not present (st 0) in the ground truth strategies set, ST'. Here 5, is the positive weight associated with the particular strategy. We add this weight the positive examples to trade off precision and recall. We put 6j # of instances not having strategy j/ of instances having strategy j-
Similarly, we use hea to predict the dialogue act for the next utterance day+1 Given the target dialogue act daj-1 and the class weights Pdo for the dialogue acts, we denote the class-weighted cross entropy loss over the set of possible dialogue acts, LDA = PPa log (softmaxx((. (h We pass h = [h;; h.5T hdd] through a linear layer to predict the negotiation success, which denoted by the sale-to-list ratio (Zhou et al., 2019). We split the ratios into 5 negotiation classes of equal sizes using the train- ing data and use those predict the success of negotiation. Therefore, given the predicted probabilities for target utterance 2441 from $2.3 target ratio class y, and the learnable param- eters W, and by. we use the cross entropy loss as the loss for the generation task (CNLg) as well the negotiation outcome prediction task (LR), thus L = Ew, En4y, log(p(i)) and
(sale price buyer target price)/ /(lsted price buyer target price)
CR = - -re(L.o) y Log(softmax(W: (W,h; + b,)).. The LR loss optimizes for encoding negotiation strategies to enable accurate prediction of negotiation outcome.
We use hyperparameters a, B and y to optimize the joint loss fornt, of strategy prediction, dialogue act prediction, utterance generation and outcome prediction together, using the Adam optimizer (Kingma & Ba 2014), get Ljoint LNLG + oL BLDA + CLn-
4

---

Published as a conference paper at ICLR 2021
3 EXPERIMENTAL SETUP
We use the CraigslistBargain datase (He et al. 2018) to evaluate our model. The dataset was created using Amazon Mechanical Turk (AMT) a negotiation setting where two work- ers were assigned the roles of buyer and seller respectively and were tasked to negotiate the price of an item on sale. The buyer was additionally given a target price. Both parties were encouraged to reach agreement while each of the workers tried to get a better deal. We remove all conversations with less than 5 turns. Dataset statistics are listed in Table in the Appendix.
Dataset:
We extract from the dataset the coarse dialogue acts described by He et al. (2018). This includes a list of 10 utterance dialogue acts, e.g., inform, agree, counter-price. We augment this list by 4 outcome dialogue acts, namely, (offer), (accept), (reject) and (quit), which correspond to the actions taken by the users. Negotiation strategies are extracted from the data following Zhou et al. 2019). These include 21 fine-grained strategies grounded in prior economics/behaviore science
research on negotiation (Pruitt 2013; Bazerman & Neale 1993 Bazerman et al. [2000a] Fisher et al.)
2011 Lax& Sebenius |006] Bazerman al. 20000)), c.g. negotiate side offers, build rapport, show
dominance. All dialogue acts and strategies are listed in Appendices A and
DIALOGRAPH refers to our proposed method. To corroborate the efficacy of DI- ALOGRAPH, we compare it against our implementation of the present state-of-the thette-attttttt model for the negotiation task: FST-enhanced hierarchical encoder-decoder model (FeHED) (Zhou et al. 2020) which utilizes FSTs for encoding sequences of strategies and dialogue acts 6 We also conduct and ablation study, and evaluate the variants of DIALOGRAPH with different ways of encoding nego- tiation strategies, namely, HED, HED+RNN, and HED+ Transformer. HED completely ignores the strategy and dialogue act information, whereas HED+RNN and HED++raanforns
Baselines:
encode them using RNN and Transformers (Vaswani al. 2017) respectively. While HED+RNN is based on the dialogue manager of He et al. (2018). HED+ TIED+TTranee Transformer has not been proposed earlier for this task. For fair comparison, we use pre-trained BERT (Devlin al. 2019) model the utterance encoder (.T)) and a common utterance decoder ($2.4) in all the models, and only vary the struc- ture encoders as described above. The strategies and dialogue acts in RNN and Transformer based encoders are fed as sequence of k--oo vectors.
For evaluating the performance on the next strategy prediction and the next dialogue act prediction task, we report the F1 and ROC AUC scores for all the models. For these metries, macro scores tell us how well the model performs on less frequent strategies/dialogue acts and the micro performance tells us how good the model performs overall while taking the label imbalance into account. Strategy prediction a multi-label prediction problem since each utterance can have multiple strategies. For the downstream tasks of utterance generation, we compare the models using BLEU score (Papineni et al., 2002) and BERTScore (Zhang et al., 2020). Finally, also evaluate on another downstream task of predicting the outcome of negotiation. using the ratio class prediction accuracy (RC-Acc) (1 out of negotiation outcome classes, as described in $2.4). Predicting sale outcome provides better interpretability potentially control to intervene when negotiation has a bad predicted outcome. Additionally, being able to prediet the sale outcome with high accuracy shows that the model encodes the sequence of negotiation strategies well.
Evaluation Metries:
the progression of sale and
RESULTS
We evaluate (1) strategy and dialogue act prediction (intrinsic evaluation), and (2) dialogue genera- tion and negotiation outcome prediction (downstream evaluation). For all metrics, we perform boot- strapped statistical tests Berrg-Kirkpatrich et al. 2012 Koehn 2004) and we bold the best results for metric in all tables (several results are in bold if they have statistically insignificant differences).
Strategy and Dialogue Act Prediction: We compare MAALOGRAPI'S effectiveness encoding the explicit sequence of strategies and dialogue acts with the baselines, using the metrics described in 3 Table [ shows that DIALOGRAPH performs on par with the Transformer based encoder in
httpp::/ thub.
st fordn fornn http:/ccca///////eo//occccccccc/ http://aaeerroooooooooe/eereerre
istbarga
We replace the utterance encoder with BERT for fair comparison. This improved slightly the performance of the FeHED model compared results published in Zhou al. (022).
5

---

Published as a conference paper at ICLR 2021
Table 1: Performance of the next strategy and dialogue-act prediction of various models. We report the Fl and ROC AUC scores. Significance tests were performed as described in $ and the best results (along with all statistically insignificant values) are bolded.
Negotiation Strategies
Dialogue Acts
F1
ROC AUC
ROC ALC
Model
Macro
Micro
Weighted
Macro
Micro
Weighted
Macro
Micro
Weighed
Macro
Weighed
FeHED
17.6
25.6
36.3
55.8
61.7
54.7
20.6
37.4
30.6
76.9
79.2
HED+RNN
23.2
26.7
42.4
65.3
65.3
60.4
33.0
46.2
42.8
83.1
84,2
HED+Transfomner
26.3
32.1
43.3
68.2
71.8
61.8
32.5
44.6
42.0
85.6
85.1
DIALOGRAPH
26.1
34.1
43.5
68.1
73.0
61.8
33.4
45.8
43.7
85.6
85.4
strategy prediction macro scores and outperforms it on other metries. Moreover, both significantly outperform the FST-based based method, prior state-of-the-art. We hypothesize that lower gains for dialogue acts are due to the limited structural dependencies between them. Conversely. validate that for negotiation strategies, RNNs are significantly worse than DIALOGRAPH. We also observe that higher macro scores show that DIALOGRAPH and Transformers are able capture the sequences containing the less frequent strategies/dialogue acts as well. These results supports our hypothesis of the importance to encode the structure more expressive model. Moreover, DIALOGRAPH also provides interpretable structures which the other baselines do not. We will discuss these findings in $
Automatic Evaluation on Downstream tasks: In this section, we analyze the impact of DIALO- GRAPH on the downstream task of Negotiation Dialogue based on the automatic evaluation metries described in 3 In Table 2 we show that DIALOGRAPH helps improve the generation of dialogue response. Even though DIALOGRAPH attains higher BLEU scores, we note that single-reference BLEU assumes only one possible response while dialogue systems can have multiple possible re- sponses to the same utterance. BERTScore alleviates this problem by scoring semantically similar responses equally high (Zhang et al. 2020). We also find that both Transformer and DIALOGRAPH have a comparable performance for negotiation outcome prediction, which is significantly better than the previously published baselines (FeHED and HED+RNN). A higher performance on this metric demonstrates that our model is able to encode the strategy sequence better and consequently predict the negotiation outcome more accurately. Additionally, ablation results in Table show that both strategy and dialogue act information helps DIALOGRAPH in improving dialogue response. The difference in BERTScore Fl scores in Tables 2nd 3 arises due different metrics chosen for early stopping, More details in Appendix D
Although, both HED+Transformer and DIALOGRAPH are based on attention mechanisms, DIALO- GRAPH has the added advantage of having structural attention which helps encode the pragmatic structure of negotiation dialogues which in turn provides an interpretable interface. The compo- nents in our graph based encoder such as the GAT and ASAP layer provide strategy influence and cluster association information which is useful to understand and control negotiation systems. This is described in more detail in $5 Though transformers have self attention, the architecture is limited and doesn't t model the structure/dependenee between strategies providing only limited understand- strong
ing. Further, our results show that DIALOGRAPH maintains improves performance models like Transformer and has much more transparent interpretability. We later show that DI- ALOGRAPH performs significantly better than HED+Trannformer in human evaluation.
Since automatic metrics only give us a partial view of the system, we com- plement our evaluation with detailed human evaluation. For that, set up DIALOGRAPH and the baselines on Amazon Mechanical Turk (AMT) and asked workers to role-play the buyer and ne- gotiate with single bot. After their chat over, we ask them fill a survey rate the dialogue on how persuasive (My task partner was persuasive.), coherent (My task partner's 's responses were on topic and in accordance with the conversation history.), natural (My task parmer was human- like.) and understandable (My task partner perfectly understood what I was typing.) the bot was Prior research entailment has shown that humans tend to get better as they chat (Mizukami al, 2016 Beñus et al. 2011) and so we restrict one user to chat with just one of the bots. We further
Human Evaluation:
We use the of httpth:://ttthbbbbocttoc
Screenshots in Appendix H

---

Published as a conference paper at ICLR 2021
Table 2:
Downstream evaluation of negotiation dia- logue generation and negotiation outcome prediction. The best results (along with all statistically insignifi- cant values to those) are bolded.
Table 3: DIALOGRAPH ablation analy- sis. This shows that all the different com- ponents provide complementary benefits. We also evaluate without BERT for com- parison with previously published works.
Generation
Outcome
BERe
Prediction
Model
BLEU
Precision
Recall
F1
RC-Ace
Model
BERT Score F-1
HED
20.9
21.8
22.3
22.1
35.2
DIALOGRAPH w/o Strategy (ST)
27.4
FelED
23.7
27.1
26.8
27.0
42.3
26.8
HED+RNN
22.5
22.9
22.7
22.8
47.9
w/o ST. Dialogue Acts (DA)
26.3
HEDdTransfomer
24.4
27.4
28.1
27.7
53.7
w/o ST, DA, BERT
22.7
DIALOGRAPH
24.7
27.8
28.3
28.1
53.1
Seller (Bot)
Hey, you need routery
Trade In
Hedge
Yroh, mase broking or the Apple ruser
Informal
Propose
Family
chi it suppettamliiie desices...
x" Person
Frestt will support taphop?
1.30
Informal
Uy
inttlyyyy
Us
0.87
0.37
Positive
Posttive
Mah.. few monthe only monongg.
Concern
Buyer (Human)
UG
Figure 3: Visualization of the learnt latent strategy sequences DIALOGRAPH where bolder edges represent higher influence. Here present only a few edges for brevity and visualize min-max nor- malized attention values as edge weights to analyze the relative ranking of strategies. For example, for family u7. informal of us has the most influence followed by propose. We present the full attention map for this example in Figure 5 in the Appendix.
prune conversations which were incomplete potentially due to dropped connections. Finally, we manually inspect the conversations extracted from AMT to extract the agreed sale price and remove conversations that were not trying to negotiate at all.
The results of human evaluations of the resulting 90 dialogues (about 20 per model) are presented in Table 4 We find that baselines are more likely to accept unfair offers and apply inappropriate strategies. Additionally, DIALOGRAPH bot attained a significantly higher Sale Price Ratio, which is the outcome of negotiation, showing that effectively modeling strategy sequences leads to more effective negotiation systems. Our model also had higher average total number of turns and words- per-turn (for just the bots) compared to all baselines, signifying engagement. It was also more persuasive and coherent while being more understandable to the user. From qualitative inspection we observe that the HED model generates utterances that are shorter and less coherent. They are natural responses like "Yes it is". but generic and contextually irrelevant. We hypothesize that this is due to the HED model not being optimized to encode the sequence of negotiation strategies and dialogue acts. We believe that this the reason for the high natural score for HED. From manual inspection we see that HED is not able to produce very persuasive responses. We provide an example of a dialogue Appendix[] We see that although DIALOGRAPH achieves better sale price outcome it tries to repeatedly offer deals to negotiate the price. We see that the HED is unable to understand the user responses well and tends to repeat itself. Both the FeHED and HED baselines tend to agree with the buyer's proposal more readily whereas
model performs well,
and DIALOGRAPH provide counter offers and trade-ins to persuade
the user.
INTERPRETING LEARNED STRATEGY GRAPHS
We visualize the intermediate attention scores generated by the GATs while obtaining the strategy node representations. These attention scores tell us what strategies influenced the representation of particular strategy and can be used observe the dependence between strategies (cf. Xie & Lu
7

---

Published as a conference paper at ICLR 2021
Table Human evaluation ratings on scale of 1-5 for various models. We also provide the average sale price ratio (224) Negative ratio means that average sale price was lower than the buyer's ss target.
Model
Persuasive
Coherent
Natural
Understandable
Sale Price Ratio
Avg Turns
Avg words/turn
HED
2.50
2.50
4,50
2.50
-2.13
11.00
4.25
FeHED
3,30
3.75
3.70
3.69
0.25
14.30
5.76
HED+RNN
2.81
3.27
3.36
3.27
-3.68
13,90
3.61
HED+ Tranffrrofommer
3.50
3.50
3,70
3.40
-0.07
11.40
4.36
DIALOGRAPH
3.58
3.94
3.75
3,70
0.49
15.72
5.84
Table 5: Examples of strategies and their least highly associated strategies based on association scores extracted using the cluster attention scores given by the ASAP layer.
Negotiation Strategy
Least associative strategies
Highly associative strategies
concern
certainty (0.1759), trade in (0.228)
politeness please (0. (0.70) politeness gratitude (0.5859)
hedge
trade in (0.4367). pos sentiment (0,4501)
propose (0.5427) friend (0.6218)
propose
factive count (0.3878), family (0.416)
politeness gratitude (0.5048), trade in (0.5223)
negutive sentiment
trade in (0.3089), informal (0.3644)
family (0.6363), propose (0.6495)
2019
Norcliffe- Browee--Bwwwnee et al., 2018). We show an example in Figure 3 where for brevity, we present a subset of few turns and only the top few most relevant edges in the figure. For visualization, we re-scale the attention values for all incoming edges of a node (strategy) using min-max normaliza- tion. This is done because the range of raw attention values would differ based on the number of edges and this allows us to normalize any difference in scales and visualize the relative ranking of strategies (Yi et al.) 2005 Chen & Liu 2004). We notice that as soon the first propose us hap- pens, the strategies completely change and become independent of the strategies before the propose point. From Figure 3 see that the edge weight from i, to ug is 0.01, signifying very low influ- ence. We noticed this trend in other examples as well, wherein, the influence of strategies coming before the first propose turn strategies coming after that, is very low. A similar phenomenon was also observed by Zhou et al. (2019) who study the conversations by splitting into two parts based on the first propose turn. Another interesting thing we note is that the trade-in and propose strate- gies at Us seem to be heavily influenced by informal from
Similarly, the informal of us was influenced by positive sentiment from u4. This indicates that the seller was influenced by previous informal interactions to propose and irade-in& at this turn, and that sellers tend to be more informal the conversation partner positive. In other examples, we see that a particular utterance, dif- ferent strategies depend separate past strategies and also observe that the attention maps usually demonstrate the strategy switch as soon as the first propose happens, which similar to what has been observed by prior work. These examples demonstrate that DIALOGRAPH can model fine grain strategies, learn dependence beyond just utterances and give interpretable representations, which previous baselines, including the FSTs, lack. Specifically, each state of the FST is explicitly rep- resented by an action distribution which can only be used to see the sequence of strategies and not observe associations or dependence information which DIALOGRAPH provides.
We utilize these cluster attention scores from the ASAP pooling layer to observe the association between various strategies which can help us observe strategies with similar contextual behaviour and structural co-occurrence. We take the average normalized value of the cluster attention scores between two strategies obtain the association score between them. In Table 5 we show some examples of strategies and their obtained association scores. We observe that negative sentiment tends to be most associated to propose. We hypothesize that this is because that people who disagree more tend to get better deals. We observe that people do not tend to associate negative sentiment with trade-in, which is in-fact highly associated with positive sentiment, because people might want to remain positive while offering something. Similarly, people tend to give vague proposals by hedging, for instance, could go lower if you can pick it up, than when suggesting trade-in. Concern also seems to be least associated with certainty. and most with politeness-basee strategies. Thus, observe that our model is able to provide meaningful insights which corroborate prior observations, justifying its ability to learn strategy associations well.
8

---

Published as a conference paper at ICLR 2021
6 RELATED WORK
Goal-oriented dialogue systems have a long history in the NLP community. Broadly, goal-oriented dialogue can be categorized into collaborative and non-collaborative sys- tems. The aim of agents in a collaborative setting is to achieve a common goal, such as travel and flight reservation (Wei al.] 2018) and information-seeking (Reddy al.) 2019]. Recent years have seen a rise in non-collaborative goal-oriented dialogue systems such as persuasion (Wang et al., 2019, Dutt al.,
Dialogue Systems:
2021), negotiation (He et al.) 2018 Lewis et al.. 2017) and strat- egy games [Asher et al. 2016) due to the challenging yet interesting nature of the task. Prior work has also focused on decision-makeee
2020;
games such as Settlers of Catan (Cuayáhuitl et al. 2015) which mainly involve decision- decision-making skills rather than communication. Lewis et al. (2017) developed the DealOrNoDeal dataset in which agents had to reach a deal to split a set of items. Extensive work has been done on capturing the explicit semantic history in dialogue systems (Kumar et al.) [2020)
Vinyals & Le 2015 Zhang et al. 2018). Recent work has shown the advantage of modeling the dia- logue history the form of belief span (Lei et al. 2018) and state graphs (Bowden et al. 2017). He
ct al.
2018 proposed a bargaining scenario that can leverage semantic and strategic history. Zhou al. 2020 used unsupervisedly learned FSTs to learn dialogue structure. This approach, however, although effective in explicitly incorporating pragmatic strategies, does not leverage the expressive power of neural networks. Our model, in contrast, combines the interpretablity of graph- ap- proaches and the expressively of neural networks, improving the performance and interpretability of negotiation agents.
Graph Neural Networks: The effectiveness of GNNs (Bruna et al. 2013 Defferrard et al. 2016
Kipf & Welling. 2017) has been corroborated in several NLP applications Vashishth et al. 2019).
including semantic role labeling (Marcheggiani & Titov 2017). machine translation [Battings et al.
2017). relation extraction (Vashishth al. 2018). and knowledge graph embeddings (Schlichtkrull et al. 2018 Vashishth et al.) 2020). Hierarchical graph pooling based structure encoders have been successful in encoding graphical structures (Zhang al. 2209]. We leverage the advances in GNNs and propose to use graph-based explicit structure encoder to model negotiation strategies. Unlike HMM and FST based encoders, GNN-based encoders can be trained by optimizing the downstream loss and have superior expressive capabilities. Moreover, they provide better interpretability of the model as they can be interpreted based on observed explicit sequences (Tu et al., 2020 Norcliffe- Brown al. 2018). In dialogue systems, graphs have been used to guide dialogue policy and response selection. However, they have been used to encode external knowledge (Tuan et al. 2019) Zhou 2018) speaker information (Ghosal et al. 2019),, rather than compose dialogue strate- gies on-the-fly. Other works (Tang et al. [019 Qin et al. 2020) focused on keyword prediction using RNN-based graphs. Our work is the first incorporate GATs with hicrarchical pooling, learning pragmatic dialogue strategies jointly with the end-to-end dialogue system. Unlike in prior work, our model leverages hybrid end-to-end and modularized architectures (Liang et al. 2020 Parvaneh et al. 2019) and can be plugged as explicit sequence encoder into other models.
7 CONCLUSION
We present DIALOGRAPII, a novel modular negotiation dialogue system which models pragmatic negotiation strategies using Graph Attention Networks with hierarchical pooling and learns plicit strategy graph jointly with the dialogue history. DIALOGRAPH outperforms strong baselines in downstream dialogue generation, while providing the capability interpret and analyze the in- termediate graph structures and the interactions between different strategies contextualized in the dialogue. As future work, we would like to extend our work to discover successful (c.g.: good for the seller) and unsuccessful strategy sequences using our interpretable graph structures.
ACKNOWLEDGMENTS
The authors are grateful to the anonymous reviewers for their invaluable feedback, and to Alissa Ostapenko, Shruti Rijhwani, Ritam Dutt, and members of the Tsvetshop at CMU for their helpful feedback on this work. The authors would also like to thank Yiheng Zhou for helping with ne- gotiation strategy extraction and FeHED model. This material is based upon work supported by the National Science Foundation under Grant No. IIS2007960 and by the Google faculty research award. We would also like to thank Amazon for providing GPU credits.

---

Published as a conference paper at ICLR 2021
REFERENCES
Nicholas Asher, Julie Hunter, Mathicu Morey, Benamara Farah, and Stergos Afantenos. Discourse structure and dialogue acts in multiparty dialogue: the STAC corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC' 16), pp. 2721- 2727, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https: www ac web. org wwb.orrg/athoooogg///
1432
Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima an. Graph convolu- tional encoders for syntax-award neural machine translation. In Proceedings of the 2017 Confer- on Empirical Methods in Natural Language Processing, pp. 1957-1967, Copenhagen, Den- mark, September 2017. Association for Computational Linguistics. doi: 10. 10D18653/1/DI1--10000
URL https: www. a c web org http://wwwwwe.ooooooooooooooooo/
Max H Bazerman and Margaret Ann Neale. Negotiating rationally. Simon and Schuster, 1993.
Max H Bazerman, Jared R Curhan, Don A Moore, and Kathleen L Valley. Negotiation. Annual review of psychology, (1):1)77 2000a.
Max H. Bazerman, Jared R. Curhan, Don A. Moore, and Kathleen L. Valley. Negotiation. An- mual Review of Psychology, 51(1):279-314 2000b. doi: 10.1 http://annrre..cc..5...... URL PMID: 10751973.
bttps::/doi.org/// otw.://.......16..a..e......rraa
Štefan Beñuš, Agustín Gravano, and Julia Hirschberg. Pragmatic aspects of temporal accommoda- tion in turn-taking. Journal of Pragmatics, 43(12):3001-3027, 2011.
Taylor Kinngkiirkat David Burkett, and Dan Klein. An empirical investigation of statistical significance in NLP. In Proceedings of the 2012 Joini Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. pp. 995-1005, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL hRLhttps::/wwrr
www.weborgg/aeo wttp.//rrroatttoooo////222 1091
Kevin K Bowden, Shereen Oraby, Jiaqi Wu, Amita Misra, and Marilyn Walker. Combining search with structured data to create more engaging user experience in open domain dialogue. ICTIR' 17 Workshop on Search-Oriente Conversational Al (SCAI' 2017), 2017.
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. CoRR, abs/1312.6203, 2013. URL latt:://axiv..rr
abs/1312. 6203
Keke Chen and Ling Liu. Clustermap: Labeling clusters in large datasets via visualization. In Proceedings of the Thirteenth ACM International Conference on Information and Knowledge Management, CIKM '04, pp. 285-293, New York, NY, USA, 2004. Association for Computing Machinery, ISBN 1581 138741. doi: 10.1145/103111111033
URL htttts::////tt o
10. 10.1145/103111 01445/1031177 031233
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn mccoder-dcoodd for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1734. Association for Computational Lin- guistics, 2014. doi: 10.3115/v1/D14-1111 119.. URI. http:/tttp/...m. /www a acl web. wtgp/otholooggy...o.oo D14 014-1179
Heriberto Cuayáhuitl, Simon Keizer, and Oliver Lemon. Strategic dialogue management via deep reinforcement learning. NIPS' 15 Workshop on Deep Reinforcement Learning, 2015.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. CoRR, 00617600099755 2016. URI. http:
// //aaxivv. wwwooraaaaa abs/ 60..666633333 09375
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
10

---

Published as a conference paper at ICLR 2021
Technologies. Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/1//N19-44233 URL hetps:
/ww. wwwwwb.wwb.... org/ http//atoooooy/111114/00
Ritam Dutt, Rishabh Joshi, and Carolyn Rose. Keeping up appearances: Computational modeling of face acts in persuasion oriented discussions. In Proceedings of the 2020 Conference Em- pirical Methods in Natural Language Processing (EMNLP), pp. 7473-7485, Online, November 2020. Association for Computational Linguisties. doi: 10. 18155//1/2000..coooooo emnlp- maapp-mmin.605... 605. URL
https:// wwwww/wwww.. ac www.wb................ org/ htt..//rr//aooooo//222220002 www.:22000.mmmmmmmmmmmmmmmmmm em- 605
Ritam Dutt, Sayan Sinha, Rishabh Joshi, Surya Shekhar Chakraborty, Meredith Riggs, Xinru Yan, Haogang Bao, and Carolyn Penstein Rosé. Resper: Computationally modelling resisting strate- gies in persuasive conversations. In 16th Conference of the European Chapter of the Association for Compulational Linguistics (EACL), 2021.
Roger Fisher, William L Ury, and Bruce Patton. Getting to yes: Negotiating agreement without giving in. Penguin, 2011.
Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. DialogueGCN: A graph convolutional neural network for emotion recognition in conversation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing 154-164, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10.11865//1//D11-1101 URL htttss::::// /wwp::::/wwww... ac web. wtt.o///a..oooogggg11-11--111111
pp.
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. Decoupling strategy and genera- tion in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. URL https: www nlp stanford. edu/ edu/pubs.
112018decouple.pde
Wenxiang Jiao, Haiqin Yang, Irwin King, and Michael R. Lyu. HiGRU: Hierarchical gated current units for utterance-level emotion recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan-
guage Technologies, Volume (Long and Short Pupers), pp. 397-406, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18655/1//N19-1037. URL
https: www. wwwwwwwwwwb......c.......www...w .ww./org//oooo haa.tonnnhhoggo//ooooo 10100y/NN19111111
Diederik P. Kingma and Jimmy Ba. Abs/1412.6968 2014.
Adam: A method for stochastic optimization. CoRR,
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations (ICLR), 2017.
Philipp Koehn. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference Empirical Methods in Natural Language Processing, pp. 388-395, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL acl wtgebborg/aan http://annho1ll//000-0-- how/hll4-oyo//2//0
Gaurav Kumar, Rishabh Joshi, Jaspreet Singh, and Promod Yenigalla. AMUSED: A multi-stream vector representation method for use natural dialogue. In Proceedings of The 12th Language Resources and Evaluation Conference, pp. 750-758, Marseille, France, May 2020. European
Language Resources Association. ISBN 179-10095544-3444 URL URLhttts::/ www http://wwwww..b.b.///p..
orq/ http//athoooyy//00 1rec-1. 94
George Larionov, Zachary Kaden, Hima Varsha Dureddy, Gabriel Bayomi T Kalejaiye, Mihir Kale, Srividya Pranavi Potharaju, Ankit Parag Shah, and Alexander I Rudnicky. Tartan: A retrieval-based socialbot powered by a dynamic finite-state machine architecture. arXiv preprint arXiv:1812.01260. 2018.
David A Lax and James K Sebenius. 3-D Negotiation: Powerful tools to change the game in your most important deals. Harvard Business Press, 2006.

---

Published as a conference paper at ICLR 2021
Junhyun Lee, Inyeop Lee, and Jaewoo Kang. Self-attention graph pooling. In Proceedings of the 36th International Conference on Machine Learning, 09-15 Jun 2019.
Wenqiang Lei, Xisen Jin, Min- Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. Sequicity: architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1437-1447, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/113 133. URL www. c web www.bb.ogggaanooooooooooooo P18 1133
Simplifying task-oriented dialogue systems with single
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. Deal deal? end- to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2443-2453, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/1//117-1222 URL https:
7/ /www. wwtp:alwwwb..ogg/aaaoooo wwt.e//www.o///ttt.ooooo///----0
Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. A
persona- based neural conversation model. Proceedings of the 54th Annual Meeting the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), Pp. 994-90041 Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16 10944 URL
https:/ www. ac web. www.orrg/aao antho ogy 094
Yu Li, Kun Qian, Weiyan Shi, and Zhou Yu. End-to-end trainable non-collaborative dialog sys- tem. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty- Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12, 2020, pp. 8293-8302. AAAI Press, 2020. URL https: http://aaaa...o wtt.a..gajaaa////ooo index. php/ AAA wal.AA//articeeevveeeee 6345
Weixin Liang, Youzhi Tian, Chengeai Chen, and Zhou Yu. MOSS: end-to-end dialog system frame- work with modular supervision. In The Thirty-Fourth Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Seccry-Second Innovative Applications of Artificial Intelligence Confer- ence, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8327-8335. AAAI Press, 2020. URL https: /aaasi www.rrggoo ojs/ indii.deni..mmmm www.nnnwpp/eiiiii....mmppppppppp /AAI/AAAAAAAA http:///A///atiicceeeeeeee
6349
Diego Marcheggiani and Ivan Titov. Encoding sentences with graph convolutional networks for
semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pp. 1506-1515, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-11155 URL httph::////// //www aclweb.
orc//anthoology//11--115
Masahiro Mizukami, Koichiro Yoshino, Graham Neubig, David Traum, and Satoshi Nakamura. Analyzing the effect of entrainment on dialogue acts. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 310-318, Los Angeles, September 2016. Association for Computational Linguistics. doi: 10. 18.18653/v 1/W16336000 URL https:
w/www.c. www.wb.wlwb.... org/ wttp///atoooog////ieeeoe.ee 3640
Will Morcliffe-Brown Stathis Vafeias, and Sarah Parisot. Learning conditioned graph structures for interpretable visual question answering. In Advances in neural information processing systems, pp. 8334-8343, 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 1033115/10330331073333 URL https:
// /www. ac web. org http://aantooooooooooo11000
Amin Parvanch, Ehsan Abbasnejad. Qi Wu, and Javen Shi. Show, price and negotiate: A hierarchical attention recurrent visual negotiator. CoRR, abs/ http/19550072/// 2019. URL http: //arxiiv.
org/ 0100 03721
12

---

Published as a conference paper at ICLR 2021
P. E. Pope, S. Kolouri, M. Rostami. C. E. Martin, and H. Hoffmann. Explainability methods for graph convolutional neural networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10764-10773, 2019.
Dean G Pruitt Negotiation behavior. Academic Press, 2013.
Jinghui Qin, Zheng Ye, Jianheng Tang, and Xiaodan Liang. Dynamic knowledge routing network for target-guided open-domain conversation. Proceedings of the AAAI Conference on Artificial Intelligence, (4(05))8657-8666 Apr 2020. ISSN 2159-5399. doi: URL http:
www.:///..........1.0 1609/ v341 00.4005..0 6390
Ekagga Ranjan, Soumya Sanyal, and Partha P. Talukdar. ASAP: adaptive structure aware pooling for learning hierarchical graph representations. In The Thirty-Fourth AAAI Conference on Artifi- cial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances Artificial Intelli- gence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 5470-5477, AAAI Press, 2020.
URL https: http://aaii...... org/ O / index. whpp/AA. /tappAAAAA//rrille/eee/e///
Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question an- swering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, March 2019. doi: 10.1162/tacl URL htt.t:://///.... http://www..lwe...broo...o wwwp:aawwwwbb.orr/aaaccoooooogo
019-1016
Alan Ritter, Colin Cherry, and Bill Dolan. Unsupervised modeling of twitter conversations. In Hu- man Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 172-180. Los Angeles, California, June 2010. As-
sociation for Computational Linguistics. URL https::////........... /www
hwtp:///wwwwwwwww../o/.. wwwweb. wwtp://w......ttttoooooogog.oo/.
NIC-1020
M. Schlichtkrull, Thomas Kipf, P. Bloem, R. V. Berg, Ivan Titov, and M. Welling. Modeling rela- tional data with graph convolutional networks. In ESWC, 2018.
Weiyan Shi. Tiancheng Zhao, and Zhou Yu. Unsupervised dialog structure learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1797-1807, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/
v1/N19-1178. Utt.httss::/ httttp:::/wwww.eee. ac ott.:ttt.s//w..eeedeee..rrttttta wttpp//www.....r.aaaoooooo////// -ttp://nnnhoollly1111111111 9-1781
Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. A neural network approach context-sensitive gen- cration of conversational responses. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 196-205, Denver, Colorado, May-June 2015. Association for Computational Linguistics. doi: 10.31 15111 1020. URL https: www. awwweb.www...... org/ http///tthhoooo11l11111
Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric Xing, and Zhiting Hu. Target- guided open-domain conversation. In Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 5624-5634 5634, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/11/P19-1555 URLhttps: / www. ac web.......... org/
lantholoy/P19- 1565.
Leigh L Thompson. The mind and heart of the negotiator, volume 3.
Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang, Xiaodong He, and Bowen Zhou. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents. In AAAI 2020 (accepted), 2020.
Yi-Lin Tuan, Yun-Nung Chen, and Hung-yi Lee. DyKgChat: Benchmarking dialogue generation grounding dynamic knowledge graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural
Language Processing (EMNLP-IJCNLP), pp. 1855-1865, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.19-93/v1/D1-- 194. URL hhtps:: //www.
acl web. wtt.oorg//aaaaaaa http://annnoooooy/11-11111
13

---

Published as a conference paper at ICLR 2021
Shikhar Vashishth, Rishabh Joshi, Sai Suman Prayaga, Chiranjib Bhattacharyya, and Partha Taluk- dar. RESIDE: Improving destantly-supervisst neural relation extraction using side information. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing, pp. 1257-1266, Brussels, Belgium, October-November 2018. Association for Computational Linguisties. URL htt.lhtttp:/e.. http://wwwwwwbrrr.. org/ http:/oorr/aattooooo/////----11- http://p/g.p////.o.....cccaaaaaa
Shikhar Vashishth, Naganand Yadati, and Partha Talukdar. Graph-based deep learning in natural language processing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Process- ing (EMNLP-IJCNLP)) Tutorial Abstracts, Hong Kong, China, November 2019. Association for Computational Linguistics.
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. Composition- multi- relational graph convolutional networks. International Conference on Leurning Representa- lions, 2020. URL htttp:://penneeeeeeeee
net met/fooumi: meet/forumidd-ByyB..... 1d-ByLA_CAtPr
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention all you need. In Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998 6008. Curran Associates, Inc., 2017. URL all youu-need..
attention
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
accepted as poster.
Oriol Vinyals and Quoe Le. A neural conversational model. arXiv preprint
2015.
Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. Persuasion for good: Towards a personalized persuasive dialogue system for social good. arXiv preprint vrXiv:1906.0677. 06725, 2019.
Wei Wei, Quoe Le, Andrew Dai, and Jia Li. AirDialogue: An environment for goal-oriented dialogue research. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3844-3854, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/w1/D18-1419 axc/annthooory/DDl--1111
httt:t/ /wwww/ aclweb.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and doi: 10.1109/amss.2200027777733 URL attp://dxx doi. org. 10. 1109/ 1111//NNLSS...... 2020 2978386
Learning Systems, pp. 1-21, 2020. ISSN 2162-2388.
Shangsheng Xie and Mingming Lu. Interpreting and Understanding Graph Convolutional Neural Network using Gradient-baid based Attribution Method. arXiv e-prints, art. 2019.
March
Soo Yi, Rachel Melton, John Stasko, and Julie Jacko. Dust & magnet: Multivari- ate information visualization using magnet metaphor. Information Visualization, 4(4):239- 256, 2005. doi: 10. http:pallaave.v..500000. URL https:://cct otr/ii.rg111....0 1057/ palgrave ivs. 9500099
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hi- erarchical graph representation learning with differentiable pooling. In Advances in neural infor- mation processing systems, pp. 4800 4810, 2018
S. Young. Using pomdps for dialog management. In 2006 IEEE Spoken Language Technology Workshop, pp. 8-13, 2006.
Ke Zhai and Jason D. Williams. Discovering latent structure in task-oriented dialogues. In Pro- ceedings of the 52nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 36-46, Baltimore, Maryland, June 2014. Association for Computational Linguistics. doi: 10.3115/v1/P44-1 1004. URL https:////// wwww. aclweb. org/anthology.
P P14-1004
14

---

Published as a conference paper at ICLR 2021
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Per- sonalizing dialogue agents: have a dog, do you have pets too? In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 2204-2213, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18- 1205.
wwww ac www.wee....ggg//aooooooo........
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evalu- ating text generation with bert. In International Conference on Learning Representations, 2020. URL htt.ps://ppenreviiii www.:neetooorrnn htt.//orrummiiikkeeeecceetee id=SkHHUVFFDT
Zhen Zhang, Jiajun Bu, Martin Ester, Jianfeng Zhang, Chengwei Yao, Zhi Yu, and Can Wang. Hierarchical graph pooling with structure learning. arXiv preprint arXiv:1911.05954 2019.
Hao Zhou, Tom Young, Minlie Huang. Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. Common- sense knowledge aware conversation generation with graph attention. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI' pp. 4623-4629, AAAI Press, 2018. ISBN 9780999241127.
Yiheng Zhou, He He, Alan W Black, and Yulia Tsvetkov. A dynamic strategy coach for effective negotiation. Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pp- 367-378, Stockholm, Sweden, September 2019. Association for Computational Linguistics. doi: 10.18653lv//W1995900 URL wwwp://wbbwwwwwwwbbbbbb.........
Yiheng Zhou, Yulia Tsvetkov, Alan W Black, and Zhou Yu. Augmenting non-collaborative dialog systems with explicit semantic and strategic dialog history. In International Conference Learn- ing Representations, 2020. URL http:///eneeeeeeeeeeeeorruraaaae
15

---

Published as a conference paper at ICLR 2021
A DIALOGUE ACTS
Here we provide the details about the dialogue acts that we have used to annotate the utterances. 10 taken from He al. (2018) and based on the actions taken by the users. The rule based acts are extracted using the code provided by them R The details are in Table 6
Table 6: The list of dialogue acts that we use to annotate the data.
Meaning
Dialogue Act Example
Detector
Greetings
intro
would love to buy
rule
Ask a question
inquiry
Sure, what's your price
Propose the first
init-price
I'm budget
could do $5
rule
Proposing counter price Unknown
counter-price pricer
How about $15 and I'll waive the deposit
rule
unknown
Hmm, let me think
rule
Agree with the proposal
agree
That works for me
rule
Disagree with a proposal
disagree
Sorry can't It agree to that
rule
Answer a question
inform
This bike brand
Using comparatives with existing price Insist on offer
vague-price
That offer is low
rule
insist
Still can I buy for S. I'm tight budget
rule
Offer the price
(offer)
agent action
Accept the offer
(accept)
agent action
Reject offer
(reject)
agent action
Quit session
(quit)
agent action
B NEGOTIATION STRATEGIES
Here we provide the details about the 15 Negotiation Strategies (Zhou et al. 2019 and Negotia-
tion Strategies (Zhou al. 2020) in Tables and g
Table 7: The details of 15 Negotiation Strategies proposed by Zhou et al. 2019
High level Negotiation Rules
Sub Strategy
Example
Detector
Describe Product
The cur has leather seats 45k miles - less than 50k miles luxury car with attractive leather seats I've just
classifier
Rephrase product
classifier
Focus on interests, not positions
Embellish product
classifier
Address concerns
to maintenance
classifier
Communicate interests
I'd like sell asap.
classifier
Propose Price
How about n/a
classifier
Do not propose first
rule
Invent options mutual gain
Negotiate side offers Hedge
can deliver it for you
rule
could come down a bit Greetings, gratitude, apology, please
rule
Communicate Politely
rule
Build Trust
Build rapport
My kid really liked this bike, he outgrew Absolutely, away!
rule
Talk informally
Show dominance
The absolute highest I Sadly, simply cannot under 500
640
rule
Insist your position
Negative Sentiment
rule
Certainty words
It has always had screen protector
rule
16

---

Published as a conference paper at ICLR 2021
Negotiation Strategies
Train set frequency
first, pesson. person_singular count pos_sentiment
26,121
24,862
number of woffiroofffffffffoofo offifffdiiccppppp .icc.pos
18,610
17,000
hedge_count
12,227
number_of caabberoofddff_diiee sddiff_dic_neee
10,402
personal concern
9,135
propose
8,449
politeness.greet
6,639
assertive_count
4,437
neg sentiment
3,680
factive count
3,429
politeness-gratiune
3,171
first.person_ www.oon.pllal..om couutal.count
2,876
liwe.cortainty
2,530
liwc informal
2,396
third. www.lpperooooppe trade in politeness.pleaat family friend <start>>
1,721
883
372
201
149
5,383
Table 8: The details of 21 Negotiation Strategies (<start>> added by us) used by Zhou et al. 2020). These used to operationalize the 15 strategies using rule based system (bttps: //github cwm/ schhuyhhengle statistics on the train set (5383 conversations) is given. A detailed description regarding the rules used by prior work to extract these are out of scope of this work, however, we intend to provide the code and extracted strategies, along with the rule based mapping to the 15 strategies upon acceptance of this work.
augment -ng-non- non- coti---onnollalaa
The frequency
C STRATEGY GRRAP VISUALIZATION
A visualization of strategy sequence graph. Refer to 2.2 for details. We also provide additional details regarding the number of nodes and edges in our strategy graphs in Table g
Ua
Ug
Greet
Dominant
Affirmative
Customer
Polite
Build Rapport
Greet
Thanks
Appeal
Sentiment
Seller (Bot)
U1
U7
U3
Visualization of strategy sequence graph. The graph connects each strategy with all previously occurring strategics. Here we present only a few edges for brevity. For example, there would be more additional edges from 44 the strategies of ug.
Figure 4:
17

---

Published as a conference paper at ICLR 2021
Table 9: We report the number of nodes and edges in our strategy-graphs. Each node corresponds to particular utterance-stratep pair.
Feature
Value
Max of nodes in graph (total strategies)
86
Avg of nodes in graph
21
Max of edges in graph
3589
Avg no, of edges in graph
308
D HYPERPARAMETERS
We present the hyper-parameters for all the experiments, their corresponding search space and their final values Table TO We also present additional details our experiments below. We use most of the hyperparameters from Zhou et al. (2020). Each training run took at most hours single Nvidia GeForce GTX 10801i GPU and all the models were saved based on Strategy Macro performance.
For experiments for Table I and we saved the best models on best Strategy Macro perfor- mance (HED being saved outcome class prediction). This is because wanted to prioritize and optimize our final model to capture sequence-structtral information owing to our focus on inter- pretability. While performing ablation studies for Table 3 not all models have structure encoders, and hence for a fair comparison we chose a metric independent of the different modules for all the models in ablations. We use the negotiation outcome class prediction (RC-Acc) scores that opti- mizes the dialogue for good negotiation outcome, which indirectly helps train the model to capture the sequence of strategies.
Table 10: Here we describe the search-space of all the hyper-parameene used in our experiments.
Model
Hyper-parameter
Search space
Final Value
BERT
bert-base-uncused fine tuning
All
BERT Dropout
0.3
ATI
Dialogue context embedding
300
All
Dialogue context dropout
0.1
All
learning-rate (Ir)
Se-3, le-3, Se-4
le-3
All
max utterances in batch weighted strategy loss
64,128,256
128
All
True,False
True
decay rate (12)
le-3
loss alpha
1,5
loss beta
10
loss gamma
10
projection layers for strategy
64
All
projection layers for DA hidden size hidden size decoder layers attention heads droout
64
HED+RNN
64, 300
64
HED+ Transformer
64,300
300
HED+ TrD+sransfomme
6
HED+ Transtomree
1,2
2
Transforeformer
0.0, 0.1
0.0
DIALOGRAPH
ASAP pooling ratio
0.2.0.5.0.8
DIALOGRAPH
hidden dim
64,128
64
DIALOGRAPH
Graph layers
1,2,3
DIALOGRAPH
Graph dropout
0.0,0.2
0.0
18

---

Published as a conference paper at ICLR 2021
E NEGOTIATION DATASET STATISTICS
In Table 11 we provide the CraiglistBargain dataset statistics along with data sizes after filtering conversations with less than 5 turns. The maximum and average number of turns in any conversation 47 and 9.2 respectively. Also, the maximum and average number of strategies in an utterance is 13 and respectively.
Table 11: Dataset statistics.
Data split
Size
Train conversations
5383
Valid conversations
643
Test conversations
656
Filtered train conversations
4828
Filtered valid conversations
561
Filtered test conversations
567
Vocabulary size
13339
F EXAMPLE CONVERSATIONS
Table 12: Examples of the generated dialogues of various models when we keep the buyer utterances same. We see that DIALOGRAPH gets the best deal for the same dialogue context and is more persistent. The FeHED and HED models accept offers more readily. We provide more examples of DIALOGRAPH in Table 13
Lästing Price: 40
Buyer's Target Price:
Title 2017 NEW Stans and 26 Tubeless Tire Kit
DIALOGRAPH
Human
HED+ Transfrranfrrmer
Humman
Hello
Hello
new?
Great, My hro destroyed my old kitt and I'm looking for new rines.
Great. My bro desmoyed old kit and I'm looking for new tires.
6a: am asking 40 for it.
< am asking 40 for it.
too
student me. about will deliver you.
That's much
like How ahout throw warranty.
can't go
could
could can come pick for 36.
go
can come pick up for 36,
can do 37.2 and you have deal.
<0: can do that.
Okay great. Thanks, < 37.2
Okay great. Thanks. offer 36
FeHED
Human
HRD
Human
Hello
Hello
Hi. kit new?
the now?
Hi thene! you interested in bike?
Yes it is.
Great. My bro destroyed my old kit and I'm looking for now tires.
Great. My bro destroyed my old kit and I'm looking for now tires.
nm asking 40 for
Yes it is.
studeat me.
That's much for student like How about
do
that.
Okay great. Thanks, < offier >- 35
Okay great. Thanks. offee 35
Final Deal: Graph = 37.2, Trans = 36, PeHED = 35, HED
19

---

Published as a conference paper at ICLR 2021
Table 13: More examples of the generated dialogues of DIALOGRAPH.
Listing Price: 40
Buyer's & Target Price: 36
Title: 2017 NEW Stans 24 and 26 Tubeless Tire Kit
DIALOGRAPH
Human
Hello
Hi.
Are you interested in the kit?
Yes am interested.
I looking for 40 for
willing to pay 35
I can't go that low
What can you take?
I could 38.8
deliver to you.
only have 36 please.
I can do 37.2 and you can pick up yourself.
Sorry only have I can do that.
Okay thank you. offer 37.2
Hello
Hi how are you.
I am good.
Is the good?
yes it is.
sure?
yes. willing negotiate price.
okay how much you willing to sell. it is in great condition. am asking 40 for
i will give you 25.
i can do 38.8 and you can pick it up.
33
can 37.2 and you can pick it up.
36 last
i can do that.
offer 36
Hello
Hi i see you are interested in my apartment
was interested the tire kit
m asking 40
would you be able to do 35?
willing negotiate little bit. what your budget?
highest can go
i cant go that low could do 38.8
okay that works offer 38.8
G INFLUENCE VISUALIZATION
Refer Figure
20

---

Published as a conference paper at ICLR 2021
5,ttrs parson pingu ar _coure
i 5,tirst person pl plaral count
summber of all ar poo
third person sinquler
2.third person, singulon
a.third, persan, singular
number at dic
jww informal
3,p sentiment
4_pos gendmmen
J.w informal
J.wwc cortaits
p sentiment
3_pulitenesss
propece
5 Sttrad
pop
4 third person singular
pos sentiment
trade
5 liwe, informal
3_livc__ce certaant
_.fist_person wwnguiirnnccccco count
5. number, of diff dic pos
5_ poss_ssee sentimett
5 propose
5. person, plural_cooe count
third singular
sentiment
7 Zesserlive_comt
third singular
7. w..hede__oom count
persosonall concern
first person_singulee count
7 7_number_od of wwl.diffdicccee neg
7, number diff dic pos
family
7_pos 7__os__ertinnne sentimant
P.iiss__erson_pppe cactopppueel__uul count
Visualization of the attention map learned by DIALOGRAPH for the example depicted in Figure 3 in the main paper. We only show it for a few turns for brevity. Here the axis labels represent the turn and the strategy. Refer the Figure 3 in the main paper for description.
Figure 5:
H HUMAN EVALUATION INTERFACE
Negotiation scenario!
You how to regotiene with snother -use for the price on femm You ane the bayes. legel ain and!
Instructions
Please use natural sentiecces as much as possitle.
Do: Would $10 wark you?
do:
deseriptien
Do presa
more once tycing an utterance. muard you money you chat our linucuryy chachet will most lkady und in fow minutes about 10 turns.
resporse bot. That might seconds.
your time while daing chat,
you made a goord afoort complete
the and of chat, after howe come a conciusion on what ofter type "cottre- set witho i amount in the chet hostt.
but dodido to and the cotvarsation wilding one "-poosst- "he "urgeets"" on r-qutt (with brackatt) ehat. Nobe that the chat windoww will closw
After this, thttr
wither wvilurs the aboue and will reach the
Please
after
with the right
Continue
Figure Screenshot of the introduction the human evaluation interface.
21

---

Published as a conference paper at ICLR 2021
Negotiation scenario!
perer antiren Yhu pee Rlppriatt
The Negotiation Scenario
Instructions
Getegey:
rutuaa aeteesaku peraall
Auooda 797
darmdd
Prec: 40
Yerr large Prke
Decoretion: have Tuketen ins Convrntonn
the Than
accind.
thme
comples
teal
ynu can buse farferrupe wupt whww oor
convenaiion
Ponject
the chen hual"
beouve cinve wven the party writee
survey prope
cheightg buffor iihtt cerve sedil
Time: 5:44
Figure Screenshot of the chat window for the human evaluation interface.
Instructions
following personally idendtylng information about yourself (or any other person] in any open text questions on the survey.
complete
dialogue
Part what extent do you agree with these statements?
underatood the task perfectly.
Strongly Disagree
Disagree
Neutral
Agree
Strongly Agroe
2. My task partner was persuasive.
Strongly Disagroe
Disagree
Agree
Strongly Agree
(natural)..
Strangly Disagree
Disagree
Neutral
Agree
Strangly Agrea
4. task partner perfectly understood what was typing.
Strongly Disagree
Disagree
Neutral
Strangly Agree
My task partner's responses were on topic accordance with conversation history, (Coherent)
Strangly Disegree
Disagree
Agrea
Strongly Agree
Figure 8: Screenshot of the survey for the human evaluation interface.
22


Reviews:
[{"review_id": "wIoTpjblNht", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with the problem of natural language generation for a dialogue system involved in complex communication tasks such as negotiation or persuasion. The proposed architecture consists of two encoders: one for the utterance and the other for dialogue acts and negotiation strategies. The decoder is an RNN that converts the encoded vectors to the output utterance. Each utterance is first passed through BERT to get an utterance-level encoding. The sequence of utterance encodings is then passed through an RNN to generate a conversation level encodings. The negotiation strategies and dialogue acts in a conversation are represented using a node-edge graph, where the nodes are one of the N different strategies/acts and there exists an edge from node a to node b if an utterance with strategy A precedes any utterance with strategy B. The entire architecture is trained in a multi-task setup where the loss function accounts for both the predictions of the model and generated language. The proposed architecture is evaluated on the CraigslistBargain dataset and compared against Zhou et al. 2020. \n\nThe paper is very clearly written and the experimental work has sufficient detail to ensure reproducibility. The main contribution and the novelty of this paper is in the use of graph neural networks for encoding dialogue acts and negotiation strategies. This choice was mainly because it helps with better interpretability of predictions and this is demonstrated anecdotally in section 5. The proposed model shows better performance in three different metrics when compared to sota from Zhou et al: (1) prediction of dialogue acts and negotiation strategies, (2) on the downstream task of dialogue generation, and (3) human evaluation to quantify the quality of generated language. \n\nThere are a few aspects of the paper unclear to me and could use more insight from the authors. (2) The input to the GNN is a node-edge graph where the edges exist between dialogue acts or negotiation strategies based on their precedence order in the conversation. It would be useful to explain why the authors chose this type of representation. What other types of representations were considered? (2) The authors use two different encoders for dialogue acts and negotiation strategies. Would it make sense to have a graphical representation that captures both dialogue acts and negotiation strategies simultaneously? (3) From my understanding, it seems the dialogue acts were annotated in the original work of (He He 2018) and strategies are obtained based on the models and rules published in (Zhou et al 2019). I am not sure if the model evaluation should entail predicting negotiation strategies which are in itself predictions of a different model. \n\nMinor comments/questions:\n1. How were the train/test/dev splits done?\n2. If I am not mistaken, it seems like the model uses predictions from time step (t) to predict and generate for time step (t+1). Does this mean errors in one of the earlier timesteps could lead to more errors in subsequent timesteps?\n3. I think it would be clearer if you were to use a single-letter variable for strategies. It helps for better readability. \n4. References: Please use the peer-reviewed version of the paper as opposed to the arxiv version when available. Eg. He He et al 2018. \n5. How many conversations were used in human evaluations?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mEHXq7kwUo7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "* Summary:\nThis paper proposes to DialoGraph, a model that utilizes Graph Attention Networks to learn the pragmatic relations between strategy and dialogue-act in negotiation dialogues, and achieves better performances than previous baselines. The use of Graph Attention Networks also enables a better interpretation of the relation between strategies.\n\n* Quality:\nThe motivation to introduce pragmatic information into negotiation dialogues is clear. The model is straightforward and effective. The experiments and analyzes are through. But the idea of directly applying Graph Attention Networks is not super exciting.\n\n* Clarity: \nThe paper is in general written clearly and easy to follow, with a few points to improve listed below. \n1. In 2.4, \"$p_j$ is the positive weight associated with the particular strategy\", is this $p_j$ the probability of the j-th strategy occurring in $u_{t+1}$ mentioned before?\n2. In section 5, \"We notice that as soon as the first propose at u5 happens, the strategies completely change and become independent of the strategies before the propose point. \" \nIt's not clear to me why the strategies are independent looking at Figure 3. And also, why is the weight in Figure 3 equal to 1 and the sum of outgoing edges bigger than 1? What's the max and min? And what's the cut-off threshold for the edges in Figure 3?\n3. In section 4, \"(several results are in bold if they have statistically insignificant differences)\"\nIt's not clear which results are significant and which are not from the tables and the descriptions\n4. I'd appreciate more dialogue examples in the appendix.\n5. How hard is it to optimize the graph networks? Is the training stable?\n\n\n* Originality:\nThe model is not very novel as Graph Networks have been used a lot in dialogue models.\n\n\n* Significance:\nDialoGraph could be potentially useful for other negotiation systems but it requires both strategy and dialogue-act annotations, which may be hard to obtain.\n\n* Pros:\n1. The interpretation of the learned strategy graph is useful, but that section needs a bit more work in clarity.\n2. The use of Graph Attention Network improves the negotiation dialogue system.\n\n* Cons:\n1. The model is applied on one negotiation dataset only and requires both strategy and dialogue-act annotations, which may not be available in other datasets, making the model not so generalizable to other tasks.\n\n2. From the results, it seems the major performance jump comes from using the embeddings from pretrained language models (transformer, BERT, etc)\n\n3. It's not clear which results are significant and which are not.\n\n4. It seems there is no connection between the \"strategy structure encoder\" and the \"dialogue-act structure encoder\"? Maybe adding a connection between these two modules would give a better performance since dialogue-act and strategies can be related? Also, are they related at all? If not, why do we need both labels?\n\n \n\n\n* Typo: \nmissing a period in Human Evaluation \"effective negotiation system Our model also...\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "dQSoP66oaJ3", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Summary\nThis paper presents DialoGraph, which introduces graph structures to encode the relations of strategies and dialog acts among utterances in a dialog history. This paper empirically demonstrated the efficacy of DialoGraph on non-collaborative negotiation dialog tasks, and the model is evaluated automatically and human-evaluation.\n\n- Strong points\n\t1. Training to control for the pragmatics of the negotiation dialogues has been less studied, and it is a crucial and interesting topic to build a logical dialog system.\n\t2. This paper is well structured and well written.\n\t3. Introducing GAT and ASAP to model strategies and dialog acts is novel, and the interpretation of their results is interesting.\n\t4. This paper also provides soundly experimental results and comparison with other SOTA models.\n\n- Weak points\n\t1. The dialog task problem setting seems unrealistic. A tuple of utterance, dialog act, and strategy at turn $I$ is given, and based on the previous tuple sequence the model predicts the next response, dialog act, and strategy. However, in reality, the user’s dialog act and strategy are hidden. \n\t2. Also, the sets of dialog acts and strategies are different depending on user and system. I agree this paper followed the task as previously defined, but it seems awkward.\n\t3. HED+Transformer vs. DialoGraph: Those two models show similar experimental results. According to the experiment configuration, the HED+Transformer used 6 decoder layers, whereas the DailoGraph used 2 graph layers. Those two models were fairly comparable with respect to the number of parameters? Or, the proposed method based on GAT+ASAP does not effective as much as Transformer, because both are basically based on attention mechanisms?\n\t4. In table 4, the average words per turn is required to be separately reported depending on users and bots.\n\n- Questions\n\t1. How the bot decide the price to offer? Does it solely depend on the language model (i.e., the decoder)?\n\t2. How the bot encode the listed price?\n\t3. In table 4, why HED got a remarkably higher score on “natural” measure than other models?\n\t4. Please address and clarify the weak points above.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qg8dCirwUeL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a end-to-end dialogue system that leverages Graph Attention Networks to model complex negotiation strategies. The main contributions that the author claims are to model negotiation strategies through a GNN and using these learned strategies to predict future strategies and generate a response leads to better negotiations.\n\nThe end to end model contains a traditional hierarchical encoder to obtain contextual representations along with a structure encoder that is designed to model strategies and dialog acts and obtain structural representations. The decoder is a simple GRU that produces the response by conditioning it on the contextual and structural representation along with the previous word.\n\nStrength & Weakness:\n1.  The aspect of generating a response based on the prediction of negotiation strategies and dialog acts is an interesting approach.\n2. The results from Table 1 show that transformers perform comparatively or even better than DialoGRAPH. What is the overall gain on using the GNN if the transformers match their performance?\n3. The results in Tables 2 & 3 are hard to interpret. Prior research has shown that BLEU and other automated metrics are not good enough to evaluate the performance of dialog systems. Why is the BERT F1 Score for Dialograph in Table 2 and 3 different? What is the major difference?\n4. What were the definitions provided for persuasive, coherence, natural, and understandable? Why is there a huge dropoff on the metric for naturalness from HED to other models? How many participants were recruited?\n\nQuestions:\n1. What is the rationale behind 5 negotiation classes? Was any ablation done with this to determine the optimal number or how this split affects the outcomes?\n2. What is the overall size of the vocabulary during decoding?\n3. How were the 4 outcome dialog acts labeled across the dataset? was this done through a human annotation process?\n4. What decoding strategy was used?\n\n\nSuggestion:\n1. Section 2.4 has inconsistencies in the usage of notations. Please fix those issues.\n2. Can the bolding of scores in Table 1 be made more consistent and highlight only those scores that are higher and lower based on the metrics being represented\n\n=======================After reading the authors response ============================\nI thank the authors for answering all the questions that been raised by the fellow reviewers. Looking at the responses and changes made to the paper, I have increased the score from 5 to 6 after the authors clarified the issues I had with the paper. Overall this paper demonstrates the effectiveness of using a GNN for negotiation dialogues. I feel that this approach can be applied for any non-collaborative dialog settings and the claims of interpretability make this approach better. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In the context of constructing negotiation dialogue strategies/policies, the authors explore the use of graph attention networks (GATs) for determining the sequence of negotiation dialogue acts -- specifically leading to a (1) hierarchical dialogue encoder via pooled BERT + GRU encoding -> (2) GAT over dialogue strategies/acts (many technical details around graph usage) -> (3) GRU decoder. While a relatively straightforward replacement relative to similar architectures with other 'structural' encoders, they provide a sound end-to-end training strategy that is shown to perform well on the buyer-seller negotiation task via CraigslistBargain dataset where they demonstrate SoTA performance.

== Pros ==
+ Studying the pragmatics component of negotiation dialogue strategies has received recent interest and this seems a good milepost that demonstrates mainstream methodological approaches for this task (i.e., this is a good baseline for future innovations)
+ The paper is well-written in that it is easy to understand intuitively while having sufficient detail to understand the details.
+ The empirical results appear promising and meet the standard within this sub-community -- showing improvements with automatic and human evaluation.

== Cons == 
- This builds on existing datasets, which are known to have undesirable properties (e.g., automatic evaluation, small number of dialogue datasets, use of explicit dialogues acts, etc.) While it still meets the standards of this sub-community, it still isn't a completely convincing task.
- While the use of GATs is novel in this setting and they get it to work within the overall architecture, this is something that many people are likely trying at this time -- so there isn't an exciting 'disruptive' step here.
- The empirical results, while satisfactory from a quantitative perspective, even in reading the Appendices, it isn't clear that these are significantly better from a planning perspective or if it is just 'pattern recognition' gains.

Evaluating along the requested dimensions:
- Quality: The underlying method is fairly straightforward and the authors incorporate up-to-date GAT-related methods to get this to work in this setting. The empirical results are sound if predicated on the general quality in this sub-community where you have the standard machine translation evaluation problem for meaning vs. lexical closeness. To mitigate, they use BERTScore and human evaluation -- which is at the higher end of what can be reasonably expected.
- Clarity: The paper is written clearly overall, especially if considering the appendices where there is significant detail. Related to empirical evaluation, it isn't easy to intuitively interpret the results, but this is again par for the course. Additionally, I believe the authors did a good job responding to reviewer concerns.
- Originality: While all of the reviewers agreed that the approach was novel in this setting, one of the reviewers explicitly pointed out that using GATs in negotiation dialogues isn't that exciting -- and I mostly agree. I view this as something that somebody would have done and will serve as a good baseline; although I think this sub-field is going to need more datasets to continue progressing.
- Significance: As stated above, it is a good baseline that I think many are likely thinking of (as the TOD community has been doing this for a bit now). However, it is done well.

Honestly, I agree with the reviewers that this is a somewhat borderline paper -- mostly due to it being a fairly 'obvious' idea and the nature of the subfield making it not entirely clear if the improvements are due to knowing the target performance while training or due to the methodological advance. Personally, I am convinced, but it isn't totally clear. That being said, it is a well-written paper and I think the reviewer issues were sufficiently addressed. Thus, I would prefer to see it accepted as I think it will be a strong methodological baseline for this problem (which hopefully will accumulate more convincing datasets and standard evaluation). 
Decision: Accept (Poster)