SlowGuess/NAACL_2022_OCR huggingface dataset:
Evaluation Metrics: Our metrics are inspired by the negotiation literature, along with related research in Dialog State Tracking (DST) and Learning-to-Rank(LTR) tasks in NLP. Our primary metric is Exact Match Accuracy (EMA): the percentage of cases where the predicted priority order is entirely correct. This is analogous to the popular Joint Goal Accuracy in DST which captures the cases where all the slots are correctly identified (Balaraman et al., 2021). For negotiation tasks, even knowing the topmost priority can be useful. Hence, we also report Top-1 Accuracy: the percentage of cases where the highest priority issue is correctly predicted. Finally, we report the Normalized Discounted Cumulative Gain (NDCG@3). NDCG has been widely used in LTR tasks with distinct relevance values (Yates et al., 2021), which is also true for the setting that we consider. In our case, we use the relevance values as 5, 4, and 3 for the most, second, and least ranked issues respectively, following the incentive design structure of CaSiNo. We compute these metrics for all  $k$  from 1 to 5, varying the number of opponent utterances seen by the model. We present the results at  $k = 5$  to analyze the performance after seeing almost all of the opponent utterances in CaSiNo. To capture the performance with partial dialogues, we report corresponding  $k$ -penalty versions that take a weighted average of the performance for different values of  $k$ , while giving a linearly higher weight to the performance at a lower  $k$ .
Methods: We call the complete model from Figure 2 that combines all the three datasets for training as  $\mathbf{CD} + \mathbf{CA} + \mathbf{DND}$ . We compare it with its ablations, including 0-shot and few-shot scenarios. We further develop two standard baselines. The Random baseline chooses the final ranking at random, from all the possible orderings. BoW-Ranker is based on the Bag-of-Words paradigm. The input features are based on the normalized frequencies of the 500 most frequent words in the training dataset, except stopwords. Instead of contextualized hierarchical representations, this method directly uses a feed-forward network on the input BoW features to predict the ranking. The model is trained on partial dialogues using the same margin ranking loss.
Training Details: The embedding dimension throughout is 768 for transformer-based models.
<sup>5</sup>https://idecisiongames.com/promo-home
These models use base variant of either BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019) for Level I encoder. The Level II encoder uses one transformer layer. The feed-forward network contains two fully connected layers with a final sigmoid activation. We train the model with Adam optimizer using a learning rate of  $2e^{-5}$  for transformer-based methods and  $2e^{-3}$  for BoW-Ranker. The margin  $c$  is kept as 0.3. We use a dropout of 0.1 to prevent overfitting. We further employ a loss-specific dropout of 0.15, in order to backpropagate the loss from fewer  $k$ s simultaneously. The models were trained for 20 epochs with a batch size of 25. We checkpoint after every epoch and the one with the highest EMA at  $k = 5$  on the held out CD dataset is chosen for evaluation. We provide the details on the computing infrastructure, hyper-parameter tuning, and validation performance in Appendix A.
5.1 Addressing Q1
We summarize the results in Table 1. Our proposed ranking-based models beat the Random and BoW-Ranker baselines by a huge margin across all metrics. This is true even for zero-shot DND and for CA + DND, attesting to the utility of the proposed ranking methodology and data adaptations. Comparing similar configurations, we observe that RoBERTa-based models outperform BERT-based models on this task. The best performing configuration is the RoBERTa CD + CA + DND that combines all the three data sources.
In Figure 3a, we plot the performance for different percentages of CD data. We only show RoBERTa-based models due to their superior performance. The plot highlights the advantage of adapting the related data sources, especially in few-shot settings, with  $\mathbf{CD} + \mathbf{CA} + \mathbf{DND}$  at  $50\%$  matching the performance of CD at  $100\%$ .
We also look at how the performance varies with the number of utterances seen in Figure 3b. We find that the performance gains are visible across all values of  $k$ . The data augmentations allow the model to perform better than the baselines, while observing a fewer number of utterances, making the model more useful in realistic scenarios.
Performance on the adapted datasets: We analyze if our joint learning also improves the per
formance on the validation sets of CA and DND datasets, showing advantages across multiple tasks. For CA dataset, we measure argument ranking accuracy: for a given input dialogue based on a pair of arguments, we consider a prediction as correct if the scores predicted by the model correctly rank the arguments. For DND, we analyze EMA at  $k = 2$  for opponent modeling, similar to our setup for CaSiNo. As evident from Tables 2a and 2b, we find support that joint learning improves the performance on CA and DND datasets as well.
5.2 Addressing Q2
Average attention: We recognize the utterances with preference statements by utilizing strategy annotations in CaSiNo (Chawla et al., 2021). We assume that an utterance contains a preference if it was annotated with at least one of Self-Need, Other-Need, or No-Need strategies. For identifying offers, we use regular expressions following prior work (He et al., 2018) (refer Appendix B.2). We consider any utterance that is not labeled with a preference or an offer as Other. Then, we observed the average attention put by the best-performing model on these categories in the Level II encoder. Preferences received an average of 0.3, offers received 0.27, and other utterances received 0.08 attention scores, without any explicit indication about these categories during model training. We consider this as preliminary evidence that the learning process matches our intuition, with preferences and offers contributing to the performance.
Performance across integrative potential: For more concrete evidence of the utility of preferences and offers, we look at how the performance varies between scenarios with low and high integrative potential. This basically captures how aligned the preferences of the two negotiators are in a negotiation. In a scenario with low integrative potential, the negotiations are more competitive, leading to a higher expression of preferences and offers and providing a better signal to our ranking models. For our best-performing model, we find EMA at  $k = 5$  to be 68.75 (4.58) for scenarios with low integrative potential against 60.31 (2.67) for those with high potential. This provides stronger evidence that the learning process sensibly takes into account the preference and offer statements in the data.
5.3 Addressing Q3
Comparison to Human Expert: Similar to the trained models, we asked a human expert (an au
$^{6}$ Training with the CA data only was not useful due to the lack of training with any partial dialogues.
Model	k=5	k-penalty
EMA	Top-1	NDCG@3	EMA	Top-1	NDCG@3
Random	16.46 (1.47)	32.49 (1.58)	48.49 (1.16)	16.59 (1.22)	33.99 (1.13)	49.76 (0.75)
BoW-Ranker	28.49 (1.3)	53.38 (2.21)	65.51 (0.62)	27.71 (1.24)	52.98 (1.97)	64.31 (1.67)
Bert-based
DND	41.12 (3.06)	64.69 (2.94)	73.88 (1.57)	34.5 (1.12)	58.75 (1.35)	68.48 (0.77)
CA+DND	41.9 (2.93)	66.98 (3.17)	75.91 (2.28)	36.01 (1.25)	61.09 (1.9)	70.09 (1.49)
CD	53.97 (3.02)	77.7 (2.85)	83.75 (1.96)	42.3 (1.53)	66.8 (1.78)	74.39 (1.45)
CD+CA	57.24 (3.09)	79.74 (2.37)	84.99 (1.87)	44.39 (1.17)	67.88 (1.16)	75.31 (1.1)
CD+DND	56.12 (4.07)	79.16 (2.57)	84.66 (1.84)	43.79 (2.07)	68.18 (1.55)	75.38 (1.6)
CD+CA+DND	56.56 (2.07)	80.13 (1.07)	85.49 (1.09)	44.22 (1.82)	69.21 (2.05)	76.03 (1.6)
RoBerta-based
DND	45.21 (3.07)	68.1 (2.8)	77.01 (1.76)	37.66 (1.41)	61.41 (2.3)	70.44 (1.5)
CA+DND	46.76 (1.89)	68.73 (1.22)	77.65 (0.9)	39.43 (1.67)	62.87 (2.5)	71.7 (1.83)
CD	60.06 (3.01)	81.98 (1.75)	86.54 (1.31)	46.57 (1.6)	69.26 (1.69)	76.17 (1.22)
CD+CA	60.01 (2.23)	80.23 (2.11)	85.85 (1.41)	46.96 (2.1)	68.59 (1.93)	76.05 (1.14)
CD+DND	62.54 (3.3)	82.56 (1.24)	87.57 (1.18)	47.69 (2.52)	69.98 (1.96)	76.71 (1.55)
CD+CA+DND	63.57 (3.44)	82.76 (2.47)	87.55 (1.58)	48.72 (2.03)	70.03 (1.63)	77.14 (1.38)
Table 1: Performance on the opponent modeling task, showing the utility of the proposed methods. EMA and Top-1 represent the accuracy in percentage. We also scaled NDCG@3 to 0-100. For all the metrics, higher is better. The numbers represent Mean (Std.) over 5-cross folds of the CD data.
(b)
Figure 3: Mean performance for two RoBERTa-based models: (a) on different percentages of CD data. The Y-Axis represents EMA at  $k = 5$ , (b) on different values of  $k$ .
CA
Model	Accuracy	Model	EMA
Random	52.4 (4.14)	Random	16.04 (0.92)
AD	63.8 (9.33)	DND	60.68 (2.05)
AD+DND	73.4 (6.19)	AD+DND	60.9 (1.87)
CD+AD	78.9 (1.39)	CD+DND	63.11 (1.77)
CD+AD+DND	76.7 (3.52)	CD+AD+DND	63.56 (0.94)
Table 2: Performance for RoBERTa-based models: (a) argument classification accuracy on the validation set of CA, (b) EMA at  $k = 2$  for opponent modeling on the validation set of DND. The numbers represent Mean (Std.) over 5-cross folds.
thor of this work) to guess the priority order of the opponent by accessing partial dialogues. The expert was allowed to make multiple guesses if she is unsure, in which case the final ranking was chosen randomly from all the guesses. We compare the expert to our best-performing model on 100 dialogues from the evaluation set. The expert achieved  $75\%$  mean EMA at  $k = 5$  against  $66\%$  for the model
while performing better on other metrics as well. We show the comparison by varying the parameter  $k$  in Appendix C.
While the model performs reasonably, there is a scope for improvement. We performed a qualitative analysis of the errors made by the model and the expert. In many cases, it is simply not feasible to predict accurately, especially when negotiators engage in small talk early on - indicating a limited scope for improvement with fewer utterances. In some cases, there is more focus on the highest priority issue, giving less explicit signals of the entire ranking. This might work for some applications but in other cases, the agent design can be modified to discuss the complete ranking more explicitly. Integrating other datasets that follow the same MIBT structure (such as (DeVault et al., 2015)) via data adaptation or multi-task learning is another potential direction. We also observed errors in the cases that included longer contextually-dense ut
terances, where preferences are shared indirectly as a response to the partner, and when the negotiators give away their higher priority issues out of empathy towards their partner. These cases are easier for the expert but can be confusing to the model. Better modeling of the prior context and handling of longer utterances are also avenues for improvements in the future.
Opponent modeling encompasses several tasks in negotiations such as priority estimation, predicting opponent limits like BATNA (Sebenius, 2017), and classifying opponents into categories such as based on personality traits (Albrecht and Stone, 2018; Baarslag et al., 2016). In this work, we focused only on inferring the opponent's priorities but in a more challenging domain involving chat-based interactions, instead of structured communication channels often used in prior work (Williams et al., 2012; Mell and Gratch, 2017; Johnson and Gratch, 2021). Using a realistic interface like natural language fundamentally alters the negotiation dynamics in terms of the exchange of information, and hence, requires a separate investigation.
For chat-based negotiations, Nazari et al. (2015) relied on heuristics and utterance-level annotations to infer the opponent's priorities using frequency-based methods. Langlet and Clavel (2018) explored a symbolic rule-based system to parse the utterances collected from a multimodal interaction. Instead, our focus is on modeling the priorities directly from partial dialogues as input. Research in negotiation dialogue systems has mainly focused on end-to-end modeling of the agent, without any explicit opponent modeling (Lewis et al., 2017; He et al., 2018; Zhou et al., 2019; Cheng et al., 2019; Parvaneh et al., 2019). However, there is evidence that even end-to-end systems can benefit from being more opponent-aware, as seen in recent work that uses dialogue acts to estimate opponent's behavior (Zhang et al., 2020; Yang et al., 2021).
A number of related data augmentation strategies have been explored in Computer Vision and NLP (Shorten and Khoshgoftaar, 2019; Feng et al., 2021). Most methods use rules or models to transform the available data or create synthetic data to avoid overfitting while training. This especially helps in low-resource languages (Li et al., 2020) and few-shot scenarios (Kumar et al., 2019).
We presented and evaluated a transformer-based approach for opponent modeling in negotiation dialogues. Our objective was to address the challenges to bridge the gap between existing research and practical applications of opponent modeling techniques. Our comparison to baselines and ablations attests the utility of our method. We found that the proposed data adaptations can be especially beneficial in 0-shot and few-shot scenarios. In the future, we will explore two primary directions: first, improving the model performance on opponent modeling by leveraging other related available datasets and by better incorporating the negotiation dialogue context, and secondly, using effective opponent modeling techniques towards the design of automated negotiation systems for applications in pedagogy and conversational AI.
8 Broader Impact and Ethical Considerations
Datasets Used: Both the datasets used in this work had been completely anonymized before their release by the respective authors. Moreover, we carefully verified the licensing details and ensured that the datasets were only used within the scope of their intended usage.
We note that both datasets follow the multi-issue structure where the priority order remains fixed throughout the negotiation. Although this may not be true for some real-world scenarios, as we noted earlier, the underlying MIBT framework used by these datasets has been extensively used in academic research and also in the industry, attesting to the generalizability and applicability of this approach. Finally, we note that both the datasets are in English. Although this means that our experiments were limited to one language, our approach makes no such assumptions and should be broadly applicable to other settings as well. We encourage researchers to extend this work and study human-machine negotiations for other languages as well. This would open up exciting avenues for cross-culture research in this space, given the well-documented differences in how humans negotiate across cultures (Luo, 2008; Andersen et al., 2018).
Human Annotations: Human annotations were used to estimate the expert performance on this task. This did not involve any additional crowdsourcing effort. Instead, the dialogues were annotated by an author of this work.
Opponent Modeling For Negotiation Dialogues: Negotiations are typically non-collaborative in nature, where the goals of the negotiating parties may not align with each other. Hence, the negotiators may not always feel comfortable in revealing their preferences for fear of being exploited. Even if they do, inferring them from natural language is challenging as preferences might be implied, and resolving these implications involves domain-specific knowledge and prior dialogue context. Regardless, incorporating such realistic communication channels is critical for designing practical and robust AI systems for downstream applications. However, most of the prior efforts in negotiations use restrictive menu-driven systems based on button clicks. Our work is a step towards bridging this gap.
This work is aligned with our broader goals for building automated negotiation systems, trained either in an end-to-end or a modular manner. For conversational AI applications, opponent modeling systems that can predict the priorities of the opponent reliably based on a partial dialogue can inform the strategy of the agent in the latter parts of the conversation. From the perspective of pedagogical applications, even the systems that can predict the priorities of a negotiator at the end of the negotiation can be helpful. For instance, consider a negotiation between two students, A and B who are asked to guess the opponent's priorities at the end of their negotiation. If the pedagogical agent is able to accurately guess the priorities of student B, while student A fails to guess correctly, this can be used to give concrete feedback to students who fail to recognize these strategies.
Ethical Recommendations: Finally, we briefly discuss the ethical considerations around the design of automated negotiation systems. A considerable amount of research in negotiations has focused on ethics. Primary concerns revolve around the acts of emotion manipulation, bias, deception, and misinterpretation (Lewicki et al., 2016). Consequently, these issues can also emerge in the systems that are developed on human-human negotiation dialogue datasets. Our central recommendation in mitigating the impact of these issues for negotiation dialogue systems or other conversational AI assistants is transparency - around the identity, capabilities, and any known undesirable behaviors of the system. Further, any data collected during the deployment phase should be properly anonymized and the users
of the system should be well-informed. In particular, we recommend extra precautions for systems that are adaptive towards their opponents or users such as having regular monitoring for any unexpected behaviors, to ensure that the systems are not offensive or discriminatory.
We would like to thank Garima Rawat, Thamme Gowda, Sarik Ghazarian, Abhilasha Sancheti, and Prakhar Gupta for their valuable comments. We also thank the anonymous reviewers for their valuable time and feedback. Our research was sponsored by the Army Research Office and was accomplished under Cooperative Agreement Number W911NF-20-2-0053. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
Stefano V Albrecht and Peter Stone. 2018. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-95.
Steffen Andersen, Seda Ertac, Uri Gneezy, John A List, and Sandra Maximiano. 2018. On the cultural basis of gender differences in negotiation. *Experimental Economics*, 21(4):757-778.
Tim Baarslag, Mark Hendrikx, Koen Hindriks, and Catholijn Jonker. 2013. Predicting the performance of opponent models in automated negotiation. In 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), volume 2, pages 59-66. IEEE.
Tim Baarslag, Mark JC Hendrikx, Koen V Hindriks, and Catholijn M Jonker. 2016. Learning about the opponent in automated bilateral negotiation: a comprehensive survey of opponent modeling techniques. Autonomous Agents and Multi-Agent Systems, 30(5):849-898.
Vevake Balaraman, Seyedmostafa Sheikhalishahi, and Bernardo Magnini. 2021. Recent neural methods on dialogue state tracking for task-oriented dialogue systems: A survey. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 239-251.
Kushal Chawla, Jaysa Ramirez, Rene Clever, Gale Lucas, Jonathan May, and Jonathan Gratch. 2021. Casino: A corpus of campsite negotiation dialogues for automatic negotiation systems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3167-3185.
Minhao Cheng, Wei Wei, and Cho-Jui Hsieh. 2019. Evaluating and enhancing the robustness of dialogue systems: A case study on a negotiation agent. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3325-3335.
David DeVault, Johnathan Mell, and Jonathan Gratch. 2015. Toward natural turn-taking in a virtual human negotiation agent. In AAAI Spring Symposia. Citeseer.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for nlp. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 968-988.
Chaim Fershtman. 1990. The importance of the agenda in bargaining. Games and Economic Behavior, 2(3):224-238.
Jonathan Gratch, David DeVault, Gale M Lucas, and Stacy Marsella. 2015. Negotiation as a challenge problem for virtual humans. In International Conference on Intelligent Virtual Agents, pages 201-215. Springer.
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018. Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2333-2343.
Emmanuel Johnson and Jonathan Gratch. 2021. Comparing the accuracy of frequentist and bayesian models in human-agent negotiation. In Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents, pages 139-144.
Emmanuel Johnson, Gale Lucas, Peter Kim, and Jonathan Gratch. 2019a. Intelligent tutoring system for negotiation skills training. In International Conference on Artificial Intelligence in Education, pages 122-127. Springer.