SlowGuess/NAACL_2022_OCR huggingface dataset:
Kushal Chawla $^{1}$  Gale M. Lucas $^{1}$  Jonathan May $^{2}$  Jonathan Gratch $^{1}$
University of Southern California, Los Angeles, USA
1{chawla, lucas, gratch}@ict.usc.edu
2jonmay@isi.edu
Opponent modeling is the task of inferring another party's mental state within the context of social interactions. In a multi-issue negotiation, it involves inferring the relative importance that the opponent assigns to each issue under discussion, which is crucial for finding high-value deals. A practical model for this task needs to infer these priorities of the opponent on the fly based on partial dialogues as input, without needing additional annotations for training. In this work, we propose a ranker for identifying these priorities from negotiation dialogues. The model takes in a partial dialogue as input and predicts the priority order of the opponent. We further devise ways to adapt related data sources for this task to provide more explicit supervision for incorporating the opponent's preferences and offers, as a proxy to relying on granular utterance-level annotations. We show the utility of our proposed approach through extensive experiments based on two dialogue datasets. We find that the proposed data adaptations lead to strong performance in zero-shot and few-shot scenarios. Moreover, they allow the model to perform better than baselines while accessing fewer utterances from the opponent. We release our code to support future work in this direction: https://github.com/kushalchawla/opponent-modeling.
Negotiations are key to our everyday interactions such as allocating available resources, salary decisions, business deals, and legal proceedings. The ability to effectively negotiate is also critical for automated systems deployed in complex social scenarios (Gratch et al., 2015). This enables these automated systems to engage in strategic conversations (Leviathan and Matias, 2018) and also assists in pedagogy by making social skills training more accessible (Johnson et al., 2019a).
Figure 1: A simplified view of a multi-issue negotiation based on the scenario in CaSiNo (Chawla et al., 2021). The negotiation involves 3 issues: Food, Water, and Firewood, each with 3 items that must be divided among the two players. From the perspective of player P1, the task of opponent modeling considered in this work involves inferring the priority order of the opponent P2 from the interaction between the two.
Consider the scenario presented in Figure 1. Two participants role-play as campsite neighbors and engage in a multi-issue negotiation (Fershtman, 1990) over three issues: food, water, and firewood (Chawla et al., 2021). Each negotiator has their own priority order depending on the relative importance assigned to each issue. The goal of the negotiation is to divide the available quantities of food, water, and firewood packages, such that each package is assigned to exactly one of the players in the final agreement.
The priority order of the opponent is typically unknown to negotiators beforehand, and can only be inferred based on the interaction between the two. Prior work argues that understanding what one's opponent wants is one of the key aspects of successful negotiations (Baarslag et al., 2013). An accurate model of the opponent can enable a dialogue system to roll out offers that work for both parties, which has implications on both its objective performance such as the final points scored from the agreed deal, and the subjective performance such as opponent's satisfaction and affinity for the dialogue system. This can also aid in pedagogy by
Findings of the Association for Computational Linguistics: NAACL 2022, pages 661 - 674
allowing the system to provide concrete feedback to students who fail to incorporate the priorities of their opponents (Johnson et al., 2019b). Discovering these priorities from an interaction with an opponent is usually referred to as Opponent modeling in the context of multi-issue negotiations.
Information about an opponent's priorities can primarily be gathered from their preference and offer statements (Nazari et al., 2015). Sharing preferences by explicitly mentioning 'We need water' or more implicitly - 'We like to go on runs' can provide information that water is of high priority to the negotiator. Further, offers such as 'I would like two food items and one water' can imply that food is of a higher priority than water.
Building techniques for opponent modeling that are useful in realistic chat-based negotiations poses several key challenges: 1) It is non-trivial to directly use counting-based methods on these preference and offer statements, which are common in prior work that does not use natural language, such as agent-agent negotiations (Williams et al., 2012) and human-agent negotiations based on button clicks (Mell and Gratch, 2017), 2) To alleviate this problem for language-based interactions, prior work has resorted to gathering additional utterance-level annotations to convert the desirable information into a more structured format, that can then be used with counting methods (Nazari et al., 2015). However, this approach remains expensive, requires expertise, and hurts generalizability. Further, these annotations are unavailable for systems that are deployed to end users, needing a separate NLU module which can potentially lead to error propagation in the downstream dialogue system pipeline, and 3) Some real-world applications require the system to guess the opponent's priorities with only partial dialogue so as to inform the future decision process of the system - a scenario which has not been well explored in prior works.
To address these challenges, we propose a transformer-based (Vaswani et al., 2017) hierarchical ranker for opponent modeling in negotiation dialogues. Our model takes a partial dialogue as input and guesses the opponent's priority order. Instead of relying on utterance-level discourse information, we devise simple and effective ways to project related data sources to this task. As opposed to multi-task learning which typically involves task-agnostic and task-specific parameters and back-to-back fine-tuning procedures that suffer
from catastrophic forgetting issues, our adaptations augment the training data available to the model, allowing end-to-end joint learning and parameter sharing. We summarize our contributions below:
1. We formulate opponent modeling as a ranking task (Section 2) and propose a transformer-based model that can be trained directly on partial dialogues using a pairwise margin ranking loss (Section 3).
2. To better capture the opponent preferences and offers, we devise methods to adapt related data sources, resulting in more labeled data for training (Section 3).
3. For a comprehensive evaluation that serves multiple downstream applications, we propose three evaluation metrics for this task (Section 4). Our experiments are based on two dialogue datasets in English: CaSiNo (Chawla et al., 2021) and DealOrNoDeal (Lewis et al., 2017), showing the utility of the proposed methodology with complete or partial dialogue as input in full, few-shot, and zero-shot scenarios (Section 5).
4. We compare our best-performing model to a human expert, discussing common errors to guide future work (Section 5), and laying out the implications for research in human-machine negotiations (Section 8).
Consider a negotiation  $C$  between two parties over  $m$  issues. We define the problem from the perspective of a specific negotiator (referred to as self, hereafter), and aim to predict the priority order of the opponent (see Figure 1). Assume that  $C$  contains an alternating sequence of  $N$  utterances between the negotiator self  $S$  and the opponent  $O$ . The partial interaction is  $C_k$ , which is obtained after  $S$  observes  $k$  utterances from the opponent. The goal is to build the model  $M$ , with  $Y_O = M(C_k)$ , where  $Y_O$  is the desired priority order of the opponent. In our experiments, we consider metrics that measure the performance for the complete dialogue and for different values of  $k$  (Section 5).
${}^{1}{C}_{k}$  will contain either  ${2k}$  or  ${2k} - 1$  utterances,depending on who starts the conversation.
Figure 2: Our proposed methodology for opponent modeling in negotiation dialogues. The approach involves three main components: Section 3.1 describes our core hierarchical encoder that takes in a partial dialogue and outputs the opponent priority order after seeing each utterance, Section 3.2.1 covers the adaptation of an argument-centric dataset (CA data) targeted towards better modeling the preference statements of the opponent, and Section 3.2.2 describes the adaptation of an offer-centric dataset (DND data) targeted towards the offer statements of the opponent.
We present our approach in Figure 2, which contains three main components: a hierarchical core model that takes in a partial dialogue and outputs the desired ranking order, and two modules for data adaptation that are designed to better model the preference and offer statements of the opponent. We first describe our core model, assuming a general input, and then describe the proposed data augmentation techniques.
3.1 Hierarchical Encoder
Our encoder (orange segment from Figure 2) uses two levels to build contextual utterance representations, which are then used to output a score for each of the  $m$  issues, representing the ranking order among them.
Utterance Encoder: First, a sentence-level module (Level I) encodes each utterance  $\mathrm{U}_j = [w_1, w_2, \ldots, w_{L_j}]$  separately. We prepend the utterances with a special token to indicate the author: <self> or <opp>. To encode a contextually-rich representation, our level I encoder uses pretrained
language models (Devlin et al., 2019; Liu et al., 2019), given their success across a wide range of NLP tasks, especially in low resource settings on similar NLU tasks (Balaraman et al., 2021). For each utterance  $U_{j}$ , the pretrained model first embeds the input words into the embedding matrix  $E \in \mathbb{R}^{L_j \times d}$ . After passing through the encoding layers, the pretrained model outputs  $d$ -dimensional word representations  $R \in \mathbb{R}^{L_j \times d}$ . Finally, this is followed by pooling to obtain the utterance representation  $U_{j} \in \mathbb{R}^{d}$ . The Level I output is essentially the conversation matrix  $U \in \mathbb{R}^{N \times d}$ , which is obtained after processing all the input utterances.
Dialogue Encoder: At Level II, we use a transformer block with masked self-attention (Vaswani et al., 2017). Self-attention enables efficient interactions for encoding partial conversations. A target utterance is only allowed to use the information from previously-seen utterances, which is accomplished by masking all the future utterances in the dialogue. In a single transformer layer, each target utterance query simultaneously assesses and encodes the information from all the unmasked key utterances, resulting in a contextualized representa
tion of each utterance - the matrix  $F\in \mathbb{R}^{N\times d}$
Output Layers: Finally, a feed-forward network acts on  $F$  to output an  $m$ -dimensional representation for each utterance. This represents the scores for each of the issues that the model is trying to rank. We then apply the sigmoid operation to constrain each score between 0 and 1, resulting in the output  $O \in \mathbb{R}^{N \times m}$ .
In comparison to text ranking tasks where the set of items that are being ranked is large and can be dynamic, the set of issues in realistic multi-issue negotiations is usually small and fixed. Hence, we predict the scores for each of these issues together, unlike text ranking literature where each item is ranked separately (Yates et al., 2021).
Training: We employ the pairwise margin ranking loss to train our model in an end-to-end manner. The loss  $\mathcal{L}_k$  after observing  $k$  utterances from the opponent is defined as:
$$
\mathcal {L} _ {k} = \sum_ {q = (q _ {1}, q _ {2}) \in Q} L _ {k} \left(o _ {q _ {1}} ^ {k}, o _ {q _ {2}} ^ {k}, y _ {q}\right), \tag {1}
$$
where  $L_{k}$  is given by:
$$
L _ {k} \left(o _ {q _ {1}} ^ {k}, o _ {q _ {2}} ^ {k}, y _ {q}\right) = \max  \left(0, - y _ {q} \left(o _ {q _ {1}} ^ {k} - o _ {q _ {2}} ^ {k}\right) + c\right). \tag {2}
$$
$Q$  represents the set of all possible pairs of issues.  $o_{q_1}^k$  and  $o_{q_2}^k$  are the scores from the final layer of the hierarchical ranker after applying the sigmoid operation.  $y_{q}$  captures the ground truth ranking between  $q_{1}$  and  $q_{2}$ .  $y_{q}$  is equal to  $+1$  when  $q_{1}$  should be ranked higher (has a larger score) than  $q_{2}$  and it is kept as  $-1$  otherwise.  $c$  is the margin.
The objective of the ranking loss is to train the model to predict a higher score for the issue that is ranked higher by the ground truth priority order. A positive margin of  $c$  ensures a nonzero loss if the score for the higher ranked item is not greater than or equal to its counterpart by  $c$ , forcing the model to predict well-separated boundaries. We experimented with different values for  $c$ , concluding that a nonzero margin is necessary for any meaningful training. For the results presented in this paper, we set  $c$  as 0.3.
Inference: Once the model is trained, the predicted scores can be used to output the desired ranking order for a given input dialogue. The model simply outputs the ranking of the issues by ordering them in decreasing order of these predicted scores.
Note on the loss formulation: The pairwise ranking loss was chosen for its suitability and simplicity. However, other potential alternatives do ex
ist. Since the number of issues is limited, one can remodel the prediction task as classification over all the possible orderings. However, this trivially does not capture that although two orderings can be wrong, one can be somewhat less wrong than the other. Hence, a ranking loss is more suitable for giving a smoother signal to the model during training, leading to a better performance in our initial experiments. We also explored more complicated ranking loss functions and a sequence-to-sequence model to directly generate the sequence of issues in their correct ranking order (Yates et al., 2021). We instead found the pairwise ranking loss to be effective and simple for our approach in this paper that involves a limited set of issues and exploits partially-masked loss functions (Section 3.2.1). Regardless, we encourage future work to explore these other formulations as well depending on the task at hand.
3.2 Data Adaptations
The transformer model discussed above learns to rank the issues directly from the partial dialogue as input without any additional supervision. Although this approach performs reasonably well in our experiments, it ignores the observations made in prior work which have primarily relied on annotations for preference and offer statements for opponent modeling (Nazari et al., 2015). This suggests that more explicit feedback for extracting information from preferences and offers is one avenue for improving the performance, especially in settings when the available dialogue data is scarce. Instead of gathering additional annotations, we devise alternate ways to better capture the preferences and offers in our hierarchical ranking model. We achieve this by adapting two additional data sources for this task, allowing the data to be directly added to the primary training dataset and enabling end-to-end parameter sharing between these related tasks.
Datasets: We leverage two datasets in this work: CaSiNo (Chawla et al., 2021) and DealOrNoDeal (Lewis et al., 2017). As discussed before, CaSiNo is grounded in a camping scenario, containing negotiations over three issues: food, water, and firewood. In addition to the dialogue, the dataset also contains metadata about the arguments used by the negotiators. DealOrNoDeal involves three arbitrarily-defined issues: books, hats, and balls. Our main goal is to perform opponent modeling for CaSiNo. To this end, we adapt DealOrNoDeal along with the available metadata in CaSiNo for
data augmentation.
We refer to the CaSiNo Dialogues as CD, CaSiNo Argument metadata as CA, and DealOrN-oDeal dialogue data as DND. While the CD data can be used as it is with our model, we adapt the other two data sources (CA and DND) to make them suitable for our approach (see Figure 2). We now describe these adaptations.
3.2.1 Capturing Preferences
In order to provide more direct supervision for the preferences, we leverage the metadata from CaSiNo (CA data), where the participants explicitly reported their arguments for needing or not needing a specific issue (blue segment from Figure 2). For instance, if food is the highest priority issue for a participant, they were asked to come up with an argument from their personal experiences as to why they would need food the most for camping. Example arguments are provided in Figure 2. The participants came up with a variety of such arguments covering Personal Care, Recreational, Group Needs or Emergency requirements. The participants were then encouraged to leverage these arguments in their upcoming negotiations.
This metadata can provide more direct feedback on which implicit preference statements can lead to a higher or a lower affinity towards a specific issue. To incorporate this, we create dummy dialogues using templates and add them to the training data for our opponent modeling task. Consider a set of arguments  $A = (A_H, A_M, A_L)$ , containing one argument for High, Medium, and Low priorities respectively. We extract two pairs:  $(A_H, A_L)$  and  $(A_M, A_L)$  and construct the dummy dialogue as per Figure 2.4 We ordered the arguments randomly to avoid any induced biases.
For each constructed dialogue, we only have ground-truth ranking order for a single pair of issues. Hence, the pairwise loss function from Equation 1 needs a special treatment to ignore the score of the issue that is not relevant for a given dialogue. More specifically, while training with these constructed dialogues, we partially mask the margin ranking loss to only consider the loss from the pair for which the relation is known. Further, since a
partial dialogue is not meaningful in this case, we only train the model with  $\mathcal{L}_2$  loss using  $k = 2$ .
Although we use the readily available metadata from CaSiNo in our work, we believe that such contextual data can be constructed for other realistic domains as well, such as by leveraging appropriate domain-specific knowledge about the negotiators' common requirements.
3.2.2 Capturing Offers
To better capture the preferences in the previous section, our approach was to construct synthetic dialogues from a resource that primarily focused on implicit preference statements, so as to teach the model in a more explicit manner. With a similar idea, we adapt DND dialogues to better use the offer statements (green segment in Figure 2). The DND dataset follows the same multi-issue framework as CaSiNo, which enables our adaptation. Each dialogue in DND involves three arbitrarily-defined issues: books, balls, and hats. Due to the arbitrary nature of these issues, there is minimal context discussed in the dialogues, reducing it to essentially an exchange of offers from both sides (see example in Figure 2). Hence, such a resource can be used to provide more explicit supervision to learn from the offer statements of the opponent. We map these dialogues to our dataset by randomly mapping the issues in this dataset to the issues in the target dataset, in our case, CaSiNo. We modify the utterances by replacing all the occurrences of the issues with the corresponding issues in CaSiNo. For this purpose, we find that simple regular expressions prove to be effective (Appendix B.1). Once mapped, this adapted data is simply added to the training data for our opponent modeling task.
Note on multi-issue negotiations: Our adaptation described above leverages the structural similarities between the two datasets. If the tasks follow a similar structure, it is relatively straightforward to use adaptations as described above for other settings as well. This can be largely done with regular expressions but if not, this relatedness still paves the way for multi-task learning. The negotiations in DealOrNoDeal and CaSiNo are based on a popular abstraction in the negotiation literature, referred to as the Multi-Issue Bargaining Task, or MIBT (Fershtman, 1990). MIBT is a generic framework that can be useful for many negotiation tasks beyond these datasets as well, for instance, salary negotiations, or negotiations between art collectors distributing the items among each other. It
2These priority orders were randomly assigned to the participants by the authors of the CaSiNo paper.
3We refer the readers to the CaSiNo dataset paper for more examples around these themes.
4We skip the third pair due to an absence of a visible difference based on our qualitative analysis.
is extensively used in NLP (Lewis et al., 2017; Chawla et al., 2021; Yamaguchi et al., 2021), beyond NLP (Mell and Gratch, 2017), and in the industry as well (e.g. iDecisionGames $^5$ ).
We address the following questions: Q1) How useful is the proposed transformer-based ranker along with data augmentations for opponent modeling in negotiation dialogues? We experiment with two pretrained language models and compare our ranker to standard baselines. To test the data augmentations, we analyze model ablations, including 0-shot and few-shot settings. We also observe if they lead to a better performance with a lower number of utterances. Q2) Do preferences and offers contribute to the performance? To further shed light on the contributions of these utterances to the final opponent modeling performance, we look at average attention scores on these utterances. Further, for a more explicit analysis, we observe whether the performance varies by the integrative potential in the negotiation, which essentially captures how aligned the preferences of the two negotiators are (Chawla et al., 2021). The scenarios with low integrative potential are usually associated with a higher expression of preferences and offers. Hence, we expected the performance to be higher in the cases with low integrative potential. Q3) How does our approach compare to a human expert? We compare our model to a human expert and recognize some of the errors that the model makes, discussing potential directions for future work.
Datasets: Each data point in CD results in two dialogues for our analysis, based on the perspectives of the two negotiators (Section 2). We report results on 5-fold cross validation for this dataset. We further leave out 100 dialogues from the training data for hyperparameter tuning, resulting in 1548 dialogues for training, 100 for tuning, and 412 for evaluation - for each cross fold. We extract CA from the metadata corresponding to the training data of CD, leaving out 200 constructed dialogues for validation (following Section 3.2.1). For DND data, we only select the dialogues with at least 4 total utterances and unique priority values for meaningful training. After adaption (following Section 3.2.2), we end up with 4074 dialogues for training and 444 for validation. All the models are
primarily validated and tested on the corresponding subsets of CD (except for some additional analysis presented in Section 5).