Venue: NAACL 2025

# KODIS: A Multicultural Dispute Resolution Dialogue Corpus

James Hale\({}^{1}\), Sushrita Rakshit\({}^{2}\), Kushal Chawla\({}^{3}\), Jeanne M. Brett\({}^{4}\), Jonathan Gratch\({}^{1}\)

\({}^{1}\)University of Southern California, \({}^{2}\)University of Michigan,

\({}^{3}\)Capital One, \({}^{4}\)Northwestern University

jahale@usc.edu, gratch@ict.usc.edu

###### Abstract

We present KODIS, a dyadic dispute resolution corpus containing thousands of dialogues from over 75 countries. Motivated by a theoretical model of culture and conflict, participants engage in a typical customer service dispute designed by experts to evoke strong emotions and conflict. The corpus contains a rich set of dispositional, process, and outcome measures. The initial analysis supports theories of how anger expressions lead to escalatory spirals and highlights cultural differences in emotional expression. We make this corpus and data collection framework available to the community1.

Footnote 1: Please fill out this Google Form to access the data-set.

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 12771–12785

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Conflicts ubiquitously arise between individuals, organizations, nations, and cultures. Conflicts can help individuals recognize and appreciate differences and learn essential social skills. Too often, conflicts escalate to verbal, legal, or physical violence [6, 1]. Individual conflicts can damage relationships and incur costly legal fees. National conflicts cost the global economy USD $19 trillion in 2023 [10]. "Culture wars" within and between nations give rise to different conceptions of reality that can perpetuate generational conflict [11].

Interest grows in using natural language processing (NLP) methods to understand how conflicts arise and are resolved through conversation [22, 23, 24]. Conflict dialogues are task-oriented non-collaborative conversations: tasks-oriented as parties aim to achieve specific goals (e.g., extract concessions), and goal achievement can be explicitly measured; non-collaborative as goals are misaligned, though not necessarily zero-sum (parties could discover win-win compromises via conversation).

This paper introduces a large (4,061 participants) and novel corpus designed to offer multicultural insights into how conflicts escalate or resolve through conversation. It is novel in that we examine _dispute resolution_ rather than _deal-making_. Deal-making is a major recent focus of NLP research [13, 14, 15, 16], though this literature has favored the term negotiation over deal-making. This creates confusion as both deal-making and dispute-resolution involve negotiation (parties converse to influence each other and extract concessions, often over multiple issues), but disputes involve unique social processes [6] and have received less attention within the fields of AI and NLP. By prioritizing deal-making over dispute resolution, the NLP community risks overlooking key processes that shape conflict, which this corpus seeks to address.

Deal-making is forward-looking as parties focus on opportunities for gain and try to establish a new relationship. As the relationship is not yet established, parties have greater opportunities to explore alternatives. When parties fail to reach an agreement, they can seek other partners - e.g., if unsatisfied with one car dealer, one can always negotiate with another. In contrast, disputes are backward-looking, typically involving an existing relationship that has gone badly. As parties are already linked, success depends on managing the costs of ending the relationship rather than opportunities moving forward [6]. As a result, disputes evoke much stronger emotions, and positions are more entrenched than dealing. This distinction is crucial as it shapes the consequence of influence attempts. For example, whereas expressions of anger promote compromise in deal-making [13], they evoke escalation in disputes [6, 16]. As disputants often become entrenched in their positions, rather than seeking

compromise, they seek to overpower their opponent through appeals to justice ("you violated my rights!") or by threatening harm ("I will see you!"), leading to a spiral of further escalation, including threats of physical violence [14, 26, 30]. Thus, the costs of disputes can greatly exceed the original perceived injury, engulfing not only the disputants but other related parties and even society at large.

The corpus is also novel in using theories of conflict to guide data collection. We collected a diverse sample of participants from over 75 countries, and participants were matched within and across countries. We measure cultural and individual differences that have previously been shown to shape negotiated outcomes. For culture, we focus on differences between Dignity, Face, and Honor cultures [9, 8]. This theoretical framework distinguishes cultures by the degree to which people's social identity is independent versus interdependent and thus shapes the importance given to norms of reciprocity and honesty. Dignity cultures (typically Western society) might respond to a norm violation with a shrug or even a smile. Honor cultures (typically the Middle East or South America) might respond with hot anger (especially if the violation involves family). Face cultures (typically East Asia) might react by shutting down all emotional expressions [1]. Additionally, we measure individual differences such as risk-propensity [10] and the specific goals parties bring to a dispute. Finally, we assess several theoretical mechanisms surrounding the dispute, including process variables (e.g., what tactics did parties use, what emotions were expressed, and did parties understand their partner's interests?) and outcome variables, including objective and subjective measures concerning the outcome of the dispute.

Finally, this corpus' novelty partially stems from including a mix of human-human and human-AI disputes. Though we focus on human disputes, when participants could not match with a partner promptly, they matched with a large language model (GPT-4), assuming their partner's role. Postconflict measures include beliefs about whether their partner was human or AI and attitudes towards using AI technology for such applications. Analysis of this data could yield insight into the current limitations of GPT-4; differences between human and AI dialogs; and how these differences shape dispute processes, outcomes, and perceptions.

We envision a wide range of theoretical and applied uses for the corpus. For social scientists, it offers a test-bed to study critical social processes such as function of emotional expressions [11], the role of perspective-taking [12, 13], the dynamics of escalation and deescalation [26], and the impact of culture [11, 12]. From a general artificial intelligence perspective, it provides a means for evaluating AI's social competence [13, 26, 27] and uncovering AI's tendency to propagate cultural stereotypes [10]. From an application perspective, it supports the development of AI agents to help teach dispute-resolution skills [11, 25], to monitor and intervene in human disputes [12], or to replace customer service agents with AI [1].

Researchers must be mindful of ethical pitfalls when pursuing these ends. A tool like GPT can yield important insights, yet its uncritical use can adversely affect knowledge production and understanding [1, 14], and our GPT-informed conclusions in Section 3 should be generalized with care. Concerning applications, AI developed to resolve conflicts could reinforce structural inequalities between cultures [11] or be repurposed to create conflict. As a result, we limit corpus access to non-commercial uses.

This work contributes the following:

* We first describe the creation and nature of the KODIS corpus, which we make available to the NLP community.
* We then summarize a recent study using the corpus to illuminate how emotional expressions shape disputes as they unfold within and between cultures.

## 2Kodis Corpus

We introduce _KObe DISpute corpus_ (KODIS), a corpus of dyadic disputes. Our data collection was inspired by the CaSiNo framework of chawla2021caffe, which allows pairs of human participants to match online and engage in a deal-making exercise via text chat. Prior NLP-based analysis of CaSiNo dialogues found that emotional expressions predict participants' satisfaction with their negotiated 

agreement (Chawla et al., 2023a), that information expressed in the dialogue revealed participant's private goals for the negotiation (Chawla et al., 2022), and that agents could be trained via reinforcement learning to negotiate effectively against people (Chawla et al., 2023c). We adapt this framework to dispute resolution and extend it to allow human-agent disputes.

### Corpus Collection Framework

Figure 1 illustrates the KODIS data collection framework, which allows dyadic text-based interaction between two people or between a person and an AI partner. Participants are recruited through an online service such as Prolific. They first enter a survey implemented in Qualtrics (qualtrics.com). This administers the consent form, pre-task measures, and describes the dispute scenario2. Participants next perform the dispute via an application created using Lioness Labs, a framework for multi-participant behavioral experiments (Giamattei et al., 2020).

Footnote 2: Note some dispute information is common knowledge between but some is private to each party.

They first enter a virtual waiting room, which displays a timer counting down from eight minutes and a message asking them to wait for their partner. If another participant with an opposing role joins in the first seven minutes, the two meet in a chat interface and work on the dispute; otherwise, at one minute left in the lobby, the participant moves on and converses with an AI counterpart. After completing the task, participants return to Qualtrics, where they complete post-task measures. Finally, they are routed to the online recruitment service, where they enter a completion code and receive compensation. When performing the task, we leave the partner's nature (e.g., human or AI) ambiguous but imply to be human. At the end of the study, participants are debriefed about the partner's true nature.

### Application Architecture

We use Giamattei et al. (2020)'s Lioness Lab to implement our design as it facilitates matching online participants into dyads. Other dialogue research has used Lioness (Chawla et al., 2021). Figure 10 (appendix) shows the user interface - on the left is a chat box; a menu on the right lets participants walk away or finalize their agreement.

### Dispute Resolution Task

Dispute resolution research advances through a mixture of field studies and scenario studies that afford measurement and manipulation of theoretically relevant variables. Here, we adopt the latter approach by designing a task to validate a theoretical model of multi-cultural negotiation proposed by Aslani et al. (2016). This model argues that culture shapes negotiation tactics (e.g., competitive vs. cooperative dialog) and proposes how this relates to creating win-win solutions.

Participants engage in a role-playing exercise simulating a bilateral multi-issue buyer-seller dispute -- other dyadic corpora also use role-play settings (Chawla et al., 2021; Yamaguchi et al., 2021; Lewis et al., 2017; He et al., 2018). We crafted the exercise in collaboration with an expert in dispute resolution (one of the authors) and designed it to evoke strong emotions and entrenched positions while adhering to the ethical guidelines for human experimental research. The dispute centers on a buyer who purchased a basketball jersey for their sick nephew and claims they received the wrong item. The seller disputes this claim, arguing the correct item was sent and no refunds are allowed. Each side is told the other has posted negative reviews attacking their counterpart's reputation. Each side receives different versions of events to encourage a dispute over facts.

We frame the task as multi-issue, where participants can discuss and potentially reach a compro

Figure 1: Participants did pre / post-dispute questionnaires and interacted with their counterpart. We first try to match the participant with a human, though we match them with GPT4 if unmatched after seven minutes.

 

mise. Participants are told they can discuss the refund, drop their review, request their opponent drop their review, and discuss who, if anyone, should apologize. In the appendix, Figure 10(a) illustrates the context presented to a participant slated to act as a buyer, and Figure 10(b) illustrates the context presented to the seller. Figure 2 demonstrates an example dialogue where participants display hostility to one another.

Participants are incentivized to take the dispute seriously by offering a substantial bonus based on performance (up to SS3). Participants receive a base pay (SS3.50) for attempting the task, which can be nearly doubled if they achieve all their objectives. Many role-playing scenarios use "assigned preferences," meaning participants are given a payoff matrix that defines their goals in the negotiation (e.g., they might receive the most bonus if they achieve a refund). To allow cultural variability, we use "elicited preferences," meaning that participants are provided a fixed number of points to allocate across the four issues: e.g., buyers might assign 70% of their points to receiving a refund but 30% to receiving an apology. They receive a bonus based on their stated preferences and the actual resolution of the dispute, receiving a fixed bonus ($0.70) if the dispute ends in an impasse. Specifically, we calculate the proportion of total possible points a participant scored (see Equation 2) and grant that proportion of the bonus.

Participants use a chat interface to communicate. A menu interface reminds them of the issues under discussion and is used to unambiguously specify the final agreement (see Figure 10 in the appendix). After exchanging at least eight messages, they use this menu to send a final offer. Their partner can accept, reject, or counter this offer. After eight messages, Participants can use this interface to "walk away" from the dispute - we refer to these as impulses. Buyers go first (unless one participant is an AI, in which case the AI always goes first). Participants alternate sending messages; the participant's interface is inactive while awaiting a new message.

### Measures

We measure several theoretical constructs claimed to shape negotiation processes and outcomes, focusing on those used in research on multicultural deal-making Aslani et al. (2016). This research suggests that cultural and individual variables (e.g., Dignity, Face, and Honor) shape negotiation tactics (e.g., tendency to express emotion, willingness to exchange information). Tactics shape perspective-taking (e.g., less sharing means less understanding of your opponent's goals), shaping the likelihood of reaching an agreement and the quality of the resulting agreement. Agreement quality can be measured in objective terms, but subjective feelings about the agreement and partner are more predictive of subsequent behavior, such as willingness to follow through on agreements and maintain a future relationship with the partner Curhan et al. (2006); Brown and curhan (2012)

#### 2.4.1 Pre-Dispute Measures

**Screening:** Participants are screened using techniques shown to improve data quality Geisen (2022). They first respond to a _commitment request_ asking if they could commit to providing thoughtful responses. They next respond to a series of attention checks, including multiple choice questions (how many legs does a cat have?) and open-ended responses (describe the flavor of a tomato). Par

Figure 2: This illustrates an example of a contentious dialogue from the KODIS corpus.

 

ticipants who fail these checks are immediately excluded without compensation.

**Questionnaires:** Participants complete a demographic survey that includes gender, education level, and country where they spent most of their life (and the number of those years). They next complete an 18-item Dignity, Face, and Honor scale measuring cultural differences in the perception of identity threats Leung and Cohen (2011). Dignity cultures view individuals as having intrinsic self-worth, making identity relatively impervious to attack. Honor cultures view self-worth as something that must be claimed and defended from external threats. Face cultures also see self-worth as conferred by others but see retaliation as further eroding self-worth. Participants are asked to consider their culture and answer questions on the strength of attitudes on a 7-point Likert scale. Examples include People must always be ready to defend their honor." and People should be very humble to maintain good relationships." Finally, Participants complete a short measure of risk propensity Meertens and Lion (2008).

**Preference Elicitation:** After reading the scenario, participants self-report their goals for the upcoming dispute by allocating 100 points over four issues to be discussed (see Figure 12a -- _refund, other drops negative review_, _you drop negative review_, and _receive apology_). Points indicate the utility of fully or partially achieving this goal and directly map onto the participant's monetary bonus. For example, assigning more points to an apology than to a refund indicates that the participant would experience greater reward from receiving an apology than a refund. Participants are also asked to write a one-sentence justification for the importance assigned to each issue. For example, one participant wrote, He is a crook and will defraud others as a justification for wanting to keep his bad review of the other party, suggesting they assign importance to reputational concerns.

**Integrative Potential:** The preferences of the Buyer and Seller determine the structure of the dispute. Depending on each party's interests, there may be an opportunity to "grow the pie" by finding mutually beneficial solutions. For example, if the Buyer only cares about a refund and the Seller only wants the Buyer to drop their negative review, both sides can maximize their bonus. However, just because a win-win solution exists doesn't mean the parties can find it. The concept of "integrative potential" measures the potential for joint gains (which may or may not be realized). We operationalize integrative potential using the preferences elicitation from each side. Given two vectors of preferences \(\vec{X}_{buyer}\) and \(\vec{X}_{selfer}\):

\[\text{IP}=1-\frac{\vec{X}_{buyer}\cdot\vec{X}_{selfer}}{\|\vec{X}_{buyer}\|\| \vec{X}_{selfer}\|} \tag{1}\]

Notably, above, we invert the cosine similarity of the two vectors -- those with dissimilar values have greater potential for joint gains.

#### 2.4.2 Post-Dispute Measures

**Perspective-Taking:** Participants are asked a series of questions to assess if they accurately understand their partner's goals in the dispute. Participants are first asked about their preferences as an attention check. They rank each issue's importance (most, middle, or least). This is contrasted with preferences they provided during the preference elicitation phase to check if they recall their initial preferences. They are asked to do the same for their partner. The distance between these estimates and their partner's actual preferences can serve as a measure of perspective-taking accuracy.

**Tactics:** Participants are asked a 10-item questionnaire Aslani et al. (2016) about the tactics they and their partner used during the dispute. On a five-point Likert scale, participants answer if they agree or disagree with a series of questions about the dispute process. E.g., I expressed frustration, "The OTHER PARTY expressed frustration," and I shared my preferences with the other party." Thus, we also capture first and second-person annotations -- i.e., participants annotated the emotions and negotiating tactics of themselves and their counterpart.

**Subjective Value of the Outcome:** Subjective perceptions of the outcome of a dispute are a better predictor of future negotiation decisions than the actual economic result Brown and Curhan (2012). We capture these impressions with an 8-item version of the Subjective Value Inventory (SVI) Curhan et al. (2006). This measures four dimensions of subjective value, including feelings about the _instrumental outcome_ (e.g., Did I get a good deal?"), the _process_ (e.g., Was the process fair?"), feelings about the _self_ ("Did I lose face?"), and feelings about _relationship_ with the partner ("Would I work with my partner again?").

**Objective Individual Outcome:** We calculate the objective value of each side's outcome from the agreed-upon outcome using the points provided in 

the preference-elicitation phase. These points are also used to determine the bonus. Specifically, the objective value of the outcome is derived using the following linear additive utility function:

\[U_{a}=\sum_{i\in I}w_{i,a}*\ell_{i,a} \tag{2}\]

Where \(U_{a}\) denotes the points participant \(a\) earns; \(I\) represents the set of all issues; \(w_{i,a}\) holds the value participant \(a\) gave to issue \(i\in I\); and \(\ell_{i,a}\in[0,1]\) holds the level agreed upon for \(i\) in the dispute (where higher is more favorable to \(a\)) -- typically binary, though the _refund_ issue uses .5 for a partial refund. Notable, this formula only applies in disputes ending in agreement -- those resulting in an impasse yield a fixed amount to each side.

**Objective Joint Outcome:** We can measure the collective benefit two parties achieve by summing the individual outcomes (\(U_{a}+U_{b}\)). If parties are effective negotiators, the joint outcome should be positively correlated with the integrative potential for the task, but if parties misunderstand each other (e.g., come from different cultures), they may fail to realize this potential.

### Participants and Corpus Characteristics

We outline the demographic and dispositional composition of the corpus.

#### 2.5.1 Demographics

We collected responses from the crowd-sourcing platform Prolific from November 2023 to June 2024. We recruited participants from countries worldwide (see Table 1). Further, our dataset comprises of 50% Female, 49% Male, and 1% other.

#### 2.5.2 Dignity, Face, Honor, & Risk

We analyze the correlations between the various dispositional measures we captured -- specifically, Dignity, Face, Honor, and Risk. We see Dignity significantly positively correlates with Face (\(r=.17\), \(p<.001\)), and Risk-seeking (\(r=.13\), \(p<.001\)); Face significantly positively correlates with Honor (\(r=.07\), \(p<.01\)) and Risk (\(r=.05\), \(p<.05\)); and Honor significantly correlates with Risk (\(r=.22\), \(p<.001\)). The recruited pool contains respondents of varying Dignity (\(M=31.22\), \(SD=4.03\)), Face (\(M=25.95\), \(SD=5.15\)), Honor (\(M=24.33\), \(SD=7.04\)), and Risk-seeking (\(M=4.03\), \(SD=1.67\)) propensities.

#### 2.5.3 Culture

We attempt to delineate culture using the previously described Dignity, Face, Honor scale, using K-means to cluster similar countries. We only consider countries with \(N\geq 10\) appearances in the dyads. Using the elbow method, we run K-means using five clusters -- Figure 3 depicts the resultant clusters. For easy visualization, the \(x\) and \(y\) axis of this figure use PCA decomposition to show the otherwise three-dimensional data; Dignity has positive loadings with PC1 (0.63) and PC2 (0.77); Face has a negative (-0.18) loading with PC1 and a positive (0.05) one with PC2; and Honor has a positive (0.75) loading with PC1 and a negative (-0.63) one with PC2. We see the first cluster comprised of Poland, Greece, Mexico, and Turkey; the second has the Netherlands, Germany, the United Kingdom, the United States, and Canada; the third has Italy, France, and Portugal; the fourth has Nigeria, and South Africa; while the fifth has Sweden.

Figure 3: K-means clusters of countries (\(N\geq 10\)).

 

Using Emotion to Analyze Disputes

We summarize an initial use of the corpus to provide insight into how emotions and culture shape dispute processes and outcomes. This highlights an example use case and demonstrates the corpus has "face validity" in that we replicate findings from the dispute resolution literature. Specifically, we use an LLM to objectively quantify the emotions expressed in the dialogue and analyze how this relates to objective (impasse vs. agreement) and subjective (subjective value inventory) outcomes.

Disputes often evoke strong emotions like anger, and expressed anger has different social consequences in disputes than in deal-making. In deal-making, expressions of anger often signal that one side has reached their limit and the other must make concessions Van Kleef et al. (2004). Thus, expressions of anger can lead to concession-making. In contrast, anger often provokes escalation in disputes Pruitt (2007); Adam and Brett (2015, 2018). Culture also plays a role in the consequences of expressions. In dignity cultures, expressions of anger are often viewed as acceptable expressions of self-interest, whereas anger can provoke retaliation in cultures where self-worth is conferred by others Adam et al. (2010).

We use the KODIS to address several theoretical claims about the role of emotion in disputes: do expressions of anger provoke escalation and impasses in disputes (as previously claimed), can negotiation satisfaction be predicted by emotional expression alone, and how does culture shape these findings?

### Approach

Prior research on the CaSiNo corpus used NLP techniques to examine the impact of emotion in deal-making, and we closely follow that approach Chawla et al. (2023a), allowing qualitative comparisons with a similar deal-making corpus. Following research demonstrating that GPT-4 currently yields the most accurate inferences on negotiation dialoges Kwon et al. (2024), we limit our reported analysis to GPT-4o.

We prompt GPT4o (run on 06/28/2024) Achiam et al. (2023) to annotate each dialogue turn. Following detailed experiments on different prompting methods and emotion labels, GPT annotates the intensity ([0..1]) of six emotions expressed in the turn (anger, compassion, sadness, joy, fear, and surprise). Anger and compassion, in particular, were chosen based on their role in prior negotiation research Van Kleef et al. (2004); Allred et al. (1997); Zhang et al. (2014). Each utterance is presented along with the preceding dialogue context, and a small amount of in-context learning guides responses. Further, GPT reports results in a machine-readable JSON format. See the appendix for justification of these choices and evidence that GPT substantially improves predictive accuracy compared to earlier methods. For brevity, we only examine human-human dyads.

### Escalation

The negotiation literature suggests that disputes are more likely to end with non-agreement than deal-making due to increased anger and entrenched positions. By comparing KODIS disputes with the CaSiNo deal-making dialogues, we replicate this finding: about 18% of the KODIS dialogues ended with an impasse (one side walked away) whereas only 3.5% of the CaSiNo dialogues ended with an impasse. This is remarkable as participants forfeit a cash bonus if they fail to achieve an agreement, even though this was merely a simulated dispute.

To examine escalatory dynamics, we divide dialogues based on whether they ended in an agreement or impasse. We then examine expressed emotion by role over time (see Figure 4). Dialogues that ended in an impasse are in red; those that ended in an agreement are in green. The results show evidence of escalation. Buyers generally enter the dialogue with greater anger. Dialogues that end with an impasse show evidence of escalation: sellers reciprocate anger, leading to even greater anger by the buyer. In contrast, sellers avoid reciprocating anger for dialogues ending in agreement, and buyers subsequently express less anger. This supports work on conflict spirals Pruitt (2007).

Figure 4: Anger by role, outcome and dialogue turn.

 



### Predicting Subjective Outcome

We next examine if emotions expressed during the dialogue can predict participants' subjective feelings about the result. We used multiple linear regression on a random subset of 406 dialogs to predict the four facets of subjective value from expressed emotion. These include feelings about the outcome, self (did I lose face?), process (was it fair?), and relationship with the partner. We construct one model over the entire dataset to assess the relationship between emotion and subjective outcomes. Overall, expressions are surprisingly accurate at predicting subjective feelings. Examining \(R^{2}\) of the regression models, emotional expressions predict almost 50% of the variance in feelings about the process and the partner. This is particularly remarkable as these models ignore the content of the dialog. Regarding the coefficients, all emotions play a significant role in the model prediction (see Figure 6).

We additionally examine how culture shapes these predictions, which could lead to systematic bias, by training a model on Western culture and testing its predictive accuracy on other cultures. Given imbalances in our dataset, we use the nation as a proxy for culture and focus our analysis on the most prevalent countries in our sample (US, UK, Canada, Mexico, and South Africa). Specifically, we trained a model on a random 50/50 split of US v. US dialogues and tested on pure and mixed-nation dialogues from the other countries and unseen US data -- we ran this 1,000 times on different splits of the US-US dyads and Table 2 displays resultant differences (averaging across all runs and SVI measures). We see the US within-culture regression explain the variance of the subjective outcome better for the countries similar and worse for dissimilar ones -- as uncovered during the cultural K-means. Lastly, Figure 5 illustrates the differences in emotions for the five countries we consider when disputing against the same or a different country.

## 4 Conclusion & Future Work

We collected a corpus of human disputes and showed some promise using NLP methods to illuminate processes. Current analyses predict cultural tendencies from the dialogues and develop models to assist human disputants (e.g., can an algorithm recognize an escalatory spiral and suggest an intervention to help parties reach an agreement?). The corpus continues to expand, focusing on additional languages and refining the theoretical measures.

## 5 Limitations

We design the corpus to address existing limitations in the literature on NLP and negotiation by emphasizing the distinction between dispute resolution and deal-making and providing a substantial corpus

Figure 5: Average GPT emotion scores for the five most common countries broken by within or cross-culture.

\begin{table}
\begin{tabular}{|c c c|} \hline  & \multicolumn{3}{c|}{\(R^{2}\)} \\
**Country** & **Within-culture** & **Cross-culture** \\ \hline \hline US & 0.236 & 0.227 \\ UK & 0.188 & 0.215 \\ Canada & 0.211 & 0.214 \\ Mexico & 0.137 & 0.170 \\ SouthAfrica & 0.047 & 0.157 \\ \hline \end{tabular}
\end{table}
Table 2: US regression on mixed and pure dyads.

Figure 6: \(R^{2}\) using GPT emotion to predict outcome.



of disputes. However, it is important to emphasize that the data is drawn from a single artificial scenario. The dispute focused on a standard consumer economic dispute, and participants were asked to role-play. Thus, care must be taken in generalizing these findings to real-world interactions.

This analysis of cultural differences contains confounds that must be unpacked. We predict dispute outcomes from expressed emotion and show that performance degrades when models trained on US participants are applied to non-US participants. Yet it is unclear if this is due to bias in emotion recognition (e.g., does GPT-4o over-estimate anger in South African dialogues) or if emotion functions differently in different cultures. Evidence from other research suggests both factors are probably in play. Thus, the analysis should be augmented by human annotations from those target cultures.

Though we collect information on demographics, personality, and culture, all information comes from self-reports, which we cannot verify. Participants act as part of a paid service, which may shape their responses in ways that do not match real-world interactions. Participants also retained anonymity during the interaction, which can shape their responses. In real interactions, parties have an existing relationship, and there can be real-world consequences. These factors can strongly shape the expression and function of emotions.

We use commercial pre-trained models to recognize emotional expressions in our dialogues. Still, we do not have independent human annotations of what emotions will likely be perceived in the text. So, while we provide some evidence of external validity (expressed emotion impacts outcomes in theoretically predicted ways), subsequent research must verify these machine-generated emotions correspond to the intended mechanism. This is particularly fraught in a cross-cultural setting as existing work shows that large pre-trained models introduce bias in interpretations Havaldar et al. (2023).

## 6 Ethical Considerations

**Data Collection** Our study was approved by USC's Institutional Review Board. The participant received informed consent, describing the purpose of the study, data policies, and noting that they could withdraw at any time. Participants could lose their compensation by withdrawing which can be seen as coercive, but aligned with current experimental norms and designed to ensure that experimental protocols match real-world decision-making Rousu et al. (2015). The compensation was set to provide a fair wage and to conform to the guidelines of the online collection service. No identifiable information was collected during the collection. Potentially identifiable information such as IP addresses, worker IDs, and location information is removed before releasing the data. Any mention of demographics or personality of participants is based on self-identified information in our pre-survey and standard procedures of collecting personality metrics.

**Potential Risks** Our work supports using NLP methods to provide insight into psychological processes. However, there are reasonable concerns that NLP can undermine the diversity of scientific research (by over-reliance on a small number of tools), create the illusion of objectivity, and reinforce cultural stereotypes Messeri and Crockett (2024). This is particularly the case for research on emotion. Recent findings in affective science emphasize that emotions are perhaps best seen as cultural constructs labeled and interpreted differently across cultures. Yet many labeling schemes used in emotion databases rely on Western representational taxonomies. This is true of the labels we adopted in the evaluation experiment. This can serve to reinforce Western biases on the interpretation of the data. As noted by Abdurahman et al. (2024), using LLMs "as an off-the-shelf 'one-size-fits-all' method in psychological text analysis--can lead to a proliferation of low-quality research, especially if the convenience of using LLMs such as ChatGPT leads researchers to rely too heavily on them." Augmenting our findings with diverse models and human judgments remains imperative.

## Acknowledgments

This work is supported by the U.S. Government including the Air Force Office of Scientific Research (grant FA9550-23-1-0320), and the National Science Foundation (grant 2150187). The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. We thank Jessie Hoegen, Kelly Tsai, and Jasmine Zhu for their contributions 



## References

* Abdurahman et al. (2024) Suhaib Abdulrahman, Mohammad Atari, Farzan Karimi-Malekabadi, Mona J Xue, Jackson Trager, Peter S Park, Preni Golazizian, Ali Omrani, and Morteza Dehghani. 2024. Perils and opportunities in using large language models in psychological research. _PNAS nexus_, 3(7):245.
* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Adam and Brett (2015) Hajo Adam and Jeanne M Brett. 2015. Context matters: The social effects of anger in cooperative, balanced, and competitive negotiation situations. _Journal of Experimental Social Psychology_, 61:44-58.
* Adam and Brett (2018) Hajo Adam and Jeanne M Brett. 2018. Everything in moderation: The social effects of anger depend on its perceived intensity. _Journal of Experimental Social Psychology_, 76:12-18.
* Adam et al. (2010) Hajo Adam, Aiwa Shirako, and William W Maddux. 2010. Cultural variance in the interpersonal effects of anger in negotiations. _Psychological Science_, 21(6):882-889.
* Allred et al. (1997) Keith G. Allred, John S. Mallozzi, Fusako Matsui, and Christopher P. Raia. 1997. The influence of anger and compassion on negotiation performance. _Organizational Behavior and Human Decision Processes_, 70(3):175-187.
* Aslani et al. (2016) Soroush Aslani, Jimena Ramirez-Marin, Jeanne Brett, Jingjing Yao, Zhalen Semnani-Azad, Zhi-Xue Zhang, Catherine Tinsley, Laurie Weingart, and Wendi Adair. 2016. Dignity, face, and honor cultures: A study of negotiation strategy and outcomes in three cultures. _Journal of Organizational Behavior_, 37(8):1178-1201.
* Brett (2007) Jeanne M Brett. 2007. _Negotiating globally: How to negotiate deals, resolve disputes, and make decisions across cultural boundaries_. John Wiley & Sons.
* Brett et al. (1998) Jeanne M Brett, Debra L Shapiro, and Anne L Lytle. 1998. Breaking the bonds of reciprocity in negotiations. _Academy of Management Journal_, 41(4):410-424.
* Brown and Curhan (2012) Ashley D Brown and Jared R Curhan. 2012. The utility of relationships in negotiation. _The Oxford handbook of economic conflict resolution_, pages 137-154.
* Chawla et al. (2023a) Kushal Chawla, Rene Clever, Jaysa Ramirez, Gale M Lucas, and Jonathan Gratch. 2023a. Towards emotion-aware agents for improved user satisfaction and partner perception in negotiation dialogues. _IEEE Transactions on Affective Computing_.
* Chawla et al. (2022) Kushal Chawla, Gale Lucas, Jonathan May, and Jonathan Gratch. 2022. Opponent modeling in negotiation dialogues by related data adaptation. In _Findings of the Association for Computational Linguistics: NAACL 2022_, pages 661-674.
* Chawla et al. (2021) Kushal Chawla, Jaysa Ramirez, Rene Clever, Gale Lucas, Jonathan May, and Jonathan Gratch. 2021. Casino: A corpus of campsite negotiation dialogues for automatic negotiation systems. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3167-3185.
* Chawla et al. (2023b) Kushal Chawla, Weiyan Shi, Jingwen Zhang, Gale Lucas, Zhou Yu, and Jonathan Gratch. 2023b. Social influence dialogue systems: A survey of datasets and models for social influence tasks. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 750-766.
* Chawla et al. (2023c) Kushal Chawla, Ian Wu, Yu Rong, Gale Lucas, and Jonathan Gratch. 2023c. Be selfish, but wisely: Investigating the impact of agent personality in mixed-motive human-agent interactions. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 13078-13092, Singapore. Association for Computational Linguistics.
* Cheng et al. (2019) Minhao Cheng, Wei Wei, and Cho-Jui Hsieh. 2019. Evaluating and enhancing the robustness of dialogue systems: A case study on a negotiation agent. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 3325-3335, Minneapolis, Minnesota. Association for Computational Linguistics.
* Cho et al. (2024) Hyundong Cho, Shuai Liu, Taiwei Shi, Darpan Jain, Basem Rizk, Yuyang Huang, Zixun Lu, Nuan Wen, Jonathan Gratch, Emilio Ferrara, and Jonathan May. 2024. Can language model moderators improve the health of online discourse? In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 7478-7496, Mexico City, Mexico. Association for Computational Linguistics.
* Curban et al. (2006) Jared R Curban, Hillary Anger Elfenbein, and Hong Xu. 2006. What do people value when they negotiate? mapping the domain of subjective value in negotiation. _Journal of personality and social psychology_, 91(3):493.
* Davani et al. (2023) Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy, and Morteza Dehghani. 2023. Hate speech classifiers learn normative social stereotypes. _Transactions of the Association for Computational Linguistics_, 11:300-319.
* Ebers (2022) Martin Ebers. 2022. Automating due process-the promise and challenges of ai-based techniques in consumer online dispute resolution. In _Frontiers in Civil Justice_, pages 142-168. Edward Elgar Publishing.
* Friedman et al. (2017) Ray Friedman, Cameron Anderson, Jeanne Brett, Mara Olekalns, Nathan Goates, and Cara Cherry Lisco.



2004. The positive and negative effects of anger on dispute resolution: evidence from electronically mediated disputes. _Journal of Applied Psychology_, 89(2):369.
* Galinsky et al. (2011) Adam D Galinsky, Debra Gilin, and William W Maddux. 2011. _Using both your head and your heart: The role of perspective taking and empathy in resolving social conflict_, volume 13. Psychology Press New York.
* Geisen (2022) Emily Geisen. 2022. Improve data quality by using a commitment request instead of attention checks. Accessed on Oct 13, 2024.
* Gelfand et al. (2001) Michele J Gelfand, Lisa H Nishii, Karen M Holcombe, Naomi Dyer, Ken-Ichi Ohbuchi, and Mitsuteru Fukuno. 2001. Cultural influences on cognitive representations of conflict: Interpretations of conflict episodes in the united states and japan. _Journal of applied psychology_, 86(6):1059.
* Giamattei et al. (2020) Marcus Giamattei, Kyanoush Seyed Yahosseini, Simon Gachter, and Lucas Molleman. 2020. Lioness lab: a free web-based platform for conducting interactive experiments online. _Journal of the Economic Science Association_, 6(1):95-111.
* Gratch et al. (2015) Jonathan Gratch, David DeVault, Gale M Lucas, and Stacy Marsella. 2015. Negotiation as a challenge problem for virtual humans. In _Intelligent Virtual Agents: 15th International Conference, IVA 2015, Delft, The Netherlands, August 26-28, 2015, Proceedings 15_, pages 201-215. Springer.
* Halperin (2008) Eran Halperin. 2008. Group-based hatred in intractable conflict in israel. _Journal of Conflict resolution_, 52(5):713-736.
* Havaldar et al. (2023) Shreya Havaldar, Bhumika Singhal, Sunny Rai, Langchen Liu, Sharath Chandra Gunutuku, and Lyle Ungar. 2023. Multilingual language models are not multicultural: A case study in emotion. In _Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis_, pages 202-214, Toronto, Canada. Association for Computational Linguistics.
* He et al. (2018) He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018. Decoupling strategy and generation in negotiation dialogues. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2333-2343, Brussels, Belgium. Association for Computational Linguistics.
* Economics and Peace (2024) Institute for Economics and Peace. 2024. Global peace index briefing 2024. Accessed on Oct 13, 2024.
* Klimecki (2019) Olga M Klimecki. 2019. The role of empathy and compassion in conflict resolution. _Emotion Review_, 11(4):310-325.
* Kwon et al. (2024) Deksin Kwon, Emily Weiss, Tara Kulshrestha, Kwalsal Chawla, Gale M. Lucas, and Jonathan Gratch. 2024. Are llms effective negotiators? systematic evaluation of the multifaceted capabilities of llms in negotiation dialogues. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, Miami, FL. Association for Computational Linguistics.
* Leung and Cohen (2011) Angela K-Y Leung and Dov Cohen. 2011. Within-and between-culture variation: individual differences and the cultural logics of honor, face, and dignity cultures. _Journal of personality and social psychology_, 100(3):507.
* Lewis et al. (2017) Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2443-2453, Copenhagen, Denmark. Association for Computational Linguistics.
* Lin and Chen (2022) Ting-An Lin and Po-Hsuan Cameron Chen. 2022. Artificial intelligence in a structurally unjust society. _Feminist Philosophy Quarterly_, 8(3/4).
* Marsella (2005) Anthony J. Marsella. 2005. Culture and conflict: Understanding, negotiating, and reconciling conflicting constructions of reality. _International Journal of Intercultural Relations_, 29(6):651-673. Special Issue: Conflict, negotiation, and mediation across cultures: highlights from the fourth biennial conference of the International Academy for Intercultural Research.
* Meertens and Lion (2008) Ree M. Meertens and Rene Lion. 2008. Measuring an individual's tendency to take risks: The risk propensity scale1. _Journal of Applied Social Psychology_, 38(6):1506-1520.
* Messeri and Crockett (2024) Lisa Messeri and MJ Crockett. 2024. Artificial intelligence and illusions of understanding in scientific research. _Nature_, 627(8002):49-58.
* Murawski et al. (2024) Alaine Murawski, Vanessa Ramirez-Zohlfeld, Johnathan Mell, Marianne Tschoe, Allison Schierer, Charles Olvera, Jeanne Brett, Jonathan Gratch, and Lee A Lindquist. 2024. Negotiage: Development and pilot testing of an artificial intelligence-based family caregiver negotiation program. _Journal of the American Geritatrics Society_, 72(4):1112-1121.
* Pruitt (2007) Dean G Pruitt. 2007. Conflict escalation in organizations. In _The psychology of conflict and conflict management in organizations_, pages 261-282. Psychology Press.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of machine learning research_, 21(140):1-67.
* Rousu et al. (2015) Matthew C. Rousu, Gregory Colson, Jay R. Corrigan, Carola Grebitus, and Maria L. Loureiro. 2015. Deception in experiments: Towards guidelines on use in applied economics research. _Applied Economic Perspectives and Policy_, 37(3):524-536 

Omar Shaikh, Valentino Emil Chai, Michele Gelfand, Diyi Yang, and Michael S. Bernstein. 2024. Reharsal: Simulating conflict to teach conflict resolution. In _Proceedings of the CHI Conference on Human Factors in Computing Systems_, CHI '24, New York, NY, USA. Association for Computing Machinery.
* Tinsley (2004) Catherine H Tinsley. 2004. Culture and conflict: Enlarging our dispute resolution framework. _The handbook of negotiation and culture_, pages 193-210.
* Van Kleef et al. (2004) Gerben A Van Kleef, Carsten KW De Dreu, and Antony SR Manstead. 2004. The interpersonal effects of anger and happiness in negotiations. _Journal of personality and social psychology_, 86(1):57.
* Yamaguchi et al. (2021) Atsuki Yamaguchi, Kosui Iwasa, and Katushide Fujita. 2021. Dialogue act-based breakdown detection in negotiation dialogues. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 745-757.
* Yao et al. (2017) Jingjing Yao, Jimena Ramirez-Marin, Jeanne Brett, Soroush Aslani, and Zhaleh Semnani-Azad. 2017. A measurement model for identity, face, and honor cultural norms. _Management and Organization Review_, 13(4):713-738.
* Yongsatanchot et al. (2024) Nutchanon Yongsatanchot, Tobias Thejll-Madsen, and Stacy Marsella. 2024. Exploring theory of mind in large language models through multimodal negotiation. In _Proceedings of 24th ACM International Conference on Intelligent Virtual Agents_, Glascow Scotland. Association for Computational Linguistics.
* Zhang et al. (2014) Qin Zhang, Stella Ting-Toomey, and John G. Oetzel. 2014. Linking Emotion to the Conflict Face-Negotiation Theory: A U.S.-China Investigation of the Mediating Effects of Anger, Compassion, and Guilt in Interpersonal Conflict. _Human Communication Research_, 40(3):373-395.

## Appendix A GPT4 Emotion Labelling

### Construct Validity

We check whether our GPT emotion labels, compared with T5 Raffel et al. (2020), accurately capture expressed emotion. We do not have ground truth, however we do have self-reported frustration. Thus, we assess how well each predicted emotion correlates with this self-report. We see GPT-4o outperform T5 in the magnitude and direction of correlation, as seen in Table 3.

### Predicting Subjective Outcomes

For a random sample of 406 dialogues, we regressed the SVI scales on the emotion scores. We analyzed how well the model fits the data (\(R^{2}\)) with GPT4 scores compared to T5. Figure 7 depicts the varying \(R^{2}\) values across the different configurations. The biggest leap in fit (\(R^{2}\)) comes from using GPT-4o rather than T5, as GPT explains almost half the variance in several measures.

### GPT Prompt

Figure 8 outlines the prompt used for GPT4o in the emotion labeling task.

## Appendix B Task Background

This section of the appendix provides further details about the instructions for the participants, as well as the interface. Figure 9 depicts what the crowd worker would have seen when recruited to the task. Figure 11 shows instructions the participants read before the task. Figure 12 shows the interfaces through which participants would enter pre-task measures, such as their preferences or aspirations.

 

You are a good emotion classification tool. Your task is to classify the emotion of the last speaker based on the contextual dialogue.

Your output should be a JSON object with an 'emotion' field, categorizing the dialogue with a score for each: joy, anger, fear, sadness, surprise, compassion, or neutral. These scores should sum to one. If an utterance is neutral, then neutral must be one with everything other label set to zero.

Here are a few examples of proper annotations:

("statement": "Hi I I 'd like to return my jersey.", "emotion": ["joy": "@", "anger": "@", "fear": "@", "sadness": "@", "surprise": "@", "compassion": "@", "neutral": "1")],

("statement": "Please understand this was for my dear nephew he loves Kobe. I understand we had a misunderstanding, last thing I want is to hurt your business. Let's resolve this together", "emotion": ("joy": "@", "anger": "@", "feran": "@", "sadness": "@", "surprise": "o", "compassion": "@", "neutral": "@")},

("statement": "Thank you!", "emotion": ("joy": "1", "anger": "0", "fear": "@", "address": "@", "surprise": "@", "compassion": "@", "neutral": "@")},

("statement": "I will report you to authorities for doing this ", " emotion": ("joy": "@", "anger": "1", "fear": "@", "sadness": "@", "surprise": "@", "compassion": "@", "neutral": "@")}

Figure 8: This outlines the prompt GPT used in the emotion annotation task.

Figure 9: Depiction of the Prolific recruitment page for crowdworkers.

 

* [10] M. C. B. K. S. Chan, and J. A. R. Williams, "The role of a new object in the cloud: The use of a new object for the cloud," _IEEE Trans. Pattern Analist._**56**(2), no. 3, pp. 479-484, 2002.
* [11] D. L. G. K. S. Chan, and J. A. R. Williams, "A new object for the cloud," _IEEE Trans. Pattern Analist._**56**(1), no. 3, pp. 101-118, 2002.

[MISSING_PAGE_POST]

 



**Rank the importance of these issues**

People usually can't get all that they want in a dispute. Of the issues discussed above, rank how important each issue is to you by assigning points. You have 100 points to distribute across the four issues. More points mean the issue is more important. You must assign all 100 points. (You will be asked about these later)

Figure 12: Mechanisms for participants to input pre-dispute responses – their preferences and aspirations.