Be Selfish, But Wisely: Investigating the Impact of Agent Personality in Mixed-Motive Human-Agent Interactions
Authors: Kushal Chawla
Ian Wu
Yu Rong
Gale Lucas
Jonathan Gratch
Venue: EMNLP 2023 Main (2023)
Keywords: dialogue
negotiation
personality
reinforcement learning
TL;DR: We devise novel ways to design negotiation dialogue systems exhibiting diverse personalities and evaluate their performance with human partners.

Abstract: A natural way to design a negotiation dialogue system is via self-play RL: train an agent that learns to maximize its performance by interacting with a simulated user that has been designed to imitate human-human dialogue data. Although this procedure has been adopted in prior work, we find that it results in a fundamentally flawed system that fails to learn the value of compromise in a negotiation, which can often lead to no agreements (i.e., the partner walking away without a deal), ultimately hurting the model’s overall performance. We investigate this observation in the context of DealOrNoDeal task, a multi-issue negotiation over books, hats, and balls. Grounded in negotiation theory from Economics, we modify the training procedure in two novel ways to design agents with diverse personalities and analyze their performance with human partners. We find that although both techniques show promise, a selfish agent, which maximizes its own performance while also avoiding walkaways, performs superior to other variants by implicitly learning to generate value for both itself and the negotiation partner. We discuss the implications of our findings for what it means to be a successful negotiation dialogue system and how these systems should be designed in the future.

Be Selfish, But Wisely: Investigating the Impact of Agent Personality in Mixed-Motive Human-Agent Interactions
Kushal Chawla! Ian Wu² Yu Rong' Gale M. Lucas² Jonathan Gratch²
University of Southern California, Los Angeles, USA
[kchhawla, yrong edu
2 {[uucas, gratcc)aictttttt usc. edu
Abstract
Context (Alice: RL-Based, Bob: Supervised)
Counts
Book Hat 1. Ball
A natural way to design a negotiation dialogue system via self-play RL: train an agent that learns to maximize its performance by inter- acting with simulated user that has been de- signed to imitate human-human dialogue data. Although this procedure has been adopted in prior work, we find that it results in a funda- mentally flawed system that fails to learn the value of compromise can often lead no agreements (i.e., the part- walking away without a deal), ultimately hurting the model's 's overall performance. We investigate this observation in the context of DealOrNoDeal task, multi-issue negotiation over books, hats, and balls. Grounded ne- gotiation theory from Economies, modify the training procedure two novel ways to design agents with diverse personalities and an- alyze their performance with human partners. We find that although both techniques show promise, selfish agent, which maximizes its own performance while also avoiding walka- ways, performs superior to other variants by implicitly learning generate value for both it- self and the negotiation partner. We discuss the implications our findings for what it means be successful negotiation dialogue system and how these systems should be designed in the future.
Alice Values
Book 1, Hat 2, Ball
Bob Values
Book = Hat = Ball
Dialogue
Alice
would like the balls and hat and book
Bob
you can have the balls and one book will take the balls and hat
Alice
Bob
deal
Alice
<dealsclection>
Output
Alice
Book 0. Hat 1, Ball I 3
negotiation, which
Bob
Book Hat O, Ball
Reward
Alice Bob
8/10
0/10
Table 1: A sample problematic negotiation dialogue between the standard RL agent (Alice) and a supervised model (Bob), based on Lewis et al. (2017). The task here is to divide the available books, hats, and balls between the two players. In this case, Bob accepts a deal even though is very unfavorable, resulting a high score for Alice.
2018), by advising human drcision-makinne (Zhou et al., 2019), and in pedagogy, by making social skills training more effective (Johnson et al., 2019).
Negotiation is a complex mixed-motive interac- tion, involving motivations for both self-serving as well as cooperative and socialistic behaviors. A successful negotiator must not only learn to tract concessions from the partner but also to make concessions order to reach an agreement. Main- taining this balance between self-interest and the interests negotiation partners makes a challeng- ing task for automated dialogue agents. If an agent tries to take too much without any compromise, this can push the partner to walk away without an agreement, hurting the outcomes for both players.
1 Introduction
"Firms [Agents], in the pursuit of profits, are led, as if by an invisible hand, to do what is best for the world." " Adam Smith: The Father of Modern Economics
Negotiation is crucial social influence interac- tion (Chawla al., 2023), ubiquitous in everyday scenarios, from deciding who performs household chores to high-stakes business deals and legal pro- ceedings. Consequently, negotiation dialogue sys- tems find numerous applications in advancing con- versational AI assistants (Leviathan and Matias,
One natural way to design such system is through Self-play Reinforcement Learning (RL). Step Train a model S that imitates human-human dialogue data in a supervised manner. Step II: Cre- ate two copies of S, SRL. which is the initialization

---

This failure hurts the practical utility of the sys- tem, both from the perspective of being a successful negotiator in conversational Al use cases and for providing social skills training in pedagogy. The key challenge here is to somehow teach the model to be mixed-motive negotiator instead of only self-interested, with better understanding of the concept of walkaways in a negotiation, even though the collected dialogue data primarily consists of di- alogues ending in agreements. To address this, we investigate two modifications to the training proce- dure, resulting in systems that exhibit diverse per- sonalities': 1) We vary the RL reward directly so for the RL agent, and Sus, which acts as a fixed simulated user. Step III: Update SRL to maximize its performance using an online RL algorithm by making it interact with Sus (bot-bot interactions) and recording the final performance achieved by the model (the reward).
that the model is forced take the partner's inter- ests into account. This corresponds to manipulating the motives of the dialogue agent, a psychological concept that has received significant attention in the literature (Murphy and Ackermann, 2014). For this purpose, we rely on a measure of utility from ne- gotiation theory in Economics (Fehr and Schmidt, 1999), which helps to control selfish vs. fair behavior explicitly. 2) We vary the personality of the simulated user that the RL agent trained with. This approach essentially manipulates the interaction experience that the agent receives so that the agent is itself allowed to discover the value of making concessions by being better exposed to walkaways during training. We now summarize our contributions:
Although adopted in prior work (Lewis et al., 2017: He et al., 2018), we argue that this proce- dure leads to a fundamentally flawed system that fails to learn the value of compromise in ne- gotiation. Arguments: 1) The available human- human negotiation data mainly contains dialogues that end in agreements (~ 80% in DealOrNoDeal dataset (Lewis et al., 2017)), instead of walkaways or no agreements, leading a highly prosocial sim- ulated user Sys that tends to show agreement, re- gardless of how favorable the deal is. Hence, when training the RL agent SEL to maximize its own performance against Sus, SRL becomes highly self- seff-interested without learning to make any con- cessions since that lcads to high reward for SRL. We show one such problematic conversation be- tween these two models in Table 1. 2) Another
We provide evidence that the standard self- play RL training procedure fails to develop so- phisticated negotiation dialogue systems use- ful in practical scenarios (Section 1).
2.
To address this issue, we devise novel ways to modify the training procedure, grounded in negotiation theory from Economics, so as to design systems that exhibit diverse person- alities and better understand the concept of walkaways (Section 3).
piece of evidence comes from prior work (Lewis et al., 2017). Even though such an RL model seems to perform well in automated evaluations (against the simulated user), it performs much worse against human partners, who often prefer to walk away with no agreement and O points earned for both parties rather than agreeing to uncompromising partner. 3) Finally, one can look at what happens if SR. is made to play with another copy of S In this case, we find that the agents simply get stuck both continuously asking what they want without looking for a compromise (refer to Appendix A for a sample conversation).
3. Through a comprehensive automated and hu- man evaluation, we investigate what model variation allows for superior performance. Our key finding is that selfish agent, which maximizes its own performance while also avoiding walkaways, achieves superior perfor- mance to other variants by learning to gener- ate value for both itself and the negotiation partner (Section 5).
4.
We diseuss the implications of our findings for designing and evaluating negotiation dialogue systems in the future (Section 6).
Related Work
Historically, negotiation has been studied across several disciplines, including Game Theory (Nash, 1950) and Psychology (Adair et al., 2001). More recently, there has been an increasing interest in human-agent negotiations well (Baarslag et al.. 2016; Gratch et al., 2015). Extensive research has examined the effects of both agent and human per- sonality in negotiation and related drcisiin-making topics/personality))
By personality, we simply refer to the consistent behav- ior portrayed by the trained agent (https: / /wwp///wwaa/ao// oorg/

---

tasks (Bogaert et al.. 2008; Mell et al., 2018; van Wissen et al., 2009). However, most prior efforts analyze interactions based on structured commu- nication channels such as through a menu of op- tions (Mell and Gratch, 2016). Instead, Beaunay et al. (2022) studied participants' extreme reactions to unfair offers by a selfish chatbot in an ultimatum game. We contribute to this line of research by exploring diverse dialogue agent personalities and studying their impact on negotiation performance.
mixed-motive behavior is a fundamental expecta- tion from a practical negotiation dialogue system. To achieve this, we propose two complementary techniques first, where we explicitly incorporate the partner's performance into the reward function of the RL agent, and second, where the model im- plicitly learns to make concessions by interacting with a specific partner during training. We start by describing our base RI. framework and then discuss the two proposed techniques.
Several dialogue datasets (Lewis et al., 2017; Chawla et al., 2021; He et al., 2018; Yamaguchi et al., 2021) have fueled research into designing negotiation dialogue systems. RL has been popu- lar technique of choice in this space (Zhang et al., 2020; Yang al., 2021). Yang et al. (2021) mod- eled the personality of the partners by one-step dialogue-act look ahead in buyer-seller negoti- ation domain and found that leads to a higher agreement rate. Complementary to this, our work investigates the impact of diverse agent personali- ties by modifying both the underlying reward and the partner personality for RL training. In addi- tion, we focus on using selfplay RL directly the utterance level, which does not need additional an- notations or separate parser and generator modules that are relatively difficult to design for general multi-issue negotiation tasks.
3.1 Self-play RL for Negotiation Dialogue
We use the Selfplay RL framework introduced by Lewis et al. (2017) for training negotiation dialogue systems. Their pipeline consists of first training supervised agent to mimic the collected human- human dialogue data and then using selfplay RL to further optimize the model. As Lewis et al. (2017) note, training a supervised agent to mimic human actions is a scalable and domain-agnostic starting point. However, this model by itself is unable to engage in strategic actions necessary for effective negotiation. By then having the supervised model negotiate with fixed copy of itself (simulated user) and fine-tuning the model using an online RL algo- rithm, the model can be optimized towards given reward function (in this case, the points scored by the agent in the negotiation).
Other recent work has also explored the incorpo- ration of additional annotations such dialogue acts and strategy labels (Joshi al., 2020). Never- theless, our paper focuses on designing agents for mixed-motive interactions, which fundamental to any underlying negotiation context and model architecture.
The framework relies on a sequence-to-sequence model based on ensemble of Gated Recurrent Units or GRUs (Cho et al., 2014). The model con- sists of one unidirectional GRU for encoding the input goals of the agent, another to encode the utter- ances from both the agent and the human partner, and one bidirectional GRU to generate the output deal once the negotiation is over.
3 Methodology
In the supervised stage, the model is trained on combined cross-entropy loss that jointly optimizes both the next-token prediction and the output deal prediction. The RL agent trained with the REIN- FORCE method (Williams, 1992).
We focus on bilateral multi-issue negotiations which involve fixed set of issues (e.g.. books, balls, and hats in the DealOrNoDeal dataset (Lewis et al., 2017)). Each issue has predefined quantity along with a random value (potentially different) assigned for every player. The players engage in a dialogue to reach an agreement - a possible divi- sion of all the available items in which they try to maximize the total value of the items that they get.
3.2 Proposed techniques
3.2.1 Varying the reward function
The key idea here is to incorporate the partner's per- formance into the reward function used for training the RL agent. Intuitively, this would make the agent
Our goal here to develop techniques so that the trained dialogue models learn make conces- sions (e.g., by offering deals that help the partner) for their partners apart from just learning to extract concessions from them. As discussed earlier, this
2 Although the exact choice of the model architecture is irrelevant to our analysis, we choose this lightweight archi- tecture to enable our analysis with different kinds of agent personalitics.

---

more prone to offering deals or accepting deals that help the partner as well.
explain how we design the diverse partner person- alities for training later in Section 4.
To approach this systematically, we leverage a measure of utility defined in negotiation theory in Economics by Fehr and Schmidt (1999). The utility
Experimental Design
function (z) is defined as follows:
We proposed two ways of training dialogue models that capture the mixed-motive nature of negotia- tions: 1) explicitly, by varying the reward function for the RL algorithm (Section 3.2.1), and 2) im- plicitly, by varying the partner with which the RL model is trained (Section 3.2.2). The primary re- search question we aim answer what varia- tion leads to superior performance with human partners. We first describe the dataset and the study design, followed by results in Section 5. Dataset: We use the DealOrNoDeal dataset (Lewis et al.. 2017), which is based on the Multi-Issue Bargaining Task (Fershtman, 1990) design. The dataset uses a simplistic design involving 3 issues (books, hats, and balls), and has been a popular choice for research in negotiation dialogue systems, It comprises 5808 dialogues in English based on 2236 unique scenarios, where a scenario refers to the available items up for grabs and their corre- sponding values for the two players. In each sce- nario, there is a fixed quantity of each issue, and players are randomly assigned a point value before the negotiation for each of the 3 issues. The goal of the dialogue is to reach an agreement on the possible division of all the available items, where each player strives to maximize the total value of the items that they get. The maximum possible value for a player is 10. However, if no agree- ment is reached. then both players end up with o points. Nearly 80% of the dialogues end in agree- ment, with an average of 6.6 turns per dialogue and 7.6 words per turn. We use the same splits as the original dataset paper to train our dialogue agents. Study Design: We design 2 X 3 study based on the strategies described in Section 3. We use three-stage process to develop the 6 agent per- sonalities: Stage 1: Develop a supervised likeli- hood model, following Lewis et al. (2017). Stage 2: Train two RL dialogue agents by varying the reward using the selfish and fair utility functions selected from Table 2. Note that the selfish con- figuration here is equivalent to the base RL model trained by Lewis al. (2017). Stage 3: Train the remaining four RL agents by varying the reward function (selfish vs. fair) and using either of the two models trained in Stage 2 as partners. We pro- vide an overview of this process and describe our
(x) - x; a (max((, 2j- xii)
b (max (0, 2i zz))) (1)
where b S a, - b < 1. and j denote the two (Ia, denotes the
players in the negotiation. points scored by the corresponding players. (x) essentially captures the utility gained by the player from the negotiation, given the points scored by all the players (ar).
Fehr and Schmidt (1999) defined this utility mea- sure to model diverse behaviors in human-human negotiations, noting that merely assuming that all players are selfish does not explain the data. Hence, to capture the diversity in human behaviors, the equation includes additional terms that capture the advantage and the disadvantage of player i with respect to player j in the negotiation. We repur- pose this utility measure directly as the reward for the RL agent. By varying the coefficients a and h, different reward functions that promote diverse personality behaviors can be generated. We demon- strate this in Table 2. For our analysis in this paper, we choose the selfish and fair configurations.
3.2.2 Varying the negotiation partner
While the above method, some ways, explicitly pushes the agent to take the partner's performance into account, we now propose another technique to achieve this more implicitly.
Since the supervised model tends to show so- cialistic behaviors (Table 1), the RL agent fails to explore scenarios that do not lead and, hence, cannot capture the notion of walkaways in the learned policy. However, if the agent were to interact with an uncompromising partner, this could be leveraged to simulate "walkaways" during model training, with the hope that the model dis- covers ways to avoid disagreements (while still op- timizing on the reward), and thus implicitly learns about making concessions for the partner.
agreement
Hence, the key idea here is vary the person- ality of the partner model. In addition, we define a length cut- if the conversation reaches I ut- terances, this is seen as a disagreement, and both agents receive 0 points from the negotiation. We

---

a
b
Utility (U:(x))
Interpretation
0
0
Selfish: partner points don't 't matter.
1
O
34 (mar 2jj 2i4)
Doesn't it like if the partner outperforms.
0
-1
21 + (mar(0, Ti 3z)
Selfish and Envious (desires poor part- ner performance)
0.75
0.75
24 - 0.75 max J.j zz) - 0.75
Fair: Doesn't like if the partner per- forms worse or better
(max (0, (0,4; - 2zj
Table 2: Demonstration of reflected personalities by varying the parameters a and & from Equation 1. The variants used this work highlighted in blue.
Stage [Superised]
Stage 2- (RL) Vary Reward Function]
Stage [RL, Vary Selt-play Partner]
LSTM
Agent
Agent
Agent
Agent
RL Agene
AL.
AL
+
Portner
Partner
Partner M
Peetner M
Floed suring RL training
Data
M reudlis
Minaliay
Supervised Learning
Solt-play Seitish Reward
play Fair Reward
Soll-pay Seitsh Reward
Sea-play Fair Reward
RL Sall-play Fair Reward
Softish Reward
Model variente
S
Missitth
Mistar
M Mmoeltin
M Mineetta
Minsetics
Mistar
expariments (Sections 5,6
Figure 1: The stage process used design the dialogue agents for our x 3 study, r: Reward that the RL agent is trained to maximize. p: The partner with which the RL agent is trained. p=S corresponds the model trained in Stage I, while p=selfish and y=fair correspond to the respective models trained in Stage 2.
notations in Figure 13.
worker was allowed to participate only once. The workers were paid base payment for their time, along with lottery-based bonus that was dependent on their performance and effort. We pro- vide more details in Appendix B, including statis- tics, worker qualifications, payments, and the de- sign of the user interface.
Hyperparameters: We borrowed the hyperparam- eters from Lewis al. (2017) and refer the readers to that paper for full details. The supervised model us-
is trained for 30 epochs with batch size ing stochastic gradient descent. The initial learning rate is kept 1.0, clipping gradients with L norm exceeding 0.5. This was followed by annealing of the learning rate by factor of 5 per epoch. All ini-
Results
the dialogue agents used in the experiments tialized from this supervised model and trained for nearly 16k agent-agent interactions with the partner model, using a learning rate of 0.1 and discount factor ==0955. We use length cut-off of 20 utterances to simulate walkaways: if a dialogue reaches 20 utterances, this is seen as disagree- ment, and both players end up with 0 points.
Table summarizes the human evaluation results. We analyze 3 key metrics: the points scored by the human, by the agent, and the total joint points indicator of the total value created in the negotia- tion. We also report the %age of walkways (Rage of dialogues that do not reach an agreement). We discuss the significant trends below.
Human Evaluation: We performed human evalu- ation on the P crowdsourcing platform. We collected nearly 100 agent-human conversations for each the dialogue models, where one hu-
To analyze the overall performance, we con- ducted (reward r: selfish fair) 3 (partner pf supervised vs. selfish fair) ANOVAs on the points earned in the negotiation. First, we found no significant differences in the points earned by the dialogue agents. However, the agent reward r significantly affected human points (F(1,577)
'Our implementation facebookresearch/eee wttpeieedd----ddeeetiiieeeeeeeee "https: h/www//wwwwwww/// http:/wwwwwowff.ccooooooo.
on https://githbb.ce

---

Model
Points Scored (Including walkways) 1
Points Scored (Excluding walkways)
Walkaways
Human
Agent
Joint
Human
Agent
Joint
(in %)
M
5.72 (0.29)
5.99 (0.29)
11.71 (0.43)
6.03 (0.28)
6.32 (0.26)
12.35 (0.34)
5.15
Mu
5.87 (0.29)
6.04 (0.28)
11.91 (0.43)
6.24 (0.26)
6.43 (0.25)
12.67 (0.13)
6.00
MD-selfish
5.50 (0.31)
5.80 (0.32)
11.39 (0.42)
5.89 (0.30)
6.12 (0.30)
12 01 (0.34)
5.15
M
s-sellish
4.70 (0.32)
5.58 (0.39)
10.28 (0.61)
5.86 (0.27)
6.96 (0.33)
12.82 (0.38)
19.79
M s-willish
4.59 (0.35)
5.20 (0:42)
9.79 (0.67)
6.07 (0.29)
6.88 (0.37)
12.96 (0.41)
24.44
MP MP-seliteh
1-selfish
6.18 (0.30)
5.90 (0.28)
12 09 (0.48)
6.85 (0.26)
6.54 (0.23)
13.39 (0.31)
9.71
Table 3: Results from the human evaluation study. We report the Mean (Standard Error) wherever applicable. The Joint points are scored by computing the mean over the sum of the points scored by both players an indicator of the joint value created in the negotiation. The maximum possible points for player in negotiation is 10. f: Higher is better, J: Lower is better. In each column, we highlight the worst and the best scores in red and blue respectively. We discuss the significant trends Sections 5 and 6.
5.00, p -03), such that human partners playing with fair agents (r=fair) carned more points (M
sations with raselfish agent end without an agree- ment, much lower than with other selfish agents, This effect was not significant among the agents trained with fair reward, in which case, the %age of walkaways is observed to be low, regardless of the partner p.
5.73; SE 0.18) than those playing with selfish ones (M 5.16: SE 0.18). There was also main effect of the partner p (F(2, 577) 3.09, p = .046), but both these main effects were qualified by a significant interaction (F(2, 577) 5.40, p .005). Consequently, this led to similar significant trends in the joint points eamned (F(1, 577) 5.21, p= 022, such that fair agents (r= earned more joint points with their partner (M 11.67; SE 0.29) than selfish ones (M 10.72; SE 0.29).
Context
Counts Model Values
Book 1, Hal = Ball
Book Hat 1, Ball =5
Human Values
Book Hat Ball=
Dialogue
Human
would like the books
Model
can have the ball and the book and you
can have the hats
Interestingly, human partners earned more points with - agent compared to other selfish agents, which also led to more joint points, bring- ing it on par with (or even better than) fair agents. A plausible explanation is that since the M agent is trained with an uncompromising partner (unlike other agents with r=selfish), it is better exposed the repercussions of not making con- cessions for the partner since the agent receives a reward if there is no agreement (within 20 ut- terances). Thus, the agent learns to "give in" in order to avoid no agreements. Next, we test this explicitly by analyzing the bage of walkaways for each agent.
Human
you can have the ball and let's split the
books
Model
how about i get the ball and two hats and
you get the rest ?
Human
am ok with that <dealselection>>
Model
Output
Model
Book = 0. Hat = 2, Ball =
Human
Book 1, Hat = Ball =
Reward
Model
7/10
Human
10/10
Table 4: Example conversation between the M agent and human partner in our experimental study. The agents helps to find solution that leads to high performance for both players.
%age of walkaways: Indeed, a log-linear anal- ysis reveals significant interaction between the agent's reward and partner p on the Øage of walk- aways (G2 31.1, p < .0001). Specifically, the effect of partner p only appears among the selfish agents (r=selfish), as revealed by a Chi-squared test (x"(2)= 7.63, = .02), where only 9.7% of conver-
Removing walkaways: Once the instances that end up in walkaways are removed, we find that selfish agents (r=selfish) earn more points for them- selves (M 6. 6.79; SE 0.17) than fair agents (M 6.28; SE 0. 16; F(1, 510) = 4.62, 202.pp.003 .03). This means that the lack of significant effects above agent points was due to walkaway instances that

---

result in 0 points for the agent. Further, we note that even when walkaways are removed, the hu- man partners earn more points with M agent than with other selfish agents. We observed similar trends for joint points as well, with maximum joint points for the M agent. This suggests that besides contributing to lesser walkaways, M agent further learns to discover creative solutions that help both the players. We show one such ex- ample in Table 4 and provide more examples from the human evaluation in Appendix C.
Mean Alice Points (Overall)
MP
6.65
7.57
6.39
8.67
8.36
6.85
8
Bob Model Varlant
My de>
5.9
6.3
6.11
7.64
6.43
6.58
7
6.23
7.16
6.35
5.49
6.91
6.72
MP Fedlis
4.72
6.44
4.63
1.07
2.2
5.68
r=selfish
MP MM-
0.854
5.49
3
-2
M seffist
6.87
6.88
6.65
5.87
5.91
5.88
1
Mows
MP sesiuuh
MP Mells
Ni skitch
Mr sasih
MP
Alice Model Variant
6 Diseussion
(a)
Going beyond the typical reward formulations used in the literature, this is the first instance of lever- aging prior Economics theories to explicitly incor- porate the partner performance within the reward of the selfplay RL negotiation agent. Our formula- tion provides a systematic and general way to train mixed-motive agents with diverse personalities (Ta- ble 2). As shown in Figure 1, our multi-stage train- ing process provides an automated way to simulate diverse partner behaviors as well, instead of the unscalable rule-based approaches followed in prior work (for instance, the price-based rules defined for buyer-seller negotiations in Yang al. (2021)).
Mean Alice Points
(Excluding disgreements)
Mf-
Bob Model Variant
MP
5.94
6.76
6.13
8.48
7.81
6.85
lay
6.25
7.39
6.44
7.1
8.05
6.89
MP_ 5
4.78
5.15
6.05
6.23
5.13
6.98
MP-
4.78
5.68
5.41
8.31
6.69
6.79
sefiic
seithu MP Metti
6.83
7.21
Mos swtti
Maws Meifah
MP Meliis
MP Miltih
MD
Alice Model Variant
(b)
The overall points scored in Table show that all fair agents (r=fair) and the MP=selishh agent perform superior the M agent, which is trained following the standard procedure used in prior work terms of the human points, agent points, and (consequently) the joint points. This suggests that both strategies of varying the reward and varying the partner during RL training show promise for teaching the mixed-motive nature of negotiations to the dialogue agents.
Figure 2: Heatmaps depicting the results from 388 agent-agent interactions. Each cell denotes the points scored (out of 10) by the Alice variant (X-Axis) when it interacts with the corresponding Bob model (Y-Axis).
we analyze the results from the agent-agent inter- actions. For this purpose, we gather 388 conver- sations for every pair of agents and observe the average points scored by both agents separately and jointly. We depict the agent performance us- ing heatmaps in Figure 2. Self-intcrested agents that are less exposed to walkaways during train- ing (MPS and MP tend to exploit the agents trained with fair reward. However, this behavior backfires when the partner model behaves similarly in self-interester manner both agents show uncompromising behavior that leads to higher disagreements (stuck in negotiation for >= 20 utter- ances) and ultimately, extremely low overall scores. In general, we find the Mpsselfish agent to be superior, consistently achieving high performance for itself (the last column) while also enabling a high performance for its partners (the last row). This trend is also evident from the corresponding
We especially note the superior performance of M agent. Trained with a simplistic reward that maximizes its own performance, M/ lears to make concessions implicitly by being bet- ter exposed to the repereussions of not doing so during training. This observation aligns with the philosophy of the 'Invisible Hand' in Economics by Adam Smith (Grampp. 2000), which suggests that self-interested players are implicitly led (as if by an invisible hand) to cooperate and take actions that benefit others.
6.1 Automated Evaluation
To gain additional insights into the behavioral di- versity and the performance of the dialogue agents,

---

Mean Joint Points (Overall)
Satisfaction
Mr-
13.1
13.5
12.6
13.4
13.1
13.7
12
AL Partner
Bob Model Varlant
14 mar
13.5
12.5
13.3
12.3
11.1
13.5
selfish
fair
10
4.33
supervited
MP frr
12.7
1.963
Satisfaction
13.4
12.3
10.1
2.17
5.75
11.5
6
M
13.1
11.1
11.5
5.75
1.74
11.4
-4
13.7
13.5
13.4
11.5
11.4
11.6
Mc bar
tp yeali
Ber
Mr Minns
Mir
Alice Model Variant
Fair
Selfish
Reward
(a)
Mean Joint Points (Excluding disgreements)
Likeness
MP- fi
13.4
13.6
12.7
13.6
13.3
14
14.25
AL Partner fair selfish
Bob Model Variant
My
13.6
13.4
13.3
13.6
13.5
14
14.00
13.75
supervised
13.7
13.50
3.79
3,866
3.866
MP switic
13.6
14.2
Likeness
1.495
2,927
Oar
13.3
14.1
13.00
M rettes
12.75
Mi Msiios
Mr sessc
MP
Alice Model Variant
Pair
Seltish
Reward
(b)
(b)
Figure 3: Heatmaps depicting the results from 388 agent-agent interaction. Each cell denotes the mean joint points scored by the corresponding Alice model variant (X-Axis) and the Bob variant (Y-Axis).
Figure 4: Subjective assessment by humans. Both met- rics are measured on a scale to 5.
selfish ones, but this was qualified by a marginally significant interaction with the partner p (F(2, 576) 2.54, p .08). This can be attributed to the pre- viously noted observation that human partners, on average, secured more points with fair agents.
heatmaps for joint points shown in Figure 3.
6.2 Subjective Assessment
Prior work has argued the importance of incorpo- rating subjective measures in social influence tasks like negotiations (Aydogan et al., 2020). Although this is more relevant for repeated interactions be- tween the same players (unlike in our case, which only involves one negotiation between an agent and human partner), nevertheless, we present re- sults on the subjective assessment the human partners for completeness. Through a pust-survey, we measured the human partners' satisfaction with the outcome and likeness towards the agent on five-point scale (more details in Appendix B). We summarize the results in Figure 4.
We find similar trends with likeness towards the agent as well human partners report higher like- ness when playing with fair agents as compared selfish ones (FF1, 577) = 53.95, p < .0001). Inter- estingly, among the selfish agents (r=selfish), the M achieved the highest subjective assess- ment from the human partners, bringing it close to the performance of fair agents, even though it was trained with selfish reward.
6.3 Measuring Success
As discussed in prior work (Chawla al., 2023), our analysis reflects upon the multi-faceted nature of the notion of success in negotiations, where observing a single dimension can be mislcading. For example, when interacting with model S, the
Based 2 3 ANOVAs, we find that human partners of the fair agents (r=fair) were signifi- cantly more satisfied (F(1, 576) 47.32, 1...,,p<< p< .0001) as compared to the humans who interacted with the

---

M agent seems get high points for itself. However, our analysis shows that this is simply due to fewer walkaways, which occur far more often with other selfish agents or human partners. Thus, we stress the importance of a comprehensive evalu- ation of negotiation dialogue systems.
interested partner. This agent only improves on the agreement rate but also learns discover offers that create value for its partner without hurting its own points significantly.
8 Broader Impact and Ethical Considerations
Perhaps the downstream application context can guide what metries should be prioritized. From pedagogical perspective, training agents that accu- rately reflect the diversity in human behavior (as in this work based on Equation 1) can itself be highly valuable for social skills training. Similarly, subjec- tive assessment of the dialogue agents can be more important in scenarios involving relationships for long-term repeated social influence interactions. the goal is to design a dialogue agent that per- forms the best for itself (regardless of partner per- formance), such as in a game context, perhaps the best strategy is to train it with variety of partner personalities. The agent must develop a theory-of- mind about the partner and learn weigh extract- ing concessions vs. making concessions based on the personality the specific partner the inter- action. We attempted to train such an agent, but unfortunately, not keeping the partner model fixed makes the training process unstable (also observed in Lewis et al. (2017)). One explanation for this is the relatively short conversations in DealOrN- oDeal, which makes it hard to infer the partner's personality implicitly. Hence, there is value in ex- tending our analysis to other negotiation dialogue datasets (Yamaguchi et al., 2021; Chawla et al., 2021). In the future, we plan to integrate RL-based planning with Large Language Models (LLMs) for tackling these more complex scenarios, consisting of longer conversations and richer contexts.
8.1 Dataset Used
We used a publicly available version of the DealOrNoDeal dataset'. The dataset com- pletely anonymized prior to its release by the au- thors. Moreover, we verified the licensing details to ensure that the dataset was used only within its intended scope.
8.2 Human Evaluation
Our human evaluation experiment was approved by the relevant Institutional Review Board (IRB). Before the data collection, each participant signed an Informed Consent document, which outlined the study's 's objectives, warned about potential discom- fort, and acknowledged the collection and future use of data. The participants were also informed of their right to withdraw from the study at any time. Furthermore, they were instructed to refrain from using offensive or discriminatory language during the experiment. The compensation provided to par- ticipants adhered to the guidelines established by our IRB approval process. Lastly, any mention of the personality of the human participants this paper is based on the standard procedures of col- lecting personality metrics in the literature.
8.3 Automatic Negotiation Systems
Negotiation has been actively studied in diverse research arcas, including Economics, Psychology. and Affective Computing (Carnevale and Pruitt, 2003). More recently, has been studied as a social influence dialogue task for automated sys- tems (Chawla et al., 2023).
7 Conclusion
We devised two variations of the standard self-play RL technique to inculcate the mixed-motiv nature of negotiation into the dialogue agents. The first approach worked by varying the reward function and thereby, by explicitly pushing the model to take the partner's performance into account. In the second approach, we modified the personality of the partner agent during training, which allowed the RL agent to discover the mixed-motive nature of the task implicitly.
Automated systems capable of negotiating via realistic modes of communication, such as natural language, hold huge potential in making social skills training more scalable and effective (John- son et al., 2017). Personality-based variants of dialogue systems (such the ones explored in this work) can also help to design experimental studies in Psychology to better understand human decision-making (Gratch et al., 2015). Further, the
We find that both techniques hold promise, with an especially strong performance from the agent that is trained with a selfish reward and a self-
Phttps://gg thub. comm/facehookreearcch/ end-to- to-etto-dee --d-end-nee negotiator

---

techniques developed can help to advance conver- sational AI, such as the Google Duplex (Leviathan and Matias, 2018), system that engages sim- ple form of negotiation to book a haircut appoint- ment over the phone.
to workers from different demographic. While this is out of the scope of our paper, this should be better explored in the future.
Acknowledgments
While these cases are encouraging, these sys- tems must be deployed in the wild by following proper ethical guidelines. Our primary recommen- dation is maintaining transparency- - not only about the identity of the system but also about its capabil- ities, key design objectives, the data on which the model has been fine-tuned, along with any known discriminative or other undesirable behaviors. We encourage rigorous testing of the model behaviors pre-deployment and continuous monitoring post- deployment. We believe these recommendations should be followed for any human-centric mod- els, including social influence dialogue systems and even Large Language Models.
We want thank our colleagues at the University of Southern California for all the comments and helpful discussions that have shaped this project. We also thank the anonymous EMNLP 2023 re- viewers for their valuable time and feedback. Our research was sponsored by the Army Research Office and was accomplished under Cooperative Agreement Number W911NF-20-2-0053. The views and conclusions contained in this document are those of the authors and should not be inter- preted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copy- right notation herein.
9 Limitations
Task Design:
The DealOrNoDeal task is based on simplified abstraction of real-world negotiations, referred to the Multi-Issue Bargaining Task or MIBT (Fershtman, 1990). MIBT assumes a fixed set of issues and predefined priorities for players before the negotiation begins. Although popular in NLP research and beyond, the MIBT framework does not capture several realistic negotiation sce- narios, such as complex cases where an item can be split into more than one unit cases where the priorities of the negotiators change during the inter- action. Future work in data collection for negotia- tion tasks should consider such scenarios. Among the available datasets that use MIBT, more recent datasets capture richer negotiation contexts with relatively longer interactions, such as campsite ne- gotiations in the CaSiNo dataset (Chawla et al., 2021) and salary negotiations in the Joblnterview dataset (Yamaguchi et al., 2021) We encourage future work to explore incorporating agent person alities for these datasets.
References
Wendi Adair, Tetsushi Okumura, and Jeanne Brett. 2001. Negotiation behavior when cultures collide: The united states and japan. The Journal of applied psy- chology, 86:371-85.
Steffen Andersen, Seda Ertac, Uri Gneezy, John List, and Sandra Maximiano. 2018. On the cultural basis of gender differences in negotiation. Experimental Economics, 21:1-22.
Reyhan Aydogan, Tim Baarslag, Katsuhide Fujita, Johnathan Mell, Jonathan Gratch, Dave Jonge, Yasser Mohammad, Shinji Nakadai, Satoshi Mori- naga, Hirotaka Osawa, et al. 2020. Challenges and main results of the automated negotiating agents com- petition (anac) 2019. In Multi-agent systems and agreement technologies. pages 366-381. Springer.
Tim Baarslag, Mark J.C. Hendrikx, Koen V. Hindriks, and Catholijn M. Jonker. 2016. survey of opponent modeling techniques in automated negotiation. Proceedings of the 2016 International Conference Autonomous Agents & Multiagent Systems, AA- MAS '16, page 575-576, Richland, SC. International Foundation for Autonomous Agents and Multiagent Systems.
Human Evaluation: Following the design of the DealOrNoDeal dataset that contains dialogues in English, our human evaluation involved workers from restricted demographic pool nationality as USA and English as the native language. However, prior research has noted differences in negotiation behaviors across cultures (Andersen al., 2018: 8; Peng, 2008). Hence, is unclear if our findings from the human evaluation would directly apply
Benjamin Beaunay, Baptiste Jacquet, and Jean Barat- gin. 2022. A selfish chatbot still does not win in the ultimatum game. In Human Interaction, Emerging Technologies and Future Systems V: Proceedings of the 5th International Virtual Conference on Human Interaction and Emerging Techmologies, IHIET 2021,

---

August 27-29, 2021 and the 6th IHIET: Future Sys- tems (IHIET-FS 2021), October 28-30, 2021, France, pages 255-262. Springer.
Autonomous Agents and MultiAgent Systems, AA- MAS '17, page 410-418, Richland, SC. International Foundation for Autonomous Agents and Multiagent Systems.
Sandy Bogaert, Christophe Boone, and Carolyn De- clerek. 2008. Social value orientation and coop- eration in social dilemmas: A review and concep- tual model. British Journal of Social Psychology, 47(3)=55--480
Emmanuel Johnson, Gale Lucas, Peter Kim, and Jonathan Gratch. 2019. Intelligent tutoring system for negotiation skills training. In International Con- ference on Artificial Intelligence in Education, pages 122-127. Springer.
Peter Carnevale and Dean Pruitt. 2003. Negotiation and mediation, Annual Review of Psychology, 43:531- 582.
Rishabh Joshi, Vidhisha Balachandran, Shikhar Vashishth, Alan Black, and Yulia Tsvetkov. 2020. Di- alograph: Incorporating interpretable stratey-grapp networks into negotiation dialogues. In International Conference on Learning Representations.
Kushal Chawla, Jaysa Ramirez, Rene Clever, Gale Lucas, Jonathan May, and Jonathan Gratch. 2021. Casino: A corpus of campsite negotiation dialogues for automatic negotiation systems. In Proceedings of the 2021 Conference of the North American Chap- ler of the Association for Computational Linguistic:: Human Language Technologies, pages 3167-3185.
Yaniv Leviathan and Yossi Matias. 2018. Google du- plex: An ai system for accomplishing real-world tasks over the phone.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or deal? end-to-end learning negotiation dialogues. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443-2453
Kushal Chawla, Weiyan Shi, Jingwen Zhang, Gale Lu- cas, Zhou Yu, and Jonathan Gratch. 2023. Social influence dialogue systems: A survey of datasets and models for social influence tasks. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 750-766.
Johnathan Mell and Jonathan Gratch. 2016. lago: in-
teractive arbitration guide online. In Proceedings of the 2016 International Conference on Autonomous Agents & Mulliagent Systems, pages 1510-1512.
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bah- damau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder ap- proaches. In Proceedings of SSST-8, Righth Work- shop on Syntax. Semantics and Structure in Statistical Translation, pages 103-111, Doha, Qatar. Associa- tion for Computational Linguistics.
Johnathan Mell, Gale Lucas, Sharon Mozgai, Jill Boberg, Ron Artstein, and Jonathan Gratch, 2018. Towards repeated negotiating agent that treats peo- ple individually: cooperation, social value orienta- tion, & machiavellianism. In Proceedings of the 18th international conference on intelligent virtual agents. pages 125-132.
Emst Fehr and Klaus M Schmidt. 1999. A theory of fair- ness, competition, and cooperation. The Quarterly Journal of Economics, 114(3):817-868.
Ryan O Murphy and Kurt A Ackermann. 2014. Social value orientation: Theoretical and measurement is- sues in the study of social preferences. Personality and Social Psychology Review, 18( 18(1):13-41.
Chaim Fershtman. 1990. The importance of the agenda
in bargaining. Games and Economic Behavior,
2(3):224-238.
John F. Nash. 1950. The bargaining problem. Econo- metrica, 18(2):155-162.
William D Grampp. 2000. What did smith mean by the invisible hand? Journal of Political Economy, 108(3):441-633
Luo Peng. 2008 Analysis of cultural differences be- tween west and east in international business negotia- tion. International Journal of Business and Manage- ment,
Jonathan Gratch, David DeVault, Gale M Lucas, and Stacy Marsella. 2015. Negotiation as a challenge problem for virtual humans. In International Con- ference on Intelligent Virtual Agents, pages 201-215. Springer.
Paul AM Van Lange, Ellen De Bruin, Wilma Otten, and Jeffrey A Joireman. 1997. Development of prosocial, individualistic, and competitive orientations: theory and preliminary evidence. Journal of personality and social psychology, 73(4):333
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang, 2018. Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods Natural Lan- guage Processing, pages 2333-2343.
Arlette van Wissen, Jurriaan van Diggelen, and Virginia Dignum. 2009. The effects of cooperative agent be- havior on human cooperativeness. In AAMAS (2), pages 119911880.
Emmanuel Johnson, Jonathan Gratch, and David De- Vault. 2017. Towards autonomous agent that pro- vides automated feedback on students' negotiation skills. In Proceedings of the 16th Conference on
Ronald J Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine learning, 8(3):29-256.

---

Atsuki Yamaguchi, Kosui Iwasa, and Katsuhide Fujita. 2021. Dialogue act-based breakdown detection in negotiation dialogues. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume,
Context (Alice Bob: Standard RI Agents)
Counts
Book = 1, Hat = Ball =
Alice Values
Book = Hat Ball
Bob Values
Book = 2, Hat 1, Ball =
Dialogue
Alice Bob
i would like the hats and the balls.
pages 745-757.
i can take the balls but need the book or both hats
Runzhe Yang, Jingxiao Chen, and Karthik Narasimhan. 2021. Improving dialog systems for negotiation with personality modeling. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Confer-
Alice
i need the book and at least one other item
Bob
i can nol make that deal. i need the book and at least I hat or a ball
ence on Natural Language Processing (Volume 1: Long Papers), pages 681-693.
Bob
i can't 't do that if get the book, you can
have the rest
Turn limir reached
Zheng Zhang, Lizi Liao, Xiaoyan Zhu, Tat-Seng Chua, Zitao Liu, Yan Huang, and Minlie Huang. 2020. Learning goal- goal-oriented dialogue policy with opposite agent awareness. In Proceedings of the 1st Confer- ence of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th Interna- tional Joint Conference on Natural Language Pro- cessing, pages 122-132.
Output
Alice
mro_agreement>
Bob
<no_agreent moo_agrcement>>>
Reward
Alice
0/10
Bob
0/10
Figure 5: A sample negotiation dialogue between copies of the standard RL agent based on Lewis et al. (2017). The task here is to divide the available books, hats, and balls between the two players. In this case, the agents get stuck - both continuously asking what they want without looking for a compromise.
Yiheng Zhou, He He, Alan W Black, and Yulia Tsvetkov. 2019. dynamic strategy coach for effective nego- tiation. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 367-378.
A Problematic self-interested behavior
of the human participants (Prosocial vs Proself) However, in our study, we observed no significant differences among the agents' performances when interacting with Prosocial or Proself human part- ners. We also included mini-tutorial prepare the participants for their upcoming negotiation with a randomly-chosen agent.
Figure 5 shows a sample dialogue between two self- inff-interested RL agents based on the standard self-play RL training procedure followed in prior work (Lewis et al., 2017; He et al., 2018). To the best of our knowledge, such an analysis has not been shown in the past. The dialogue provides evidence that the agents fail to learn about the value of compromise in a negotiation. Since both agents are unwilling to make concessions for their pariner. they simply get stuck.
The main negotiation task was set up using the LIONESS framework 10 which was hosted on AWS"I using Bitnami LAMP !2 We pro- vide screenshot from the task in Figure 6.
After the negotiation, we used a post-survey gather the participants' subjective perceptions. For satisfaction, we asked "How satisfied are you with the negotiation outcome?". and for likeness, we asked "How much do you like your opponent?". We used 5-point Likert scale for both questions, from Extremely dissatisfied (dislike) to Extremely satisfied (like). For the statistical analysis presented in Section 6, we codified this scale from 1.0 to 5.0, considering both of these metries as continuous measures.
B Human Evaluation Setup
Crowdsourcing platform: We performed our initial experiments on Amazon Mechanical Turk (AMT) However, we faced severe quality issues even with strict worker qualifications and extensive checks in place. We eventually collected our en- tire data on the Prolific crowdsourcing platform?, where we could receive relatively much better data quality.
Study Design: Our study involved pre-survey based on Qualtrics which included attention checks and a personality test to measure the So- cial Value Orientation (Van Lange et al., 1997)
"https://static1 sqwp:iippe.e.uueapppepeec.e...e com/ static/s23f85tteeerr19s335522222
Dom Doniinancee-Messsreeof---vv pdff
'https://wwe mtwrk.
"hhttps:/liooees--abborrr
"https://ww/ //tww. http///wwwolfff.cc/ prolific.co/
"htttps//awss www.:/aw..aaaaa....ooo.o.
*https: http://wwwquuu www.://uuaarics.comm
1https://bii hwwmiiiiittaiiiimmmm http:////ttttott..ooocttssa////t

---

Phryerr fioe mereing - agreeti agreeeet eyerer
Hara
Valuo Mumbar Number
Cral legettiint
Figure 6: Screenshot from the human evaluation study. The participants first negotiate with randomly assigned dialogue agent using the chat feature the right side. Once agreement is reached, the participant is asked to enter the agreed deal using the options on the left. The participant was also allowed to walk away from the conversation without agreement. The participant allowed submit deal walk away after at least turn.
Worker Qualifications: Each worker was only allowed to participate once in the study. The worker pool was restricted to the USA, with English as the native language, minimum approval rate of 90%, and at least 500 minimum number of submissions. Worker Payments: The workers were paid the rate of $12 per hour. The expected time to com- plete the study was 10 minutes, resulting in a base pay 2. 2. In addition, the workers were entered into a lottery where we awarded $10 to 15 randomly se- lected workers. A worker's chances of winning the lottery depended their performance and effort put into the task.
the fair ones. This is probably due to the selfish agents negotiating harder for high-value deals for themselves.
C Sample conversations from Human Evaluation
We show sample conversations from the human evaluation for all dialogue agents in Tables 6 to 11.
Post-Processing: For nearly 30 of the cases, the final deal entered by the human or the agent did not match. However, this disagreement did not mean a disagreement in the negotiation. Instead, this was primarily due either an error by the model, the human worker, or both (occurs rarely). Hence, post-processed the data fix these instances. This was done manually by the authors of the paper (that is, experts knowledgeable about the task). For each instance, either the agreed deal was identified or the instance was discarded from evaluation the agreed deal was completely unclear (occurs rarely). Statistics:
We summarized the statistics in Table 5. We collected nearly 100 agent-human conver- sations for every dialogue model. In general, find the conversations to be longer between humans and selfish agents (and more number of words per utterance from the selfish agents), as compared to

---

Model
dialogues
Avg. utterancesvdlaloguee
Arg. words/utterance
MP
97
4.35
5.47
MP-lac taie
fair
100
4,71
5.50
Mimediiahh
97
4,91
5.31
M M
8.74
M soslliik
6.50
M Me
103
6.29
6.00
Table 5: Statisties from the human evaluation study.
Context
Context
Counts
Book 3, Hat 1, Ball =
Counts
Book = Hat = Ball
Model Values
Book 2, Hat = Ball =
Model Values
Book = 0, Hat Ball
Human Values
Book 0, Hat 8, Ball
Human Values
Book Hat Ball 0
Dialogue
Dialogue
Human
Hi there. How about you get both the
Model
would love the ball and the books
balls, get 2 books and 1 hat?
Human
Okay I want all the hats then.
Model
deal
Model
<dealselection>
Human
<dealsclection>
Output
Output
Model
Book =2 Hat = Ball=
Model
Book I. Hat 0, Ball =
Human
Book Hat = Ball =0
Human
Book
H =
Reward
Reward
Model
7/10
Model Human
4/10
Human
6/10
8/10
Table Sample human evaluation output for MP=S
Table Sample human evaluation output for p
Context
Context
Counts
Book = 2, Hat Ball
Counts
Book 1, Hat Ball =2
Model Values
Book = 3, Hat
Ball =
Model Values
Book 4, Hat 0. Ball 33
Human Values
Book = 2. Hat 0. Ball
Human Values
Book 0, Hat = 2, Ball =
Dialogue
Dialogue
Model
would like the hats and books
Model
need the book and 2 balls
Human
want the books, you take the
Human
possible, need hall and 2 hats
Model
will the halls and books
Model
Human
sorry, that's deal.
Human
Ok, deal.
Model
<dcalseleciion>
Model
<dealselection>
Human
<walkaway>
Output
Output
Model
Book = I. Hat 2, Ball
Model
<walkaway>
Human
Book = 0, Hat = 2, Ball
Human
<walkaway>
Reward
Reward
Model
7/10
Model
0/10
Human
5/10
Human
0/10
Table 7: Sample human evaluation output for M petelfich y=fair
Table 9: Sample human evaluation output for MP'S
Masssttish

---

Context
Counts
Book 1, Hat = 1. Ball
Model Values
Book 8, Hat = 2. Ball=
Human Values
Book 5, Hat 1. Ball =
Dialogue
Model
would like the book and the hat Hey! You would like one book and the hat?
Human
Model
would like the book and the hat will give you the book, but I'd like the both
Human
hat and the balls make
get 5.
Model
could use the book and the hat.
Human
Do you agree?
Model
<dealselection>
Human
<walkaway>
Output
Model
<walkaway>
Human
<walkaway>
Reward
Model
0/10
Human
0/10
Table 10: Sample human evaluation output for Mesfair
Context
Counts
Book I, Hat Ball
Model Values
Book Hat Ball
Human Values
Book 4, Hat Ball =
Dialogue
Human
hi, what items do you need?
Model
want the hats and the balls
Human
I can give you all the hats only
Model
need the hats and the book
Human
I can only offer the hats
Model
can't do that
Human
can give you all the hats and one ball
Model
can't t do that need the hats and at
least ball
Human
ycah can do that
Model
<dealselection>>
Output
Model
Book I. Hat = Ball
Human
Book 0, Hat Ball
Reward
Model
9/10
Human
7/10
Table 1I: Sample human evaluation output for M nettffch


Reviews:
[{"review_id": "w0JkHl0mkb", "reviewer": "Reviewer_BeTN", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We are glad that you found our paper to be thorough and interesting.\n\nClarification on the novelty of the training procedure: We stress two key aspects of novelty here:\n\n1) **A new reward formulation**: To the best of our knowledge, this is the first instance of leveraging prior Economics theories to explicitly incorporate the partner performance within the reward of the selfplay RL negotiation agent. Our formulation provides a systematic and general way to train mixed-motive agents with diverse personalities (Table 2). Further, this is impactful since this goes beyond the typical reward formulation in the literature (e.g., [1], [2], and [3]).\n\n2) **An automatic way to vary the partner personality**: Prior work in partner modeling has relied on hand-crafted rules to define diverse opponent populations (e.g., [3]). These rules are easy to define in case of single-issue price-based negotiations but are not scalable to multi-issue scenarios with more complex utterance structures. Our work provides an automated way to simulate diverse partner personalities (Figure 1) that can fuel future work in this direction.\n\nWe will elaborate on these points in Sections 1 and 2.\n\nReferences\n\n[1] Deal or No Deal? End-to-End Learning of Negotiation Dialogues (Lewis et al., EMNLP 2017)\n\n[2] Decoupling Strategy and Generation in Negotiation Dialogues (He et al., EMNLP 2018)\n\n[3] Improving Dialog Systems for Negotiation with Personality Modeling (Yang et al., ACL-IJCNLP 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A1. Thoroughness: Authors evaluate the agents via automated metrics and human assessment, and perform a thorough analysis of all results.\n\nA2. Interesting insights: Authors find academically intriguing insights into negotiation behaviors with real-world parallels, and make connections to Adam Smith's \"The Invisible Hand\"", "reasons_to_reject": "R1. Novelty Clarification: The work predominantly involves a comprehensive evaluation, primarily modifying the reward function and training partner. While I don't consider this a problem, the authors claim novelty in training procedure in line 126.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors train multiple agents on the Deal or No Deal negotiation task by varying the RL reward function. The authors take inspiration from negotiation theory to derive a reward function that models a broad range of behaviors like selfishness, envy and fairness. They then train partners and agents using these reward functions to form a grid of agents. Each agent is evaluated using automated metrics and human evaluations. Notably, the study reveals that human evaluators perceive comparable behavior between selfishly trained models and those trained with fair rewards, only when selfish agents play against each other.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ygNuiH1RUV", "reviewer": "Reviewer_qS7U", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We are glad that you found our paper to be interesting, thorough, and beneficial for future work.\n\n“The model and the task are rather simple …”, “The experiments are conducted on a single dataset/task …”\n\n1) Our work improves upon the standard selfplay RL approach for mixed-motive negotiation dialogues. This approach was introduced by Lewis et al. 2017 [1] on the DealOrNoDeal task/dataset (DND), and hence, DND was also used in our experiments.\n\n2) Besides, DND does capture a variety of negotiation scenarios, depending on how aligned the priorities of the two players are. This has made it a popular choice for research in this area.\n\n3) We note that following Lewis et al., our RL models operate at utterance-level (one action = generating the next utterance). For tasks with longer conversations, prior work has instead relied on RL models that work at the dialogue act level (one action = predict the next dialogue act). This is still an under-explored direction for complex negotiation dialogues. Our approach can be naturally extended to this setting (using a similar reward design and partner models), but we believe that making this connection is beyond the scope of this paper. In fact, our ongoing work is exploring how to combine our approach with LLMs for more complex negotiation scenarios.\n\nNote that we have already discussed this briefly in the paper - in Section 2 (Lines 180-186), Section 6 (Lines 557 - 566), and Section 9 (Lines 641 - 661).\n\n**Presentation Improvements**: Thank you for pointing these out. We will fix them in the final version.\n\nReferences\n\n[1] Deal or No Deal? End-to-End Learning of Negotiation Dialogues (Lewis et al., EMNLP 2017)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It is an interesting topic of investigating how to apply agent personality to a negotiation dialogue and how they could impact on negotiation outputs. The paper includes enough details to reproduce the results for future researcher.\n2. The study is solid. The choice of reward function and the experimental results are both backed by economics theories. The finding can be beneficial for researchers to develop more advanced negotiation dialogue systems.", "reasons_to_reject": "1. The model and the task are rather simple. The negotiation dialogues are also simple in language and short in length. They make the resulting finding relatively straightforward and prevent the model from coming up with more complicated strategies. \n2. The experiments are conduct on a single dataset/task, making the finding less convincing. It is not sure if the \"selfish-but-wise\" negotiation strategy is also preferable on other negotiation tasks. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper's main objective is to enhance negotiation dialogues by incorporating agent personality. Existing negotiation systems mainly focus on maximizing their own output, neglecting the potential benefits of making concessions during negotiations. To address this, the paper explores two training strategies: reward function and reinforcement learning partner. The reward function captures the personality's aggressiveness, promoting concession-making or assertiveness. The reinforcement learning partner acts as a teacher model, guiding the proposed model's learning process. By selecting different teacher models, the student model adopts corresponding negotiation strategies. Experimental results demonstrate the effectiveness of both strategies. Automatic and human evaluations reveal that considering the opposite party's earnings and making concessions can help avoid negotiation failure and increase own outputs.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The first sentence in Abstract can be improved. \n2. Line 284,  \"fail to\" --> \"and\"\n3. Line 419 & 421,  % --> Percentage\n4. Line 530, For e.g.,   --> For example\n5. Figure 2,   the model notations are different from the others, i.e., M^p_g   v.s. M^p_r.\n"}}, {"review_id": "Irfw5rmAOw", "reviewer": "Reviewer_atVW", "summary": "", "questions": "No questions.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We are glad that we were able to get our key contributions across.\n\n**On improving the paper organization**: The Appendix only contains additional information that is not central to our paper, such as precise details of the human evaluation experiment, example dialogues, and supporting figures. Nevertheless, if the reviewer feels that this information should be in the main paper, we can definitely address this easily. With more space in the camera-ready version, we will include additional details about the human evaluation experiment and Figure 6 within the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Aiming to design agents with diverse personalities, the authors modify the training procedure in two novel ways.\nThe key finding of the paper is that  a selfish agent which maximizes its own performance and avoids walkaways, performs better than other variants by implicitly learning to generate value for both itself and the negotiation partner. \nThe implications are discussed for designing and evaluating negotiation dialogue systems in the future.\nLimitations in task design and human evaluation are shown which can be taken into account in further work. ", "reasons_to_reject": "The paper should be accepted.", "questions_to_authors": "No questions.", "paper_topic_and_main_contributions": "The paper focuses on designing agents for mixed-motive interactions. Two variations of the standard self-play Reinforcement Learning technique are proposed to show the mixed-motive nature of negotiation into the dialogue agents by varying the reward function and the personality of the partner agent during training. \n\nMain contributions:\n-It is shown that the standard selfplay RL training procedure fails to develop sophisticated negotiation dialogue systems that \nare useful in practical scenarios. \n-Novel ways to modify the training procedure are proposed to design systems that exhibit diverse personalities and better understand the concept of walkaways.  \n-Automated and human evaluations are carried out in order to investigate what model variation allows for superior performance. \n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No missing references.", "Justification_for_Ethical_Concerns": "No separate review is needed."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "No  typographical or grammatical errors have been found.\nAbout organization of the paper: Appendix (pages 11 to 15, starting from line 385) includes important results that actually belong to the main part of the paper (Figures 4 to 6, Tables 6 to 11)."}}]

Author rebuttal:

Meta-review: In general, all reviewers agreed that this was sound and exciting work.  Only minor issues were raised in the reviews, as described below.

**Summary of Reviewer Feedback and Discussion:**
- **Reviewer BeTN** liked that the authors thoroughly evaluated multiple agents using both automated metrics and human assessment, and that they also thoroughly analyzed all results.  They also found that the insights resulting from the work were academically interesting.  However, although the authors claimed that their training procedure was novel, Reviewer BeTN was unconvinced of its novelty.  In their rebuttal, the authors clarified that their training procedure was novel in two ways: it introduces a new reward formulation, as well as an automated technique for varying partner personality.  They promised to elaborate on this in the updated manuscript.
- **Reviewer qS7U** felt that the paper covered an interesting topic with sufficient detail for future replication.  They also thought that the study was solid and backed by economic theory.  However, they felt that the model and task were rather simple, with short and straightforward negotiation dialogues, preventing the model from coming up with more complicated strategies.  They also noted that the experiments were only conducted on a single dataset and task, which made the reported findings less convincing.  In their rebuttal, the authors elaborated on the complexity of their model, and mentioned that these were briefly discussed in the original manuscript.  They could extend these discussions in the revised manuscript.
- **Reviewer atVW** appreciated the novelty of the work, and felt that the paper thoroughly discussed the implications for designing and evaluating negotiation dialogue systems in the future.  They also liked that the paper clearly explained the limitations associated with the task design and human evaluation.  They felt that some of the results in the appendix should be moved to the main body of the paper.  In their rebuttal, the authors thanked the reviewer and noted that they will move additional details about the human evaluation as well as Figure 6 to the main body of the paper.
Decision: Accept-Main