INA: An Integrative Approach for Enhancing Negotiation Strategies with Reward-Based Dialogue Agent
Authors: Zishan Ahmad
Suman Saurabh
Vaishakh Sreekanth Menon
Asif Ekbal
Roshni Ramnani
ANUTOSH MAITRA
Venue: EMNLP 2023 Findings (2023)
Keywords: Negotiation
Dialogue Agent
Prompting

Abstract: In this paper, we propose a novel negotiation agent designed for the online marketplace. Our dialogue agent is integrative in nature i.e, it possesses the capability to negotiate on price as well as other factors, such as the addition or removal of items from a deal bundle, thereby offering a more flexible and comprehensive negotiation experience. To enable this functionality, we create a new dataset called Integrative Negotiation Dataset (IND). For this dataset creation, we introduce a new semi-automated data creation method, which combines defining negotiation intents, actions, and intent-action simulation between users and the agent to generate potential dialogue flows. Finally, the prompting of GPT-J, a state-of-the-art language model, is done to generate dialogues for a given intent, with a human-in-the-loop process for post-editing and refining minor errors to ensure high data quality. We first train a maximum likelihood loss based model on IND, and then  employ a set of novel rewards specifically tailored for the negotiation task to train our Integrative Negotiation Agent (INA). These rewards incentivize the agent to learn effective negotiation strategies that can adapt to various contextual requirements and price proposals. We train our model and conduct experiments to evaluate the effectiveness of our reward-based dialogue agent for negotiation. Our results demonstrate that the proposed approach and reward functions significantly enhance the negotiation capabilities of the dialogue agent. The INA successfully engages in integrative negotiations, displaying the ability to dynamically adjust prices and negotiate the inclusion or exclusion of items in a deal bundle.

INA: An Integrative Approach for Enhancing Negotiation Strategies with Reward-Based Dialogue System
Zishan Ahmad" Suman Saurabh', Vaishakh Sreekanth Menon', Asif Ekbal',
Roshni Ramnani*, Anutosh Maitra²
Department of Computer Science and Engineering
Indian Institute of Technology Patna, India
{ [zeeman. zzemannzzihan,, saurabhm. saurabhm.ss89, vaishakhsm,
(agmale com
*Acenture Labs, Bangalore, India (roshni. ramnani,, anutosh. mai wwwrrraaacccture..ommm com
Abstract
(He et al., 2018). In such setting, negotiation between the customer and the seller is core facet of discourse that ultimately decides the profit sale and customer satisfaction. Negotiation on the price of a product is very common, however, cus- tomers have an open-ended approach to negotiation often also involving negotiation on certain aspects related to the deal. For example, while buying chair the customer may negotiate a deal without the cushions, or even negotiate between delivery and in-store pick-up. As a result, a dialogue sys- tem for negotiation in an online marketplace should be capable of engaging in negotiation on different aspects such as price, product, and delivery. Addi- tionally, such system should also be capable of responding to product inquiries with relevant and knowledge-grounded information.
this paper, we propose a novel negotiation dialogue agent designed for the online market- place. Our agent is integrative in nature i.e, it possesses the capability to negotiate on price well as other factors, such as the addition or removal of items from a deal bundle, thereby offering more flexible and comprehensive ne- dataset
gotiation experience. We create called Integrative Negotiation Dataset (LND) enable this functionality. For this dataset cre- ation, we introduce new semi-automated data creation method, which combines defining ne- gotiation intents, actions, and intent-action sim- ulation between users and the agent to generate potential dialogue flows. Finally, the prompt- ing of GPT-J, state-of-the-art language model, is done to generate dialogues for a given in- tent, with human-inh -n-un-ii--ae thop process for post- editing and refining minor errors to ensure high data quality. We employ a set of novel rewards, specifically tailored for the negotiation task to train our Negotiation Agent, termed as the In- tegrative Negotiation Agent (INA). These re- wards incentivize the chatbot to learn effective negotiation strategies that can adapt to various contextual requirements and price proposals. By leveraging the IND, train our model and conduct experiments to evaluate the effective- ness of our reward-based dialogue system for negotiation. Our results demonstrate that the proposed approach and reward system signif- icantly enhance the agent's negotiation capa- bilities. The INA successfully engages in inte- grative negotiations, displaying the ability to dynamically adjust prices and negotiate the in- clusion or exclusion of items in bundle deal'.
A systemic survey conducted by (Zhan et al.. 2022) discussed various datasets, evaluation met- rics, and methodologies in common literature. From this, it can be implied that bargain in the marketplace typically follows "Distributive" strat- egy where cach party involved aims to maximize their gain rather than mutually benefiting outcomes. This strategy follows a win-lose model, where one party can gain only if the other party loses. The CraigslistBargains dataset (He et al., 2018) is the most prominent dataset in the price bargain domain with other datasets having less than 1,000 dialogues. This dataset contains dialogues between two hu- man agents assigned the role of customer and seller negotiating over a product on Craigslist, the strat- egy used in the dialogues are largely distributive in nature. In contrast to a distributive approach, an "Integrative" approach to negotiation aims to reach a win-win situation by understanding the other party's needs and reaching a mutually satis- fying consensus. has been shown that an integra- tive approach to negotiation in retail e-commerce is more effective and leads to better customer sat- isfaction, than distributive approaches (Guttman
1 Introduction
In an online marketplace, customers and sellers en- gage in discussions involving product inquiry and bargaining before reaching common consensus
'Codes and datasct available at https://githul: com/zishan-ail ainee and httpp://ww.. www.wiitti aic in/ "ai-nlp-ml/resource"" html#INA

---

and Maes, 1998) that typically utilize agents that negotiate only on price. It is common in online marketplaces for products to have several items, such as a "A chair and its cushion", a negotiation agent that is capable of satisfying customers that only want select items from the product such as cus- tomers that only want chair or customers that only want a cushion is beneficial since the agent better understands customer requirements and may lead win-win outcomes. Hence, treating product as a "bundle" of items that customers can choose is a more integrative approach than treating the product as a single entity.
ond, we propose efficient approach for automat- ically generating data with minimal manual inter- vention, addressing the challenge of data scarcity in certain domains. This contribution will drive the de- velopment of more robust dialogue systems. Third, we create a unique dataset of integrative negotia- tion dialoges.. Finally, we leverage the strengths of both supervised and reinforcement learning to construct powerful dialogue system empowered by integrative negotiation strategies.
Related Work
Thompson et al. (2010) studied the effects of var- ious intra-personal processes, such as mood, and interpersonal processes, such as emotion, nego- tiation outcomes. They defined integrative nego- tiation as "the extent to which the negotiated out- come satisfies the interests of both parties in a way that the outcome cannot be improved upon with- out hurting one or more of the parties involved". They also reported that the studies on the effec- tiveness of computer-mediate negotiation with re- spect to face-to-face negotiation give mixed results. Laroche and Genevay (2016) stated the importance of user adaptation in negotiation dialogue systems by performing experiments using different policies on simulated users in a newly designed negotiation dialogue game. Zhao et al. (2018) proposes a semi- automatic negotiation wherein a dialogue manager decides the intent after which a natural language generator presents conversational strategies to hu- man expert that writes the final utterance. Lewis et al. (2017) prepares dataset and proposes end-to- end dialogue systems for "multi-issue bargaining". In this type of bargaining, two agents are presented with a set of items and asked to assign each item to one agent, each agent is also given a value function to decide the value of an item. He et al. (2018) prepares the CraiglistBargains dataset where two human agents negotiate over the price of product listed on Craigslist, further, they decouple negotia- tion strategy and dialogue generation by proposing dialogue manager to decide the intent of the next utterance and a generator that uses the intent to gen- erate the utterance. Following this work, Yang et al. (2020) proposes a framework to integrate "The- ory of mind" (Premack and Woodruff, 1978) for inferring personality types to enhance negotiation dialogues.
To incorporate this integrative approach, this paper, we propose a novel dialogue system for ne- gotiation in the online marketplace domain, which can respond to customers' inquiries and engage in negotiation with the customer. Unlike existing systems (He et al., 2018) that primarily focus on negotiation over the price of a product, our system follows more integrative approach wherein nego- tiation involves different aspects such as adding or removing products from the aforementioned "bun- dle" of products, the price of the bundle, and the delivery of the product. Datasets for negotiation such as the CraigslistBargains dataset do not explic- itly model the product as a bundle of smaller items. Hence, we construct dataset (IND) consisting of integrative negotiation dialogues where the deal is modeled as a bundle of products. To avoid com- plete manual data creation, we design prompts for the GPT-J model (Wang and Komatsuzaki, 2021) to generate integrative negotiation utterances. To ensure the dataset's 's quality, we use humans in the loop for minor edits and filtering of the generated dialogues.
Using the constructed dataset, we build an inte- grative negotiation poweeedd dialogue agent (INA) using a supervised learning (SL) + reinforcement learning (RL) approach. To train our system, we leverage a novel reward function and maxi- mize it using PPO loss (Schulman et al., 2017) to ensure aspects of negotiation consistency, ne- gotiation power, and intent consistency. As per our knowledge, this is the first attempt to build negettiatiinnno based dialogue system. Therefore we present pioneering effort in develop- ing an integrative-negotiation- based dialogue sys- tem, making several key contributions. First, we introduce a new task of integrative negotiation, ex- panding the scope of dialogue system research. Sec-
Unlike these previous works, our proposed nego- tiation agent (INA) is capable of doing integrative

---

negotiation. Our agent not only capable of nego- tiation with respect to the price of item but can also modify the deal to better suit the customer's preference. Similarly, our agent can also handle the customization of a deal proposed by the customer and decide on accepting or rejecting the deal. These capabilities are currently absent in any negotiation agent.
Greet: The utterances with general greetings like welcome and thank you come under this category.
Ask: This intent is triggered when a user ex- plicitly asks for information about an item or the ongoing negotiation.
The agent may use the 'inform' in- tent to share detailed information about the products or services involved in the negotia- tion.
Inform:
3 Dataset Creation
We construct the IND dataset for the task of inte- grative negotiation. To save on human effort and resources, we come up with a novel mechanism based on prompting large language model for dataset creation. We keep human annotators in the loop only for making minor edits and filtering the automatically generated dialogues to ensure the quality of the conversations. The overall pro- cess consists of creating skeleton of dialogues by dynamically deciding the correct intent for any arbitrary conversation. Our overall dataset creation process consists of 5 steps: (i). Background Data Creation, (ii). Intent Definition, (iii). Dialogue Flow Generation, (iv). Prompting for Dialogue Generation, and (v). Data Correction.
-akCClarffication: This intent captures the user's intention seek further explanation clarification regarding certain aspects of the negotiation or the overall deal according the current negotiation state.
Negotiate-Price- Ergetiate-Price--ncresse This intent indi- cates that the agent is seeking to increase the pricing terms of product service during the negotiation process.
Negotiate- Pwwec-rriieeeeeee This intent indi- cates that the agent is seeking to decrease the pricing terms of a product or service during the negotiation process.
3.1 Background Data Creation
This is an intent by the agent in a negotiation system indicat- ing the system's 's intention to propose or assert that the price remain unchanged during the negotiation pro- cess. This is ideally done by highlighting the value and fairness of the current deal.
Pegotiate-PriceNoCCCCC
Although our method can be adapted to any prod- uet negotiation, mainly focus list of 10 different electronic items: (i). Air Conditioning. (ii). Television, (iii). Refrigerator, (iv). Oven, (v). Washing Machine, (vi). Printer, (vii). Smart Phone, (vili). Laptop. (ix). Tablet, and (x). Camera. Along with these products, the deal bundle consists of a set of accessories related to the product. Therefore, our background database consists of the following information, such Product Name, Product De- scription, Product Features, Price, Accessory List, and Accessory Description.
product or service should
Negotiate-Add-X: This intent by the agent user refers to the intention propose or sug- gest the addition of specific item or feature to enhance the value a product or service during the negotiation process. This may or may not lead to an increase in the price of the deal.
3.2 Intent Definition
Negotiate- Remove-X: This intent by the agent or user refers to the intention to pro- pose or suggest the removal of a specific item or feature from the deal in the negotiation pro- cess. This may may not lead to decrease in the price of the deal.
In order to build a robust negotiation system it is vital to define intents that can cover diverse range of scenarios during negotiation. For an integrative negotiation agent, the scenario in the scope of the agent not just price negotiation, but also item- level negotiation the given bundle. To cover these properties, we come with the following i
Accept: This refers to the agent or user's in- tention to agree or accept proposal, offer, or condition reached during the negotiation process.
FExmmpl utterances for cach intent provided in Table 6 of the appendix.

---

Product bundle: tublet called Lenovo P11 in.5-nch OLED Stapdragon 7300 processor Stylns pen callidi Conpusibbe Amdrid clsiisss 2048 locels prosoue Memory callal Protessioaa SDXC Inttial price:
Card memory cord has capacity up ITB
Final price: 8.300
User: Hello. interested purchassng your Lenovo Tab Pi1 How
intenc: ćireer
Bot:
Hellal The Lenova Tab P11 Pro almn with stylis pen and memory card is bring eoki for S92500 Inser
Grusstofooe
User: Exeuse me, I do not need the snytus pei, it possible to buy the taller without the stylus pen? What would be the price? Intour: Negodiane-Rowene Remuse-X
Bol:
That is very popular product, I can sell you the tabler without the stylus pen for $9L100,0 Antent Inform
User:
That is very expeusive tablet, I'm ofradd I cun't affood it. How about $747007 Jolent: NeyotiatePPise Price-
Deewwar
understand very expensive However, I'm willing give you great deal at 583300. Pleeor me you would to purchase Insear: Negotione-Priee Prie-leemove
perfect
to buy
Accepe
grest, we can proczal with pusment Ackoosinige
Figure I: An example conversation between the negotiation agent and a customer
This refers to the agent or user's in- tention to agree or reject a proposal, offer, or condition reached during the negotiation pro- cess.
Reject:
for the computation of the proposed price by cus- tomer (Pt) or seller (P₂) at dialogue turn t. In the equations, A is constant to control the rate of price change from one turn to the next. If it k is larger there will higher rate of concession, at low value the rate of concession provided by the seller low. For our setting, we have assumed a higher k value for the seller and a lower l for the customer, considering the customer is strict with their budget.
The above intents can occur either individually or in combination with other intents (e.g.: Greet-Ask).
3.3 Dialogue Flow Generation
Our dialogue flow generator module assumes that the dialogue flow (intent sequence) during negoti- ation can be random. However, we also put some obvious constraints on this datasct-generation pro- cess. One simple constraint is that the conversation would be initiated by the customer with greet intent. This greet intent could be accompanied by a request for clarification or one of the 'negotiatc" intents for the customer. The agent can respond by the inform intent or one of the agent 'negotiate" intents.
Pst Pbt 1 + (Pst 1 Pbt 1)e i)e-
(1)
Pb. = Ps- (Pst Pb 1),-1)e
(2)
The seller will choose intent 'Accept' when the customer offered price is less than or equal the amount Ps, - tol * Psy. The customer will choose intent 'Reject' when the conversation has crossed the negotiation deadline, and the seller is no more ready to lower the bundle price. The dialogue flow terminates with the acknowledgment of 'accept' intent the 'reject' intent.
For all the deal bundles, we maintain negotia- tion details of the ongoing deal with the customer, which consist of: (i). Minimum Seller price, (ii). Current Seller price, (iv). Tolerance value (tol) and (iii). Current Customer price. To enforce the integrative nature of our agent, we limit only price-based negotiations to turns after which the Siegotiate-Add- or Remotiiatt-Rmove-"" in- tents would take over. To propose a price for the next turn, assume that decay in price differ- ence (increment for customer and decrement for seller) over dialogue turns. This is in line with Faratin et al. (1998) where a similar function is used to model the price negotiation between the customer and seller. Equations and 2 are used
3,4 Prompting for Dialogue Generation
We design few-shot prompts (Brown et al., 2020) for each intent, with around four shots for each prompt (due to the token limit of 2,048 in GPT-J). Each shot contains three parts, description of the task, summary of the relevant information from the dialogue, and an utterance following the intent, all in a natural language format. The summary of the relevant information is designed considering the intent flow of the previous utterances of the di- alogue. The description of the task is the sentences
EExmple prompts provided in Section B.I of the Ap- pendix

---

Dialogue Flow Generator
Stop
Constraint Rules (Con)
flow
empty
Deal Bundle Sampling
Random Next Intent Generator
intent Con
False
Dialogue Flow Queue
False
Pop
True
Pop first Intent
Add utterance
DB (tems accessories)
Intent
Generate prompt
Generate utterance
Figure 2: Overall data creation process
in the prompt that explains the situation and the goal of the intent, for instance, the task descrip- tion for the "Acknowledge acceptance" intent is "A customer has agreed purchase product from seller, the seller wants thank the customer and proceed with the transaction". The utterance fol- lowing the intent a manually designed utterance following the task description and the information summary of the shot.
ings, pretending to be human, were to be rephrased or removed (to maintain authenticity). The experts were also responsible for correcting minor gram- matical errors. Furthermore, they were asked to rate the fluency of each utterance scale of 0-2, where represented non-fluency and indicated complete fluency. Dialogues containing utterances rated as fluency were dropped from the dataset. These measures were implemented to uphold the quality standards of the dataset.
The flow generation module creates an ordered list of intents along with relevant information for each intent, for instance, for the intent "Negotiate- Add-X" the item to be added is mentioned, and for Errrotiate-Prie-Deeeree posed is mentioned. Our algorithm uses the list created by the flow generation module to create shot that is augmented to the prompt of the respec- tive intent, this prompt is then passed to the GPT-J model produce the utterance.
Dataset Statistic
The statistics of the dataset created are given Table 1. The dataset has a total of 4, 163 utterances and we follow an 12:8 split between train, test, and validation sets. The average number of turns per dialogue in the dataset is 13 and the number of unique words in the dataset, excluding numbers is 12,219, both these metries are comparable the metries in the Craigslist Bargain dataset (avg. turns:9; unique words: words:11,799) 1,799). Following (Wang et al., 2021), to automatically measure the vari- ability of conversations of our dataset, we com- pute BLEU-1 and METEOR scores between the utterances. We obtain low BLEU-I and METEOR scores of 0.08 and 0.05, respectively, indicating high variability between the utterances in IND. We ask three human experts rate the 'engagingness' and 'fainness' of dialogues on scale of to 3 (higher the better). The dialogues obtained an aver- age rating of for 'engagingness' and 2.26 for
the price to be pro-
3.5
Data Correction
To ensure the quality of the automatically generated dataset, we implemented manual correction and fil- tration steps. We engaged three human experts who possess post-graduate qualifications and have two years of experience in the field. Their instructions were make edits the generated dialogues in order ensure grounding the provided back- ground database, intent, action, and negotiation flow. Additionally, any utterances produced by the agent that referred to its own experiences or feel-

---

Train
Test
Valid
(R2), negotiation strategy reward (Rt) and interac- tiveness (R4) in generated responses. R intuitively nudges SLDM towards these aspects by providing appropriate respective aspect penalization/rewarr for generated responses. For example, if the model generates intent inconsistent response then R3 will penalize the model to discourage it from generating similar type of content. All five rewards can be written
#Dialogues
3330
500
#Utterances
45,914
6887
4592
Avg # of words in Cus-
19.30
19.32
19.29
tomer Utterance
Avg # of words in Sales-
33.13
33.32
33.27
Person Utterance
Table Statistics of the dataset created (IND)
faimnsss
5 Methodology
Intent consistency: In a negotiation system with complex intents there can often be divergence be- tween the predicted intent and the intent of the generated utterance. To enforce this consistency, we propose Intent Consistency (IC) reward. This reward function is implemented by first training a BERT model (Devlin et al., 201 8) on the training set of IND for the task of intent prediction. This task is modeled as a classification task where the input to the BERT model is an agent utterance turn L, Ua, and the expected output is the intent of the utterance Ja, The accuracy of the trained intent classifier 71.2%. We use the [CT.S] token for computing the probability distribution of the intent classes. We sample the probability value Pa of the intent predicted by our end-to-end SLDM dialogue model and use it as Rı (Eq. 6).
To force language model to negotiate with the user while following its own price goal as well as approach, we fine-tune it using novel-designed reward function in reinforcement learning set- ting. Here, first, pre-trained language model (GPT-2-medium) is fine-tuned in supervised setting using traditional cross-entropy loss be- tween the ground truth and predicted utterances probability distributions. For a supervised dia- logue dataset D = {do, d1, .., d where, {0o, (o,..., ai, u,, ar 1, ur 1) multi-turn di- alogue with u; + crt, (u, user's utterance at jth turn and crt; {ao, u0, () imput and aj (agent's utterance 1t turn) as output. The supervised learning dialogue model po(d) can be expressed as:
R1 = Pa()))
(6)
(d(d) II Pa (ureeeeaeeeaappa/ueaaaaee T=0
Price Gap Reward:
The purpose of negotiation is to find a win-win solution for both the customer and the seller. The winning scenario for a seller would be as little reduction in the initially proposed price as possible. In line with this logic, we propose Price Gap (PG) reward. This reward is simply the fraction of the initial proposed price by the agent Pai and the final selling price after negotiation Paf (Eq 7). The higher the final price the greater the reward.
(3) where Pa and Pa are the user's and agent's utter- ances prohability distributions. This trained SLDM is fine-tuned in an RL setting using the PPO loss formulated as below:
L CLIP (0)
cl))(pry(O), 1- =)A (4)
Paf P.:
R₂ =
(7)
where (0) Pge /'Pold, and Á, denote the clipping range and normalized rewards, respec- tively. Finally, parameters' updation done as follows:
Negotiation Strategy Reward: A successful ne- gotiation might not always entail deal acceptance. In cases where the customer wants to go below the minimum selling price of the agent Pa-min it would not be judicious for the seller to satisfy the customer. In such situations where the negotiation could result in win-lose situation, the deal should be rejected. Hence, the success criterion of the negotiation lies not just acceptance of the deal but also the faimess of the deal. To ensure that our negotiation succeeds only in win-win scenarios we design the Negotiation Strategy (NS) reward.
Ok+1 argmax E CLIP
(5)
s.a-PPoy
Here, normalized rewards are obtained by novel designed reward function (R) incorporating intent consistency reward (R1), price gap reward
'The overall inter-annotator agreement using Krippen- dortt's 's alpha (Krippendorff, 201 1) was found to he 0.84

---

Human evaluations were conducted by three postgraduate evaluators who possess proficiency in similar tasks. Each evaluator interacted with the proposed system 15 times and assessed the conver- sations hased on: (i). Negotiation Consistency (N- Con): It is the measure of consistency (absence of arbitrariness) in the negotiation approach within dialogue (ii). Bargaining Efficacy (B-Eff): It mea- sures the ability of the negotiation system present compelling arguments, reasoning. or incentives that influence the other party's 's decision-makinn pro- oess., (iii). Outcome fairness (O-fair): assesses the faimess or equity of the final outcomes reached during the negotiation process., (iv). Dialogue- fluency (D-F): It measures the overall grammat- ical correctness of the generated responses, and (v). Dingonne-Engagingness (D-E): Measures the extent to which conversation or dialogue inter- esting, captivating, and able to hold the attention of the participants. The evaluators assigned scores on scale of to for each metric (The higher the better).
Po - Pa Pa-min G[Intenty)
R3 F(
(8)
Po Pa-min
1, Intenty = accept
G(Intenty)
(9)
-1, Intenly = rejecl
0, < 0
F(x)
(10)
6², I 2 0
In the above equations, Po is the customer's pro- posed price, and Intent, E (Accept, Reject) is the final intent in the conversation used to capture the negotiation result. The reward incentivizes ac- ceptance of a deal when the negotiated price is within the limit of a minimum price for the seller, and rejection when the negotiated price is below this minimum price.
Interactiveness: To ensure interactiveness, repe- titions, and conversation blackholes are penalized such that system can engage the user for longer duration with interactive responses. To penalize the generation of similar utterances for a given intent in the dialogue we use Equation 11.
7 Results and Analysis
Automatic Evaluation: It can be noticed from Ta- ble 2 that the proposed INA performs better than all the four baselines viz. ARDM, ARDM BK (Background Knowledge). ARDM + In (Intent), and Neg-TOD (Hosseini-Asl et al., 2020), terms of all the five metrics viz. METEOR, BS-F1, WM, PPL, and R-LEN. For the evaluation metrics mea- suring similarity (of the generated utterance) with the gold utterance i.e METEOR, BS-FI and WM, INA attains scores of 0.43, 0.86 and 0.57, respec- tively. The obtained scores significant improve- ments <0.141, 0.042, 0.04>, <0.158, 0.032, 0.04>, <0.144, 0.032, 0.03> and <0.137, 0.029, 0.03> over the baselines, ARDM, ARDM+BK, ARDM+In, and Neg respectively.
R4 =
(11)
where otth is the vector (bag of words) representing the generated utterance with intent in. of" to oin are the vectors representing the previously gener- ated utterances in the dialogue with the same intent. The final normalized reward function is formu- lated as:
R = R4 + R2 + Ra + R4
(12)
with 71 + + + + 74- = 1.
6 Experiments
It can also be inferred that the difference of BS- F1, and WM scores decrease in the following order: INA> -ANA-NNAA---NN--GGIIMA importance of task specifie rewards in our proposed system INA.
6.1 Evaluation Metrics
To properly assess INA's performance, we perform both automatic and manual evaluations. In auto- matic evaluation to measure the surface similarity with the gold responses, we compute METEOR (Banerjee and Lavie, 2005). For semantic simi- larity, we compute BERT Score (BS-F1) (Zhang al., 2019) and Word Mover distance (WM). We also report the Perplexity (PPL) and the av- erage response length (R-LEN) of the generated responses.
This shows the
It can also be observed from Table 2 that INA obtains lower (better) PPL 1.56 score than that of ARDM, ARDM+BK. ARDM+In, and NogTod with difference of 1.39, 1.19, 1.24, and 1.37 points, respectively. Further, we obtain score of R-LEN = 39.93 is also better than that of ARDM, ARDM+BK, ARDM+In, and Neg-TOD with a dif-

---

Model
METEOR
BS-F1
WM
PPL
R-LEN
ARDM (Wu al., 2021)
0.289
0.823
0.53
2.95
25.22
ARDM BK
0.272
0.833
0.53
2.75
35.29
ARDM
0.286
0.833
0.54
2.80
30.30
NegTOD
0.293
0.836
0.54
2.63
32.32
INA-
0.39
0.833
0.54
2.16
39.15
INA- PG
0.31
0.832
0.53
2.34
39.11
INA NS
0.33
0.833
0.54
2.05
39,08
INA-1
0.34
0.831
0.53
2.30
39.13
INA
0,4.
0.865
0.57
1.56
39.93
Table 2: Results of automatic evaluation
Model
N-Con
B-Eff
O-fair
D-F
D-E
ARDM (Wu et al., 2021)
0.5
1.2
1.8
ARDM BK
1.2
0.4
1.4
2.4
2.2
ARDM In
0.8
0.4
1.2
INA IC
1.8
1.4
2.7
2.4
INA PG
1.4
2.6
2.4
INA NS
1.8
1.6
INA-I
1.6
2.5
2.0
INA
1.8
Table 3: Results of human evaluation
ference of 15.72, 2.28, 13.5, and 1.76, respectively. This indicates that the INA is able to generate longer responses, hence, showcasing more engag- ingness with the user. It can be due to the incor- poration of all four rewards where R1, R2, and R3 play the crucial role in handling negotiation and price consistency, and R& helps maintaining the non-repetitivennss, hence, driving the agent to build the rapport with a user as well as be on the goal by generating diverse and interactive negotiation responses.
Conclusion
In this paper, we have presented novel dialogue agent for negotiation (INA) in the online market- place domain, focusing on an integrative approach that goes beyond price negotiations. Our system can respond to customer inquiries and engage in ne- gotiations that encompass various aspects, such modifying product bundles, adjusting prices, and arranging product delivery. Unlike existing sys- tems that mainly concentrate on price negotiations, our approach provides a more comprehensive and versatile solution. To achieve this, we constructed a dataset of negotiation dialogues (IND) where the product is represented as bundle of smaller items. To minimize manual effort in data creation, we em- ployed prompts for the GPT-J model to generate integrative negotiation utterances. Using IND, we developed INA through combination of super- vised learning and reinforcement learning. Our training process incorporated a novel reward func- tion that suits the negotiation task, which we op- timized using the Proximal Policy Optimization (PPO) loss. Our results show that INA is able to perform integrative negotiations with the customer and enable engaging negotiations that can lead to win-win win-win&ee deal for the seller and the eustomer.
Human Evaluation: Table shows the human evaluation results for all the eight models viz. ARDM, ARDM+BK, ARDM+In, NegTOD, INA- IC, INA-PG, INA-NS and INA-I. It may be noted that INA yields better scores for N-Con. B-Eff, O-fair, D-F, and D-E compared the baselines. Scores of N-Con: 2.4, D-F: 2.8, and D-E: 2.6 shows that the consitecnn (IC) and interac- tiveness (l) rewards play crucial role in obtaining consistent, fluent, and engaging, responses as com- pared to other models. Further, in terms of B-Eff and O-fair, INA attains score of 1.8 for both. The ablation of the price-gap (PG) and negotiation- strategy (NS) rewards showcases the importance of these rewards in terms of B-Eff and O-fair. There- fore, it can be inferred that employing intent consis- tency, price gap, and negotiation strategy rewards help in a more consistent, persuasive, and overall fair negotiation with the customer.
In the future, it would be interesting to explore the role of the customer persona like age, gender, hobbies, etc. during negotiation.

---

9 Limitations
learners. Advances neural information processing systems, 33:1877-1901.
Our data creation steps and modeling have some limitations. First, to create the data, GPT-J is used which requires a large GPU memory size (here, 40 GB). Another limitation of GPT-J is that has a context window of 2,048 tokens, which constrains our prompting mechanism. Within this context window, we need to fit background data as well as dialogue history with a few shot examples. This allows us to only go for a maximum of 4 shots while prompting leading some hallucinations in the created data which needed to be fixed manually.
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. Bert: Pre-training deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:/810004805.
Peyman Faratin, Carles Sierra, and Nick R Jennings. 1998. Negotiation decision functions for autonomous agents, Robotics and Autonomous Systems, 24(3- 4)-15--182.
Robert H Guttman and Pattic Maes. 1998. Agent- mediated integrative negotiation for retail electronic commerce. In International Workshop on Agent- Mediated Electronic Trading, pages 70-90. Springer.
10 Ethical Considerations
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018. Decoupling strategy and gener- ation in negotiation dialogues. arXiv preprint arXiv:1888.06633
Since negotiation by nature entails bargain with the customer, it should be done ethically. Our integra- tive approach to negotiation gives greater flexibility to the customer and hence leads to a win-win sce- nario in negotiation. Our negotiation is not aimed at as zero-sum game where a party has to lose in order for the other to win. The customer at any point of the conversation can reject the deal and thus not compelled continue with the negotia- tion if it does not suit them.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019, The curious case of neural text degeneration. arXiv preprint 10XXv:190440997
Ehsan Hosseini-Asl, Bryan McCann, Sheen-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. Advances in Neural Information Processing Systems, 33:201 79- 20191.
The dataset created in this work will be made available only after filling and signing an agree- ment declaring that the data will be used only for research purposes. The annotation, filtering/edit- ing of data, and manual evaluations were done by human experts, who are regular employees of our research group and are paid accordance with the institute's policy. There are no other issues to declare.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
Romain Laroche and Aude Genevay. 2016. The negoti- ation dialogue game.
Mike Lewis, Denis Yarats, Yann N Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning for negotiation dialogue.. arXiv preprint arXiv:1706.05/25.
11 Acknowledgement
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint
Authors acknowledge the grant received from Accenture LLP for the project T"Convvrsational Agents with Negotiation and Influencing ability".
arXiv:1711.05100
David Premack and Guy Woodruff. 1978 Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4):515-522 526..
References
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188):
Satanjcev Banerjce and Alon Lavic. 2005. Meteor: An automatic metric for mt evaluation with improved cor- relation with human judgments. In Proceedings of the acl workshop on intrinsic and extiinicc evaluation measures for machine translation and/or summariza- tion, pages 65-72.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. arXiv preprint arXiv:
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot
Leigh L. Thompson, Jiunwen Wang, and Brian C Gunia. 2010. Negotiation. Annual review of psychology. 61:491-515.

---

Ben Wang and Aran Komatsuzaki. 2021. Gpt-j-6b: A billion parameter autoregressive language model.
A.1.1 Specifications of Computational Resource
Xiaoyang Wang. Chen Li, Jianqiao Zhao, and Dong Yu. 2021. Naturalconv: chinese dialogue dataset towards multi-tury turn topic-driven conversation. In Pro- ceedings of the AAAI Conference Artificial Intelli- gence, volume 35, pages 14006-14014.
To train the MLE-loss-based conversational model. and proposed INA, following configurations are used:
-PU:: A100-PCIE-40GB.
Qingyang Wu, Yichi Zhang. Yu Li, and Zhou Yu. 2021. Alternating recurrent dialog model with large-scale pre-trained language models. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume. pages 1292-1301.
CUDA Support: CUDA 11.x (or later.
Memory clock: 1215 MHz.
Total board power: 250 W.
Runzhe Yang, Jingxiao Chen, and Karthik Narasimhan. 2020. Improving dialog systems for negotia- tion with personality modeling. arXiv preprint arXiv:2010.0995..
GPU clocks: Base: 765 MHz, Boost: 1410 MHz.
Memory Size: 40 GB.
Haolan Zhan, Yufei Wang, Tao Feng, Yuncheng Hua, Suraj Sharma, Zhuang Li, Lizhen Qu, and Gho- lamreza Haffari. 2022. Let's negotiate! survey of negotiation dialogue systems. arXiv preprint
Memory Type: HBM2.
Bus Width: 5120 bits.
arXiv:2212.09072.
Tianyi Zhang, Varsha Kishore. Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint
B
Dataset
We ensure that the utterances in INA are grounded on the background knowledge consisting of product and deal details. Table shows example utterances from our dataset for different agent intents. It can clearly be observed that the utterances are well- grounded in the background knowledge and do contain factual hallucinations.
arXiv:1904.090 09675.
Ran Zhao, Oscar J Romero, and Alex Rudnicky. 2018. Sogo: a social intelligent negotiation dialogue system. In Proceedings of the 18th International Conference intelligent virtual agents, pages 239-246.
A
Appendix
Table 6 shows example utterances for each in- tent defined in Section 3. The table also shows examples of custom intents obtained through the combination the defined intents.
A.1 Implementation Details
For generating the INA corpus, GPT-J model (Wang and Komatsuzaki, 2021) with 6 billion parameters was used. INA is trained in an RL framework by employing fine-tuned GPT-2 small (Radford et al., 2019) model (117 million param- eters) on our proposed IND dataset. For dia- logue flow generation, we keep the value of d as 2. In each iteration of RL-training, didate responses are generated. It is selected as per PPL score, after experimenting with differ- ent values i.e. n = 2,3, 4, 10. Nucleus sam- pling (Holtzman et al., 2019) with temperature T = 0.8 and probability = 0.9 is used to de- code the generated utterances. INA trained is done using seed_ value = 10, human_reward = 10, max max_candae cata___adddaeee_. hengghh 50 with optimizer AdamW (Loshchilov and Hutter, 2017) and leamn ing rate 2e 05, 0.2 and epochs 17.
B.1 Prompts
Each prompt contains around 4 shots and each shot contains task description, required context, and an example utterance.
3 can-
An example shot for the intent "Negotiate- Priic--tt--riee- NoChange" is follows:
seller negotiating with a customer for a laptop called Dell X8, has GB ram, 11-inch screen and Intel i7 processor. The seller idcally wants it for S770 and not willing to reduce the price."
"TThe customer tells "I appreciate that you need to make profit this item, but unfortunately, $770 is above my budget for a laptop. I was ideally hoping to purchase the Dell X8 for $500, but I'm willing negotiate up $570 if necessary. there any way you could lower the price to meet me somewhere the middle?"
The reward weight combination of 0.2, 0.2, 0.3, 0.2 are chosen as the final weights for 71, 72, 73. and 74 respectively.

---

Keustomer
kseller
utilityeustomee
utilityagend
0.2
0.8
0.73
0.45
0.4
0.6
0.64
0.53
0.6
0.4
0.51
0.65
0.8
0.2
0.33
0.82
Table 4: Table showing price simulation result for different value of k for counter offer generation for seller Keetter and buyer Knuyer. the most practical and fair outcome where average utility (final price above reserve preference) not one sided in more than 50% simulations assumption we had taken was customer will be more stricter to his/her budget. Therefore customer will increase his/her budget in counter offer in lower rate than seller will decrease the price of bundle in counter offer. Final value of k was 0.4 for customer and 0.6 for seller.
getting was value 0.4 for seller and 0.6 for buyer. One
"RRemember, the seller cannot go lower than $770) The seller endorses the product by saying." "<start> While appreciate your offer, I cannot as low as $570. I cannot lower the price further since the laptop is high-end and is well worth $770. It has 16 GB ram and an Intel i7 processor, making it ideal for heavy duty applications. I'm sure that you would be pleased with it even for S770!"
Here, we provide context in terms of the pre- vious utterance the customer and the product description. The task mentioned to endorse the product and not go below particular price.
An example shot for the intent "Negotiate- Remove-X" X" is as follows:
"A customer is negotiating with seller about product. The eustomer wants to ask for another deal to the seller."
"The initial deal was laptop called Dell X8, has 16 GB ram, 11-inch screen and Intel i7 proces- sor along with gaming mouse. The price for this deal was $800. The customer wants to remove the gaming mouse from the deal."
"The customer asks for the new deal by saying."
"<start> do not really need the mouse, is it possible to just sell the laptop?"
Here the task description is mentioned explicitly in lines. The context is provided as the previous deal.
C INA Generated Conversation Sample
In Table 7 we show one sample interaction between human and INA. The negotiation for bundle deal of TV, TV Stand, and Extension Cord. The interaction shows the capability our dialogue agent (INA) in handling and integrative negotiation.

---

1. An Oven called Frigidaire FGFT3066UF. This double-wall oven has total capacity of 10.2 ft. and features true convection cooking system with a third heating element and fan. also has built-in temperature probe, self-cleaning system, and cleaning option. (Main product)
A Microwave Safe Coffee Maker Mug called Corelle Coordinates Microwave Safe Coffee Mug. This 1 11-unne coffee mug made of durable, lightweight stoneware and mierowave safe. features simple, classic design and is dishwasher safe for easy eleaning (Side product
Background DB
Microwave oven cleaner called Weiman Microwave & Cooktop Cleaner. mullii-prrpoeeeeee cleaner works on the inside outside of your microwave. formula designed dissolve tough stains, grease, grime, leaving your microwave looking new. (Side product 2)
understand your budget constraints, but I'm sorry cannot reduce the price of the oven. It has total capacity of 10.2 cu. ft. and features true convection cooking system with a third heating element and fan. also has a built-in temperature probe, a self-clean system, and steam cleaning option. along with a microwave oven cleaner called Weiman Microwave Cooktop Cleaner. It is designed to dissolve tough stains, grease, and grime, leaving your microwave looking like new. [Intent: Negotiate-Price Pricc-NoChangee]
Utterance
The microwave oven cleaner is multi- purpose cleaner that works on the inside and outside of your microwave. It works to dissolve tough stains, grease, and grime, leaving your microwave looking like new, (Reply to: What is the Microwave oven cleaner?) [Intent: Provide Clarification/Inorru
Utterance
A AC called Voltas 1.5 Ton Star Split AC. Comes with 3-star energy rating, cooling capacity of $100 W. and anti-dust filter. (Main product)
A Smart thermostat called Honeywell Home T9. The Honeywell Home T9 smart thermostat features multi-room sensors that measure temperature and humidity in different of your home, helping keep your home the right temperature in every controlled using smartphone tablet and features huilt-in Amazon Alexa voice control. (Side product)
Background DB
While I appreciate your offer. cannot reduce price of AC since
3-star energy rating
Utterance
and cooling capacity 5100 W. It also built-in anti-dust filter. I'm sure that will worth the
investment even for $23900.0 0 [Intett: Negotiate- Peeotiate-PiceNNoCCCC Pricc-NoChangel
Yes, the cooling capacity is $100 W. (Reply to: Is the Cooling capacity of the Voltas 1.5 Ton 3 Star
Utterance
Split AC 5100 W?) [Intent: Provide Clarification/nnfrr))
Table Examples showing the intents expressed in generated utterances in the dataset while being grounded in the background database

---

Intent Greet-Ask
Example
Hello, I interested in buying the refrigerator you have listed. How much
get for?
Greet-Inform
Hi! selling the Speed Queen TR3000WN washing machine for $11000.0
Negotiate- Remove-X Negotatte-Pric-Deerreas
really like the phone but don't need the charging case, earphones, or charging
case. Can you sell the phone for $42200.0
Negoliate-Price-emoove
Certainly! can remove the refrigerator and covers from deal and sell price is $19300.0
refrigerator to you, however,
Negotiate-Removv-eeivvee
Excuse me, do not want the delivery. Can have the TV and stand without delivery? What would be the new price?
Negotiate-Prrie-NoCCC
understand your budget constrants. However, I I'm sorry cannot reduce the price of the Nikon Z7 IL. It is a top-natch camera, and is one of the best available in the market. It has full-frame mirrorless sensor; which gives it a 493 facus points and a 45.7 megapixel resolution. It also has weather-resistant flap and a comfortable shoulder strap. also has 4K vidco recording feature, which makes it ideal for shooting video. I'm sure that it will be worth the investment even for $51300.0
Negotiate-Pricet Increase Increee
understand that you' re looking for refrigerator with a lower price. However. I' afraid that the price of this refrigenator is already very low. I'm willing to reduce the price to $16700.0 Are you interested?
Ask Clarification-y
What's the relresh rate this TV?
Provide Clarification-
The refresh rate of this TV 60Hz.
Negotiate-Add-X
What will he the total price if I buy the laptop together with the keyboard and the USB Hub?
Tell price
For the laptop, external hard drive, and mouse it would be $34900.0
Segotiate-rrice-eeceee
I'm interested in the Samsung QN90A Neo QLED TV. Unfortunately, I'm not willing to go above $78600.0 Is there any way you could meet me in the middle and sell it to at $68500.0
Reject
Sorry, my budget is limited to $29800.0 I'm sure will find better deals, thank you for your interest.
Ask Price
What would be the price for the thermostat?
Segotiate-Remove.
not really need stylus, it possible to just sell me tablet?
Accept
I'm willing to purchase the laptop at that price. Let's 's go ahead with the transuc- tion.
Acknowledge acceptance
Great! am pleased that you are buying and I "m sure you will be happy with your purchase.
Greet-Ask
Hello, I'm interested buying your Samsung Galaxy S21 Ultra 5G, but was hoping to get better price. Can get for $62900.0 Ok, in that case we can proceed with $39500.0 Shall we proceed to checkout?
Srreotiate-rric--eeeeee
Avoid_rejectinn
Greet-Inform
Hello! I'm glad you't interested in the washing machine and dryer. While understand your budget, I'm sorry cannot lower the price. This is an amazing machine, which is why can only offer $23200.0 While understand that above your budget, I'm sure my customers will love it.
Negotiate-Prie-NoChang
Greet-Inform
Hello there! Thank you for your interest in the A Carrier 1.5 Ton Star Split AC. I'm sorry, but not able go down $53400.0 but am willing lower the price to $57700.0 If you are interested, please let know.
Negootate-Prric-Ineree
Table 6: Example utterances for each intent type

---

Product Description: A TV called Skyworth Q71 Series TV. This 4K TV features a 6012 refresh rate. 4K resolution, 65-inch screen, and Dolby Vision and Atmos for enhanced audio and visual experiences., (A TV Stand called WALI Universal TV Stand.
This stand fits most TVs from 22 to 65 inches and can support up to 110 pounds. It has tempered glass base. adjustable height, and cable management system., An Extension Cord called AmazonBasies Indoor Extension Cord. This is 6-foot extension cord that can handle for: TV, (TV Stand, Extension Cord)
13 amps buyer asking
Bundle I: TV (TV Stand, Extension Cord)
Initial Buyer Proposed Price: 55000
Initial Seller Proposed Price: 67000
Minimum Seller Price: 57000
Context
Bundle 2: TV (TV Stand)
Initial Buyer Proposed Price: 52500 Initial Seller Proposed Price: 65000
Minimum Seller Price: 55000
Bundle 3:
(Extension Cord)
Initial Buyer Proposed Price: 53000
Initial Seller Proposed Price: 65500
Minimum Seller Price: 55500
Bundle 4:
Initial Buyer Proposed Price: 51500
Initial Seller Proposed Price: 6450000
Minimum Seller Price: 53500
Hello, I'm interested in purchasing the Skyworth Q71 Scries TVV, Bose Soundhar 700,
Buyer
the GE Designer Extension Cord
Hello! I' m glad to hear that you re interested in the Skyworth Q71 Series TV,
Agent
Bose Soundbar 700. the GE Designer Extension Cord
Buyer
I'd like to know about the TV. What is the resolution of the model? How much does it cost?
The TV has 4K resulution. The TV includes remote control and remote shutter release,
Agent
will you $66600
am interested in buying the set of TV, but not want the stand. What will be cost?
Buyer
understand that $66600 is above your budget. However, I'm willing to
Agent
meet you in the middle at $64,600 after removing TV stand
Buyer
Also do not need the extension cord, can we remove extension cord from the dcal.Also
Agent
Yes, we can remove extension cord, it will cost you $62000
Buyer
have budget of $59,500, Can you give more concession?
Certainly!
offer you the TVV from the deal for $61,000. however, $59,000 is less
Agent
cannot afford more than $59500, You should understand am interested this deal, could you finalize the deal $60000
Buyer
Agent
Ok, we can have deal, Thank you
Table 7: Sample conversation between human and INA


Reviews:
[{"review_id": "tIkVZRTB6b", "reviewer": "Reviewer_ndma", "summary": "", "questions": "Any reason for choosing GPT-J over GODEL or Koala?\n\nWhat is the accuracy of the BERT intent classifier?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1) Considering limited compute available in the research setting we could not use LLMs larger than 7 billion parameters. For this reason we could not use the Koala-13b model. The Koala-7b model (which we could use) is benchmarked lower than the GPT-6b model on the open-llm leaderboard. GODEL models on the other hand has a maximum size of 2.7 billion parameters and although trained for dialogue setting, our in context prompts consist of dialogue summaries instead of raw dialogues. For this reason GODEL was not found to be suitable for data generation purposes for the given task. \n2) The test accuracy of BERT intent classifier is 71.2%, training:validation:test 2000:500:1500 instances. We will make sure to mention this in the camera ready version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Authors present a novel new dataset of reasonable size, containing discussion on integrative negotiation dialogues.  The dialogue contain clarification intents, which is frankly useful in itself as a totally different task worth studying.  3,300 training dialogues is a respectable amount.\n\nFor quantitative metrics (METEOR, BERT-Score, WMD, Perplexity, and Length), the proposed INA model outperforms all baselines.  In human evaluation across five qualitative metrics, the INA model also does better than all ARDM and NegTOD baselines.", "reasons_to_reject": "The quality of the dialogues is still fairly unrealistic.  To highlight a couple of areas:\na) The users don't really dive deep into the description or product details.  (ie. \"The SnapDragon processor is 2 generations old now, you should lower the price.\")\nb) There are no emotional appeals that would occur (ie. \"Oh, please!  I really need it for my daughter's birthday gift!\")\nc) Conversations are all roughly the same length, real life chats can go on for 100+ turns easily (or conversely, will end abruptly).\nd) The prices are off by a factor of 100x?  (ie. Tablet costs $92,800, rather than $928)\n\nThere are no ablations or extra analysis on the rewards used for RL-training.  In general, the modeling is not too novel, relying largely on pre-trained SL followed by RL, which is fairly common for dialogue system training.  This is not a huge deal though if we view this as resource paper, rather than a modeling paper.", "questions_to_authors": "Any reason for choosing GPT-J over GODEL or Koala?\n\nWhat is the accuracy of the BERT intent classifier?", "paper_topic_and_main_contributions": "This paper introduces a new task of integrative negotiation along with an accompanying new dataset.  Most prior works establish agents that attempt to bargain for the best price possible during a discussion, resulting in a win-loss dynamic.  In contrast, this work designs an agent that attempts to complete a negotiation in a win-win manner, by bargaining on items on unequal value, such as partial items in a bundle.  \n\nThe dataset is created by (a) coming up with 10 product categories, 11 turn-level intents, various pieces of meta-data such as item price (b) randomly select these intents to create a dialogue flow skeleton (c) use GPT-J to generate dialogues using 4-shot ICL following the dialogue flow (d) Authors finally review the generated data to ensure quality.   This results in ~4K total dialogues, with an average of 13 turns per dialogue. \n\nFor modeling, the authors first fine-tune a GPT-2 Medium on the ground truth dialogues.  This model is then further updated with PPO in an RL setting where the reward contains four parts.  (1) Intent consistency - train a BERT model to ensure that the intent for the dialogue flow is the one that is generated (2) Price gap - higher reward for higher price obtained.  (3) Negotiation strategy - nudges the model to only accept a deal when within a certain threshold (4) Interactiveness - a penalty (negative reward) for producing utterances that are too similar.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 441: four rewards?"}, "extra_text": {}}, {"review_id": "XTiwyxsqhb", "reviewer": "Reviewer_gPQD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1) We thank the reviewer for pointing out the figures, we will make sure to fix this issue in the camera-ready version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work provides substantial contents including:\n1. A concrete pathway to collect data\n2. A detailed training framework containing SFT and RL, as well as the delicate reward design.\n3. Relatively positive experimental results both on automatic and human evaluation.", "reasons_to_reject": "Figures in this paper should be polished up. For example, several lines in Figure 2 are actually crooked.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Towards negotiation in the online marketplace, this paper creates a new semi-automated data creation method and release a new dataset called Integrative Negotiation Dataset (IND). It further builds an integrative negotiation-powered dialogue agent (INA) with a process similar to RLHF.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "povrqjyeVo", "reviewer": "Reviewer_PVmB", "summary": "", "questions": "1) How was k decided? What were the final values?\n\n2) Eq 7 seems to define the reward at the end of the dialogue. How did the authors tackle the issue of back-propagation through such long token sequences? It is unclear how the RL rewards are defined currently. I suggest providing more details about the reward design for clarity.\n\n3) The data collection makes several assumptions that hurt the diversity of the collected data, such as assuming concession rates for buyers and sellers. Further, semi-automatic methods make it challenging to capture other variables, such as user personality. I wonder if the authors have any thoughts on this and how it can be improved in the future - it would be nice to discuss this in the paper and can improve the overall contributions made in this work.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1)  In the proposed approach the rewards were defined at the dialogue level. We will endure to elaborate on the approach in a more thorough manner in the camera ready version.\n2) In our proposed approach, negotiation is on the bundle, where a customer can add an item with some concession price or he/she can remove any item and go for the deal with primary item of the bundle. On adding any accessory to the bundle it is actually offering a reasonably negotiated price to the customer and also it is giving some better margin to seller (above the reserve preference of cost of bundle), that way our proposed approach is doing negotiation where one issue is definitely price but the other issue is addition and removal of an accessory from the bundle and negotiation on bundle with added accessory or removed accessory.\n3) The test accuracy of BERT intent classifier is 71.2%, training:validation:test 2000:500:1500 instances, however we have not mentioned classification accuracy of proposed RL model but as per the human evaluation results, we can see on absence of intent consistency reward, model has shown below average bargaining efficacy and dialogue engagingness, which shown intent consistency contribute to the conversation or dialogue in makings it interesting, captivating, and able to hold the attention of the participants and also it helps in presenting compelling argument in response to the customer counter offer or issue with price. The proposed model is aiming for the best possible outcome for the customer and seller, therefore accessories addition and removal accessory decision is one of the factor which help in finalizing the deal, the decision is based on a counter offer from the customer. We will ensure to better explain this point in the camera ready version of the paper.\n4) We have taken different value of k for counter offer generation for seller and buyer, We performed price simulation for different value of k for both seller and customer , the most practical and fair outcome where average utility (final price above reserve preference) was not one sided in more than 50% simulations we were getting was on value 0.4 for seller and 0.6 for buyer. One assumption we had taken was customer will be more stricter to his/her budget. Therefore customer will increase his/her budget in counter offer in lower rate than seller will decrease the price of bundle in counter offer. Final value of k was 0.4 for customer and 0.6 for seller. We will explain this process in detail in the camera ready version\n5) Because of O(e^n) possibility of conversation during negotiation where a customer can add or remove accessory any number of items, we have taken one assumption of maximum 2 accessories in bundle however user have choice to add or remove 1 or 2 accessories. Final value of k is not purely assumption, we should have mention our basis of k value assumption for buyer and seller, since we have taken different value of k for counter offer generation for seller and buyer, we performed price simulation for different value of k for both seller and customer , the most practical and fair outcome where average utility (final price above reserve preference) was not one sided in more than 50% simulations we were getting was on value 0.4 for seller and 0.6 for customer. Our assumption here was customer is more strict to his budget then seller. However, concession can be given based on entailment of persona and the accessory if we have some prior information about the customer persona or we can capture the persona during conversation that is his/her likes and dislikes, based on that we can decide how much entailed offer we have for customer and eventually this will help in deciding the final concession on the counter offer from seller. In future work Artificial theory of Minds (AToM) will be also very helpful in deciding the counter offer for the customer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The semi-automatic data collection technique is an interesting step and potentially useful in this space where collecting quality data is challenging. This naturally has pros and cons (I discuss some of them below). The process followed was also rigorous, involving post-processing and human evaluation to ensure quality.\n\n2) In their approach, the authors defined useful rewards that are all relevant to negotiation and can potentially help future work.\n\n3) The paper consists of a comprehensive evaluation, showing the superiority of the proposed approach on a variety of metrics based on both automatic and human evaluation.", "reasons_to_reject": "1) The proposed approach is unclear. RL formulation has not been stated properly. It is unclear what the state and actions are and whether the rewards are defined at an utterance level or a dialogue level. I suggest the authors to describe the approach in a more thorough manner.\n\n2) The paper argues for win-win negotiations that “see” a product as a bundle rather than negotiating for a single issue of price. However, neither the proposed approach nor the evaluation focuses on the other items in the bundle. The rewards are based only on the price. In other words, the paper set out with the motivation of handling a scenario with multiple items in a bundle, but the spirit of the approach seems no different than prior work by He et al 2018. What am I missing here?\n\n3) In addition, the pitch of the paper ignores a body of prior work on multi-issue negotiations that do capture multiple items rather than just one. These datasets already involve integrative negotiations (along with distributive negotiations). For instance, work on DealOrNoDeal task (Lewis et al. 2017) and more recent datasets like CaSiNo (Chawla et al. 2021) and JobInterview (Yamaguchi et al. 2021). These should be discussed in related work. One difference is that in this paper, the set of issues varies depending on the given context. Thus, the claims in Lines 126-128 and 189-191 are misleading.\n\nTo fix these, I believe either the pitch of the paper should be changed or the experiments should be better designed in a way that captures the multiple items in a bundle (just like modeling work on multi-issue negotiation tasks). One way to do this can be by defining a reward that captures the other items in the bundle.\n\n4) The paper lacks additional critical analysis, such as comparing the intent accuracy of the final models and analyzing how the model deals with the other items in the bundle - this is important since this relates to how the paper has been motivated. Hence, I suggest the authors to include this in the paper.", "questions_to_authors": "1) How was k decided? What were the final values?\n\n2) Eq 7 seems to define the reward at the end of the dialogue. How did the authors tackle the issue of back-propagation through such long token sequences? It is unclear how the RL rewards are defined currently. I suggest providing more details about the reward design for clarity.\n\n3) The data collection makes several assumptions that hurt the diversity of the collected data, such as assuming concession rates for buyers and sellers. Further, semi-automatic methods make it challenging to capture other variables, such as user personality. I wonder if the authors have any thoughts on this and how it can be improved in the future - it would be nice to discuss this in the paper and can improve the overall contributions made in this work.", "paper_topic_and_main_contributions": "The authors propose a new dataset and dialogue agents for agent-human negotiations. The authors claim that their work looks at each product as a bundle of multiple items, which can all be negotiated together, unlike prior work that focuses on win-lose conversations over the price of products. The dataset is collected in a semi-automated way using randomly generated (with some sensible constraints) dialogue flows based on intents and subsequent utterance generation by prompting a language model. Their approach is based on a supervised + reinforcement learning paradigm using a combination of multiple relevant rewards. A comprehensive set of experiments is provided based on automated and human evaluation.\n\nOverall, I believe this is an exciting direction to pursue, and the proposed dataset and approach are reasonable. However, the pitch of the paper is misleading, which questions the overall novelty and usefulness of the work.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) 755: Section reference error\n\n2) 313: remove “it”\n\n3) The citation for the PPO method is missing.\n\n4) Figures and tables: Improve the captions - currently, they do not provide any valuable info about the metrics shown, acronyms used, etc.\n\n5) Baselines and ablations have not been described properly.\n\n6) SLDM and ARDM are never defined. ARDM paper citation is missing (Wu et al 2021)."}}]

Author rebuttal:

Meta-review: This paper contributes a valuable new dataset of negotiation dialogues, a proposed new model for a negotiation agent, and automatic and human evaluation experiments that demonstrate superior performance over several baselines.  The reviewers especially appreciate the value of the newly created dataset and methodology, and is seen as the paper's main asset.  They also acknowledge the comprehensive evaluations and positive results in terms of both objective and subjective metrics.  On the other hand, the reviewers are less excited about the proposed approach, considering it to have only limited novelty.  The paper could be improved by pitching it more as a resource paper.
Decision: Accept-Findings