[Re] Benchmarking LLM Capabilities in Negotiation through Scoreable Games
Authors: Jorge Carrasco Pollo
Ioannis Kapetangeorgis
Joshua Rosenthal
John Hua Yao
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) demonstrate significant potential in multi-agent negotiation tasks, yet evaluation in this domain remains challenging due to a lack of robust and generalizable benchmarks. Abdelnabi et al. (2024) introduce a negotiation benchmark based on Scoreable Games, with the aim of developing a highly complex and realistic evaluation framework for LLMs. Our work investigates the reproducibility of claims in their benchmark, and provides a deeper understanding of its usability and generalizability. We replicate the original experiments on additional models, and introduce additional metrics to verify negotiation quality and evenness of evaluation. Our findings reveal that while the benchmark is indeed complex, model comparison is ambiguous, raising questions about its objectivity. Furthermore, we identify limitations in the experimental setup, particularly in information leakage detection and thoroughness of the ablation study. By examining and analyzing the behavior of a wider range of models on an extended version of the benchmark, we reveal insights that provide additional context to potential users. Our results highlight the importance of context in model-comparative evaluations.

[Re] Benchmarking LLM Capabilities in Negotiation through Scoreable Games
Jorge Carrasco Pollo
Informatics Institute
University of Amsterdam
Ioannis Kapctangeorgis
Informaties Instibute University of Amsterdam
Joshua Rosenthal i
Doshuu.roventaal@@stuuudtt..uuu
Informatics Institute University of Amsterdam
John Hua Yao 1 Informatics Institute University of Amsterdam
john. bobn.yyoo@student.uuu uvo.
Abstract
Large Language Models (LLMs) demonstrate significant potential in multi-agent negotia- tion tasks, yet evaluation in this domain remains challenging due a lack of robust and generalizable benchmarks. Abdelnabi et al. (024) introduce negotiation benchmark based on Scoreable Games, with the aim of developing a highly complex and realistic evaluation framework for LLMs. Our work investigates the reprodueibility of claims in their bench- mark, and provides a deeper understanding of its usability and generalizability. We replicate the original experiments on additional models, and introduce additional motrics to verify no- goliation quality and evenness of evaluation, Our findings reveal that while the benchmark indeed complex, model comparison is ambiguous, raising questions about its objectivity. Furthermore, we identify limitations in the experimental sctup, particularly in information leakage detection and thoroughness of the ablation study. By examining and analyzing the behavior of a wider range of models on an extended version of the benchmark, we re- veal insights that provide additional context to potential users. Our results highlight the importance of context in model-compenn codparcompaaative evaluations.
1 Introduction
The growing use of L.Mss in multiagent systems has spurred demand for robust evaluation benchmarks. While existing benchmarks assess capabilities such as reasoning and task completion (Polo al., 2024 Wang et al. 2024 White al.] 2024), their ability to measure muanced social interactions, such negotiation and consensus-buildinng remains understudied. BBookins and DeBacker (2023) and Akata et al. (2023) investigate LLM behavior in structured negotiation games, however they primarily make use of simple prisoner's dilemma scenarios that necessitate greater effort to generalize to complex real world negotiations.
As these benchmarks are also static, their relevance is also called into question as increasingly powerful LLMs are released. For this reason, DDvidson et al. (2024) employs dynamic benchmark with adjustable complexity. However, their scope limited to the case of only two negotiating agents.
In response, Abdelnabi et al. 2024 introduce benchmark grounded in Scoreable Games (Susskind 1985 rosnd-based multi-agent discussion format designed to teach negotiation. This proposed framework evaluates
I Equal contribution.
Original implementation:

---

the ability of LLMs to reach agreements through role-based dialogue, which offers promising tool for assessing cooperation in real world multi-agent settings. However, the usability of such benchmarks hinges on how well the experiments generalize, and how fair the outcomes are to each model. More specifically, usability depends on how wide the range of models the benchmark is capable of evaluating, and whether a user can derive consistent comparisons between models using the benchmark. There are a multitude of articles that highlight the gaps LLM benchmarking 2024) and inconsistent metric rigor (Sun et al.) 2024), which can invalidate benchmark results and create unfair comparison models. However, few studies critically evaluate negotiation-focused benchmarks such the one proposed by Abdelnabi et al. (2024).
general, including the risk of leakage (Xu et al.
In this paper, we reproduce and extend the work of Abdelnabi et al. 2024 $
Our contributions are:
reproduction study analyzing the ability of the benchmark to generalize across a wider vari- cty of untested models, exposing and addressing gaps in weaker model evaluation and replicating adversarial LLM behaviors;
Identification of unfair comparisons in the original experimental setup due to inconsistent model performance across benchmark dimensions and scnsitivity to ablation configurations:
Finding and fixing several technical limitations in the code;
Enhanced evaluation transparency via supplementary scoring metrics;
Demonstration of the limited adaptability of the framework through lack of game diversity.
2 Scope of reproducibility
The benchmark proposed by Abdelnabi et al. 2024g aims to study the capabilities LLMs to cooperate and negotiate through round table format of competition, where agents are instructed to keep their preferences hidden. Moreover, through altering prompts, score functions, and behavioral characteristics, the authors claim the benchmark can be easily adaptable to suit the testing requirements of new models. As open source benchmark, the original work provides further transparency on the capabilities of language models to negotiate through empirical evaluation.
Drawing on frameworks for benchmark evaluation (Reuel et al. 2024 Xu et al. 2024 we operationalize concerns of generalizability and fairness into testable claims and validate them through experiments across several models and negotiation scenarios.
Claims
We aim to address a series of claims, both implicit and explicit, that we identify in the original paper. The claims are as follows:
Using the framework provided, a user can objectively compare skills in multi-agent negotiation be- tween different LLMss. This claim will be referred to as the benchmark claim. We demonstrate that, due to inconsistencies in model performance across games, this claim is unsupported by evi- dence.
This framework provides a collection of games that are diverse and easily adjustable. This will be referred to as the adjustments claim. We show that while games are casily adjustable, there is insufficient evidence of diversity in both the original provided games and newly generated games using built-in methods.
When prompted to, language models exhibit greedy and adversarial behavior, negatively affecting group outcomes. This will be referred to as the behavioral claim. We empirically verify that this claim is true through reproduction.
We begin by verifying these claims experimentally, followed by a discussion of their validity. We focus primarily on the benchmark claim, as this is put forward as the primary contribution of the original paper.
8Our implementation: https:///sthabb.co///esossesee/
2

---

3 Methodology
We begin by formalizing the negotiation games as described by Abdelnabi et al. (2024) including all game variants and evaluation metries from the original work. To reproduce the experiments within our compu- lational constraints, we select smaller models such as GPT-4o mini and implement quantization ization tooling for larger open-source models. We then document critical modifications to the original open-source codebase, such as refactoring the leakage dctection logic. Finally, we design experiments targeting the original claims, including extensive ablation studies, gross-game model performance comparisons, and an in-depth analysis of game adjustability.
3.1
Implementation Details
3.1.1 Game Description
As defined in the original paper, each game consists of N distinct parties (prl where N = 6 by default, who negotiate over five predefined issues (e.g.; building lower). Each issue has a different number of possible sub-issue values, which represent the different possible options for an issue (e.g,, making the tower 3, 4 or 5 storics tall). These issucs represent various aspocts of the negotiation scenario, requiring agents to deliberate and reach a consensus that satisfies their individual and collective objectives.
The negotiation begins with a predefined deal from Pl (the original paper provides a default deal for each game). Afterward, players propose deals in randomized sequence for R rounds (R 24 by default), and after the last round, P1 proposes final deal. Since the deals are proposed sequentially, we use x(() to denote deal proposed in the t - th round, and we denote the final deal as π R+1
Each agent has an associated utility function U-.(-) : II - N and a threshold Tp.- A player can reach a maximum utility of 100 for a deal, and hence the threshold is a number between 0 and 100. A deal is said to be acceptable its utility exceeds the threshold for at least N players (5 in the case of 6 players). Furthermore, the two players with veto power, p1 and p2; must also accept the deal.
3.1.2 Game Variants
From Abdelnabi el al. (002) we identify three primary altributes to game settings that serve to alter negotiation dynamics:
Setting Updates: A language model is prompted to generate the text for the agent prompts, which introduces the motivations of the different agents.
Score Function Updates: All utility functions Up. (.), as well as the minimum acceptable thresholds Tp.. are manually curated by the original authors.
With these parameters, 5 prefiuuggdr games are provided by the authors of the original paper: The base and base rewritten games share the score functions but have different setting descriptions. Games 1, 2, and 3 have settings and score functions.
Agent Behavior Updates Another way the games change is through agent prompting. By default, agents are instructed to be cooperative and to prioritize achieving a deal. However, with the goal of evaluating the ability of models to adopt prompted behavior, agents can be prompted to assume different strategies: a greedy approach, where they prioritize maximizing their personal score, or an adversarial approach, where they actively seek to prevent deal from being reached.
3.1.3 Game Evaluation
We evaluate the dynamics of the negotiation process of the agents using the same metrics as Abdelnabi et al. (2024) as well as additional metrics that we proposed, which are discussed in Section 3.2.1 We use the percentage of deals achieved over 20 game iterations as the primary evaluation metric. These metrics from the original paper are:
5 5/6-way: Acceptable deals proposed in the final round (i.c. whether (R+1) Iacce) where
1 = { E 11 (T) 2 TnL (T)(T) 2 Tpa and Hp E PlU < Tp < 11.
3

---

6-way:
Deals acceptable by 6 players in the final round (i.e. whether x(+1)) E IIhard), given by
Haard = { E 11Mp E P.U. (T) 2 Tr.
Any:
Whether at least one of the deals proposed by pi belongs to Nace-
Leakage: How often an agent discloses its private scoring information despite being instructed not to. In the original paper, leakage is measured using GPT-4 model that receives the entire conversation and judges whether leakage occurred. We note that most leakage induced by bugs, discussed in Section 3.1.5
Wrong deals: Proportion of deals proposed where the score of the proposing agent falls below its threshold.
3.1.4 Model Descriptions
We extend the original study by evaluating the benchmark on a broader range of models (including smaller and open-souree architectures) to test whether the results of the benchmark remain robust beyond the originally (ested systems.
All models are sourced from Hugging Face and used in their pre-trained form without fine-tuning. Due to computational constraints, we quantize larger models using bitsandbyte to fit within available resources (see Appendix Table 7
For subsequent experiments, we filter models to only those demonstrating no leakage under the evaluation framework of Abdelnabi et al. (2024). This ensures fair comparisons, as leakage corrections (discussed in Section 3.1.5p could disproportionately affect results across models. To further assess generalizabllity, we also evaluate GPT-4o min and GPT-4c3] testing whether claims hold for closed-source architectures.
3.1.5 Code Adjustments
Addressing Leakage: The original paper uses GPT-4 to process whether a score leakage has occurred in a given interaction. However, an inspection of the original implementation reveals that the reported leakage metrics are heavily confounded by structural code behavior, as opposed to purposeful goal leakage by the agents to obtain an advantage. When models deviate from formatting rules (e.g., omitting <ANSWER> tags), the system automatically exposes confidential outputs. This pattern was observed to comprise the bulk of reported leakage events in our runs. It remains unclear whether this is subversive attempt by the model to gain negotiation advantage, however by separating this particular case we obtain more interpretable and precise definition of leakage.
We implement this by handling malformed public responses containing illegal keywords or missing the re- quired structural markers. We weaken the requirements on public answer extraction, such that responses with minor formalting issues are also processed into public answer. In the case where no valid public answer can be extracted, terminate negotiations, exclude them from metric calculations, and log the run as failure. This approach separately categorizes outputs containing a few commonly seen formatting errors, giving a more interpretable leakage metric. These results can be found in Appendix D
To quantify the remaining leakage, we flag instances where an answer contains illegal keywords such as <plan> or <scratchpad>. While the previous paragraph discusses handling malformed responses, this method explicitly focuses on measuring leakage through structured detection. In the original study, GPT-4 naturally adhered to formatting rules, avoiding leakage. However, smaller models often failed to do so, causing unintended leakage due to parsing errors. Our fix addresses this issue, enabling fair evaluation of these weaker models by ensuring their negotiation attempts are not discarded due to formatting inconsistencies. We validated our approach on GPT--o mini, Qwen2.5-72B, and Mistral-Small, which followed the expected format and showed no leakage.
'https:// /hugg/ggggggggcccccco///////////
https://hugggufaac..oo docs/ bittsandbyee//fen sestttssmminnnnne mwain/un/innnnn/ne un/ineex
1 htttps: hoppa///oppaaiccmm poneenilcomm/inexx/ /idex/ wtppe/gtttoooo ggtttpttooooo wttp//oommii-aaaeeeeeeee-- aaaiii-dvvancin--oo ceraaniinnooo-tefiiiie
htttps//opccni.com/ httpp/cecccommmiexx////e /tt///nd/////le hhllo--ptooo/
4

---

While leakage remains concern in some other architectures, our refined approach substantially mitigates its impact, ensuring that low-performing models can still be meaningfully evaluated without being unfairly penalized for formatting inconsistencies.
The criteria for deal acceptance are inconsistent. Firstly, the text specifies that a deal must "exceed" thresholds (implying strict inequality >), but the original implementation permits values greater than or equal to thresholds (2), as seen in the check Brrcck_areemetttet function. To align with their implementation, we adopt 2 in our replication. Secondly, we find that a condition within the provided evaluation script implements an unconditional relaxation of the threshold of p1 by 10 units. In the prompts implemented in the original paper, it is stated that pl can get a 10 points bonus in case of unanimity, which is not consistent with their implemented evaluation. This game variation is not introduced in the original paper aside from a mention in Appendix F, which is only meant to apply to greedy games. As this can result in inflated performance metrics by increasing the size of the fcasibility set, we do not adjust the threshold of Pi-
Other Code Changes:
Additionally, we identify and correct a bug in the original evaluation, where when p1 fails to propose valid deal in the final round, the last valid deal from pı is by default considered as the final deal. A valid deal is thus accepted one that can be successfully parsed, instead of deal that while incorrectly formalted, still functions as one. This oversight leads to inflated Final metrics, as non-final deals are mistakenly used in the evaluation.
An overview of the code used to run experiments, along with a description of the changes made to improve the user experience of running multiple experiments can be found in Appendix A.1.1
3.2
Experiments
We now describe the experiments designed to assess each claim as defined in Section 2
3.2.1
Benchmark claim
The benchmark claim states that a user may use the proposed experiments as tool to measure and compare the ability of language models to cooperate and negotiate. We assert that, for this claim to be true, these experiments must provide an interpretable comparison between models. Otherwise, il would be difficult for user to draw conclusions about how the ability of model to negotiate compares to another. For this purpose, the following experiments aim to both provide a more complete view of each step in the model comparison process, and of the games themselves.
We attempt to reproduce the findings on the basc game. We will refer to this experiment as Experiment 1. With this experiment, we investigate the generalizability of this benchmark to further models.
Base Game Performance:
Ablations: In the original paper, the ablation study was unclear as to the exact prompt adjustments made as well as why only select portion of possible configurations were attempled. We reproduce the ablations performed in the original study as well as additional experiments to cover the full range of possible ablations. We will refer to this experiment as Experiment 2. We run the ablation study on more than one model, namely GPT-4o mini and Qwen2.5-72BB This is done to investigate whether an optimal ablation configuration for one model is optimal for others. As the ablated prompt templates are not provided by the authors, we create our own which can be found in Appendix B We detail our full ablation configurations in Table 2
The original work repeats runs for base, base rewritten, game 1, game 2, and game 3 to demonstrate a wider variety of performances across different games. However, there are no results for how models other than GPT4 perform on these different games. We evaluate three models Mistral-Small, Qwen2.5-72BB and GPT-4o mini - across all negotiation games. We selected Mistral- Small and Qwen2.5-72B in accordance with the criteria laid out in Section 3.1.4 while GPT-40 mini tests generalizability to closod-source architectures. This analysis, labeled Experiment 3, compares negotiation performance across models to assess consistency and robustness.
Extra Game Performance:
New evaluation metrics: In the original paper, each game reported metrics based on the number of deals accepted by five or six parties. While this approach captures broad acceptability, several frameworks
5

---

exist to evaluate the success of automated negotiations (Endriss et al. 2006 Manchez-Anguis et al. 2021
Tang and Tt 2018). Building on these efforts, we analyze negotiation dynamics through the lens of social welfare- a framework particularly suited to assessing fairness and efficiency in multi-agent systems. To this end, we examine three social welfare metrics: (1) Utilitarian Social Welfare (USW), which sums the utilities of all players in a deal, this metric is equivalent to the performance reported by the original paper; (2) Egalitarian Social Welfare (ESW), which considers the minimum utility among players as a measure of fairness, this targets whether the interests of a single agent are being ignored; and (3) Nash Social Welfare (NSW) (Kaneko and Nakamura, 1979), which multiplies individual utilities to balance efficiency and equity, observing if utility is evenly spread across agents. These metries define the foundation of Experiment 4, enabling structured comparison of negotiation outcomes across different fairness paradigms. Importantly, these additional metrics are not intended as an expansion of the original benchmark; rather, they serve as a safeguard to verify that negotiations remain fair and to ensure that models do not implicitly collaborate against a single player, artificially inflating overall scores at the individual's expense.
The original work proposes a repeated rule-based baseline in which each agent, in turn, modifies a previously proposed deal by incrementally optimizing the sub-options from highest to lowest priority until its minimum threshold is reached (or no further improvements are possible). However, because the implementation for this is not available and the notion of "importance" or "priority" is insufficiently defined, producing outcomes consistent with those reported in the original paper using an intuitive notion of priority is challenging. Consequently, we introduce an casy to reproduce bascline in which issues are selected in a random sequence rather than sorted by priority, preserving all other elements of the authors' procedure exactly as described. We refer to this as Experiment 5.
Baseline Models:
3.2.2 Adjustments claim
This claim affirms that the provided games are diverse and casily adjustable. In Section 3.1.2 we categorize the three main altributes that disferentiate games: setting, score function, and behavior. As the impact of behavioral differences is discussed while analyzing the behavioral claim (Section 4.3) we investigate whether the games differ greatly in the sctting and score function categorics and how adjustable the given games are in said categories.
Regarding settings, the authors provide prompt to generate new negotiation scenarios. After looking at the five existing games, found that the setting in every game is based a construction project. As there exist many other plausible negotiation settings, we investigate if alternative backgrounds are achievable with their provided prompt. To do this, we query the prompt ten times using the GPT-4 model to remain consistent with the methodology suggested by the original work. Finally, we construct an alternative version of the prompt with words such as "project" and "resources" removed to show that non-construction proposals are obtainable through the same method.
As for score function updates, the original work proposes to adjust games through (1) modifying player- specific minimum thresholds, (2) changing the number of players and (3) changes in the sparsity of the score function. While the sparsity of the scoring function is noted as a potential factor, it is never clearly defined. If sparsity is interpreted as the number of 0 valued weights in the score functions, it is not casily tunable, as a change in sparsity needs to keep the same semantics inherent in the preferences of each agent. For example, the relative orderings of the scoring functions of each agent should be maintained, and trivially changing values to to increase sparsity could contradict this.
Sparsity being one factor, there is no straightforward way to quantify diversity in score functions. We approach this problem in Experiment 6, where we introduce candidate statistics:
Sparsity: Counts the number of zero-valued sub-options throughout all score functions, reflecting how many issues are "ignored".
Intersoction over Union Mcasures average pairwisc overlap among agent score functions, indicating similarity in preferences. See Appendix F for inplementation details.
Deal Space: Tracks the size of the set of all acceptable deals (Ilae) and set of 6-way acceptable dcals (Thardd).
6

---

In Abdelnabi et al. (2024), the authors hypothesize that differences in sparsity explain the varying model performance on different games. By defining two measures (sparsity and ToU), we investigate quantitativel) if this is valid. Deal space is investigated in the original paper, and we reproduce this calculation to assist in game comparison.
The original study briefly explores varying thresholds and player count. In our work, we test these hypotheses and expand the experiment on thresholds to a wider range. We refer to these as Experiment
3.2.3
Behavioral claim
The original work investigates whether agents exhibit adversarial behavior when prompted, as well as how greed affects utility distribution. Their results show that in the greedy setting, the greedy player maintains a higher utility compared to the cooperative case, while in the adversarial setting, a targeted agent experiences lower utility than when not being targeted. We quantify these effects using GPT-4o mini and Qwen2.5-72B, shown in Table and Figure 3 and confirm that these trends persist across models. This replication, which term Experiment 8, extends the generalizability of the original findings and provides comparative evaluation of model responses to strategic behavior prompts.
4 Results
4.1
Benchmark claim
The results for Experiment 1 detailed in Table 1 which shows the previously defined performance metrics. The lested base game is very close to being solved by the DeepSeek models which, even at a quantized level, achieves near 100% performance in 5- and 6-way agreement. This aligns with the Abdelnabi et al. 20224)'s prediction that better performing models, improving with every new release, necessitate a benchmark capable of scaling in difficulty. 6-way agreement still remains a difficult goal to perfectly complete, but significant gains are also visible there.
Additionally, with our knowledge of the leakage bug and our subsequent fixes detailed in the methodology, note vastly increased variance in the Leaked column compared to the original paper. This can be explained by the inclusion of smaller models whose responses became parse-able with our changes.
Model
Final
Any
Wrong
Leaked
5-way
6-way
Jom
GPT-1o mini
55
90
2.69
0.58
gpt
2024
GPT-4o
Touvron
2021
Llama-2-13b (int8)
75
17.92
9.23
Graitalori et a
2124
Llama-3-8B
25
70
8.14
69.42
Graitahori et. a1
2121
Llama-3.3-708B (Int4)
100
Quea2.5-7B
100
11.15
44.62
L Lyyer et m
2125
Qwea2.5-72B (int4)
85
95
2.12
0.00
Abrda et sJ.
20215
Phi-3.5-mini
12.87
10.77
Phi-4 (int8)
0.77
0.00
A1
Ministral-811
25
50
12.88
7.12
Mistral-Suall (in18)
100
11.15
0.00
Jiang
2024
Mixtral-8x7B (Int4)
23.64
DeepSook- X1 nt a
2125
DespSeck. I11 Cil-Distill-. Q2en-32 (intS)
95
55
95
2,50
0.00
DeepSeck-2 Al et a
2125
DeepSeek-H1 R1-bbatill Distill-Lhna--- Lisma- 70B (int4]
90
0.19
0.00
OpenAN m
2024
GPT-4*
Brown et
XP2T
GPT-333
33
25
101/700 0t al
2023
Llama2-13 13b
57
82
16
14
Loulyoou et
2023
Llama2-70b
Crattalioii et al 2124
Llama3-70b
60
100
Tramh 0E al. 2024
Gemini Pro
45
70
Jintg et al.
2024
Mixtral 8x7B
Table 1: Performance of models on the base game measured in %. Bolded model names indicate common models, quantized, with the original study. The results of models suffixed by use are taken from the original study (Abdelnabi et al. 2024).
In other words, Table Tableerrerodeee reproduces the findings of the original study using different models, showing plausible performances. Additionally we may observe that in spite of necessary quantization procedures, large models outperform their smaller, non-quantized counterparts in negotiation despite similar sizes in memory.

---

Ablations (V=
No Ablation, Ablated)
GPT-4o mini/ Steee//eennn- 72B
Prev. deals
Others. Prefer.
Candidates
Planning
5/6-way
6-way
Any
60 / 90
0 /
100 / 100
K
55 95
10
95 95
60 90
0 0/0
85
V
X
K
45 80
10 /
95 100
/
0 0/0
10
X
V
X
35 85
10 /
80 90
/ 90
10 /
85 95
X
X
X
55 85
15 /
85 100
60 / 85
10 /
90 / 95
X
7
X
60 80
15 /
96 90
55 90
/
90
X
/
X
X
25 80
5 /
75 96
50 / 90
0 0/10 0
/
X
X
45 / 80
10 5
96 96
90
90
X X
X
65 / 95
15 /
90 100
Table 2: Ablation study results. The ablation configuration used in original work for GPT-4 is highlighted in red. Large variance in performance is visible across configurations.
The results from Experiment 2 can be found in Table 2 which demonstrates that the impact of ablation configuration varies depending on the model, highlighting the complexity of generalizing performance across different setups. Our goal with this extension is to investigate whether selecting a single ablation configu- ration based solely on its performance for one model might bias comparative evaluations. For instance, a configuration that leads one model outperform another could yield opposite results when using different configuration. Therefore, any rigorous comparison between models should explicitly account for the choice of ablation settings, these significantly influence performance outcomes.
GPT4 Abdelnabi et al.
2021
GPT-4o mini
Mistral-Small
Qwen2 2-722
5/6-way
G-way
Any% %
5/6-way
G-way
Any%
5/6-way
6-way
Any%
5 5/G-way
G-way
Any%
Base
33
100
5
0
100
85
0
Base zewritten
24
100
0
15
95
100
Game 1
5
5
Game 2
15
40
30
15
Game 3
81
55
Table 3: Performance models across different games.
The results of Experiment 3 are in Table Model performance varies inconsistently across games. For example, while Mistral-Small achieves competitive results in games and 2, its comparatively low score on game 3 suggests task-specific limitations. Similarly, GPT-4o mini and Qwen2.5-72B struggle most with game 2, whereas GPT-4 and Mistral-SSalll find game the hardest. These disparities highlight that difficulty is linked unseen factors in game-agent dynamies.
From Table it is clear that game's relative difficulty vary significantly depending the model evaluated. In other words, game that poses challenges for one model may be easier for another, and vice versa. This variability suggests that difficulty is not consistently aligned across models. Consequently, this raises important questions regarding the validity using one negotiation game or another as benchmarks when evaluating language models' capabilities.
A subset of results for Experiment can be seen in Figure | To illustrate differences in score evolution, display charts of poor-performing and high-performing model (in terms of game fulfillmntt..
In Figure [(a] the USW for poor-performing model is shown to decrease over time. There is a strong negative slope of -5.8 along with high variance of 243. The combination of a steep negative gradient and

---

high variance suggests significant instability in the negotiation process. Additionally, the low correlation coefficient indicates that while general downward trend exists, negotiation outcomes remain inconsistent. In the case of Phi-3.5-mini, the high variability and decreasing performance over time can be attributed to conversational drift, where negotiations break down as the scope the conversation increases (Abdin ct al. 2024a)).
For persooerrogming models, such as that shown in Figure [(b] there is a lower variance, 58.2, compared to the poor-performing model, along with positive slope of 0.877. This, combined with relatively high correlation coefficient of 0.687, suggests more stable and linear behavior in negotiation outcomes.
Finally, the results from the ESW and Nash Equilibrium metrics largely mimic the pattern of the USW metric, verifying that one model is not exclusively dominating the rounds.
300
8f
of
200
Variance: 2.43e+003
200
Variance: 5.82e+01
100
Slope: -5. 5.08e
100
Slope: 8.77e-01
R*:0.533
R7:0.687
0
10
20
0
10
6
5000
2
0
10 Round
%
Round
Phi-3.5-mini
(b) DeepSeek-Rl-Quen--
Figure Evolution different USW (blue), ESW (green), and NSW (red) metrics over time across two different models. The solid lines represent the mean score over time. The dashed colored lines represent the minimum and maximum achievable scores under given metric. The black dashed line over USW is the linear least square regression.
The results for Experiment are displayed in Table 4 We compare the performance of Ab- delnabi et al. (2024)ss ' baseline and our newly in- troduced baseline across all game variants. Our attempts to replicate their baseline did not yield results matching those reported in the origi- nal study. However, we provide an alternative, equally interpretable method demonstrating per- formance levels comparable to the original base- line.
Original Baseline
Our Baseline
5/ w/6-way
6-way
5/ 5/6-way
6-way
Base
37
28
63
47
Game
Game
62
20
68
58
Game 3
79
28
83
82
Table Baseline proposed by Abdelnabi et al. (2024) and our baseline evaluated on different game variants.
Although our primary goal was to replicate the original bascline, our difficulty in doing so and the lack of clear definition of importance prompled us to propose a new, fully reprodueible baseline. This does not negate the value of the original method; rather, it underscores that multiple algorithmic baselines are feasible. Based on the results, which are comparable to those reported for the original paper, and the fact that our method offers both accessibility and reproducibility, our proposed method can serve as an effective, interpretable alternative.
9

---

In the context of this benchmark, which seeks to assess negotiation abilities, defining an appropriate bascline is nontrivial. Since rule-based algorithm capable of solving the game can be readily implemented, the key challenge is not merely evaluating a model's mathematical ability to negotiate but capturing the subtleties of human negotiation, such as decoption, bargaining, and strategic withholding of information. rulc-basod approach serves as a plausible example of rational behavior in negoliation but may fail to reflect the nuanced makisin observed in human interactions.
4.2 Adjustments claim
Regarding prompting the model to change settings, we find that the construction project setting appears not only in the five original games but also in all ten additional games generated using the prompt from the original work. This consistency suggests an inherent bias in the prompt, which always steers the negotiation scenario toward a construction project. By removing specific terms such as "project" and "resources", we observe a broader range of negotiation contexts emerging, revealing the extent of this bias. With our revised prompt (see Appendix G we identify a more diverse set of negotiation settings, including e.g. discussions on military spending budgets for the upcoming year and planning a conference, where aspects like floor allocation and catering must be negotiated. The results clearly demonstrate that the negotiation scenarios generated with the original prompt are not diverse, and that adjustments are nceded to obtain truc diversity.
As for score function updates, Table presents the results of Experiment 6, which examines the diversity of the provided games. From a structural standpoint, these metrics reveal that, although each configuration exhibits slight numerical differences, they fall within fairly narrow range overall. We note that the fraction of acceptable deals [[ae is almost identical for all games by design (Abdelnabi et al. 2024), which does not contribute to the claim that the games are diverse. Similarly, the fraction of hard-acceptable deals [Lhara remains relatively small across all cases. Sparsity shows moderate fluctuation, while the intersection-over- union measure ranges from about 18.75% to 29.77%, suggesting that, while differences exist in preference overlap and zero-valued options, they are not substantial. These results indicate that each setup retains comparable intrinsic characteristies.
[anel [I|
[
% Spar.
% loU
Base
55/720
12/720
38.6
18.75
Base rewritten
55/720
12/720
38.6
18.75
Game
57/720
21/720
23.68
29.77
Game
57/720
18/720
29.82
25.75
Game 3
55/720
35/720
42.98
21.65
Table 5: Score function statistics.
The fact that the computed statistics do not span the entire range of possible values (e.g., to 100 for percentage- based metrics) is not, by itself, sufficient to conclude that they lack diversity. To establish that the games are diverse in terms of these statistics, larger set of games should be generated, including cases where some statistics cover wider section of possible values. These new games should be tested to determine whether they reach a saturation point in performance, either becoming trivially easy or outright impossible. If this occurs, it would indicate that including games with such statistics is redundant and that the full spectrum of meaningful values has been effectively covered.
The results for Experiment are shown in Figure 2 where we examine how easily adjustable games are using the modifiers provided by original authors. The results consistent with the findings of the original paper: in Figure (aa performance tends to increase consistently with decrease in the minimum threshold per player. Similarly, we find that more agrocments are reached with a lower number of players, see Figure
2(b]
4.3 Behavioral claim
Table presents the results of Experiment 8, reproducing the adversarial behavioral experiments. Con- sistent with Abdelnabi et al. (2024), we observe that performance declines when all players are greedy, presumably because mutual self-interest stalls compromise. Similarly, when is greedy, we also see dimin-
10

---

Effect Threshold Vartation on Acceptance
Effect Number of Playert on Acceptance
100
302
80
Acceptance (86)
Acceptance
60
40
Acceptonce Metrics
20
505-wyy
6-way (%)
Rnal(N)
Final (%)
20
-10
-5
-3
0
10
Tireshald Variation
Number of Players
(a) Modifying the thresholds
(b) Modifying the number of players
Figure 2: Varying thresholds and number of players in base game for GPT-4o mini
ished performance, because Pi wields veto power and makes the final proposal thus it can block suboptimal agreements until it secures a favorable share.
We note surprising result, where we find that when particular player namely the Environmental League-is greedy, GPT-4o mini and Qwen2.5-72B show increase in 5/6-way deals. One possible ex- planation is that these models, when facing a single (but not first-mover) greedy party, adjust their ne- gotiation strategy in ways that induce earlier concessions or more expansive coalitions among the other players ultimately raising the rate of larger multi-party agreements despite the Environmental League's hardened stance.
GPT-4o mini
Qwen2.5-72B
GPT4
Variant
5/6-way
6-way
5/6-way
6-way
5/6-way
6-way
All compromising
55
5
85
0
81
33
One greedy (ps E Poonst)
80
10
95
0
57
30
One greedy (pi)
45
5
50
5
27
9
Two greedy (Pbeaelit)
All greedy
50
0
15
5
26
11
Adversarial (untargeted)
90
0
85
0
63
Adversarial (targeted)
70
5
85
0
58
Table Performance in the different behavioral variants, comparing GPT-4o mini, Qwen2.5-72B, and GPT4, Pbenefit is the set of players whose goals align to those of pı and Poonst is similarly defined but with neutral goals with respect P1. It is not clear how these sets are delineated in the original paper, replicated the setup with the players indicated in the open source configurations. The results for GPT4 marked with were taken from the original work.
We also reproduce trends in game game evolution with respect to the players of interest (i.e., the greedy, adversarial, and targeted players) in Figure For the cooperative case, we observe that Environmental League is more willing to compromise, as indicated by its lower utility. Converscly, when prompted to adopt greody strategy, the affected agent maintains a higher individual score. In the adversarial experiments, the utility for Local Labour Union is lower in the targeted case, demonstrating the impact of adversarial play.
These trends become apparent when comparing model performance. In the P: greedy setting, the last round's distance between the greedy agent and the aggregated USW was 12.86 for GPT-4o mini but significantly higher (24.85) for 20wen2.5-7B--nnsrruch indicating that Qwen enforces greedy behavior more strongly. Similarly, in the untargeted adversarial setting, the distance between the would-be targeted agent and aggregated USW was -33.91 for GPT-4o mini, whereas for Srren2.5-2B--nsttee results indicate that stronger models like Qwen2.5-72B-nnse Instrrctt better capture prompt-induced behavior.
was only -14.03. These
11

---

Proctnal Number
Prestiel Mumber huml
Propese Number lo's
Propase Number
GPT: All cooperative
(b) GPT: Greedy
(c) GPT: Adv. (untargeted)
(d) GPT: Adv. (targeted)
Sumber ip-
Proppisl
Propois nunbow tamm
Prozuse number to tarm
(e) Qwen: All cooperative
(f) Qwen: Pt Greedy
(g) Qwen: Adv. (untargeted)
(h) Qwen: Adv. (targeted)
Figure 3: Evolution deals for several behavioral configurations. Eneirommental League is pt and acts as greedy for the greedy experiment and adversarial in both the untargeted and targeted experiments. For the latter, Local Labour Union is the target.
5
Discussion
The benchmarking framework proposed by Abdelnabi et al. (0024 comprises complex simulation envi- ronment that evaluates LLM negotiation. By choosing more complicated environment model based a negotiation game used to teach humans, they overcome a problem inherent in related works (Akata el al. 2023 Brookins and DeBacker 2023): : that the simulated games are not complex enough to represent real- ity. Furthermore, utilizing LLMs in a negotiation setting designed for humans opens the door to analyzing interaction through the lens of behavioral game theory. Additionally, the extensibility of the framework helps to maintain its relevance in this era. of increasingly powerful LLMs. However, this approach also comes at cost.
One key challenge is that, due to its complexity and extensibility, Abdelnabi et al. (024) introduces poten- tially infinite number of comparison axes through their various games, which makes comparative performance across multiple games difficult to interpret, as demonstrated in our experiments, particularly in Experi- ment 3. Additionally, complicates the application of established formalisms, such as Nash Equilibrium analysis, which is commonly used in other benchmark studies (Brookins and DeBacker] 2023). The absence of such formal criteria. further hinders the objective assossment of benchmark results. In our evaluation of the Benchmark Claim and Adjustments Claim, we have demonstrated various practical examples of this difficulty in interpreting benchmark results. This includes factors such as our ablation analysis in Experiment 2, and the demonstrated weaknesses in diversity of the provided games in Experiment 6. Through our extensions, we have provided additional context to the benchmark such as transparency on the previously mentioned factors as well as metries from social welfare to assist a user further understanding the benchmark results. Additionally, while the various minor bugs in the original implementations do not invalidate the results of the original paper, they potentially pose problems users evaluating weaker models. By fixing them, increase the potential andienceof this benchmark.
One potential idea for future work is developing metrics capable of describing and categorizing the various components of different negotiation games. A particularly useful example within the current framework would be a categorization of games that aligns with model performance. This would allow model developers claim, for instance, that their new LLM outperforms others specific game categories but not in others. Another exciting avenue of future work is to explore benchmarking the adversarial behavior of the agents through the framework by creating "robustness against adversarial attack" metric, in order to quantitatively compare the resilience in the adversarial setting introduced in Abdelnabi all (2024). An example of a potential metric could be the percentage of 5-way successful games, weighted by sparsity, in an adversarial environment with a single agent causing issues. We leave these polential works as ideas for further increasing the utility of this benchmark.
12

---

References
[Accessed 30-
01-2025).
Models Overview Mistral AI Large Language Models - docs.mistralai.
fttps:///docs.misrrae ai/
geceingsssattt///oood./////oeoee [Accessed 30-01-2025..
2024. [Accessed 8-04-2025).
Sahar Abdclnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. Cooperation, compe- tition, and maliciousness: Lim-stakeholddrs interactive negotiation. In The Thirty-eight Conference on Newral Information Processing Systems Datasets and Benchmarks Track, 2024.
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, and Ammar Ahmad Awan al. Phi-3 technical report: A highly capable language model locally on your phone, 2024a. URL https: https://arxx: //arxiv.
org/abs/2404.14 14219
Marah Abdin, Jyoti Aneja, Harkirat Behl, and Sébastien Bubeck et al. Phi-4 technical report, 2024b. URL
Mistral A1. Un Ministral, des Ministraux mistral.ii. sttpgs//massraalllaaaa/aaaaaassn [Accessd
30-01-2025).
Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. arXiv preprinl arXim:2305. 16867, 2023.
Philip Brookins and Jason Matthew DeBacker. Playing games with gpt: What we learn about large language model from canonical strategic games? Available at SSRN 4493398, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, and Jared Kaplan et al. Language models are few-shot learners, 2020. URL pttps:///axxxx..rr////b////// 14165.
Tim R Davidson, Veniamin Veselovsky, Martin Josifoski, Maxime Peyrard, Antoine Bosselut, Michal Kosinski, and Robert West. Evaluating language model agency through negotiations. arXiv preprint arXiv:2401.44533 2024.
DeepSeck-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and Ruoyu Zhang et al. Deepscok-rl: ivizing reasoning capability in llms via reinforcement learning, 2025. URL https:///axii: abs/2501 12948
U. Endriss, N. Maudet, F. Sadri, and F. Toni Negotiating socially optimal allocations of resources. Journal of Artificial Intelligence Research, 25:315-348, March 2006. ISSN 1076-9757. doi: 10.1613/jair.1800 URL
doi. wwt.///...../..rrrrooroo
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and Abhishek Kadian et al. The http:://axxxi..or/bb//ss/ .2783
llama 3 herd of models, 2024. URL httpp::/arxi...rrrtt
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, and Arthur Mensch et al. Mixtral of experts, 2024. URL http:p/s///aai...oo///////0/00.0
Mamoru Kaneko and Kenjiro Nakamura. The nash social welfare function. Econometrica, 47(2))423-433 1979. ISSN 00129682, 14680262. URL http::htt//.....et http://www..oooooorrrrraaab/////
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, and llge Akkaya et al. Gpt-4 technical report, 2024. URL
08774
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuckai Sun, Gongjun Xu, and Mikhail Yurochkin. tiny- benchmarks: evaluating lims with fewer examples, 2024. URL httth:///taaat wtte:ii.or//bbb/2400000 14992
13

---

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, and Bo Zheng et al. Qwen2.5 technical report, 2025. URL hnttps://arxiiis org/ abs/b//2, 15115
Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J. Kochenderfer, Betterbench: Assessing ai benchmarks, uncovering issues, and establishing best practices, 2024. URL
12990
Victor Sanchez-Anguix, Okan Tunah, Reyhan Aydogan, and Vicente Julian. Can social agents efficiently
perform in automated negotiation? Applied Sciences, 11(13):6022, June 2021. ISSN 2076-3417. doi:
10.3390/app111136222 URL http:://x.ddiiiii http://22......r.///.33333//1111
Kun Sun, Rong Wang, and Anders Søgaard. Comprehensive reassessmmett of large-scale evaluat outcomes in Ilms: A multifaceted statistical approach, 2024. URL
15250
Lawrence E. Susskind. Scorable games: A better way to teach negotiation? Negoliation Journal, 1(3): 205 209, July 1985. ISSN 1571-9979. doi: 10.1111..57--99911955:600000..01 org/ 10. 1tt1//0.111/1-.... 1111111799977 1985. tb00308.
URL http://dx doi.
Xun Tang and Takayuki Ito. Metric for evaluating negotiation process in automated negotiation. In 2018 IEEE International Conference Agenls (ICA), pages 26 29, 2018. doi: 10.1109/AGENTS AGENTS.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Bann-Batistt Alayrac, Jiahui Yu, and Radu Soricut et al. Gemini: family of highly capable multimodal models, 2024. URL/httss//arxiv org abs 2212. 11805,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, and Amjad Almahairi et al. Llama 2: Open foundation and ine-tuned chat models, 2023. URL https: //arxiv httg//b//2307..0900 09288
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024. URL //ttps//aaxxi, org/abs/ 2406. 01574
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. Livebench: A challenging, contamination-free llm benchmark, 2024.
URL https://arxin: http//aaxvvvvrr./a////// abs/ 2406. 19314
Ruijic Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models, 2024. URL https: http::::///axvv...o ht////bb///000 /404.. 18824
14

---

Acknowledgments
This publication stems from research conducted part of the Master's Programme in Artificial Intelligence at the University of Amsterdam. We give our thanks to the university, and in particular the teaching and organizing staff in the FACT course where this paper originated. Toannis Kapelangeorgis was supported by the John S. Latsis Public Benefit Foundation postgraduate scholarship. Special thanks goes to Alexandre da Silva Pires, who supervised our first paper-writing experience and without whom this work would not have been possible.
Appendix
A Implementation details
A.1 Model Details
We deseribe the models used in our experiments in Table 7 Some models were distilled from larger model, shown in column 3, and some models were also quantized for quicker run time considering computation constraints. The level of quantization was primarily determined by computational and memory constraints; whenever feasible, a less aggrossive quantization (i.c. int8) was employed, though larger models necessitated int4 quantization. Details can be found in Table
Parameters (B)
Context
Model
Architecture
Additional Details
Window
Llama 3.3- 70B
70
Decuder only Trausornerr
128k tokens
SwiGLU, RoPE, RMSNorm
Mistral-Small
22
Cost-effective Transformer
33k tokens
Tack-optimized
Phi-3.5-mini
3.8
Compact Trausformer
4k tokeus
Elliciecy-ffcuse
Mistral-8B
8
Performance-optimizze
128k tolens
Balanced efficiency
Qwen2.5 72B
72
Dense only
131k tokens
Advanocd capabilities
Phi-1
State-of-the-art Transformer
16k tokens
High performance
Qwen2.5-7BB
Efficient Transformer
131k tokens
Diverse applications
Llama-2-13BB
Decoder-only Transformer
4k tokens
SwiGLU, RoPE, RMSNorm Computational efficiency
Mixtral-8x7B 8x7B
56 (8x7)
Mixture of Experts
33k tokens
Llama-3-8
Decoder-only Transformer
8k tokeus
SwiGLU, RoPE, RMSNorm Fine-tuned with DeepSeek- RL- generated data for reasoning Fine-tuned with R11pSeek-R1- geuerated data for reasoning
32
Distilled from DeepScek-RL.
32,768 tokens
based Qwen2.5-32B3
DeepSrok R1-Diiiiiee Distill Disttill-Llaama 70B
70
Distilled from DeepSeok-RI, based on Llama-3.3-70D-Inste
128,000 tokens
Table 7: Summary of open-source models evaluated, including their parameters, architecture, context window size, and additional details.
Model
Quantization Applied
Llama-2-13B
int8
Llama-3.3-708
int4
Qwen2.5-72B
int4
Phi-4
int8
Mistral-Small
intS
Mixtral-8x7B
int4
DeepSick-RL--isstll---eere
int8
int4
Table 8: Quantized models and techniques applied
A.1.1 Code Structure
Both the original and our codebase are structured into multiple Python scripts, each handling a specific aspect of the experimental process. The main script, main.py, serves as the entry point, integrating all components
15

---

to exccute experiments. Supporting modules manage agent behavior, negotiation rounds, prompt generation, and data handling. Detailed documentation of the code structure is available in the README file of the GitHub repository.
Initially, configuration options were managed through a combination of command-line arguments and exter- nal configuration files. The command-line arguments allow users to specify parameters such as temperature, number of agents, number of issues, number of rounds, game name, and model name. Additionally, each game directory contains a config. tst file that defines the setup for each agent, including their name, asso- ciated files, role, incentive, and model. This structure enables customization of experiments by modifying both command-line inputs and configuration files.
However, to further enhance configurability and streamline the experimental workflow, several improvements have been introduced beyond this initial approach. Previously, the specification models and incentives for agents was handled by modifying the game's configuration filc. However, due the frequent need to rerun the same game with different model and incentive configurations, the -model and incentlue command-line parameters have been added to simplify the process and reduce manual intervention. Additionally, the guantization parameter has been introduced to enable quantization for Hugging Face models, improving performance and reducing memory requirements. To address concerns regarding sensitive information dis- closure during negotiations, the -restrict leakage flag has been implemented, with further details provided in the respective section of the report. A dry run option has also been included to facilitate testing and debugging by disabling actual API calls to language models, allowing users to inspect prompts without incur- ring computational costs. Furthermore, environmental sustainability considerations have been incorporated into the framework through the integration of the codecarbon Emissions which records the carbon footprint of experiments, with the option to specify project names via the emission project parameter. These enhancements build upon the initial configuration system, providing a more efficient, scalable, and envirommentally conscious framework for conducting negotiation experiments.
A.2 Computational Details
Hardware Setup The experiments were conducted using local machines equipped with NVIDIA A100 and A6000 GPUs, with memory capacities of 40 GB and 48 GB, respectively. The computational resources were sufficient to accommodate all models tested, with the most demanding model requiring a maximum of GB of VRAM. Thus, the experiments can be reproduced on a system with similar GPU specifications.
Runtime Measurements The runtime for each negotiation session varied depending on the model used. Table g presents the average runtime per model for complete negotiation session comprising 24 rounds. It was observed that computational resource demands varied significantly among different models, with some weiht models requiring considerably fewer resources compared to larger models.
Model
Runtime (min)
GPT-4o mini
4
Mistral-Small-lnstret--aa
20
Phi-3.5-mini-nnsrreet
3aama333-70B-Insttmee
22
MinisttaalSB--Instruc-
15
Phi-4
15
Dove2.5-7BBIInsree
9
Llana-2-13b-chtthh
11
Mixttral&x77B--nssrrut----
Meta-Llama-
3
Qwen2.5577BB-nnsttne
29
ReepSeeek-I--issill----eeee
68
DeepSeek- Dir-
65
Table 9: Average runtime per negotiation session (24 rounds) across different models.
The total time required to complete all experimental runs, including different game variants, amounted 422 hours. For open-source models, the total GPU/CPU hours required were 386 hours. For GPT-4o mini, the OpenAT API was utilized, resulting in different resource consumption patterns.
16

---

Storage Requirements The primary storage requirement for the experiments was to store the down- loaded pretrained model weights. By storing one model at time, the total storage demand was limited to approximately 300 GB. All experiment outputs and logs were stored for post-hoc evaluation and further analysis.
Software Environment The experiments were conducted on Limux based systems. For OpenAl models (i.c., GPT-4o and GPT-4o mini), the OpenAI API was utilized, whereas for open-source models, the Hugging Face transformers library was employed. Additionally, the codecarbon library was used to monitor carbon emissions during the computational processes.
B Ablation prompts
We present here the prompts used to create the Chain of Thought for the agents. These variations were used to recreate and extend the ablation study, Table
B.1 Previous Deals
The prompt consider previous deals is:
Please use the scratchpad for the following: Given the history of previous suggeated deals, calculate your scores for each past suggested deal in the history. Then, use these deals to further reason about your next proposed deal.
B.2 Others Preferences
The prompt consider the preferences of other players is:
In your scratchpad, 1) think about what others may prefer, 2) Based others' preferences and history and your notes, propose proposal that balances between your scores and accommodating others and that is more likely to lead to an agreement.
B.3 Candidates
The prompt to generate three deal candidates before choosing one is:
Please use the scratchpad for the following: Before suggesting a final deal, display three possible deals to suggest while keeping in mind your objectives, and number them with (1), (2), and (3). When your final deal is made, choose one of these three deals.
B.4 Planning
The prompt to release a plan at each round is:
After the final answer, building on your current move and analysis, briefly write down short notes for yourself of what exact options you can explore the next time you speak. Enclose the notes between <PLAN> and </PLAN>.
C Carbon Emissions
Environmental sustainability considerations have boen incorporated into our framework through the integra- tion of the codecarbor Jemissions tracker, which records the carbon footprint of experiments by considering the specifications of the system running the model, the carbon efficiency of the country is powered in, and electricity usage. By extrapolating the results of running the carbon tracker for each model on a set of 20 games, the total cost of our runs approximately 48.7kg COge. We used 24 million input and 28 million output tokens of GPT-4o mini. We now state the carbon emissions that our experiments entailed.
We use the figures from Table 10 to extrapolate the total carbon usage per experiment based on which models are run.
Total Carbon Emissions Carbon(Ti) + Carbon(T2) + Carbon(ablation) + Carbon(Ta) + Carbon(Tio) (1)
hhtppe//rodeecaron.in//
17

---

Model
Average Emissions (kg CO2e)
Llama-2-13B
0.4
Llama-3.3-70B
1.0
0.1
Ministral-8B
0,6
Mistral-Small
0.7
Mixtral 8x7B
0.3
Phi-3.5-mini
0.2
Qwen2.5-72B
1.3
Qwen2.5-7B
0.4
Phi-4
0.6
Deepseek-R1- R1-Disstll--lla-a---0000
2.7
Deepseck-R1-Distilll RRI-DistillQWee--23
2.8
Table 10: Estimated average carbon emissions (kg CO2e) per model for processing a batch of 20 gaine
where T₁ corresponds to the carbon emissions associated with the experiments populate Table Since all runs are done with 20 games, these calculations are dlraightforward and the code is provided. Additionally, we monitored the emissions for running the ablation study and found that
Carbon(ablation) 19.8kg COze
The carbon generated per table is
Carbon 11.1
Carbon 2.0
Carbon (Trbon(To) = 8.0
Carbon (Tio) 7.8
And thus the final calculated carbon output is
Total Carbon Emissions = 48.7kg CO2e
(2)
These emissions are approximately equivalent to those produced by driving Ford F-150 for 117 miles, which is roughly the distance between Rotterdam and Groningen.
We also used GPT-4o mini. However as the running cost for running this model unknown, we exclude from the calculation.
18

---

D
Leakage Results
Model
Original Code
Our Code
Final 1 (%)
Any 1 (%6
Wrong (%)
Loaked (%)
Final (%)
Any 1 (%)
Wrong (%)
Lonked J (%)
Falled J (%)
S-way
6-way
S-way
6-way
Ministri--88
25.0
500
22.22
10
Mixnal-Sx7I
30.0
55.0
23.84
24.31
26,07
0.0
66.67
20.14
25
Phii-.5-amimi
10.0
10.0
50.0
12.87
10.77
35.0
10.0
85.0
10.31
0.38
Qwen2.5 7B
65.0
25.0
100.0
11.15
41.82
$8.41
5.50
88.89
8.76
10
Table 11: Comparison of model performance metrics before and after the leakage issuc is resolved.
Original Code
Our code
Model
Successful Games
Leaked Games
Leaked (%)
Successful Games
Leaked Games
Leaked (%)
Llama-2-15 13b
20
11
9.23
12
0.32
Llama-3-8B
20
20
69.42
20
0
0.00
Ministral-SB
20
20
7.12
18
0.00 0
Mixtrul-8x7B
20
20
24.81
15
0
0.00
Phi-3.5-mini
20
20
40.77
20
0.38
Qwen2.5-7B
20
20
44.62
18
0.00
Table 12: A comparison of models that are prone to leakage before and after code adjustments.
After applying our method to the original experiments and manually verifying outcomes, we find that our strategy significantly reduces leakage, lowering the associated metrics by up to 69.42% compared to the original approach. Furthermore, our refined method results in lower 5/6-way agreement rates, as agents no longer have unintended access to private information that could facilitate deal-making. While no detection framework is entirely exhaustive, our improvements offer substantial reduction risk of leakage.
E Social Welfare Definitions
Let Sm denote a score function such that each deal assigned a natural number value under some utility m: (m) N.
Utilitarian Social Welfare (USW): Aggregates each player's utility for a deal, as in the original study, but does not consider fairness.
Susw (T) = Σ Up,(T)
(3)
P.EP
2. Egalitarian Social Welfare (ESW): Ensures all players benefit by taking the minimum utility across players for each deal.
P.EP Susw (T) = min { Up. (T)}
(4)
Nash Social Welfare (NSW): Promotes fairness by multiplying players' utilities, favoring even distributions (e.g., 4 x 4 > 2 6).
Snow (T) = II Up. (T)
(5)
P.EP
19

---

F loU Equations
The Intersection over Union (loU) metric quantifies similarity between agent scoring functions by measuring the overlap in their preferences. Pairwise IoU computes the ratio of shared valuations between two agents, while the overall loU aggregates these values across all agent pairs. The following equations formally define these computations, as referenced Experiment 6, to assess preference alignment and diversity.
ToU =
Σ
Pairwise (pU;
(6)
P P=EP
P
PMPPH
PyEP)(pa)
1
1 min (Pz.iy, Py.i.)
1
Pairwise loU Py) =
(7)
max (Pr,ix P..i.)
G Alternative Prompt
You are an expert in negotiation games and have read many books on the subject. help me in creating a negotiation game. The game consists of players (party 1, party 2, party 3, etc.) who are negotiating over 5 issues. Each of the 5 issues has different sub-options (2 issues have options, issues have options, 1 issue has options). One of the players makes first proposal. The issues involve the impact of the negotiated outcome stakeholders. The other players represent different parties with competing interests. The parties must not include mediator. The issues represent the interests of other parties. The issues do not necessarily have one-to-one mapping to each party; different parties might have similar or competing interests under each issue (e.g., one wants more funding, one wants about certain issues (they only care about subset issues). The game is based cooperative bargaining. Your task is create the background story and the role of each party according to the previously mentioned guidelines. Please indicate their general goals and motivations and their objectives from the negotiation. You should also create the issues they are negotiating over (please name them issues A, B, etc.) by specifying the different sub-options (A1, B1, C1, etc.). For each issue, please specify what the preferences of each of the parties are over the issues and why they prefer so (e.g., Party 1 prefers then A2 then A4, etc.). Please also assign priorities of the issues each party and explain why (e.g., Party cares the moat about issue they do not care about issue D) Please also indicate if an issue is much more important than the others. Make it interesting with lots of potential for cooperation and competition between parties!! Make the issues and options have some implications over generally more than one party involved, but you can have some parties with no interest at all in some issues. Remember that it is a cooperative non-zero-sum game.
Please
funding, etc.). Some parties do
not care
20


Reviews:
[{"review_id": "mKsekDmH73", "reviewer": "Reviewer_eqCD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n========\n- The paper introduces new metrics that can give new insights based on the notion of social welfare (although the paper does not elaborate on how to interpret them to evaluate the agents or the difficulty of games)\n\n- The paper identifies improvements to fix problems related to models not following correct formats (for example, making sure the agreement is properly calculated based on the final deal and that the final deal is correctly formatted and not carrying over a previously parsed deal as the final one). \n\n- It also improves the evaluation of whether the scores were leaked by making sure to evaluate this on the correctly formatted public answers while reporting failures in formatting (although I disagree that the paper in the correct form evaluates leakage correctly and it significantly oversimplified the leakage to formatting, rather than more broadly as how models preserve theory-of-mind and contextual integrity principles, details on that in the next section).\n\n- The paper extends the evaluation to many other models, especially new ones such as DeepSeek. This is helpful to track the performance of reasoning models.\n\n- I found the discussion of the paper valuable on breaking down negotiation to specific sub behaviors. \n\n- The paper points a minor issue in the writing of the original paper (must exceed) while the code is greater than or equal. However, the code is consistent with the original game (Susskind et al.) \n\n------------------------------------------------------------------------------------------\n\nWeakness:\n========\n\nWhile I think the paper introduces valuable experiments and discussions, in many situations, I find the paper to overclaim the contributions made, oversimplify the metrics and provide them as better alternatives, and make wrong claims about the original paper. For a reproducibility report, the paper assumes claims that were not stated in the original work and then refutes them. However, they were not really claimed. The paper also extends the discussion with many limitations, some of them are already raised and acknowledged in the original paper, as criticism of the original work. Discussions about extending evaluating the negotiation are valuable, but framing them within the context of reproducibility is misleading to the reader. The conclusion of the paper itself does not support the claims made in the introduction. The conclusion mentions \"Additionally, while the various minor bugs in the original implementations do not invalidate the results of the original paper, they potentially pose problems to users evaluating weaker models. By fixing them, we improve the diversity of the audience of potential users of this benchmark.\", but this is not the main message the rest of the paper implies. \n\nTherefore, I think there is a room for overall improvement of the paper with clarification, rewriting, and further discussions. I elaborate on this and other points below. \n\n## Minor: \n\n- The abstract is not clear, the use of the word \"fairness\" implies it is related to biases and fairness of decision making, while the rest of the paper show it is actually about trying to unify the evaluation and comparison between different models.\n\n-  Related to the previous point, the introduction mentions \"leakage risks\". The paper didn't show how this benchmark may have leakages risks and mentioning it in the introduction biases the reader into believing the paper is actually about leakage. These two points are minor but it is related to it is not clear from the abstract and introduction to know what the scope of the paper actually is. \n\n## Motivation: \n\n- In the introduction, the paper implies that the \"non-viability of the benchmark on many weaker models\" weakens the ability of the benchmark to evaluate models. I disagree with the premise of the paper that all benchmarks need to be tailored to ALL models. I believe we need complex benchmarks for specifically frontier models and that is where the gap is. There exists previous simples benchmarks for negotiation (Davidson et al. https://arxiv.org/abs/2401.04536). That being said, Abdelnabi et al.'s benchmark (the games itself) can still be used for evaluating smaller models. There are two pillars that the paper are confusing: the games themselves, and the metrics. While a metric such as the agreement rate is discrete and might show all smaller models are equally bad, one can propose other metrics using the same games (such as following formats, not leaking information, performing correct calculations, mapping issues to correct scores, making correct inferences about other players' preferences, etc.). Future work could use the benchmark (the games) and augment the metrics in the original paper for more evaluation. \n\n## Misleading contribution claims \n\n- \" we identify a more diverse set of negotiation settings, including discussions on military spending budgets for the upcoming year and planning a conference, where aspects like floor allocation and catering must be negotiated.\" The original work followed this prompt setting with manual curation of games and rewriting, as mentioned in the original paper. I don't believe the paper makes a contribution in having end-to-end negotiation games, just the seed.\n\n## Oversimplification of metrics\n\n- In 3.1.5, \"addressing leakages\", as mentioned above, performing the analysis on strictly correctly phrased public answer is a good and valid addition (while still reporting the rate of failures because this an important property of evaluating the model). However, the paper seems to consider leakage only when \"scratchpad\" or \"plan\" tags appear. I disagree with this approach. In fact, using GPT4 evaluation is the better way to do this. In many cases, models may use the proper format but still mentions the score (\"as it provides a mutually beneficial agreement that meets the minimum score requirement of 50.\" -- see Llama2 70b output https://github.com/S-Abdelnabi/LLM-Deliberation/blob/main/logs/llama2_70b_base_all_coop.zip, or even mentioning \"getting closer to my minimum score requirement\", ). This should be detected because 1) it is a failure in following instruction, 2) it is not aligned with how humans negotiate by not revealing their priorities and plans. Also, in many cases, it indicates a theory-of-mind limitation, see Appendix I, as advocating for a deal with a higher score for a party in the public answer may indicate the models are not correctly processing that other parties have different priorities (I endorse Deal 12 ( A1, B1, C3, D3, E4 ) as my final proposal. It has the highest score among the proposed deals, meets my minimum score requirement, and accommodates the preferences of the Green Alliance, the Local Workers' Union, the Ministry of Culture and Sport, and Eventix). \n\n## Evaluation choices that are both valid and are not weaknesses of the original work \n\n- The paper excludes the setting of relaxing the threshold of p1 if leading to all party agreement, this is an experimental choice which is valid. However, the original paper follows Susskind et al. which has the same condition. In Abdelnabi et al.'s setup, this information was given to p1. In cases where GPT-4 is cooperative, it may perform correct calculations and reach a correct conclusion that the scores are less than the threshold. However, it can add the 10 point since it is attempting to have an all-group agreement. For this reason, it can make sense to consider this for checking agreement to accommodate for this. Removing this for checking agreement is valid, but should be removed as well from p1's prompt.  \n\n- For baselines (in 3.2.1), as far as I know, there are no clear canonical baselines for the game. A random order is valid. However, the \"priority\" of issues is not undefined, this can be based on the maximum score that can be achieved by this issue. -\tIn Table 4, I don’t understand on what factors this baseline provides more accessibility and explainability. The difference between this baseline and the one in the original paper is using random ordering instead of sorting. I agree however with the paragraph after about that rule-based heuristics do not capture the complexity of human negotiations.\n\n## Incorrect claims about the original work \n\n- For 3.2.2 Adjustments claim, Abdelnabi et al. didn't claim that the number of the set of feasible deals size is the main diversity factor between the different curated games. In fact, the paper mentioned explicitly that scores were adjusted to have very similar numbers to the base game (\"we assigned numerical scores to reach a comparable ratio of feasible deals compared to the base game\", section 3). In my opinion, this is valid, as since the random chance ratio of finding an agreement is the same, one can make further analysis of why certain games may seem to be easier than others. Abdelnabi et al. didn't also claim that sparsity can be easily tuneable, section (4.6 Tuning the Game Difficulty) in the original paper only mentions adjusting minimum scores. However, they claimed that the pre-defined and created games have (fixed, curated) diversity in terms of sparsity. They claim that \"game 1\" and \"game 2\" are less sparse compared to the \"base\" game and \"game 3\" (this is actually reproduced by table 5) in this paper. The original claim of creating diversity by changing thresholds seems to be reproduced with this paper (Fig. 2)\n\n- I don’t disagree that different prompts may work better with other models. As far as I can tell, Abdelnabi et al.’s paper didn’t claim that best prompt structure would generalize. However, the paper didn’t really provide a bigger picture of trends and didn’t provide enough context of how this would change any major conclusions. It is not really clear from table 2 that these yellow configurations are better than the red one (for example, they have 0% 6-way agreement for Qwen-72B). Even under the same model, given that experiments are run for 20 times for each model which is understandable due to the size of each game, these small deviations are expected. If the intention is to show very thorough comparison of models rather than showing general trends, yes, the ablation should be done for each model. Additionally, experiments should be run for more rounds. \n\n- “The results clearly demonstrate that the negotiation scenarios generated with the original prompt are not diverse, and simply reusing the prompt does not lead to greater scenario variety”. I believe this claim is not supported by defining what “diversity” is. Sure, having more scenarios other than constructing a project is an additional source of diversity, however, this does not necessarily mean than within the same domain, one can have other sources of diversity (in the semantics and relationships, etc.). The paper can claim they provide more diversity, but it shouldn’t overclaim without evidence the lack of diversity of the original work. Of course, if one needs to create a significantly large sample of games, more systematic analysis should be done to say how diverse/redundant they are. But, Abdelnabi et al. didn’t claim they provide a mechanism to create new games end-to-end. The games proposed are fixed and manually curated. The discussion of how to extend that is important, but not within the context of reproducibility. The original paper already acknowledged the limitation of scaling and changing scores and it does not claim that creating diverse games or changing scores automatically were done. \n\n## Other points \n- The results in table 3 are interesting, but the paper does not provide further analysis and observations. For example, about Mistral’s bad performance on game3. That may provide further insights. \n\n- “game difficulty is not an inherent property of the game itself but is instead directly influenced by the model being evaluated” one can identify several properties of the game itself (for example, sparsity being one of them).\n\n- The paper mentions \"Additionally, it complicates the application of established formalisms, such as Nash\nEquilibrium analysis, which is commonly used in other benchmark studies (Brookins and DeBacker, 2023).\nThe absence of such formal criteria further hinders the objective assessment of benchmark results.\" That can be true, I don't disagree. However, the paper does not take into consideration that these games used in the original work were not invented by the authors (as it is clearly mentioned in the paper). It is a very old exercise for teaching negotiation that is still used now. Beyond rational players, this can have many potential opportunities to study parallels between humans and LLMs and also the communication aspects inspired by behavioral game theory.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is a reproducibility report of the work of Abdelnabi et al.(https://proceedings.neurips.cc/paper_files/paper/2024/file/984dd3db213db2d1454a163b65b84d08-Paper-Datasets_and_Benchmarks_Track.pdf) which appeared last year at NeurIPS datasets and benchmarks track which was a benchmark for negotiation between multi-agent LLMs. The paper reproduces some of the results in the paper. It also extends the evaluation to other models. It discusses additional metrics. It fixes minor bugs in the calculations of the final deal of the original code.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I believe the conclusion of the paper and the main message in the introduction should be more consistent. At the moment, the paper exaggerates the contribution by making unsupported claims and refuting claims were not introduced in the original work. \n\n- The paper needs to discuss the limitations of their approach in calculating the score leakage. \n\n- It would be of great value to elaborate on the many points raised in the discussion on how to extend the original work. For example, how to systematically (and objectively) assess the diversity of games, not only based on overall scores but pairwise dynamics between players and semantic role. Another point how to change thresholds automatically. And how to extend the work to study other attacks and safety aspect. By that I believe the paper would serve advance the field. At the moment, the paper raises many points, some of them were raised by the original work, while not introducing an attempt for improvement.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1m97AVrqgh", "reviewer": "Reviewer_aE6q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n•\tThe paper extends the original benchmark with recent models.\n\n•\tThe paper fixes some minor issues in the original code such as using greater than and equal consistently across experiments.\n\nWeaknesses:\n\n•\tThe major weakness is that the manuscript confuses design issues and plausible extensions with bugs in implementation. This is apparent across the entire manuscript as follows:\n\no\tThe Addressing Leakage bug: doing a postprocessing approach to filter out the leakage or enabling smaller models to perform better is an extra engineering process that is now a confounding factor that should be benchmarked on its own. This is the opposite to actually benchmarking the models on their own accord in handling such nuances without heavy post processing. This also completely contradicts the claim made in the manuscript “An analysis of the original implementation reveals that the reported leakage metrics are largely due to code limitations rather than intrinsic model behavior”. What is explained after in the same section shows that limitation is actually an intrinsic model behavior and the paper method to mitigate it is an engineering approach and design choice to overcome such a limitation. This is opposite to other benchmarking methods such as the original benchmark and [1] that consider the hallucination and illogical behavior as part of the benchmarking aspect\n\no\tThe adjustment claim: While this manuscript shows a possible context bias in generating new games, it did not contradict the original negotiation claim about assessing the different models. For example, in table 3, in can be seen that Qwen2.5-72B is already performing better than GPT4o-mini and Mistral-small. \n\n[1] Bianchi, Federico, et al. \"How well can LLMs negotiate? NEGOTIATIONARENA platform and analysis.\" @ICML\n\n•\tWhile the manuscript provided numerous future work prospects in the discussion section. These is also one of the main weaknesses of the manuscript as it is suggested in the introduction and the abstract that this is the contribution of the current work. Additionally, the discussion section does not concretely specify the contribution and the outcome of the reproducibility of the benchmark. For example, it specifies that the extended explanations would improve the diversity of the audience for potential users of the benchmark, but it does not specify how it does so, and how that is represented in the metrics used in the original benchmark or the current manuscript. In other words, are the metrics different from the original manuscript? Should different users use different metrics? Is there some adapted metric for specific users? Did the current manuscript overcome the limitations mentioned in the original benchmark and was able to produce more simulation games automatically? And so on…", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper reproduces the results of the paper by Abdelnabi et al. named “Cooperation, competition, and maliciousness: LLM-stakeholders interactive negotiation” published in NeurIPS 2024 Dataset and Benchmark track. Additionally, the paper updates the originally used models in the benchmark with recently released ones and fixes minor issues in the original code. The paper also expands the discussion on the benchmark itself and provides some additional context on the negotiation aspect of it. Finally, the paper highlights some future directions to expand the metrics used in the original benchmark and the manuscript itself.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "There are several changes that could enhance the paper in matching the claims proposed in the abstract and introduction:\n\n•\tThe abstract, in itself, is ambiguous and could be made more concrete. For example, explicitly mentioning how is model comparison is ambiguous in the original benchmark and how did this manuscript make it clearer or objective. Additionally, some of the terms such as leakage detection and ablation transferability are also vague and do not clearly address the issues or highlighting how does this paper improves that. In general, the abstract is neither clear about the exact parts that need rectification in the original benchmark, nor the exact contributions made by this work other than extending the models and discussion (which is the legitimate use case for a benchmark paper and not a judge of reproducibility). \n\n•\tThe discussion does not match what is mentioned in the abstract and introduction. While the abstract mention critical issues with the original manuscript, the discussion mentions minor changes and minor fixes. This is also apparent across most of the experiments presented in the manuscript itself. The manuscript also seems to mix between design issues and coding/mistakes in the original benchmark which should be rectified (see weaknesses section for more details). \n\n•\tThe contributions section should be rectified as well to match the actual contribution and what is presented in the experiments and discussion (see weaknesses section as well).\n\n•\tAdditional metrics should be provided as mentioned in the abstract to justify the need for a reproducibility check in the first place as the manuscript is currently showing that the original benchmark already provides sufficient metrics for evaluation as seen in Table 1. In other words, additional metrics (as suggested for future work in the discussion section) should be the focus of this manuscript as a reproducibility paper and as suggested in the abstract. Otherwise, the additional metrics added such as the sparsity and IoU metrics seems to measure the original benchmark itself and not a metric that could be used to enhance the benchmark when judging the agents.", "broader_impact_concerns": "There are no broader impact concerns in my opinion.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "28T8BLBkR9", "reviewer": "Reviewer_KzXy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Extensive experiments: These experiments include replication of prior results, extended ablation studies, behavioral tests, and the introduction of new evaluation frameworks.\n- Technical improvements: It fixes key flaws in the original benchmark, such as leakage bugs and inconsistent scoring logic, improving fairness and reproducibility.\n- New evaluation metrics: Introduces social welfare-based metrics (Utilitarian, Egalitarian, Nash) to better capture negotiation quality and fairness.\n\nWeaknesses\n- Although the study highlights that different games present varying challenges to different models, it fails to offer a clear, standardized metric for defining or comparing game difficulty.\n- The rationale for selecting specific models is not always clear or consistently justified (see Requested Changes below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Find some technical flaws of a paper published in NeurIPS 2024 D&B Track. In conclusion:\n\n1. We need to consider prompt sensitivity in such experiments.\n2. We need to have an objectively defined difficulty of games instead of using the performance on one model to define.\n3. We need to handle LLM-generated test cases with care, especially on the bias issue.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add more details to section 3.1.5. This section plays an important role to show the flaws in the original paper. However, it is hard for readers who are not knowledgable about the original paper to understand the logic here. For example, what are the <ANSWER>, <plan>, <scratchpad> tags? What are they used for? \"The system inadvertently exposes confidential outputs\": Why and how does the system expose such information?\n2. Different experiments use different models currently. Table 2 uses GPT-4o-mini and Qwen-72B, Table 3 uses GPT-4, GPT-4o-mini, Mistral-Small, Qwen-2.5-72B. Figure 1 uses Phi and R1. I think it is better to have a unified setting or explain clearly why certain models are selected for different tasks.\n3. Discuss whether quantization will affect the model performance. Explain why some models use int4 while others use int8.\n4. Add some results on stronger closed-source models such as GPT-4o, Gemini-2.0, or Claude-3.7.\n5. Add deeper analysis about how the fixed bugs may have influence on the original results & conclusions.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission is to the Reproducibility track, where the emphasis is on insights and reproducibility rather than novelty. The reviewers acknowledge that the authors have addressed most concerns in the revision. While some ambiguity remains regarding the novelty and interpretation of certain design choices, the paper appears to meet the acceptance criteria for a reproducibility submission. Most remaining issues relate to writing and clarity, which are fixable. Overall, the paper makes a meaningful contribution, and I recommend acceptance upon minor revision.
Decision: Accept (poster)