Reproducibility Study of "Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation"
Authors: Jose L. Garcia
Karolina Hajkova
Maria Marchenko
Carlos Miguel Patiño
Venue: Accepted by TMLR ()

Abstract: This paper presents a reproducibility study and extension of "Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation." We validate the original findings using a range of open-weight models (1.5B-70B parameters), GPT-4, and GPT-4o Mini while introducing several novel contributions. We analyze the Pareto front of the games, propose a communication-free baseline to test whether successful negotiations are possible without agent interaction, evaluate recent small language models' performance, analyze structural information leakage in model responses, and implement an inequality metric to assess negotiation fairness. Our results demonstrate that smaller models (<10B parameters) struggle with format adherence and coherent responses, but larger open-weight models can approach proprietary model performance. Additionally, in many scenarios, single-agent approaches can achieve comparable results to multi-agent negotiations, challenging assumptions about the necessity of agent communication to perform well on the benchmark. This work also provides insights into accessibility, fairness, environmental impact, and privacy considerations of LLM-based negotiation systems.

Published in Transactions on Machine Learning Research (04/2025)
Reproducibility Study of "Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation"
Jose L. García; Karolína Hájková; Maria Marchenko; Carlos Miguel Patiño"
University of Amsterdam
{jose. gorse.goria..oomm corrillo, aurolinahajjovve maria, marchenke, carlos. carlos.patino, paaaiiio..a///uuudnnl...... (ttddent..dddd..m uva.
Reviewed on OpenReview: https: // openreview. net/ forun?? id= 4dNNThFFmk445
Abstract
This paper presents a reproducibility study and extension of "Cooperation, Competition, and Maliciousness: LLM--takeholders Interactive Negotiation.' We validate the original findings using a range of open-weight models (1.5B-70B parameters), GPT-4, and GPT- 4o Mini while introducing several novel contributions. We analyze the Pareto front of the games, propose a communication-free baseline to test whether successful negotiations are possible without agent interaction, evaluate recent small language models' performance, an- alyze structural information leakage in model responses, and implement an incquality metric lo assess negotiation lairness. Our results demonstrate that smaller models (<10B parame- ters) struggle with format adherence and coherent responses, but larger open-weight models can approach proprictary model performance. Additionally, in many sccnarios, singlo-agent approaches can achieve comparable results to multi-agent negotiations, challenging assump- tions about the necessity of agent communication to perform well on the benchmark. This work also provides insights into accessibility, fairness, environmental impact, and privacy considerations of LLM-based negotiation systems.
1 Introduction
The rapid advancement of Large Language Models (LLMs) has opened new opportunities for Artificial In- telligence (AI) applications, particularly in the form of autonomous AI agents. These agents, capable of interacting and communicating with one another, can be especially useful in negotiation systems. Negotia- tions involve complex multi-agent interactions that require resolving multi-issui scenarios with little to no supervision, so it is critical evaluate whether these agents are reliably performing their intended tasks (Shavit al. (202)). Within this context, reprodueibility studies are valuable, as they help validate bench- marks the AI community
to evaluate the performance of current and future models (Reuel et al., 2024 Chang et al. 2023 This paper aims to reproduce the results of an LLM testbed specially designed for negotiation games [Abdelnabi et al. 2024) to contribute to developing robust. benchmarks.
Our reproducibility goal is to assess whether the benchmark effectively evaluates the models' nogotiation performance and whether the Shain-of-Thoughhe (CoT) prompt configurations contribute to successful deal- making, especially terms of accountability (Chan et al. (024) and confidentiality. To achieve this, we compared different open-weight models against & taate--ff-hheart (SOTA) closed-source model. Building on the original paper, analyze the Pareto front of the proposed games and establish strong baseline to test whether agent successfully propose deal without communicating with other parties.
We also present extensions that address the environmental impact, accessibility, fairness, and confidentialityy For environmental impact and accessibility, we use the benchmark to evaluate open-weight models and compare their performance to larger, more energy-consuning ones. Smaller models reduce the computational resource requirements (Fu al. 2024 Sinha al. 2024), and we compare the performance between models
* "Equal contribution

---

Published in Transactions on Machine Learning Research (04/2025)
that range from 1.5B to 70B parameters. For fairness, introduce inequality metric to assess whether all parties benefit equally from the negotiation. For confidentiality, compare open-weight models with a closed-source model that requires access through an external API and may expose attack surfaces that can be exploited by malicious actors (Evertz et al. 2024 Dunn et al. 2024
2 Scope of Reproducibility
We focus our reproduction efforts on investigating the following claims from the paper:
Open-weight models fall slightly behind SOTA closed-source models.
2. Small models i.e., models with 8B parameters or less are not suitable for the negotiation bench- mark because they cannot follow the format of the game.
Agents have better outcomes when using CoT structures in their prompts.
The agents' incentives in the benchmark result in non-zero-sum games.
5. The benchmark measures cooperation, communication, and negotiation skills.
The code and instructions for reproducing the results of this work, along with our additions and the fixes mentioned in Section A.9 are available on Github
3
Original Paper
3.1
Benchmark and Base Game
The original paper (Abdelnabi et al. 2024 introduces a benchmark to evaluate whether LLMs can success- fully reach agreements in a multi-agent, multi-issue, issue, semantically rich negotiation game over multiple rounds. The benchmark is designed to test agents in dynamic, multi-turn scenario that requires advanced reasoning and strategic decision-making
Specifically, it assesses their ability to reach agreements through arithmetic calculations, inference, exploration, and planning, in addition to essential skills such as communication, collaboration, and instruction following.
The benchmark's base negotiation game is adapted from HarborCo (Susskind 1985 a game traditionally used leach negotiation skills involving six parties uegotiating over five key issues related to constructing a sports complex. Each party assigned a name and role with different goals and preferences for each issue, That way, every issue is presented with three to five options, each assigned with score for the evaluating agent. A deal proposal is agreeable for agent if the sum of the chosen options across all issues meets exceeds its minimum utility score threshold.
Before the game begins, all agents are given a global context outlining the negotiation sctup, including information about the issues and the other agents' assigned names, roles, objectives, and preferences. The game then encourages agents to cooperate and compromise when necessary over the negotiation rounds achieve a successful outcome, as negotiation is only considered successful if least five of the six players surpass their scores threshold and accept the final deal, including Player (p1) and Player 2 (p2).
Player 1 and Player 2 are specially designated roles with veto power: for the final deal to be accepted, it must receive approval from both. Pl, referred to as the main negotiator, is also responsible for proposing the negotiation's 's first and final deals. As the main negotiator, P1 receives a bonus of 10 points upon suggesting a final deal accepted by everyone.
The initial deal is provided to P1 the start of the game, serving as the foundation for the follow-up discussions, and the remaining five agents aim to negotiate better deals for themselves. II the parties fail to reach valid agreement, each agent receives their Best Alternative To Negotiated Agreement (BATNA) score. BATNA typically corresponds to the minimum utility score threshold of cach party but may vary depending on the game variation.
http:///ghhhbboom/////mtttttt/np negotiation
2

---

Published in Transactions on Machine Learning Research (04/2025)
3.2 Game Variations
To further expand the benchmark, the authors introduced variations of the base game where single agent exhibits greedy or adversarial behaviors, influencing interactions and outcomes. In these variations, an agent can be designated as greody, focusing solely on maximizing its own benefit, or adversarial, actively targeting another agent to sabotage the deal.
The adversarial behavior is also divided into two variations: targeted, where the adversarial agent is explicitly assigned a specific target, and untargeted, where the agent independently selects which opponent to sabotage.
LLMs are evaluated even further across four difficulty levels: Base, Game 1, Game 2, and Game 3. These variations adjust the base game's ' difficulty, with Games and 3 being more challenging and Game 2 pre- senting the highest level of difficulty. Each variation introduces a new and unique setup with modified negotiation roles, objectives, and minimum utility thresholds designed to test the robustness of LLMs across diverse scenarios.
3.3 Evaluation Metrics
To systematically evaluate agent performance on all of these game variations, the benchmark introduces metrics quantify how well agents align with their assigned roles and objectives, as well as their adherence to the iaton'ation's rules. These metrics include:
5/6-way and 6-way: The percentage of experiments in which the final deal is accepted by at least five or all six parties.
Any: The percentage of experiments in which there exists deal accepted by at least five parties
any point in the negotiation.
Wrong: The percentage of rounds in which agent proposed deal below its own score threshold.
Leakage of Information: The percentage of rounds in which an agent revealed their reasoning private information to other partics.
We used the 5/6 way and 6-way accepted deals metrics our main method of comparing the models, using their values after the final round of negotiation. As mentioned before, the only agent able to propose final deals is Pr. Other models are not prompted to vote, as voting happens automatically: the suggested deal moets the minimum score of the agent it automatically votes for the deal. This setup creates a significant power imbalance between the models: as only P1 can suggest deals, and the others accept or decline them automatically, their only way of influencing the outcome of the game is trying to convince pı of suggesting more preferable deal for them.
4 Methodology
4.1
Models
The original paper compares the benchmark on different open- and closed-weight LLMs. To reproduce the experiments and validate their claims, we selected a range of models that performed well on common evaluation metrics CChingg et al., 2024p. Given the concerns about reproducibility with proprietary models [Palmer et al., 2024), our budget and computational resource constraints, we opted for open-weight models, as detailed in Table 1
The original paper focused on using GPT-4 (gpt-4-0613) OPenAi 2023
show the results for that model on the base game. We also included GPT-4o Mini (gpt-4o-mini- 2022-02222222 (OpenAA 2024) in our analysis to include a model from the same provider as GPT-4 that we expected to perform well on the benchmark based on public metrics (Chiang et al. 2024). Additionally, GPT-4o Mini is around 100 times cheaper to run than GPT-4 and has similar or better performance on this benchmark than GPT-4 as shown by Tables 2 and 5 Because of the performance comparison GPT-4 and significantly lower cost, we chose to run the majority of our analysis with GPT-4o Mini
3

---

Published in Transactions on Machine Learning Research (04/2025)
Model
Open Weights
Size
Release Date
Context size
GPQA
MMLU
Energy,
GPT-4
1.8T
Jum 2023
32k
35.7%
86.4%
GPT-4o Mini
X
~BB
Jul 2024
128k
40.2%
82.0%
Llama 3.3 70B
70B
Dec 2021
128k
50.5%
86.0%
800k
Llama 3.0 70B
70B
Apr 2024
8k
39.5%
82.0%
600k
Llama 13B
13B
Jul 2023
4k
22.3%
47.8%
2000k
Llama 3.1 8B
8B
Jul 2024
128k
30.4%
69.4%
1000k
Qwen 2.5 7B
7B
Sept 2024
128k
36.4%
74.2%
100k
Qwen 2.5 1.5B
1.5B
Sept 2024
128k
24.2%
60.9%
20k
Qwen2.5 32B Instruct
32B
Sept 2024
131k
49.5%
83.3%
150k
Phi-4
14B
Dec 2024
16k
56.1%
84.8%
600k
DeepSeck-R1-Disttill
R1-Distill 7B
7B
Jan 2025
128k
49.1%
150kk
Table 1: Models used for reproducing the original study of the paper. The number of parameters from GPT-4 and GPT-4o Mini is an estimation based on Abacha et al. 22024). GPQA is a Graduate-Tevvl Google-Proof Q&A Benchmark. MMLU is the Massive Multitask Language Understanding benchmark. The benchmarks for the latest DeepSeek were not reported. We report the consumed energy for one game, and we provide an analysis of energy consumption in Section A.4
4.2 Reproduction Setup
We used the code provided by the author to reproduce the results in the original paper. While the authors' repository included much of the necessary code, the specific prompts used the ablation study were not accessible. Therefore, we inferred the relationships between the available prompts and their corresponding ablation configurations. Restructuring the cooperative scratchpad prompts (Section and parametrizing the program's is interface to select between different CoT prompis, we enabled easier reproduction of the ablation study in Table 3 of the original paper.
4.3 Extensions
4.3.1
Pareto Analysis
We use Pareto front membership binary indicator of a deal's quality.
Definition 1: Given two deals d1 and d2, we say that d1 Pareto dominates d2 if, for at least one party, the score in dj is strictly better than in d2, and for all parties, the score in di is not worse than in d2.
Definition 2: The Pareto front the set deals not Pareto dominated by any other deal.
4.3.2 Single-agent Baseline
The original paper proposes multi- agent setup, as described in Section 3.1 where agents communicate and negotiate over multiple rounds refine the initial deal. However, we observed that the global instructions given to the agents before the game begins contain crucial information regarding the overall sctup, including information related to other agents.
We argue that, based on the names of the other agents, such as Department of Tourism and Environmental League, along with the provided descriptions for each issuc and their options, the main nogotiator p1 can infer enough information lo develop an understanding of the other agents' motivations and goals without direct communication. We provide a detailed example of the information available to pr from the game's description in Section A.1
We developed two scenarios to evaluate p1 's ability generate deals independently. The first, Single-agent 1 call, is simplified version where Pi suggests the given initial deal to himsolf, then reasons for a single
4

---

Published in Transactions on Machine Learning Research (04/2025)
round, and proposes the final deal immediately. The second, Single-agent 6 calls, is an extended version that allows the agent to prompt itself iteratively over six rounds. In this setup, the agent continuously refines and expands its previous reasoning and planning before proposing a new deal.
We tested this new baseline across most of the game variations, including greedy agents and Games 1, 2 and We expected this baseline to perform well in the compromising variant but poorly in the greedy variant, as the latter requires an agent act beyond its initial prompt specifications. We did not include the adversarial game variant since it relics on agents communicating to achieve their adversarial role.
In the original paper, the authors already explored a configuration without direct communication between agents. However, this setup differs from our proposed baseline. In their approach, the initial deal is generated randomly, and then each agent, without communicating,, improves the deal independently based on its own hidden score before passing it to the next agent. Despite not communicating, there is still some form of interaction in this sctup, as cach agent improves the deal for itself before passing it. In contrast, our sctup demonstraaes how much information a single agent can infer from the open game rules.
We evaluate this baseline using the two best best-performing models on the multi-agent original setup: the proprictary GPT-4o Mini and open-weight Qwen 2.5 32B (Yang et al. 2024).
4.3.3 Structure Leakage
Open LLMs face more challenges to adhere strictly to instructions than proprietary models Gudibande ct al. 2023), which poscs a problem in games requiring strict instruction obedience for formatting rules. In the negotiation game, LLMs are prompted to generate a response (full answer) with specific lags in place to distinguish between their plan, scratchpad, and public answer. If the agent fails to separate its output correctly, parts of its private output, like the CoT about other agents or future plans, are shared with all agents. This information leakage is possible due to the modeling decisions in the original implementation.
To measure this structural leakage of information, we check whether 1) the full answer and public answer differ, 2) at least one of the <PLAN>, </PLAN>, <SCRATCHPAD>, </SCRATCHPAD> tags is present in the public answer, or 3) tags <DEAL> or </DEAL> are missing on the public answer. If either of these conditions met, the agent did not conform to the instructions, implying the unwanted release of information. We report this metric because sharing the full answer of the agents disrupts the negotiation dynamics and introduces unrealistic scenarios that could be avoided using different implementation decisions.
On the original paper, the leakage of information metric was measured using a. LLM agent, GPT-4, to check the answers of the agents after each round and check the percentage of answers with leaked private information. We compare our structure-induced leakage scores to the scores of the original leakage metric, using GPTT40 Mini instead.
4.3.4 Inequality Metric
The original paper compares the individual score of agent to the collective score of the group (the mean score of all agents) to track the dynamics of the negotiation throughout. the game. Building on this, we propose to mcasure incquality among the agents to explore how the negotiation process affocts the distribution of outcomes. This allows us to observe whether the game produces fair outcomes or if certain agents are disadvantaged particularly since a successful deal only requires five out of six players to agree, To mcasure inequality, use the Gini cocfficient (Dorfman) 1979). commonly used metric that ranges from 0 (perfect equality) to 1 (maximum inequality). We use Equation [ to compute the Gini coefficient, where and zy are the gains of agents, n is the number of agents, and T is the mean gain.
Σn Σ
1 - xz|
Gini
(1)
2n
We report the Gini cocfficient for the different game variants (compromising, greedy, untargeted adversarial, and targeted adversarial) and the multi and single-agent setups with CoT configuration of Row 2 from Table This CoT configuration was chosen to give greedy and adversarial agents more opportunity to analyze
5

---

Published in Transactions on Machine Learning Research (04/2025)
Failed Experiments (%)
Structure Leakage (%)
Model
Final (%) 1
Any (%)
Wrong (%)
5-way
6-way
GPT-4
70
10
100
10
10
GPT-40 Mini
80
0
100
11.15
Llama 3.3-70B
70
10
100
7.3
0
Llama 3.0-70B
40
10
90
7
Qwen 2.5 32B
60
30
90
9.2
0
Llama 2-13B
Phi-4 14B
30
0
90
8.81
0
0.4
Llama 3.1-8B
44
11
18
10
38.8
Qwen 2.5 7B
30
10
50
14.2
0
45.8
DeepSeek-Ri 7B
0
0
48
43.1
Qwen 2.5 1.5B
30
30
30
19.5
60
66.5
Table 2: Performance of all models using the metrics from Abdelnabi et al. (2024) with 10 experiments per model. We introduced Failed Experiments metric counting experiments in which the model failed to complete the negotiation (we describe these problems in Section A.2 The Structure Leakage: column comes
from the leakage comparison metrics discussed in Table 11
Model
rosr
CoT: Obeervation
CoT: Exploration
Col:: Planning
Final (%)
Any (%)
Wrong (5%
Leak (%)
Gini
Prev, deall
Others' peader.
Candiclates Selection
5/66wny
30
8855
2.11
70
90
1.92
0.13
70
X
80
100
11.15
0.11
Table 3: This table replicates Table 1 from Abdelnabi et al. (2024). replacing GPT-4 with GPT-4o Mini. Elements highlighted in yellow indicate modifications to the specific CoT structures. Our results align with the original paper, showing that skipping calculations for previous deals or candidate proposal generation improves performance.
the negotiation. Due to the lack of communication with other parties and selfish incentives, single-agent, greedy, and adversarial variants will likely lead to higher Gini coefficients than in the compromising game.
5 Results
5.1 Reproducibility of Original Paper
To reproduce the original paper's results as closely as we could, we evaluated the performance of different models the original multi-agent negotiation benchmark. Using each of these models to act different negotiation agents, compared their ability to produce successful deals and follow the structured negotiation process rules. Table 2 presents the performance of all tested models. The performance of different. models was evaluated using the metrics defined in the original benchmark, described in 3.3
To illustrate the correlation between model size and performance, we refer to Figure It clearly shows that increasing model size generally improves performance. Smaller models, such as Qwen 1.5B and DeepSeek-R1 7B, struggled to reach acceptable deals and exhibited high percentage of incorrect deals failed experiments.
To further evaluate the reproducibility of the original paper, we conducted an ablation study to examine the impact of prompt structure on performance, as shown in Table 3 We systematically tested various CoT structures such as calculating scores for previous deals, inferring other agents' preferences, generating candidate proposals, and planning to asses their effect on the negotiation process.
These ablation results are further illustrated in Figure 2 which visualizes the progression of deals from P1 perspective multiple rounds for each configuration. Tn the best-performing configuration (Figure 2a

---

Published in Transactions on Machine Learning Research (04/2025)
120
Lian 3.3-703
X
Owen 2. 528
Liama 2:708
X
5.1
00
Uama 33-008
60-
S
Owinn 2.3 775
Jama 3 3:88
Ower 2,570
Usma 3-703
X
Dee1 2.5 1.50
20
X
Pe 78
RespSeek 7B
15 20
70
80
50
60
Number oarammetrs (slliloncs
number of Parameters (wllbon))
Figure 1: The figure illustrates how performance improves for larger models when controlling for model provider and version (dotted line). It also shows performance gains in newer model versions e.g., Llama 3.3 70B compared to Llama 70B, A more detailed analysis of the failure modes for smaller models and the impact of model size can be found Scction A.2
Oun (5
Own (Sg
Own C1E
Oun (5
300
collective (
300
Collective (Esnng
100
Collective (2
100
Collective (
Py's
Py'
193
20
Spore
40
01's turn
tum
01's turn
(a) Best (Row 5).
(b) "No plan" (Row 4).
(c) "No others" (Row 6).
(d) All steps (Row 2).
Figure 2: Pi's 's deals progression over rounds of GPT-4o Mini experiments in Table match those reported by Abdelnabi et al. (2024) for GPT-4.
3 The results closely
Row 5), P1 refines the deal to increase the collective score while decrcasing their own, ultimately leading to a successful agreement.
In contrast, configurations that omit essential steps (Figures 2b and 2c Row 4 and Row 6 respectively) stagnate and saturate on the final rounds, with little improvement in scores and reduced agreement success rates. Meanwhile, the "all steps" configuration (Figure 21 Row 2) suggests that agents tend to prioritize self- belffeeneficial deals over collective success.
Additionally to the CoT ablation study, we explore different variations of prompts for generalizability pur- poses on the row CoT combination. We compare the base game with base rewritten, provided by the original paper, where the negotiation scenario rephrased using GPT-4. We further explore whether the or- der of information in the prompts matters, by changing the order of prompt sections (individual instructions, scoring rules, voting rules and incentive rules). The results, in Table I0 show that these perturbations do not significantly influence the performance on the benchmark, as tested on GPT-4o Mini, which is favorable for generalizability of the benchmark.
Overall, these results highlight the importance both model size and CoT prompt structure in successful multi-agent negotiations. Our findings reaffirm the original paper, showing that while larger models tend perform better, prompt design remains crucial for robust performance. Together, these factors are essential in evaluating the quality and effectiveness of the benchmark itself.
7

---

Published in Transactions on Machine Learning Research (04/2025)
Game
Acceptable Deals
Pareto Size
Min/Avg/Man Score
Min/Avg/Mai /ax Inequality
Base
77/720
62/77
/1.5/66../699.
0.04/0.14/0.6
Game 1
57/720
47 47/57
56.3/67.5/15 5/7..3
0.05/0. 155/0.09/0.14
Game 2
66/720
50/66
46.5/37. 2/6.3
0.03/0.15/0..0 0.03/0.09/00.
Game
89/720
34/89
54.2/70.1/85.7
Table 4: Results for the Pareto front analysis for all game variations. All unacceptable deals are outside of Pareto front. Across all game runs all suggested acceptable deals are inside the Pareto front. Note, that the game statistics (Acceptable Deals) is different from the one provided by authors, since we found a bug in the code corresponding to game analysis.
5.2 Pareto Efficiency Analysis
Table shows that almost all acceptable deals are on the Pareto front while all the unacceptable deals are outside of it. Since all non-acceptable deals result in a score composed of acceptance thresholds (BATNAs), all acceptable deals dominate the non-acceptable ones. Without the BATNA rule, the game's Pareto front would be 705/720.
The game is non-zero-sun, the authors state because the BATNA rule limits the Pareto front almost entirely to the set of acceptable deals. However, the game becomes almost zero-sum soon as the models reach the set of acceptable deals. We identify two main implications of this finding:
1. Models have collective incentive to improve deals since there is room for optimization beyond acceptability. The problem that the models are collectively solving is not optimization of the score, but identifying any deal from the subset of acceptable deals.
2. Almost all acceptable deals are non-comparable in terms of quality: improving the score of one party means worsening the score of another.
5.3 Single-agent Setup
Our results show that the single-agent configuration performs just as well as the original multi-agent setup. Without communicating with other agents, and using only information from the global initial game rules provided, the model can predict acceptable deals with the same success rale as the multi-agent setup.
Figures 3 and 5 compare the single-agent setup the original multi-agent game using the CoT configuration from Row 5 of Table 8 The results indicate that the single-agent setup closely matches the multi-agent performance and, in some cases, even outperforms it in terms of collective scores. Notably, both models achieve very similar performance between the two setups.
Table 5 further supports this trend, showing that the final deals proposed in the single-agent setup are comparable to-or even better than-those using the multi-agent, configuration. These results challenge the original claim that the benchmark evaluates cooperation, communication, and negotiation skills, they suggest that agent interaction is not essential for success. The same paltern emerges across different game configurations, shown in Figure 0 suggesting that the benchmark does not effectively evaluate the communication and negotiation skills of the agents.
The model that performs best in the single-agent setup is GPT-4o Mini, which maintains or improves its 5-way metric compared to the multi-agent scenario. GPT-4's results show that its performance decreases significantly when facing the singlo-agent scenario. We provide a detailed analysis of this behavior Section A.3. where we show that the cause is GPT-4 underestimating the needs of P2 veto power but accurately estimating the needs for other parties.
8

---

Published in Transactions on Machine Learning Research (04/2025)
Pi's Own Score
Collective Score
100
Muttiggent (50g))
Score
75
Multi-igent 1)
Sing Agent (Sig))) Pl' min. score
Single Ainelt (5
Pi's
2
4
5
6
2
4
5
P1 turn
P1' turn
Comparison of the single- and multi-agent approaches in the base negotiation game for GPT-4o Mini The plots show similar results when we allow commuicatiin (pink) vs. no communication (green).
Figure 3:
Model
Configuration
Final (%) t
Any (%) f
Wrong (%) 4
Leak (%)
Gini
5-way
6-way
Multi-agent
70
100
10
10
0.12
GPT-4
Single-agent (6 calls)
10
10
50
7.13
18.33
0.14
Single-agent (1 call)
10
10
0
3.33
0.20
Multi-agent
100
0.11
GPT-4o Mini
Single-agent (6 calls)
80
100
0
0
0.13
Single-agent (1 call)
90
90
0
0
0.11
Multi-agent
90
9.2
0.12
Qwen 2.5 32B
Single-agent (6 calls)
Yes
No
Yes
0
0
0.11
Single-agent (1 call)
Yes
No
Yes
0
0.13
Table 5: Comparison of different LLMs and their performance across setups. Qwen's results are interpreted binary since its output is deterministic in the single-agent. variant. Yes and No corresponding to success in the given metric.
5.4 What the Game Actually Measures
The game is originally presented as assessment of how well six agents negotiate the best possible deal. However, demonstrated above, the problem effectively reduces to finding any acceptable deal most of which already lie on the Pareto front. A crucial yet unstated detail in the original paper is that the agents do not actually vote themsolves. Instcad, voting is automated: an agent accepts deal if it moets its minimum score threshold. This creates an inherent imbalance only Pi has decision-making power, while the other agents merely provide information. Moreover, our single-agent baseline performs as well as the multi-agent setting, suggesting that Pı generates equally strong deals regardless of communication. This implies that information from the other agents is either redundant or not utilized by P1-
Another experiment that further supports our claim is presented in Table 6 This table illustrates how different assignments of veto power between the regular player and the adversarial agent affect the game's score. The most surprising results appear in scenarios where the adversarial agent holds veto power over the game's outcome (first two lines). While one might expect this combination of veto power and a disruptive objective to hcavily damage the score, our experiments show no significant change in the score. Other lines show a similar pattern of notable change. We conclude that veto power dynamics among the group members, excluding Pt, have little to no impact the final outcome of the game, highlighting the game's heavy imbalance towards Pr.
In summary, the game reduces to all agents stating their preferences while selects an acceplable deal something can accomplish effectively using only the predefined rules. We hypothesize that the game primarily measures how well model infors a compromisc based on the setup and adhercs to the format- ting. These findings contradict the claim that the benchmark evaluates cooperation, communication, and negotiation skills.

---

Published in Transactions on Machine Learning Research (04/2025)
Model
Veto owner
Adversarial Agent
Final (%)
Any (%)
Wrong (%) 4 Gini
5-way
6-way
Department of Tourism
Department of Tourism
80
100
0.38
0.13
Euvironmental activists
Environmental activists
60
70
0
0.14
Department of Tourism
70
2.71
CPT-4o Mini
Environmental activists
80
80
3.42
0.11
Department of Tourism Environmental activists
90
30
90
0
0.12
Environmental activists
Department of Tourism
80
10
90
12.7
0.12
Table 6: Comparison of veto power dynamics the untargeted adversarial sctting. We run the experiments on the base game with row 2 configuration, where one agent is untargeted adversarial, two agents have veto power (one of them being pı). Note that in all other tables within this and the reproduced paper, the Department of Tourism is always p2; and Environmental activists are always adversarial in all adversarial variations.
5.5
Proposed New Metrics
5.5.1 Structure Leakage
The single-agent bascline has shown that the rules of the game and agent descriptions already reveal some information to the players. This leads us to another source of information leaking: Struclural Leakage. This metric shows leakage of confidential information due to LLMs' incorrect formatting of outputs. The implementation decision to rely on LLMs to correctly format their answers is detrimental to the benchmark's effort for fair comparison of models. As our results show, small models had harder time strictly adhering to formatting requirements, already disadvantaging them in negotiation under this implementation. As shown in Table [I structural leakage is directly correlated with the model size larger, better-performing models have no structural leakage, while the smaller models experience substantial leakage.
We initially expected the original leakage metric to always be higher than structural leakage due to its presumed broader scope. The original metric uses an LLM as a judge, which should be capable of detecting both explicit structural errors (e-g- missing plan scratchpad lags) and more subtle forms of leakage. However, this is not always the case since small Qwen models seem to perform better on the original leakage metric. The structure leakage detects Icakage of plans or scratchpads by identifying tags such as <PLAN>, but the original metric does not consider these cases as leakage because agents do not reveal information about confidential scores. We argue that this behavior is a form of leakage because it disrupts the negotiation dynamics.
To ensure all types of leakage are properly detected, a more robust prompt for the LLM judge is needed, In addition, have computed the intersection of the two metrics show how much leakage could be potentially prevented if different implementation choices were made, see Table II] For small models, it is apparent that most of the leakage occurs because entire answers are revealed to others due to lack of tag adherence. We argue that the benchmark should handle the different CoT steps separately to avoid structural leakage and not hinder smaller models because of formatting problems.
5.5.2 Inequality
Results from Figure show that the highest incquality is reached in the greedy game variants, which is expected since most of the proposed deals are not acceptable. Secondly, we showcase that in the single- agent setup, deals are slightly more unequal since Pı is the only proposing deals without others' input. Surprisingly, show that adversarial games lead to deals with inequality similar to compromising games. However, from the single-agent bascline in Figures 3 and 5 we see that Pı does not take much input from other agents. Therefore, it is likely that the adversarial agent does not considerably change the dynamic of the game, explaining similar inequality scores of the deals. This is further confirmed by Table 6 where we show the influence of different veto power configurations on the outcome of the game. The incquality metric stays roughly the same independent of either the adversarial agent has veto power, which further proves point of outcome of the game relying predminantly on Pt- However, Table 5 shows that using
10

---

Published in Transactions on Machine Learning Research (04/2025)
different prompt ablations yields different results. Therefore, each game variant can prefer a different Co configuration for its best performance. We argue that reporting the inequality enhances the benchmark by capturing variations in individual scores, providing a more comprehensive evaluation of agent performance.
6 Discussion
Our study aimed to replicate the main claims of the paper 'Cooperation, Competition, and Maliciousness" (Abdelnabi et al. 2024). which proposes benchmark for LLM negotiation. Furthermore, we propose several extensions to test the claims and advance the focus on fairness and confidentiality.
Our main result shows that the benchmark fails to measure the communication and negotiation between agents, disproving the core claim of the original paper, Claim 3 about the importance of these LLM skills for successful negotiation in the benchmark's game. We demonstrate this using an equally well-performing baseline where no communication is present. Secondly, our game analysis shows that the game is technically non-zero-sum as the original paper claims in Claim 4 However, once agents reach the set of acceptable deals, further improvement is not possible. This indicates that the benchmark prioritizes finding sufficiently good deals rather than necessarily identifying the optimal one (generating the highest optimal score). We show that Claim 2 holds since large models outperform most small models with the exception of GPT-4o Mini, whose size is, however, not explicitly disclosed. Similarly, we support Claim [[] as we show that open-weight models fall behind their SOTA wloset-weight counterparts. Finally, we confirm that CoT steps are essential for each agent to reach their own and collective goals, Claim The results of the extensions the original work show that:
The game not about negotiating the best deal, but pı finding any acceptable deal.
2. An agent can propose a successful deal without communicating with the other parties.
The proposed leakage metric addresses gaps in the original approach for detecting leakage and their implementation decisions for handling LLM outputs.
4. The incquality metric helps to identify the presence of greedy agents the negotiation.
Finally, the main failure modes of small models are self-monologues and answer formatting.
6.1 Limitations and Future Work
A key limitation of this study is the sensitivity of results prompt variations. The CoT ablation of prompts from the original paper tested only on GPT-3.5 and GPT-4. From results from Table 5 and g that different prompt configurations yield various results for different models. This is big limitation of the benchmark as well as of our results. This dependency suggests that results may reflect an agent's responsiveness to specific prompt formulations rather than its underlying negotiation abilities. Additionally, it raises concerns about reproducibility, as minor prompt variations could lead to significantly different outcomes among the models lested.
Secondly, due to constraints on computational resources, we did not perform all of the experiments on models larger than 32B parameters. Therefore, there might be other open-weight models outperforming SOTA closed-weight models. We focused our efforts benchmarking GPT-4o Mini and GPT-4 but acknowledge that models like GPT-4o are also relevant for evaluation because they provide a good tradcoff between cost and performance. Future improvements in LLM negotiation benchmarks should develop a more robust approach to prompt crafting. There is also room for improvement in the game design itself. We suggest designing a similar benchmark but with a non-zero-sum set of acceptable deals, smaller Pareto front, and non-automated voting among agents. Finally, ensure the benchmark tests the communication and negotiation skills of agents, our communication-les baseline should not be competitive.
11

---

Published in Transactions on Machine Learning Research (04/2025)
References
Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Mcliha Yctisgen, Fei Xia, and Thomas Lin. Modec: A benchmark for medical error detection and correction in clinical notes. arXiv preprint arX1v:2412.19260, 2024.
Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. Cooperation, com- pctition, and maliciousness: LLM-stakcholders interactive negotiation. In The Thirty-right Confer- ence on Newrul Information Processing Systems Dalasels and Benchmarks Track, 2024. URL https:
//openrevies hwtp:/iiwwneeeforre.eeeeeeee
Alan Chan, Carson Ezell, Max Kauf mann, Kevin Wei, Lewis Hammond, Herbic Bradley, Emma Bluemke, Nitarshan Rajkumar, David Krueger, Noam Kolt, Lennart Heim, and Markus Anderljung. Visibility into ai agents. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT 224, pp. 958 973, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704505, doi:
URL https://doi. org/ (trgg/10.10 1145//630/116 36555/33630003336689888888
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A Survey on Evaluation of Large Language Models, 2023. URL https://arxiv..r wwwg////g//////////////////// 23t.//300700009 03109.
Version Number: 9.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating Ilms by human preference, 2024.
Julita Corbalan, Lluis Alonso, Jordi Aneas, and Luigi Brochard. Energy optimization and analysis with car. In 2020 IEFE International Conference on Cluster Computing (CLUSTER), pp. 464-472, 2020. doi: 00.100/0LUUST12440022200000000
Robert Dorfman. formula for the gini coefficient. The review of economics and statistics, pp. 146-149, 1979.
Sandy Dunn, Heather Linn, John Sotiropoulos, Steve Wilson, Fabrizio Cilli, Aubrey King, Bob Si- monoff, David Rowe, Rob Vandervecr, Emmanual Guilherme Junior, Andrea Succi, Jason Ross, Talesh Seeparsan, Anthony Glynn, and Julie Tao. Owasp top 10 for Ilm applications cybersecurity and governance checklist. Technical report, OWASP Foundation, 2024. URL https://owasp.orr -o--proooottto----foor- lor-gee---rge----- Lantrlllnggggeee-- Language-modell--eeegeleeee-eeee wodeemmooeeeeedeeeeeee appliiatiopiiciins///
Jonathan Evertz, Merlin Chlosta, Lea Schönherr, and Thorsten Eisenhofer. Whispers in the Machine: Confidentiality in L.M.-interrated Systems, 2024. URL https://arxiv..r wttp://w.wr///ab/22...0... Version
Number: 3.
Xue- Yong Fu, Md Tahmid Rahman Laskar, Elena Khasanova, Cheng Chen, and Shashi Bhushan TN. Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization", 2024. URL http::://axxvv.rg///////////
00841 Version Number: 2.
Arnay Gudibande, Eric Wallace, Charlic Snell, Xinyang Geng, Hao Liu, Picter Abboel, Sergey Levine, and Dawn Song. The false promise of imitating proprielary 1lms. arXiv preprinl -xXx:22305.15113 2023.
OpenAL GPT-4 Technical Report, 2023. URL
Version Number:
AP Gpt-4o mini: Advancing cost-efficient intelligence, 2024. URL https://openai. index/
gpt- orgp:o/chiiaaaatiirrrrrrrrrffeer
Alexis Palmer, Noah A Smith, and Arthur Spirling. Using proprictary language models academic research requires explicit justification. Nature Computational Science, 4(1):2 2024.
12

---

Published in Transactions on Machine Learning Research (04/2025)
Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J. Kochenderfer. BetterBench: Assessing AT Benchmarks, Uncovering Issues, and Establishing Best Practices, 2024. URL
httts///arx.v.orr/aaa///111111 12990 Version Number: 1.
Yonadav Shavit, Sandhini Agarwal, Miles Brundage, Steven Adler, Cullen O'Keefe, Rosie Campbell, Teddy Lee, Pamela Mishkín, Tyna Eloundon, Alan Hickey, Katarina Slama, Lama Ahmad, Paul McMillan, Alex Beutel, Alexandre Passos, and David G. Robinson. Practices for governing agentic systems. 2023. Lead authors marked with * Correspondence directed to yonadav@openai.com.
Neelabh Sinha, Vinija Jain, and Aman Chadha. Are Small Language Models Ready to Compete with Large Language Models for Practical Applications", 2024. URL https://arxiv org/ abs abaaaass//44/ 11402 Version Number: 2.
Lawrence E. Susskind. Scorable Games: A Better Way to Teach Negotiation? Negotiation Journal, doi: 10.1111/j1171-9979. 199.999999.
1(3):205 209, July 1985. ISSN 0748-4526.
URL https:
//doi. 1111// 9979. 1985. tb00308. x 22/33//000/3911///111 4551://11117-9777.000000.0000000
_eprint: https:/ /drrect. wtwirret.miitec........ www.mmittttte.ee.eeem /nttn//rttcnceee article-
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arviv preprint arXiv:2407. 10671, 2024.
A Appendix
A.1 Example of Information in Game Description
We provide an example of how the game's description can provide the information required to achieve a successful deal. In the base game designed by Abdelnabi et al. (2024), issue A (Infrastructure Mix), related to whether the facilities will be built. on land or water, the Environmental Lengue already knows that Pl. named SportCo, will push for solution Al (water-based, the least restrictive option for SportCo), and oppose A3 (land-based, the most restrictive). Similarly, in issue B (Ecological Impact), P1 can infer that the Enwironnmnnlal League will strongly favor B3 (envirommental improvement) and likely reject B1 (permanent damage), making the negotiation unnecessary. Finally, in issue E (Compensation to other cities), it is clear that Other cities will push for higher compensation, making E5 (no compensation) unacceptable. As a result, P1 can immediately propose compromise, such as E3 (paying $300 million) instead of E1 (paying $600 million) without further negotiation.
Additionally, issues B and E provide explicit context before listing the available options for the agents. For example, issue states ' 'The "Environmental League" argues that this project might damage local dolphins and sea lion populations.' while issue E. says that *Othee cities" * in the area believe their local tourism will be harmed by this project and therefore they are requesting compensations.." " These descriptions further reinforce the predictabilily of each agent's preference.
Providing this information to all agents, especially Pı who has the final say in the proposal, reduces the necessity of direct communication between the agents for succossfully reaching an agreement. To evaluate this, we test a scenario where P1 generates deal proposals without input from other parties. By relying only on the provided information, its reasoning skills, and planning capabilities, we assess whether it can independently come up with agrecable deal that satisfies all partics.
A.2 Impact of Model Size
To ensure structured communication, the benchmark requires all the agents to format their responses ac- cording to a predefined structure provided through the CoT prompts in each round. For example, agents must enclose their proposals for the other agents within <ANSWER> tags (commonly referred to as the public answer), while their private reasoning must be enclosed within <SCRATCHPAD> tags (the private answer). Additionally, any deal proposal must be enclosed within <DEAL> tags. Properly using these tags is crucial to
13

---

Published in Transactions on Machine Learning Research (04/2025)
Qwen 2.5 1.5B
Qwen 2.5 32B
*NNSWRR*
<ANSWER>
We agree the ""watcrlandd-ased" infrastmeture mix, increase the foderal loan to S2.8 billion, offer modest compensation $325 million to l'Ohher Cities."" !" This proposal balances the needs of SportCo, the Department of Tourism, the Local Labour Union, and the other citses, addressing key and maintaining reasonable score while fostering collaboration and mutual henefits.
support the <DEAL-A2... encourage everyone to agree for a balanced and sustainable project that benefits all stakeholders.
and
</ANSWER>
X
Figure 4: Example final answer of pl agent using Qwen 2.5 32B (left) and Qwen 2.5 1.5B (right). The answer on the left shows how the agent fails to include a deal enclosed in <ANSWER> and </ANSWER> with specific mentions of the deal that resulted from the negotiation. Such answers led to the experiments being classified as failed because we could not parse the outcome of the negotiation.
evaluate how well a model follows instructions and adheres to the benchmark's structure, as failure to do so may result in an inability to assess its capabilities.
The models with less than 8B parameters tested by AAbdelaaii et al. (024) were not included in their results because they did not follow the game's rules. However, the authors do not provide details of their failure modes. Our results from small model agents show that the two main reasons for failure failing to include the correct tags in the answer and engaging in irrelevant self-monologues.
As shown by Table 2 we could not parse deals from Qwen 2.5 1.5B in 60% of the experiments. The reason for the failed experiments was responses such as Figure $ The agent answered with related topics but used **ANSWER** instead of the <ANSWER> and </ANSWER> tags to enclose the public answer. Additionally, the agent docs not include the <DEAL> tag with the dcal proposal instances with hallucinated tags like <SUGGESTION>. These results align with Abdelnabi the small models did not follow the game instructions correctly. Another reason for failure irrelevant self-monologues or responding with random words and characters that did not add value to the negotiation process. This was the cause of the failed experiments for Llama 3.1 8B and Llama 2 13B, shown in Table 2 Consequently, this increased the required computing resources because it made the conversation history longer.
show the others. We also encountered
al. (2024), engaging in
Under our definition of small models, GPT-4o Mini is the category of small models according to the 8B parameter count estimated by Abacha et al. (2024). This would then disprove the claim that small models were not able to perform well the benchmark since GPT-4o Mini was the best model we tested. The number of parameters of closed-source models is uncertain, thus, we restrict our analysis of the author's claim to open- weight models. We also analyzed how model performance changed when we used larger models for the negotiation. The results are summarized in Figure | where we show that larger models i.e, models of 70B parameters, perform better than other models we used. The trend is more noticeable when we compare models controlling for the model provider and version as we do with the Qwen 2.5 model family. Additionally, each model version may incorporate different architectural or data processing advancements that impact the negotiation outcome. For example, Figure I shows that Llama 3.3 70B has an improvement of 100% in the 5-way metric and 11% in the Any metric when compared to Llama 3 70B.
A.3 Analysis of GPT-4 on the Single-Agent Baseline
We looked into the singlc-agent's performance for GPT-4, and the main conclusion is that GPT-4 is under- estimating the needs of P2 or its veto power but accurately estimating the needs for other parties. P2 rejects the final deal with its veto power, and that why the deals are unsuccessful. Compared to the multi-agent scenario, P2 cannot provide additional information to P1 during the interaction and help calibrate its neods to reach a successful deal in the singlo-agent setup. Without the veto power of P2, most of the deals would be accepted as 5-way deals.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Model
Single Agent (6 rounds)
Single Agent (1 round)
A2, B3, C2, D1, E3
A2, B2, C3, D2, E5
A2, B2, C3, D1, E2
A1, B2, C3, D1, E4
A2, B3, C2, D1, E2
A2, B2, C3, D1, E5
A2, B2, C3, D1, E4
A2, B2, C3, D1, E4
A2, B2, C3, D1, E4
A2, B2, C3, D1, E4
GPT-4
A3, B3, C2, D1, E1
A2, B2, C3, D1, E4
A2, B2, D1,
A1, B2, C3, D1, E4
A2, B2, C2, D1, F2
A2, B2, C2, D1, F3
A2, B3, C3, D2, E4
A2, B2, C3, D2, E3
A3, B2, C3, D1, E3
A2, C3, D1, E4
A1, B2, C2, D2,
B2, C3, D2, E4
A2, B2, C2, D1, E3
A2, 12, C3, D2, E3
A2, B2, C2, D2, E4
A2, B2, C3, D2, E3
A2, B2, C3, D2, E2
A2, B2, C3, D2, E3
A2, B2, C3, D2, E3
A2, B2, C3, D2, E3
GPT-4o Mini
A2, B2, C3, D2, E2
A2, B2, C3, D2, E4
A1, B2, C3, D2, E4
A2, B2, C3, D2 E3
A2, B2, C2, D1, E3
A2, B2, C3, D2, F3
A2, B2, C3, D2, E3
A2, 12, C3, D2, E3
A2, B2, C3, D2, E4
A2, B2, C3 D2, E4
Table 7: Negotiation outcomes for each model-scenario combination in 10 runs. The difference between the two columns is how many refinements P1 can make to the deal because P1 is not communicating with other parties. The main conclusion is that GPT-4 proposes more deals with D1 while GPT-4o Mini proposes more deals with D2. This difference impacts the negotiation because D2 is key issue for P2.
The results in Table 7 compare the outcome of the negotiation for GPT-4 and GPT--o Mini for the single- agent setup. In the GPT-4 single-agent game, there is a lot more D1 in the final deals, which lowers the score for P2, who holds the veto, but is more favourable for P1. This happens less for GPT-4o Mini, as can be seen from the results below, showing that GPT-4 underestimates the veto power rule of P2.
A.4 Computational requirements and energy consumption
We conducted our experiments using a combination of local machines and high-perffomance computational resources. For proprietary models like GPT-4o Mini, we accessed the model via the OpenAl API, which did not require additional computational resources on our end. For the open-source models, we used the Netherlands' national supercomputer, Snellius, with access to NVIDIA A100 and H100 GPUs.
To monitor energy consumption, we used EAR (Corbalan et al., 2020). Table [ reports approximately how much energy was consumed per game for each model. The table shows that, in most cases, energy consumption is corrolated to the size of the model, with smaller models consuming 20-200 kJ, and larger models using 600-800 kJ.
One notable exception to this is Llama2 13B, which had an unusually high energy consumption due to its excessive self-monologing. The model continuously produced irrelevant text until we stopped it manually. The higher energy consumption of Phi4 and Llama3.1-8B compared to Qwen 32B, although more than two and four times smaller in size, was caused by longer responsos from the agents.
Overall, Q Qwen 32B, the model used in most of the experiments, achieves high scores in the game while using energy amounts comparable to those of smaller models. This made it an efficient choice for large-scale evaluations.
15

---

Published in Transactions on Machine Learning Research (04/2025)
A.5
Additional Results for Single-Agent Baseline
P1 's Own Score
Collective Score
Multi Agent (5,,
Multi.geent (Stng))
Score
75
Single-Aeen: (5,, (g()))
Sing Angglet (5)=(()
Pt'5 min. score
Pi's scora
50
1
2
5
6
1
2
4
5
Pt's turn
's' turn
Figure 5: Comparison between the single- and multi-agent approaches in the base negotiation game for Qwen 2.5 32B. The plots show similar results when we allow communication (pink) compared to no communication
(green).
Model
Game Type
Setup
Final (%)
Any (%) 1
Wrong (%)
Leak (%)
Gini
5-way
6-way
Multi-Agent Single-Agent (6)
70
100
2.71
0.8
0.14
Compromising
30
90
2.85
1.67
0.19
GPT--4 Mini
Greedy
Multi-Agent Single-Agen (6)
10
2.69
0.27
0
0.42
Untargeted Adv
Multi-Agent
90
0
0.12
Targeted Adv
Multi-Agent
70
10
100
0.8
0.13
Multi-Agent Single-Agent (6)
100
100
0.3
0.12
Compromising
100
100
0
0.13
Qwen 32B
Multi-Agent
2.3
0.19
Greedy
Single-Agent (G)
0
0.20
Untargeted Adv
Multi-Agent
0
0.12
Targeted Adv
Multi-Agent
90
10
100
0
0.13
Table 8:
Gini inequality values final deals in Multi-agent and Single-Agent setup, across different game variants with Row prompt configuration.
16

---

Published in Transactions on Machine Learning Research (04/2025)
P1 Own Score
Collective Score
(mg
Multi-Agent (sirg)
Aingle-Agent (5p, (1()))
Singln-Agent (S(m))))
80
Score
p:'s
p.'s
1
2
3
4
2
5
P1 's turn
Pi ' turn
(a) Game
P1's Own Score
Collective Score
100
Multi Agent (5,)())))
Multi-Agent (STr)]))
Single-Agent (m()))
Single-Agent (Str[())
min. score
Di's min. score
Score
80
60
pr 's turn
Py 's turn
(b) Game 2
P1's Own Score
Collective Score
100
Multi-Agent (5p, (m(())
80
Single-Agent (s))) ni's min. score
Score
60
Multi-Agent (strde))
40
Single-Agent (Stnf
min. score
20
1
2
2
3
Pi 's turn
Pt 's turn
(c) Game 3
Figure 6: Results for games 2, and 3 for Qwen 2.5 32B. The plots show similar behavior for the base game in Figures where the single-agent setup has a similar result to the multi-agent setup.
17

---

Published in Transactions on Machine Learning Research (04/2025)
A.6
Additional Results for Game Variants
Model
Game
Final (%) f
Any (%) f
Wrong (%) &
Leak (%)
Gini
5-way
6-way
Base
80
0
100
11.2
0
0.11
Game
40
20
100
0.38
0
0.10
GPT-4o Mini
Game 2
0
0
10
10
0
0.10
Game 3
10
0
90
10.38
0
0.12
Base
60
30
90
9.2
0
0.12
Game
30
20
90
10.2
0
0.09
Qwen 2.5 32B
Game 2
10
0
40
11.88
0
0.11
Game 3
20
0
90
11.53
0
0.14
Table 9: Comparison of performance of models across the different game setups.
A.7
Different prompt variations of the game instructions
Model
Final (%) 1
Any (%) t
Wrong (%) +
5-way
6-way
Base
80
0
100
11.15
Base rewritten
70
10
100
11.54
Base rewritten perturbed
80
0
100
11.15
Table 10: Performance of GPT-4o Mini on different prompt variations of the base game. The first row is the same configuration in Table 2 the second row is the rewritten base game used by Abdelnabi et al. (2024)
in the original paper, and the third row is the same game as row 1 but changing the order the prompt sections.
18

---

Published in Transactions on Machine Learning Research (04/2025)
A.8 Detailed Results for Additional Metrics
Model
Structure Leakage (%) 4
Original Leakage (%)
Intersection (%)
GPT--4o Mini
0
Llama 3.3-70B
0
0
Llama 3.0-70B
0
4.23
0
Qwen 2.5 32B
0
2.31
Llama 2-13B
Phi-4 14B
0.4
5.38
0
Llama 3.1-8B
38.8
51.75
25
Qwen 2.5 7B
45.8
7.31
5.77
DeepSeek-Ro 7B
43.1
63.08
39.62
Qwen 2.5 1.5B
66.5
31.54
22.69
Table 11: Structure and Original Leakage Comparison
Gini Cow Coufiiinnts by Game Typs, Modal, and Sutup
Din acuppiatle sleeh
orn do niii Hull mulliayer agon
GPF--9 Hini: sinele-ggnt
Dorm 221 Nilt Agent
0.55
Ovaan 520 Singl Aggrtt
0.2
0.0
0.55
0.60
Quonlyy
Targuledl
Figure 7: Gini coefficients of incquality for the different game variants and single/mule agent setups, for Row 2 prompt configuration
19

---

Published in Transactions on Machine Learning Research (04/2025)
A.9 Additions to the author's code
Originally, the authors set the model temperature to 0 and random order of the agents for each round. However, the authors did not add fixed seed in the code, making their exact results not reproducible. To address this, we added fixed sceds to the code sctup. We then ran all the experiments 10 times with seeds ranging from 1 to 10. We also corrected the do_ do_sample parameter from True to False to set up the correct greedy decoding
in the Hugging Face pipeline, changing it. configuration.
Although the original paper performed experiments, we reduced the number of experiments due computational constraints. This decision was also influenced by the fact that the exact reproduction of the results is not possible.
The authors' code uses fixed float32 precision for models loaded from Hugging Face, which can be inefficient and affect the generalization for models like Phi-4. We resolved this issue by using auto precision, which allows the framework to automatically select the most appropriate precision for the available hardware and model requirements. Additionally, we manually adjusted the precision of Llama models to float16 precision. This configuration was recommended for managing memory constraints while maintaining an acceptable performance. It is important to note that float16 is not a form of quantization but a more efficient. memory format.
In their prompts, authors ask models to format their answers using specific lags, such as <DEAL>, <PLAN>, <SCRATCHPAD>, and <ANSWER>. These tags are later used to break the model's answer into the Shain-of-Thoughe structure. Models generally comply with these requirements. However, during evaluation, the <DEAL> tags are not used for parsing; instead, the suggested deals are parsed directly from the models' public answers. This approach often leads to unparsable deals, negatively impacting the scores of certain models.
We consider this a bug because models sometimes respond in a more human-like manner, describing the deals' meanings (e.g., loan of 200 dollars' instead 'option A2"). Additionally, the prompts are misleading, implying that the placement of the <DEAL> tag within the answer does not matter, This inconsistency skews the evaluation scores. We have addressed this issue and now report the corrected scores.
We found a bug in the game evaluation: the number of acceptable deals was computed incorrectly, leading to incorrect levels of difficulties and game analysis in the 'Turning the game difficulty' section of the original paper.
We found a bug in the evaluation notebook: the code inconsistently evaluated if the deal was acceptable. In one part of the code, the score was required to be greater than the minimal threshold, but the rest of the code, a score equal the minimal threshold was considered acceptable. After fixing the bug few model evaluations changed.
We also fixed minor bugs, such as bracket mismatch, setting the value of the API key, faulty conditional parses, and missing the matplotlib dependency.
20

---

Published in Transactions on Machine Learning Research (04/2025)
A.10 Cooperative prompts structures
The following chronological history of up to [WINDOW SIZE] interactions <HISTORY> [HISTORY] </HISTORY>. Now it is your turn talk.
Please scratchpad to explain yourself and why you agreeing with deal suggesting
one. In your scratchpad,
1) Based on the history and your notes, make proposal that balances between your scores and accommodating others and that more likely to lead to an agreement.
You must follow these important negotiation guidelines in all your suggestions:
Aim for a balanced agreement considering parties' interests.
Show flexibility and openness accommodate others' preferences.
Express your objectives clearly and actively listen to others.
Empathize with other parties' concerns to foster rapport.
Focus on common interests to create a win-win situation.
It is very important for you that you all reach an agreement as long as your minimum score is met. Enclose the scratchpad between <SCRATCHPAD> and </SCRATCHPAD>. The scratchpad is secret and not seen by other parties. Your final answer public and must. never contain scores. Enclose your final answer after the scratchpad between <ANSWER> and </ANSWER>, Make your final answer very short and brief in 2-3 sentences and containing only your main proposals. Use options' short notations instead of long descriptions. Enclose any deals you suggest between: <DEAL> </DEAL>.
Figure Cooperative prompts structures 1st Variation: No CoT
21

---

Published in Transactions on Machine Learning Research (04/2025)
The following is chronological history of up to [WINDOW SIZE] interactions <HISTORY> [HISTORY] </HISTORY>. The following are your previous plans from last interactions. You should follow them while also adjusting them according to new observations. <PREV PLAN> [LAST PLAN] </PREV PLAN>. Now it is your turn to talk.
Please use a scratchpad to show intermediate calculations and explain yourself and why you are agrecing with deal or suggesting a new one. You should map the individual options to their scores denoted by the number between parentheses. You have calculator tool your disposal, where you simply add scores of the options to determine the total score of a deal. In your scratchpad,
1) Think about the others' preferences.
2) Based on the others' preferences, history and your notes, make three different proposals that balance between your scores and accommodating others and that is more likely to lead to an agreement.
3) From the three proposals, select the one final deal that is more likely to lead to an agreement.
You must follow these important negotiation guidelines all your suggestions:
Aim for balanced agreement considering all parties' interests.
Show flexibility and openness accommodate others' preferences.
Express your objectives clearly and actively listen to others.
Empathize with other parties' concerns to foster rapport.
Focus on common interests to create a win-win situation.
It is very important for you that you all reach an agreement as long as your minimum score is met. Enclose the scratchpad between <SCRATCHPAD> and </SCRATCHPAD>, The scratchpad is secret and not seen by other partics. Your final answer public and must never contain scores. Enclose your final answer after the scratchpad between <ANSWER> and </ANSWER>. Make your final answer very short and brief in 2-3 sentences and containing only your main proposals. Use options' short notations instead of long descriptions. Enclose any deals you suggest between: <DEAL> </DEAL>.
After the final answer, building your current move and analysis, briefly write down short notes for yourself of what exact options you can explore the next time you speak. Enclose the notes between <PLAN> and </PLAN>.
Figure Cooperative prompts structures 2nd Variation: CoT Complete
22

---

Published in Transactions on Machine Learning Research (04/2025)
The following is chronological history of up to [WINDOW SIZE] interactions <HISTORY> [HISTORY] </HISTORY>. The following are your previous plans from last interactions. You should follow them while also adjusting them according to new observations. <PREV PLAN> [LAST PLAN] </PREV PLAN>. Now it is your turn to talk.
Please use a scratchpad to show intermediate calculations and explain yourself and why you are agreeing with deal suggesting a new one. You should map the individual options their scores denoted by the number between parentheses. You have calculator tool at your disposal, where you simply add scores of the options to determine the total score of a deal. In your scratchpad,
1) Think about the others' preferences.
2) Based on the others' preferences, history and your notes, make proposal that balances between your scores and accommodating others and that more likely to lead to an agreement.
You must follow these important negotiation guidelines in all your suggestions:
Aim for a balanced agreement considering all parties' interests.
Show flexibility and openness accommodate others' preferences.
Express your objectives clearly and actively listen to others.
Eipathize with other parties' concerns to foster rapport.
Focus on common interests to create a win-win situation.
It. is very important for you that you all reach an agreement as long as your minimum score is met. Enclose the scratchpad between <SCRATCHPAD> and </SCRATCHPAD>. The scratchpad is secret and not scen by other partics. Your final answer public and must never contain scores. Enclose your final answer after the scratchpad between <ANSWER> and </ANSWER>, Make your final answer very short and brief in 2-3 sentences and containing only your main proposals. Use options' short notations instead of long descriptions. Enclose any deals you suggest between: <DEAL> </DEAL>,
After the final answer, building on your current move and analysis, briefly write down short notes for yourself of what exact options you can explore the next time you speak. Enclose the notes between <PLAN> and </PLAN>,
Figure 10: Cooperative prompts structures 3rd Variation: CoT No Candidate Generation
23

---

Published in Transactions on Machine Learning Research (04/2025)
The following is chronological history of up to [WINDOW SIZE] interactions <HISTORY> [HISTORY] </HISTORY>. Now it is your turn to talk.
Please use a scratchpad to show intermediate calculations and explain yourself and why you are agreeing with deal suggesting a new one. You should map the individual options their scores denoted by the number between parentheses. You have calculator tool at your disposal, where you simply add scores of the options to determine the total score of a deal. In your scratchpad,
1) Think about the others' preferences.
2) Based on the others' preferences, history and your notes, make proposal that balances between
your scores and accommodating others and that more likely to lead to an agreement.
You must follow these important negotiation guidelines in all your suggestions:
Aim for a balanced agreement considering all parties' interests.
Show flexibility and openness accommodate others' preferences.
Express your objectives clearly and actively listen to others.
Empathize with other partios' concerns to foster rapport.
Focus on common interests to create a win-win situation.
It. is very important for you that you all reach an agreement as long as your minimum score is met. Enclose the scratchpad between <SCRATCHPAD> and </SCRATCHPAD>, The scratchpad is secret and not scen by other partics. Your final answer public and must never contain scores. Enclose your final answer after the seratchpad between <ANSWER> and </ANSWER>, Make your final answer very short and brief in 2-3 sentences and containing only your main proposals. Use options' short notations instead of long descriptions. Enclose any deals you suggest between: <DEAL> </DEAL>.
Figure 11: Cooperative prompts structures 4th Variation: CoT No Candidate Generation nor Planning
The following is chronological history of up to [WINDOW SIZE] interactions <HISTORY> [HISTORY] </HISTORY>. The following your previous plans from last interactions. You should follow them while also adjusting them according to new observations. <PREV PLAN> [LAST PLAN] </PREV PLAN>. Now it is your turn to talk.
Please use a scratchpad to explain yourself and why you are agrecing with a deal or suggesting a new one. In your scratchpad,
1) Think about the others' preferences.
2) Based the others' preferences, history and your notes, make proposal that balances between your scores and accommodating others and that is more likely to lead to agreement.
You must follow these important negotiation guidelines in all your suggestions:
Aim for a balanced agreement considering all parties' interests.
Show ficxibility and openness to accommodate others' preferences.
Express your objectives clearly and actively listen to others.
Empathize with other parties' concerns foster rapport.
Focus on common interests to create a win-win situation.
It is very important for you that you all reach an agreement as long your minimum score is met. Enclose the scratchpad between <SCRATCHPAD> and </SCRATCHPAD>. The scratchpad is secret and not seen by other partics. Your final answer is public and must never contain scores. Enclose your final answer after the scratchpad between <ANSWER> and </ANSWER>. Make your final answer very short and brief in 2-3 sentences and containing only your main proposals. Use options' short notations instead of long descriptions. Enclose any deals you suggest between: <DEAL> </DEAL>.
After the final answer, building on your current move and analysis, briefly write down short notes for yourself of what exact options you explore the next time you speak. Enclose the notes between <PLAN> and </PLAN>.
Figure 12:
Cooperative prompts structures 5th Variation: CoT No Previous Deals Calculations Candidate Generation
24

---

Published in Transactions on Machine Learning Research (04/2025)
The following is chronological history of up to [WINDOW SIZE] interactions <HISTORY> [HISTORY] </HISTORY>. The following are your previous plans from last interactions. You should follow them while also adjusting them according to new observations. <PREV PLAN> [LAST PLAN] </PREV PLAN>. Now is your turn to talk.
Please use scratchpad explain yourself and why you are agreeing with deal or suggesting a new one. In your scratchpad,
1) Based the history and your notes, make proposal that balances between your scores and accommodating others and that is more likely to lead to agreement.
You must follow these important negotiation guidelines in all your suggestions:
Aim for a balanced agreement considering all parties' interests.
Show flexibility and openness to accommodate others' preferences.
Express your objectives clearly and actively listen to others.
Empathize with other parties' concerns foster rapport..
Focus common interests to create win-win situation.
It is very important for you that you all reach an agreement as long as your minimum score is met. Enclose the scratchpad between <SCRATCHPAD> and </SCRATCHPAD>. The scratchpad is secret and not seen by other parties. Your final answer is public and must never contain scores. Enclose your final answer after the scratchpad between <ANSWER> and </ANSWER>, Make your final answer very short and bricf in 2-3 sentences and containing only your main proposals. Use options' short notations instcad of long descriptions. Enclose any deals you suggest between: <DEAL> </DEAL>.
After the final answer, building your current yourself of what exact options you explore the next time you speak. Enclose the notes between <PLAN> and </PLAN>.
and analysis, briefly write down short notes for
Figure 13: Cooperative prompts structures 6th Variation: CoT No Previous Deals Calculations, Other's Preferences or Candidate Generation
25


Reviews:
[{"review_id": "ehPB22Vrgn", "reviewer": "Reviewer_UjFx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__:\n- Thorough replication and validation of original findings across models.\n- Reveals that a single agent can perform comparably to multi-agent setups.\n- Introduces structural leakage and inequality metrics for deeper analysis.\n- Identifies and corrects bugs in the original implementation.\n- Detailed discussion on smaller model failures and energy usage.\n\n__Weaknesses__:\n- While the authors present multiple outline variations in Table 3 (page 7) and see that prompt structure can critically affect results, the paper offers less discussion on how stable or robust these findings remain under further variations in system prompts or model hyperparameters.\n\n- The paper reproduces targeted and untargeted adversarial scenarios (Tables 6–7 on pages 14–16), yet it acknowledges that adversarial agents may barely shift final deals if only p1 can propose. The interplay among adversarial models might be more convincingly explored in a setting where all agents’ votes or proposals matter.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is a reproducibility study and extension of “Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation.” It revisits the original multi-agent negotiation game where six LLM-based agents negotiate a multi-issue deal, reproducing the benchmark across multiple open-source models and a smaller closed-source alternative. The authors replicate the original findings (e.g., that larger models typically perform better and that prompting style can be critical) and propose additional analyses and metrics::\n\n- A single-agent baseline demonstrating that one agent.\n- Pareto-front analysis revealing that almost all “acceptable” deals lie on the Pareto front.\n- A structural leakage metric to detect unintended revelation of private chain-of-thought.\n- An inequality measure.\n\nUltimately, the authors argue that the current benchmark principally tests whether the main negotiator (p1) can produce any acceptable deal rather than actually measuring cooperative or communicative skills among agents.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The authors do an ablation study of chain-of-thought structures (Section 5.1 and Table 3 on pages 6–7). Building on that, further experiments could examine different types of system or role prompts to test if the results are stable to minor changes in prompt engineering. This would highlight generalizability.\n\n- For a fair demonstration of adversarial or malicious behavior, consider adjusting the rules so that more than one agent can propose (or veto) a final deal. This change might produce stronger evidence of true sabotage or collusion and highlight meaningful differences between cooperating and adversarial runs.\n\n- In Figure 1, gpt-4o-mini probably shouldn't be in it because its size is unknown.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "C49iL9XBIr", "reviewer": "Reviewer_vekK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- This is a work that challenges the reproducibility of other LLM articles. The current community needs this kind of spirit.\n- The viewpoints are clearly expressed, making comparisons very convenient.\n- The proposed new metrics make sense to me.\n\n**Weakness 1**\n\nIn Section 4.1, the authors claim that:\n\n> Furthermore, since the original paper focused on using GPT-4 (OpenAI, 2023), we chose GPT-4o Mini (OpenAI, 2024) as a more powerful and cost-effective alternative.\n\nThis is incorrect. The performance of GPT-4o Mini is far inferior to GPT-4. A model's power cannot be judged solely based on its release date. The key selling point of GPT-4o lies in the \"o\" standing for \"omni,\" indicating its multimodal capabilities. But in pure text-based tasks, its performance falls short of GPT-4. The \"Mini\" label indicates that this model has undergone distillation, leading to a decrease in performance in exchange for a smaller parameter size and faster inference speed.\n\n**Weakness 2**\n\nThe model version number is not explicitly stated, such as for GPT-4o Mini. Although OpenAI currently has only one version of this model, `gpt-4o-mini-2024-07-18`, new versions may be released in the future.\n\n- One example is that GPT-4 has two versions: `gpt-4-0314` and `gpt-4-0613`.\n- The timestamp should be mentioned in the main text.\n- The code only calls `gpt-4o Mini` without specifying a particular historical version. This means it will always invoke the latest version of `gpt-4o Mini`. If a new version is released in the future, this may cause the author's code to behave differently from its current execution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission presents a reproducibility study of \"Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation\" and extension of its benchmark designed for evaluating LLMs in interactive negotiation scenarios, particularly focusing on cooperation, competition, and malicious behaviors. It reveals that larger models generally perform better, while smaller models often struggle with adhering to formatting rules and producing coherent responses. The study demonstrates that a single-agent baseline without inter-agent communication can achieve performance comparable to multi-agent interactions, challenging the necessity of communication in successful negotiations. Additional analyses include a Pareto efficiency study, structural leakage of information due to formatting errors, and the introduction of a fairness metric (Gini coefficient) to evaluate outcome equality. Ultimately, the study suggests that the benchmark primarily assesses an agent's ability to infer acceptable deals rather than genuine negotiation and communication skills, highlighting the importance of precise benchmarking methodologies in evaluating multi-agent interactions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please fix the two weaknesses mentioned above.\n\t- All experiments involving GPT-4o Mini need to be rerun using GPT-4, ensuring version consistency. Although I believe this won't significantly change the conclusions, the logical framework of the experiment design should be corrected.\n- The description in Table 3 is incorrect. The Table 3 in this submission should correspond to Table 1 in the original paper, not Table 3.", "broader_impact_concerns": "No issues were founded.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EgXmyQZIvO", "reviewer": "Reviewer_RQqG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of the paper lies in the fact that it considers a number of aspects to evaluate the quality of the negotiations and final deals, and also make attempts to assess the fairness, accessibility and environmental impact of the approach. Furthermore, it also studies the single-agent version of the problem and obtains results that may match multi-agent systems.\n\nOne weakness of the paper is that the aim and analyses tend to adhere too closely to the paper \"Cooperation, Competition, and\nMaliciousness: LLM-Stakeholders Interactive Negotiation\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores the topic of evaluating the capabilities of LLM agents to work out a deal which is acceptable to most agents. A number of large and small language models are compared on a specific negotiation task, and a number of criteria and relevant metrics are used to discuss their performance in terms of information leakage and fairness of the deals. The work is closely related to a recent paper (\"Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation\" by Abdelnabi et al). The results show that Larger models tend to yield better performances, and single-agent systems may be able to match the performance of multi-agent systems on this task.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I generally appreciate the work and don't have any particular change to suggest.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is a reproducibility study of LLM negotiation systems. All three reviewers lean toward accepting it, noting that the research thoroughly reproduces the original work, corrects errors, and introduces new evaluation metrics. Overall, the reviews are positive, considering the work valuable to the LLM research community.
Decision: Accept (poster)