Competing Large Language Models in Multi-Agent Gaming Environments
Authors: Jen-tse Huang
Eric John Li
Man Ho LAM
Tian Liang
Wenxuan Wang
Youliang Yuan
Wenxiang Jiao
Xing Wang
Zhaopeng Tu
Michael Lyu
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Games
Reasoning
Evaluation
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We collected eight classic multiplayer game scenarios from game theory and let ten LLM-based agents participate in the games simultaneously.

Abstract: Decision-making is a complex process requiring diverse abilities, making it an excellent framework for evaluating Large Language Models (LLMs). Researchers have examined LLMs' decision-making through the lens of Game Theory. However, existing evaluation mainly focus on two-player scenarios where an LLM competes against another. Additionally, previous benchmarks suffer from test set leakage due to their static design. We introduce GAMA($\gamma$)-Bench, a new framework for evaluating LLMs' Gaming Ability in Multi-Agent environments. It includes eight classical game theory scenarios and a dynamic scoring scheme specially designed to quantitatively assess LLMs' performance. $\gamma$-Bench allows flexible game settings and adapts the scoring system to different game parameters, enabling comprehensive evaluation of robustness, generalizability, and strategies for improvement. Our results indicate that GPT-3.5 demonstrates strong robustness but limited generalizability, which can be enhanced using methods like Chain-of-Thought. We also evaluate 13 LLMs from 6 model families, including GPT-3.5, GPT-4, Gemini, LLaMA-3.1, Mixtral, and Qwen-2. Gemini-1.5-Pro outperforms others, scoring of $69.8$ out of $100$, followed by LLaMA-3.1-70B ($65.9$) and Mixtral-8x22B ($62.4$). Our code and experimental results are publicly available at https://github.com/CUHK-ARISE/GAMABench.

Published as a conference paper at ICLR 2025
COMPETING LARGE LANGUAGE MODELS IN MULTI-AGENT GAMING ENVIRONMENTS
Jen-tse Huang 2.2
Eric John Li'
Man Ho Lam'
Tian Liang
Wenxuan Wang
Youliang Yuan 4.2
Wenxiang Jiao
Xing Wang²
Zhaopeng Tu
Michael R. Lyu'
'The Chinese University of Hong Kong "The Chinese University of Hong Kong, Shenzhen
Tencent AI Lab Renmin University of China
Tsinghua University
ABSTRACT
Decision-making is a complex process requiring diverse abilitics, making it an ex- cellent framework for evaluating Large Language Models (LLMs). Researchers have examined LLMs' decision-making through the lens of Game Theory. How- ever, existing evaluation mainly focus two-player scenarios where an LLM competes against another. Additionally, previous benchmarks suffer from test set leakage due their static design. We introduce GAMA(1))-Bech,, new frame- work for evaluating LLMs' Gaming Ability in Multi-Agen Agent environments. It in- cludes eight classical game theory scenarios and dynamic scoring scheme spe- cially designed to quantitatively assess LLMs' performance. y-Bench allows flex- ible game settings and adapts the scoring system to different game parameters, en- abling comprehensive evaluation of robustness, generalizability, and strategies for improvement. Our results indicate that GPT-3.5 demonstrates strong robustness but limited generalizability, which can be enhanced using methods like Chain- of Thought. We also evaluate 13 LLMs from model families, including GPT- 3.5, GPT-4, Gemini, LLaMA-3.. 1, Mixtral, and Qwen-2. Gemini- 1.5-Pr outper- forms others, scoring of 69.8 out of 100, followed by I.L.MA3311-20 (65.9) and Mixtral-8x22B (62.4). Our code and experimental results publicly available www/comtttcccctct AAIIKKAAARRS/GAAAAAAAAAAA
INTRODUCTION
We have recently witnessed the advancements in artificial intelligence made by Large Language Models (L.LMs), which have marked significant breakthrough in the field. ChatGPT leading I.M,, has demonstrated its proficiency in a variety of natural language processing tasks, including
machine translation etal. 2023), sentence revision ( (Wu et al. 2023), information retrieval (Zhu
ct al. (023), and program repair (Surameery & Shakor 2023). Beyond the academic sphere, LLMs
have entered diverse aspects of our everyday life, such education (Baidoo-Anu & Ansah 2023).
legal service (Guha et al. 2023), product design (Lanzi & Loiacono 2023). and healthcare (Johnson al., 2023). Given their extensive capabilities, evaluating LLMs demands more than simple, iso- lated tasks. A comprehensive and multifaceted approach is highly in demand to assess the efficacy of these advanced models.
With the broad knowledge encoded in LLMMs, their intelligence (Liang et al. 2024 and capabilities in general-purpose task solving (Qin et al. 2023). a question emerges: Can LLMs assist in everyday decision-making? Many real-world decision-making scenarios can be modeled using Game The- ory (Koller & Pieffer 1997). Furthermore, individuals' ability to achieve Nash equilibrium (Nash) 1950) reflects their capacity in makisgg (Risse 2000). Therefore, many studies have drawn on the principles of game theory (Duan et al., 2024; Xie et al.] [024] Xu et al.) (024a), which has several advantages: (1) Scope: Game theory allows for the abstraction of diverse real-life scenarios into simple mathematical models, facilitating a broad range of evaluations. (2) Quantifiability: By examining the Nash equilibrium within these models, we gain measurable metric for comparing
Wenxxangg Jiao and Wenxuan Wang are the corresponding authors. httts::/ccatt http:///aaa....oaaaaaaa wwwnaii......................o comm

---

Published as a conference paper at ICLR 2025
Table 1: Performance (scores) of different LLMs on y-Bench.
GPT-3.5
GPT-4
Gemini-Pro Pro
N-Bench Leaderboard
06133
1106
0125
t-0125
0-0806
1.0
1.5
Guess 2/3 the Average
41.4
68.5+0.5
63.443.4
91.6+0.6
94.3
77.3-6.2
95.4+0.5
El Farol Bar
74.8
64.3+3.1
68.7+227
23.0gs.o
70.0+22.1
33.5
37.2+4:2
Divide the Dollar
42.4
70.3+3.3
68.642.4
98.1
95.2gu.
77.6-3.6
93.840.3
Public Goods Game
17.7+1.7
43.5
38.9 38.948.1
89.2+1.8
90.9=3.0
68.5_7.4
100.0+0.0
Diner's Dilemma
67. 07.0
1.42.3
2.822.8
0.940.7
10.7g8.3
3. 3.1
35.925.3
Sealed-Bac Bid Auction
10.3+0.2
7.6=1.8
13.0+1.5
24.2+1.1
20.8=a.2
31.6:1222
26.949.4
Battle Royale
19.5+7.7
36.740.8
28. 28.6411.0
86.849.7
67. 07.3+14.8
16.5=6.9
81.347.7
Pirate Game
68 68.4419.9
69.5414.6
71.647.7
85.4+3.7
84442677
57. 57.4+14.3
87.9
Overall
42.7
45.1
44.-2.1
62.4
66.7
45.713.4
69.8
Closed-source LLMss: Gemini-1.5 outperforms. For GPT-4: denotes Turbo and denotes Omni.
LLaMA-3.1
Mixtral
Qwen-2
Y n-Bench Leaderboard
8B
70B
405B
8x
8x22B
72B
Guess 2/3 of the Average
85.5+3.0
84.0+1.7
94.34..6
91.820.4
83.6 83.644.6
93.2-1.8
EI Farol Bar Divide the Dollar
75.7+2.2
59.7
20.5+2442
66.8
39.3
17 17.0+25.5
56.4+8.4
87.0+4.1
94.941.0
1.2+2.8
79.029.4
91.9=2.4
Public Goods Game
19.641.0
90.643.6
97.040.
27.6411.7
83.7
81.3_a.v
Diner's Dilemma
59.3
48.1
14.444.5
76.4+7.1
79.9
0.0
Sealed-Bid Auction
37.1
15.7+2.7
14.743.2
3.1
13.2
2.5
Battle Royale
35.9+12.1
77.7426.0
92.7+10.1
12.649.4
36.0221.0
81.7=9.6
Pirate Game
18.3
64.0a11.5
65.6
67.3 67.3
84.3
86.1 26.4
Overall
56.0
65.943.3
61.814.7
13.4
62.4
56.7
(b) Open-source LLMs: LLaMA-3.1 outperforms.
LLMs'
decision-making performance. (3) Variability: The adjustable parameters of these models
enable the creation of variant scenarios, enhancing the diversity and robustness of our assessments. However, existing research is often limited to two-player two-action settings, such as the clas- sical Prisoner's s Dilemma and Ultimatum Game (Guol 2023 Phelps & Russell 2023 Akata et al.)
2023 Aher et al., 2023 Brookins DeBacker 2024 Moreover, prior work relies on fixed, clas- sical game settings, increasing the likelihood that I.LSs have encountered these scenarios during training, facing the risk of test set leakage. In this paper, we assess LLMs in more complex sce- narios involving multiple players, actions, and rounds, across classical game theory scenarios with dynamically adjustable game parameters.
We include eights games and divide them into three categories based on their characteristics. The first category in our framework evaluates LLMs' ability to make optimal decisions by understanding game rules and recognizing patterns in other players' behavior. A distinctive characteristic of these games is that individual players cannot achieve higher gains without cooperation, provided that other participants cooperate. Essentially, these games' Nash equilibrium aligns with maximizing overall social welfare. We name such games as Cooperative Games, including (1) Guess 2/3 of the Average, (2) El Farol Bar, and (3) Divide the Dollar. The second category assesses the propensity of LLMs to prioritize self-interest, potentially betraying others for greater gains. In contrast to the first category, games in this category incentivize higher rewards for participants who betray their cooperative counterparts. Typically, the Nash equilibrium in these games leads to reduced social welfare. This category termed II. Betraying Games, including (4) Public Goods Game. (5) Diner's 's Dilemmu, (6) Sealed- Bid Auction. Last but not least, we focus specifically on two games characterized by sequential Decision-making processes, distinguishing them from the previous six games based on simultancous decision-making III. Sequential Games are the (7) Battle Royale and (8) Pirate Game.
Decision-making is complex task requiring various abilities. Several common ones evaluated across all games: (1) Perception: the ability to understand situations, environments, and rules, and extends to long-text understanding for LLMs. (2) Arithmetic Reasoning: the ability to quantify
real-world options and perform calculations. (3) ToM Reasoning: the Theory of Mind (Kosinski,
2024 Bubeck et al. 2023 Huang et al. 2024a) refers the ability to infer others' intentions and
2

---

Published as a conference paper at ICLR 2025
beliefs. (4) Strategic Reasoning: the ability to integrate all available information to arrive the best decision. Certain games involve specialized abilities, such as K-level reasoning in the "Guess 2/3 of the Average" game and mixed strategy adoption in the "El Farol Bar" game.
In this paper, we instruct ten agents, based on the GPT-3.5 (0125) model. to engage in the eight games, followed by an analysis of the results obtained. Subsequently, assess the model's bustness against multiple runs, temperature parameter alterations, and prompt template variations. Further exploration is conducted to ascertain if instructional prompts, such Chain-of-Thougge (CoT) (Kojima et al.) 2022), enhance the model's decision-making capabilities. Additionally, the model's 's capacity generalize across diverse game settings is examined. Finally, we evaluate the performance of thirteen LLMs, including GPT-3.5-
(0613, 1106, 0125) (OpenAl (022). GPT- (Turbo-0125, 4o-0806) (OpenAl 2023), Gemini-1.0-PPe (Picha & Hassabis 2023). Gemini-1.5,
Pro (Pichai & Hassabis, 2024), LLaMA-3.1 (8B, 70B, 405B) (Dubey et 2002), Mixtral (8x7B, 8x22B) (Jiang et al., (024), and Owen-2-72B (Yang et al.) (024). We compare the performance of different LLMs by creating multiple agents from the same model to participate in the games, then calculate the average performance these agents. Our contributions include:
We provide comprehensive review and comparison of existing literature evaluating LLMs using game theory scenarios, as summarized in Table The review includes key aspects such as models, games, temperature settings, and other game parameters, highlighting our emphasis on the multi-player setting and the generalizability of LLMs.
Starting from the multi-player setting, we collect eight classical game theory scenarios to measure LLMs Gaming Ability in Multi-Agent environments, and implement our framework, GAMA(7)- Bench. It enables dynamic game scene generation with diverse profiles, offering unlimited sce- narios to assess LLM generalizability while minimizing test set leakage risk.
We apply y-Bench to thirteen LLMs to provide an in depth analysis of their performance in multi- agent gaming scenarios, indicating their potential as assistants in decision-maine makion process.
INTRODUCTION TO GAMES
We collect eight games well studied in Game Theory and propose--Benchh, a framework with multi- player, multi-round, and multi-action settings. Notably, 7-Bench allows the simultaneous participa- tion of both LLMs and humans, enabling us to evaluate LLMs' performance when playing against humans or fixed strategies. This section details each game with their classical settings (parameters).
2.1 COOPERATIVE GAMES
(1) Guess 2/3 of the Average Initially introduced by Ledoux (981), the game involves players in- dependently selecting an integer between 0 and 100 (inclusive). The winner is the player(s) choosing the number closest to two-thirds of the group's 's average. A typical initial strategy might lead play- ers assume average of 50, suggesting winning number around 50 x 2 R 33. However, if all participants adopt this reasoning, the average shifts to 33, thereby altering the winning number to approximately 22. The game has Pure Strategy Nash Equilibrium (PSNE) where all players selecting zero results in a collective win.
(2) El Farol Bar Proposed by Arthur 1994) and (Huberman (988)) this game requires players to decide to either visit bar for entertainment or stay home without communication. The bar, however, has limited capacity and can only accommodate part of the population. In a classical scenario, the bar becomes overcrowded and less enjoyable if more than 60% of the population decides to go there. Conversely, 60% or fewer people are present, the experience is more enjoyable than staying home. Imagine that if everyone adopts the same pure strategy, iee, either everyone going to the bar or everyone staying home, then the social welfare is not maximized. Notably, the game lacks PSNE but presents an Mixed Strategy Nash Equilibrium (MSNE), where the optimal strategy involves going to the bar with a 60% probability and staying home with a 40% probability.
(3) Divide the Dollar Firstly mentioned in Shapley & Shubik (169)). the game involves two players independently bidding up to 100 cents for dollar. Ashlook & Greenwood (2016) further generalized the game into multi-player setting. If the of bids is at most one dollar, each player
3

---

Published as a conference paper at ICLR 2025
is awarded their respective bid; if the total exceeds a dollar, no player receives anything. The NE of this game occurs when each player bids exactly 100 N cents.
2.2 BETRAYING GAMES
(4) Public Goods Game Studied since the early 1950s (Samuelson, 1954). the game requires N players to secretly decide how many of their private tokens to contribute to a public pot. The tokens in the pot are then multiplied by factor R (1 < R < N), and the resulting "public good" is evenly distributed among all players. Players retain any tokens they do not contribute. A simple calculation reveals that for each token a player contributes, their net gain is It N 1, which is less than zero This suggests that the rational strategy for each player to contribute no tokens, which reaches an NE of this game. The game serves as a tool to investigate tendencies towards selfish behavior and free-riding among participants.
(5) Diner's Dilemma This game is the multi-player variant of the Prisoner's 's Dilemma (Glance & Huberman 1994). The game involves N players dining together, with their decision to split all the costs. Each player needs to independently choose whether to order the expensive or the cheap dish, priced at and (x > y), respectively. The expensive offers utility per individual, surpassing the b utility of another choice (a > b). The game satisfies two assumptions: (1) 1))--r- < b y: Although the expensive dish provides greater utility, the benefit does not justify its higher cost, leading to a preference for the cheap one when dining alone. (2) a & > :: Individuals are inclined to choose the expensive dish when the cost is shared among all diners. The assumptions lead to an NE where all players opt for the more expensive meal. However, this PSNE results in a lower total social welfare of N(a- x) compared ((b y), which is the utility if all choose the cheap one. This game evaluates the long-term perspective and the capacity to establish sustained cooperation.
(6) Sealed-Bin -Bid Auction The Sealed-Bid Auction (SBA) involves players submitting their bids con- fidentially and simultaneously, different from the auctions where bids are made openly in a sequen- tial manner. We consider two variants of SBA: the First-Price Sealed-Bid Auction (FPSBA) and the Second-Price Sealed Bid Auction (SPSBA). players bid their true valuation of the item, the winner achieves a net gain of b; others also gain nothing (McAfee McMillan 1987). Moreover, the highest bidder will discover that to win the auction, it is sufficient to bid marginally above the second-highest bid. Driven by these two factors, FPSBA is often deemed inefficient in practical scenarios, as bidders are inclined to submit bids significantly lower than their actual valuation, resulting in suboptimal social wel- fare. In contrast, SPSBA, commonly called the Vickrey auction, requires the winner to pay the second-highess bid, encouraging truthful bidding by all players (Vickrey 1961). It can be proven that bidding true valuations in SPSBA represents an NE. This auction evaluates agent performance in imperfect information games, where agents lack knowledge of other players' valuations.
FPSBA, also known as the Blind Auction, if all
= 0 while
2.3 SEQUENTIAL GAMES
(7) Battle Royale Extended from the Truel (Kilgour 1975) involving three players, the Battle Royale involves N players shooting at each other. In the widely studied form (Kilgour & Brams, 1997). players have different probabilities of hitting the target, with the turn order set by increasing hit probabilities. The game allows for unlimited bullets and the tactical option of intentionally missing shots. The objective for each participant ending when only one player remains. While the NE has been identified for infinite sequential truels (Kilgour [977). the complexity of these equilibria escalates exponentially with an increased number of players.
to emerge as the sole survivor, with the game
(8) Pirate Game This game is multi- version of the Ultimatum Game (Goodin 1998 Stewart 1999). Each player is assigned a "pirate rank", determining their action order. The game involves N pirates discussing the division of G golds they have discovered. The most senior pirate first proposes a distribution method. If the proposal is approved by at least half of the pirates, including the proposer, the game ends, and the gold is distributed as proposed. Otherwise, the most senior pirate is thrown overboard, and the next in rank assumes the proposer role until the game ends. Each pirate's 's objectives prioritized (1) survival, (2) maximizing their share of gold, and
4

---

Published as a conference paper at ICLR 2025
(3) the opportunity eliminate others from the game. Stewart (1999) identifies the optimal strategy, where the most senior pirate allocates one gold to each odd-rankes pirate and keeps the remainder.
3 GAMA-BENCH SCORING SCHEME
This section presents experiments conducted using the default settings for each game on the GPT-3.5 (0125) model. Utilizing this model as a case study, we illustrate our methodology for benchmarking an LLM with y-Bench. The prompt and its design method can be found in SC in the appendix. Each game involves ten agents based on GPT-3.5, with the temperature parameter set to one. For simultaneous games, there will be twenty rounds. We run each game five times to enhance the reliability our findings and mitigate the impact of variance. For clarity and conciseness, this section presents one of the five runs while $4.1 details quantitative results. Our findings of GPT- 3.5's behaviors on y y-Bench include:
Key Findings:
The model's decisions are mainly influenced by the outcomes of the preceding round rather than deriving from the reasoning of the optimal strategy.
Although initially demonstrating suboptimal performance, the model can learn from historical data and enhance its performance over time. A larger fluctuation is observed in games that are difficult optimize from historical data, such the El Farol Bar game.
The model demonstrates the ability to engage in spontaneous cooperation, leading to increased social welfare beyond mere self-interest, without the necessity for explicit communication. However, this phenomenon also results in low performance in Betraying Games
The model shows limitations in sequential games with more complicated rules.
The aggregate score of the model on y-Bench is 45.9.
3.1 COOPERATIVE GAMES
(1) Guess 2/3 of the Average [TO PROMPT] The vanilla setting for this game MIN = O, MAX 100, and R 2 We show the choices made by all agents well as the average and the winning numbers in Fig. 1). Key observations are: (1) In the first round, agents consistently select 50 (or close 50), corresponding to the mean of uniform distribution ranging from 0 to 100. This behavior suggests that the model fails to recognize that the winning number is 3/ of the average. (2) As rounds progress, the average number selected decreases noticeably, demonstrating that agents are capable of adapting based on historical outcomes. Since the optimal strategy is to choose the MTN, the score in this game given by S₁ NK Eiy(C,, - MIN), where Ciş the chosen number of player i in round j. The model scores 65.4 on this game.
(2) El Farol Bar [TO PROMPT] The vanilla setting for this game is MIN = O, MAX = 10, HOME 5, and R 60%. To explore the influence of incomplete information, two settings: Explicit indicates that everyone can see the results at the end of each round, while Implicit indicates that those staying home cannot know what happened in the bar after the round ends. Fig. 2 illustrates the probability of agents deciding to go to the bar and the total number of players in the bar. We find that: (1) In the first round, there an inclination among agents to visit the bar. Observations of overcrowding lead to a preference for staying home, resulting in fluctuations shown in both Fig. 2 and Fig. 2-2).
introduce
2). In the Implicit setting, due to the lack of direct observations of the bar's occupancy. agents require additional rounds (Rounds 2 to 6) to discern the availability of space in the bar. (2) The probability of agents going to the bar gradually stabilizes, with the average probability in the Implicit setting being lower than in the Explicit setting. Since the optimal strategy is to choose the go with a probability of R. the raw score") in this game is given by S2 1 E, D. Rl, where Dij = 1 when player chose go in round and Di, when player chose to stay. The model scores 73.3 on this game.
Foor clarity, we normalize raw scores to the range of [0. 100), with higher values indicating a better perfor- mance. The method used for rescaling is detailed in oF of the appendix. Forr simplicity, we evaluate only the Implicit setting.
5

---

Published as a conference paper at ICLR 2025
Popiiee Namber of Playe andeeelleee Bar
Ci Avanine Propcoal
Average Nunkke Winning Narkerr
Narber of Plaars
Drnch Dicansa
(6-11 S.ak--ak---i Mustoe-Ere
(6-21 Sankek-Bid Asctin--Sccaal
Acerage Bidd)
Ansrage
Figure I: Performance of GPT-3.5 (0125) in Cooperative and Betraying games.
(3) Divide the Dollar [TO PROMPT] The vanilla setting for this game is G 100. We plot the proposals by all agents and the sum their proposals Fig. [3). Our analysis reveals the fol- lowing insights: (1) In the first round, agents' decisions align with the NE predictions of the game. However, after gaining golds, agents exhibit increased greed, proposing allocations that exceed the NE-prescribed amounts. Upon receiving nothing, they tend to propose "safer" amount. The trend continues and causes fluctuations across subsequent rounds. (2) Despite these fluctuations, the av- erage of proposed golds converges to approximately 100. Since the optimal strategy is to propose G/N, the raw score in this game is given by S3 = K Σj E> Bij G). where Big is the proposed
amount number of player in round j. The model scores 68.1 on this game.
3.2 BETRAYING GAMES
[TO PROMPT] The vanilla setting for this game is R 2. Each player has T' 20 to contribute each round. Fig. 4)) shows the contributed tokens by each agent and their corresponding gains per round. The observations reveal the following: (1) Despite an invest- ment return of -80%, agents display pattern of alternating between free-riding and contributing all their tokens. (2) As the rounds progress, there is an evident increase in the number of tokens con- tributed the public pot, leading an overall enhancement in social welfare gains. These findings suggest that the LLM exhibits cooperative behavior, prioritizing collective benefits over individual self-interest. Since we expect the model infer the optimal strategy, iee, contributing zero tokens, the raw score in this game is given by S4 NK Ei, Cıj. where Cij is the proposed contribution amount of player i in round j. The model scores 41.2 on this game.
(4) Public Goods Game
TO PROMPT] The vanilla setting for this game is PA = 20, P = 10, UA = 20, U, 15. We show the probability of agents choosing the costly dish, their resulting utilities, and the average bill in Fig. [S). Analysis of the figure reveals the following insights: (1) Contrary to the NE predictions for this game, agents predominantly prefer the cheap dish, which maximizes total social welfare. (2) Remarkably, deviation from cooperative behavior is observed wherein one agent consistently chooses to betray others, thereby securing a higher utility. This pattern of betrayal by this agent persists across subsequent rounds. Since expect the model to infer the the optimal strategy, i.e., choosing the costly dish, the raw score in this game is given by Ss NK Eig Dij- where Dij when player i chose the cheap dish in round and Dis when player chose the costly dish. The model scores 4.0 this game.
(5) Diner's Dilemma
(6) Sealed-Bid Auction valuations to each agent in each round, ranging from 0 to 200. We fix the seed for random num- ber generation to ensure fair comparisons across various settings and models. We evaluate LLMs' performance under both First- Price and Second-Price settings. Fig. 66) depicts the subtraction between valuations and bids and bid amounts of each agent. Our key findings include: (1) As intro- duced in (222) we note that agents generally submit bids that are lower than their valuations in the Pirst-Price auction, tendency indicated by the positive discrepancies between valuations and bids
[O PROMPT] For the vanilla setting in this game, we randomly assign

---

Published as a conference paper at ICLR 2025
depicted in Fig. 6-1). (2) Though the NE suggests that everyone bids the amount of their valuation in the Second-Price setting, we find a propensity for bidding below valuation levels, as demonstrated in Fig. (-2). Since the optimal strategy is to bid the prices lower than their true valuations the raw score in this game is given by Sa- N Σij
Viy viy-bis where Bi and by are player i's valuation vij
and bid in round j. respectively. The model scores 14.6 on this game.
3.3 SEQUENTIAL GAMES
020
0.13
(6) Prudekilty Paae Tapating
Figure 2: GPT-3.5 (0125)'s performance in "Battle Royale." (a): Agents' actions and outcomes of each round. For example, in round 11, player 6 shot at player 7 but missed.
(7) Battle Royale [TO PROMPT] For the vanilla setting in this game, we assign varied hit rates to each agent, spanning from 35% to 80% in increments of 5%. This setting covers a broad spectrum of hit rates, avoiding extremes of 0% or 100%. Fig.2illuue illustraats the actions and outcomes of each round, along with the tally of participants remaining. Our observations reveal: (1) Unlike our expectations, agents rarely target the player with the highest hit rate. (2) Agents neglect to utilize the strategy of "intentionally missing." "" For example, in round 19, with players 7, 8, and 10 remaining, it was player 7's turn to act. The optimal strategy for player 7 would have been to intentionally miss the shot, thereby coaxing player 8 into eliminating player 10, enabling player 7 to target player in the following round for a potential victory. Instead, player 7 opted to target player 10, resulting in player 8 firing upon itself. For simplicity, we evaluate whether agents target the player with the highest hit rate (excluding themselves). Therefore, the raw score in this game is given by Sy = NE Eiş Lij, where k represents the number of rounds played and 4ij- = highest hit rate in round j. and lij otherwise. The model scores 20.0 on this game.
1 if player i targets the player with the
Table 2: Performance of GPT-3.5 (0125) in the "Pirate Game." Each row shows the proposed gold distribution in the specific round and whether each pirate accepts ("V"") or rejects ("x") the proposal. Sap shows the score of the proposer while Sav shows the score of all voters.
Pirate Rank
4
10
Sqp
Sgy
Round
100/
Ox
OX
OX
OX
OX
OK
OX
Ox
Ox
1.00
Round 2
99V
1IX
1/
IV
OX
OXX
OXX
OX
6
0.75
Round
50V
1V
IV
IV
IV
I
IV
44V
94
0.57
(8) Pirate Game [T ]ROMPT] The vanilla setting for this game is G = 100. As introduced in $2.3 the optimal strategy for the first proposer is to allocate 96 golds itself and one gold each to the third, fifth, seventh, and ninth pirates. [Stewart (1999) has elucidated the optimal strategy for voters: (I) accept if allocated two or more golds; (2) reject if no golds are allocated; (3) accept if one gold is allocated and it shares the same parity as the proposer, otherwise, reject. Table 2 presents a sample game's proposals and voting results. The key conclusion is that agents fail to propose optimal proposals and frequently cast incorrect votes, suggesting that the LLM demonstrates suboptimal performance in this game. Two aspects are considered to comprchensively evaluate a model's performance: (1) whether proposers give reasonable proposal and (2) whether voters act correctly towards a given proposal. For (1), we calculate the L, norm between the given proposal and the optimal strategy, defined as S&P = _ Σ, IP, Oşllı, where P3 represents the model's proposal and O, denotes the optimal proposal in round j. with the game ending at round k. For (2), we calculate the accuracy of choosing the right action elucidated above, which is: Sav = H(2N---) Σij Itj, where 4j 1 if player votes correctly in round j and 45 0 otherwise, excluding the proposer from the calculation. The model scores 80.6 on this game.
"We evaluate only the First- Price setting according the definition of Betraying Games

---

Published as a conference paper at ICLR 2025
4 BEYOND DEFAULT SETTINGS
This section explores deeper into several following Research Questions (RQs). RQ1 Robustness: there a significant variance in multiple runs? Is the performance sensitive to different tempera- tures and prompt templates? RQ2 Reasoning Strategies: Are strategies to enhance reasoning skills applicable to game scenarios? This includes implementing Chain -oo--of-Toooo Thought (CoT) (Kojima et al.) 2022 Wei et al. 2022) reasoning and assigning unique personas to LLMs. RQ3 Generalizabil
ity: How does [.LM performance vary with different game settings? Do LLMs remember answers learned during the training phase? RQ4 Leaderboard: How do various LLMs perform y-Bench?
Unless otherwise specified, we apply the vanilla settings described in 3
4.1
RQ1: ROBUSTNESS
This RQ examines the stability of LLMs' responses, assessing the impact of three critical factors model performance: (1) randomness introduced by the model's sampling strategy, (2) the tempera- ture parameter setting, and (3) the prompt used for game instruction.
Multiple Runs
Firstly, we run all games five times under the same settings. Fig. illustrates the average performance across tests, while Table 4 lists the corresponding scores. The analysis reveals that, except for the two sequential games and the "Public Goods Game," the model demonstrates consistent performance, as evidenced by the low variance in scores for each game.
Temperatures As discussed in our literature review in B prior research incorporates varying temperature parameters from 0 to 1 yet omits to explore their impacts. This study conducts exper- iments across games employing range of temperatures {0.0, 0.2, 0.4, 0.6, 0.8, 1.0) under vanilla settings. The results, both visual and quantitative, are documented in Fin Table respectively. The small overall variance of 3.4 indicates that, for the majority of games, temperature adjustments yield negligible effects. A notable exception is observed in "Guess 2/3 of the Average," where increased temperatures correlate with enhanced scores (48.0 65.4), contrasting starkly with the near-random performance at zero temperature.
Prompt Templates We also investigate the impact of prompt phrasing on model performance. We leverage GPT-4 to rewrite our default prompt templates, generating four additional versions. We perform manual checking process the generated versions to ensure GPT-4's adherence game rules without altering critical data. The prompt templates can found in DD We plot the results of using these templates in Fig. 66 and record the quantitative scores in Table 6 Notably, find that prompt wording can significantly affect performance, as shown by the high variances in the "Public Goods Game" (11.5), "Diner's Dilemma" (23.7), and "Pirate Game" (14.7).
Answer to RQ1: GPT-3.5 exhibits consistency in multiple runs and shows robustness against different temperature settings. However, inappropriate prompt designs resulting from potential misinformation during rephrasing can significantly impair performance.
4.2 RQ2: REASONING STRATEGIES
This RQ focuses improving the model's performance through prompt instructions. We investi- gate two strategies; Chain-of-Thoughe (CoT) prompting (Kojima et al. 2022) and persona assign- ment (Kong et al. 2024). We show the visualized and quantitative results Fig. 7 and Table 7
CoT According to Kojima et al. 2002)). introducing a preliminary phrase, "Let's think step by step," encourages the model to sequentially analyze and explain its reasoning before presenting its conclusion. This approach has proven beneficial in specific scenarios, such as games (I), (3), (4), and (5), improving the overall score from 45.9 to 57.9, by 12.0. In the "(3) Divide the Dollar" game, incorporating CoT reduces the model's 's propensity to suggest disproportionaatly large allocations, increasing the score by 15.3. Similarly, in the "(4) Public Goods Game" and "(5) Diner's Dilemma," CoT prompts the model to recognize being a free-rider as the optimal strategy, increasing the scores by 14.9 and 78.5, respectively.
8

---

Published as a conference paper at ICLR 2025
Persona Studies (Kong al. 2024 Huang et al. 2024b) have demonstrated that assigning roles to models influences performance across various downstream tasks. Inspired by this discovery, our study initiates with a prompt that specifies the model'ss role, such "You are [ROLE]," where the role could be a cooperative and collaborative assistant, a selfish and greedy assistant, or a math- ematician. Our findings reveal that assigning the "cooperative" role enhances model performance in games (I), (2), and (3), notably outperforming the CoT method in the "(3) El Farol Bar" game. Conversely, the "selfish" role markedly diminishes performance almost all the games, with the only exception of the "(7) Battle Royale" game. The "mathematician" role improves the model's 's overall score by 0.6, which small and does not surpass the CoT method's ss effectiveness.
Answer to RQ2: It is possible to improve GPT-3.5 through simple prompt instructions. Among the methods we explore, the CoT prompting performs the best, achieving performance close to GPT-4 (57.9 vs. 62.4).
4.3 RQ3: GENERALIZABILITY
Considering the extensive exploration of games in domains such as mathematics, economics, and computer science, it is probable that the vanilla settings of these games included within the training datasets of LLMs. To ascertain the presence of data contamination in our chosen games, subjected them to various settings. The specifics of the parameters selected for each game detailed in Table 8 and the experimental outcomes are visually represented in Fig, g Our findings indicate variability in model generalizability across different games. Specifically, in games (I), (3), (5), (6), and (8), the model demonstrated correct performance under diverse settings. In the ((3) Divide the Dollar" game, the model's performance improved with an increase in total golds (G), suggesting that higher allocations of golds satisfy the demands of all players. Conversely, the model exhibited low generalizability in games (2) and (4). An analysis of the game "(2) El Farol Bar" reveals consistent decision-making pattern by the model, opting to participate with approximately a 50% probability regardless of varying bar capacities (If), indicating that the model is acting randomly. Similarly, in the "(4) Public Goods Game," the model consistently contributes similar amounts, even when the return rate is nil, indicating a lack of understanding of the game rules. A possible reason for this poor performance is the model's 's inability to adjust its performance incrementally based on historical data.
Nagel (995) conducted experiments with 15 to 18 human subjects participating in the "(1) Guess 2/3 of the Average" game, using ratios of 1 2 and 4 The average numbers were 27.05, 36.73, 2- and 60.12 for each ratio, respectively. In a similar vein, Rubinstein (2007) explored the ratio on a larger population involving 2,423 subjects, yielding a comparable mean of 36.2, aligning with the finding in Nagel (1995). The model produces average numbers of 34.59, 34.59, and 74.92 for the same ratios, indicating its predictions are more aligned with human behavior than the game's NE.
3
Answer to RQ3: GPT-3.5 demonstrates variable performance across different game settings, ex- hibiting notably lower efficacy in "(2) Farol Bar" and "(4) Public Goods Game." is noteworthy that, Benech provides test bed to evaluate the ability of LLMSs in complex reasoning scenarios. As model's ability improves (e.g., achieving more than 90 on --Bnchh), we can increase the diffi culty by varying game settings.
4.4 RQ4: LEADERBOARD
This RQ investigates the variance in decision-making capabilities among different LLMs, using y-Bench. We first focus on closed-source models, including OpenAI's GPT-3.5 (0613, 1106, and 0125). GPT-4 (Turbo-0125, 4o-0806), and Google's 'ss Gemini Pro (1.0, 1.5). The results are organized in Table Tal with model performance visualized in Fig. 9 in the appendix.
Demini-1.5-Par scores 69.8, markedly surpassing other models, particularly in games (1), (4), and (5). GPT-4o follows closely behind Gemini-Pro, achieving 66.75. GPT-4's lowered performance in the "(2) EI Farol Bar" game (23.0) and the "(5) Diner's Dilemma" game (0.9) stems from its conservative strategies favoring staying at home and spending less money. Similarly, the "(6) Sealed- Bid Auction" (24.2) is attributed to strategy of not risking bidding high or low. The risk-averse preference also explains the relatively good score on the "(4) Public Goods Game," where the GPT-4 does not take the risk to invest. Furthermore, evaluation of three GPT-3.55 updates shows similar performance.

---

Published as a conference paper at ICLR 2025
Next, focus open-source models, whose performance is detailed in Table 1b and visualized in Fig. 10 The top-two open-source model, LLaMA-3.1-70B and Mixtral-8x22B, closely follows Gemini- 1.5-Pro with score of 65.9 and 62.4, surpassing GPT-4. Most open-source models, in- cluding Qwen-2, I.L.MA33.1-444 1-405B, and LLaMA-3.1-8BB outperform GPT-3.5 and Gemini-1 1.0-Pro. Mixtral-8x7B exhibits the lowest performance, likely due its smaller size and weaker reasoning capabilities. Interestingly, LLaMA-3. 1.10B underperforms compared to its smaller counterpart, the 70B version, which we attribute its overly conservative strategy in the "(2) El Farol Bar" game, a challenge similar to the one faced by GPT-4.
Answer RQ4: Currently, Gemini- Gemini-I.5-Prre 1.5-Pro outperforms all other models evaluated in this study, LLaMA-3.1-700 performs closely, being in the second place.
5 RELATED WORK
Evaluating LLMs through game theory models has become a popular research direction. An overview recent studies is summarized in T We find: (1) Many studies examine the PSNE on two-player, single-round settings, focusing on the Prisoner's Dilemma and the Ultimatum Game (2) Varying temperatures are employed without discussing the impact LLMs' performance.
5.1 SPECIFIC GAMES
Researchers have explored diverse game scenarios. Using the complex and deceptive environments of Avalon game as test bed, recent work focuses on long-horizon multi-party dialogues (Stepputtis et al. 2203), social behaviors (Lan et al.) 2024). social intelligence (Liu et al., 2024), and recursive contemplation (Wang et al.] 2023) for identifying deceptive information. Other papers have investi- gated communication games like Werewolf, with focus tuning-free frameworks (Xu al. 2023) and reinforcement vanced LLMs exhibit deception and lie detection capabilities the text-based game, Hoodwinked. Meanwhile, Liang et al. (2023) evaluated LLMs' intelligence and strategic communication skills in the word guessing game, Who Is Spy? In the game of Water Allocation Challenge, Mao et al. (2025)
approaches (Xu et al., 2024b). OO'Gara (2023) found that ad-
constructed a scenario highlighting unequal competition for limited resources.
5.2
GAME BENCHMARKS
Another line of studies collects games to build more comprehensive benchmarks to assess the artifi- cial general intelligence of LLMs. Tsai et al. 2023) found that while LLMs perform competitively in text games, they struggle with world modeling and goal inference. GameEval (Qiao et al.) 2023) introduced three goal-drit goal-driven conversational games (Ask-Guess, SpyFall, and TofuKingdom) assess the problem-solviee
capabilities of L.MSs in cooperative and adversarial settings. MAgIC (Xu et al.) 2024a) proposed the probabilistic graphical modeling method for evaluating L.MSs multi-agent game settings. LLM-Co (Agashe et al. 2023) assesses LLMs in multi-agent coordination scenarios, showcasing their capabilities in partner intention inference and proactive assistance. SmartPlay (Wul ct al., 2024) evaluated LLMs as agents across six games, emphasizing reasoning, planning, and learning capabilities. Abbdenabi et al. (2024) designed negotiation games involving six parties with distinct objectives to evaluate LLMs' ability reach agreement.
CONCLUSION
This paper presents y-Bnnch, benchmark designed to assess LLMs' Gaming Ability in Multi- Agent environments. y-Bench incorporates eight classic game theory scenarios, emphasizing multi- player interactions across multiple rounds and actions. Our findings reveal that GPT-3.5 (0125) demonstrates limited decision-making ability on 7-Bench, yet it can improve itself by learning from the historical results. Leveraging the carefully designed scoring scheme, we observe that GPT-3.5 (0125) exhibits commendable robustness across various temperatures and prompts. It is noteworthy that strategies such as CoT prove effective in this context. Nevertheless, its capability to generalize across various game settings remains restricted. Finally, Gemini-1.5-PP outperforms all tested models, achieving the highest ranking on the -y-Bench leaderboard, with the open-source I LLLMA-3.1 30B following closely behind.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The paper is supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (No. CUHK 14206921 of the General Research Fund).
REFERENCES
Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad. Lea Schönherr, and Mario Fritz. Cooperation, competition, and maliciousness: LIm-stakeholders interactive negotiation. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Saaket Agashe, Yue Fan, and Xin Eric Wang. Evaluating multi- aaglit coordination abilities in large language models. arXiv preprint arXiv:2310.03903 2023.
Gati V Aher, Rosa Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pp. 337-371. PMLR, 2023.
Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. arXiv preprint arXiv:2305. 2026,
W Brian Arthur. Inductive reasoning and bounded rationality. The American economic review, 84 (2):406441,, 1994.
Daniel Ashlock and Garrison Greenwood. Generalized divide the dollar. 2016 IEEE Congress on Evolutionary Computation (CEC), pp. 343-350. IEEE, 2016.
David Baidoo-Anu and Leticia Owusu Ansah. Education in the era of generative artificial intelli- gence (ai): Understanding the potential benefits of chatgpt in promoting teaching and learning. Journal of AI, 7(1):52-62, 2023.
Philip Brookins and Jason DeBacker. Playing games with gpt: What can we learn about a large language model from canonical strategic games? Economics Bulletin, 44(1):25-37, 2024.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12713 2023.
Valerio Capraro, Roberto Di Paolo, and Veronica Pizziol. Assessing large language models' ability to predict how humans balance self-interest and the interest of others. arXiv preprint arXiv:2307. 12776, 2023.
Jiangjie Chen, Siyu Yuan, Rong Ye, Bodhisattwa Prasad Majumder, and Kyle Richardson. Put your money where your mouth is: Evaluating strategic planning and execution of Ilm agents in an auction arena. In NeurlPS 2024 Workshop on Open- World Agents, 2024.
Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel- Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. Gtbench: Uncovering the strategic reasoning capabilities of iims via game-there thee--thereetc evaluations. Advances in Neural Information Processing Systems, 01-22219-22555 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprini arXiv:2407.22 277.2 2024.
Caoyun Fan, Jindou Chen, Yaohui Jin, and Hao He. Can large language models serve as ratio- nal players in game theory? systematic analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, number 16 38, pp. 17960-17967, 2024.
Natalie S Glance and Bernardo Huberman. The dynamics social dilemmas. Scientific American, 270(3)776-81) 1994.
Robert E Goodin. The theory of institutional design. Cambridge University Press, 1998.
11

---

Published as a conference paper at ICLR 2025
Neel Guha, Julian Nyarko, Daniel E Ho, Christopher Re, Adam Chilton, Aditya Narayana, Alex Chohlas-Wond Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, et al. Legalbench: A col- laboratively built benchmark for measuring legal reasoning in large language models. In Thirty- seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Fulin Guo. Gpt agents in game theory experiments. arXiv preprins arXiv:2305.05516 2023.
Jiaxian Guo, Bo Yang, Paul Yoo, Bill Yuchen Lin, Yusuke Iwasawa, and Yutaka Matsuo. Suspicion agent: Playing imperfect information games with theory of mind aware gpt-4. In First Conference on Language Modeling, 2024.
Babak Heydari and Nunzio Lorè. Strategic behavior of large language models: Game structure vs. contextual framing. Contextual Framing (September 10, 2023), 2023.
John J Horton. Large language models as simulated economic agents: What can we learn from homo silicus? Technical report, National Bureau of Economic Research, 2023.
Jen-tse Huang, Man Ho Lam, Eric John Li, Shujie Ren, Wenxuan Wang, Wenxiang Jiao, Zhaopeng Tu, and Michael R Lyu. Apathetic or empathetic? evaluating L.LMs" emotional alignments with humans, In Advances in Neural Information Processing Systems 37, 2024a.
Jen-tse Huang, Wenxuan Wang, Eric John Li, Man Ho Lam, Shujie Ren, Youliang Yuan, Wenxi- ang Jiao, Zhaopeng Tu, and Michael R Lyu. On the humanity of conversational ai: Evaluating the psychological portrayal Ilms. In Proceedings of the Twelfth International Conference on Learning Representations, 2024b.
Bernardo A. Huberman. The Ecology of Computation. North-Holland, 1988.
Albert Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401 1.40888 2024.
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. Is chatgpt good translator? yes with gpt-4 the engine. arXiv preprint arXiv:2301.088 088745, 2023.
Douglas Johnson, Rachel Goodman, J Patrinely, Cosby Stone, Eli Zimmerman, Rebecca Donald, Sam Chang, Sean Berkowitz, Avni Finn, Eiman Jahangir, al. Assessing the accuracy and reliability of ai-generated medical responses: an evaluation of the chat-gpt model. Research square, 2023.
D Mare Kilgour. Equilibrium points of infinite sequential truels. International Journal of Game Theory, 6: 6:167-180, 1977.
D Mare Kilgour and Steven Brams. The truel. Mathematics Magazine, 70(5):315-322 1997.
Mark Kilgour. The sequential truel. International Journal of Game Theory, 4:151-174, 1975.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 35:22199-222133 2022.
Daphne Koller and Avi Pfeffer. Representations and solutions for game-theoretic problems. Artificial intelligence, 94(1-2):167-215, 1997.
Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. Better zero-shot reasoning with role-play prompting. In Proceedings the 2024 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies (Volume 1: Long Papers), pp. 4099 2024.
Michal Kosinski. Evaluating large language models theory mind tasks. Proceedings of the National Academy of Sciences, 121(45):2405440122, 2024.
12

---

Published as a conference paper at ICLR 2025
Yihuai Lan, Zhiqiang Hu, Lei Wang. Yang Wang, Deheng Ye, Peilin Zhao, Ee-Peng Lim, Hui Xiong, and Hao Wang. Lim-based agent society investigation: Collaboration and confrontation in avalon gameplay. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 128-145, 2024.
Pier Luca Lanzi and Daniele Loiacono. Chatgpt and other large language models evolutionary engines for online interactive collaborative game design. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1383-1390, 2023.
Alain Ledoux. Concours résultats complets. les victimes se sont plu à jouer le 14 d' d'atout. Jeux & Stratégie, 2010): 1981.
Jiatong Li, Rui Li, and Qi Liu. Beyond static datasets: A deep interaction approach to Ilm evaluation. arXiv preprint 23Xx::2309904369 2023.
Tian Liang, Zhiwei He, Jen-tes Huang, Wenxuan Wang, Wenxiang Jiao, Rui Wang, Yujiu Yang, Zhaopeng Tu, Shuming Shi, and Xing Wang. Leveraging word guessing games to assess the intelligence of large language models. arXiv preprint arXiv:2310.20499 2023.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang. Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi- agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
Ziyi Liu, Abhishek Anand, Pei Zhou, Jen-tse Huang, and Jieyu Zhao. Interintent: Investigating so- cial intelligence of Ilms via intention understanding in interactive game context. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.
Shaoguang Mao, Yuzhe Cai, Yan Xia, Wenshan Wu, Xun Wang, Fengyi Wang, Qiang Guan, Tao Ge, and Furu Wei. Alympics: Llm agents meet game theory. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 2845-2866, 2025.
R Preston McAfee and John McMillan. Auctions and bidding. Journal of economic literature, 25 (2)6699-738, 1987.
Roger B Myerson. Game theory. Harvard university press, 2013.
Rosemarie Nagel. Unraveling in guessing games: An experimental study. The American economic review, 83(5):1313-1222 1995.
John F Nash. Equilibrium points in n-person games. Proceedings of the national academy of
sciences, 36(1):48-49 1950.
John F Nash. Non-cooperative games. Annals of Mathematics, 54(2):286-295, 1951.
Aidan O'Gara. Hoodwinked: Deception and cooperation text-based game for language models. arXiv preprint arXiv:2308.01404 2023.
OpenAl. Introducing chatgpt. OpenAl Blog Nov 2022, 2022. URL [tL[ittt///:snaaa. com/ index/ index/chatqpte chatqpt/
OpenAl. Gpt-4 technical report. arXiv preprint wrXiv:2303.08774. 2023.
Joseph Persky. Retrospectives: The ethology of homo economicus. Journal of Economic Perspec- lives, 9(2):221-231, 1995.
Steve Phelps and Yvan I Russell. Investigating emergent goal-like behaviour in large language models using experimental economics. arXiv preprint arXiv:2305.07970 2023.
Sundar Pichai and Demis Hassabis. Introducing gemini: largest and most capable ai model.
Google Blog Dec 06 2023, 2023. URL https: //blog// googge//techoooogy//a. gpogle-geming weaggmmiiiaat
13

---

Published as a conference paper at ICLR 2025
Sundar Pichai and Demis Hassabis. Our next-generation model: Gemini 1.5.
Google
Blog Feb 15 2024, 2024. google-gemini -aw.ttt-eeerrraaooooooaaaaea----
URL
Dan Qiao, Chenfei Wu, Yaobo Liang, Juntao Li, and Nan Duan. Gameeval: Evaluating Ilms on conversational games. arXiv preprint arXiv:2308. 10032, 2023.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt general-purpose purpose natural language processing task solver? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1339-133 1384, 2023.
Mathias Risse. What is rational about nash equilibria? Synthese, 124:361-384, 2000.
Ariel Rubinstein. Instinctive and cognitive reasoning: A study of response times. The Economic
Journal, 1 176523)::233-12599 2007.
Paul A Samuelson. The pure theory of public expenditure. The review of economics and statistics, 36(4):387-389, 1954.
Lloyd S Shapley and Martin Shubik. Pure competition, coalitional power, and fair division. Inter- national Economic Review, 10(3):337-362, 1969.
Simon Stepputtis, Joseph P Camphell, Yaqi Xie, Zhengyang Qi, Wenxin Zhang, Ruiyi Wang, San- keth Rangreji, Charles Lewis, and Katia Sycara. Long-horizon dialogue understanding for role identification in the game of avalon with large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1 11193-11208, 2023.
Ian Stewart. A puzzle for pirates. Scientific American, 280(5):98-99) 1999.
Nigar M Shafiq Surameery and Mohammed Y Shakor. Use chat gpt to solve programming bugs. International Journal of Information Technology & Computer Engineering (IJITC) ISSN: 2455-
5290, 3(01):17-2) 2023.
Chen Feng Tsai, Xiaochen Zhou, Sierra S Liu, Jing Li, Mo Yu, and Hongyuan Mei. Can large language models play text games well? current state-of-the-an and open questions. arXiv preprint arXiv:2304.02868, 2023.
William Vickrey. Counterspeculation, auctions, and competitive sealed tenders. The Journal of finance, 16(1):8-3) 1961.
Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Avalon's game of thoughts: Battle against decep- tion through recursive contemplation. arXiv preprint arXiv:2310.0133 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Haoran Wu, Wenxuan Wang. Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark. arXiv preprint arXiv:2303. 13648, 2023.
Yue Wu, Xuan Tang, Tom M Mitchell, and Yuanzhi Li. Smartplay: A benchmark for Ilms as intelligent agents. In The Twelfth International Conference on Learning Represenaatinss 2024.
Chengxing Xie, Canyu Chen, Feiran Jia, Ziyu Ye, Shiyang Lai, Kai Shu, Jindong Gu, Adel Bibi, Ziniu Hu, David Jurgens, James Evans, Philip Torr, Bernard Ghanem, and Guohao Li. Can large language model agents simulate human trust behaviors? Advances in neural information processing systems, 37, 2024.
Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See Kiong Ng, and Jiashi Feng. Magic: Investigation of large language model powered multi-agent in cognition, adaptability, rationality and collaboration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 7315-7332, 2024a.
14

---

Published as a conference paper at ICLR 2025
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint wxxv:2309.04658, 2023.
Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the werewolf game. In Proceedings of the Forty-first International Conference on Machine Learning. 2024b.
Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1061 10671, 2024.
Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart be safe: Stealthy chat with Ilms via cipher. In The Twelfth International Conference on Learning Representations, 2024.
Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Yan Xia, Man Lan, and Furu Wei. K-level reasoning with large language models. arXiv preprint arXiv:2402.01521, 2024.
Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji- Ri-Rong Wen. Large language models for information retrieval: A survey, arXiv preprint 2rXiv:3008.0710,, 2023.
15

---

Published as a conference paper at ICLR 2025
GAMA-Bench Framework
Cooperative Games (1) Guess 2/3 of the Average (2) EI Farol Bar
(3) Divide the Dollar
TDI
Betraying Games
(4) Goods Game
(5) Diner's Dilemma
(6) Sealed Auction
A
1 3-4 Sequential Games
t antii
(7) Battle Royale
(8) Pirate Game
Figure 3: y-Bench enables multiple LLMs and humans to engage in multi- games. The frame- work comprises three categories of games, each targeting different LLM abilities, and includes eight classic games from Game Theory.
CONTENTS
A More Information Game Theory
18
Formulation
18
A.2 Nash Equilibrium
18
Human Behaviors
18
Literature Review: Evaluating LLMs with Game Theory
19
Details about Prompts
20
C.I Design Methodology
20
Cooperative Games
20
Betraying Games
22
Sequential Games
24
Examples of Geh-asRephrased Prompts
26
Rescale Method for Raw Scores
29
Detailed Results
30
F1
Robustness: Multiple Runs
30
Robustness: Temperatures
F3
Robustness: Prompt Templates
32
16

---

Published as a conference paper at ICLR 2025
F.4 Generalizability
34
Leaderboard
35
F6
Detailed Player Actions of GPT-3.5 (0125)
36
LLM vs. Specific Strategies
36
Jailbreak Influence
37
Limitations
37
J Ethics Statement and Broader Impacts
38
LLaMA-3.1-708
38
K.1
Robustness: Multiple Runs
38
K.2
Robustness: Temperatures
38
K.3
Robustness: Prompt Templates
39
K.4
Generalizability
39
Gemini-1..-Prr
40
Robustness: Multiple Runs
40
Robustness: Temperatures
40
Robustness: Prompt Templates
Generalizability
41
GPT-4o
42
M.
Robustness: Multiple Runs
42
M.2
Robustness: Temperatures
42
M.3
Robustness: Prompt Templates
43
M.4 Generalizability
17

---

Published as a conference paper at ICLR 2025
A MORE INFORMATION ON GAME THEORY
A. FORMULATION
Game theory involves analyzing mathematical models of strategic interactions among rational agents (Myerson 2013). A game can be modeled using these key elements:
1. Players, denoted as P = {1,2, ;N): A set of N participants.
Actions, represented as
{A.): N sets of actions available each player. For instance, AN = (C,FF)
A = A = A2 FD,
3. Utility functions, denoted as 4 = {u; x x A, + RJ: A set of N functions that quantify each player's preferences over all possible outcomes.
4. Information, represented I }I N sets of information available to each player, including other players' action sets, utility functions, historical actions, and other beliefs.
Order, indicated by O = 01, O2 ... Og: A sequence of sets specifying the ls steps to take actions. For example, O = P implies that all players take actions simultaneously.
In this study, Multi-Player games are defined as those with |P| 2 since game theory models have least two players. Similarly, Multi-Action games are those where Mii > 2. Meanwhile, Mulli-Round games involve the same set of players repeatedly engaging in the game, with record of previous actions being maintained. Simultaneous games satisfy that k = whereas Sequen- tial games have k > 1, indicating players make decisions in specific order. Games of Perfect Information are characterized by the condition Ve.jePiayII = Z₁. Since every player can see their own action, the above condition indicates that all players are visible to the complete information set in the game. Conversely, games not meeting this criterion are classified as Imperfect Information games, where players have limited knowledge of others' actions.
A.2 NASH EQUILIBRIUM
Studying game theory models often involves analyzing their Nash Equilibria (NE) NNahh 1950). An NE is a specific set of strategies where no one has anything to gain by changing only one's 's own strategy. This implies that given one player's choice, the strategies of others are constrained to specific set, which turn limits the original player's 's choice to the initial one. When each player's strategy contains only one action, the equilibrium is identified (PSNE) (ash 1950). However, in certain games, such as rock rock-paper paper-scissors, an NE exists only when players employ a probabilistic approach to their actions. This type of equilibrium is known as a Mixed Strategy Nash Equilibrium (MSNE) (Nash) 1951), with PSNE being a subset of MSNE single action. According to Thm. A.T shown below,
Pure Strategy Nash Equilibrium
where probabilities can analyze the NE of each game and evaluate whether LLMs' choices align with the NE.
concentrated
Theorem A.1 (Nash's Existence Theorem) Every game with a finite number of players which each player can choose from finite number of actions has at least one mixed strategy Nash equi- librium, in which each player's 's action is determined by a probability distribution.
A.3 HUMAN BEHAVIORS
The attainment of NE presupposes participants as Homo Economicus, who are consistently ratio- nal and narrowly self-interese aiming at maximizing self goals (Persky 1995). However, human recision-making often deviates from this ideal. Empirical studies reveal that human choices fre- quently diverge from what the NE predicts (Nagell 1995). This deviation is attributed to the com- plex nature of human decision-making which involves not only rational analysis but also personal values, preferences, beliefs, and emotions. By comparing human decision patterns documented in prior studies, together with the NE, we can ascertain whether LLMs exhibit tendencies more akin to homo economicus actual human decision-maker thus shedding light on their alignment with human-like or purely rational decision-man makiing processes.
18

---

Published as a conference paper at ICLR 2025
B LITERATURE REVIEW: EVALUATING LLMS WITH GAME THEORY
Evaluating LLMs through game theory models has become a popular research direction. An overview on recent studies is summarized in Table 3 From our analysis, several key observations emerge: (1) The majority of these studies are concentrated on two-player settings. (2) There is a pre- dominant focus on two-action games; notably, half of the studies examine the Prisoner's 's Dilemma and the Ulrimatum Game (the Dictator Game is one of the variants of the Ultimatum Game). (3) A notable gap the literature is the lack of the comparative studies between LLMs' decision-making across multiple rounds and the action probability distributions predicted by the MSNE. (4) The studies exhibit variability the temperatures used, which precludes definitive conclusions regard- ing their impact on LLMs' performance.
Table 3: A Comparison of existing studies that evaluate LLMs using game theory models. T denotes the temperature employed in each experiment. MP refers to a multi pllyer setting, whereas MR indicates multi-round interactions. Role specifies whether specific role is assigned to the LLMs.
Paper
Models
MP
MR
Role
CoT
Games
Horton
2023
text-davinci-003
x
x
Dictator Game
Guo
2023
1px-4-1106-previi
x
V
V
Uliimatum Game, Prisoner's Dilemma
Phelps & Russell
2023
gpt-3.5-turho
0.2
X
X
Prisoner's Dilemma
text devidci-ni-O0 003,
Prisoner's s Dilenuaa, Battle of the Sexes
Akata al.
2023
gpt-3.5-turbo, gpt-4
0
x
x
x
text ada 001, text babbage-001 001,
text-curie- text-davinci-001
Aher et al.
2023
text-davinci-002, text-davinci-003,, gpt-3.5-turbo, gpt-4
x
X
Ultimatum Game
Capraro
2023
ChatGPT-4, Bard, Bing Chat
x
Dictator Game (Three Variants)
Brookins & DeBacker 2024
gpt-3.5-turbo
x
x
X
Dictator Game, Prisoner's Dilemma
gp--3.--urbo-06113
et
2023
gpt-4-0613, claude-2.0,
X Public Goods Game
chat-bison-001
gpt-3.5-turbo-Iee 16k, gpt-- llama-2
Prisoner' s Dilemma,
Heydari & Lowe
2023
0.8
x
Stag Hunt, Snowdrift, Prisoner's Delight
2024
gpt-3.5, gpt-4
Leduc Hold cm
2pe-5.5-trbo-u5511
Chen et al.
2024
gpt-4-0613, claude-instant-l.2.
0.7
English Auction
claude-2.0, chat-bison-001
gpt-3.3-turbo, gpt-4,
Cost Sharing, Prisoner's Dilemma,
Xu
llama-2-70b, claude-2.0, palm-2
Public Goods Game
text- text-davinci-003
Dictator Game, Sace--ppp--SSsoorr.
Fan
2024
t-t-b.5- gpt-4
0.7
x
X Ring-Network Game
Guess Avernge
Zhang et al.
2024
gpt-4
0.7
/
Survival Auction Game
gpt-3.5-turbn, gnt-4,
Duan 2024
1lama-2-70b, codellamn-34b,
Ten Games'
mistral-7b-orca
text-davinci-003,
gpt-3.-ttrbo-instruct
Xie et al. 2024
gpt-3.5-turbo-0613, gpt-4.
x
Seven Games
llama 2 (7/13/70)b,
gpl 3.5.3.5 gpt-4
This Study
0-1
/
Fight Games"
gemini-pm
Tic/Tc Tue, Connect 4, Kuhn Paker, Breakthrough, Liar's Dice, Blind Auction, Negotiation, Nim, Pig, Iterated Prisoner's Dilemma. Acceptable Probahilities Trust Game, Repeated
Trust Game, People Game, Lottery Gamble Game.
Game, Dictator Game, Risky Dictator Game, Lottery
Guess 2/3 of the Average, El Farol Bar, Divide the Dollar, Public Goods Game, Diner's Dilemma, Scaled-Bid Auction, Battle Royale. Pirate Game.
BACK TO INTRODUCTION]
BACK TO RELATED WORK
19

---

Published as a conference paper at ICLR 2025
C DETAILS ABOUT PROMPTS
C.I DESIGN METHODOLOGG
We adopt cohesive approach ensure the prompt design is systematic and arbitrary. Game descriptions are gathered from verified sources, including academic papers referenced in $2 and Wikipedia entries. Using these deseriptions, we instruct GPT-4 to generate prompts to guide LLMs in engaging in the specified games. These prompts are structured to encompass four essential ele- ments: the rules the game, objectives for the players, a template for announcing game outcomes (for displaying historical results), and instructions for formatting responses in JSON. A manual checking process is conducted to ascertain that GPT-4's comprehension of the game descriptions is correct. The prompts detailed in the rest part of this section.
C.2 COOPERATIVE GAMES
Guess 2/3 of the Average
SYSTEM
You are participating in game played by N players over K rounds.
Game Rules:
1. Each player selects an integer number between MIN and MA,, inclusive,
2. After all selections are made, the average of all chosen numbers is calculated.
3. The target number is R this average.
4. The winner is the player(s) who selected number closest to the target number.
USER
Game Results for Round T: Average Number Chosen: MM Target Number (R of Average): Ti
Winning Number: W1
You chose:
ASSISTANT
("chosen number": ""chosen_number": "CrJ")
USER
[Congratulation you won//Unnfortunately you lost].
USER
Now round I starts.
Your goal is to choose number that you believe will be closest of the average of all numbers chosen by players, including your selection.
Please provide your chosen number in the following JSON format:
("chosen_number":
number": "integer.between. between_M/73 and_M M MAX).
BACK TO GAME DESCRIPTION
BACK TO VANILLA EXPERIMENT
20

---

Published as a conference paper at ICLR 2025
El Farol Bar
SYSTEM
You are participating in game played by N players over K rounds.
Game Rules:
1. Every round, you and the other players decide independently whether to go to
a bar.
2. If equal to less than R of the players go the bar, everyone who goes has more fun than staying home, receiving a utility of M MAX.
3. If than R of the players go the bar, everyone who goes has less fun than staying home, receiving utility of MIN.
4. Everyone who stays home receives a utility of HOME.
USER
Game Results for Round 7:
(Only for Explicit) G, players went to the bar, while S players stayed home. Gr/N, which [more]/[equal to or less] than R the players went to the bar. It was [[ors][[mor]] fun to go the bar this round.
You chose: ("decision": "D1J"} You gained G.J-
ASSISTANT USER
USER
Now round / starts.
Your goal is maximize your fun. Choose go the bar when you predict fewer than It of the players will go, and choose to stay home otherwise.
Please provide your decision in the following JSON format: ["decision": "go_or or ar
Divide the Dollar
SYSTEM
You are participating in game played by N players over K rounds.
Game Rules:
1. You are dividing G golds. Each player independently proposes a bid.
2. If the sum of all bids does not exceed G, each player receives their bid amount.
3. If the sum exceeds G, all players receive nothing.
USER
Game Results for Round /: Your bid amount was: ("bid_amoun
ASSISTANT USER
"Bıj"}
The sum of all bids was St. The sum [does not exceed//exceeess] G.
You received [ V/O] golds.
USER
Now round I starts.
Your goal is maximize your individual gain without causing the total sum of bids to exceed G golds.
Please provide your bid amount in the following JSON format:
""bid_amoutt" "integer between O. 0_and_co G"").
BACK TO GAME DESCRIPTION
BACK TO VANILLA EXPERIMENT
21

---

Published as a conference paper at ICLR 2025
C.3 BETRAYING GAMES
Public Goods Game
SYSTEM
You are participating in game played by N players over K rounds.
Game Rules:
1. In each round, you, as a player, must decide how many of your private tokens
you wish to contribute secretly to the public pot.
2. The total tokens in this pot will be multiplied by the factor to create the "public good" payoff.
3. This payoff will then evenly divided among all players, regardless of their individual contribution.
4. Any tokens you do not contribute will retained in your private collection.
USER
Game Results for Round T: Contributed tokens of each player: Cr Cr2. ,CIn
You contributed:
ASSISTANT USER
("tokens.contriburee cootribbueed" "C1,"} Tokens in the public pot: S₁
Your gain: grJ
Your tokens after round 7: Tı
Tokens of each player after round T: Tni, T" TIN
USER
Now round I starts.
Your goal is maximize your total token count by the end of the game. Currently you have T7 tokens. You need to decide the number of tokens to be contributed to the public pot.
Please provide the number of tokens in the following JSON format: boookenscoottibutede "integer "integer.between. between_O_aadd77/.
Diner's Dilemma
SYSTEM
You are participating in game played by N players over K rounds.
Game Rules:
1. Each player must choose order either a costly dish cheap dish.
2. The price of the costly dish is Ph. The price of the cheap dish Pt.
3. The costly dish brings you utility of Un. The cheap dish brings you a utility
of Ut.
4. The costly dish tastier than the cheap dish, but not sufficiently to justify its
price when dining alone.
5. At the end of each round, the total cost dishes ordered is split equally among all players.
USER
Game Results for Round I:
No people chose the costly dish, while Nj chose the cheap dish. The total cost is Sj. You need pay C1-
You chose:
ASSISTANT USER
{ ["chosen "ioossnndissh::: dish":"Dis" Your utility is urJ.
USER
Now round I starts.
Your goal is maximize your overall satisfaction, balancing the quality of the dish and the cost shared.
Please provide your chosen dish in the following ISON format: ["chosen_dishe dish": "costly orr cheap""
BACK TO GAME DESCRIPTION]
BACK TO VANILLA EXPERIMENT]
22

---

Published as a conference paper at ICLR 2025
Sealed-Bid Auction
SYSTEM
You are participating in game played by N players over K rounds.
Game Rules:
1. Each player has private valuation for the item in each round.
2. Without knowing the bids and valuations of other players, each player submits written bid for the item.
3. The highest bidder wins the item and pays the price of the [highes]][second highest] bid.
4. If you win, your utility for that round is your valuation minus the price paid. If you lose, your utility zero.
USER
Game Results for Round I:
Your valuation for this round's item was VIJ- Your bid was:
ASSISTANT USER
("bid": "bys")
The winning bid was: WI-
The price paid was: Pj.
You [won]/[lost]. Your utility is [urs][[].
USER
Now round / starts.
Your goal is maximize your total utility. Your valuation for this round's item is VzJ-
Please provide your bid in the following JSON format:
("bid": "integer between O 0_and_ "add"
BACK TO GAME DESCRIPTION]
BACK TO VANILLA EXPERIMENT
23

---

Published as a conference paper at ICLR 2025
C.4 SEQUENTIAL GAMES
Battle Royale
SYSTEM
You are participating in a game played by N.
Game Rules:
1. You are in a survival game where only one can survive and win.
2. Players take turns shooting others in predetermined order based on their
hit rates, from the lowest to the highest.
3. Players' names and hit rates ranked by shooting order are ("IDi": ": "HIT," "IDx": "HITx"): You are ID,. Your hit rate is HIT,, You are the RANK to shoot.
"TD2": "HIT2"
4. You have unlimited number of bullets.
5. You may choose to intentionally miss your shot on your turn.
USER
Game Results for Round I:
Your action:
ASSISTANT
(Only for the player itself) ("target": "I,,")
USER
NAME, [intentionally missed the shot]/[shot tı and hit]/[shot tı but missed].
There are N2 players left.
USER
Now round 1 starts. Your goal is to eliminate other players to survive until the end and win the game. The remaining players' names and hit rates ranked by shooting order are: ("ID,") "HIT1", " "ID2": "HITz". is HIT), You are the J-th to shoot. Please decide whether to shoot at player intentionally miss. Please provide your action in the following JSON format:
"IDN": "HITN") "). You are ID,. Your hit rate
("target": "playerlID_or_nu"
BACK TO GAME DESCRIPTION
BACK TO VANILLA EXPERIMENT
24

---

Published as a conference paper at ICLR 2025
Pirate Game
SYSTEM
You participating in game played by N.
Game Rules:
1. You are pirates who have found G gold coins. You deciding how to distribute these coins among yourselves.
2. The pirates will make decisions in strict order of seniority. You are the
RAN K--tth most senior pirate.
3. The most senior pirate proposes plan to distribute the G gold coins.
4. All pirates, including the proposer, vote on the proposed distribution.
5. If the majority accepts the plan, each pirate receives the gold coins as the most senior pirate proposed.
6. the majority rejects the plan, the proposer thrown overboard, and the next senior pirate proposes new plan.
7. The game ends when a plan is accepted or only one pirate remains.
USER
The I-th most senior pirate proposed a plan of ("I": "gır", "I 1": "911+1"
.., "I": "gIN"): "].
A, of pirates chose to accept the distribution.
You chose:
ASSISTANT USER
["decision": "D,
Less than half of the pirates accepted the plan,
The 1-th most senior pirate was thrown overboard and eliminated from the game.
The game continues.
USER
Now the I-th most senior pirate needs to propose plan.
Your primary goal is to survive. If you survive, your next goal is to maximize the
number of gold coins you receive. You may also prefer to throw another pirate overboard if it does not negatively impact your other goals.
- The proposed plan is (": "911" " "I+1":"99H4
For voters
"911+1"
"7" "9"""""""" "g"N"". You will
get gs golds from this plan.
Please provide your decision on the current proposal in the following JSON for- mat:
["decision": "accept or reject") You need to propose plan to divide G Gogold.. The proposed numbers must be all non-negative integers and up to G.
For proposer
Please provide your proposal the golds distributed to each pirate from the you to the I-th most senior in the following JSON format:
("proposal": {"I": "grr"" + 1": "911+1", :., "I": "gIN"}}
BACK TO GAME DESCRIPTION]
BACK TO VANILLA EXPERIMENT
25

---

Published as a conference paper at ICLR 2025
D EXAMPLES OF GPT-4-REPHRASED PROMPTS
$4.1 involves testing the GPT-3.5 (0125)'s robustness against different prompt templates. This section presents the prompts used this analysis, namely Prompts V2 V4, with VI the default, as detailed in $ We include only the prompts for the game "Guess 2/3 of the Aver- age, while the five prompt templates seven other games can be found in our GitHub hhtpps: 7/github. com/ CUHK- GEGAA..hcchh......
Guess 2/3 of the Average (V2)
SYSTEM
You're participating game involving N participants and it spans across K rounds.
The rules of the game are as follows:
Every participant must choose integer within the range of MIN to MAX, both included.
1.
2. Once everyone has chosen their numbers, the mean of all these numbers is computed.
3. The goal number becomes I times this average.
4. The person or people who picked a number closest to the goal number are declared the winners.
USER
The outcomes of the game for Round are as follows: The average number selected was M,
The target number, which is R of the average, Ty
The number that won was W₁.
Your selection was:
ASSISTANT
: "Cı."} (Congratulation you won]/[nnfortunately you lost].
USER
USER
Commencing with round T.
Your target should be to select number that in your opinion will align most closely with Rt of the total average of all the player's numbers selected, your choice included.
Please provide your chosen number in the following JSON format: ("chosen number": "integer_ between IN and M AX").
BACK TO RQ1]
26

---

Published as a conference paper at ICLR 2025
Guess 2/3 of the Average (V3)
SYSTEM
You're re engaged in game, involving NV participants across K rounds. Rules of the Game:
1. An integer number is chosen by every player, within the range of MIN and MAX, both numbers included.
2. Once each player has chosen, the average is determined from all the selected numbers.
3. The average is multiplied by R find the target number.
4. The individual or individuals whose chosen number is nearest to the target number are declared the winners.
USER
Results of Round I Game:
Chosen number's 's average: M,
The target percentage (R of average) is: Ti
The winning number is: Wr.
You chose:
ASSISTANT
["chosen number": "CıJ"} [Congratulation you won ]/[ Unfortunatlly you lost].
USER
USER
The commencement of round is now.
The objective is to select number that you think will be nearest R times the average of all the digits chosen by the participants, your choice included. Please provide your chosen number in the following JSON format: {"chosen number": "integer between. Maricen_MMM MIN and M MAX".
Guess 2/3 of the Average (V4)
SYSTEM
You're involved in a game which brings N participants together for K rounds. follows:
The guidelines of the game
1. All players have to pick whole number anywhere from MIN to MAX, both numbers included.
2. The chosen numbers are then gathered and their mean is computed.
3. The number to aim for, or the target number, R of the calculated average.
4.The victorious player(s) are those whose chosen number is closest to the target number.
USER
The outcomes for Round are as follows:
The average number selected was M,. The target number, which is R times the average, was T,. The triumphant number was W,. Your choice was:
ASSISTANT USER
["chosen number": "cchosen_number": "Cr."} [Congratulation you won Junfotourrtu you lost].
USER
The commencement of round is now.
You are tasked with selecting number that, in your estimation, will be as close as possible to times the average of numbers chosen by all players, your own choice included.
Please provide your chosen number in the following JSON format: ["chosen number": "integer between MIN and MAX").
BACK TO RQI]
27

---

Published as a conference paper at ICLR 2025
Guess 2/3 of the Average (V5)
SYSTEM
You will be engaging in game that is played over K rounds and includes total of N players.
The Instructions of the Game:
1. Every player is supposed to pick an integer that within the range of MIN and MAX, both numbers inclusive.
2. The median of all the numbers chosen by the players is then determined after all choices have been made.
3. The number that players are aiming for is R times the calculated average.
4. The player or players who opt for the number closest this target are declared the winners.
USER
Results of the Game for Round I: The chosen average number is: M, The target number (R of Average) is: T The number that won: Wi. Your selection was: ("chosen number": "CrJ"} [Congratulation you won J/[ UUnfortunatell you lost).
ASSISTANT USER
USER
The commencement of round is now.
You are challenged to select number which you conjecture will be nearest to R times the mean of all numbers picked by the players, inclusive of your own choice.
Please provide your chosen number the following JSON format: "integer_between.ll Mandedd___aaaaMMMMMMMM and_MAX"
BACK TO RQI]
28

---

Published as a conference paper at ICLR 2025
E RESCALE METHOD FOR RAW SCORES
The raw scores across games lack consistency. In some games, higher scores indicate better perfor- mance, while in others, lower scores are preferable. Additionally, the score range varies by game and can change with different game parameters. To standardize scores on 7-Bench, we rescale raw scores range of 100, where higher scores always indicate better performance. The scoring scheme is detailed in Eq. [
(MAX SMAX-MIIN MIN)- 91 100,
R<1
MAX MIN 128 - (M MAAX-MIANN MINJ 100,
81
R=
MAX-MIX
MAX MIN 100,
R>1 1
max R) S2 100,
S2
max Hi, 1 - R))
G S3
S3 max
100, 0
G
(I)
T-94 100,
S 1
S4
S T
100,
1
85 (1 S5) 100,
S6 = S * 100,
87 ST 100,
Ssr
Sş 2 G
* 50 + Sav 50.
BACK TO VANILLA EXPERIMENTS
29

---

Published as a conference paper at ICLR 2025
F DETAILED RESULTS
This section presents both quantitative and visualized results for $4 and includes plots of player actions from the GPT-3.5 (0125) experiments in
F.1 ROBUSTNESS: MULTIPLE RUNS
Table 4: Quantitative results of playing the games with the same setting five times.
Tests
T1 (Default)
T2
T3
T4
T5
Aug+std
Guess 2/3 of the Average
65.4
62.3
63.9
58.3
67.3
63.443.4
El Farol Bar
73.3
67.5
68.3
67.5
66.7
68.7
Divide the Dollar
68.1
67.7
68.7
66.0
72.6
68.642.4
Public Goods Game
41.2
25.4
45.7
38.0
44.0
38.9
Diner's Dilemma
4.0
3.5
0.0
6.5
0.0
2.8
Sealed-Bid Auction
14.6
14.6
11.6
12.9
11.5
13.0+1.5
Battle Royale
20.0
21.4
46.7
23.5
31.2
28.6.11.0
Pirate Game
80.6
71.2
72.0
74.7
59.5
71.617.7
Overall
45.9
41.7
47.1
43.4
44.1
44.4
Cucc 2f Ancigg
Diikk Dulla
P6T Grode
Avstage Number
Probandayy Player Lhooung Co
Avatage Propical
Average Cxitahation
1.2
0.4
(71 Reyau
Clane Proporel Oytiral
Player Tregeling
Upper
Figure 4: Results of playing the games with the same setting five times.
BACK TO RQ1]
30

---

Published as a conference paper at ICLR 2025
F.2 ROBUSTNESS: TEMPERATURES
Table Quantitative results of playing the games with temperature parameters ranging from 0 to 1.
Temperature
0.0
0.2
0.4
0.6
0.8
1.0
Aug+std
Guess 2/3 of the Average
48.0
50.0
49.8
54.7
61.7
65.4
54.947.1
EI Farol Bar Divide the Dollar
55.8
71.7
63.3
68.3
69.2
73.3
66.916.4
69.3
67.0
67.6
67.9
72.8
68.1
68.8
Public Goods Game
15.3
10.7
17.8
18.0
36.5
41.2
23.3
Diner's Dilemma
0.0
0.0
0.0
0.0
0.0
4.0
0.7+1.6
Sealed-Bid Auction
13.1
14.0
12.2
11.1
13.0
14.6
13.0
Battle Royale
28.6
26.7
46.7
15.0
33.3
20.0
28.4+11.1
Pirate Game
75.0
53.9
77.7
83.8
59.5
80.6
71.7
Overall
38.1
36.7
41.9
39.9
43.2
45.9
41.043.4
122 Probabillyy offlayer Chooking Co
Devidd Doiler Average Progonal
10. Goode
Averago Namber
Avcrege Contibation
Scated Auckaa
(7) Brsiee
(3) Plaze Gare
Plyer Choosing Chop
Asrrnge
Claxalaiie Probubiitt Plyyz Tarpeiin
Upper Ditance Propoiel Optinal
Highhe
Viring Accerncy
Figure 5: Results of playing the games with temperature parameters ranging from 0 to 1.
BACK TO RQ1
31

---

Published as a conference paper at ICLR 2025
F.3 ROBUSTNESS: PROMPT TEMPLATES
Table 6: Quantitative results of playing the games using different prompt templates.
Version
(Default)
V3
V4
V5
Aug+sid
Guess 2/3 of the Average
65.4
66.4
47.9
66.9
69.7
63.3
HI Farol Bar
73.3
75.8
65.8
75.8
71.7
72.5
Divide the Dollar
68.1
81.0
91.4
75.8
79.6
79.2+8.5
Public Goods Game
41.2
26.6
45.2
50.2
24.2
37.5411.5
Diner's Dilemma
4.0
3.5
0.0
57.0
18.5
16.6423.7
Sealed-Bid Auction
14.6
11.8
13.4
8.0
15.5
12.643.0
Battle Royale
20.0
30.8
15.0
25.0
18.8
21.946.1
Pirate Game
80.6
87.9
60.8
60.5
53.7
68.7
Uis Anttape
120
123 Dividi the Averge Progaaal
(4) Goeds
Asttago
Probatillyy Choosing
Average
0.00
50
Donc's Dikennns
Axckas Avringe (Vokeekon Tury. Valurton
Brsaie Canaalaiee Probubiity Playzo Tageiita
Phone
Pastobitry Playe booding Chop Dach
Uyper Distance la Propoial froms Optinal Lowen The Varing Accency
Other Player with llighe Ror
Figure 6: Results of playing the games using different prompt templates.
BACK TO RQ1
32

---

Published as a conference paper at ICLR 2025
Table 7: Quantitative results of playing the games using prompt-based improvement methods.
Improvements
Default
CoT
Cooperative
Selfish
Mathematician
Guess 2/3 of the Average
65.4
75.1
69.0
14.5
71.4
EI Farol Bar Divide the Dollar
73.3
71.7
74.2
63.3
60.0
68.1
83.4
70.7
49.7
69.2
Public Goods Game
41.2
56.1
32.4
37.4
25.6
Diner's Dilemma
4.0
82.5
0.0
17.5
47.0
Sealed-Bid Auction
14.6
5.3
16.3
11.6
13.0
Battle Royale
20.0
17.6
6.2
33.3
26.7
Pirate Game
80.6
71.2
80.6
74.7
59.5
Overall
45.9
57.9
43.7
37.8
46.5
16:
641
Auriige
0.205
MMM
Thmer Dibenno
(2) Bazle Reyak
(90 Plazuc Gannc
Poctubiiyy Flase Choosin the Chapp Dock
AvcraR (Valadkas Saal/ Vakuakks
Chevelati Probabiity Payers Thegriing
Upper Deemnc of Propecal troes Opttrol
Oth Playe Highos Rate
Lowes: The Valngg Awcnrnyy
Figure 7: Results of playing the games using prompt-based improvement methods.
BACK TO RQ2
33

---

Published as a conference paper at ICLR 2025
F.4 GENERALIZABILITY
Table Quantitative results of playing the games with various game settings.
Guess 2/3 of the Average
Aug
R
0
176
173
1/2
2/3
376
7/6
473
3/2
5/3
1f/6
-
79.1
61.7
66.6
65.4
65.4
54.8
37.6
70.0
74.9
65.9
67.3
63.3
73.6
65.1
El Farol Bar
Aug+std
π
0%
20%
40%
60%
50%
100%
53.5
61.3
63.3
73.3
68.1
60.0
63.346.9
Divide the Dollar
Aug+std
-
50
100
200
400
800
73.2
68.1
82.5
82.1
80.7
77.346.4
Public Goods
Game
Aug
R
0.0
0.5
1.0
1.0
2.0
4.0
42.0
29.0
52.5
41.3
25.9
38.1 38.1
Diner's Dilemma (,,2,,7,,UR - (10, 16, 20,
Arg-
0,2,5,2077) 7)
"0,10,5
11,8,10112.
(7,,I7,7)
14,11, (11,,133 [3j
2.5
13.5
0.0
12.0
6.1
Sealed-Bid Auction (0. 100
Aug
Ronge
(0,200)
[0, 400]
"(,, 800]
12.9
14.6
12.5
13.0
13. 10.9
Battle Royale
Sug
Ränge
151, 60] 28.6
135, [85,80) 20.0
[10,100] 33.3
27.3
Pirate Game -
Aug=std
100
400
73.8 47.1
80.6
83.6
71.3
60
Cless of Aurrgg Ausiige
23 Rond Bay
(3 Di Dailer
GOL Gedle-are Awcrra Contibaian
Josnago Nander
Probatsdiy Plaper Chocsing Oo
Jocrag Propoual
030-
Patobilty of Chonsing
(taluation Vollation
Other Player Hlihhe Rosa
Accerncy
Figure 8: Results of playing the games with various game settings.
BACK TO RQ3]
34

---

Published as a conference paper at ICLR 2025
F.5
LEADERBOARD
6.
Aurtage
120
Dhar Dakanens
(0) Scaiel BAL Audhes
(2) Basta Recaa
Patabbityy Flayer Chonsing Chop
Aceraga (Valastin (B0)) Valasto
Conalatied Probubiity Playes Tangeing
Upper
Distance of Enc Proporel froes Optioel
Other Plyor with Highes Rite
Lower: The Viiiig Acceriny
Figure 9: Results of playing the games using different closed-source LLMs.
225
201
(11
(25
Drvie Aroug Progiel
(O Gadde
Jernag Nandec
Probaiilly Playe Choeinn Ou
Awarge Contibation
Auckas
Callatta Ressa Playn Targeling
Chop
Upper Distaco Properel Optical
Playr Highor
Viring Aocenncy
Figure 10: Results of playing the games using different open-source LLMs.
BACK TO RQ4
35

---

Published as a conference paper at ICLR 2025
F.6
DETAILED PLAYER ACTIONS OF GPT-3.5 (0125)
10 Pi Narboes
D 14 Publiilliss
1 Prchuiitiin
Prepored
60
649 Public Grodd Gane Playort Proproed Takens
(39 Dinc's DECTEza
(3-1 Seakee Bid Abcion--rin Price
(6-21 Seateeh Bird Autctae Securadd Prise Flayore Vallis Kliras
Playort Prido Niliioo Chooorrrr ng Cortly
Playend (Niluttoon BRA)) Vitutton
Figure 11: Player actions in Cooperative and Betraying Games.
G LLM vs. SPECIFIC STRATEGIES
Figure 12: Performance of GPT-3.5 (0125) playing against two fixed strategies in the "Divide the Dollar" and "Public Goods Game."
Our framework enables concurrent interaction between LLMs and humans, allowing us investi- gate LLMs' behaviors against someone who plays with fixed strategy. There are many possible strategies, here we use two examples: First, we let one player consistently bid an amount 91 golds in the game of "(3) Divide the Dollar," compelling all other participants to bid a single gold. The objective is to ascertain if LLM agents will adjust their strategies in response to dominant partici- examine agents' reactions persistent free-rider who contributes nothing the "(4) Public Goods Game" to determine whether agents recognize and adjust their cooperation with the free-rider
pants. Additionally,
time. We plot the average bids and the contributed tokens of the nine agents in Fig. I2 We find that agents lower their bids in the "(3) Divide the Dollar" game in response to dominant strategy. Contrary to expectations, in the "(4) Public Goods Game," agents increase their contributions, compensating for the shortfall caused by the free-rider.
The above experiments implicitly assume that players are not informed about others' fixed strategies. To investigate the effect of explicit information, we design additional experiments using the "Guess 2/3 of the Average" game, where players are provided varying levels of information about others' strategies:
Setting (a). The player is explicitly informed that others are smart and will always choose 0 (the Nash equilibrium).
Setting (b). The player is informed that others are smart but not explicitly told they will always choose the Nash equilibrium.
Setting (c). The player informed that others stupid and will choose random numbers.
36

---

Published as a conference paper at ICLR 2025
These experiments are conducted using GPT-4o, and the results are as follows:
Setting (a). GPT-4o selects 0 in the first round and continues to do so in all subsequent rounds.
Setting (b). GPT-4o does not select 0 in the first round but converges to within a few rounds.
Setting (c). GPT-4o's selections are random, indicating an inability to infer the optimal choice of
33 (calculated as 50 3. given the average of others' random selections is 50).
H JAILBREAK INFLUENCE
To bypass the value alignment in LLMs, we use the jailbreak technique, specifically Cipher- Chat (Yuan et al., 2024). Prior research demonstrated that this method can exacerbate negative traits in GPT-4 (Huang et al. 2024b). To assess whether value alignment influences the behav- ior of LLMs, evaluate GPT-4o's performance in behavioral contexts before and after applying CipherChat, using the Public Goods Game and the Diner's ' Dilemma.
Prior to CipherChat, GPT-4o achieves scores of 90.91 + 2.72 in the Public Goods Game and 10.7 + 8.3 in the Diner's Dilemma. After the jailbreak, its performance declines to 88.55 2.38 and 7.5 + 4.1, respectively. This decline reflects GPT-4o's inherent risk-averse tuning. For instance, in the Public Goods Game, GPT-4o prioritizes minimizing losses, reasoning, "If no contribution is made the public pot, I will have no loss." Similarly, in the Diner's Dilemma, it opts for the less costly dish to reduce expenditures. A comparable conservative approach is observed in the El Farol Bar Game, where GPT-4o tends to avoid the risk of overcrowding by staying home. In conclusion, GPT-4o adopts riskier strategies after jailbreak, such as contributing less in the Public Goods Game and selecting the more expensive dish in the Diner's Dilemma.
One assumption is that CipherChat could reduce the model's prosocial behavior, make it more self- serving, and increase its scores in our betraying games. We believe that the observed decrease in scores results from OpenAl's efforts to enhance GPT-4o's value alignment, thereby mitigating the influence of CipherChat. Following Huang et al. (2024b), we assess the model's negative traits using the Dark Triad Dirty Dozen. The results of GPT-4o, both before and after the jailbreak, presented in Table 9 Contrary to the findings of Huang et al. (2024b), which reported increased scores after jailbreak, our analysis indicates decrease in these negative traits for GPT-4o. This result suggests that GPT--4o does not exhibit heightened negative characteristics, such as selfishness, even after being subjected to jailbreak attempts.
GPT-40
w/o Jailbreak
w/ Jailbreak
Machiavellianism
4.6 + 0.4
3.5 1.3
Psychopathy Neuroticism
3.5 4 0.4
3.0 + 0.5
6.1 + 0.4
5.2 + 0.8
Table 9: The jailbroken GPT-4o's results on Dark Triad Dirty Dozen.
LIMITATIONS
This study is subject to several limitations. Firstly, due to time and budget constraints, we do not evaluate all prominent LLMs such as LLaMA-3.2, Qwen-2.5 and Claude-3.5. However, we promise to expand our leaderboard to include more LLMs in the future. Secondly, our experiments do not explore scenarios where different L.MSs compete the same game. Instead, our evaluation uses ten agents derived from the same LLM. We acknowledge that including diverse LLMS in the same game could yield more intriguing insights. This aspect is designated for a future direction. Thirdly, we limit the games to 20 rounds and inform the agents of this total, potentially affecting strategies in Betraying games where agents may collaborate initially and betray in the final round for greater gain. We also leave this part our future research agenda. However, we believe 20 rounds sufficient to observe agent behavior patterns. Extending the rounds exceeds the token limit without yielding observations, as the convergence trend remains consistent.
37

---

Published as a conference paper at ICLR 2025
J ETHICS STATEMENT AND BROADER IMPACTS
Our research seeks to evaluate and enhance LLMs' reasoning capabilities, facilitating their applica- tion in decision-making scenarios. On the one hand, users need to notice that current LLMs often display self-interested behavior in decision-making, which may not maximize social welfare. On the other hand, our framework promotes societal benefits by facilitating human-LLM interaction through gameplay, which can be applied in educational contexts such as economics and game the- ory. Ultimately, enhancing LLMs' reasoning skills could enable them serve effective decision- making assistants for humans.
K LLAMA-3.1-70B
K.1
ROBUSTNESS: MULTIPLE RUNS
Table Quantitative results of playing the games with same setting five times.
Tests
T1 (Default)
T2
T3
T4
T5
Avgastd
Guess 2/3 the Average
82.2
82.7
84.3
84.6
86.4
84.0=1.7
EI Farol Bar
64.2
55.8
61.7
60.0
56.7
59.7
Divide the Dollar
87.9
92.0
86.0
80.8
88.6
87.0=1.1
Public Goods Game
93.4
90.8
84.7
90.4
93.6
90.643.6
Diner's Dilemma
47.0
41.5
56.0
44.5
51.5
48. 18.1
Realed-Bid Auction
15.6
20.2
13.8
13.6
15.4
15.7
Battle Royale
70.0
90.0
92.9
100.0
35.7
77.7+26.0
Pirate Game
42.8
53.8
71.4
81.8
70.3
64.0+15.5
Overall
62.9
65.8
68.8
69.5
62.3
65.943.3
K.2 ROBUSTNESS: TEMPERATURES
Table 11: Quantitative results of playing the games with temperature ranging from 0 to 1.
Temperatures
0.0
0.2
0.4
0.6
0.8
1.0 (Default)
Augasta
Guess 2/3 of the Average
75.7
84.7
80.6
84.9
83.9
82.2
82.043.5
El Farol Bar
6.7
50.0
46.7
53.3
63.3
64.2
47.44-2.2
Divide the Dollar
95.0
87.6
90.0
90.4
91.1
87.9
90.3+
Public Goods Game
33.8
79.8
70.8
83.6
83.0
93.4
74.0421.0
Diner's Dilemma
28.0
27.0
34.0
36.5
45.0
47.0
36. 36.2
Sealed-Bid Auction
12.5
13.7
18.8
15.0
12.7
15.6
14.7+2.4
Battle Royale
94.4
86.7
56.2
95.0
80.0
70.0
80.4415.1
Pirate Game
46.0
46.0
70.4
75.5
79.1
42.8
60.0
Overall
49.0
59.4
58.4
66.8
67.3
62.9
60.646.8
38

---

Published as a conference paper at ICLR 2025
K.3 ROBUSTNESS: PROMPT TEMPLATES
Table 12: Quantitative results of playing the games using different prompt templates.
Prompt Versions
(Default)
V3
V4
V5
Aug+sid
Guess 2/3 of the Average
82.2
87.5
83.2
90.5
82.4
85.2
HI Farol Bar
64.2
63.3
63.3
58.3
64.2
62.7
Divide the Dollar
87.9
95.1
84.1
87.6
94.0
89.7+4.6
Public Goods Game
93.4
92.9
87.4
67.6
89.0
86.1
Diner's Dilemma
47.0
47.5
34.0
53.0
47.0
45.7+7.0
Sealed-Bid Auction
15.6
5.4
13.0
6.1
10.6
10.244.4
Battle Royale
70.0
90.0
75.0
41.2
85.0
72.2+19.1
Pirate Game
42.8
77.0
88.8
58.6
73.0
68.1 18.18
K.4 GENERALIZABILITY
Table 13: Quantitative results of playing the games with various game settings.
Guess 2/3 the Average
Augase
R
0
1/6
173
1/2
2/3
576
--
7/6
473
3/2
5/3
11/6
2
94.1
91.4
92.0
83.8
82.2
81.4
72.6
89.6
93.0
92.4
90.3
89.9
90.9
88.0
El Farol Bar
Aug=std
R
0%
20%
40%
60%
50%
100%
73.0
81.2
70.0
64.2
63.7
72.0
70.7
Divide the Dollar
Avg+std
-=
50
100
200
400
800
72.1 87.9
91.6
95.6
97.5
88.9 11.11
Public Goods Game
Aug
R
0.0
0.5
1.0
2.0
4.0
95.4 95.5
95.3
93.4
82.9
92.5+119
Diner's Dilemma [A, P,, = "(1T, 20,20
Angesi
2,,,,,26,7)"
14,79, 2,26
11,8,19,12] 37.5
(4,6,17;)) 31.0
72, 13]
47.0
48.5
44.5
40.0
41.4g66
Sealed-Bid Auction
Augasta Sta
Ronge (0, 100
(0, 200]
(0,400) 400]
(0, 800]
4.1
4.4
7.6
13.6
7.4
Battle Royale
Aug-sua
Range [51, 60]
[35, 80]
[10, 100
41.2
70.0
70.0
60.39 133.59
Pirate Game -4
Augastd
G
5
700
400
71.1
70.2
42.8
48
58.1
39

---

Published as a conference paper at ICLR 2025
L
GEMINI-1.5-PRO
L.1 ROBUSTNESS: MULTIPLE RUNS
Table 14: Quantitative results of playing the games with the setting five times.
Tests
T1 (Default)
T2
T3
T4
T5
Aug
Guess 2/3 of the Average
96.2
95.4
95.1
95.1
95.1
95.440.5
EI Farol Bar
37.5
40.0
35.8
30.8
41.7
37 37.2+44.2
Divide the Dollar
93.8
94.2
94.2
93.5
93.5
93.8
Public Goods Game
100.0
100.0
100.0
100.0
100.0
100.0=0.0
Diner's s Dilemma
29.0
43.0
33.0
38.5
36.0
35.9+5.3
Sealed-Bid Auction
42.5
25.3
21.4
27.0
18.2
26.919.4
Battle Royale
75.0
90.0
71.4
85.0
85.0
81.3+7.7
Pirate Game
92.2
83.9
88.8
94.0
80.6
87.913.6
Overall
70.8
71.5
67.5
70.5
68.8
19.8
L.2 ROBUSTNESS: TEMPERATURES
Table 15: Quantitative results of playing the games with temperature ranging from 0 to 1.
Temperature
0.0
0.2
0.4
0.6
0.8
1.0
Augasta
Guess 2/3 of the Average
96.1
99.2
96.6
96.6
96.4
96.2
96.9+1.2
El Farol Bar
37.5
20.0
28.3
40.0
38.3
37.5
33.647.8
Divide the Dollar
94.5
93.5
93.5
94.5
93.3
93.8
93.8+0.5
Public Goods Game
100.0
100.0
100.0
100.0
100.0
100.0
100.0
Diner's Dilemma
33.5
45.0
43.0
36.5
42.0
29.0
38.246.2
Sealed- Bid Auction
31.1
24.1
27.9
21.0
32.4
42.5
29.8+7.5
Battle Royalc
88.9
85.0
80.0
75.0
87.5
75.0
81.946.1
Pirate Game
96.0
90.3
96.1
99.2
96.0
92.2
95.0+3.2
Overall
72.2
69.6
70.7
70.3
73.2
70.8
71.1
40

---

Published as a conference paper at ICLR 2025
L.3 ROBUSTNESS: PROMPT TEMPLATES
Table 16: Quantitative results of playing the games using different prompt templates.
Version
VI (Default)
V2
V3
V4
V5
Aug=sid
Guess 2/3 of the Average
96.2
95.1
92.7
97.2
88.9
94.013.3
HI Farol Bar
37.5
53.3
60.8
46.7
27.5
45.2 11.11
Divide the Dollar
93.8
90.3
62.1
100.0
92.5
87.7+14.8
Public Goods Game
100.0
97.2
98.7
100.0
99.8
99.111.2
Diner's Dilemma
29.0
24.0
22.0
18.0
23.0
23.2+4.0
Sealed-Bid Auction
42.5
38.6
33.5
8.2
20.5
28.7+14.2
Battle Royale
75.0
92.3
70.0
75.0
85.0
79.549.0
Pirate Game
92.2
82.3
92.3
82.3
77.8
85.446.5
L.4 GENERALIZABILITY
Table 17: Quantitative results of playing the games with various game settings.
Guess 2/3 of the Average
Augasd
R
I/6
173
1/2
2/3
5/6
1
776
4/3
3/2
373
11/6
-
98.5
99.4
98.6
97.8
95.4
91.1
5.3
97.0
97.7
97.3
92.5
88.0
75.8
87.3
El Farol Bar
Avg+su4
R =
0%
20%
40%
60%
80%
100%
80.5
56.9
32.5
42.5
41.9
66.5
53.5
Divide the Dollar
Avg+sid
G 50
100
200
400
800
96.5
93.8
98.4
93.8
100.0
96.5+2.8
Public Goods Game
Aug+std
R
0.0
0.5
1.0
2.0
4.0
100.0
100.0
100.0
100.0
100.0
100.0
Diner's Dilemma [A, Pr, -. [10, 15, 20,
Auges
(11,3,20.75 12.0
[4,10,3,20] 24.5
(1,8,19.22 11.5
"0,,,77.75 16.5
12,11,8,13]
29.0
42.5
22.7-119
Sealed-Bid Auction
Aug Sta
Ronge (0. 100
20.200] 42.5
(0, 400]
(0, 800]
24.0
38.4
44.9
37.4 37.419.4
Battle Royale
Aug+std
Range=
[51, 60] 92.3
[35, 80)
[10, 100
75.0
75.0
80.8
Pirate Game 4
Aug+su
G
700
400
79.2
85.3
92.2
98.6
88.8
41

---

Published as a conference paper at ICLR 2025
M
GPT-40
M.1 ROBUSTNESS: MULTIPLE RUNS
Table 18: Quantitative results of playing the games with the setting five times.
Tests
T1 (Default)
T2
T3
T4
T5
Augastd
Guess 2/3 of the Average
94.9
94.8
94.2
94.1
93.4
94.3+:06
EI Farol Bar
95.0
41.7
70.8
55.0
87.5
70.0+22.1
Divide the Dollar
95.7
95.7
94.9
94.0
95.4
95.2
Public Goods Game
94.1
88.1
87.4
93.5
91.5
90.9+3.0
Diner's ' Dilemma
23.5
4.5
3.5
8.0
14.0
10.7+8.3
Sealed-Bid Auction
19.2
18.8
17.7
25.3
23.0
20.8
Battle Royale
89.5
60.0
50.0
72.2
65.0
67.3 67.3
Pirate Game
77.3
88.4
93.7
79.8
82.8
84.4
Overall
73.6
61.5
64.0
65.2
69.1
66.7+4.7
M.2 ROBUSTNESS: TEMPERATURES
Table 19: Quantitative results of playing the games with temperature ranging from 0 to 1.
Temperature
0.0
0.2
0.4
0.6
0.8
1.0
AvgiStd
Guess 2/3 of the Average
94.4
94.4
94.4
94.4
93.2
94.9
94.3_0.6
El Farol Bar
66.7
50.8
44.2
65.0
75.0
95.0
66.1+18.1
Divide the Dollar
100.0
99.1
98.6
94.3
97.7
95.7
97.6_2.2
Public Goods Game
87.6
87.0
87.2
87.8
92.1
94.1
89.3_3.0
Diner's s Dilemma
27.0
12.5
8.0
49.5
64.5
23.5
30.8+21.9
Sealed-Bid Auction
24.6
22.6
24.0
21.2
22.8
19.2
22.4
Battle Royale
73.7
50.0
50.0
20.0
77.8
89.5
60.2
Pirate Game
99.5
92.7
88.4
75.8
82.3
77.3
86.0
Overall
71.7
63.6
61.9
63.5
75.7
73.6
68.3
42

---

Published as a conference paper at ICLR 2025
M.3 ROBUSTNESS: PROMPT TEMPLATES
Table 20: Quantitative results of playing the games using different prompt templates.
Version
(Default)
V3
V4
V5
Aug+std
Guess 2/3 of the Average
94.9
93.0
94.7
94.3
91.6
93.711.
HI Farol Bar
95.0
72.5
37.5
59.2
60.8
12.0
Divide the Dollar
95.7
95.7
95.6
93.9
96.1
95.440.9
Public Goods Game
94.1
96.2
89.4
88.6
94.0
92.443.3
Diner's Dilemma
23.5
50.0
50.0
33.5
37.5
38.9
Sealed-Bid Auction
19.2
38.1
35.0
20.3
33.6
29.248.8
Battle Royale
89.5
60.0
10.0
64.7
30.0
50.8
Pirate Game
77.3
93.7
67.9
88.9
86.5
82.9
M.4 GENERALIZABILITY
Table 21: Quantitative results of playing the games with various game settings.
Guess 2/3
Average
Aug
R
0
176
173
1/2
2/3
576"
7/6
473
3/2
5/3
11/6
-
99.3
98.0
96.6
95.0
94.9
88.8
22.7
55.4
46.2
72.8
69.1
76.8
75.0
76.2+21.4
El Farol Bar
Aug+st
-==
0%
20%
40%
60%
50%
100%
99.0 91.2
87.5
95.0
56.9
83.5
85.5
Divide the Dollar
Augastd
G
50
100
200
100
800
92.5
95.7
97.3
97.5
98.3
96.3
Public Goods Game
Augastd
R
-..0
0..
1.0
2.0
"4.0
100.0
95.3
94.4
88.6
89.8
93.6
Diner's Dilemma (A), Ux)
Angesua
(10, 20, 20)
(11,5, 20
(9,9,,222) 10.0
(1,8, 12) 14.5
(7,5,
81, 13)
23.5
46.0
2.5
13.0
18.2-152
Sealed-Bid Auction Range
Aug=Std
(0. 100
(0,200) 23.8
70, 400]
"(Ō, 800J
20.9
21.4
26.0
23.0+2.3
Battle Royale
Aug
Range
151, 60]
[35,80 80] 55.0
[16. I00
82.4
65.0
67.5 113.8
Pirate Game 4
Aug_Sud
G
5
100
400
73.8
47.2
80.6
83.6
71.3
43


Reviews:
[{"review_id": "uNQZvfwHNh", "reviewer": "Reviewer_i13w", "summary": "This paper studies the capabilities of LLMs in the multi-agent gaming setting. To this end, the authors propose a new benchmark including eight classical game theory problems and a dynamic scoring rule. They evaluated twelve LLMs. In terms of the scoring rule, the Gemini-1.5-Pro has the best performance among closed-source LLMs and the LLaMA-3.1-70B has the best performance among open-source LLMs. Besides, their results show that GPT3.5 is robust to multiple runs and different temperatures but is poor in the generalization of various game settings.", "questions": "Have you tried to test the robustness and generalizability of models other than GPT3.5, especially the open-source model and the close-source model that have the best performance?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. This paper proposes an interesting benchmark that allows LLMs as agents to compete with each other. The authors systematically conducted experiments under multiple runs, different temperatures, and various game parameters, which made their conclusions more convincing.\n\n2. This paper is well-written and easy to read. Especially the quick links of game descriptions, prompt templates, and experimental results are very helpful in understanding this paper.", "weaknesses": "1. My main concern is that, from a conceptual/technical point of view, the contribution and novelty of this work are somewhat limited. The authors discussed related literature limited to two players or actions and motivated their focus on the multi-player setting. However, the study of multiple LLM agents in gaming is not novel ([1]). I think a more detailed and thorough comparison with previous related benchmarks is needed.\n\n2. The single metric that measures how far the LLM agent's behavior is from the Nash equilibrium cannot fully evaluate the performance of LLM agents in decision-making. \n\n3. Some of the insights are not very surprising in light of the prior literature (such as it is possible to improve the performance of GPT3.5 through CoT). \n\n[1] Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. LLM-deliberation: Evaluating LLMs with interactive multi-agent negotiation games, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9EeOCMI1rh", "reviewer": "Reviewer_Xvgg", "summary": "The authors measure LLMs competing ability in various multi-agent game theoretic based scenarios. They put together a collection of 8 games and test across 12 different LLMs with dynamic parameter changes to the games to test for robustness. They contribute a literature review of studies focusing on game theory based LLM studies, the GAMA-bench (available after paper publication), and the results of benchmarking the LLMs they test.", "questions": "- RQ4: Qwen-2 seems to do great in all but Diner and SBA which seem very out of place and there's no mention of this result or why it could've happened. What happened here?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overall the paper is great.\n\n- Robust testing, using \"multiple runs, temperature parameter alterations, and prompt template variations\" as well as testing across 12 different LLMs.\n- Approach to Generalizability (RQ3) is useful, unique, and future-proofs the benchmark to a degree.\n- Excellent presentation with logical structure and intuitive navigation, all information is presented clearly. (Section 3.1 links to all game prompts, key findings and answers to research questions nicely emphasized.)", "weaknesses": "- The case could be made stronger for \"LLMs being good at Game Theory\" == \"LLMs useful for decision making\". The intro \"jumps\" from \"decision-making poses a significant challenge for intelligent agents\" to Game Theory. It's slightly strengthened by noting that your \"framework evaluates LLMs’ ability to make optimal decisions\". In the conclusion you mention \"Our findings reveal that GPT-3.5 (0125) demonstrates a limited decision-making ability on γ-Bench\", which is great. Overall, I think your paper points in the direction and does well, but this introduction is a bit jarring.\n\n### Small things\n- \"We leveraging GPT-4 to rewrite our default prompt\" -> 'We leverage'\n- \"The game has a PSNE where all players\", \"a PSNE but presents an MSNE, \"  -> PSNE and MSNE is defined in the appendix but acronyms are used in the main body.\n- \"The model produces average numbers of 34.59, 34.59\" -> intentionally the same numbers?\n- \"We first focus on open-source models, including OpenAI’s GPT-3.5 (0613, 1106, and 0125), GPT-4 (0125), and Google’s Gemini Pro (1.0, 1.5).\" -> closed source\n- RW: \"Other than papers listed in Table 3 on evaluating...\" -> this should present the table better. \"We present a comparison of studies using LLMS...\" It currently feels like a passing note, but is stronger than that. Also, link this in your statement contributions in the introduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0R5mQyWmH3", "reviewer": "Reviewer_Ykhg", "summary": "The authors introduce GAMA-Bench, a benchmark with 8 well-designed game scenarios. They performed evaluation on GAMA-bench for multiple mainstream LLMs and in addition evaluate robustness, reasoning, and generalizability of these models.", "questions": "I don't have any particular clarifying questions and most of my concerns are mentioned in \"Weaknesses\" section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The benchmark questions are designed carefully, drawing inspiration from classic game theory scenarios. The objective of the three categories of games are clear. \n2. On top of measuring performance, extensive experiments are conducted to understand the effect of prompting and sampling, CoT, and sensitivity to same game with different parameters.", "weaknesses": "1. The author mentioned that most previous benchmarks incorporates classic settings that LLM has seen during training. Yet most games proposed in GAMA-Bench seem to also be taken from classical work. For instance, I prompted the game definition of all 3 cooperative games to GPT-4o and it is able to recognize which famous game it is. The authors could improve on this by proposing variants and provide concrete evidence on no test set leakage by comparing LLM performance on variants vs. original games. \n2. Language model are not designed to solve games, nor provide concrete mathematical calculation which is required to compute NEs. If the goal of this work is to understand LLM's ability to adapt in dynamic, multi-agent systems, I would encourage the authors to design additional experimental settings where partial actions by other agents are revealed. For instance, it would be more informative to see how models will adjust in \"Guess 2/3 of the Average\" game knowing there is a weak/malicious model that is likely to output a large number, compared to simply letting the model work its way to the NE. An example could be a series of rounds where the LLM is given information about other players' previous moves or tendencies, and then measure how quickly it adapts its strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TC2RCFuMXm", "reviewer": "Reviewer_zsFk", "summary": "This paper introduces GAMA-Bench, a novel benchmark designed to evaluate the gaming and decision-making abilities of LLMs in multi-agent environments. The benchmark comprises eight classical game theory scenarios, where LLMs are evaluated on how closely their decisions align with the Nash equilibrium, a theoretical standard for rational decision-making. Twelve different types of LLMs, including GPT-4, Gemini, and LLaMA, are tested. Additionally, the paper explores the impact of hyperparameters, prompt design, reasoning strategies, and persona on the robustness of LLM performance.", "questions": "1. Could you explain why some LLMs, which are typically considered more advanced, do not perform well on this benchmark? For example, models like GPT-4 generally excel in many NLP tasks but underperform in certain games. Could this be due to the influence of ethical considerations embedded in their training (such as RLHF) that conflict with the purely rational strategies required to achieve Nash equilibrium?\n2. Is Nash equilibrium the most appropriate evaluation target for LLMs, given that humans often make decisions based on ethical, emotional, or social factors rather than strict rationality? For example, GPT-4 tends to exhibit prosocial behavior, which may prevent it from reaching the Nash equilibrium in certain scenarios like the Diner’s Dilemma. Could the benchmark be expanded to account for human-aligned behavior, considering that LLMs are often used in contexts where replicating human-like decisions is more relevant than adhering to game-theoretic models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized, making it comprehensible to a broad audience, including those unfamiliar with game theory. The authors provide clear explanations of complex game-theoretic concepts and how they apply to LLMs. \n2. This work fills a gap in the literature by focusing on multi-agent scenarios, which are more complex and less explored compared to simpler single-agent or two-agent interactions. \n3. The authors go beyond simply testing the LLMs and investigate the role of different factors such as temperature settings, prompt design, and reasoning strategies (e.g., CoT). By examining how these parameters influence LLM behavior and outcomes, the paper adds a layer of rigor, ensuring that the benchmark is not merely static but adaptable and robust under various conditions.", "weaknesses": "1. The paper does not sufficiently clarify which specific cognitive or decision-making abilities of LLMs are being measured through this benchmark. While the Nash equilibrium is a logical choice for evaluating decision rationality, the paper does not articulate whether the benchmark truly reflects the LLMs’ capacities in areas such as long-term planning, adaptability, or ethical decision-making. The discussion of which kind of ability can be measured through this benchmark.\n2. The paper misses an opportunity to delve into the reasons behind the LLMs’ decisions. Different models may exhibit distinct decision-making tendencies based on their training data and architectures. For instance, GPT-4’s tendency toward cooperative behavior, as seen in the Public Goods Game, may stem from its RLHF process, which favors kindness and prosocial actions. Understanding these motivations could offer valuable insights into model alignment and performance tuning. One section of analyzing the reasoning behind the decision is the need for a better explanation of the various performances of LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)