GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations
Authors: Jinhao Duan
Renming Zhang
James Diffenderfer
Bhavya Kailkhura
Lichao Sun
Elias Stengel-Eskin
Mohit Bansal
Tianlong Chen
Kaidi Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Game Theory
Strategic Reasoning
Benchmark
Primary area: evaluation

Abstract: As Large Language Models (LLMs) are integrated into critical real-world applications, their strategic and logical reasoning abilities are increasingly crucial. This paper evaluates LLMs' reasoning abilities in competitive environments through game-theoretic tasks, e.g., board and card games that require pure logic and strategic reasoning to compete with opponents. We first propose GTBench, a language-driven environment composing 10 widely-recognized tasks, across a comprehensive game taxonomy: complete versus incomplete information, dynamic versus static, and probabilistic versus deterministic scenarios. Then, we (1) Characterize the game-theoretic reasoning of LLMs; and (2) Perform LLM-vs.-LLM competitions as reasoning evaluation. We observe that (1) LLMs have distinct behaviors regarding various gaming scenarios; for example, LLMs fail in complete and deterministic games yet they are competitive in probabilistic gaming scenarios; (2) Most open-source LLMs, e.g., CodeLlama-34b-Instruct and Llama-2-70b-chat, are less competitive than commercial LLMs, e.g., GPT-4, in complex games, yet the recently released Llama-3-70b-Instruct makes up for this shortcoming. In addition, code-pretraining greatly benefits strategic reasoning, while advanced reasoning methods such as Chain-of-Thought (CoT) and Tree-of-Thought (ToT) do not always help. We further characterize the game-theoretic properties of LLMs, such as equilibrium and Pareto Efficiency in repeated games. Detailed error profiles are provided for a better understanding of LLMs' behavior. We hope our research provides standardized protocols and serves as a foundation to spur further explorations in the strategic reasoning of LLMs.

GTBENCH: Uncovering the Strategic Reasoning
Limitations of LLMs via Game-Theoreti Evaluations
Jinhao Duan't Renming Zhang?* James Diffenderfer' Bhavya Kailkhura
Lichao Sun Elias Stengel-Eskin' Mohit Bansal Tianlong Che7 Kaidi Xu
Drexel University Boston University SLLNL *Lehigh University UNC Chapel Hill MIT Harvard University
Abstract
As Large Language Models (LLMs) are integrated into critical real-world applica- tions, their strategic and logical reasoning abilities are increasingly crucial. This paper evaluates LLMs' reasoning abilities competitive environments through game-theoretic tasks, c.g., board and card games that require pure logic and strategic reasoning to compete with opponents. We first propose GTBENCH, language- driven environment composing 10 widely-recognized tasks, across comprehensive game taxonomy: complete versus incomplete information, dynamic versus static, and probabilistic versus deterministic scenarios. Then, we 1 Characterize the game-ther theoretic reasoning of LLMs; and @ Perform LLM-vs.-LLM. competitions reasoning evaluation. We observe that O LLMs have distinct behaviors regarding various gaming scenarios; for example, I.LMs fail in complete and determinis- tic games yet they are competitive in probabilistic gaming scenarios; @ Most open-source LLMs, c.g., CodeLlaam-334--nnttn competitive than commercial LLMs, e.g., GPT-4, in complex games, yet the cently released -aemaa----b-Isstiiiie makes up for this shortcoming, addition, code telerrretainnngg greatly benefits strategic reasoning, while advanced reasoning methods such as Chain-of- -hhou-tf-Thought (CoT) and Tree-of-Though (ToT) do not al- ways help. We further characterize the game-theoretic properties of LLMs, such equilibrium and Pareto Efficiency in repeated games. Detailed error profiles are provided for better understanding of L.MMs' behavior. We hope research pro- vides standardized protocols and serves as foundation to spur further explorations in the strategic reasoning of L.MSs.
and L-ama-2-70b-chat, are less
Introduction
Large Language Models (LLMs) are increasingly being integrated into critical real-world applications, such as cybersecurity (Ameri et al., 2021; Aghaei et al., 2022), decision science (Jiang et al., 2023b), and finance (Wu al., 2023). These areas involve advanced strategic thinking and logical reasoning skills, including the ability to foresee possible dangers and weaknesses (Yao et al., 2024b; Duan et al., 2024a), systematically examine difficulties, and make informed decisions based on provided evidence. However, evaluation environments that thoroughly assess these situations are not sufficiently explored.
There has been emerging trend where LLMs evaluated in various interactive role-playing envi- ronments, including collaborative environments such as CAMEL (Li et al., 2023), ReConcile (Chen al., 2023), and competition environments such Diplomacy (Bakhtin et al., 2022). Werewolf (Xu et al., 2023a), Avalon (Light et al., 2023; Stepputtis et al., 2023), multi-agent debate (Liang et al., 2023; Du et al., 2023; Chan al., 2023; Xiong al., 2023), board and card games (Duan al.,
Equal contribution.
t Correspondence to: Tianlong Chen tianlongdmit. edu, Kaidi Xu kx460drexel.edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Prompt Adapter
Environments
Corplete and Debernenvii Qurritg
Observation Pronspt
Participant 1
iilor olged
HNur Plyyerr
Tike Frandee
warch
Oeseriation Prompt
Nan
Bremhhrrugh
Apre
Incorplett or Probeteivtic Garinn
Responas to Action
Find Aucticn
Kieas Pubei
Frem
KG355+ -C300
welach
d and
Baroad Prissna's Dilumans
Naguriation
(4)
Figure I: The overall schematic of GTBENCH. There are three main components from right to left: Environments (c) for game hosting, observation providing, and action execution; Prompt Adapter (b) for converting observation to prompt and extracting actions from participants' generations; Participants (a) for reasoning and action generation.
2024b). By engaging LLMs in simulated scenarios, role-playing-basee environments offer useful potential for analyzing the cognitive reasoning abilities of LLMs. However, the extensive background and intricate details involved in role- aale-pay-bassd games dilute the pureness of logic and strategic reasoning that is typically found in same-theoretic tasks. Additionally, the evaluation is primarily verbal as it hinges on spoken or written exchanges between the LLMs. This could mask instances where LLMs might lack concrete reasoning abilities but navigate the scenario effectively through the proficient use of language.
Why are game- tame-theoretic tasks unique and necessary for LLM reasoning evaluation? Game- theoretic tasks are typically conceptualized based on prevalent trade-offs and dilemmas manifesting in real-life scenarios and are designed to be easy to understand yet require difficult skills to be mastered. In contrast to the rich narrative contexts afforded in verbal- or role-playing- srasingg-baeedddd games, e.g., Werewolf (Xu et al., 2023a) and Avalon (Light et al., 2023), the reality of game-theorete games such as Chess and Go involve: pure logic and strategic reasoning without the added complexity of backgrounds character roles; @ embracing rigorous rules with well-defined action/state space, which allow for an in-depth examination of the strategic reasoning of LLMS.
Hence, in order to spur more research in the LLM Came-Theoretic evaluation domain, propose GTBENCHI an environment consisting of 10 widely recognized game-theoretic tasks, across a com- prehensive taxonomy of games, e.g., complete- (Tic-Tac-Tode Connect-4, Breakthrough) versus incomplete-information (Kuhn Poker, Liar's Dice) gaming, deterministic (Nim) versus proba- bilistic ENegotiation Pig) gaming, static versus dynamic (Iterated Prisoner's Dilemma, Blind Auction) gaming. These environments require a variety of abilities including board strategy, collaboration, auction, and bidding. There two key issues investigated in this paper:
Characterizing Strategic Reasoning of LLMs: How LLMs will perform when facing various game- theoretic scenarios? How do they perform compared conventional solvers? How essential factors, e.g., pertaining, parameter sizes, and reasoning methods, affect strategic reasoning?
L.LM-vs.-I.LN Competitions as New Reasoning Evaluation: A new automated and adaptive bench- mark that can be effective in evaluating reasoning errors even for future L.MMs.
To address these crucial problems, we conduct experiments over two configurations: (a) LLM- os-Conventional where conventional solvers such as optimization- or search-based solvers, e.g..
Monte-Carle Tree Search (MCTS) (Chaslot et al., 2008), are taken as the opponent of LLMs; (b) L.LM-vs.-LLM where two LLMs compete directly to reveal the reasoning limitations an automated manner. We find that: O LLMS almost always fail when playing against simple MCTS opponents in complete and deterministic gaming scenarios (Section 4.1), while @ LLMS remain competitive in incomplete and probabilistic scenarios (Section 4.2); O Code-pretraining benefits game-theoretic

---

reasoning, e.g., CodeLlama- Ins-tuntt (Roziere et al., 2023) achieves comparable results as GPT- 3.5-turbo, and significantly outperforms (lama-2-70b-chan (Touvron et al., 2023) (Section 4.3); o Advanced reasoning methods, such as Chain-of-Thouggh (Co'T) (Wei et al., 2022), Self-Consistent CoT (SC-CoT) (Wang et al., 2022b), Tree-of-Thought (ToT) (Yao al., 2024a) are not always helpful; @ Most open-source LLMs are less competitive than commercial LLMs in games with complex rules and large action/state space, while the recently released 1aama-3-70b-Instrree (Meta, 2024) makes up for this shorteoming. The interfaces of GTBENCHI leaderboard can be found in Appendix A11. Our contributions can be summarized as the following:
LLM Gaam-TThorree Evaluation (GTBENCH); An LLM environment supporting 10 well- recognized tasks across comprehensive game-theorete theoretic taxonomy, is presented to spur future work for the community. The code and leaderboard will be public and continuously updated for future reasoning agents and LLMs.
We investigate how essential factors, e.g., parameter size, code-pretraining. and reasoning methods, affect strategic reasoning. A detailed error profile is provided for a better understanding of LLMs' behaviors.
Essential Factors for the Strategic Reasoning of LLMs:
Characteriz the Saae-TThorettic Properties of LLMs: We characterize distinct LLM behav- iors when facing different game-theoretic scenarios, such as LLMs fail in complete-information and deterministic gaming yet remain competitive in probabilistic gaming. We further study the equilibrium and Pareto efficiency during the gameplay.
2 Background and Problem Definition
2.1 Background and Related Work
LLM-aa-Agent Evaluation. Several studies have been conducted to measure the effectiveness of LLMs as agents in recent years. Hausknecht et al. (2020) carried out extensive study to evaluate the performance of LLMs interactive fiction games. Zhu et al. (2023) provides a valuable dataset for finetuning LLMs to improve usefulness in the strategic game Dungeons & Dragons. GRUE (Ramamurthy et al., 2023) uses reinforcement learning-basee based metrics to benchmark the performance of generation tasks in six different languages. Gandhi et al. (2023) test the use of LLMs as a broker with human contestants in the negotiation game "Deal or No Deal". A few studies have explored the use of text-based games ALFWorld (Shridhar et al., 2020) introduced a novel virtual environment that allows agents to acquire learning in text- environment while executing in a visual environment. The environment was developed in conjunction with Building Understanding in Text world via Language for Embodied Reasoning (BUTLER) agent, which can acquire abstract text knowledge in the text world. Similarly, Worl (Côté et al., 2019) is introduced as an environment that enables RL agents to play text games. Wang et al. (2022a) proposed ScienceWorld, reasoning ability, and their findings showed that transformer- reasoning in novel contexts. MTBench (Zheng et al., 2024) introduces LLM-as-a-Judee where GPT-4 (Achiam et al., 2023) is utilized as judge to evaluate the quality of LLM generations. It indicates that GPT-4 shares close criteria as humans. There have been works evaluating LLMs in solving weal-world tasks, such as graph reasoning (Besta et al., 2023), WebShop ( (Yao et al., 2022), AgentBench (Liu et al., 2023) for pragmatic missions, MINT (Wang et al., 2023b) for tool utilization.
means of facilitating learning in such environments.
benchmark used for evaluating agents'
models are not effective at
Multiple LLMss-asAgents in Gaming, A key research area is the competition and collaboration between L.Mss. Many studies examine LLMs' strategic reasoning and performance, using evaluation frameworks to assess multiple LLM agents in individual games, such as: Social deduction or deception games (Xu et al., 2023a.b; O' O'Gara, 2023; Light et al., 2023), diplomacy games (Mukobi et al., 2023; FAIR), negotiation games (Abdelnabi et al., 2023; Davidson et al., 2023), coordination and cooperation games (Akata al., 2023), and Minecraft (Gong et al., 2023; Wang et al., 2023a; Fan et al., 2022). These works not only provide evaluation frameworks for games and demonstrate the flexibility of LLMs to variety of gaming tasks but some provide meaningful datasets for fine-tuning, policies for reinforcement learning to produce better strategies, or evaluate the strategic reasoning of LLMs. However, many of these standalone works quantify either individual or a subset of desirable strategic reasoning capabilities of LLMs,, such as negotiation, deception, or coordination. Further, they often evaluate these capabilities for L.MS using one or two games which may produce less robust assurances of LLM abilities.
3

---

Table 1: Game environments explored in GTBENCH.
Taxonomy of Games
Preferred Ability
Zeno-
First-player Advantage
Complete Incomplete
Dynamic
Probubulistic
Stralogy
Collaboration
Bluff
Tic Tac Toe
X
X
Donsect-4
K
K
X
x
X
V
Breakthrough
X
V
X
X
Liar's Dice
X
K
X
V
ind Auction
X
X
X
V
X
X
Negotiation
X
V
V
Vim
X
X
2ig Prisoner'e
X
X
X
X
X
X
X
/
X
Dileema
1: Breakdhough has slight llay-r advantage which is not as significant as others.
4 The ilerated vansoon of Prisoner's Dilemma allows puricipunts the action made by their opponents in rounds, achiesing implici collaboration.
H1 Imppliicable comples combiratioon dynamic
We make an additional crucial contribution this line of work by measuring strategic reasoning capabilities with games that such as clembench (Chalamalasetti et al., 2023) focusing on conversational agents over non-zero-sum games and LMRL-Gym (Abdulhai et al., 2023) verbal reinforcement learning tasks. (Chen et al., 2024) and Duan et al. (2024b) also proposes multi-agent strategic reasoning evaluation. However, they overlooked the analysis of L.M behaviors response different game-theoretic scenarios and their associated properties. Differently, GTBENCH seeks to provide unified suite of games that are carefully curated to (1) evaluate comprehensive collection of strategic reasoning abilities for a given agent and (2) enable competition-base scenarios (i.e., LLM agent-1 vs LLM agent-2) allowing for competition-bai based comparisons strategic reasoning capabilities by LLM-based agents.
found in the existing unified benchmark suites (Zhang al., 2024),
2.2 Problem Definition
Notation: Gameplay. We formulate the gameplay as a Markov Decision Process AS, A,M, M.O) O) under given game environment, among the alternating interaction of two participants. This process composes of an infinite state space S, an infinite action space AA the participants M = (M1,M2)) and observation space O. Considering the decision of M, (i 1,2) the 1-th step of the process, we denote by & E S the state that M, are placed and o E O the observation that M, are observing. We assume M; follows policy ne, (ar]st, (;) for state transition T
A S, where a E As the action sampled by тe, under conditions 8; and or. 0; is determined by the implementation by M,, e.g., optimization-biann based solver, LLM-driven agents, which will be discussed in Section 3.2 in detail. In this way, the two-participate gameplay can be represented as (8u, Gg, 41 82, so is the initial state and s,, is a terminal state, i.e., end of the game. The progress is driven by the alternating execution of actions sampled by participants. Please refer to Section 3.1 and Appendix A2 for all the supported games with the corresponding actions and observations.
sn)). where
Evaluation Metric: Normalized Relative Advantage. We introduce Normalized Relative Advan- tage (NRA), denoted (RA(M,,MM) fe). to measure to relative advantage of M, when competing against Mo, under the score calculation fs:
Em fa(M.. - E.,. fa(Mo: m)
NRA(Mi,
Em feM, m) + En. fa(Mo: m))
where fMM, m) refers to the score earned by M, at the m-th match (1 S S K, K is the number performed matches):
For zero-sum games, e.g., Tic-Tac-Toe.
1, M, wins at the m-th match
f&(M, m)
0, M, loses the with match
0.5, if M, and M. achieve draw
For non-zero-sum games, e.g., Blind Auction, (M, m) the rewards earned by M; at the m-th match.
NRA(MM; Ma: fa) is naturally normalized to [-1, 1]. providing an interpretable meaning regarding the performance of M,: NRA(M,, (o)ss S&) > means M, is better than Ma: (RA,M; Mo.fs) < 0 means M; is worse than M.; NRA(M, fa) 0 means M, is competitive as Mo

---

Evaluation Metric: Elo Rating, Following the conventional rating mechanism in the real world, e.g., Chess, we employ the popular Elo Rating (Elo, 1960) for calculating the relative skill levels of players in zero-sum games. Please refer to Appendix A7 for more details of Elo rating.
3 GTBENCH: Game-Theoretic Evaluation of LLMs
GTBENCH is a language-driven environment, making participating agents compete against each other in game-theoreti, manner. It is designed to be flexible and extensible, providing unified interfaces to participants and games, and supporting various multi-turn-basee games which can be extended in the future. The overall framework is presented in Figure 1. There three main components: Environment, Prompt Adapter, and Participant. Please refer to Appendix A1 for a detailed introduction of each component.
3.1 Taxonomy of Came-Theoretic Tasks
The chosen tasks and their detailed configurations are presented in Table 1. To comply with the common taxonomy (Lanctot et al., 2019) of game-theoretic tasks and provide diverse gam- ing scenarios, GTBENCH supports 10 different gaming environments, including Tic-Tac-Toe, Connect-4, Kuhn Poker, Breakthrough, Liar's Dice, Blind Auction, Negotiation, Nim, Pig. Iterated Prisoner's Dilemma, covering 6 mainstream cluding complete- and incomplete-informatinn gaming, dynamic and static gaming, and probabilistic and deterministic gaming. The preferred abilities of each game could be characterized as the combi- nation of board strategy, bids, collaboration, bluff, and math. Please refer to Appendix A2. I for the rules of each game and Appendix A2.2 for explanation of game-thhereee taxonomy.
theerthe configurations, in-
3.2 Participants and Protocols
Conventional Agents output actions through a conventional optimization or searching process. To provide fair comparisons, we employ the powerful Monte Carol Tree Search (MCTS) (Chaslot et al.. 2008) as the conventional agent for most of the games, with the number of simulations as 1000. Since Iterated Prisoner's Dilemma is dynamic gaming with very limited action space, i.e. <TESTIFY>> or <SILENT>, we utilize the more popular Tit-for-Tat (Axelrod, 1981) strategy, which simply repeating the opponent's s last action, as the conventional agent. We also include Random Agent that randomly selects action at each turn, serving as baseline and sanity check. Please refer to Appendix A3.1 for more details about MCTS Agent and Tit-for-Te Tat Agent.
LLM-Driven Reasoning Agent consists of backbone LLMs and reasoning paradigms. For reasoning schemes, we consider the following reasoning paradigms as they are widely known to be effective for general reasoning tasks: Prompt: Directly Prompt LLMs to generate responses, without additional reasoning steps; @ Chain-of-Thougee (CoT) (Wei et al., 2022): CoT Agent prompts LLMs by thinking step by step: 2 Self- Consistnnttt CoT (Wang et al., 2022b): SC-CoT Agent prompts LLMs by generating multiple step-by-step thinking trajectories and performing majority voting get the final response. The number of trajectories is set to 5 in this paper; O Tree-of-Thoughl (ToT) (Yao et al., 2024a): ToT Agent prompts LLMSs to generate responses by incorporating exploration and deliberate decision-making, e.g. self-evaluation. The number of sequences for both answer generation and answer evaluations
Prompt Templates. Prompts are designed to be modular, consisting of four individual components: System Prompt, Head Prompt, Observation Prompt, and Reasoning Prompt. Reasoning prompts, e.g., CoT7TOT, are designed to only focus on instructing LLM how to think, regardless the game environment. Thus, they could be automatically adapted when adding new game. Please refer to Appendix A5 for the detailed prompts and observations for each game and agent.
Sanity Check. We provide the task completion rates of all the LLMs and reasoning agents in Ap- pendix A5.6. We show that all the LLM agents achieve 2 90% completion rate, indicating that the prompts are properly configured and LLMs are capable of following instructions to finish the game.
5

---

GPT-4
GPT-3.5-turb
Llama-3-70b 3-b---nttutt
Llama-2- 2-mna---00b-hat
CodeLlama- 1aamma--4---nsttucc
0.5
NRA
3.9
Opponent
Ramdom Agent
MCTS Agant
Prompt
SC-CoT
Prompt
SC-CoT
Prompt SC-CoT
Figure The NRA state-of-the-an LLM-driven reasoning agents when against MCTS Agents and Random Agents, over complete and deterministic scenarios. Red and gray lines mean the maximum NRA achieved by LLM agents.
Liar's Dice
Blind Auction
Negotiation
Opporent
Ransom Agent MOTT//TI Apent
NRA
Kuhn Poker
Pig
Prisoner's Dilemma
6P74
GPT-3 5-trt
Liama-2-7000 7Db-dial
Mictral7e Orea
GPT4
GPT-3 0PT-3.6-urto 6-uutt
Codelt CodelLanms lans- 34th-34h-h- Inni-nnttutt
Liama-2-70 2-70h-hhat
Micciial ABcthal-7b-Oma 7b-Omaa
GPT-4
GPT-3 5-uuba
CodeLlama- 34b-iretuud Llama-2 7-b--chl
Mistral-7b Oroa
Codel Jaman 34b
Figure 3: The game-wise NRA of LLMS when against MCTS/TfT Agents and Random Agents, incomplete and probabilistic scenarios. Error bars are obtained over different reasoning methods. Green and gray lines mean the maximum NRA achieved by LLM agents.
Are LLMs Capable of Strategic Reasoning?
In this section, we evaluate the strategic reasoning capabilities of LLMs by conducting experiments among conventional solvers and LLM-driven agents.
Experimental Settings. We consider well-recognized L.MSs such as commercial LLMs: GPT- 3.5-turbo-1 106 and GPT-4-0613 (Achiam et al., 2023), and open-source LLMs: Llama-3-70b- Instruct (Meta, 2024), Deepseek-1LMM-66bhchhl (Bi et al., 2024), Llamm-2-70b-chat (Touvron et al., 2023), CodeLlama (Roziere et al., 2023), and Mistral-7b-Orca (Jiang et al., 2023a; Mukherjee et al., 2023). For all the LLMs, the temperature is set to 0.2 and the max number of generated tokens run 50 valid matches. The final performance is measured by
1024. For each competition, the averaged NRA over the 50 valid matches. To mitigate the first-player advantage, we have each participant take the first turn in 25 matches.
4.1 Complete and Deterministic Gaming
There four complete and deterministic tasks supported in GTBENCH: Tic-Tac-Toe, Connect-4, Breakthrough, and Nim. We compare LLM--riven agents with Random Agent and MCTS Agent. Results summarized in Figure 2. In general, show that all LLMs achieve substantial relative advantages when competing against the Random Agent. Among all the agents, GPT-4 w/ CoT reasoning achieves the highest NRA. For open-source LLMs,, Llama-3-70b-Instre outperforms other open-source I.Mss, achieving comparable capabilities as GPT-4.
However, when competing against the MCTS Agent, all the LLM agents equipped with various reasoning methods achieve NRA as -1, meaning that LLM agents can barely win even a single match. This is because for board games with moderate action/state space such the four involved complete and deterministic games in GTBENCH, MCTS agents with a sufficient number of simulations achieve near-optimal strategies. Consequently, LLMs not competitive in complete and deterministic games.
4.2 Probabilistic and Dynamic Gaming

---

There are five probabilistic ghme-theoretic gaming tasks: Kuhn Poker, Liar's Dice, Blind Auction, Negotiation, Pig. and one dynamic task: Iterated Prisoner's Dilemma. We group these games together as they all involve stochasticity in the gameplay, which is essentially different from complete and deterministic games. The Random Agent as the opponent is omitted for both Negotiation and It- erated Prisoner's s Dilemma because the Random Agent rarely chooses to collaborate, resulting in meaningless evaluation. Results are summarized in Figure 3. When competing against the MCTS Agent, it is shown that Liar's s Dice shares similar trend the complete and determin- istic scenarios (Figure 2), where LLM--riven agents achieve near 1 NRA. This is because the 2-player Li Liar's Dice has very limited stochasticity, making the gameplay tend to be complete information. For other tasks, we found that LLMs do not always fail. We observe that the NRA of LLM agents is close to 0 over all the tasks, indicating that they are equally competitive as conventional solvers or even better (e.g., Kuhn Poker where GPT-4 outperforms MCTS Agent).
Table 2: Code- soode-retraining benefits strategic reason- ing. Gray rows are code-pretrained LLMs.
Model
aeg. NRA in Det. Games
arg. NRA
Proh.
NRA
GPT4
0.09
0.33
0.13
Liama 3 70t Inseruct
0.07
0.11
0.04
Codel -am-------eeeee Ab-Innmmct
4006
4.01
Dorporek-1 LLM To-chat
-0.09
-0.08
4.06
Despacdk-11
Deepseek:
insiruct
4.3 -LM- Competition
We investigate whether popular LLMs remain competitive in game-theoretic scenarios. Specif- ically, we take GPT-3.5-turbo with Prompt Agent as the common opponent and make other LLM-driven agents compete against it. Please refer to Figure A6 for the full leaderboard eval- uated by NRA. The Elo rating results are placed in Table 6. In general, GPT-4 is the most power- ful LLM in strategic reasoning among all the ex- amined LLMs. Morcover, Llama-3- 3-70b-Instrus achieves comparable performances as GPT-4 and outperforms GPT-3355turbo Here break the results into 3 takeaways:
Model
GPT-3.6-uul
Nistal-D Orca
Liana 2 706 chat
Completo and Detorminttcc
Codel Rodellamma 340-
Incomplnt and Probebiistc
0.4
NRA
0.2
0.5
0.0
Prompt
CoT
SC-CoT
ToT
Pode-Pretraining Benefits Game-Theoreite Tasks. Table 2, we show code-pretrained LLMs, e.g.. CodeLLlmma334--Insit -o-.e--.7.--..-nsruccc
Figure 4: The NRA of LLM agents when compet- ing against Random Agent. Advanced reasoning does not always result in better results.
and Deepseek-Coders Insttruct,, significantly out- perform larger chat LLMs, c.g., Llama-2-70b-chh and Deepseek- Deepseck-LLM-67b-chaa pretrained LLMs have less than half of the parameters, suggesting that code ande-pretrainin benefits game-theoree theoretic tasks. This verifies recent discoveries where code- pretraining benefits logical reason- ing (Madaan et al., 2022; Liang et al., 2022; Ma et al., 2023).
These code-
Advanced Reasoning Methods Do Not Always Help. We observe that advanced reasoning methods may lead to worse results in game-theoretic scenar- ios. To make more clear, we present the averaged NRA obtained by reasoning methods across differ- ent LLMS when against Random Agent in Figure 4, In general, only Mistral-7b- -rbbrraa has a substantial improvement when equipped with CoT reasoning while advanced reasoning leads to worse results for other LLMs.
Table 3: The NRA of LLM agents w/ CoT rea- soning. Cyan cells mean CoT results in better performance. Magenta cells mean CoT results in worse performance.
Opponent
Model
Reasoning
avg. NRA
Prompt CoT
0.00
G- Prompt Agent
Llama-
Prompt Cal
GPT4
Prompt
Col
0.15
0.13
In Table 3, we present the results when against GPT- 3.5-turbo Prompt Agent. We show that advanced reasoning benefits powerful LLMs, e.g., GPT-3.5- turbo, while it results worse results for other LLMs. It suggests that advanced reasoning is
CodeLham- 34b-
Prompt
0.01
Instract
CUT
0.09
CodeL 34b-
0.01
Prompt Agent
Llama-2-70b-char
Prompt
Col
0.29
7

---

double-edged sword: powerful LLMs are capable of leveraging advanced reasoning to achieve better results; advanced reasoning may also impose reasoning errois and risks during the inference of ordinary L.Mss. In Appendix A8, we further examine five different CoT strategies over the GPT-3.5-turbo model to mitigate the effect brought by prompt sensitivity, along with some failure cases presented. These CoT prompts resulting in different performances are all worse than the naive Prompt Agent.
Most Open-source LLMs are Less Competitive than Commercial LLMs in Complex Games. We observe that most of open-source LLMs such as 1-ama-2-70b-hher and CodeLlama-3455 Instruct are not good at games with com- plex rules and board states. In Table 4, we present the average NRA when in- cluding and excluding Breakthrough? It is shown that both alama-2-70b-onn and CodeLlama- 34b-Instruct fail Breakthrough, resulting in worse NRA scores than GPT-4. However, we found that the recently released 1arma---770bInsttie in breakthrought This indicates that open open-source LLMs achieve comparable capabilities when dealing with complex tasks and environments commercial LLMs.
Table 4: The average NRA of LLM-driven agents when Breakthrough is included and excluded.
Llama-3. 70t Iis-Instruct
Codal
Lluma-22 2
Taxonomy
GPT4
34b Iastucttuct
70b chat
Breakthrough wio Breaktarough
0.11 (-0.22)
.001 (-0.55) 0.08 (10.09) -0.18 (10.22)
(Meta, 2024) has significant performance
4.4 Error Profiles
We introduce the most prevalent mistake patterns observed across different games, comprising Misinterpretation, Factual Inaccuracies, Overconfidence, Calculation Mistakes, and Endgame:
Misinterpretation denotes the misinter-
Table 5: Quantitative results of error patterns.
pretation the game's 's current state by LLMs, including errors like misattribut- ing piece ownership and failing to rec- ognize vacant spots on the board. Fac- tual Errors refer situations where the player has reasonable plan but their actions do not align with their plan. For instance, in Breakthrough, GPT-4 w/ CoT agent plans to fend off frontal attacks by the opponent, which reasonable. However, it takes scribes a scenario where a player overlooks potential risks in pursuit of greater rewards. Calculation Errors refer errors that occur in arithmetic, such calculating XOR in Nim. Endgame Misdetec- tion means a failure to recognize immediate win/lose situations, e.g., a player fails to recognize potential winning move. Demonstrations of each mistake pattern are presented in Appendix A9.
Percentage of Error patterns (%)
Model
Endgame Misdetcction
Mis- interpretation
Calculation Error
Factual
confidence
Error
GPT-
33.33
9.80
15.69
9,80
45.10
pieces to achieve that, which is impossible. Over-confidence de-
In Table we present the quantitative results regarding these error patterns. It obtained from GPT-4 w/ CoT agent when playing against conventional solvers, e.g., MCTS/TYT agent, the opponent. We manually examined a total of 157 turns (50 matches, with turns per match). We observe that LLM agents capable of generating reasonable planning/strategies. However, they have difficulties in selecting the correct actions to align with their thoughts. Also, LLMs miss endgame situations, leading to failure to recognize winning and losing moves.
Table 6: The Elo rating results of LLM vs. -LMM experiments.
Model
TicTTacToe
Breakthrough
Blind Auction
Kuhn Poker
Liar's Dice
avg. Elo
GPT-4
1554.34 34
1667.11
1581.94
1479.87
1676.70 70
1591.99
Alama-3-70b-Insmicc
1669.42
1625.46
G-T--.5-turbo
1579.80
1576.37
1514.27
1441,80
1459.26
1514.30
Sodee-Juma-34b-Insste
1589,94
1398.10
1533.48
1414.57
1374.40
1462.10
1479.08
1320.42
1484.32
1521.82
1485.00
1458.13
Mistral-7B-Insteee
1338.57
1516.48
1310.00
1393.42
'Beekkthrough has larger action/state space than other complete-information information games.

---

OPT4
100
Rogret Vallue
1.5
Co0.Jeraeeeiiiitt Mrnae7B-Doenrre
Player 2 Rowards
80
System Reward
0.0
Optimal
-0.5
1.0
-1.0
0.5
-1.5
Gerepkoy
Open
0.0
80
Blind Auction Prisoner's Dilemma
Player Rewards
Round Index
Regret
(b) Resource Distribution
(c) Pareto Efficiency
Figure 5: Camee-heorett properties. The results are obtained when competing against GPT-3.5- ttrbbo w/ Prompt Agent as the opponent. In (b), each dot (x, y) represents an agreement resource distribution with Player obtaining reward x and Player 2 obtaining reward y. In (c), the system reward is calculated by the sum of the payoffs of all players.
5
The Game-Theoretic Properties of LLMs
Nash Equilibrium with Regret. In game theory, being close to a Nash Equilibrium (Nash Jr, 1950) indicates that the strategies chosen by the players are near to optimal. It has been popular to approximate Nash Equilibrium with Regret" (Johanson et al., 2012; Nisan and Noti, 2017: Zinkevich al., 2007). In Figure 5a, we present the regret values of LLMs on Blind Auction and Iterated Prisoner's 3 Dilemma. Please refer to Appendix A10 for how regret values are calculated for these two tasks. For Blind Auction, GPT-4 shows lower Regret, indicating achieving closer to optimal solutions than other LLMs. However, in Iterated Prisoner's ' Dilemma, CodeLlama- 34b-Instruct exhibits lower regret compared to GPT-4. Through human examination, we found that this is because GPT-4 tends to <Silent> more frequently, whereas Codellama has significantly higher probability of <Testtff>>. This discrepancy may be due to the human preference alignment in GPT-4, such as a higher emphasis on morality (Pan et al., 2023) or maximizing system reward which makes GPT-4 less likely to <Testify>>
Pareto Efficiency. We study Pareto Efficiency in two games: Negotiation and (Iterated) Prisoner's ' Dilemma. In Figure 5b, we count all agreements reached by participants and record the values attributed to each based on the agreed division. Most agreements result in substantial values for both participants, though some LLMs, like Llama-2-70b-cha. and CodeLlama- 3ee--33bb---s-e Instruct, may accept unfair resource divisions. In contrast, GPT-4 and Mistral struggle to reach agreements and tend to negotiate for Pareto improvements. A repeated game is a standard game that is played multiple times by the same players, with each player is able to observe the history of past plays (Aumann et al., 1995; Akata al., 2023). In Figure 5c, we investigate the Pareto Improvement in Iterated Prisoner's a Dilemma and ordinary Prisoner's Dilemma, i... each round is played individually. The Pareto Improvement is observed in the repeated-gand scenario during the rounds, indicating that LLMs are capable of leveraging history to adjust their strategies.
6
Conclusion
This work investigated LLMs' strategic and logical reasoning abilitics under competitive scenarios. To achieve this, we created broad evaluation scope by considering various classic and LLM- based gaming agents and 10 representative games. We conducted the benchmark study of game- theoretic evaluations for LLMs, shedding light their reasoning performance. Our extensive evaluations revealed insightful LLMs' gaming behavior, such as their intrinsic failure in complete and deterministic games, impressive reasoning in incomplete and probabilistic games, and benefiting from code-generation pertaining and appropriate prompt designs.
Limitations
This research prompts LLMS to generate actions regarding various game scenarios, relying on pre-defined prompt templates. Thus, the results may suffer from certain variances introduced by prompt sensitivities. Although the introduced games are popular, their actions/state
'Regret (Zinkevich al., 2007) measures how much a player would have improved their outcome by choosing a different strategy, given what they know now after the game has played out.
SSilntt> maximizes the system reward in Iterated Prisoner's Dilemma

---

space is limited, which may not be well-distinguished for LLMs in the same skill levels. The generated actions may be illegal due the incapabilities of the following instructions.
This paper examines the game-theoretic task proficiency of Al models. We acknowledge concerns about models becoming autonomous entities with their own objectives. especially in deception or negotiation scenarios. It's important to note that our research measures the current capabilities of models, rather than enhancing their abilities. We do not train Al models to be competent in game theory tasks or to bluff or defect. Instead, we assess existing competencies, contributing to deeper understanding that can inform innovative measures against potential risks. We believe our work paves the way for responsible and effective Al safety.
Impact Statements
Acknowledgement
This work was performed under the auspices of the U.S. Department of Energy by the Lawrence Livermore National Laboratory under Contract No. DE- AC52-07NA27344 and was supported by the L.NNL LDRD Program under Project No. 23-ERD-030 and 24-ERI--058. This work was partially It also partially supported by NSF-AI Engage
supported by the NSF award FMitF-231922 Institute DRL-21 12635 and DARPA MCS Grant N66001-19-2-4033 The views, opinions, and/or findings contained in this article those of the authors and not of the funding agency.
References
Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. Lim-deliberation: Evaluating Iims with interactive multi-agent negotiation games, 2023.
Marwa Abdulhai, Isadora White, Charlic Snell, Charles Sun, Joey Hong. Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models, 2023.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Ehsan Aghaei, Xi Niu, Waseem Shadid, and Ehab Al-Shaer. Securebert: A domain specific language model for cybersecurity. In International Conference on Security and Privacy in Communication Systems, pages 39-56. Springer, 2022.
Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models, 2023.
Kimia Ameri, Michael Hempel, Hamid Sharif, Juan Lopez Jr, and Kalyan Perumalla. Cybert: Cybersecurity claim classification by fine-tuning the bert language model. Journal of Cybersecurity and Privacy, 1(4):615-637, 2021.
Robert J Aumann, Michael Maschler, and Richard E Stearns. Repeated games with incomplete information. MIT press, 1995.
Robert Axelrod. The emergence of cooperation among egoists. American political science review, 75 (2):306-318, 1981.
Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis. Alexander H. Miller, Sandra Mitts. Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei. David Wu, Hugh Zhang, and Markus Zijlstra. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378: 178:1067- 1074, 2022.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph thoughts: Solving elaborate problems with large language models. arXiv preprint a30iv:2308. 09687, 2023.
10

---

Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint urXiv:2401.0299 2024.
Kranti Chalamalasett, Jana Götze, Sherzod Hakimov, Brielen Madureira, Philipp Sadler, and David Schlangen. clembench: Using game play to evaluate chat- chat-optimized language models as conversa- tional agents. arXiv preprint arXiv:2305. 13455,, 2023.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308. 07200, 2023.
Guillaume Chaslot. Sander Bakkes, Istvan Szita, and Pieter Spronck. Monte-carlo tree search: A new framework for game ai. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, pages 216-217, 2008
Junzhe Chen, Xuming Hu, Shuodi Liu, Shiyu Huang, Wei- Wei Tu, Zhaofeng He, and Lijie Wen. Limarena: Assessing capabilities of large language models in dynamic multi-agent environments. arXiv preprint arXiv:2440.. 16499, 2024.
Justin Chih- Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. arXiv preprint arXiv:2309. 13007, 2023.
Maee-Alex Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. In Computer Games: 7th Workshop. CGW 2018, Held in Con- junction with the 27th International Conference on Artificial Intelligence IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers 7, pages 41-75, Springer, 2019.
Tim Ruben Davidson, Veniamin Veselovsky, Michal Kosinski, and Robert West. Evaluating language models through negotiations. In The Twelfth International Conference on Learning Representations, 2023.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factual- ity and reasoning in language models through multiagent debate. arXiv preprint 13X15:2305.14325. 2023.
Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5050-5063, 2024a.
Jinhao Duan, Shiqi Wang. James Diffenderfer, Lichao Sun, Tianlong Chen, Bhavya Kailkhura, and Kaidi Xu. Reta: Recursively thinking ahead to improve the strategic reasoning of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers), pages 2232-2246, 2024b.
Arpad Elo. Elo rating system.
1960.
Meta Fundamental AI Research Diplomacy Team (FAIR)+. Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624): 1067-1074, 2022.
Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35: 18343-18362, 2022.
11

---

Kanishk Gandhi, Dorsa Sadigh, and Noah D. Goodman. Strategic reasoning with language models, 2023.
Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Long-Chun Zhu, Demetri Terzopoulos, Li Fei-Fei, and Jianfeng Gao. Mindagent: Emergent gaming interaction, 2023.
Matthew Hausknecht, Prithviraj Ammanabrolu, Mare-Alexandre Côté, and Xingdi Yuan. Interac- tive fiction games: A colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7903-7910, 2020.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, I Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXlv:2310.06825, 2023a.
Haitao Jiang, Lin Ge, Yuhe Gao, Jianian Wang, and Rui Song. Large language model for causal decision making. arXiv preprint urXiv:23/2.171222 2023b.
Michael Johanson, Nolan Bard, Mare Lanctot, Richard G Gibson, and Michael Bowling, Efficient nash equilibrium approximation through monte carlo counterfactual regret minimization. In Aamas, pages 837 007-846,22 2012.
Mare Lanctot, Edward Lockhart, Jean- Saan-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, et al. Openspiel: A framework for reinforcement learning in games, arXiv preprint arXiv: (Et.::9900009553,, 2019.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09) 10, 2022.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang. Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305. 19118, 2023.
Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. Avalonbench: Evaluating iims playing the game of avalon. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang. Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong. and Jie Tang. Agentbench:
Evaluating 1lms as agents, 2023.
Yingwei Ma. Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help llms reasoning? arXiv preprint arXiv:2309. 16298, 2023.
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1384-1403, 2022.
Meta. Introducing meta llama 3: The most capable openly available Ilm to date. https://aai http::///aa.aaaaa.... meta. com/blogg/eeta-llams -llaaa 3/, 2024. Accessed: 2024-05-18.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023.
Gabriel Mukobi, Hannah Erlebach, Niklas Lauffer, Lewis Hammond, Alan Chan, and Jesse Clifton. Welfare diplomacy: Benchmarking language model cooperation. In Socially Responsible Language Modelling Research, 2023.
John Nash Jr. Equilibrium points in n-person games. Proceedings of the national academy of sciences, 36(1):48-49 1950.
12

---

Noam Nisan and Gali Noti. An experimental evaluation of regret- based econometrics. In Proceedings of the 26th International Conference on World Wide Web, pages 73-81, 2017.
Aidan O'Gar,, Hoodwinked: Deception and cooperation in text-based game for language models, 2023.
Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. In International Conference on Machine Learning, pages 26837-26867. PMLR, 2023.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Conference on Learning Representations, 2023.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Mohit Shridhar, Xingdi Yuan, AlexxAdee Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference Learning Representations, 2020.
Simon Stepputtis, Joseph Campbell, Yaqi Xie, Zhengyang Qi, Wenxin Sharon Zhang, Ruiyi Wang, Sanketh Rangreji. Charles Michael Lewis, and Katia P Sycara. Long-horizon dialogue under- standing for role identification in the game of avalon with large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.0922 2023.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291, 2023a.
Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1279-11298, 2022a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2022b.
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating 1lms in multi-turn interaction with tools and language feedback. arXiv preprint 2rX1v:2309.10669 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of thothhtght prompting elicits reasoning in large language models. Advances in 2022.
Neural Information Processing Systems,
Shijie WU, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhan- jan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint 2rXiv:2303.11566 2023.
Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. Examining the inter-consistency of large language models: An in-depth analysis via debate. Association for Computational Linguisties, 2023.
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.0466 2023a.
13

---

Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the werewolf game, 2023b.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024a.
Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. A survey on large language model (Ilm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, page 100211, 2024b.
Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, and Furu Wei. Llm mastermind: survey of strategic reasoning with large language models. arXiv preprint arXiv:2404.01230, 2024.
Liammin Zheng, Wei-Lin Chiang. Ying Sheng. Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing. et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Andrew Zhu, Karmanya Aggarwal, Alexander Feng, Lara Martin, and Chris Callison-Burch Fire- ball: A dataset of dungeons and dragons actual-play with structured game state information. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2023.
Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information, Advances neural information processing systems, 20, 2007.
14

---

Appendix
A1 Overall Architecture
There are three main components GTBENCH:
Environment. The environment ( Figure (c)) is responsible for overseeing the crucial processes related to gameplay. Specifically, it is tasked with building up observations, managing gameplay, and applying the actions obtained from participants. In this paper, all of the gaming environments are built top of OpenSpiel Lanctot et al. (2019).
Prompt Adapter. The prompt adapter ( Figure 1 (b)) plays a vital role in facilitating effective communication between the environment and the virtual participants. It serves as an intermediary between the two entities by receiving observations from the environment, which it then translates into unified observation prompts. The prompts are then parsed and sent to the participating agents to formulate their responses. The adapter is also responsible for obtaining actions from the participants, which transforms into legal actions before parsing them the environment for game execution.
Participant. The participants Figure 1 (a)) involved in the gaming process generate responses according to the observation prompts received from the Prompt Adapter. These responses consist of actions that participants intend to take in this turn.
A2
Gameplay Configurations
A2.1
Games Introduction
paper-and-pencil game for two players who take turns marking the spaces in three-by-three grid with X O. The player who succeeds in placing three of their marks horizontal, vertical, or diagonal row is the winner. It is a solved game, with forced draw assuming optimal play from both players.
Tic- Tic-TacTToe"
Observation (input): Our observation contains "opponent moves" and "self moves". "Opponent moves" contains all the current opponent agent's historical actions. "Self moves" contains all the current agent's history actions.
Actions: We define our action in the following format: <CrRy>, which C and R mean columns and rows respectively, while x and y mean the index of column and row. Each player may make their own action in turn.
Prisoner's Dilemma' 7 is a game theory thought experiment that involves two rational agents, each of whom can cooperate for mutual benefit or betray their partner ("defect") for individual reward.
Observation (input): Our observation contains "opponent moves" and "self moves". "Opponent moves" contains all the current opponent agent's historical actions. "Self moves" contains all the current agent's history actions.
Actions: We define our action in the following format: <Silent> or <Testify>> All players must take their action simultaneously.
Breakthrough Breakthrough is abstract strategy board game invented by Dan Troyka in 2000 and made available as Zillions of Games file (ZRF). It won the 2001 8x8 Game Design Competition. The first player reach the opponent's home row the one farthest from the player - is the winner. In our work, we scale the size of the board to 3*8 while maintaining its competitiveness.
Observation (input): Our observation contains "opponent moves", "self moves", and "board preview"". "Opponent moves" contains all the current opponent agent's 's historical actions. "Self moves" contains all the current agent's history actions. The "board preview" feature maintains the status of each grid on the board through a list of strings, denoting whether it contains a black piece, a white piece, or is empty.
'https://enn www.wewiiiiiiiiioimmom hwtppp/di...g//iiiiiiii-ttt-atet
phttpp://en.wikiieee: halkippdia.or//iii////iiseeeea.
http://ew.w.iiiiiaa..oo.o hwk.ippdda..rg///aa///teeettttto (boardddu (board_game)
A15

---

We define our action in the following format: Az->By, in which A and B mean the current column index and destination column index respectively, while r and y mean the index of current row and destination row. Each player may make their own action in turn.
Actions:
Connect Four a game in which the players choose a color and then take turns dropping colored tokens into six-row, seven-column vertically suspended grid. The pieces fall straight down, occupying the lowest available space within the column. The objective of the game to be the first to form horizontal, vertical, or diagonal line of four of one's own tokens.
Observation (input); Our observation contains "opponent moves" and "self moves". "Opponent moves" contains all the current opponent agent's historical actions. "Self moves" contains all the current agent's history actions.
Actions: We define our action in the following format: <Cr> in which C means column, while means the index of column. Each player may make their action in turn.
Blind Auction 10 is a common type of auction. In this type of auction, all bidders simultancously submit sealed bids that no bidder knows the bid of any other participant. The highest bidder pays the price that was submitted. All players must take their action simultaneously.
Observation (input): Our observation contains "valuation" "Valuation" contains each of the values of all the items for the current player.
Actions: We define our action in the following format: <r>, in which x represents the amount that certain player would like to bid for.
Kuhn Pl 1 is a simplified form of poker. Kuhn is a simple model zero-sum two-player imperfect- information game, amenable to complete game-theoretic analysis. In Kuhn poker, the deck includes only three playing cards, for example, a King, Queen, and Jack. One card is dealt to each player, which may place bets similarly to standard poker. If both players bet or both players pass, the player with the higher card wins, otherwise, the betting player wins.
Observation (input): Our observation contains "card", and "moves". Among these, "card" denotes the current player's hand card in this match, while "moves" represents the history of all characters' moves together with the index of the rounds.
Actions: We define our action in the following format: <Pass> or <Bt>>. Each player may make their own action in turn.
Liar's Dice is a class of dice games for two more players requiring the ability to deceive and detect an opponent's 's deception.
Observation (input): Our observation contains: "Self dice face value" and "last move". "Self dice face value" describes all the face values of dices the current player has, while "last move" represents the previous player's action.
We define our action in the following format: < x dices, y value> or <Liar>, Among these, x means the quantity of dice, and y means the face values of the dice. The option "Liar" denotes the current player wants to stop and challenge the previous players. Each player may make their own action in turn.
Actions:
Pig! 13 is simple dice game. Players take turns to roll single dice many times they wish, adding all roll results to a running total, but losing their gained score for the turn if they roll a
Observation (input): Our observation contains: "self current score", "opponent current score", and "turn total score". "Self current score" and "opponent current score" represent the game culminated score of the current player and opponent player respectively. While "turn total score" denotes the sum of the score of the current turn.
"nttps://en. htwp:///daaa.g//aa////momeeeteee
sealleeeeeeeeeeeeeeeeeeeeeeeeeee bid_auction pokeereuuuunoororre_
(dice_gardd_gc
A16

---

Actions: We define our action in the following format: <ssop or <roll>, Each player may make their own action in turn.
Nim 14 is a mathematical game of strategy in which two players take turns removing objects from distinct heaps piles. On each turn, player must remove at least one object and may remove any number of objects provided they all come from the same heap or pile.
Observation (input): Our observation contains: "piles". "Piles" denotes the number of matches different piles have.
Actions: We define our action the following format: <pile:r, take:y>> Among these, x represents the index of the pile that the current player takes, and y represents the number of matches the current player takes. Each player may make their own action in turn.
Negotiation
Observation (input): Our observation contains: "turn type". "item pool", "most recent proposal", "most recent utterance", and "self value vector". "turn type" is an enum variable, it has two options: proposal and utterance. The "Proposal" the turn that the current player could think about the desired quantities of the itemss, and the "Utterance" is the turn that the current player states the values to its opponent. "item pool" represents the quantities of all the items. "most recent proposal" and "most recent utterance" represent the opponent's 's latest proposal and utterance. "self value vector" represents how much the value of the items to the current player.
Actions: We define our action in the following format: <Agree> or < x. y. >, Among these, <Agre>> represents the current player agreeing on the opponent's utterance. æ. y, and represent the quantities of different items that the current player wants to get.
A2.2 Gaming-Theorrff Taxonomy
Complete and Incomplete Information One fundamental dimension along which games are classi- fied is the level of information available to players. In complete information games, players possess perfect knowledge regarding the game's structure, including the available strategies, payoffs,, and the actions taken by other players. Examples of complete information games include canonical examples like chess and the game. Conversely, incomplete information games involve situations where players must make decisions without having full knowledge of the game's 's parameters or the actions of other players. Classic examples of incomplete information games include strategic interactions in economics, such as auctions negotiations, where players have limited knowledge about the valuations or preferences of other participants.
where all relevant information is transparent to all players throughout
Dynamic and Static Another crucial dimension for classifying games is the timing of players' decisions. In static games, players make decisions simultancously, without the opportunity observe or react to other players' moves, Examples of static games include simultaneous-move games like the Iterated Prisoner's Dilemma. In contrast, dynamic games involve sequential decision-maker where players observe previous moves before choosing their actions. Dynamic games encompass a wide range of strategic environments, from bune-ased board games like chess to dynamic settings like Kuhn Poker, where players strategically make their actions based on the unfolding dynamics of the game.
Probabilistic and Deterministic Games can also be differentiated based on the role of uncertainty in decision-making In deterministic games, the outcomes of players' actions are fully determined by the game's 's rules and the strategies chosen by players. Deterministic games include classic examples like chess or Tic-Tac- Toe, where each move leads to a predictable outcome based on the game's 's rules and the players' strategies. Conversely, probabilistic games involve randomness or uncertainty in determining outcomes. This uncertainty can stem from elements such as dice rolls, card draws. Examples of probabilistic games include games of chance like Kuhn Poker, Liar's Dice, or Pig. where players must contend with the inherent uncertainty of probabilistic outcomes.
I "nttpa://en.wiikiiii walliipedaa.org/vkkke... "https://arxx" ttttpp://axxxv.org/pp///////e 05125. .pf
A17

---

A3
Participants
A3.1 Conventional Agent
MCTS Chaslot et al. (2008) is a heuristic search algorithm that has gained prominence in recent years, particularly in the domain board games and decision-makinn under uncertainty. characterized by its ability efficiently explore large search spaces by sampling potential future outcomes through Monte Carlo simulations. The algorithm iteratively builds a search tree by simulating random sequences of moves from the current game state and evaluating their outcomes through repeated simulations. By focusing computational resources on promising branches of the search tree, MCTS aims guide the search towards regions of the game space that outcomes. MCTS has demonstrated remarkable success in various domains, including games like Go, where traditional search algorithms struggle due to the game's immense complexity and branching factor.
likely to lead favorable
Tit-for-Tat Axelrod (1981) is a simple but powerful strategy in the realm of repeated games and social dilemmas. The strategy is based on the principle of reciprocity, where an agent initially cooperates and then mimics the opponent's previous action in subsequent rounds. Specifically, Tit-for-Tas starts by cooperating in the first round and then replicates the opponent's ' last move in each subsequent round. Despite its simplicity, Tit-for-Tat has been shown to be remarkably effective in promoting cooperation and achieving favorable outcomes in various scenarios, including Iterated Prisoner's s Dilemma and evolutionary simulations. Its success stems from its ability balance cooperation and retaliation, fostering reciprocal behavior and encouraging cooperation among interacting agents.
A4 LLM-vs-LLM Results
In Figure A6, we present the confusion matrix of NRA when various LLM agents are against GPT-3.5-turbo and GPT-4.
Prisenar s Dilemmaa
Tie The Te
Connact 4
Breaktiraigh
Liar's Dice
Bind Aucton
Negstaation
Kuhn Paker
bvg NRA
Nin
Pg
Compete Against
1.00
Stturo Promet
GPT4
GPT4 + CoT -0.02
-0
0.56
24
0.07
0.13
0.44
0.04
0.04
00.16
0.13
0.75
(umm--t-0b--mstnntt + CoT 0.17
-011
0.22
0:83
4.12
0.06
4.16
-0.16
0.12
0..06
0.07
Uemo-3-70b notruct + Prompt- -0.15
-046
0.60
0.84
0.04
0.00
00.07
-0.28
0.00
00002
0.04
GFT3 +
Strbo SC CoT
GPT turbo TGT
0.05
0.24
0.36
0.00
0.12
0.18
0.00
0.12
0.40
0.19
0.01
Codel, ama 34b rstruct Promgt
0.33
10
40 0.80
0.40
40.25
0.22
0.16
0.36
0.12
0.60
0..01
Uama-2-70b-chatt SC-Oo
-016
0.16
-0.04
0.05
0.12
0.04
0.04
0.30
0.00
Lbmm-2-70-stht
Mistral 76 Orca + CoT
0.08
00
0.32
0.56
0:13
0.08
0.00
0.36
0.68
0.05
0.11
Codollama 34b notruct SC CoT
0:12
0.60
0.56
0.28
0.35
0.09
0.00
0.16
0.12
0.01
0.13
7b-Orca
0.01
0.50
Ukma-2-70h-cnet Prompt
4.37
-M4
-016
-007
-0.0
-0 04
80
-6 02
71
+0 20
-0.75
Mistral-P 76 Tr-Orra ToT -018
< 80
.0 44
..00
0 30
<0 20
00 08
+0 84
0.16
+0.24
Mistro 76 Orca Prompt
0.43
84
0.58
0.88
0.07
0.11
0.04
0.08
0.00
0.18
0.31
1.00
Figure A6: NRA confusion matrix of LLM vs. LLM across games ranked by average NRA. GPT-3.5-turbo with Prompt Agent serve as the common opponent against multiple combinations of LLMss with agents.
A18

---

A5 Prompt and Protocol
A5.1 Modular Prompt Structure
When prompting LLMs to generate the next action during the course of a game, the prompt is com- posed of four individual components, make sure all the participants access the same observations and information from environments:
System Prompt provides general guidance how the LLMs should perform.
Head Prompt provides the general background and rules of the game.
Observation Prompt formatted by fixed game-wise template, providing sufficient observations from the environment regarding the current gaming state, make L.MSs capable of making decisions.
The following provides the template used in the Blind Auction environment:
Your budget is <VALUATION>. Your bid must be strictly lower than or equal to <VALUATION>> Your opponent also has expected valuation and you do not know it. The legal actions are: <OVESI_
Here <VALUATION> and ELEGAL_MOVESS are variables and are obtained from a unified <observation> object. In this way, all the participants are guaranteed to assess the same infor- mation.
Reasoning Prompt guides the LLM's generation process, e.g., "Let's think step by step" for the CoT Agent.
AS.2 System Prompt
The system prompt is the initial text or context provided by the user to prompt the model to generate response. This prompt serves as the starting point for the model to understand the user's query or input and generate relevant and coherent response based on the provided context. We use the same system prompt for different tasks in work. Our system prompt presented as follows:
You are powerful gaming agent who can make proper decisions to beat the user in gaming tasks. You are helpful assistant that strictly follows the user's instructions. You must answer your questions by choosing one of the legal moves given by the user!
System Prompt:
A5.3 Head Prompt
The head prompt the text that explains the rules the games the LLMs. In our work, we designed different prompts for each of the games.
Tic- Tae-Toe
Head Prompt: Tic Tae Toe is a two-player game played on a grid. Players take turns marking space with their respective symbols. The goal is get of one's symbols in row, either horizontally, vertically, or diagonally. before the opponent does. all nine squares are filled and no player has three in row, the game is draw. The Tic Tac Toe game is played a by 3 grid, with the winning length as 3. Each move is represented by a string consisting of parts: the column (C) and the row (R), in that order. For instance, C1R2 means the movement at the position of the first column and the second row of the grid. You are playing this game with the user (opponent).
Iterated Prisoner's Dilemma
You and your partner are in the Prisoner's Dilemma situation. Specifically, you <Testify>> against your partner and your partner remains <Silent>> you will go free while your partner will get 3 years in prison the main charge. If you remain <Silent> but your partner <Testifyy>> against you, you will serve 3 years in prison and your partner will be set free. If you and your partner <Testifyy> against each other, you and your partner will each
Head Prompt:
A19

---

serve 2 years. If both you and your partner remain <Silent>, you and your partner will each serve year.
Breakthrough
Breakthrough a two-player game played on a rectangular board. Players take turns moving their pieces, which move one space straight or diagonally forward if the target square is empty. piece can also move diagonally forward to capture an opponent's piece. Capturing is optional, and player can only capture one piece per turn. The goal to be the first to reach the opponent's s home row, the farthest row from the player. If all of player's pieces only move forward or be captured. The Breakthrough board is identified by columns labeled starting from A (from left to right) and rows numbered 1 to 8 (from bottom to top). The intersection of column and a row specifies unique square on the board.
Head Prompt:
captured, they lose. The game does not allow draws, pieces
Connect Four
Connect a two-player connection board game, where the players choose color and then take turns dropping colored discs into vertically suspended grid. The pieces fall straight down, occupying the next available space within the column. The objective of the game is to be the first to form a horizontal, vertical, or diagonal line of four of one's own discs. You are a gamiing agent who aims to beat me in Connect games. Each move is represented by string consisting of two parts: the column (C) and the row (R), in that order. For instance, C1 means the first column.
Head Prompt:
First-price sealed-bid auction
Head Prompt: first-price sealed-bid auction (FPSBA) a common type of auction. It is also known as the blind auction. In this type of auction, all bidders simultancously submit sealed bids so that no bidder knows the bid of any other participant. The highest bidder pays the price that was submitted.
Each action is represented by <>> where refers the bid.
Kuhn Poker
Head Prompt: Kuhn poker is simple model pero-sum two-player imperfect- imperfect-information information game, amenable to a complete game- theorete analysis. Kuhn poker, the deck includes only three playing cards: King (K), Queen (Q), and Jack (J), One card is dealt to each player, and the third put aside unseen. The players take turns either <Bet> match the bet raised by the opponent <Pass>> to concede the game.
If a player bets, the other player must either call the bet by matching fold by conceding the game. If both players pass, the game is over, and the player with the higher-ranking card wins. The card rankings
follows: King (K) Queen (Q) Jack (J).
You are playing Kuhn poker with the opponent. The actions are denoted by <Bet>> and <Pass>.
Liar's Dice
Head Prompt: Liar's Dice is game of bluffing and probability, played with two players and each player has dice. During each turn, a player can either bid a higher quantity any particular face value or the same quantity of higher face value than the previous bid. Each player tries to outbid their opponent without being caught in a lie. The move this game is denoted in < dices, y value>, meaning there are at least dices with face values y.
Pig
Head Prompt: Pig is a fast-paced dice game where players risk accumulating points with each roll but risk losing them all if they roll a 1. Each player must decide when stop rolling and bank their points, aiming to be the first to reach 100 points. You are playing Pig with the other.
Nim
A20

---

In Nim, a strategic game with set of four piles containing 1, 3, 5, and matches respectively, players aim to avoid taking the last match. During each turn, player may take any number of matches from a single pile, but must take at least one and cannot exceed the number remaining in that pile. The objective is to force the opponent to pick up the final match, thereby winning the game.
Head Prompt:
The action is presented in <pilerr, take:y>> which means take match(es) from the æ-th pile.
Negotiation
You are negotiating the division of Peppers, Strawberries, and Cherries with the opponent. Different values these items hold for both you and your opponent. The process is structured into two stages per round: the proposal stage and the utterance stage.
Head Prompt:
A5.4 Observations
Our research team has developed a range of observation prompts tailored to different types of games. The list of these prompts is presented below.
Tic-Tac- Tac-Toe
Observation Prompt: Your opponent has finished actions: <OPPONENT KOPPONENT_MOVES>> You
have finished actions: <SELF_MOVER
Iterated Prisoner's Dilemma
You have been through this situation in the past and here are the decisions you and your partner made: (In the idæ + th round, you decided to <MOVE> and your opponent decided to <OPPONENT_MON MOVE>>> n round
Observation Prompt:
Breakthrough
Among which, the letter "b' represents black piece, while the letter 'w' represents a white piece. And character "" represents vacant space. The numbers the board are the indexes of the rows. Your opponent has finished actions: COPPONANT_MOVER You have finished actions:
Observation Prompt:
The board now looks like <BOARD
PREVIEW>, EBOARD_PREVIEEW
<SELF_ MOVES>.
Connect Four
Observation Prompt: Your opponent has finished actions: COPPONENT_MOVER You have finished actions: <SELF MOVES>.
First-price sealed-hid auction
Now, you are in an auction with an opponent. You want to win the object and at the same time, your budget <VALUATION>. Your bid must be strictly lower than or equal to <VALUATION>> You shall bid wisely against your opponent. Your opponent also has an expected valuation and you do not know
Observation Prompt:
Kuhn Poker
Observation Prompt: In this match, your card is <CARD>>. Here are the past moves in this match: <SELF MOVES>, <OPPONENT MOVES>.
Liar's Dice
Observation Prompt: Currently, the face value of your dice is <FACE, VALUE>, Last time, the opponent called CONOOONNNTLLSTTAACIIIO You are playing the Liar's Dice with another opponent. Therefore, there only two dice in total.
Pig
A21

---

Right now, your current score <AGENT your opponent's current score is earned
Observation Prompt:
<AGENT_CURRENT CURRENT SCORE>> CURRENT_SCORE>> and SCORE>> In this turn, you have
score.
Nim
Observation Prompt: Currently, the 1st pile has <PILES[0]> match(es), the 2nd pile has <PILES[[]> match(es), the 3rd pile has <PILES[2]> match(es), 4th pile has <PILES[3]> match(es).
Negotiation We proposed two different prompts for the "proposal" turn and "utterance" turn respectively.
For the "proposal" turn, we have:
Observation Prompt:
Now, the opponent propose take peppers, SL__PPOOOOOPOPPPEEEESSLL
CETPONENT_RRPPSAALLAAANN strawberries, and item pool. Last time, the utterance the opponent was to take <OPPONENT UTTERANCE, UTTERANCE_TAKE(0)> peppers, COPPONENT UTTERANCE TAKE[]]> strawberries, and SUPPONENTUTTTRRANGELL (AAAAA [F[]}>
[AASE]] cherries from the
cherries from the item pool.
Now, it is your decision. If you find the proposal raised by the opponent is acceptable, you should output Agree. Otherwise, you should output your proposal in the format <Proposal: [a, b, c|>.
For the "utterance" turn, we have:
Observation Prompt:
Last time, you propose to take <AGENT _ROPOSALR [AKE[ pep- pers, <AGENT <AAGEN_PPRPPOSSTS PROPOSAT__AAAAE TAKE[ }> strawberries, and <AGENT SOCINT__RRPOSAA_AAKCCOO cherries from the item pool. Last time, the utterance of the opponent was to take peppers, <OPPONENT <OPPONETTUTTRRNNEE,, SOTTT_TTERAN__TAAEET [ANKF_]ATK[[]>>>>>>
<OPPONENT COPPONENT_UTTERRNCE LE__LTTTNT__AAAAEEEE strawberries, and CEPPONENTTUUTTRRANEE [_ cherries from the item pool.
Now, it your turn to provide your utterance regarding the division of items. The utterance is what you want to tell your opponent and does not mean your real intent. You should output your utterance in the format <Utterance: [o, <]>
A5.5 Reasoning Prompt
Prompt agent: Prompt agent does not necessitate the use of LLMs to apply any predetermined strategy prior pocision-making. Rather, it simply requests LLMs for inference and subsequently provides the outcome.
You must choose legal action to set up advantages. Your output must be in the following format:
Action:
Your action wrapped with <>, i.e., <format>>
Please return your answer without explanation!
CoT agent: CoT agent makes LLMs consider the given observation first, then give out the action according to its thinking.
First think about your current situation, then you must choose one action from legal actions set up advantages.
Your output must be in the following format strictly:
Thought: Your thought.
Action:
Your action wrapped by <>, i.e., <format>>
Remember, you can only choose one move from the legal actions.
SC-CoT agent: SC-CoT agent an advanced version of the CoT agent. It obtains actions from multiple CoT trajectories. It employs the same prompt templates as in the CoT agent.
A22

---

First think about your current situation, then you must choose one action from legal actions set up advantages.
Your output must be in the following format strictly:
Thought: Your thought.
Action: Your action wrapped by <>, i.e. <format>
Remember, you can only choose one move from the legal actions.
ToT agent: we follow the text generation task implementation in the official codebase of ToT 16, Specifically, the ToT factorized into candidate thought generation, 2). thought voting, 3). candidate action generation, 4). action voting:
Here provide the basic prompt template used in ToT.
Step Prompt:
First think about your current situation, then choose one move from legal positions to set up advantages.
Your output should be of the following format:
Thought:
Your thought.
Move:
Your action wrapped with <>, e.g., <format>
After executing step prompts in breath-first search manner, utilize the original ToT vote prompt:
Vote Prompt: Given an instruction and several choices, decide which choice is most promis- ing. Analyze each choice in detail, then conclude in the last line "TThe best choice is s", where s the integer id of the choice.
A5.6 Sanity Check
To evaluate the effectiveness of our framework, perform sanity check by calculating the
50 completion rates of each game. The completion rates are calculated as N where N is the number of matches that will take achieve 50 valid matches. Here, a valid match means all the participants will always generate legal moves at each turn of the match. Results are summarized in Table A7. We show that all the L.M agents achieve 2 90% completion rate, showing that the prompts are properly configured and LLMSs are capable of following instructions to finish the game.
Table A7: Sanity check. The completion rates of LLM agents over all the games.
Rackiuns
Heaesning: Eac
Connect
Broaktbenagh
Lärt' Auction
Nupritsttin
Kntn
Nm
Flg
Prteners Ullemm
JT2
1075
1095
45%
45%
5005
1095
1015
100%
500%
1095
S-arte
Cst
1172
11972
45%
10075
1005)
1197%
103/
0852
1975
5005:
1175
2amm-2XXb-dae Mb-chal
Col
31%
95%
64%
1075
305
05%
100%
192%
1022
08%
90%
1107
9/7
1975
1972
1005
07%
CCT
1075
1175
1007
197%
1975
5005
91%
SC.CUT
718
1002
71%
1005
1006
716
1002
100%
1072
5306
926
Mistal-7s-Oe
SC.CUT
03%
100%
1005
1005
1006
1075
100%
100%
1079
1006
996
A6 How Temperature Affects LLM Performance
To study how the temperature used in generating LLMs' responses affects performances, we conduct experiments by making LLMs with 0.2 temperature (the default setting in our paper) play against LLMs with 0.4/0.6/0.8 temperature, over Baaeelll-aa33--sstttuu and GPT-3.5-turbo- 106. For each experiment, we run 20 matches. The reasoning method is the PromptAgent. The results are summarized as in Table A8. We show that larger temperature will result in worse performance for deterministic games, while has a model-specific effect for probabilistic games.
"nttps://github http::////gghhhuoooooooeem wowp:rincco/o.onncomooooooooooo. wttpiiicttooonnnntteeeeeeeee nlp/tree- oottttree--fttttee of thothttt http////////atat/rrr/////// prompts/text.py
A23

---

Table A8:
The affect of various temperatures for generation sampling.
Model
Temperature
avg. NRA Probabilistic Games
avg. NRA in Deterministic Games
CodeL 34b--Inttuucc
0.4
-0.13
-0.01
CodeLLlama3350 Saaa--3db---nstte
-0.16
-0.05
-0.16
-0.10
GPT-35-turbo
0.04
-0.10
GPT-35-turbo
GPT-35-turbo
0.02
-0.34
A7
Elo Rating System
The Elo rating system Elo (1960) is popular method for calculating the relative skill levels of players in two-player games such as Chess. It was used by various organizations rank players. Assume there are two players: A and B, and each player has a rating, R.A. RB, which is a numerical value representing their skill level. The expected score for a player is the probability that the player will win against another player:
EΛ 10( 1Q(RB-RA)/400 RA)/400
EB + 10(RA- A40)
After match between A and B, the real values, SA and SB, are defined as
Player wins, S = and BB =
If Player B wins, S and Sn 1 the game a draw, S = Sg = 0.5
Then, the updated rating It', and Rtn are calculated as:
R RA + K (SA EA))
Rg = RB K * (S - En));
where K constant that determines how much the rating changes after game. higher K results in larger change. In our paper, the initial rating is set to 1500, i.c., RA = Rg = 1500, and K = 20.
0.4
0.3
NRA
0.2
Prompts
Prompt Agent
0.1
CoT (used)
Other CoT
0.0
Prompt
Template 0 (used) Template 1
Template 2
Template 3
Template 4
Figure A7: Investigating the sensitivity of Chain-of Chain-of-Thought prompt. Prompt (used) and CoT (used) refer to the prompts utilized by the Prompt Agent and the CoT Agent in this paper. Results obtained from the model GPT-3.5-turbo over all the game-theoretic tasks. Please refer to Table A9 for Template 0 Template 4.
A24

---

A8 Chain-of-Thought Sensitivity
We provide five different CoT strategies over the GPT-3.5-turbo model as shown in Table A9 the results presented in Figure A7.
We also include instances shown in Table A10 where CoT agents unable produce legal outcomes, which serves as evidence of their limitations.
Table A9: Different Chain-of-Thought strategies.
Index
Chain Chain-of-Thought Prompt
0
First, think about your current situation. First, think about your current situation by thinking carefully, logically, and step by step. First, provide your thoughts.
2
First, analyze and think about your current state and provide your strategy beat your opponent.
4
First, analyze the state and provide your strategy to beat your opponent.
Table A10: Fail cases different Chain-of- Chain-of-Thouugt prompts.
Game
Chain-of-Thouue Prompt
Thought:
my opponent advanced their pieces quite far,
and I need to make strutegic moves to gain advantage. should prioritize moving my pieces forward while also considering potential capture opportumities.
Breukthrough
Action: <c4->b5>
Thought: Considering the current board position and the opponent's seems that the best move would be to advance the white piece at b3 to c4, capturing the black piece c4. This move would help gaining control of the board and putting pressure the opponent.
Breakthrough
Action:
<cc3 >c0*>>
Thought: I see that the opponent's 's pieces are advancing towards my home sow, and they have already captured one my pieces. advancing my pieces and try create opportunities for capturing the opponent's pieces
Breakthrough
need to gain an advantage.
Action: <b2->b3>>
Thought:In this state, total number matches which losing position Nim. To win. I need to force the total number of matches to be multiple of 4 after my turn. This means should uim to leave 3, 7, 11, or 15 matches for my opponent after my move.
Nim
Action: <pile:4, take:3>
The current of game such that the number matches remaining 3, which is a losing position for the current player.
Thought:
Nim
To win, the strategy should be to force the opponent into losing position by making move that leaves 2 matches remaining.
Action: Actiom:<pile:4, take:6>
A9 Error Pattern Demonstrations
present demonstrations of error patterns Table AIl.
A25

---

Table A11: We conclude common error patterms in GTBENCH, including misinterpretation, factual error, math calculation, and over-confidence. For each demonstration, the explanation explains why L.M generation incorrect.
Game
Category
Demonstrations
"Considering the current situation, moving the piece at b6 to either a5, b5, or c5 would al- low me to advance further into the opponent's territory. However, this could potentially expose my piece be- ing captured. On the other hand, moving piece from the back row (a8, b8, or c8) to b7 could potentially block the opponent's 's advance and provide a defensive line. Action:<bbs <b8->>>>>>
LLM Generation:
Breakthrough
Factual Error
Explanation: In the third round of this match, moving the back row renders it impossible to hinder the oppo- nent's forward movement.
State: matches in Pile 5 matches in Pile 3, and 7 matches Pile 4.
Nim
Calculation Error
"The current state of the game is such that the 2nd pile has 3 matches, the 3rd pile has 5 matches, and the 4th pile has matches. The goal is to avoid being the one to pick up the last match. To achieve this, need to consider the binary representa- tions of the pile sizes and aim to make the XOR of all pile sizes equal zero. This will put me win- ning position. Currently, the XOR all pile sizes (011 in binary). To make this zero, I need to remove 4 matches from the 3rd or 4th pile. Action:<pile:3, take:4>"
LLM Generation:
Explanation: The XOR the binary representations 3, 5, 7 is 1, not 4.
Iterated Prisoner's Dilemma
LLM Generation: "Given the history the previous round, where both my partner and chose to remain silent, it seems that we are both inclined towards coop- eration. This is positive sign, it indicates level trust and mutual understanding. Considering the potential outcomes, the safest option is to remain silent, If my partner also remains silent, we both serve the least possible time. Action: <Silent>" Explanation: The player overconfident that the part- ner will cooperate and keep silent.
Over-contidence
have analyzed the current sit- uation and identified that my opponent has taken the center column, which gives them significant advan- tage. To counter this, will take the corner column to create potential winning line. Action: <CI>." Explanation: Currently, the center column is unoceu- pied.
LLM Generation:
Connect-4
Misinterpretation
The opponent finished actions <C2R1> and <C2R2>, the agent finished moves <CIR1> and <CRR>>
State:
Tic-Tac-Toe
Endgame Misdetection
LLM Generation: have mark in the first column, and one in third. could potentially win get the chance to mark the third row of the first column in the next turn. Action: <CIR3>
Explanation: This immediate loss situation be- cause the opponent has two pieces connected in row and <C2R3> is unoccupied. The agent missed that they should take action <C2R3> to avoid lose
A26

---

A10
Regret Value
A10.1
Regret Value for Blind Auction
Assume that the last round of bidding was b1 for the player and by for the opponent. Assume the player's valuation is u, then the regret value calculated by
if b_ > b_2 + 1: regret b_1 (b_2
else:
(b_2 1) < v:
regret
(b_2 + 1)
else:
regret 0
A10.2 Regret Value for Iterated Prisoner's Dilemma
The regret value Iterated Prisoner's Dilemma is simply the accumulation of the regret value of per-turn Prisoner's s Dilemma:
if player_move
'Testify' and opponent_move 'Silent';
regret = 0
elif player_men _moee 'Testify' and opponent_mov move 'Testify':
regret 0
elif player_mov 'Silent' and opponent_move Testify':
regret 1
else:
regret 2
A11
User Interfaces of GTBench Leaderboard
The user interfaces of GTBENCH leaderboard are presented in Figures A8 and A9.
A27

---

GTBench: Uncovering the Strategic Reasoning Limitation of LLMs via Game- Theoretic Evaluations
mesoring
pirse-thoveeie prove. mongr ge sapporedd Gpentizzieee
eokktt will meag sibed LMM agents aa aguge dlier manner collatio code pro tampaats can fo Ellient
The pempbian boorowed form Gram Leadertiand
GTBench
About
Candent Pear
Adra-laccion
Mulla Polee
Gale aoth sodhrworeat
Aepelanii
Bwoand loren''
Maaul medrica
Apera
Pronpe
Gel
socall
Moed
pompleget
Model
Eppament Rodal
Avezaage
Tic-Tac-To
Connact Fouz
Dessichcnug
Lixe's Oice
E2didd Auctdes
agert
pooepp Agprt
GPT-3 S- Butbe
SFT-2 2.5 www.o 110-00-00000 5406
proepp Agertt
0
0.237
0.32
-0.12
2:55
0.025
OPT-3 5- Hutube
50-5o7
0FT-3 tittttooii 5.000
0.425
0..04
3.50
2.52
peovept: Agert
SC-SUT
proept:
Agertt
48.56
2.58
Coole 34b- Inatouct
EeT
SFT- 3, 2i-teboo-110
ptoeep: agortt
-0.000
0.336
08.36
4.75
-3.32
0,300
Liom 78t chat
5.5 txato 1300
0,089
0.8
8:30
Mictmal-
Figure A8: The user interface of GTBENCH leaderboard.
A28

---

GTBench: Uncovering the Strategic Reasoning Limitation of ML via Theorete Evaluations
Otteach ainu evaluato and raskiLLM LLd ruser ing abilltioo ie congediivee envinomennt nough Peoreiiciass, e.g. board and card garnes it atilless wide recggried games supported by Decaa ond mel une wel recog rird LUN LiNagamm a longa drwwnmmn the not cndon pno pemplaaes can Soandie Ellbenst
relor to
garnei metrics.
borroaed
Leatectours.
GT CTBBench
About
Selecrook
Moddl
Message
Draithhough
Coeweri
Dindtkuction
2014
diruseettee
Cocalarablo
Jarl/Dice
Aegolatton
Min
Hamtaed Prtseerrs Slammn
Agait
scor
OpporarsHoce
Model
Agent
Oppoment Podel
Oppanent Agent
Averaee
Drwakthough
Connect Fest
Blded Auction
Kam Pekoc
Liee's
Segettatt
kaabo
EFT-3 205-3.5-00 tacby Aile
posnet agant
Pronet
19tV Aetteia 11ts
presst agost
27T-2.5-tatbe
Tol
posnat agest
0.36
0.54
-5.32
0.100
1FT-3 tucho A1ds
2,000
2.085
-2.2
3.205
Figure A9: The user interface of GTBENCH leaderboard when various LLMss/geets and opponents are selected.
A29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We summarize main empirical observations and conclusions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section Limitation
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't tt acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
A30

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide details about models and generative hyper-parameters' in Section 4, and all the prompts utilized in Appendix A5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
A31

---

Answer: [Yes]
Justification:
We provide access to the data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
on oolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips.co
wwt.:ssubmsssioosooooo.oo onsllicy)) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 3, Section 4, Appendix A5, A6, A7. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
A32

---

It is OK to report 1- I-sigma enror bars, but one should state it. The authors should preferably report 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The results are obtained from endpoint API providers, e.g., OpenAl (Section
4).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't tt make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https http:////euusppeeeeeeeeeeer wwwpuurps.ccpuuuuueeeeee
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:[Yes]
Justification:
Section Impact Statements
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
A33

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: Section Impact Statements
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Section 4
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
A34

---

13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code//modll as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
A35


Reviews:
[{"review_id": "IuXeFqX6t2", "reviewer": "Reviewer_VyBq", "summary": "This paper introduces GTBench, a set of 10 different games to test how well large language models can think strategically. The author found that while LLMs struggle with complete and deterministic games like Tic-Tac-Toe and Connect-4, they perform better in incomplete uncertain games like poker and negotiation. Code-pretraining improves their strategic thinking abilities. However, advanced thinking methods like Chain-of-Thought and Tree-of-Thought don’t always help and can sometimes make things worse. The latest open-source models, like Llama-3, are getting closer in performance to commercial models like GPT-4. Common mistakes LLMs make include misunderstanding game rules, being over-confident, and making calculation errors.", "questions": "Could you address the weakness 1, and try to discuss weakness 2-4?", "limitations": "If LLMs are trained on biased data, they might reinforce existing biases in strategic decision-making. Testing the LLM with different personas could show if this changes the game results and reveal potential biases.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to understand. \n2. The problem of evaluating LLMs' strategic reasoning abilities is meaningful. Creating such a benchmark is valuable for the research community.\n3. The paper provides a detailed evaluation of LLMs across different game tasks. These tasks indeed measure the strategic reasoning of LLMs, even if some models already understand the optimal algorithms for those games. (For example, you could ask GPT-4 about the optimal strategy for some of these games, and it knows the optimal algorithm.)\n4. The authors conducted extensive experiments using various base models, including reasoning methods like ToT and CoT. They had some interesting findings and analysis (concluded in the summary).", "weaknesses": "1. The paper claims that measuring strategic reasoning capabilities with games is missing in existing benchmarks. However, there are other benchmarks, such as MAgIC released last year, that consider benchmarking LLMs' strategic behavior using games. While there are differences, this weakens the claim of novelty.\n2. Some of the selected games, like Tic-Tac-Toe, have known optimal strategies and are not complex enough. These games might not fully challenge the advanced strategic reasoning capabilities of LLMs. Even though the current evaluation is useful, as a benchmark intended for future use, it should be capable of evaluating more advanced or adapted LLM agents.\n3. The benchmark focuses on a set of 10 games. It’s unclear how well the findings generalize to other strategic scenarios, even similar types of tasks. The results appear to be quite case-by-case. A broader range of tasks and scalable evaluation frameworks would make the benchmark more comprehensive.\n4. The experiments primarily involve LLMs and traditional solvers. There is a lack of evaluation against human opponents, which could provide more insights into the models' performance in real-world strategic interactions. As a benchmark, I also expect to have other opponents (for example, the optimal algorithm, the RL based agent).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable and insightful comments!\n\n>W1: The paper claims that measuring strategic reasoning capabilities with games is missing in existing benchmarks. However, there are other benchmarks, such as MAgIC released last year.\n\nIn Line 119, we meant to convey that some of the games and the taxonomy introduced in GTBench are not used by current benchmarks for evaluating LLM strategic reasoning. We recognize existing benchmarks that evaluate LLMs through games, such as Clemency, LMRL-Gym, and LLMArena, discussed in Section 2.1. While MAgIC focuses on the complex social and cognitive dimensions of LLMs through various games, our focus is on pure logical game-theoretic evaluation, which could complement our work. We will add more discussion about MAgIC and revise our statement in the next revision.\n\n>W2: Some games have known optimal strategies that might not fully challenge the capabilities of LLMs. A benchmark intended for future use should be capable of evaluating more advanced or adapted LLM agents.\n\nWe would like to mention that only Tic-Tac-Toe and Connect-4 are solved games, meaning they have optimal strategies, while other games in GTBench remain complex and challenging without known optimal strategies. We are aware that some strategies for poker games are \"approximately\" optimal, but these strategies incorporate bluffing, which introduces uncertainty and prevents them from being universally optimal.\n\nAdditionally, we would like to emphasize the following points:\n\n1. GTBench includes both simple games like Tic-Tac-Toe and complex ones like Breakthrough, with up to 48 actions per turn. These complex games, popular in strategy competitions, pose a significant challenge for LLMs already struggling with simpler games.\n\n2. LLM agents, even with optimal strategies, may still produce errors like hallucinations. Thus, comparing performance using statistics like the number of draws and deviations from the optimal strategy is valuable.\n\n3. GTBench is flexible and extensible, with modular game implementations independent of LLM agent design. It will be maintained long-term, supporting more complex games and advanced agents in the future.\n\nWe believe that the proposed GTBench will continue to be valuable for advancing LLM research. We will incorporate this discussion in our next revision.\n\n>W3: It’s unclear the generalization of findings to other scenarios. A broader range of tasks and scalable evaluation frameworks would make the benchmark more comprehensive.\n\nWe would like to mention that GTBench focuses on gameplay and includes various strategic scenarios like competition, negotiation, and collaboration (Table 1). Our conclusions are based on general trends across all scenarios, not focus on specific types. For example, we compare the NRA of code-pretrained LLMs and chat LLMs across all scenarios to assess the effect of code pretraining, providing a broad overview of LLM agent performances.\n\nTo address scalability, we offer adjustable action/state spaces and complexity, such as varying board sizes for Breakthrough and player numbers and winscore for PIG. As complexity increases, the advantage of powerful LLM agents like GPT-4 over GPT-3.5-turbo decreases, but our overall conclusions remain valid. Here are related results (GPT-3.5-turbo PromptAgent vs MCTS and GPT-4, reporting NRA/win rate over 50 matches):\n\n|Breakthrough|Column=3 (default) | Column=4|\n| --- | --- | --- |\n|Gpt-3.5-turbo vs MCTS |-1 |-1|\n|Gpt-4 vs GPT-3.5-turbo |0.32 |0.26|\n\n|3-Player Pig (GPT-3.5-turbo)|PromptAgent |CoTAgent |ToTAgent|\n| --- | --- | --- | --- |\n|Win rate | 46% | 42% | 12% |\n\n|Pig |Winscore=20 (default) |Winscore=30|\n| --- | --- | --- |\n|Gpt-3.5-turbo vs MCTS |-0.44 |-0.40|\n|Gpt-4 vs GPT-3.5-turbo |-0.04 |-0.06|\n\nWe conclude that the relative advantage between LLM agents diminishes as games become more challenging. For example, increasing the board size in Breakthrough from 3 to 4 reduces GPT-4's NRA from 0.32 to 0.26, though it still outperforms GPT-3.5-turbo. This occurs because participants tend to show similar skill levels in more complex games. We will explore more extensible settings and support diverse games in future revisions.\n\n\n>W4: There is a lack of evaluation against human opponents and other opponents (for example, the optimal algorithm, the RL based agent).\n\n\nWe conducted a preliminary human opponent evaluation by selecting 5 games (2 deterministic and 3 probabilistic) from GTBench. Five graduate students who are familiar with the rules but not experts in these games, played 10 matches against MCTS and GPT-3.5-turbo for each game:\n\n| setting | Tic-Tac-Toe | Connect-4 | Kuhn Poker | Liar’s Dice | Blind Auction |\n| --- | --- | --- | --- | --- | --- |\n|Human vs MCTS|-0.25|-0.55|0.09|-0.60|0.15|\n|MCTS vs Human |0.25 |0.55 | -0.09 | 0.60 | -0.15 |\n|GPT-3.5-turbo vs Human | -0.55 | -0.60 | -0.53 | -0.88 | -0.46 |\n|Human vs GPT-3.5-turbo | 0.55 | 0.60 | 0.53 | 0.88 | 0.46|\n\nThe average NRA across participants showed that humans with moderate knowledge significantly outperform LLM agents in strategic reasoning in both complete and incomplete information scenarios. This highlights the limitations of current LLMs and the need for improvement. We will include these results and provide more detailed evaluations in our next revision.\n\nWe agree that MCTS may not be optimal for probabilistic games like poker. We implemented the **Counterfactual Regret Minimization (CFR) solver**, known for finding Nash equilibria in incomplete games[1]. In 100 matches of Kuhn Poker, the CFR solver had a 54% win rate over MCTS, showing a slight advantage. We re-ran GPT-4 experiments against CFR, and GPT-4 with Prompt achieved a 0.33 NRA, consistent with our paper's conclusions. We will include more solvers and these results in our next revision.\n\t\nReference:\n\n[1]Zinkevich, Martin, et al. \"Regret minimization in games with incomplete information.\" Advances in neural information processing systems 20 (2007).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U12aTHNgPA", "reviewer": "Reviewer_HsvL", "summary": "The paper proposes a benchmark to understand the strategic reasoning capabilities of llms. The authors present a suite of game theoretic tasks with different structures to do this. They use different evaluation metrics like ELOs and Relative advantage to compare different llms and prompting methods.", "questions": "See weaknesses.", "limitations": "The authors do a good job of addressing limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is clearly written and well motivated. It provides some structure to the growing literature of strategic reasoning with llms.\n- A wide range of closed source, open source models are tested. A good set of prompts are used to test the models too!\n- I particularly liked table 1 and the selection of different tasks with different characteristics.\n- The normalized relative advantage is a good, interpretable metric\n- The framework and taxonomy are clear and easy to understand.\n- Section 4.4 gave some good insight into the types of errors made by llms\n- I also liked reading the analysis in section 4.3, in particular that code pretraining helps with strategic reasoning.", "weaknesses": "- Characterizing human performance would strengthen the paper\n- Including some qualitative reasoning traces of successes and failures might be insightful.\n- Minor: This paper would be an ideal fit for the datasets and benchmarks track, instead of the main track. I dont think it should be penalized for this though!\n\nTypos\n\nLine 79: Characterize\n\nLine 171: dynamic gaming → dynamic game", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable and insightful comments!\n\n>W1: Characterizing human performance would strengthen the paper\n\nWe provide a preliminary human opponent evaluation. Specifically, we selected 5 games from GTBench and organized matches with 5 graduate students. These participants are familiar with the rules but are not trained experts. Each human competed against MCTS and GPT-3.5-turbo in 10 matches for each game. We report the average NRA across these human participants.\n\n| setting | Tic-Tac-Toe | Connect-4 | Kuhn Poker | Liar’s Dice | Blind Auction |\n| --- | --- | --- | --- | --- | --- |\n|Human vs MCTS|-0.25|-0.55|0.09|-0.60|0.15|\n|MCTS vs Human |0.25 |0.55 | -0.09 | 0.60 | -0.15 |\n|GPT-3.5-turbo vs Human | -0.55 | -0.60 | -0.53 | -0.88 | -0.46 |\n|Human vs GPT-3.5-turbo | 0.55 | 0.60 | 0.53 | 0.88 | 0.46|\n\nOverall, our results indicate that even humans with moderate knowledge of these games outperform LLM agents significantly in strategic reasoning, in both complete and incomplete information scenarios. This underscores the limitations of current LLMs in strategic reasoning and highlights the need for improvement in this area. We will include these results and provide more detailed human-as-opponent evaluations in our next revision.\n\n>W2: Including some qualitative reasoning traces of successes and failures might be insightful.\n\nWe have identified five common error patterns and documented the corresponding failure cases in Table A11 of our manuscript to provide a qualitative analysis. Here, we present a pair of success and failure logs from the Tic-Tac-Toe environment, with detailed annotations explaining the intent behind each step (the intent is derived from the CoT reasoning):\n\n**Success Traces (GPT-4-turbo w/ CoT Agent  beats Human)**\n\n**Step 1 (CoT Agent)**:\nThought: As the first player, I have the advantage of making the first move. The center square is the most strategic position to start with, as it provides the most opportunities for creating a line of three. Action:<C2R2>\n\n**Step 2 (Human)**:<C3R2>\n\n**Step 3 (CoT Agent)**: Thought: Looking at the current state of the game, I can see that the center and one of the side squares are occupied. To maximize my chances of winning, I should aim to control one of the corners. This will give me multiple ways to create a line of three. Considering the available moves, I can choose either <C1R1> or <C3R1> to achieve this.Action:<C1R1>\n\n**Step 4 (Human)**: <C3R3>\n\n**Step 5 (CoT Agent)**: Thought: Looking at the current state of the game, my opponent has two marks in the third column. If they get another turn, they might place their mark in the first row of the third column and win the game. Therefore, I need to block this potential winning move. Action:<C3R1>\n\n**Step 6 (Human)**: <C2R1>\n\n**Step 7 (CoT Agent)**: Thought: Looking at the current state of the game, I notice that the opponent has two of their symbols in the third column. If they get a chance to place their symbol in the third row of the first column, they will win the game. Therefore, I need to block this potential winning move by placing my symbol in the third row of the first column. Action:<C1R3>\n\n**CoT Agent wins!** In this match, the CoT agent successfully blocked the human's connected pieces and identified its own winning move.\n\n**Failure Traces (Human Beats GPT-4-turbo w/ CoT Agent)**\n\n**Step 1 (Human)**: <C2R2>\n\n**Step 2 (CoT Agent)**: Thought:The opponent has taken the central position on the board, which is a strategic advantage because it opens up the most possibilities for creating a line of three symbols. To counter this, I should aim to block potential winning moves and set myself up for multiple winning paths. Placing my symbol in a corner can be a good strategy because it gives me two potential ways to create a line of three symbols (**NOTE: correct and popular strategy**). Among the corners, choosing <C1R1> or <C3R3> seems most advantageous as it does not immediately threaten by the opponent's current position but sets up a potential win in multiple directions. Action:<C1R1>\n\n**Step 3 (Human)**: <C2R1>\n\n**Step 4 (CoT Agent)**: Thought: Given the current state of the game, my opponent has placed their symbols in C2R2 and C2R1, forming a potential line vertically in column 2. My symbol is in C1R1. To maximize my chances of winning or at least blocking my opponent's path to victory, I should consider creating a two-front threat that forces my opponent to defend rather than attack (**NOTE: A hallucination occurred here; the opponent already has two connected pieces, so the first priority should have been blocking the opponent**). Placing my symbol in C3R3 would start to form a diagonal line from C1R1 to C3R3, and it also positions me to potentially create a vertical line in column 3. This move doesn't immediately block my opponent's potential win but sets up a strategic position for future moves.Action:<C3R3>\n\n**Step 5 (Human)**: <C2R3>\n\n**Human Wins!** The CoT agent lost because it failed to correctly prioritize its actions. With the opponent already having two connected pieces and no immediate winning move available for the CoT agent, the primary focus should have been on blocking the opponent.\n\nWe will add more qualitative results in our next revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zdT1ypHD1c", "reviewer": "Reviewer_PA1v", "summary": "This paper proposes a benchmark for evaluating the strategic reasoning of LLMs. The benchmark includes ten games of various types. The authors use these games to conduct competitive experiments between LLMs and traditional methods, as well as LLM-vs.-LLM. The paper then analyzes the experimental results and model behavior, and examines the game-theoretic properties of LLMs.", "questions": "1. In Section 4.1, why does the tree-like prompting strategy ToT still lag significantly behind MCTS?\n2. Is there any reference to classifying games in the benchmark? Why is it classified this way?\n3. Why does the model perform better in probabilistic and dynamic games than in completely deterministic games? Is it that LLM performs better or that MCTS performs worse, making LLM appear better?", "limitations": "The authors have fully addressed the limitations in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is logically clear, understandable, and well-written.\n2. The experiments are comprehensive. The authors evaluate comparisons between LLMs and traditional methods and LLM-vs.-LLM competitions. They include multiple open-source and closed-source models and tests of various prompting methods.\n3. The authors evaluate game-theoretic properties, including Nash equilibrium with regret and Pareto efficiency.", "weaknesses": "I didn't find any significant weaknesses, only a few questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable and insightful comments. \n\n>Q1: In Section 4.1, why does the tree-like prompting strategy ToT still lag significantly behind MCTS?\n\nThere are two potential reasons:\n\n1. **Exploration Space**: MCTS has a significantly larger exploration space compared to ToT. In our experiments, MCTS is allowed to execute up to 1000 simulations to determine the next action. However, due to the complexity of the game and the cost of tokens, it is infeasible for ToT to traverse such a large number of simulations. For complex games like Breakthrough, completing even one full game simulation is impossible for ToT.\n\n2. **Reward Backpropagation**: MCTS uses the actual win/loss signal obtained from simulations as a reward to determine which action should be selected. In contrast, ToT relies on LLMs voting or grading the actions, which is less accurate compared to the reward mechanism used by MCTS.\n\n>Q2: Is there any reference to classifying games in the benchmark? Why is it classified this way?\n\nWe primarily classify games based on their game-theoretic properties, such as complete versus incomplete information, dynamic versus static, and probabilistic versus deterministic scenarios, which are fundamental and widely recognized in the field of game theory [1-3]. After categorizing the games according to these properties, we then identified and summarized the most commonly preferred abilities (Table 1).\n\nReference:\n\n[1]Fraser, Niall. Conflict analysis. Ed. Keith W. Hipel. North-Holland, 1990.\n\n[2]Osborne, Martin J. An introduction to game theory. Vol. 3. No. 3. New York: Oxford university press, 2004.\n\n[3]Lanctot, Marc, et al. \"OpenSpiel: A framework for reinforcement learning in games.\" arXiv preprint arXiv:1908.09453 (2019).\n\n>Q3: Why does the model perform better in probabilistic and dynamic games than in completely deterministic games? Is it that LLM performs better or that MCTS performs worse, making LLM appear better?\n\nCompletely deterministic games are those in which all players have full knowledge of the game's state and the actions of other players. In this case, search-based solvers such as MCTS exhaustively explore possible future states and use statistical sampling to make highly informed decisions based on full information. Thus, MCTS approaches near-optimal performance in complete games, which significantly outperforms LLM agents. However, some information is hidden or unknown to the players (such as dice values or poker cards) in incomplete and probabilistic games, making MCTS simulation unavailable due to the hidden full-state information or uncertainty. \n\nWe agree that MCTS may not be optimal for probabilistic games such as poker games. We further implemented the well-known **Counterfactual (CFR) solver** that is proven to be effective in finding Nash equilibria for in-complete games [1-3]. Then we conduct MCTS-vs-CFR experiments in 100 matches on the Kuhn Poker environment. The win rate of the CFR solver is 54%, which shows a slight advantage compared to the MCTS solver. We then re-run GPT-4 experiments when playing against the CFR solver. We observe that GPT-4 w/ Prompt achieves 0.33 NRA when against CFR, indicating that the conclusions presented in our paper are still consistent. We will include more CFR results in our next revision.\n\t\nReference:\n\n[1]Zinkevich, Martin, et al. \"Regret minimization in games with incomplete information.\" Advances in neural information processing systems 20 (2007). \n\n[2]Tammelin, Oskari, et al. \"Solving heads-up limit texas hold'em.\" Twenty-fourth international joint conference on artificial intelligence. 2015. \n\n[3]Moravčík, Matej, et al. \"Deepstack: Expert-level artificial intelligence in heads-up no-limit poker.\" Science 356.6337 (2017): 508-513.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4dpPjtSuq7", "reviewer": "Reviewer_L1Yq", "summary": "This paper tries to evaluate the strategic reasoning abilities of LLM. Therefore, 10 games are chosen where LLMs is trying to solve the game. This paper includes various open- and closed-source LLMs into consideration and build a benchmark for easy evaluation.", "questions": "1. Does the evaluation really evaluate the strategic reasoning? Basically the evaluation is letting the LLM to play as one of the player in the game. However, this is much like a decision making problem, especially when the opponent is also a LLM-agent, where the LLM agent is largely stationary. Therefore, I would like to ask the authors provide the justification about why the evaluation is about strategic reasoning, not the decision making? \n\n2. Also about the strategic reasoning. The selected games only focus on competitive zero-sum games. what about general-sum and multi-player games? Are cooperative games, e.g., hanabi, also requiring strategic reasoning? Even further, mixed cooperative and competitive, e.g., soccer, need the strategic reasoning? I think the strategic reasoning is not well-defined and fully discussed. \n\n3. Does the evaluation really unlock the abilities of LLMs? The evaluation is focusing the prompting. However, for games, especially unfamiliar games for LLMs, exploration is important. Therefore, a memory or long in-context learning of the exploration experience should be included for the evaluation of strategic reasoning in games.", "limitations": "More limitations should be discussed.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "Evaluating the strategic reasoning is important and the evaluation includes various LLMs into consideration.", "weaknesses": "The evaluation protocol is questionable. More comments and questions are in the following section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable and insightful comments!\n\n>Q1: Does the evaluation really evaluate the strategic reasoning? Basically the evaluation is letting the LLM to play as one of the player in the game. However, this is much like a decision making problem, especially when the opponent is also a LLM-agent, where the LLM agent is largely stationary. Therefore, I would like to ask the authors provide the justification about why the evaluation is about strategic reasoning, not the decision making?\n\t\nConceptually, strategic reasoning and decision making are related yet distinct in their focus[1-2]: Decision making tends to be more (1) **immediate** and (2) **focused on selecting the best option among alternatives**, whereas strategic reasoning involves (1) **long-term planning** and (2) **anticipating others’ actions**. In Section 2.2, we explicitly describe the gameplay process as a Markov Decision Process, involving multiple players and multi-turn action execution by participants.\n\nGames in GTBench involve long-term planning and emphasize competition arising from others’ actions. Specifically, each game comprises multiple turns, with Breakthrough often taking over 20 turns per match. In each turn, the decision of an LLM agent is constrained by factors such as game rules, past moves from opponents, and the current game state. The agent must achieve long-term planning, such as constructing a **FORK** in board games which requires multiple turns, and maintain long-context reasoning, such as in Negotiation which involves reviewing past moves and actions, with participation from at least two players. From this perspective, GTBench aligns more closely with strategic reasoning than with immediate decision making. We agree that decision making is the foundation of strategic reasoning, and we will add further discussion on the scope of this paper in our next revision.\n\nReference:\n\n[1]Kahneman, Daniel. Thinking, fast and slow. macmillan, 2011.\n\n[2]Dixit, Avinash K., and Barry Nalebuff. The art of strategy: a game theorist's guide to success in business & life. WW Norton & Company, 2008.\n \n\n>Q2: Also about the strategic reasoning. The selected games only focus on competitive zero-sum games. what about general-sum and multi-player games? Are cooperative games, e.g., hanabi, also requiring strategic reasoning? Even further, mixed cooperative and competitive, e.g., soccer, need the strategic reasoning? I think the strategic reasoning is not well-defined and fully discussed.\n\nWe would like to highlight that GTBench does **NOT** solely focus on zero-sum games: the taxonomy of zero-sum versus general-sum is a major classification in GTBench (Table 1, column 2), and 4 out of the 10 games are general-sum, specifically Negotiation, Blind Auction, Pig, and Iterated Prisoner’s Dilemma.\nWe agree that cooperative games and multi-player environments provide valuable evaluations for strategic reasoning. Two of our games, Negotiation and Iterated Prisoner’s Dilemma, already involve collaboration. The mentioned Hanabi is a purely cooperative environment, which poses challenges in identifying individual relative skills during gameplay. We will extend our GTBench to multi-player games. In fact, GTBench already supports it for some games. For instance, we implemented the game PIG with 3 players: GPT-3.5-turbo PromptAgent vs CoTAgent vs ToTAgent. We report the win rate of each player over 50 matches:\n\n|agent|PromptAgent|CoTAgent|ToTAgent|\n|---|---|---|---|\n|Win rate|46%|42%|12%|\n\nWe would like to emphasize that LLM strategic reasoning evaluation is still in its early stages. This paper aims to evaluate the strategic reasoning of LLM agents in competitive environments. Experimental results indicate that LLM agents are largely ineffective in these simple competitive games. Therefore, GTBench could serve as a unified starting point for this research domain. Our GTBench will be maintained long-term and will support more diverse and complex strategic scenarios in the future.\n\n>Q3: Does the evaluation really unlock the abilities of LLMs? The evaluation is focusing the prompting. However, for games, especially unfamiliar games for LLMs, exploration is important. Therefore, a memory or long in-context learning of the exploration experience should be included for the evaluation of strategic reasoning in games.\n\nWe acknowledge that designing advanced exploration mechanisms and unlocking the abilities of LLMs in strategic reasoning is impactful. However, our current paper **does not** aim to fully realize the potential of LLMs in these games. This is because a unified strategic reasoning benchmark is needed before delving into such designs. The scope of this paper is to (1) provide a unified evaluation framework and (2) benchmark common LLM reasoning agents in various game-theoretic scenarios. Designing advanced strategic reasoning agents and fully unlocking model capabilities will be our next step in this domain. It is also worth noting that the implemented Tree-of-Thought (ToT) agent [1] integrates exploration into both thought space and action space through thought decomposition and generation. However, ToT is less effective due to (1) limited exploration depth and (2) inaccurate reward estimation. This underscores that designing effective exploration mechanisms for LLM agents in strategic reasoning remains an open problem, requiring specific design considerations. We will address this in future work and discuss it further in our next revision.\n\nReference:\n\n[1]Yao, Shunyu, et al. \"Tree of thoughts: Deliberate problem solving with large language models.\" Advances in Neural Information Processing Systems 36 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General Response

We appreciate all the valuable comments from the reviewers. We are pleased to know that our work is considered meaningful (Reviewer **VyBq**, **L1Yq**), valuable (Reviewer **VyBq**), comprehensive (Reviewer **VyBq**, **PA1v**), and insightful (Reviewer **PA1v**, **HsvL**).

Here are the major changes we have made:

1. We have added Human-as-Opponent experiments for further analysis (Reviewer **VyBq**, **HsvL**).
2. We have included the Counterfactual Regret Minimization (CFR) solver as a more powerful opponent in probabilistic games (Reviewer **VyBq**, **PA1v**).
3. We have provided detailed success/failure traces for qualitative analysis (Reviewer **HsvL**).
4. We have introduced extensible experiments for scalability, such as variations in board size, multi-player settings, and different gameplay win scores (Reviewer **VyBq**, **L1Yq**).
5. We have clarified the scope and the goal of this paper.

Further details are available in our individual responses. We are also open to providing additional clarifications and addressing any other concerns the reviewers may have.

Meta-review: The paper introduces a benchmark designed to evaluate the strategic reasoning capabilities of LLMs using a suite of ten diverse games. These games range from deterministic ones like Tic-Tac-Toe and Connect-4 to uncertain, incomplete information games like poker and negotiation. The study finds that LLMs generally struggle with deterministic games but perform better in uncertain scenarios. Code-pretraining enhances their strategic thinking, though advanced methods like Chain-of-Thought and Tree-of-Thought can be counterproductive in some cases. The paper highlights that while open-source models like Llama-3 are closing the gap with commercial models like GPT-4, common errors persist, such as misunderstanding game rules, overconfidence, and calculation mistakes. The benchmark also facilitates competitive experiments between LLMs and traditional methods, as well as LLM-vs.-LLM, providing a comprehensive analysis of the game-theoretic properties of these models.

In general, reviewers were relatively positive about the benchmark, although it would have been nice (as always) to see a larger number of games and solvers. Nonetheless, the games selected provide a good range over categories (such as collaborative vs competitive, perfect vs hidden information, etc). The main point of concern is around the central claim as to whether the benchmark does in fact probe the ability of LLMs to perform strategic reasoning or whether this is simply a case of decision making.  Certainly the title is slightly misleading, in that the benchmark merely serves to evaluate, as opposed to "unlock" strategic reasoning abilities (which hopefully work down the line will do). Nonetheless, I think that the current benchmark is useful and intersting enough to be of utility to the field.
Decision: Accept (poster)