GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations

Authors: jinhao  duan
renming  zhang
james  diffenderfer
bhavya  kailkhura
lichao  sun
elias  stengel-eskin
mohit  bansal
tianlong  chen
kaidi  xu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 22:28:52

Abstract: As Large Language Models (LLMs) are integrated into critical real-world applications, their strategic and logical reasoning abilities are increasingly crucial. This paper evaluates LLMs' reasoning abilities in competitive environments through game-theoretic tasks, e.g., board and card games that require pure logic and strategic reasoning to compete with opponents. We first propose GTBench, a language-driven environment composing 10 widely-recognized tasks, across a comprehensive game taxonomy: complete versus incomplete information, dynamic versus static, and probabilistic versus deterministic scenarios. Then, we (1) Characterize the game-theoretic reasoning of LLMs; and (2) Perform LLM-vs.-LLM competitions as reasoning evaluation. We observe that (1) LLMs have distinct behaviors regarding various gaming scenarios; for example, LLMs fail in complete and deterministic games yet they are competitive in probabilistic gaming scenarios; (2) Most open-source LLMs, e.g., CodeLlama-34b-Instruct and Llama-2-70b-chat, are less competitive than commercial LLMs, e.g., GPT-4, in complex games, yet the recently released Llama-3-70b-Instruct makes up for this shortcoming. In addition, code-pretraining greatly benefits strategic reasoning, while advanced reasoning methods such as Chain-of-Thought (CoT) and Tree-of-Thought (ToT) do not always help. We further characterize the game-theoretic properties of LLMs, such as equilibrium and Pareto Efficiency in repeated games. Detailed error profiles are provided for a better understanding of LLMs' behavior. We hope our research provides standardized protocols and serves as a foundation to spur further explorations in the strategic reasoning of LLMs.

Main paper: 
 introduction: 
 Large Language Models (LLMs) are increasingly being integrated into critical real-world applications, such as cybersecurity (Ameri et al., 2021; Aghaei et al., 2022) , decision science (Jiang et al., 2023b) , and finance (Wu et al., 2023) . These areas involve advanced strategic thinking and logical reasoning skills, including the ability to foresee possible dangers and weaknesses (Yao et al., 2024b; Duan et al., 2024a) , systematically examine difficulties, and make informed decisions based on provided evidence. However, evaluation environments that thoroughly assess these situations are not sufficiently explored.There has been an emerging trend where LLMs are evaluated in various interactive role-playing environments, including collaborative environments such as CAMEL (Li et al., 2023) , ReConcile (Chen et al., 2023) , and competition environments such as Diplomacy (Bakhtin et al., 2022) , Werewolf (Xu et al., 2023a) , Avalon (Light et al., 2023; Stepputtis et al., 2023) , multi-agent debate (Liang et al., 2023; Du et al., 2023; Chan et al., 2023; Xiong et al., 2023) , board and card games (Duan et al., 2024b). By engaging LLMs in simulated scenarios, role-playing-based environments offer useful potential for analyzing the cognitive reasoning abilities of LLMs. However, the extensive background and intricate details involved in role-play-based games dilute the pureness of logic and strategic reasoning that is typically found in game-theoretic tasks. Additionally, the evaluation is primarily verbal as it hinges on spoken or written exchanges between the LLMs. This could mask instances where LLMs might lack concrete reasoning abilities but navigate the scenario effectively through the proficient use of language.Why are game-theoretic tasks unique and necessary for LLM reasoning evaluation? Gametheoretic tasks are typically conceptualized based on prevalent trade-offs and dilemmas manifesting in real-life scenarios and are designed to be easy to understand yet require difficult skills to be mastered. In contrast to the rich narrative contexts afforded in verbal-or role-playing-based games, e.g., Werewolf (Xu et al., 2023a) and Avalon (Light et al., 2023) , the reality of game-theoretic games such as Chess and Go involve: ➊ pure logic and strategic reasoning without the added complexity of backgrounds or character roles; ➋ embracing rigorous rules with well-defined action/state space, which allow for an in-depth examination of the strategic reasoning of LLMs.Hence, in order to spur more research in the LLM Game-Theoretic evaluation domain, we propose GTBENCH, an environment consisting of 10 widely recognized game-theoretic tasks, across a comprehensive taxonomy of games, e.g., complete-(Tic-Tac-Toe, Connect-4, Breakthrough) versus incomplete-information (Kuhn Poker, Liar's Dice) gaming, deterministic (Nim) versus probabilistic (Negotiation, Pig) gaming, static versus dynamic (Iterated Prisoner's Dilemma, Blind Auction) gaming. These environments require a variety of abilities including board strategy, collaboration, auction, and bidding. There are two key issues investigated in this paper:2 Background and Problem Definition 2.1 Background and Related Work LLM-as-Agent Evaluation. Several studies have been conducted to measure the effectiveness of LLMs as agents in recent years. Hausknecht et al. (2020) carried out an extensive study to evaluate the performance of LLMs in interactive fiction games. Zhu et al. (2023) provides a valuable dataset for finetuning LLMs to improve usefulness in the strategic game Dungeons & Dragons. GRUE (Ramamurthy et al., 2023) uses reinforcement learning-based metrics to benchmark the performance of generation tasks in six different languages. Gandhi et al. (2023) test the use of LLMs as a broker with human contestants in the negotiation game "Deal or No Deal". A few studies have explored the use of text-based games as a means of facilitating learning in such environments. ALFWorld (Shridhar et al., 2020) introduced a novel virtual environment that allows agents to acquire learning in a text-based environment while executing in a visual environment. The environment was developed in conjunction with Building Understanding in Text world via Language for Embodied Reasoning (BUTLER) agent, which can acquire abstract text knowledge in the text world. Similarly, TextWorld (Côté et al., 2019) is introduced as an environment that enables RL agents to play text games. Wang et al. (2022a) proposed ScienceWorld, a benchmark used for evaluating agents' reasoning ability, and their findings showed that transformer-based models are not effective at reasoning in novel contexts. MTBench (Zheng et al., 2024) introduces LLM-as-a-Judge where GPT-4 (Achiam et al., 2023) is utilized as a judge to evaluate the quality of LLM generations. It indicates that GPT-4 shares close criteria as humans. There have been works evaluating LLMs in solving real-world tasks, such as graph reasoning (Besta et al., 2023) , WebShop (Yao et al., 2022) , AgentBench (Liu et al., 2023) for pragmatic missions, MINT (Wang et al., 2023b) for tool utilization.Multiple LLMs-as-Agents in Gaming. A key research area is the competition and collaboration between LLMs. Many studies examine LLMs' strategic reasoning and performance, using evaluation frameworks to assess multiple LLM agents in individual games, such as: Social deduction or deception games (Xu et al., 2023a,b; O'Gara, 2023; Light et al., 2023) , diplomacy games (Mukobi et al., 2023; , FAIR) , negotiation games (Abdelnabi et al., 2023; Davidson et al., 2023) , coordination and cooperation games (Akata et al., 2023) , and Minecraft (Gong et al., 2023; Wang et al., 2023a; Fan et al., 2022) . These works not only provide evaluation frameworks for games and demonstrate the flexibility of LLMs to a variety of gaming tasks but some provide meaningful datasets for fine-tuning, policies for reinforcement learning to produce better strategies, or evaluate the strategic reasoning of LLMs. However, many of these standalone works quantify either individual or a subset of desirable strategic reasoning capabilities of LLMs, such as negotiation, deception, or coordination. Further, they often evaluate these capabilities for LLMs using one or two games which may produce less robust assurances of LLM abilities. Dilemma ‡ † : Breakthrough has a slight first-player advantage which is not as significant as others. ‡ : The iterated version of Prisoner's Dilemma allows participants access to the actions made by their opponents in the past rounds, achieving implicit collaboration. † † : Inapplicable due to complex combination and dynamic environment.We make an additional crucial contribution in this line of work by measuring strategic reasoning capabilities with games that are not found in the existing unified benchmark suites (Zhang et al., 2024) , such as clembench (Chalamalasetti et al., 2023) focusing on conversational agents over non-zero-sum games and LMRL-Gym (Abdulhai et al., 2023) on verbal reinforcement learning tasks. (Chen et al., 2024) and Duan et al. (2024b) also proposes multi-agent strategic reasoning evaluation. However, they overlooked the analysis of LLM behaviors in response to different game-theoretic scenarios and their associated properties. Differently, GTBENCH seeks to provide a unified suite of games that are carefully curated to (1) evaluate a comprehensive collection of strategic reasoning abilities for a given agent and (2) enable competition-based scenarios (i.e., LLM agent-1 vs LLM agent-2) allowing for competition-based comparisons of strategic reasoning capabilities by LLM-based agents. 
problem definition: 
 Notation: Gameplay. We formulate the gameplay as a Markov Decision Process (S, A, M, O) under a given game environment, among the alternating interaction of two participants. This process composes of an infinite state space S, an infinite action space A, the participants M = {M 1 , M 2 }, and an observation space O. Considering the decision of M i (i = 1, 2) at the t-th step of the process, we denote by s t ∈ S the state that M i are placed and o t ∈ O the observation that M i are observing. We assume M i follows policy π θi (a t |s t , o t ) for state transition T : S × A → S, where a t ∈ A is the action sampled by π θi under conditions s t and o t . θ i is determined by the implementation by M i , e.g., optimization-based solver, LLM-driven agents, which will be discussed in Section 3.2 in detail. In this way, the two-participate gameplay can be represented as (s 0 , a 0 , s 1 , a 1 , s 2 , • • • , s n ), where s 0 is the initial state and s n is a terminal state, i.e., end of the game. The progress is driven by the alternating execution of actions sampled by participants. Please refer to Section 3.1 and Appendix A2 for all the supported games with the corresponding actions and observations. Evaluation Metric: Normalized Relative Advantage. We introduce Normalized Relative Advantage (NRA), denoted NRA(M i , M o , f s ), to measure to relative advantage of M i when competing against M o , under the score calculation f s :NRA(Mi, Mo, fs) = m fs(Mi, m) -m fs(Mo, m) m fs(Mi, m) + m fs(Mo, m) ,where f s (M i , m) refers to the score earned by M i at the m-th match (1 ≤ m ≤ K, K is the number of performed matches):• For zero-sum games, e.g., Tic-Tac-Toe,fs(Mi, m) =      1, if Mi wins at the m-th match 0,if Mi loses at the m-th match 0.5, if Mi and Mo achieve a draw • For non-zero-sum games, e.g., Blind Auction, fs(Mi, m) is the rewards earned by Mi at the m-th match.i , M o , f s ) is naturally normalized to [-1, 1], providing an interpretable meaning regarding the performance of M i : NRA(M i , M o , f s ) > 0 means M i is better than M o ; NRA(M i , M o , f s ) < 0 means M i is worse than M o ; NRA(M i , M o , f s ) = 0 means M i is as competitive as M o .Evaluation Metric: Elo Rating. Following the conventional rating mechanism in the real world, e.g., Chess, we employ the popular Elo Rating (Elo, 1960) for calculating the relative skill levels of players in zero-sum games. Please refer to Appendix A7 for more details of Elo rating. 
gtbench: game-theoretic evaluation of llms: 
 GTBENCH is a language-driven environment, making participating agents compete against each other in a game-theoretic manner. It is designed to be flexible and extensible, providing unified interfaces to participants and games, and supporting various multi-turn-based games which can be extended in the future. The overall framework is presented in Figure 1 . There are three main components: Environment, Prompt Adapter, and Participant. Please refer to Appendix A1 for a detailed introduction of each component.The chosen tasks and their detailed configurations are presented in Table 1 . To comply with the common taxonomy (Lanctot et al., 2019) of game-theoretic tasks and provide diverse gaming scenarios, GTBENCH supports 10 different gaming environments, including Tic-Tac-Toe, Connect-4, Kuhn Poker, Breakthrough, Liar's Dice, Blind Auction, Negotiation, Nim, Pig, Iterated Prisoner's Dilemma, covering 6 mainstream game-theoretic configurations, including completeand incomplete-information gaming, dynamic and static gaming, and probabilistic and deterministic gaming. The preferred abilities of each game could be characterized as the combination of board strategy, bids, collaboration, bluff , and math. Please refer to Appendix A2.1 for the rules of each game and Appendix A2.2 for an explanation of game-theoretic taxonomy.Conventional Agents output actions through a conventional optimization or searching process. To provide fair comparisons, we employ the powerful Monte Carol Tree Search (MCTS) (Chaslot et al., 2008) as the conventional agent for most of the games, with the number of simulations as 1000. Since Iterated Prisoner's Dilemma is dynamic gaming with very limited action space, i.e., <TESTIFY> or <SILENT>, we utilize the more popular Tit-for-Tat (Axelrod, 1981) strategy, which simply repeating the opponent's last action, as the conventional agent. We also include Random Agent that randomly selects action at each turn, serving as a baseline and sanity check. Please refer to Appendix A3.1 for more details about MCTS Agent and Tit-for-Tat Agent.LLM-Driven Reasoning Agent consists of backbone LLMs and reasoning paradigms. For reasoning schemes, we consider the following reasoning paradigms as they are widely known to be effective for general reasoning tasks: ➊ Prompt: Directly Prompt LLMs to generate responses, without additional reasoning steps; ➋ Chain-of-Thought (CoT) (Wei et al., 2022) : CoT Agent prompts LLMs by thinking step by step; ➌ Self-Consistent CoT (Wang et al., 2022b) : SC-CoT Agent prompts LLMs by generating multiple step-by-step thinking trajectories and performing majority voting to get the final response. The number of trajectories is set to 5 in this paper; ➍ Tree-of-Thought (ToT) (Yao et al., 2024a) : ToT Agent prompts LLMs to generate responses by incorporating exploration and deliberate decision-making, e.g., self-evaluation. The number of sequences for both answer generation and answer evaluations is set to 3.Prompt Templates. Prompts are designed to be modular, consisting of four individual components: System Prompt, Head Prompt, Observation Prompt, and Reasoning Prompt. Reasoning prompts, e.g., CoT/ToT, are designed to only focus on instructing LLM how to think, regardless of the game environment. Thus, they could be automatically adapted when adding a new game. Please refer to Appendix A5 for the detailed prompts and observations for each game and agent.Sanity Check. We provide the task completion rates of all the LLMs and reasoning agents in Appendix A5.6. We show that all the LLM agents achieve ≥ 90% completion rate, indicating that the prompts are properly configured and LLMs are capable of following instructions to finish the game. 4 Are LLMs Capable of Strategic Reasoning?In this section, we evaluate the strategic reasoning capabilities of LLMs by conducting experiments among conventional solvers and LLM-driven agents.Experimental Settings. We consider well-recognized LLMs such as commercial LLMs: GPT-3.5-turbo-1106 and GPT-4-0613 (Achiam et al., 2023) , and open-source LLMs: Llama-3-70b-Instruct (Meta, 2024), Deepseek-LLM-67b-chat (Bi et al., 2024) , Llama-2-70b-chat (Touvron et al., 2023 ), CodeLlama (Roziere et al., 2023) , and Mistral-7b-Orca (Jiang et al., 2023a; Mukherjee et al., 2023) . For all the LLMs, the temperature is set to 0.2 and the max number of generated tokens is 1024. For each competition, we run 50 valid matches. The final performance is measured by the averaged NRA over the 50 valid matches. To mitigate the first-player advantage, we have each participant take the first turn in 25 matches.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
complete and deterministic gaming: 
 There are four complete and deterministic tasks supported in GTBENCH: Tic-Tac-Toe, Connect-4, Breakthrough, and Nim. We compare LLM-driven agents with Random Agent and MCTS Agent. Results are summarized in Figure 2 . In general, we show that all LLMs achieve substantial relative advantages when competing against the Random Agent. Among all the agents, GPT-4 w/ CoT reasoning achieves the highest NRA. For open-source LLMs, Llama-3-70b-Instruct outperforms other open-source LLMs, achieving comparable capabilities as GPT-4. However, when competing against the MCTS Agent, all the LLM agents equipped with various reasoning methods achieve NRA as -1, meaning that LLM agents can barely win even a single match. This is because for board games with moderate action/state space such as the four involved complete and deterministic games in GTBENCH, MCTS agents with a sufficient number of simulations can achieve near-optimal strategies. Consequently, LLMs are not competitive in complete and deterministic games. 3 . When competing against the MCTS Agent, it is shown that Liar's Dice shares a similar trend as the complete and deterministic scenarios (Figure 2 ), where LLM-driven agents achieve near -1 NRA. This is because the 2-player Liar's Dice has very limited stochasticity, making the gameplay tend to be complete information. For other tasks, we found that LLMs do not always fail. We observe that the NRA of LLM agents is close to 0 over all the tasks, indicating that they are equally competitive as conventional solvers or even better (e.g., Kuhn Poker where GPT-4 outperforms MCTS Agent). We investigate whether popular LLMs remain competitive in game-theoretic scenarios. Specifically, we take GPT-3.5-turbo with Prompt Agent as the common opponent and make other LLM-driven agents compete against it. Please refer to Figure A6 for the full leaderboard evaluated by NRA. The Elo rating results are placed in Table 6 . In general, GPT-4 is the most powerful LLM in strategic reasoning among all the examined LLMs. Moreover, Llama-3-70b-Instruct achieves comparable performances as GPT-4 and outperforms GPT-3.5-turbo. Here we break the results into 3 takeaways:4.3 LLM-vs.-LLM Competition 0.2 0.4 Model GPT-3.5-turbo Llama-2-70b-chat CodeLlama-34b-Instruct Mistral-7b-Code-Pretraining Benefits Game-Theoretic Tasks. In Table 2 , we show code-pretrained LLMs, e.g., CodeLlama-34b-Instruct and Deepseek-Coder-6.7b-Instruct, significantly outperform larger chat LLMs, e.g., Llama-2-70b-chat and Deepseek-LLM-67b-chat. These codepretrained LLMs have less than half of the parameters, suggesting that code-pretraining benefits game-theoretic tasks. This verifies recent discoveries where code-pretraining benefits logical reasoning (Madaan et al., 2022; Liang et al., 2022; Ma et al., 2023) . Advanced Reasoning Methods Do Not Always Help. We observe that advanced reasoning methods may lead to worse results in game-theoretic scenarios. To make it more clear, we present the averaged NRA obtained by reasoning methods across different LLMs when against Random Agent in Figure 4 .In general, only Mistral-7b-Orca has a substantial improvement when equipped with CoT reasoning while advanced reasoning leads to worse results for other LLMs.In Table 3 , we present the results when against GPT-3.5-turbo w/ Prompt Agent. We show that advanced reasoning benefits powerful LLMs, e.g., GPT-3.5turbo, while it results in worse results for other LLMs. It suggests that advanced reasoning is a double-edged sword: ➊ powerful LLMs are capable of leveraging advanced reasoning to achieve better results; ➋ advanced reasoning may also impose reasoning errors and risks during the inference of ordinary LLMs. In Appendix A8, we further examine five different CoT strategies over the GPT-3.5-turbo model to mitigate the effect brought by prompt sensitivity, along with some failure cases presented. These CoT prompts resulting in different performances are all worse than the naive Prompt Agent. Most Open-source LLMs are Less Competitive than Commercial LLMs in Complex Games. We observe that most of open-source LLMs such as Llama-2-70b-chat and CodeLlama-34b-Instruct are not good at games with complex rules and board states. In Table 4 , we present the average NRA when including and excluding Breakthrough3 . It is shown that both Llama-2-70b-chat and CodeLlama-34b-Instruct fail in Breakthrough, resulting in worse NRA scores than GPT-4. However, we found that the recently released Llama-3-70b-Instruct (Meta, 2024) has a significant performance in Breakthrough. This indicates that open-source LLMs achieve comparable capabilities when dealing with complex tasks and environments as commercial LLMs.We introduce the most prevalent mistake patterns observed across different games, comprising Misinterpretation, Factual Inaccuracies, Overconfidence, Calculation Mistakes, and Endgame: Misinterpretation denotes the misinterpretation of the game's current state by LLMs, including errors like misattributing piece ownership and failing to recognize vacant spots on the board. Factual Errors refer to situations where the player has a reasonable plan but their actions do not align with their plan. For instance, in Breakthrough, GPT-4 w/ CoT agent plans to fend off frontal attacks by the opponent, which is reasonable. However, it takes rear pieces to achieve that, which is impossible. Over-confidence describes a scenario where a player overlooks potential risks in pursuit of greater rewards. Calculation Errors refer to errors that occur in arithmetic, such as calculating XOR in Nim. Endgame Misdetection means a failure to recognize immediate win/lose situations, e.g., a player fails to recognize a potential winning move. Demonstrations of each mistake pattern are presented in Appendix A9.In Table 5 , we present the quantitative results regarding these error patterns. It is obtained from GPT-4 w/ CoT agent when playing against conventional solvers, e.g., MCTS/TfT agent, as the opponent.We manually examined a total of 157 turns (50 matches, with 5 turns per match). We observe that LLM agents are capable of generating reasonable planning/strategies. However, they have difficulties in selecting the correct actions to align with their thoughts. Also, LLMs miss endgame situations, leading to a failure to recognize winning and losing moves. 5 The Game-Theoretic Properties of LLMs Nash Equilibrium with Regret. In game theory, being close to a Nash Equilibrium (Nash Jr, 1950) indicates that the strategies chosen by the players are near to optimal. It has been popular to approximate Nash Equilibrium with Regret4 (Johanson et al., 2012; Nisan and Noti, 2017; Zinkevich et al., 2007) . In Figure 5a , we present the regret values of LLMs on Blind Auction and Iterated Prisoner's Dilemma. Please refer to Appendix A10 for how regret values are calculated for these two tasks. For Blind Auction, GPT-4 shows lower Regret, indicating achieving closer to optimal solutions than other LLMs. However, in Iterated Prisoner's Dilemma, CodeLlama-34b-Instruct exhibits lower regret compared to GPT-4. Through human examination, we found that this is because GPT-4 tends to <Silent> more frequently, whereas Codellama has a significantly higher probability of <Testify>. This discrepancy may be due to the human preference alignment in GPT-4, such as a higher emphasis on morality (Pan et al., 2023) or maximizing system reward5 , which makes GPT-4 less likely to <Testify>.Pareto Efficiency. We study Pareto Efficiency in two games: Negotiation and (Iterated) Prisoner's Dilemma. In Figure 5b , we count all agreements reached by participants and record the values attributed to each based on the agreed division. Most agreements result in substantial values for both participants, though some LLMs, like Llama-2-70b-chat and CodeLlama-34b-Instruct, may accept unfair resource divisions. In contrast, GPT-4 and Mistral struggle to reach agreements and tend to negotiate for Pareto improvements. A repeated game is a standard game that is played multiple times by the same players, with each player is able to observe the history of past plays (Aumann et al., 1995; Akata et al., 2023) . In Figure 5c , we investigate the Pareto Improvement in Iterated Prisoner's Dilemma and ordinary Prisoner's Dilemma, i.e., each round is played individually. The Pareto Improvement is observed in the repeated-game scenario during the rounds, indicating that LLMs are capable of leveraging history to adjust their strategies. 
conclusion: 
 This work investigated LLMs' strategic and logical reasoning abilities under competitive scenarios. To achieve this, we created a broad evaluation scope by considering various classic and LLMbased gaming agents and 10 representative games. We conducted the benchmark study of gametheoretic evaluations for LLMs, shedding light on their reasoning performance. Our extensive evaluations revealed insightful LLMs' gaming behavior, such as their intrinsic failure in complete and deterministic games, impressive reasoning in incomplete and probabilistic games, and benefiting from code-generation pertaining and appropriate prompt designs.Limitations This research prompts LLMs to generate actions regarding various game scenarios, relying on pre-defined prompt templates. Thus, the results may suffer from certain variances introduced by prompt sensitivities. Although the introduced games are popular, their actions/state space is limited, which may not be well-distinguished for LLMs in the same skill levels. The generated actions may be illegal due to the incapabilities of the following instructions.Impact Statements This paper examines the game-theoretic task proficiency of AI models. We acknowledge concerns about models becoming autonomous entities with their own objectives, especially in deception or negotiation scenarios. It's important to note that our research measures the current capabilities of models, rather than enhancing their abilities. We do not train AI models to be competent in game theory tasks or to bluff or defect. Instead, we assess existing competencies, contributing to a deeper understanding that can inform innovative measures against potential risks. We believe our work paves the way for responsible and effective AI safety. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: The results are obtained from endpoint API providers, e.g., OpenAI (Section 4).Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Section Impact Statements Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: Section Impact Statements Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Section 4 Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
 
 Appendix: There are three main components in GTBENCH:• Environment. The environment ( Figure 1 (c) ) is responsible for overseeing the crucial processes related to gameplay. Specifically, it is tasked with building up observations, managing gameplay, and applying the actions obtained from participants. In this paper, all of the gaming environments are built on top of OpenSpiel Lanctot et al. (2019) . • Prompt Adapter. The prompt adapter ( Figure 1 (b) ) plays a vital role in facilitating effective communication between the environment and the virtual participants. It serves as an intermediary between the two entities by receiving observations from the environment, which it then translates into unified observation prompts. The prompts are then parsed and sent to the participating agents to formulate their responses. The adapter is also responsible for obtaining actions from the participants, which it transforms into legal actions before parsing them to the environment for game execution. • Participant. The participants ( Figure 1 (a) ) involved in the gaming process generate responses according to the observation prompts received from the Prompt Adapter. These responses consist of actions that participants intend to take in this turn.A2.1 Games Introduction Tic-Tac-Toe6 is a paper-and-pencil game for two players who take turns marking the spaces in a three-by-three grid with X or O. The player who succeeds in placing three of their marks in a horizontal, vertical, or diagonal row is the winner. It is a solved game, with a forced draw assuming optimal play from both players.• Observation (input): Our observation contains "opponent moves" and "self moves". "Opponent moves" contains all the current opponent agent's historical actions. "Self moves" contains all the current agent's history actions. • Actions: We define our action in the following format: <CxRy>, in which C and R mean columns and rows respectively, while x and y mean the index of column and row. Each player may make their own action in turn.Prisoner's Dilemma7 is a game theory thought experiment that involves two rational agents, each of whom can cooperate for mutual benefit or betray their partner ("defect") for individual reward.• Observation (input): Our observation contains "opponent moves" and "self moves". "Opponent moves" contains all the current opponent agent's historical actions. "Self moves" contains all the current agent's history actions. • Actions: We define our action in the following format: <Silent> or <Testify>. All players must take their action simultaneously.Breakthrough8 Breakthrough is an abstract strategy board game invented by Dan Troyka in 2000 and made available as a Zillions of Games file (ZRF). It won the 2001 8x8 Game Design Competition.The first player to reach the opponent's home row -the one farthest from the player -is the winner.In our work, we scale the size of the board to 3*8 while maintaining its competitiveness.• Observation (input): Our observation contains "opponent moves", "self moves", and "board preview". "Opponent moves" contains all the current opponent agent's historical actions. "Self moves" contains all the current agent's history actions. The "board preview" feature maintains the status of each grid on the board through a list of strings, denoting whether it contains a black piece, a white piece, or is empty.• Actions: We define our action in the following format: Ax->By, in which A and B mean the current column index and destination column index respectively, while x and y mean the index of current row and destination row. Each player may make their own action in turn.Connect Four9 is a game in which the players choose a color and then take turns dropping colored tokens into a six-row, seven-column vertically suspended grid. The pieces fall straight down, occupying the lowest available space within the column. The objective of the game is to be the first to form a horizontal, vertical, or diagonal line of four of one's own tokens.• Observation (input): Our observation contains "opponent moves" and "self moves". "Opponent moves" contains all the current opponent agent's historical actions. "Self moves" contains all the current agent's history actions. • Actions: We define our action in the following format: <Cx> in which C means column, while x means the index of column. Each player may make their action in turn.Blind Auction10 is a common type of auction. In this type of auction, all bidders simultaneously submit sealed bids so that no bidder knows the bid of any other participant. The highest bidder pays the price that was submitted. All players must take their action simultaneously.• Observation (input): Our observation contains "valuation"."Valuation" contains each of the values of all the items for the current player. • Actions: We define our action in the following format: <x>, in which x represents the amount that a certain player would like to bid for.Kuhn Poker11 is a simplified form of poker. Kuhn is a simple model zero-sum two-player imperfectinformation game, amenable to a complete game-theoretic analysis. In Kuhn poker, the deck includes only three playing cards, for example, a King, Queen, and Jack. One card is dealt to each player, which may place bets similarly to a standard poker. If both players bet or both players pass, the player with the higher card wins, otherwise, the betting player wins.• Observation (input): Our observation contains "card" , and "moves". Among these, "card" denotes the current player's hand card in this match, while "moves" represents the history of all characters' moves together with the index of the rounds. • Actions: We define our action in the following format: <Pass> or <Bet>. Each player may make their own action in turn.Liar's Dice12 is a class of dice games for two or more players requiring the ability to deceive and detect an opponent's deception.• Observation (input): Our observation contains: "Self dice face value" and "last move". "Self dice face value" describes all the face values of dices the current player has, while "last move" represents the previous player's action. • Actions: We define our action in the following format: < x dices, y value> or <Liar>. Among these, x means the quantity of dice, and y means the face values of the dice. The option "Liar" denotes the current player wants to stop and challenge the previous players. Each player may make their own action in turn.Pig13 is a simple dice game. Players take turns to roll a single dice as many times as they wish, adding all roll results to a running total, but losing their gained score for the turn if they roll a 1.• Observation (input): Our observation contains: "self current score", "opponent current score", and "turn total score". "Self current score" and "opponent current score" represent the game culminated score of the current player and opponent player respectively. While "turn total score" denotes the sum of the score of the current turn.• Actions: We define our action in the following format: <stop> or <roll>. Each player may make their own action in turn.Nim14 is a mathematical game of strategy in which two players take turns removing objects from distinct heaps or piles. On each turn, a player must remove at least one object and may remove any number of objects provided they all come from the same heap or pile.• Observation (input): Our observation contains: "piles". "Piles" denotes the number of matches different piles have.• Actions: We define our action in the following format: <pile:x, take:y>. Among these, x represents the index of the pile that the current player takes, and y represents the number of matches the current player takes. Each player may make their own action in turn.Negotiation15 • Observation (input): Our observation contains: "turn type", "item pool", "most recent proposal", "most recent utterance", and "self value vector". "turn type" is an enum variable, it has two options: proposal and utterance. The "Proposal" is the turn that the current player could think about the desired quantities of the items, and the "Utterance" is the turn that the current player states the values to its opponent. "item pool" represents the quantities of all the items."most recent proposal" and "most recent utterance" represent the opponent's latest proposal and utterance. "self value vector" represents how much the value of the items to the current player.• Actions: We define our action in the following format: <Agree> or < x, y, z >. Among these, <Agree> represents the current player agreeing on the opponent's utterance. x, y, and z represent the quantities of different items that the current player wants to get.Complete and Incomplete Information One fundamental dimension along which games are classified is the level of information available to players. In complete information games, players possess perfect knowledge regarding the game's structure, including the available strategies, payoffs, and the actions taken by other players. Examples of complete information games include canonical examples like chess and Tic-Tac-Toe, where all relevant information is transparent to all players throughout the game. Conversely, incomplete information games involve situations where players must make decisions without having full knowledge of the game's parameters or the actions of other players. Classic examples of incomplete information games include strategic interactions in economics, such as auctions or negotiations, where players have limited knowledge about the valuations or preferences of other participants.Another crucial dimension for classifying games is the timing of players' decisions. In static games, players make decisions simultaneously, without the opportunity to observe or react to other players' moves. Examples of static games include simultaneous-move games like the Iterated Prisoner's Dilemma. In contrast, dynamic games involve sequential decision-making, where players observe previous moves before choosing their actions. Dynamic games encompass a wide range of strategic environments, from turn-based board games like chess to dynamic settings like Kuhn Poker, where players strategically make their actions based on the unfolding dynamics of the game.Probabilistic and Deterministic Games can also be differentiated based on the role of uncertainty in decision-making. In deterministic games, the outcomes of players' actions are fully determined by the game's rules and the strategies chosen by players. Deterministic games include classic examples like chess or Tic-Tac-Toe, where each move leads to a predictable outcome based on the game's rules and the players' strategies. Conversely, probabilistic games involve randomness or uncertainty in determining outcomes. This uncertainty can stem from elements such as dice rolls, card draws.Examples of probabilistic games include games of chance like Kuhn Poker, Liar's Dice, or Pig, where players must contend with the inherent uncertainty of probabilistic outcomes.A3.1 Conventional Agent MCTS Chaslot et al. (2008) is a heuristic search algorithm that has gained prominence in recent years, particularly in the domain of board games and decision-making under uncertainty. It is characterized by its ability to efficiently explore large search spaces by sampling potential future outcomes through Monte Carlo simulations. The algorithm iteratively builds a search tree by simulating random sequences of moves from the current game state and evaluating their outcomes through repeated simulations. By focusing computational resources on promising branches of the search tree, MCTS aims to guide the search towards regions of the game space that are more likely to lead to favorable outcomes. MCTS has demonstrated remarkable success in various domains, including games like Go, where traditional search algorithms struggle due to the game's immense complexity and branching factor.Tit-for-Tat Axelrod ( 1981) is a simple but powerful strategy in the realm of repeated games and social dilemmas. The strategy is based on the principle of reciprocity, where an agent initially cooperates and then mimics the opponent's previous action in subsequent rounds. Specifically, Tit-for-Tat starts by cooperating in the first round and then replicates the opponent's last move in each subsequent round. Despite its simplicity, Tit-for-Tat has been shown to be remarkably effective in promoting cooperation and achieving favorable outcomes in various scenarios, including Iterated Prisoner's Dilemma and evolutionary simulations. Its success stems from its ability to balance cooperation and retaliation, fostering reciprocal behavior and encouraging cooperation among interacting agents.In Figure A6 , we present the confusion matrix of NRA when various LLM agents are against GPT-3.5-turbo and GPT-4.Figure A6 : NRA confusion matrix of LLM vs. LLM across ten games ranked by average NRA. GPT-3.5-turbo with Prompt Agent serve as the common opponent against multiple combinations of LLMs with agents.serve 2 years. If both you and your partner remain <Silent>, you and your partner will each serve 1 year.Head Prompt: Breakthrough is a two-player game played on a rectangular board. Players take turns moving their pieces, which can move one space straight or diagonally forward if the target square is empty. A piece can also move diagonally forward to capture an opponent's piece. Capturing is optional, and a player can only capture one piece per turn. The goal is to be the first to reach the opponent's home row, the farthest row from the player. If all of a player's pieces are captured, they lose. The game does not allow draws, as pieces can only move forward or be captured. The Breakthrough board is identified by columns labeled starting from A (from left to right) and rows numbered 1 to 8 (from bottom to top). The intersection of a column and a row specifies a unique square on the board.• Connect Four Head Prompt: Connect 4 is a two-player connection board game, where the players choose a color and then take turns dropping colored discs into a vertically suspended grid. The pieces fall straight down, occupying the next available space within the column. The objective of the game is to be the first to form a horizontal, vertical, or diagonal line of four of one's own discs. You are a gaming agent who aims to beat me in Connect 4 games. Each move is represented by a string consisting of two parts: the column (C) and the row (R), in that order. For instance, C1 means the first column.• First-price sealed-bid auction Head Prompt: A first-price sealed-bid auction (FPSBA) is a common type of auction. It is also known as the blind auction. In this type of auction, all bidders simultaneously submit sealed bids so that no bidder knows the bid of any other participant. The highest bidder pays the price that was submitted. Each action is represented by <x> where x refers to the bid.Head Prompt: Kuhn poker is a simple model zero-sum two-player imperfect-information game, amenable to a complete game-theoretic analysis. In Kuhn poker, the deck includes only three playing cards: a King (K), a Queen (Q), and a Jack (J). One card is dealt to each player, and the third is put aside unseen. The players take turns either <Bet> to match the bet raised by the opponent or <Pass> to concede the game. If a player bets, the other player must either call the bet by matching it or fold by conceding the game. If both players pass, the game is over, and the player with the higher-ranking card wins. The card rankings are as follows: King (K) > Queen (Q) > Jack (J). You are playing Kuhn poker with the opponent. The actions are denoted by <Bet> and <Pass>.Head Prompt: Liar's Dice is a game of bluffing and probability, played with two players and each player has 1 dice. During each turn, a player can either bid a higher quantity of any particular face value or the same quantity of a higher face value than the previous bid. Each player tries to outbid their opponent without being caught in a lie. The move in this game is denoted in <x dices, y value>, meaning there are at least x dices with face values as y.Head Prompt: Pig is a fast-paced dice game where players risk accumulating points with each roll but risk losing them all if they roll a 1. Each player must decide when to stop rolling and bank their points, aiming to be the first to reach 100 points. You are playing Pig with the other.• Nim Head Prompt: In Nim, a strategic game with a set of four piles containing 1, 3, 5, and 7 matches respectively, players aim to avoid taking the last match. During each turn, a player may take any number of matches from a single pile, but must take at least one and cannot exceed the number remaining in that pile. The objective is to force the opponent to pick up the final match, thereby winning the game. The action is presented in <pile:x, take:y>, which means take y match(es) from the x-th pile.• Negotiation Head Prompt: You are negotiating the division of Peppers, Strawberries, and Cherries with the opponent. Different values these items hold for both you and your opponent. The process is structured into two stages per round: the proposal stage and the utterance stage.Our research team has developed a range of observation prompts tailored to different types of games. The list of these prompts is presented below.• Tic-Tac-ToeObservation Prompt: Your opponent has finished actions: <OPPONENT_MOVES>. You have finished actions: <SELF_MOVES>.• Iterated Prisoner's Dilemma Observation Prompt: You have been through this situation in the past and here are the decisions you and your partner made: (In the idx + 1 th round, you decided to <MOVE> and your opponent decided to <OPPONENT_MOVE>) * n round• BreakthroughObservation Prompt: The board now looks like : <BOARD_PREVIEW>. Among which, the letter 'b' represents a black piece, while the letter 'w' represents a white piece. And the character "." represents vacant space. The numbers in the board are the indexes of the rows. Your opponent has finished actions: <OPPONENT_MOVES>.You have finished actions: <SELF_MOVES>.Observation Prompt: Your opponent has finished actions: <OPPONENT_MOVES>. You have finished actions: <SELF_MOVES>.• First-price sealed-bid auction Observation Prompt: Now, you are in an auction with an opponent. You want to win the object and at the same time, your budget is <VALUATION>. Your bid must be strictly lower than or equal to <VALUATION>. You shall bid wisely against your opponent. Your opponent also has an expected valuation and you do not know it.Observation Prompt: In this match, your card is <CARD>. Here are the past moves in this match: <SELF_MOVES>, <OPPONENT_MOVES>.Observation Prompt: Currently, the face value of your dice is <FACE_VALUE>. Last time, the opponent called <OPPONENT_LAST_ACTION>. You are playing the Liar's Dice with another opponent. Therefore, there are only two dice in total.• Pig Observation Prompt: Right now, your current score is <AGENT_CURRENT_SCORE> and your opponent's current score is <OPPONENT_CURRENT_SCORE>. In this turn, you have earned <TURN_TOTAL_SCORE> score.• Nim Observation Prompt: Currently, the 1st pile has <PILES[0]> match(es), the 2nd pile has <PILES[1]> match(es), the 3rd pile has <PILES[2]> match(es), 4th pile has <PILES[3]> match(es).• Negotiation We proposed two different prompts for the "proposal" turn and "utterance" turn respectively.For the "proposal" turn, we have:Observation Prompt: Now, the opponent propose to take <OPPONENT_PROPOSAL_TAKE[0]> peppers, <OPPONENT_PROPOSAL_TAKE[1]> strawberries,and<OPPONENT_PROPOSAL_TAKE[2]> cherries from the item pool.Last time, the utterance of the opponent was to take <OPPONENT_UTTERANCE_TAKE[0]> peppers, <OPPONENT_UTTERANCE_TAKE[1]> strawberries, and <OPPONENT_UTTERANCE_TAKE[2]> cherries from the item pool. Now, it is your decision. If you find the proposal raised by the opponent is acceptable, you should output Agree. Otherwise, you should output your proposal in the format <Proposal: [a, b, c]>.For the "utterance" turn, we have: Observation Prompt: Last time, you propose to take <AGENT_PROPOSAL_TAKE[0]> pep- pers, <AGENT_PROPOSAL_TAKE[1]> strawberries,• Prompt agent: Prompt agent does not necessitate the use of LLMs to apply any predetermined strategy prior to decision-making. Rather, it simply requests LLMs for inference and subsequently provides the outcome.You must choose a legal action to set up advantages. Your output must be in the following format: Action: Your action wrapped with <>, i.e., <format>Please return your answer without explanation!• CoT agent: CoT agent makes LLMs consider the given observation first, then give out the action according to its thinking.First think about your current situation, then you must choose one action from legal actions to set up advantages. Your output must be in the following format strictly: Thought: Your thought. Action: Your action wrapped by <>, i.e., <format> Remember, you can only choose one move from the legal actions.• SC-CoT agent: SC-CoT agent is an advanced version of the CoT agent. It obtains actions from multiple CoT trajectories. It employs the same prompt templates as in the CoT agent.First think about your current situation, then you must choose one action from legal actions to set up advantages. Your output must be in the following format strictly: Thought: Your thought. Action: Your action wrapped by <>, i.e., <format> Remember, you can only choose one move from the legal actions.• ToT agent: we follow the text generation task implementation in the official codebase of ToT16 . Specifically, the ToT is factorized into 1). candidate thought generation, 2). thought voting, 3). candidate action generation, 4). action voting:Here we provide the basic prompt template used in ToT.Step Prompt: First think about your current situation, then choose one move from legal positions to set up advantages. Your output should be of the following format: Thought: Your thought. Move: Your action wrapped with <>, e.g., <format>After executing step prompts in a breath-first search manner, we utilize the original ToT vote prompt:Vote Prompt: Given an instruction and several choices, decide which choice is most promising. Analyze each choice in detail, then conclude in the last line "The best choice is s", where s the integer id of the choice.To evaluate the effectiveness of our framework, we perform a sanity check by calculating the completion rates of each game. The completion rates are calculated as 50 N where N is the number of matches that will take to achieve 50 valid matches. Here, a valid match means all the participants will always generate legal moves at each turn of the match. Results are summarized in Table A7 . We show that all the LLM agents achieve ≥ 90% completion rate, showing that the prompts are properly configured and LLMs are capable of following instructions to finish the game. To study how the temperature used in generating LLMs' responses affects performances, we conduct experiments by making LLMs with 0.2 temperature (the default setting as in our paper) play against LLMs with 0.4/0.6/0.8 temperature, over CodeLlama-34b-Instruct and GPT-3.5-turbo-1106. For each experiment, we run 20 matches. The reasoning method is the PromptAgent. The results are summarized as in Table A8 . We show that a larger temperature will result in worse performance for deterministic games, while it has a model-specific effect for probabilistic games. The Elo rating system Elo (1960) is a popular method for calculating the relative skill levels of players in two-player games such as Chess. It was used by various organizations to rank players. Assume there are two players: A and B, and each player has a rating, R A , R B , which is a numerical value representing their skill level. The expected score for a player is the probability that the player will win against another player: E A = 1 1 + 10 (R B -R A )/400 E B = 1 1 + 10 (R A -R B )/400 .A = R A + K * (S A -E A )) R ′ B = R B + K * (S B -E B )),where K is a constant that determines how much the rating changes after a game. A higher K results in a larger change. In our paper, the initial rating is set to 1500, i.e., R A = R B = 1500, and K = 20. for Template 0 to Template 4.Table A11 : We conclude 5 common error patterns in GTBENCH, including misinterpretation, factual error, math calculation, and over-confidence. For each demonstration, the explanation explains why LLM generation is incorrect.Breakthrough Factual Error LLM Generation: "Considering the current situation, moving the piece at b6 to either a5, b5, or c5 would allow me to advance further into the opponent's territory. However, this could potentially expose my piece to being captured. On the other hand, moving a piece from the back row (a8, b8, or c8) to b7 could potentially block the opponent's advance and provide a defensive line. Action:<b8->b7>".Explanation: In the third round of this match, moving the back row renders it impossible to hinder the opponent's forward movement.State: 3 matches in Pile 2, 5 matches in Pile 3, and 7 matches in Pile 4. LLM Generation: "The current state of the game is such that the 2nd pile has 3 matches, the 3rd pile has 5 matches, and the 4th pile has 7 matches. The goal is to avoid being the one to pick up the last match. To achieve this, I need to consider the binary representations of the pile sizes and aim to make the XOR of all pile sizes equal to zero. This will put me in a winning position. Currently, the XOR of all pile sizes is 4 (011 in binary). To make this zero, I need to remove 4 matches from the 3rd or 4th pile. Action:<pile:3, take:4>". Explanation: The XOR of the binary representations of 3, 5, 7 is 1, not 4.Over-confidence LLM Generation: "Given the history of the previous round, where both my partner and I chose to remain silent, it seems that we are both inclined towards cooperation. This is a positive sign, as it indicates a level of trust and mutual understanding. Considering the potential outcomes, the safest option is to remain silent. If my partner also remains silent, we both serve the least possible time. Action: <Silent>" Explanation: The player is overconfident that the partner will cooperate and keep silent.Misinterpretation LLM Generation: "I have analyzed the current situation and identified that my opponent has taken the center column, which gives them a significant advantage. To counter this, I will take the corner column to create a potential winning line. Action: <C1>." Explanation: Currently, the center column is unoccupied.State: The opponent finished actions <C2R1> and <C2R2>, the agent finished moves <C1R1> and <C3R1>.LLM Generation: I have one mark in the first column, and one in the third. I could potentially win if I get the chance to mark the third row of the first column in the next turn. Action: <C1R3> Explanation: This is an immediate loss situation because the opponent has two pieces connected in a row and <C2R3> is unoccupied. The agent missed that they should take action <C2R3> to avoid lose • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Breakthrough has larger action/state space than other complete-information games.Regret(Zinkevich et al., 2007) measures how much a player would have improved their outcome by choosing a different strategy, given what they know now after the game has played out.<Silent> maximizes the system reward in Iterated Prisoner's Dilemmahttps://en.wikipedia.org/wiki/Tic-tac-toehttps://en.wikipedia.org/wiki/Prisoner%27s_dilemmahttps://en.wikipedia.org/wiki/Breakthrough_(board_game)https://en.wikipedia.org/wiki/Connect_Fourhttps://en.wikipedia.org/wiki/First-price_sealed-bid_auctionhttps://en.wikipedia.org/wiki/Kuhn_pokerhttps://en.wikipedia.org/wiki/Liar%27s_dicehttps://en.wikipedia.org/wiki/Pig_(dice_game)https://en.wikipedia.org/wiki/Nimhttps://arxiv.org/pdf/1706.05125.pdfhttps://github.com/princeton-nlp/tree-of-thought-llm/blob/master/src/tot/ prompts/text.py 


Decision: The paper introduces a benchmark designed to evaluate the strategic reasoning capabilities of LLMs using a suite of ten diverse games. These games range from deterministic ones like Tic-Tac-Toe and Connect-4 to uncertain, incomplete information games like poker and negotiation. The study finds that LLMs generally struggle with deterministic games but perform better in uncertain scenarios. Code-pretraining enhances their strategic thinking, though advanced methods like Chain-of-Thought and Tree-of-Thought can be counterproductive in some cases. The paper highlights that while open-source models like Llama-3 are closing the gap with commercial models like GPT-4, common errors persist, such as misunderstanding game rules, overconfidence, and calculation mistakes. The benchmark also facilitates competitive experiments between LLMs and traditional methods, as well as LLM-vs.-LLM, providing a comprehensive analysis of the game-theoretic properties of these models.

In general, reviewers were relatively positive about the benchmark, although it would have been nice (as always) to see a larger number of games and solvers. Nonetheless, the games selected provide a good range over categories (such as collaborative vs competitive, perfect vs hidden information, etc). The main point of concern is around the central claim as to whether the benchmark does in fact probe the ability of LLMs to perform strategic reasoning or whether this is simply a case of decision making.  Certainly the title is slightly misleading, in that the benchmark merely serves to evaluate, as opposed to "unlock" strategic reasoning abilities (which hopefully work down the line will do). Nonetheless, I think that the current benchmark is useful and intersting enough to be of utility to the field.

Reviews: review: paper_summary: paper_summary: This paper introduces GTBench, a set of 10 different games to test how well large language models can think strategically. The author found that while LLMs struggle with complete and deterministic games like Tic-Tac-Toe and Connect-4, they perform better in incomplete uncertain games like poker and negotiation. Code-pretraining improves their strategic thinking abilities. However, advanced thinking methods like Chain-of-Thought and Tree-of-Thought don’t always help and can sometimes make things worse. The latest open-source models, like Llama-3, are getting closer in performance to commercial models like GPT-4. Common mistakes LLMs make include misunderstanding game rules, being over-confident, and making calculation errors.
strength_weakness: strength_weakness: 1. The paper claims that measuring strategic reasoning capabilities with games is missing in existing benchmarks. However, there are other benchmarks, such as MAgIC released last year, that consider benchmarking LLMs' strategic behavior using games. While there are differences, this weakens the claim of novelty.
2. Some of the selected games, like Tic-Tac-Toe, have known optimal strategies and are not complex enough. These games might not fully challenge the advanced strategic reasoning capabilities of LLMs. Even though the current evaluation is useful, as a benchmark intended for future use, it should be capable of evaluating more advanced or adapted LLM agents.
3. The benchmark focuses on a set of 10 games. It’s unclear how well the findings generalize to other strategic scenarios, even similar types of tasks. The results appear to be quite case-by-case. A broader range of tasks and scalable evaluation frameworks would make the benchmark more comprehensive.
4. The experiments primarily involve LLMs and traditional solvers. There is a lack of evaluation against human opponents, which could provide more insights into the models' performance in real-world strategic interactions. As a benchmark, I also expect to have other opponents (for example, the optimal algorithm, the RL based agent).
questions: questions: Could you address the weakness 1, and try to discuss weakness 2-4?
limitations: limitations: If LLMs are trained on biased data, they might reinforce existing biases in strategic decision-making. Testing the LLM with different personas could show if this changes the game results and reveal potential biases.
paper_summary: paper_summary: The paper proposes a benchmark to understand the strategic reasoning capabilities of llms. The authors present a suite of game theoretic tasks with different structures to do this. They use different evaluation metrics like ELOs and Relative advantage to compare different llms and prompting methods.
strength_weakness: strength_weakness: - Characterizing human performance would strengthen the paper
- Including some qualitative reasoning traces of successes and failures might be insightful.
- Minor: This paper would be an ideal fit for the datasets and benchmarks track, instead of the main track. I dont think it should be penalized for this though!

Typos

Line 79: Characterize

Line 171: dynamic gaming → dynamic game
questions: questions: See weaknesses.
limitations: limitations: The authors do a good job of addressing limitations.
paper_summary: paper_summary: This paper proposes a benchmark for evaluating the strategic reasoning of LLMs. The benchmark includes ten games of various types. The authors use these games to conduct competitive experiments between LLMs and traditional methods, as well as LLM-vs.-LLM. The paper then analyzes the experimental results and model behavior, and examines the game-theoretic properties of LLMs.
strength_weakness: strength_weakness: I didn't find any significant weaknesses, only a few questions.
questions: questions: 1. In Section 4.1, why does the tree-like prompting strategy ToT still lag significantly behind MCTS?
2. Is there any reference to classifying games in the benchmark? Why is it classified this way?
3. Why does the model perform better in probabilistic and dynamic games than in completely deterministic games? Is it that LLM performs better or that MCTS performs worse, making LLM appear better?
limitations: limitations: The authors have fully addressed the limitations in the paper.
paper_summary: paper_summary: This paper tries to evaluate the strategic reasoning abilities of LLM. Therefore, 10 games are chosen where LLMs is trying to solve the game. This paper includes various open- and closed-source LLMs into consideration and build a benchmark for easy evaluation.
strength_weakness: strength_weakness: The evaluation protocol is questionable. More comments and questions are in the following section.
questions: questions: 1. Does the evaluation really evaluate the strategic reasoning? Basically the evaluation is letting the LLM to play as one of the player in the game. However, this is much like a decision making problem, especially when the opponent is also a LLM-agent, where the LLM agent is largely stationary. Therefore, I would like to ask the authors provide the justification about why the evaluation is about strategic reasoning, not the decision making? 

2. Also about the strategic reasoning. The selected games only focus on competitive zero-sum games. what about general-sum and multi-player games? Are cooperative games, e.g., hanabi, also requiring strategic reasoning? Even further, mixed cooperative and competitive, e.g., soccer, need the strategic reasoning? I think the strategic reasoning is not well-defined and fully discussed. 

3. Does the evaluation really unlock the abilities of LLMs? The evaluation is focusing the prompting. However, for games, especially unfamiliar games for LLMs, exploration is important. Therefore, a memory or long in-context learning of the exploration experience should be included for the evaluation of strategic reasoning in games.
limitations: limitations: More limitations should be discussed.

Comments: title: We thank you for your valuable comments!
Reply to the Rebuttal by Authors
Thanks for the response
comment: Thank you for your valuable feedback and support. We will incorporate additional results and details in our next revision.
I appreciate the author's effort and the response to my comments. I was hoping for more solid work to give a higher score (like 7 or above). For example, adding more games, comparing with more solvers, or including some additional insights (perhaps thinking out-of-the-box). I understand that it’s hard to make big changes in a short time. I have increased my score and lowered my confidence.
Dear reviewer VyBq,

We would greatly appreciate your esteemed feedback, as this could be our last chance to resolve any outstanding issues or inquiries you might have. Should any elements of our work need additional explanation, we kindly ask you to inform us. We look forward to your valuable input and the chance to have a productive discussion to enhance our submission. Thank you!
Thanks for your response. I think it's an appropriate score.
Thank you for your response! I'll keep my current score!
We thank you for your prompt response.

1. In our last response, we explicitly mention that the major difference between decision making and strategic reasoning is the consideration of other players, i.e., “**(2) anticipating others’ actions**” (also described in Section 2.2). We would appreciate further clarification on how the proposed definition ("**simulate the play process of the two or more players**") differs from ours.


2. Cooperative games do require strategic reasoning. We already highlighted that collaboration is one of the major categories in the taxonomy of GTBench in Table 1. Games such as the Iterated Prisoner’s Dilemma require cooperative strategies to maximize system rewards (please refer to the definition of Prisoner's Dilemma **L575-576**, and the definition of Tit-for-Tat **L695-702** which both clearly mention cooperation). Thus, **we would like to politely disagree with the claim that we ignore cooperative games**.

3. In Lines 41-48, we mention that games provide rigorous rules and well-defined action/state spaces, involving multi-player interaction (section 2.2, the gameplay is denoted as Markov decision process), which is well-aligned to the goals of strategic reasoning. We appreciate further clarification on the concerns of evaluating strategic reasoning through games.  **Two-player games, i.e., considering one player’s action, are well-recognized as strategic reasoning in game theory literature [1-3]**. Besides, the reviewer has mentioned the definition of strategic reasoning as “simulate the play process of the two or more players”, indicating that two-player games are strategic reasoning. Also, we have provided the 3-player Pig experiments in our previous response to support that GTBench supports multi-player environments. 


Reference:

[1] Hedden, Trey, and Jun Zhang. "What do you think I think you think?: Strategic reasoning in matrix games." Cognition 85.1 (2002): 1-36.

[2] Abramson, Bruce. "Control strategies for two-player games." ACM Computing Surveys (CSUR) 21.2 (1989): 137-161.

[3]Gutierrez, Julian, Paul Harrenstein, and Michael Wooldridge. "Expressiveness and complexity results for strategic reasoning." (2015).
1. About the definition of the decision making and strategic reasoning. I do not agree about the strategic reasoning. Generally, the strategic behavior is mostly investigated in game theory. What game theory differs from RL is that it need to simulate the play process of the two or more players, thus the fictitious play is proposed. In this paper, it is still decision making, not strategic reasoning. Surely this would be a definition problem, but I think this is important. 

2. Sorry for the zero-sum competitive claim. but your response do not address my concerns. does the cooperative, mixed cooperative & competitive games also need strategic reasoning? ——based on your definition, cooperative game is definitely need to infer other's action. but is ignored in your paper. 

3. I agree that benchmarking LLM in games still in its early stage. but I am just skeptical about the claim that benchmarking LLM in games = evaluating the strategic reasoning, also that considering one player's action is strategic reasoning. 

Given that my score is the border reject, which is not that negative, and the author's rebuttal do not convince me about the suitability of the methods, I would keep my score.