LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models
Authors: Yadong Zhang
Shaoguang Mao
Tao Ge
Xun Wang
Yan Xia
Wenshan Wu
Ting Song
Man Lan
Furu Wei
Venue: COLM (2024)
Keywords: Large Language Model,Strategic Reasoning,Multi-Agent System,Decision-Making,Advanced Reasoning
TL;DR: A comprehensive overview of the state of LLMs in strategic reasoning, shedding light on their capabilities, applications, and the road ahead for harnessing their potential more effectively.

Abstract: This paper presents a comprehensive survey of the current status and opportunities for Large Language Models (LLMs) in strategic reasoning, a sophisticated form of reasoning that necessitates understanding and predicting adversary actions in multi-agent settings while adjusting strategies accordingly. Strategic reasoning is distinguished by its focus on the dynamic and uncertain nature of interactions among multi-agents, where comprehending the environment and anticipating the behavior of others is crucial. We explore the scopes, applications, methodologies, and evaluation metrics related to strategic reasoning with LLMs, highlighting the burgeoning development in this area and the interdisciplinary approaches enhancing their decision-making performance. It aims to systematize and clarify the scattered literature on this subject, providing a systematic review that underscores the importance of strategic reasoning as a critical cognitive capability and offers insights into future research directions and potential improvements.

Published as a conference paper at COLM 2024
LLM as Mastermind: A Survey of Strategic Reasoning with Large Language Models
Yadong Shaoguang Mao² Ge², Xun Wang", Yan Xia?, Wenshan Wu?, Ting Song', Man Lan', Furu Wei?
'Department of Computer Science and Technology, East China Normal Uiversity Microsoft Research Asia
(yadongzhangostu, mlan@cs)... ceancss..ccmm ecnu. edu. en, (shaoguang mao, tage, xunwang, yanxia, wenshan wu, tsong, fuwei [iii com
Abstract
This paper presents a comprehensive survey of the current status and op- portunities for Large Language Models (LLMs) in strategic reasoning, a sophisticated form of reasoning that necessitates understanding and pre- dicting adversary actions in multi-agent settings while adjusting strategies accordingly. Strategic reasoning is distinguished by its focus on the dy- namic and uncertain nature of interactions among multi-agents, where comprehending the environment and anticipating the behavior of others is crucial. We explore the scopes, applications, methodologies, and eval- uation metrics related to strategic reasoning with LLMs, highlighting the burgeoning development in this area and the interdisciplinary approaches enhancing their decision-makinn clarify the scattered literature on this subject, providing systematic review that underscores the importance of strategic reasoning critical cognitive capability and offers insights into future research directions and potential improvements.
performance. It aims to systematize and
1 Introduction
Large Language Models (LLMs) have ushered new era in artificial intelligence, partic- ularly highlighting the potential in performing reasoning tasks, including common sense question answering (Tarrin Talmor al. 2022), and mathematical problems Miao et al. 2021 etc.
Strategic reasoning Van Der Hoek et al. 2005 Duan et al. 2024 Gandhi et al. 2023 repre-
sents a distinct art of reasoning. Generally, it involves reasonably choosing the best strategy of action in multi-agent setting, considering how others will likely act and how one's own decisions will influence their choices. The necessity of strategic reasoning with large language models extends beyond academic curiosity; it is integral to understanding and navigating the complexities of the physical and social worlds. Human intelligence not only predicts the outcome of behaviors in the physical and social environments but also adjusts strategies based on these predictions. In order to endow Al with social attributes-male more intellectual, responsible, and equipped with an empathetic perspective-delyee into strategic reasoning with LLMs is imperative.
Strategic reasoning differentiates itself from other forms of reasoning by the dynamism of the reasoning environment and the uncertainty of adversary actions. We compare core cognitive skills required for various reasoning tasks in T requires not only profound understanding of the dynamic environment (context) but also the ability to make rational decisions within predictions of other participants. Strategic reasoning challenges are highly relevant to real-world issues, including business analysis and policy making.
"Work was done when interning at Microsoft Research Asia.
1 *Correspondence to:
1

---

Published as a conference paper at COLM 2024
Society
Economics
Game Theory
Gaming
Scenarios
Participant: "I'll bid the last moment avoid driving the price prema turely.
Player: bluff with weak hand, baiting opponent into betting then surprise them by winning the pot at the last moment."
Analyst: choose equilibrium strategy. even knowing my opponent's chaice, wouldn'tchang decision."
Debater: counter my opponent's points with facts and logic, strengthening position."
LLMs
Prompt Engineering
Module Enhancement
Theory Mind
Fine-tuning
Learning Gased Prompting
Memory
Cognitive hierarchy
Imitation Learning
Speciic Promoting
Knowledge
First-order Theory Mind
Reinforcement Learning
Panning
Search, Value, Belief
Methods
Figure Strategic reasoning with Large Language Models.
Due to the intriguing characteristics, strategic reasoning has attracted increasing attention from the academic community.
Prior to the widespread adoption of large language models, strategic reasoning has been confined intricate digitalized environments such as spatial action games, board games, and competitive video games, where agents' decision-making capabilities heavily rely on extensive simulation through reinforcement learning (Gronauer & Diepold [022 Arulku maran et al., 2017 Browne et al. 2012 Silver et al. 2017). These hindered the application scope and transterability of strategic reasoning. Fortunately, the advent of LLMSs has brought new opportunities for strategic reasoning. Firstly, the text generation capabilities of Large Language Models (LLMs) facilitate a wider range of strategic applications through the implementation of dialogue-baset generative agents. Secondly, the powerful contextual understanding capabilities of LLMs (Ouyang et al.) 2022 enable them to quickly grasp new scenarios, significantly extending the scope of Al-based strategic reasoning settings beyond the previous confines. Lastly, the text-based reasoning process offered by LLMs serves as simulation of human thought (Wei et al., 2022: Kojima al., 2022: Wang et al. 2023b), making decision-making more transparent and interpretable.
Leveraging the advantages of LLMs in decision-making and reasoning, there has been a flourishing development in enlarging scenarios recently. Meanwhile, methods from interdisciplinary fields such as cognitive hierarchy (Zhang al., (024c) and theory of mind (Guo et al.) 2023) are being adapted to enhance the decision-makinn performance of L.MMs. Despite the proliferation of applications and methodologies, there notable absence of systematic review on the use of LLMs in strategic reasoning that would organize and elucidate the differences and connections among these works. Compared to the literature on multi-agent reinforcement learning (Huh & Mohapatral 2023), utilizing
LLMs for strategic reasoning significantly diverges in methodology and application scope. The review literature using large language models for agent (Guo et al. 2024b Wang et al.) 2024 Xi et al. 2023), simulation (Gao et al. 2023) and game-play playing (Xu et al.) (024b) does
mention some aspects of strategic reasoning, but strategic reasoning as critical cognitive
2

---

Published as a conference paper at COLM 2024
Logical Deduction
Contextual Intelligence
Predictive Analytics
Abstract Thinking
Cognitive
Reasoning Task
Empathy
Common Sense Reasoning
Mathematical Reasoning
Symbolic Reasoning
Causal Reasoning
Strategic Reasoning
Table 1: An analysis of common reasoning tasks and their alignment with different cognitive skills. O, 00, and O indicate low, medium, and high levels, respectively. We have not exhaustively listed all the cognitive skills related to reasoning. Instead, we have primarily selected few representative cognitive skills associated with different reasoning tasks. The meaning of each cognitive skill explained in Appendix A.1
capability should be focused on and systematically analyzed. This paper aims to provide a comprehensive overview of the state of LLMs strategic reasoning, shedding light on their capabilities, applications, and the road ahead for harnessing their potential more effectively.
The rest of this survey is organized in the following order: Section 2 delves into the defi- nitions and scopes of strategic reasoning, outlining how strategic reasoning differentiates from other reasoning scenarios. Section explores the applications of LLMs in strategic reasoning, categorizing tasks and application domains. Section discusses existing methods to enhance LLMs in strategic reasoning, classifying approaches for employing LLMs in strategic thought processes. Section 5 discusses how to evaluate LLMs' performance in strategic reasoning, incorporating both quantitative assessments and qualitative analysis of capabilities. Lastly, Section engaage in a discussion on the challenges and opportunities presented by applying LLMs strategic reasoning, offering insights into future research directions and potential improvements based on the current limitations of the research.
6
Definition: What is Strategic Reasoning with LLMs
Strategic reasoning can be defined as the ability to anticipate and influence the actions of others competitive or cooperative multi-agent setting. This involves understanding the motives, intentions, and potential actions of others, as well as the causal relationships within the environment. Unlike other forms of reasoning, which may focus on static problem- solving or single-agent decision-making, strategic reasoning is inherently dynamic and interactive, requiring a continuous assessment of the evolving situation and the intentions of other agents. In Appendix A.2] we provide formal definition of strategic reasoning with LLMs.
The core characteristics of strategic reasoning include:
Goal-Oriented: The reasoning process is directed towards achieving specific objectives, often within competitive or cooperative framework.
Interactivity: Strategic reasoning involves interaction among multiple agents, each influenc- ing and being influenced by the decisions of others.
Predictive Nature: requires the ability to predict the actions and responses of other agents based on limited information and uncertain outcomes.
Adaptability: Agents must be able to adapt their strategies in response to the actions of others and changes in the environment.
is also important to define what falls outside our discussion's scope. Specifically, we will not address environments lacking in strategic complexity such as simulations with Generative Agents (Park et al. 2023) that do not engage in obvious strategic reasoning. Additionally, scenarios of multi-agent collaboration task solving that not require dynamic environmental adjustments or feedback from collaborators are also excluded from the analysis of strategic reasoning. This exclusion includes both the environments and use cases
3

---

Published as a conference paper at COLM 2024
where the strategic reasoning is either absent significantly minimized, ensuring focus remains on the strategic applications of LLMs in contexts that demand comprehensive understanding of goals, competition, and environmental dynamics.
Scenarios: Where to Apply Strategic Reasoning with LLMs
Bog (Gandhi et al.
2024 SOTOPIA (Zou at al.
2023
UGI (Xu al.
2023a
Suzuki & Arita
2024
Open ToM
Societal behavior
Xuu et
2024a Xie
(2024) Mou
2024
Zhou
(0044). Yang
2024
Strachan
2024
InterIn-
Societal Simulation
Fu et al. 2023 Abdelnabi et al.
2023 TRIP [Zhang et al.
2024a). WarAgent (Hua al.
2023
Flamino al. (2024).
Debate & Negotiation
Taubenfeld
2024
2023
Gemp
2024
Hua
2024
Schneider
(2023 Lamparth
2024). Davidson
2024).
Negotiation Chan
Horton
2023
Xie et al. (002)) Chen et al.
2023b Li et al.
Economics
2023b
EconAgent et 2024
Economic Simulation
Han al.
(2023).
TradingGPT
al.
2023c
CompeteAI
Business
Zhao et al
2023 Chen et al.
2023a
DG-Narrtoor Xia
et al 2024
InvAgent Quan & Liu
2024
2023
Phelps & Russel
2023). MAgiC (Xu
Matrix Game
2023h
et [0023 Brookins DeBacker 2023
Fan et al.
2024
Fontana et al. (2024)
Game Theory
Akata al.
2023
Alympics (Mao et
2023
K Level
Scenarios
Repeated Game
Reasoning
Zhang al 2024c Wu
2024b
Bench
(Huang
2024b)
Xu et al.
2023d
Xu et al.
(023cc
Thinker
WereWalf
(Wu et al.
2024a
Werewolf Arene
Bailis et al.
2024
Avalonbench
Light et al.
2023
ReCon WWng
Avalan
Conversational
ct 2023a
Lan ct
2023
Games
Welfare diplomacy Mukobi et
Cicero
Diplomacy
[FAIR). Richelieu
Guan
MAgiC Xu
2023b
Hoodwinked 0 Gara
Other
2023
Tsai et al
2023
et al
2023g
GameBench
Costarelli
2024
SuspicionAgent (Gno al.
2023
PokerGPT
(Huang et al. 2024a Agent-Pi Pro Zhang et al.
Poker
2024b BIDDER (Zhang al
2024d
UND
Board & Card Game
Arena
Qin et
2024).
Gaming
ChessGPT
Feng et
2024
Kuo et
2023
Chess
Fang ct al 2023
Guo ct
2024g)
Topsakal
Harper
2024
LLM-Co Co (Agashe
2023
TextStarCraft II (Ma
SwarmBrain (Shao
PokéLLMon
Electronic Game
2024
CivRealm
et
2024
(de Wynter 2024
Cradie
Tan al.
2024
Figure Taxonomy of scenarios of strategic reasoning by I.LM-basee agents.
This paper delineates the distinct aspects of LLM applications strategic reasoning scenar- ios, illustrating how these models forecast and adapt within various settings. As shown in Figure we categorize these scenarios into societal simulation, economic simulation, game theory, and gaming. Each category represents different environment or set of conditions under which strategic reasoning is required, and together they showcase the versatility and depth of LLMss in understanding and influencing multi-agent dynamics:

---

Published as a conference paper at COLM 2024
Societal Simulation focuses on the simulation of social systems and interactions, where LLMs are used to model and predict human behavior in complex societal contexts. It involves multiple agents (individuals or groups) whose interactions are influenced by social norms, cultural values, and collective behaviors. By simulating these interactions, LLMs can help in understanding societal trends, decision-making processes, and the impact of policies or interventions. To advance the study of the social intelligence of LLMs, BigToM (Gandhi et al. )024), SOTOPIA (Zhou et al., 2023), , and OpenToM (Xu et al. 2024a) have been introduced as pivotal frameworks. These tools are engineered to assess LLMs' abilities to comprehend human psychological states as well as their social skills, such as empathy, persuasiveness (Zhou et al. 2023), and trsut (Xie et al. 2024). In the realm of political debates, Taubenfeld et al. (2024) and [lang et al. (023) critically assess the limitations of LLMs in simulating human-like interactions, pointing out tendency for LLM agents to adhere to inherent social biases despite attempts to engage them in diverse political perspectives. This highlights the challenges in achieving unbiased and representative simulations of societal discourse. The simulation of historical conflicts, as presented in WarAgent (Hua et al. 2023), exemplifies the potential of LLLM-powered AI systems to recreate and analyze international disputes, offering a novel perspective on understanding the decisions and outcomes of major historical events such as the World Wars and the Warring States Period.
Economic Simulation involves modeling market dynamics, business operations, and finan- cial decision-making processes. In this setting, LLMs are applied to understand and predict the outcomes of economic decisions, simulating scenarios like market competition, resource allocation, and investment strategies. These simulations require strategic reasoning to navigate complex economic landscapes, optimizing outcomes based on predictions of other agents' behaviors. LLMs demonstrate their capability analyze and participate in economic systems, showcasing strategic thinking in monetary and business environments. H Chen et al. Xie al. Li et al.] have contributed to understanding how -LMeempoeeeed agents can simulate hiring scenarios, demonstrate rational decision-making in economic experiments, and predict stock movements, These studies underscore the capacity of LLMs to mimic realistic work and consumption decisions, potentially reshaping macroeconomic modeling. CompeteAI (Zhao et al., 2023) framework introduces a competitive environment the interaction between restaurant and customer agents,
simulated with GPT-4, focusing which illustrates the dynamics of business competition. Furthermore, AucArena (Chen et al. 2023a) demonstrates how LLMs can engage effectively in auctions, emphasizing the adaptability and strategic thinking capabilities of these models.
Game Theory is the study of strategic interaction among rational meaksss It is inherently about strategic reasoning, as it involves predicting and countering the moves of other players in various game settings. LLMs engaged in game-theoretic simulations are tested on their ability to formulate strategies in competitive, cooperative, and mixed situations. This not only shows LLMs' strength in abstract strategic reasoning but also their application in practical scenarios where understanding and anticipating the actions of others are crucial. In the realm of game theory, LLMs have been instrumental in analyzing and engaging in strategic play, demonstrating their ability to model fairness and cooperation in matrix and repeated games, as highlighted in studies by(X Xy et al. (2023b), Gandhi et al. (2023), and Brookins & DeBacker (2023). The ongoing research into frameworks like Alympics (Mao et 2022) and approaches like k-level reasoning (Zhang et al. 2024c) showcases LLMs' proficiency in multi-round strategic thinking, providing insights into their long-term strategic planning capabilities.
In the context of Gaming, which includes board games (Feng et al. 2024 Kuo et al. 2023)
card games (Guo et al.) 2023 Huang et al.) 2024a Zhang et al. 20246), and video games (Ma
ett 2023 Agashe et al.] (203) Hu et al. 22024) strategic reasoning critical for success. LLMs are used to understand game mechanics, develop winning strategies, and adapt to opponents' tactics. This ca category demonstrates L.MS'' ability to engage in and enhance the strategic depth of interactive entertainment, reflecting their potential to reason and make decisions in dynamic and often unpredictable environments. In conversational games like Werewolf, Chameleon, and Avalon, research by Xu et al. (2023d), Wu et al. (2024a), ) and Light let al. (2023) demonstrates how LLMs can enhance communication, reasoning, and deception detection among agents. In board and card games, Guo et al. (2023) and Feng et al. (024
5

---

Published as a conference paper at COLM 2024
have shown how LLMs outperform traditional algorithms in poker and integrate policy learning in chess, respectively. These findings suggest a broader applicability of LLMs beyond mere simulation, potentially transforming strategic game play. Electronic gaming, including StarCraft and Pokémon, has also benefited from LLM integration. TextStarCraft II (Ma et a 2023) and PokeLLMon (Hu et al. 2024) showcase the capability of LLMss to process game information, recommend strategies, and exhibit human-parity performance in tactical battles.
Overall, LLMs are pivotal in elucidating and enhancing strategic reasoning across diverse simulations, each category offering unique insights and challenges.
4 Methods: How to Improve Strategic Reasoning with LLMs
In order to enhance the performance of LLM in strategic reasoning challenges, numerous methods have recently emerged. We categorize these methods into four types based on their underlying motivations, as illustrated in Figure
Prompt Engineering
LLM
Action
Learning- Based Prompting
Iask-Specfic Prompting
Theory of Mind
Firs-orderr Theory of Mind
Search, Value, Betef
Cognitive Hierarchy
example
d
Reasoning Action
LLM
Module Enhancement
Fine-tuning
Imitation
Reinforcement
Memory
Policy
LLM
knowledge
Preparstion
LLM
Tweccback
Supervised Fine-tuning
Environment
Summorization
Planning
Ressoning Actiion
Figure Methods for Improving Strategic Reasoning with Large Language Models.
Prompt Engineering refers to the techniques and methodologies employed in construct- ing effective prompts to guide Large Language Models (LLMs) in generating impactful prompting (In-context Learning (Brown et al. 2020 Wei et al. 2022 and task-specific prompting (zero-shot
outputs. This includes learning-Basee
(Kojima et al. 2022)). Given the more complex contextual backgrounds of tasks involving strategic rea- soning as compared to mathematical reasoning, utilizing Prompt Engineering to facilitate clearer understanding of scenarios by LLMs presents direct approach. To augment the situational awareness of Large Language Models (LLMs) and leverage learning from gaming history, the investigations by Fu et al. (2023), Xu et al. (2023c), Wu al. (2023), HudHaa et al. (2024) have focused on the retrieval of historical gaming data for Incontext Learning (Brown et al. 2020). These endeavors aim to enhance the proficiency LLMs in negotiation and communication games through feedback titepfuz223improving and reflection (Xu et al. 2023c). These studies illustrate how prompt engineering not only boost LLMs' understanding and engagement in strategic games and systems but also their ability to adapt and refine these skills over time, highlighting the potential of LLMs in strategic thinking and decision-making.
Hua

---

Published as a conference paper at COLM 2024
Modular Enhanced Agents demonstrate superior performance in strategic reasoning sce- narios, such as games, by integrating functionalities like memory modules for the reuse of successful strategies and leveraging external knowledge bases for the retrieval of use- ful information or domain-specific data. To augment the efficacy of communication and interaction in LLMs,, Lan et al. (2023) proposes innovative and encompassing framework designed for seamless adaptation to the Avalon game, including modules dedicated to sum marization, analysis, planning, and action. Wthin the bargaining context, the OG-Narrator (Xia et al., 2024) incorporates deterministic quote generator that regulates the price range of buyer propositions, along with an LLM-based narrator that formulates natural language sentences for these quotes, achieving a decuple increase in profitability relative to the base- lines. In complex gaming environments, PokéLLMon (Hu et al. 2024) and Thinker (Wu et al. 2024a) address the phenomenon of illusions faced by LLM-based agents through the retrieval of external knowledge. The strategic capabilities of agents in StarCraft have been subject of long-standing research interest. In this vein, TextStarCraft I1(Ma et al., 2023 applies Large Language Models (LLMs) to StarCraft, introducing Chain of Summarization method encompassing both single-frame summaries, aimed at processing raw observations, and multi-frame summaries, designed for the analysis of game information, provision of command recommendations, and generation of strategic decisions. This holistic enhance- ment of cognitive abilities makes agents more autonomous and effective in wide range of scenarios, from simple decision-making to complex strategic reasoning and planning in dynamic scenarios.
2024.7
Richelieu( Guan al.. 24)
InvAgent( Quan & Liu, 2024)
EconAgent (1 et al. 2 24)
Yang et al. (2244)
UNO Arena Qin at al., 2 2(24)
Cradie Jan ot al.
TimeToM (Hoo al, 24)
2024.4
(22))
OG- OG-Narrator> (a al. 2024)
K-Level Reasoning
Agent-Pu Agent-Proo(ee hhang al., 244b)
Thinker[ (u ot al, 20243)
Zhang at al., (244))
Guo et at. (2021a)
Hua (224)
PokiLLMon Ho et ar. 2p29)
ThiP
LLaMAC Zhang al., 2/23)
2024.1
INA AAhaa at al., 2023)
TestStarCraft 11 et al. 2433)
MAgiC( Ku et al.. (033b
PokerGPT (Hrung et 2024
Wu 0233
SwarmBrain[ Shoo et al., 2024)
SimToMM Vil et al. (222))
CivRealm (or at.,
2023.10
SuspicionAgent Guo et al., 20133
Retroformer( Nao et al., 2023)
ReCon Wang al., (123a)
Fong at
Xu (233c)
Chen (20254),
Xu
2023.7
Xie et al.
023)
Cicorol AIR)
al.
023)
Lord & Heydari (0123)
ChessGPT et al., 2024)
Prompt Engineering
Modular Enhancements
Theory Of Mind
Fine-tuning
Methods
Figure 4: Overview of methods for strategic reasoning work based on LLMs.
Theory of Mind (ToM) is a crucial concept in strategic reasoning, enabling agents to antici- pate and strategize based on the mental states of others. Gandhi al. (2023) and Suspicion- Agent (Guo et al. (203)) employ the ToM framework for decomposing the strategic reasoning process into search algorithmss, value assessments, and belief tracking environments, tai- lored specifically for Matrix Games and Poker, respectively. This method significantly elevates the decision-making prowess of Large Language Models (LLMs). Sim WWif et al., 2023) and K-Level Reasoning (Zhang et al. 2024c) demonstrate that predictions about opponents' behaviors become markedly more precise when utilizing opponent-specific sessions. K-Level Reasoning further elucidates that broader historical record of opponents' actions can enhance prediction accuracy, illustrating the dynamic adaptability of L.MMs. This adaptability has notably improved LLMs' intelligence in DOOM (de Wynter 2024). Additionally, Li et al. (2023a) finds that LLMs exhibit ToM capabilities in cooperative tasks, and their performance levels are comparable to reinforcement learning baselines in these
7

---

Published as a conference paper at COLM 2024
tasks. Together, these works illustrate the significant role ToM plays in enriching LLMs' strategic reasoning capabilities, demonstrating its potential to revolutionize mecision-making processes across various domains.
The fusion of Imitation
Learning and Reinforcement Learning (RL) with LLMs also marks significant advancement in strategic reasoning capabilities. The initiatives of Feng et al.. (2023), Guo et al. (2024a), , and ChessGPT (Feng et al. 2024) are pivotal in integrafing Large Language Models (LLMs) within the chess domain. 1o augment LLMs' chess performance, a bifurcated approach is adopted: firstly, by emulating the experiential wisdom of human players, thereby imbibing the sophisticated strategies and tactical decisions inherent ex- pert gameplay; and secondly, by hamnessing LLMs' inherent pre-trained reasoning, prowess as a value function to directly boost their operational efficacy. Gemp et al. (2024) offers broader perspective by conceptualizing dialogue processes as game-theoretica constructs. Herein, the reinforcement learning framework is deployed to refine LLM performance in intricate interactive contexts such as meeting scheduling and public debates. Collectively, these advancements underscore the utility of incorporating imitation and reinforcement learning into LLMs, showcasing their potential to navigate and reason within complex decision-making landscapes with unprecedented sophistication.
It is important to note that the boundaries between the above methodological categories are not entirely orthogonal. For instance, methods pertaining to Theory of Mind can be implemented through prompt engineering, yet their essence lies in leveraging game theory principles to enhance LLM performance, rather than merely providing examples to refine LLM's understanding of task definitions. Finally, Figure offers an overview of the methods employed to improve the efficacy of LLMs in strategic reasoning tasks.
Evaluations: How to Assess Strategic Reasoning with LLMs
The Evaluation in strategic reasoning includes measuring outcomes in controlled environ- ments, where the efficacy of a model can be gauged through its performance metrics like win rates (Qia et al. 2023), survival rates (Mao et al. 2023), and rewards. Researches such as GTBench (Duan et al. 2024) and LLMArena (Chen et al. 2024), with their sophisticated scoring systems like the Normalized Relative Advantage (NRA) and TrueSkill (Herbrich) eet al., 2006), respectively, provide a structured framework for this analysis. These tools monllyuaaiify
not only quantify success but also allow for comparisons across various game types and difficulty levels, offering a comprehensive view of an LLM's strategic prowess.
The evaluation in strategic reasoning with LLMs also includes a Quantitative Analysis of the reasoning processes. Metrics that target the processes within games focus assessing the LLM's ability to perceive, predict, and adapt to the dynamic environment and oppo- nents' strategies. For instance, MAgiC (Xu et al., 2023b) evaluates the accuracy of LLMs in analyzing opponents' moves under conditions of incomplete information, and K-Level Reasoning (Zhang et al. 2024c) assesses the precision in predicting behaviors based public information. Process-oriented evaluations are critical in multi-agent environments where nonstationarity, due to uncertain opponent behavior, significantly impacts perfor- mance. Accurate predictions of opponent behavior are essential to mitigate the effects of this nonstationarity, offering more clear view of an LLM's strategic capabilities.
Moreover, considering the intrinsic advantages of LLMs, such as their ability to generate reasoning processes, provides a unique angle for evaluating strategic reasoning. Unlike reinforcement learning methods that focus merely on outcomes, LLMs offer explainability by detailing the inferential steps they take. This characteristic enables more focused assessment, where the model's outputs themselves can be analyzed to understand the decision making process better. Therefore, it is imperative to integrate these insights into the quantitative evaluation of LLMs.
Qualitative Evaluation shifts towards understanding the underlying mechanics of strategic reasoning in LLMs, encompassing capabilities like deception, cooperation, discernment, and so on. These aspects are crucial for navigating the intricacies multi-agent interactions, where the effectiveness of strategy is often contingent on the dynamic and often unpre-
8

---

Published as a conference paper at COLM 2024
dictable nature of opponent behavior and game states. For example, in games like Werewolf (Xu et al. 2023c) or Poker (Guo et al. 2023), the ability to bluff or cooperate effectively is as indicative of strategic reasoning as the ultimate game outcome.
The interplay between quantitative and qualitative evaluations is crucial for a holistic understanding of LLMs' rerategic capabilities. While quantitative analysis offers objective benchmarks, qualitative insights expose the strategic depth and adaptability of LLMs in complex, real-world scenarios. This dual approach not only enhances the robustness of the evaluation framework but also metigates the challenges inherent in measuring cognitive processes in strategic reasoning.
The selection of these quantitative and qualitative metrics is intricately linked to specific contextual scenarios. For instance, in survival games, metrics such as agent survival rates are pertinent, whereas, in games without elimination, metrics like win rates or rewards are more relevant. Additionally, different scenarios necessitate distinct focuses on the reasoning capabilities of the employed strategies. Regarding the rationale behind selecting these quantitative metrics, we aimed two primary objectives:
These metrics evaluate the rationality of LLMs, highlighting the disparity between LLMs and other methods. Moreover, these metrics provide benchmark for comparing the relative rationality of different methods.
Rationality Assessment of LLMs:
Enhanced Transparency in Decision-Making: As strategic reasoning with LLMs aims to make decision-making more transparent and interpretable, these metrics should analyze the performance of LLMs in the process of strategic reasoning.
Discussions: An Outlook on Strategic Reasoning with LLMs
6.1 Can the LLM agents really simulate human strategic reasoning?
Although LLMs and LLM agents have been applied in variety of strategic reasoning scenarios, with some studies claiming the emergence of human-like intelligence capabilities in certain simulations, we argue that there is a lack of systematic and rigorous research into what level of LLM can be employed to simulate tasks of varying complexity and cognitive difficulty in strategic reasoning. This absence of systematic and rigorous study has led to a gap in understanding the scalability and limitations of LLMs in these contexts. Specifically, it remains unclear how different sizes and configurations of LLMs correlate with their ability to handle the decision-making and predictive tasks required in complex strate- gic environments. Without this knowledge, the application of LLMs in strategic reasoning risks being haphazard, potentially overlooking critical insights into the model's capabilities, decision-making processes, and potential biases or shortcomings. Therefore, a more struc- tured approach to researching and categorizing LLM competencies in strategic reasoning essential for fully leveraging their potential and ensuring responsible development and deployment in multi-agent strategic simulations.
6.2 Bridging the Divide: The Urgent Need for Unified Benchmarks
A key challenge in strategic reasoning is the absence of unified benchmarks. While recently there are some benchmarks (Xu et al. 2023b; Duan et al. 2024 Chen et al. 2024 derived from classical game theoretical problems for strategic reasoning, the vast application range of strategic reasoning, from business strategy complex system simulations, has led pro- liferation of customized solutions focused on novel scenarios rather than deep exploration within well-defined benchmarks. This trend hinders direct method comparisons and stifles progress under a common standard. Also, as mentioned in Section 5| in tasks of strategic reasoning, it is often necessary to use a combined quantitative and qualitative evaluation approach to comprehensively evaluate the performance of LLMs both in the reasoning process and the outcome, which presents challenges for the design of a unified benchmark. The strategic reasoning community urgently needs to collaborate difficulty level, recognized benchmarks that cover its diverse applications. Specifically, believe that unified benchmark should possess the following characteristics:
creating a suitable

---

Published as a conference paper at COLM 2024
Well-Controlled Environments: The benchmark should operate within controlled environments with minimal randomness, facilitating reproducibility and ease of experi- mentation.
Reasonable (Objective or Subjective) Evaluation: As discussed in section 5, unified benchmark for strategic reasoning with Large Language Models (LLMs) should be capable of objectively or subjectively assessing their strategic reasoning abilities. This includes evaluating long-term planning capabilities and dynamic adaptability, among other facets of strategic reasoning.
Scalability Across Scenarios:
The benchmark should be adaptable to broad spectrum of strategic reasoning scenarios, enabling researchers to explore diverse applications and challenges.
These benchmarks would facilitate algorithm performance assessment, method comparison, and drive innovation by defining clear metrics, representative datasets, and evaluation protocols. Such efforts could unify the field, enhance knowledge sharing, and accelerate technological development.
6.3 Strategic Reasoning: Challenging yet Promising for Large Language Models
Strategic reasoning presents a unique challenge LLMs. These models, which rely on next token prediction during pre-training stage, are adept at learning patterns from vast amounts of static textual data (Sap et al., 2022) but struggle to inherently grasp the subtlety of strategic reasoning. This limitation stems from the fact that strategic reasoning requires understanding complex, dynamic interactions between multiple agents, something that not directly inferable from static text data alone. Despite this, the massive volume of data used to train LLMs allows them to model a wide range of behaviors and scenarios, indirectly capturing elements of strategic thought. By crafting prompts or algorithms that frame problem within a strategic context, these models can generate responses that reflect strategic considerations.
However, the question remains whether scaling up-merely increasing the number of pa- rameters and the volume of training data for a general-purpose LLM could suffice for general-p general-purpose purpose LLMs to fully master strategic reasoning. While larger models can capture more fine and complex patterns, strategic reasoning fundamentally involves understanding intentions, predicting future actions based on those intentions, and dynamically adjusting strategies response to evolving situations. These aspects are not purely function of model size data quantity. We speculate that even the most powerful general-purpose. LLMs in may fall short of fully realizing strategic reasoning capabilities.
7 Conclusion
In conclusion, our review highlights the pivotal role of LLMs in strategic reasoning, showcas- ing their evolution and significant advantages in complex decision-making across various domains. Future efforts should focus on interdisciplinary collaborations to bridge theoret- ical advancements and practical applications, enhancing decision-making processes and strategy development. As we advance, the exploration and refinement of LLMs promise to offer substantial advancements in artificial intelligence, opening new pathways for solving complex problems and enriching strategic decision-making in interconnected world. This calls for a concerted effort from researchers and practitioners to unlock the transformative impact of LLMs on strategic reasoning.
Acknowledgments
We would like to express our gratitude to our colleagues at Microsoft (Jindong Wang, Adrian de Wynter) and East China Normal University (Li Cai and Xinshu Shen) for their valuable internal discussions and feedback. We also appreciate the reviewers' thorough review and valuable comments.
10

---

Published as a conference paper at COLM 2024
References
Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. Llm- deliberation: Evaluating Ilms with interactive multi-agent negotiation games. arXiv preprint ar Xit:2309.17234, 2023.
Saaket Agashe, Yue Fan, and Xin Eric Wang- Evaluating multi-agent coordination abilities in large language models. arvio preprint ar arXiv:2310.03903 2023.
Zishan Ahmad, Suman Saurabh, Vaishakh Sreekanth Menon, Asif Ekbal, Roshni Ramnani, and Anutosh Maitra. Ina: An integrative approach for enhancing negotiation strategies with reward-based dialogue agent. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. ar Xiv preprint ar XrXvv22305.168 2023.
Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath.
Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 2666)226-38, 2017.
Suma Bailis, Jane Friedhoff, and Feiyang Chen. Werewolf arena: A case study in Ilm evaluation via social deduction. arXiv preprint arXtm:2407.13943, 2024.
Philip Brookins and Jason Matthew DeBacker. Playing games with gpt: What can we learn about a large language model from canonical strategic games? Available at SSRN 4493398, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1):1-43, 2012.
Chunkit Chan, Cheng Jiayang, Yauwai Yim, Zheye Deng, Wei Fan, Haoran Li, Xin Liu, Hongming Zhang, Weiqi Wang, and Yangqiu Song. Negotiationtom: A benchmark for stresss--estingggge machine theory of mind on negotiation surrounding. ar preprint ar Xii:2404.13622 2024.
Jiangjie Chen, Siyu Yuan, Rong Ye, Bodhisattwa Prasad Majumder, and Kyle Richardson. Put your money where your mouth is: Evaluating strategic planning and execution of llm agents in an auction arena. arXiv preprint ar drXi::3310005746, 2023a.
Junzhe Chen, Xuming Hu, Shuodi Liu, Shiyu Huang, Wei-Wei Tu, Zhaofeng He, and Lijie Wen. LImarena: Assessing capabilities of large language models in dynamic multi-agent environments. Xiv preprint arXiv:2402.16499 2024.
Yiting Chen, Tracy Xiao Liu, You Shan, and Songfa Zhong. The emergence of economic rationality of gpt. Proceedings of the National Academy of Sciences, 120151))e3301205122, 2023b.
Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, and Arjun Yadav. Gamebench: Evaluating strategic reasoning abilities of I1 agents. arxiv preprint ar Xitx:2406006613, 2024.
Tim R Davidson, Veniamin Veselovsky, Martin Josifoski, Maxime Peyrard, Antoine Bosse- lut, Michal Kosinski, and Robert West. Evaluating language model agency through negotiations. ar Xit preprint ar Xix: 004566,,6 2024.
11

---

Published as a conference paper at COLM 2024
Adrian de Wynter. Will gpt-4 run doom? arxiv preprint arXit:2403.055468 2024.
Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel-Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. Gtbench: Uncovering the strategic reasoning limitations of Ilms via game- game-theoretic evaluations. arXiv preprint ar drXiv:2402.12348, 2024.
Meta Fundamental AI Research Diplomacy Team (FAIR)+, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 37866624)11067-1774, 2022.
Caoyun Fan, Jindou Chen, Yaohui Jin, and Hao He. Can large language models serve as rational players in game theory? a systematic analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17960-17967, 2024.
Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero- like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.17179 2023.
Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Mguni, Yali Du, and Jun Wang. Chessgpt: Bridging policy learning and language model- ing. Advances in Neural Information Processing Systemss, 36, 2024.
James Flamino, Mohammed Shahid Modi, Boleslaw K Szymanski, Brendan Cross, and Colton Mikolajczyk. Limits large language models in debating humans. arXiv preprint arXit:2402.06049, 2024.
Nicoló Fontana, Francesco Pierri, and Luca Maria Aiello. Nicer than humans: How do large language models behave in the prisoner's 's dilemma? ar preprint ar (tt:::24061360055 2024.
Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self-play and in-context learning from ai feedback. arXiv preprint arXio:2305.10142, 2023.
Kanishk Gandhi, Dorsa Sadigh, and Noah D Goodman. Strategic reasoning with language models. Xrviv preprint vrXiv:2305.19166 2023.
Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah Goodman. Under- standing social reasoning in language models with language models. Advances in Neural Information Processing Systems, 36, 2024.
Chen Gao, Xiaochong Lan, Nian Li, Yuan Yuan, Jingtao Ding, Zhilun Zhou, Fengli Xu, and Yong Li. Large language models empowered agent-based modeling and simulation: A survey and perspectives. arXio preprint ar Xix::2312.11970, 2023.
Ian Gemp, Yoram Bachrach, Marc Lanctot, Roma Patel, Vibhavari Dasagi, Luke Marris, Georgios Piliouras, and Karl Tuyls. States as strings as strategies: Steering language models with game-theoretic solvers. arXiv preprint ar arXio:2402.01704 2024.
Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: survey. Arlificial Intelligence Review, 55(2):895 943,, 2022.
Zhenyu Guan, Xiangyu Kong, Fangwei Zhong, and Yizhou Wang. Richelieu: Self-evolving based agents for ai diplomacy. arxiv preprint ar arXio:2407.06813, 2024.
Fulin Guo. Gpt agents game theory experiments. Technical report, 2023.
Hongyi Guo, Zhihan Liu, Yufeng Zhang, and Zhaoran Wang. Can large language models play games? a case study of self-play approach. arxiv preprint arXivv2403.05632, 2024a.
Jiaxian Guo, Bo Yang, Paul Yoo, Bill Yuchen Lin, Yusuke Iwasawa, and Yutaka Matsuo. Suspicion-agent: Playing imperfect information games with theory of mind aware gpt-4. ar Xiv preprint X:Xtt:2309.17277, 2023.
12

---

Published as a conference paper at COLM 2024
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agentss survey of progress and challenges. aro preprint arXiv:2402.016680 2024b.
Xu Han, Zengqing Wu, and Chuan Xiao. " guinee pig trials" utilizing gpt: A novel smart agent-based modeling approach for studying firm competition and collusion. arXiv preprint ar Xiv:2308. 10974, 2023.
Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill bayesian skill rating system. Advances in neural information processing systems, 19, 2006.
John Horton. Large language models simulated economic agents: What we learn from homo silicus? Technical report, National Bureau of Economic Research, 2023.
Guiyang Hou, Wenqi Zhang, Yongliang Shen, Linjuan Wu, and Weiming Lu. Timetom: Temporal space is the key to unlocking the door of large language models' theory-of-mind. Xix preprint ar XrXt::2407.01455, 2024.
Sihao Hu, Tiansheng Huang, and Ling Liu. Pokk 'ellmon: A human-parity agent for pok) 'emon battles with large language models. arXiv preprint ar xrXiv:2402.01118, 2024.
Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. War and peace (waragent): Large language model-based multi- agent simulation of world wars. arXiv preprint ar Xri::2311.172221 2023.
Yuncheng Hua, Lizhen Qu, and Gholamreza Haffari. Assistive large language model agents for socially-aware negotiation dialogues. arviv preprint ar XrXio22402.01737,2000 2024.
Chenghao Huang, Yanbo Cao, Yinlong Wen, Tao Zhou, and Yanru Zhang. Pokergpt: An nold'em em via large language model.
end-to-end lightweight solver for multi-player texas hold' arXiv preprint ar XxX::2401.0077
2024a.
Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Michael Lyu. How far are we on the decision-making of Ilms? evaluating lims' gaming ability in multi-agent environments. arviv preprint ar arXiv:2403.11807 2024b.
Dom Huh and Prasant Mohapatra. Multi-agent reinforcement learning: A comprehensive
survey. arXiv preprint arXit:2312.10256 2023.
Philip N Johnson-Laird. Deductive reasoning. Annual review of psychology, 50(1):109-135, 1999.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systemss, 35:22199-22213,, 2022.
Mu-Tien Kuo, Chih-Chung Hsueh, and Richard Tzong-Han Tsai. Large language models on the chessboard: A study on chatgpt's formal language comprehension and complex reasoning skills. Xiv preprini arXiv:2308. 155188 2023.
Max Lamparth, Anthony Corso, Jacob Ganz, Oriana Skylar Mastro, Jacquelyn Schneider, and Harold Trinkunas. Human vs. machine: Language models and wargames. arxiv 2024.
preprint ar
Yihuai Lan, Zhiqiang Hu, Lei Wang, Yang Wang, Deheng Ye, Peilin Zhao, Ee-Peng Lim, Hui Xiong, and Hao Wang. Llm-based agent society investigation: Collaboration and confrontation in avalon gameplay. arv preprint ar Xriv:2310.14985, 2023.
Huao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Michael Lewis, and Katia Sycara. Theory of mind for multi-agent collaboration via large language models. ar Xiv preprint arX1::3310.1000,, 2023a.
13

---

Published as a conference paper at COLM 2024
Nian Li, Chen Gao, Yong Li, and Qingmin Liao. Large language model-empowered agents for simulating macroeconomic activities. arXiv preprint ar arXic:2310.10436,220 2023b.
Nian Li, Chen Gao, Mingyu Li, Yong Li, and Qingmin Liao. Econagent: Large language model-empowered agents for simulating macroeconomic activities. Preprint, 2024.
Yang Li, Yangyang Yu, Haohang Li, Zhi Chen, and Khaldoun Khashanah. Tradinggpt: Multi-agent system with layered memory and distinct characters for enhanced financial trading performance. ar preprint ar Xii::3399003736 2023c.
Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. Avalonbench: Evaluating Iims playing the game of avalon. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023.
Ziyi Liu, Abhishek Anand, Pei Zhou, Jen-tse Huang, and Jieyu Zhao. Interintent: Investigat- ing social intelligence of Iims via intention understanding in an interactive game context. arviv preprint arXiv:2406.12221 2024.
Nunzio Lorè and Babak Heydari. Strategic behavior of large language models: Game structure vs. contextual framing. arXiv preprint ar Xiio:2309.05898, 2023.
Weiyu Ma, Qirui Mi, Xue Yan, Yuqiao Wu, Runji Lin, Haifeng Zhang, and Jun Wang. Large language models play starcraft ii: Benchmarks and a chain of summarization approach. arXiv preprint ar drXiv:2312.11866 2023.
Shaoguang Mao, Yuzhe Cai, Yan Xia, Wenshan Wu, Xun Wang, Fengyi Wang, Tao Ge, and Furu Wei. Alympics: Language agents meet game theory. arXiv preprint arXi::2311.03220, 2023.
Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing englishh math word problem solvers. arXio preprint ar #rXi::2106.1572, 2021.
Xinyi Mou, Zhongyu Wei, and Xuanjing Huang. Unveiling the truth and facilitating change: Towards agent-based large-scale social movement simulation. ar preprint 1rXiv:2402.16333, 2024.
Gabriel Mukobi, Hannah Erlebach, Niklas Lauffer, Lewis Hammond, Alan Chan, and Jesse Clifton. Welfare diplomacy: Benchmarking language model cooperation. arxiv preprint arXiv:2310.08901, 2023.
Aidan O'Gara. Hoodwinked; Deception and cooperation in text-based game for language models. arXiv preprint ar Xiv:2308 000.2308..0144,, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-277444 2022.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposiuon on User Interface Software and Technology, pp. 1-22, 2023.
Steve Phelps and Yvan Russell. Investigating emergent goal-like behaviour in large language models using experimental economics. Xrv preprint ar a/X1v:2305007970, 2023.
Siyuan Qi, Shuo Chen, Yexin Li, Xiangyu Kong, Junqi Wang, Bangcheng Yang, Pring Wong, Yifan Zhong, Xiaoyuan Zhang, Zhaowei Zhang, et al. Civrealm: learning and reasoning odyssey in civilizàtion for decision-making agents. ar preprint ar #xXi::2401.1066 2024.
Dan Qiao, Chenfei Wu, Yaobo Liang, Juntao Li, and Nan Duan. Gameeval: Evaluating lims conversational games. Xiv preprint arXiv:2308.1003 2023.
14

---

Published as a conference paper at COLM 2024
Zhanyue Qin, Haochuan Wang, Deyuan Liu, Ziyang Song, Cunhang Fan, Zhao Lv, Jinlin Wu, Zhen Lei, Zhiying Tu, Dianhui Chu, et al. Uno arena for evaluating sequential decision-making capability of large language models. arXiv preprint ar arXio:2406.16382, 2024.
Yinzhu Quan and Zefang Liu. Invagent: A large language model based multi-agent system for inventory management in supply chains. ar preprint arXiv:2407.11384, 2024.
Maarten Sap, Ronan LeBras, Daniel Fried, and Yejin Choi. Neural theory-of-mind? on the limits of social intelligence in large lms. arXiv preprint arXiv:2210.13312, 2022.
Johannes Schneider, Steffi Haag, and Leona Chandra Kruse. Negotiating with Ilms: Prompt hacks, skill gaps, and reasoning deficits. arxiv preprint ar arXo:2332..03720 2023.
Simone G Shamay-Isoory, Judith Aharon-Perrtz, and Daniella Perry.
Two systems for empathy: double dissociation between emotional and cognitive empathy in inferior frontal gyrus versus ventromedial prefrontal lesions. Brain, 132(3):617-627, 2009.
Xiao Shao, Weifu Jiang, Fei Zuo, and Mengqing Liu. Swarmbrain: Embodied agent for real- time strategy game starcraft ii via large language models. Xiviv preprint 17X11:2401.17749, 2024.
Eric Siegel. Predictive analytics: The power to predict who will click, buy, lie, or die. John Wiley & Sons, 2013.
David Silver, Julian Schrittwieser, Karen Simonyan, Toannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7666)3354-3333 2017.
James WA Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Ruto, Stefano Panzeri, Guido Manzi, et al. Testing theory of mind in large language models and humans. Nature Human Behaviour, PP. 1-11, 2024.
Reiji Suzuki and Takaya Arita. An evolutionary model of personality traits related to cooperative behavior using a large language model. Scientific Reports, 14(1):5989, 2024.
Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. Commonsenseqa 2.0: Exposing the limits of ai through gamification. arv preprint arXiv:2201.05320, 2022.
Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, and Zongqing Lu. Cradle: Empowering foundation agents towards general computer control, 2024. URL hnttps://arxiv..r wwtpxii.org/abb////0 200bb///333333666666
Xiangru Tang, Anni Zou, Zhuosheng Zhang, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning. Xiv preprint ar xrXo::2311.1053: 2023.
Amir Taubenfeld, Yaniv Dover, Roi Reichart, and Ariel Goldstein. Systematic biases in Ilm simulations of debates. arXiv preprint ar Xio:2402.004099 2024.
Oguzhan Topsakal and Jackson B Harper. Benchmarking large language model (Im) performance for game playing via tic-tac-toe. Electronics, 13631:133,, 2024.
CF Tsai, X Zhou, SS Liu, Li, M Yu, and H Mei. Can large language models play text games well. Current State-of-ther thethf-hhrrtttt and Open Questions, 2023.
Fons JR van Vijver and Madde E Willemsen. Abstract thinking. In Advances in psychology, volume 103, pp. 317-342. Elsevier, 1993.
15

---

Published as a conference paper at COLM 2024
Wiebe Van Der Hoek, Wojciech Jamroga, and Michael Wooldridge. A logic for strategic reasoning. In Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems, PP. 157-164, 2005.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):1-26, 2024.
Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Avalon's game of thoughts: Battle against deception through recursive contemplation. arXiv preprint arXio:2310.01320 2023a.
Zhenhailong Wang. Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Un- leashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaborationn arXiv preprint ar Xxxf:23007005300 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24877, 2022.
Alex Wilf, Sihyun Shawn Lee, Paul Pu Liang, and Louis-Philippe Morency. Think twice: Perspective-taking improves large language models' theory-of-mind capabilities. Xiv preprint arXi::2311.1022 2023.
Dekun Wu, Haochen Shi, Zhiyuan Sun, and Bang Liu. Deciphering digital detectives: Understanding I1m behaviors and capabilities in multi-agent mystery games. arviv preprint Xio:2312.007466, 2023.
Shuang Wu, Liwen Zhu, Tao Yang, Shiwei Xu, Qiang Fu, Yang Wei, and Haobo Fu. Enhance reasoning for large language models in the game werewolf. ar Xiv preprint ar xrXiv22402.02330, 2024a.
Zengqing Wu, Shuyuan Zheng, Qianying Liu, Xu Han, Brian Inhyuk Kwon, Makoto Onizuka, Shaojie Tang, Run Peng, and Chuan Xiao. Shall we talk: Exploring spontaneous collaborations of competing 11m agents. arXiv preprint ar XrXiv:2402.12327, 2024b.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model 2023.
based agents: A survey. ar Xix preprint XiX:::2309.0788
Tian Xia, Zhiwei He, Tong Ren, Yibo Miao, Zhuosheng Zhang, Yang Yang, and Rui Wang. Measuring bargaining abilities of llms: A benchmark and buyer-enhancement method. ar arxiv preprint ar Xio::2440 158833 2024.
Chengxing Xie, Canyu Chen, Feiran Jia, Ziyu Ye, Kai Shu, Adel Bibi, Ziniu Hu, Philip Torr, Bernard Ghanem, and Guohao Li. Can large language model agents simulate human trust behaviors? ar preprint arXio:2402.04555 2024.
Qianqian Xie, Weiguang Han, Yanzhao Lai, Min Peng, and Jimin Huang. The wall street neophyte: A zero-shot analysis of chatgpt over multimodal stock movement prediction challenges. arv preprint arXiv:2304.00 05351, 2023.
Fengli Xu, Jun Zhang, Chen Gao, Jie Feng, and Yong Li. Urban generative intelligence (ugi): A foundational platform for agents in embodied city environment. arXiv preprint ar dxXio:2312.11813, 2023a.
Hainiu Xu, Runcong Zhao, Lixing Zhu, Jinhua Du, and Yulan He. Opentom: A comprehen- sive benchmark for evaluating theory-of-mind reasoning capabilities of large language models. Xiv preprint arXiv:2402.0 26044, 2024a.
Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See-Kiong Ng, and Jíashi Feng. Magic: Investigation of large language model powered multi-agent in cognition, adaptability, rationality and collaboration. In ICLR 2024 Workshop on Large I Language Model (L.M) Agents, 2023b.
16

---

Published as a conference paper at COLM 2024
Xinrun Xu, Yuxin Wang, Chaoyi Xu, Ziluo Ding, Jiechuan Jiang, Zhiming Ding, and Börje F Karlsson. A survey on game playing agents and large models: Methods, applications, and challenges. arXiv preprint ar arXii:2403.10249, 20244, 2024b.
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. Xiv preprint ar AiXiv::309. Xit:2309.004688, 2023c.
Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXi::2310.18940, 2023d.
Diyi Yang, Caleb Ziems, William Held, Omar Shaikh, Michael 5 Bernstein, and John Mitchell. Social skill training with large language models. arXiv preprint ar arXio:2404.04204 2024.
Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, et al. Retroformer: Retrospective large language agents with policy gradient optimization. arviv preprint arXiv:2308.02157 2023.
Bin Zhang, Hangyu Mao, Jingqing Ruan, Ying Wen, Yang Li, Shao Zhang, Zhiwei Xu, Dapeng Li, Ziyue Li, Rui Zhao, et al. Controlling large language model-based agents for large-scale decision-making: An actor-critic approach. ar Xiv preprint ar 1rXi:22311.1384, 2023.
Tong Zhang, Chen Huang, Yang Deng, Hongru Liang, Jia Liu, Zujie Wen, Wenqiang Lei, and Tat-Seng Chua. Strength lies differences! towards effective non-collaborative dialogues via tailored strategy planning. ar Xiv preprint ar Xrfo:2203306666 2024a.
Wenqi Zhang, Ke Tang, Hai Wu, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tan, Peng Li, Yueting Zhuang, and Weiming Lu. Agent-pro: Learning to evolve via policy-level reflection and optimization. arXiv preprint ar arXio:2402.17574, 2024b.
Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Yan Xia, Man Lan, and Furu Wei. K-level reasoning with large language models. arXio preprint ar xxie:24402.0152, 2024c.
Yadong Zhang, Shaoguang Mao, Wenshan Wu, Yan Xia, Tao Ge, Man Lan, and Furu Wei. Enhancing language model rationality with bi-directional deliberation reasoning. arXiv preprint ar Xix::247.006112 2024d.
Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xie. Competeai: Understanding the competition behaviors in large language model-based agents. Xiv preprint ar xri::2210.17512, 2023.
Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis- Philipppe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. Sotopia: Interactive evaluation for social intelligence in language agents. arXiv preprint ar Xrio:2330.11667, 2023.
Xuhui Zhou, Zhe Su, Tiwalayo Eisape, Hyunwoo Kim, and Maarten Sap. Is this the real life? this just fantasy? the misleading success of simulating social interactions with 1lms. ar Xiv preprint ar Xth::2403.00020, 2024.
A Appendix
A.1 Cognitive Skills of Reasoning
Logical deduction refers to the capacity to derive conclusions from premises through the application of explicit logical rules(johnson-Laire adheres to the principles formal logic, including deductive and inductive reasoning. Deductive reasoning involves a process from the general to the specific, where conclusions are drawn based on universal truths. Inductive reasoning, conversely, is the process from the specific to the general, where general conclusions are inferred from specific observations.
1999). This mode of reasoning typically
17

---

Published as a conference paper at COLM 2024
Logical deduction necessitates the ability to identify and apply logical relationships, such as causality, equivalence, and contradiction.
Contextual intelligence denotes the ability to comprehend and interpret information within a specific context or background. It involves recognizing and interpreting the context, social norms, and implied meanings. This ability requires capturing subtle cues within given context and understanding the significance of conversations, events, or texts. Contextual Intelligence indispensable for language comprehension, empathetic resonance, and social interaction.
Predictive analytics refers to the capacity to forecast future events or trends based existing information(Sieen (Siegel 2013). This includes analyzing data, identifying patterns and trends, and utilizing this information to make informed predictions. Predictive ability demands the integration of past and present information, employing probabilistic and statistical methods, and reasoning about possible future scenarios.
Abstract thinking is the ability to understand concepts, principles, and models beyond con- crete and direct experiences de Vijver & Willemsen 1993). This type of thinking involves generalization, categorization,, and conceptualization capabilities, enabling individuals to identify similarities and differences across different situations and apply broad principles to solve problems. Abstract thinking is crucial for innovation, theoretical development, and complex problem-solving.
Cognitive empathy is the skill of understanding how others think and feel(Shamay-1sow Isoory et al. 2009). Cognitive empathy includes several aspects: 1. Perspective-taking Naturally seeing things from someone else's point of view; 2. Fantasy: The ability to connect with fictional characters as they were real; 3. Tactical empathy: Intentionally using perspective- taking to achieve specific goals; 4. Emotion regulation: The capability to empathize with others' emotions without becoming overwhelmed by them.
A.2 Symbolic System of Strategic Reasoning
A.2.1 Formulation of Strategic Reasoning Environment
Player (P,)
Observation
History (H)
Persona setting
Target alignment
Infostate (4,)
Functionality dedlaration
Action (A)
Communication maniiiiaaion
Reward (1t]
Reward (M)
Player (Pw)
Observation
Memory
Infostate (Ix)
Planning
Summarization
Action (4)
Figure 4: Environment in strategic reasoning of multi-agent systems (MAS).
We term strategic reasoning environment a 'GAME'. Formally, GAME is a tuple (N, A, H. 3, u, where:
. (1,2, ..., n) a set of n agents. the initialization of LLM based agent partic- ipant, system message is required for configuration, commonly including persona
18

---

Published as a conference paper at COLM 2024
setting, target alignment, and functionality declaration. The system messages passed to the LLM in the form of messages to influence the LLM's performance.
is a action set agents may take. It is a global set of state-independent actions; generally, only subset of legal actions are available to each player at each decision point. The actions for LLMs essentially refer to the LLMs textual response given dialogue histories prompts, In conversational environments, such as debate scenarios, any output from the LLM is considered as actions. Whereas, in scenarios with a defined finite action set, like Voting, Bidding, Poker, etc., the LLM's output needs to be parsed into a legitimate functional space.
is a set of game history. A historical record is a sequence of actions (including chance node "actions" or outcomes) taken from the start of the game. In environ- ments based on LLM, the historical information consists of the union of dialogue histories of all players.
E C H is set of terminal histories that each represent finished (completely played) game.
u 2 A S Ri", where A = Miinn Mmax] is utility (or payoff) function assigning each player payoff at the end of the game, and Umins umax are lower and upper bounds on those payoffs.
I is a set of information states. In general, I is partition of H such that each i E I contains histories. Decisions are made by players at these states. In LLM-based environments, each player's s infostate is the observable dialogue history along with their own action history and private information.
A.2.2 Target
In the realm of strategic reasoning, we explore an environment populated by multiple agents, each endowed with the capability engage in sophisticated reasoning processes. These agents navigate the environment with the objective of fulfilling their individual goals. Through the application of strategic reasoning, each agent assesses the potential outcomes of various actions, considering not only their own objectives but also the possible actions of other agents within the same environment. This intricate dance of decision-making and anticipation allows each agent to select the actions most likely to lead to the achievement of their goals.
Let (s) denote the expected utility for player i under strategy profile In strategic reasoning, a player aims to maximize their expected utility, which can be symbolically represented as:
max (si :)] S,CS;
where s-i represents the strategies of all other players except player i, and E[-] denotes the A expectation operator, based on player i's beliefs about the others' actions.
A.3 Strategic Reasoning with Large Language Models vs. Reinforcement Learning
In strategic reasoning, L.MSs and Reinforcement Learning (RL) represent distinct yet comple- mentary approaches. LLMs excel in generating coherent language and leveraging extensive knowledge, making them ideal for complex problem-solving that requires creativity and deep understanding, such as in business strategy formulation or geopolitical analysis. RL, in contrast, thrives on learning optimal actions through trial-and-error interactions with the environment, suited for scenarios demanding dynamic decision-making: like autonomous systems and game optimization. However, both methodologies face challenges: LLMs may inherit biases from their training data, while RL's effectiveness hinges on precise reward definitions and environmental modeling. The future might see synergistic models that blend LLMs' comprehensive knowledge handling with RL's adaptive decision-making, promising enhanced strategic reasoning across diverse and complex scenarios.
19

---

Published as a conference paper at COLM 2024
Criteria
Large Language Models
Reinforcement Learning
Knowledge Base
Extensive from diverse datasets
Acquired from specific environments Limited to specific state spaces Numerical, based on rewards
Contextual Understanding
Excellent in language-based tasks Abstract, human-liks like reasoning High with text explanations
Decision-making
Transparency
Low, often black box Tailored to specific tasks
Flexibility
Adaptable various scenarios
Generalization
Transfers knowledge well across domains
Limited to trained environments
Interactivity
Suitable for dialogue and negotiation Lower, uses pre-trained models Limited without updates
Optimizes actions in environments Higher, needs reward system design Excellent in dynamic environments
Implementation Complexity
Real-time Adaptation
Table 2: Comparison of Strategic Reasoning with Large Language Models vs. Reinforce- ment Learning
20


Reviews:
[{"review_id": "KFubU9WH3r", "reviewer": "Reviewer_ppn3", "summary": "This paper provides a survey regarding strategic reasoning with LLMs. The paper well motivated why LLMs are game changers for strategic reasoning and frame the existing works according to application, methods, and evaluation. Interesting challenges and insights are discussed at the end.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and valuable comments on our work.\n\n**Response to Question \"How could we define any proposed benchmark as unified and standard enough so that every related research should evaluate with?\"**\n\nWe believe a unified benchmark should embody three key characteristics:\n\n1. **Well-Controlled Environments:** The benchmark should operate within controlled environments with minimal randomness, facilitating reproducibility and ease of experimentation.\n2. **Reasonable (Objective or Subjective) Evaluation:** As discussed in section 5, a unified benchmark for strategic reasoning with Large Language Models (LLMs) should be capable of objectively or subjectively assessing their strategic reasoning abilities. This includes evaluating long-term planning capabilities and dynamic adaptability, among other facets of strategic reasoning.\n3. **Scalability Across Scenarios:** The benchmark should be adaptable to a broad spectrum of strategic reasoning scenarios, enabling researchers to explore diverse applications and challenges.\n\nAs mentioned in our paper, some benchmarks derived from classical game-theoretical problems for strategic reasoning lack higher-level abstraction, which limits their scalability and applicability to real-world applications. One feasible approach involves building upon existing RL-based environments, such as Robot Warehouse (RWARE)[1] and the StarCraft Multi-Agent Challenge [2]. These environments offer well-defined and controllable settings, laying a solid foundation for benchmarking strategic reasoning abilities. However, further refinement is necessary to ensure the benchmark meets the criteria of reasonable and comprehensive evaluation and scalability.\n\nBy leveraging established environments as starting points, we can enhance the benchmark's capacity to assess LLMs' strategic reasoning capabilities while fostering innovation and advancement in the field. Thank you for prompting this critical discussion, and we welcome any additional insights or suggestions you may have.\n\n[1] F. Christianos, L. Schäfer, and S. V. Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. 2020.\n\n[2] M. Samvelyan, T. Rashid, C. S. de Witt, G. Farquhar, N. Nardelli, T. G. J. Rudner, C.-M. Hung, P. H. S. Torr, J. Foerster, and S. Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043, 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Strategic reasoning is an important topic. Though many existing works (especially relying on LLMs) are published recently, their effort indeed widely spread in terms of topics and evaluation criteria. Thus, it becomes hard to follow and compare across papers. The survey paper calls out a unified framework and benchmark, which is crucial and necessary for the related research community. Researcher should align on the same standard.\n\n- The survey provides good angles to well unify most of the related works.", "reasons_to_reject": "- As a survey paper, the paper has limited novelty.\n\n- Some points on outlook section can go deeper with more interesting insights and designs among existing work.", "questions_to_authors": "I agree with the point that we should have a unified framework and benchmark. I am curious how could we define any proposed benchmark as unified and standard enough so that every related research should evaluate with?", "extra_scores": {}}, {"review_id": "vxXy2Swwzv", "reviewer": "Reviewer_5o6Z", "summary": "In this paper, the authors give a comprehensive review of recent work on using LLMs in strategic reasoning environments (e.g., gaming environments, economic simulation). This area of research has gained much traction in the last year (e.g., all of the papers they cite in Figures 2 and 4 involves work done exclusively in 2023 and 2024) and is integral to new applications of LLMs in neighboring disciplines such as economics, business and many others. I therefore think that a survey paper of this kind is very timely and important and I generally found their paper to be easy to read and accessible. \n\nAs with any broad survey article, certain liberties were taken with how to classify different work and concepts. For example, they exclude generative agents [Park et al] and work like this as being out of scope given that the lack of focus on goal-oriented task modeling (I’d like to see more discussion of this). They also attempted to quantify the different types of reasoning skills involved with strategic reasoning tasks versus other tasks and I found this part to be overly subjective without further discussion. Given that virtually all of this work involves prompt-based LLMs, distinguishing between “prompt-engineering” approaches versus other approaches, virtually all of which involve prompting, to be a bit confusing (in fairness, the authors point this out when they write that “it is important to note that the boundaries between the above methodological categories are not entirely orthogonal”.)\n\nGiven that this article is only a survey article and offers no new empirical or technical results, I don’t have much to criticize outside of some presentation-specific issues that I note below. Baring these concerns, my main concern is about whether a survey paper fits within the COLM technical track (I don’t think it does, I will say more below. I am willing to be convinced otherwise if other reviewers and the area chair(s) are not concerned about this).", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Regarding your main concern, \"**whether a survey paper fits within the COLM technical track,**\" we have inquired with the Program Chairs:\n\n> Q: As this is the first COLM, we seek your clear guidance on whether a survey paper is appropriate for the COLM track. … If the survey paper will naturally be rejected by COLM2024, we will withdraw the submission accordingly.\n\n> A: Please address this in your rebuttal. We don't have a specific policy about survey papers, and **they will be judged according to the contribution**, similar to conferences like ICLR/NeurIPS and ACL venues.   -Yoav\n\nAs noted by the PC, survey papers are also valued as a form of contribution by COLM'24, similar to other top-tier conferences like ACL, which even categorizes surveys as a distinct type of contribution on its submission site. One can find survey papers such as \"Reasoning with Language Model Prompting: A Survey\" accepted by prestigious venues like ACL/EMNLP/NAACL. Unlike well-defined reasoning tasks, Strategic Reasoning with LLMs remains unorganized and lacks systematic consolidation despite numerous related efforts. Our paper precisely provides a timely and comprehensive review, contributing to the community by standardizing research efforts in this field.\n\n**Response to \"What major challenges and opportunities exist in applying LMs to strategic reasoning scenarios, and which areas are receiving more or less attention in LLM research?\"**\n\nIn our paper, we highlighted three key benefits of utilizing LLMs in the introduction. Regarding the replacement of traditional methods, we acknowledge that LLMs have not yet reached a stage where they can outperform state-of-the-art RL models in domains such as poker or Go. However, LLMs have shown advantages in novel research areas like conversational games (e.g., Diplomacy), where traditional RL methods may fall short. LLMs have broadened the application scenarios of strategic reasoning, presenting new challenges and opportunities.\n\nWe aim to offer significant value through comprehensive synthesis and critical analysis of existing works, alongside our in-depth insights on this direction. To address your concern about the depth of discussion, we will expand the \"scenarios\" section and delve into specific challenges and opportunities across scenarios, explaining the motivation for using LLMs and assessing the prospects for success.\n\nThank you for your insightful suggestions. We look forward to further discussions with you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A comprehensive and very easy to read review of LLMs and strategic reasoning, which is motivated by the considerable interest in the topic in the last year. I think it provides a useful overview of this emerging area and that researchers will cite it for this reason.", "reasons_to_reject": "- No new technical or empirical results; not original research but a survey article. Given the call for papers (https://colmweb.org/cfp.html) and the reviewer guidelines (https://colmweb.org/ReviewGuide.html, in particular, the stated goal of having a “technical deep, exciting, forward-looking, insightful and impact program”), I don’t the paper meets the criterion for inclusion in COLM. \n\n- (a complicated criticism) While their review is comprehensive, it fails, in my mind, to communicate what the big problems are in the different application areas where strategic reasoning has been investigated. For example, in the scenarios involving economics or game theory, what are the big or open problems in these fields that motivate using language models in place of traditional tools? What are the prospects of success in using LLMs for these problems? Of these problems, are there particular sets of problems that the LLM field has tended to focused on, or ones that people are not paying attention to? \n\nA lot of the discussion along these lines is left vague (e.g., they end the “scenarios” section by saying: “each category [or application area of LLMs] offer[s]… unique insights and challenges”. I was really keen to get deeper into these challenges). I think it would benefit the LLM community to know a bit more about what the core problems are in these different to better motivate interest in strategic reasoning. It might also make the article easier for others outside of LLMs to engage with this work.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ue70EkCxNL", "reviewer": "Reviewer_3S3R", "summary": "This survey analyzes the problem definition, solutions, and evaluation of LLM strategic reasoning from various perspectives. It also provides future directions for research in the field of LLM strategic reasoning. I really like this paper and recommend it to be accepted.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for affirming our work and providing valuable suggestions regarding the existing issues.\n\n**Response to the Concern \"The lack of discussion on the applicable conditions of metrics and explanation for the selection of metrics\".**\n\nThe selection of these metrics is intricately linked to specific contextual scenarios. For instance, in survival games, metrics such as agent survival rates are pertinent, whereas, in games without elimination, metrics like win rates or rewards are more relevant. Additionally, different scenarios necessitate distinct focuses on the reasoning capabilities of the employed strategies.\n\nRegarding the rationale behind selecting these quantitative metrics, we aimed at two primary objectives:\n\n1. **Rationality Assessment of LLMs**: These metrics evaluate the rationality of LLMs, highlighting the disparity between LLMs and traditional methods like reinforcement learning. This comparison helps showcase the differences in rationality between LLMs and conventional approaches. Moreover, these metrics provide a benchmark for comparing the relative rationality of different methods.\n2. **Enhanced Transparency in Decision-Making:** As discussed in our introduction, LLMs aim to make decision-making more transparent and interpretable. Therefore, these metrics analyze the performance of LLMs in the process of strategic reasoning.\n\nWe greatly appreciate your additional references, which undoubtedly enhance the quality of our work.\n\nOnce again, thank you for your valuable feedback and constructive recommendations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is clearly written and easy to follow.\n2. The perspective proposed in this paper is excellent. Strategic reasoning is a crucial issue within LLMs, serving as an important dimension for evaluating LLM capabilities and a significant indicator for achieving AGI\n3. The classification basis for the method of strategic reasoning proposed in this article is very reasonable, and the literature review is conducted with great thoroughness.\n4. In the discussion of future work, the authors note that we still lack studies on the boundaries of strategic reasoning capabilities in large models, which is indeed a crucial point.", "reasons_to_reject": "1. A minor critique is the lack of discussion on the conditions under which quantitative and qualitative evaluation metrics are applicable when analyzing the evaluation, specifically why these particular metrics were chosen.", "questions_to_authors": "I do not have questions but I recommend adding more related work:\n[1] TimeArena: Shaping Efficient Multitasking Language Agents in a Time-Aware Simulation\n[2] AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation\n[3] SCIENCEWORLD: Is your Agent Smarter than a 5th Grader?", "extra_scores": {}}, {"review_id": "XwmvDPGrq9", "reviewer": "Reviewer_dKAH", "summary": "This paper provides a comprehensive survey about the field of strategic reasoning with LLMs.\nThe authors give a clear definition of strategic reasoning, and core characteristics compared to other reasoning types.\nThey categorize and systematically review the scenarios applying strategic reasoning with LLMs, existing methods enhancing the LLMs' capability for strategic reasoning, and the evaluation in strategic reasoning with LLMs.\nThe challenges of strategic reasoning with LLMs are discussed in the end.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and encouraging comments. We would like to take this opportunity to clarify the raised questions.\n\n**Response To Question \"How scenarios uniquely test the reasoning capabilities of LLMs?\"**\n\nWe acknowledge the necessity for a detailed discussion regarding specific tasks within each scenario, and correspondingly discussed the advantages of LLMs in various scenarios, such as Societal Simulation, where \"LLMs are utilized for the modeling and prediction of human behavior within intricate societal frameworks\", and \"LLMs are applied to understand and predict the outcomes of economic decisions, simulating scenarios like market competition, resource allocation, and investment strategies\" for Economic Simulation.\n\nThe further dicussion will cover the high-level capabilities of LLM required for different scenarios, such as planning, problem decomposition, and world modeling [1], and it will facilitate more accurate assessments of LLM strengths and limitations.\n\n**Response To Criticism \"The survey could benefit from a deeper analysis of the related works rather than just listing them.\"**\n\nWe appreciate your suggestion on a deeper analysis of the related works. We have made efforts in this regard. For instance, in Section 4 (Methods), we categorized the methods based on their underlying motivations and implementations. Additionally, we utilized a timeline (Figure 4) to delineate the developmental roadmap of these methods, providing readers with a clearer developmental trajectory.\n\n**Response To Concern \"It would be beneficial to include the authors' perspectives on practical approaches to realizing them\"**\n\nAs mentioned in 6.2, the establishment of a universal and standardized benchmark is highly needed. Such a benchmark should adhere to three principles: **Well-Controlled Environments**, **Reasonable Evaluation**, and **Scalability Across Scenarios**. From a practical standpoint, we consider Welfare Diplomacy [2] a good initiative. This game, which incorporates cooperative and deceptive elements typical of multi-agent systems, stems from classic diplomacy games [3]. However, there is a need for the development of more quantitative metrics to evaluate the strategic reasoning abilities in such scenarios.\n\n[1] Reasoning with Language Model is Planning with World Model.\n\n[2] Welfare diplomacy: Benchmarking language model cooperation.\n\n[3] Human-level play in the game of diplomacy by combining language models with strategic reasoning", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This is a timely survey for strategic reasoning with LLMs, which is clearly an important research topic.\n\n2. The survey is well-organized and clearly written. I especially like the formalism of the definition of strategic reasoning.\n\n3. A broad range of works related to strategic reasoning with LLMs are well placed.", "reasons_to_reject": "1. The categorization of scenarios only distinguishes topic differences and does not discuss how they uniquely test the reasoning capabilities of LLMs or how they present distinct features and challenges compared to other scenarios.\n\n2. In the first few sections, the survey could benefit from a deeper analysis of the related works rather than just listing them. It would be valuable to see a more detailed examination that connects different studies, evaluates their contributions critically, and outlines a developmental roadmap of the research area. This is essential for readers to grasp the progression and interplay of the works within the field.\n\n3. While the discussion and invitation for future contributions from the community is appreciated and well-timed, it would be beneficial to include the authors' perspectives on practical approaches to realizing them.", "questions_to_authors": "Typos: \n- Mind the space before citations.\n- public debatesCollectively", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper provides a survey on strategic reasoning with LLMs. Most reviewers appreciated the comprehensive and timely nature of the survey, recognizing its importance in the burgeoning field of LLM-based strategic reasoning. This paper received mixed reviews but after clarifying the suitability of survey papers in COLM, one reviewer raised their score and all reviewers reach a consensus on this work. The paper was commended for its clear organization, thorough literature review, and insightful future research directions. Concerns regarding deeper analysis of related work and more specific discussion of unique challenges and evaluation metrics within different scenarios seem to be mostly addressed in the rebuttal. Overall, the consensus leaned towards acceptance, emphasizing the paper's value in systematizing research efforts and contributing to the strategic reasoning community.
Decision: Accept