Suspicion Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4
Authors: Jiaxian Guo
Bo Yang
Paul Yoo
Bill Yuchen Lin
Yusuke Iwasawa
Yutaka Matsuo
Venue: COLM (2024)
Keywords: Imperfect Information Games, Large Language Models, Theory of Mind
TL;DR: LLMs for imperfect information games

Abstract: Unlike perfect information games, where all elements are known to every player, imperfect information games emulate the real-world complexities of decision-making under uncertain or incomplete information. 
GPT-4, the recent breakthrough in large language models (LLMs) trained on massive passive data, is notable for its knowledge retrieval and reasoning abilities. This paper delves into the applicability of GPT-4's learned knowledge for imperfect information games. 
To achieve this, we introduce \textbf{\agentname{}}, an innovative agent that leverages GPT-4's capabilities for imperfect information games. With proper prompt engineering to achieve different functions, \agentname{} based on GPT-4 demonstrates remarkable adaptability across a range of imperfect information card games. Importantly, GPT-4 displays a strong high-order theory of mind (ToM) capacity, meaning it can understand others and intentionally impact others' behavior. Leveraging this, we design a planning strategy that enables GPT-4 to competently play against different opponents, adapting its gameplay style as needed, while requiring only the game rules and descriptions of observations as input.
In the experiments, we qualitatively showcase the capabilities of \agentname{}  across three different imperfect information games and then quantitatively evaluate it in Leduc Hold'em. {As an exploration study, we show that \agentname{} can potentially outperform traditional algorithms without any specialized training or examples, but still cannot beat Nash-Equilibrium algorithms}. In order to encourage and foster deeper insights within the community, we make our game-related data publicly available.

Published as a conference paper at COLM 2024
Auseicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4
Jiaxian Guo +11
Bo Yang 11
Paul Yoo 1
Bill Yuchen Lin 2
Yusuke Iwasawa
Yutaka Matsuo
The University of Tokyo!
Allen Institute for Al 2
Abstract
Unlike perfect information games, where all elements are known to every player, imperfect information games emulate the real-world complexities of decision-making under uncertain or incomplete information. GPT-4, the recent breahthrough in large language models (LLMs)) trained on massive passive data, is notable for its knowledge retrieval and reasoning abilities. This paper delves into the applicability of GPT-4's learned knowledge for imperfect information games. To achieve this, we introduce Suspicion- Agent, an innovative agent that leverages GPT-4's capabilities for imperfect information games. With proper prompt engineering to achieve differ- ent functions, Suspicion- cuspicion-AAent based on GPT-4 demonstrates remarkable adaptability across a range of imperfect information card games. Impor- tantly, GPT-4 displays a strong high-order theory of mind (ToM) capacity, meaning it can understand others and intentionally impact others' behav- ior. Leveraging this, we design a planning strategy that enables GPT-4 to competently play against different opponents, adapting its gameplay style as needed, while requiring only the game rules and descriptions of observations as input. In the experiments, we qualitatively showcase the capabilities of Suspicion-Agen across three different imperfect information games and then quantitatively evaluate it in Leduc Hold'em. As an explo- ration study, we show that Suspicion-Agent traditional algorithms without any specialized training or examples, but still cannot beat Mash-Equilibrium algorithms. In order to encourage and foster deeper insights within the community, we make our game-related data publicly available.
can potentially outperform
1 Introduction
Recently, large language models (LLMs) (Brown et al., 2020b; Chowdhery et al., 2022; Touvron et al., 2023), which are trained extensively on text corpora and code datasets and aligned with instructions (Ouyang et al., 2022; Wei et al., 2021; Longpre et al., 2023), have demonstrated remarkable knowledge retrieval and reasoning capabilities (Kojima et al., 2022; Wei et al., 2022b;a) on natural language benchmarks and exams (Hendrycks et al., 2020; Cobbe et al., 2021). Given few-shot examples or specific instructions as prompts, these models, especially GPT-4 (OpenAI, 2023), can understand human intentions and make informed decisions in open-ended scenarios, and tackle intricate tasks by gathering observations and utilizing the learned prior knowledge, such as Voyager (Wang et al., 2023a), ReAct (Yao et al., 2022) and SwiftSage (Lin et al., 2023).
However, most of these methods typically assume that the agent has access to all relevant information, an assumption that is often unrealistic in real-world settings. Take diplomacy (Team et al., 2022; Gray et al., 2020) as an example: representatives must discern the veiled intentions of other countries based on incomplete information and decide accordingly to maximize benefits for their own nation. This challenge is not unique to diplomacy but extends to other domains as well, such as poker (Moravčík et al., 2017; Brown &
"Equal Contribution
I *Corresponding to jiaxian. siiiiiannguwwwblbbe u-toooo. jc. (Guo is now working at Google)

---

Published as a conference paper at COLM 2024
Sandholm, 2018) and economic simulations (Holmström & Myerson, 1983; Harsanyi, 1968). The inherent unpredictability in these games makes it impractical for a learned agent to adopt single, optimal strategy for every scenario (Brown et al., 2019). This necessitates predictive capabilities for handling incomplete information, along with a theory of mind (ToM) ability (Frith & Frith, 2005) to comprehend decisions from others' perspectives. Such complexities, both strategic and cognitive, represent challenges in the field of Al research.
Furthermore, recent advancements in imperfect information games, such ReBel (Brown et al., 2020a), DeepStack (Moravčík et al., 2017) and Libratus (Brown & Sandholm, 2018), typically start training from scratch, and thus they normally need millions of data to understand the game rules and learn the adequate strategies for each new game. Such a high sample complexity hampers their ability to generalize across different games and poses challenges when applying them into complex and open-ended imperfect information scenarios. By contrast, as alluded to previously, LLMS have undergone training on massive passive datasets. This leads to an intriguing proposition: Can we harness pre-trained LLMs' knowledge and reasoning capabilities to navigate imperfect information games without additional training data?
To achieve this, we propose Suspicion-Agent, an innovative autonomous agent based on GPT-4. This agent harnesses its extensive prior knowledge and cognitive adaptability to effective strategies against a range of adversaries without any specialized training. Con- cretely, we first decompose the process of solving such games into multiple sub-modules like observation interpreter and planning module to understand the game rules and game states (as Figure 1 shows) so that GPT-4 can make decisions accordingly. Each module employs different prompts to enable GPT-4 to fulfill specific functions. However, unlike perfect information games, planning strategies in imperfect information games can have varying effectiveness depending on the opponent's 's behavior (Brown et al., 2020a; Moravčík et al., 2017; Brown& & Sandholm, 2018). To tackle these challenges, we introduce a theory of mind (ToM) aware planning approach that leverages the higher-order ToM capability (Frith& & Frith, 2005) present in LLMs. Specifically, the model utilizes its understanding of human cognition to predict opponents' thought processes, susceptibilities, and actions. This aligns with the idea that individuals use their own minds as models to understand and affect others (Montes et al., 2022). For instance, the model might consider, "If execute Plan 1, how would this influence my opponent's 's beliefs about my cards, and what actions might they take based on their behavioral patterns?"
Concretely, given the gameplay history as the input, we find that GPT-4 can identify an opponent's 'ss strategic tendencies and analyze how our actions influence the opponent's behavior, egg. if Suspicion-Agent identifies a weak hand held by the opponent, coupled with the cautious strategy, it might strategically raise the bet to encourage the opponent to fold, even when Suspicion-Agent itself holds a similarly weak hand (as illustrated in Figure and A.11). Remarkably, by using some simple prompts, e.g., GPT-4 can even self- examine its behavior through the lens of the opponent (Refer to A.7). Leveraging its ToM capabilities, GPT-4 can predict and even influence an opponent's actions effectively (Röska- Hardy, 2008). Integrating these simulated actions into our planning module can mitigate the information asymmetry inherent in imperfect information games and more accurately assess the effectiveness of various strategies. As result, our AapiciionAAent can adjust its strategy to play effectively against a range of opponents, as shown in Section 4.1. In the experiments, we first conduct qualitative assessment of Suspicion-Agent' in 3 two-player imperfect information games, aiming to showcase the generalization capabilities of our method. Subsequently, we perform a quantitative analysis in Leduc Hold'em (Southey et al., 2012). The results reveal that Suspicion-Agen exhibits varying behaviors when interacting with previous works such as CFR+ (Tammelin, 2014) and NFSP (Heinrich & Silver, 2016) while outperforming some of them in terms of overall performance. In summary, our contributions are as follows:
1.
We introduce Suspicion-Agent, the first agent framework designed to empower GPT-4 with theory of mind (ToM) ability to compete in various imperfect information games by understanding game rules and observational data without requiring any specialized training or examples. By incorporating the ToM capability into the planning process, Suspicion-Agent captures the inherent uncertainty of opponent behavior in our strategic
2

---

Published as a conference paper at COLM 2024
GERMA SPotz
Obseraation
Interpatation
Slsaration
Fude
Play Hotery
All Game
Reflesion (LLM)
March Clock
Proporeds
(LLM)
and holding Jack pati can pat clips aeo the eppoaeed bus put
Flanl Boad eppaneet
The coproeent tends
(LUM]
card.
Canrent
Opponene
The
Plans:
AREES
Hiatory
Informations Prediction (LLM)
rits compattion (ve)
Auqpisiow-Agent
Lywlenawin
Opponent
Kaaning with First-Order Theory Mind
Figure 1: Left Figure. The illustration of the Suspicion-Agent when playing against an unknown opponent. Right Figure. The illustration about the first-order ToM planning method, where the texts in yellow blocks are outputs, and green blocks are inputs.
deliberations. This enables Suspicion-Agent to adapt its tactics dynamically when facing opponents with differing behavioral patterns.
We are the first to demonstrate that an agent based on GPT-4 can potentially outperform traditional algorithms in imperfect-informattio et al., 2012), when compared to established learning-basee methods like NFSP (Heinrich Silver, 2016) and DMC (Zha et al., 2021), still unperformed Vash-Equilibrinn algorithms like CFR+ (Tammelin, 2014) which may inspire more subsequent use of LLMs in imperfect- information games.
games, such as Ledue Hold'en (Southey
3. We make all interaction data between Suspicion-Agent and traditional algorithms for imperfect-informane
games in Leduc Hold' em publicly available. This will enable the research community to scrutinize the capabilities of GPTT-4based agents and inspire further work, particularly in fine-tuning smaller language models.
2 Problem Definition
Two-Player Imperfect Information Game In this paper, we propose to employ LLMs to play imperfect information games. As a preliminary exploration, we concentrate primarily two-player imperfect information games, such as Leduc Hold'em (Southey al., 2012), which involves two players, denoted by V = [1,2], who share the same action space, A. Let #1 E A and #2 E A represent the actions chosen by player 1 and player 2, respectively. Each player has access to two types of observations: private observation, denoted as Spri(1) where N the player index, and a public observation, shared among both players, denoted as Spub-
As the game progresses in discrete timesteps indexed by j, each player i observes history h of the game. This history comprises the series of public and private obser- vations and actions up to timestep j-1 and the result of game rí, formally given as pub/ pri(1)) af d a rf- Simultaneously, players re- ceive the current private and public observations, si pri(1) and Spub and select the next action
h (Spbb 50 (L)( af, a 10 ...,, si-1
of according to policy πΙ, All game histories are constructed as dataset D, denoted as D = (h, h,. where M indexes individual games. The goal of each player is to select the next action of with the imperfect observation according to the game rules, aiming for victory over many games. Specifically, the order of players is not fixed and depends on the game rule for each game. For example, the role of the small blind rotates among players in Texas Holed' em, dictating the order of play.
3 Method
To enable LLMs to play various imperfect information games without specialized training, we break down the overall task into several modules shown in Figure 1, such as the observa- tion interpreter, game pattern analysis, and planning module. In the following sections, will demonstrate how we craft specific prompts to guide LLMs to use its prior knowledge, reasoning ability, and psychological ability in performing these modular functions and
3

---

Published as a conference paper at COLM 2024
batender action van
keww ais iateriar and ane tas Dasad card the quer
koow hAs
Raxeel on ee cord, weeld Et wis aknintze the
eatinate Ais
de dxtsnate
kis cnliine
18
Planning
Mauning
Themyy of Mind
Planningg with Seeond-Cvder
Theory
Figure 2: Left Figure. The decision-making of the vanilla planning of Guspicion-Agent, Middle Figure. The decision-making of the planning with first-order ToM of Suspicion- Agent. Right Figure The decision-making of the planning with second ToM of Auspicion-Agent.
explain how we combine these functions to equip the model with the capability to navigate the intricacies of imperfect information games. Áll prompts and codes will be made public on our codebase (Please refer to our supplementary material).
3.1 Game Rule & Observation Understanding
While LLMs excel in processing text data, it can be misled in imperfect information games because they normally provide only brief, low-level descriptions. To mitigate this issue, we initially develop structured prompts that assist LLMss comprehending both the game's rules and its current state. For each type of imperfect information game, one can write structured rule description as follows:
General Rules: A brief game introduction, the number of rounds, and betting rules;
Action Descriptions: (Description of Action ( (Description of Action 2).
Single Win/Loss Rule: The conditions for winning, losing, or drawing in single game;
Win/Loss Payoff Rule: The rewards or penalties for winning or losing single game;
Whole Win/Loss Rule: The number of games and the overall win /loss conditions.
In most imperfect information game environments (Zha et al., 2019), game states are often represented as low-level numerical values, such one-hot vectors, to facilitate machine learning. Leveraging LLMs, we can convert these low-level game states into natural lan- guage text (Wu et al., 2023; Wang et al., 2023a; Guo al., 2022; Lin et al., 2023), thereby aiding the model's 's understanding. For each game, it is also essential to define an observation conversion rule. Similar structuring game rules, we organize the observation conversion rule as follows:
The type of inputs received, such as dictionaries, lists, or other formats, and describes the number of elements the game state along with the name of each element;
Input Explanation:
Element Descriptions: (Description of Element 1}, (Description of Element 2),
Conversion Tips: More guidelines for transforming the low-level game states into text. By leveraging both the game rule and the observation conversion rule, we can effi- ciently transform low-level game states into readable text, denoted as Obsy. This read- able text serves as the input for LLMs. Using the prompts Promplabs, the conditional distribution for each element Obs, il
in the generated text can be modeled as: Obs, ~ M Fe (Obs, [i]] Promptoss/ Rule, Rulcots Obs,[1,....
1]). Here, Fa represents the lan- guage model parameterized by 0; M the length of the generated text Obs,. The concrete definition can be found in Appendix A.2. We name this module an Observation Interpreter.
This formulation allows for more understandable interaction with the model in imperfect information games.
3.2 Vanilla Planning Module and Reflexion
After understanding the game rules and converting the game states into a readable format, we can craft prompts to guide LLMs in formulating strategies. Inspired by advancements in LLMs-agent and prompt prompteengineering (Ganguli et al., 2023; Wang et al., 2023e; Liu et al., 2023b; Shinn et al., 2023), we introduce a vanilla planning method which features a Reflexion module aimed at automatically scrutinizing game history to enable LLMs to learn and
4

---

Published as a conference paper at COLM 2024
improve planning from the experience of the history, as well separate planning module dedicated to making decisions accordingly.
Reflexion The Reflexion module takes input the history of games played against the current opponent and outputs a Reflexion. In the j-th round of the i-th game, we gather all prior game histories, denoted as Dí = (1,,12,..,, and prompt LLMs to carry out these analytical functions to get the Reflexion output Oj
I (o [ Reomp Rule, D', Oy 1]), ie,, O 0;~~ Fe Pefflexion which covers why we won or lost in specific previous games, and suggests how to improve strategies for future games. Importantly, the Reflexion module empowers Suspicion-Agent to enhance its strategies during gameplay, even without previous examples.
Planning After obtaining the Reflexion Oj, we proceed to use the game rules, the cur- rent game history H², the current readable observation Obs,, and the set of valid ac- tions {a} in the current game as inputs. We then prompt LLMs to formulate multi- ple textual plans based on its understanding of the imperfect information, i.e., Optan
M cally, the vanilla planning method assumes the marginal distribution of the actions of the opponent is uniform, and thus it can be regarded as special case of planning with the p plan
plam F Promppt Rule, Obs, W , Opan plan[[,...., iI,..,,i 1]), Opian ~ Fi Specifi-
zero-order ToM. In this way, we can further denote pplans as Fg
Evaluator To assess the likely success of each plan, we introduce an evaluation module. This module takes into account factors such as the game's 's current state, i.e., readable observation Obs, the Reflexion Oefftexion the game rule Rule and estimated plans Opian as the input, to estimate the win rates for each of the proposed plans and output the next action by prompting LLMs, i.e., the next action aj
paero- eval (Obs,, Reflexioon Rule, Optan).
3.3 Planning with Theory of Mind (ToM)
However, the vanilla planning method often struggles against the inherent uncertainties that typify imperfect information games, particularly when faced with opponents skilled at exploiting others' strategies. Inspired by this adaptability, we seek to devise new planning method that capitalizes on LLMs' ToM capabilities (Frith & Frith, 2005; Kosinski, 2023) to understand the opponent's 's behavior and thus can adjust the strategy accordingly. In the following sections, we will detail how we employ LLMs to analyze the behavior patterns of other agents and predict their subsequent actions in response to various plans using different orders of ToM (results are shown Table 3), thereby facilitating more informed decision-making. Note that all sample outputs are given in Section A.11 and A.7.
Planning with First-Order ToM Modelling: In the first-order ToM modeling approach (as Figure 2 shows), Suspicion-Agent goes step further by inferring the probable hand of the opponent based on their actions to that point, e.g,, the opponent raised, they likely have a strong hand. Consequently, Suspicion-Agent can adapt their strategy to maximize winnings when holding a strong hand and minimize losses with a weak hand. To forecast the opponent's ss actions, we first introduce behavior pattern analysis process. In this process, we feed the game history Di (h- ,12,...,, ......,,,-_)) and the game rules into LLMs, prompting it to analyze the opponent's s behavioral pattern. The formulation and the prompts can be expressed as: Opp
...
M io | [romppppptteen Rule, D', O i
Sample Prompts for First-Order Behaviour Pattern Analysis (Incomplete): From my perspective, please infer several beliefs about the opponent's game pattern/ppeference for each round when holding different cards and the public card (if have).
Through this approach, we can deduce the opponent's ''s behavior pattern. Notably, since the input for behavior pattern analysis is the same as that for the Reflexion module, we have integrated them into single module to reduce inference time, as shown in Figure 1. After identifying the opponent's behavior pattern, LLMs can be prompted to predict the strength of the opponent's current hand or observations in the game. This is expressed as:
5

---

Published as a conference paper at COLM 2024
O crrdd.pred 11).
I Fard Prompt pred, Rule, Obp Obs/, O card crrdd (1, ....,
Sample Prompts for First-Order Cards Prediction (Incomplete): Understanding the game rule, your observation, progress summarization in the current game, the estimated behaviour pattern of the opponent, and your knowledge about the game, please infer the probabilities about the cards of the opponent (number 100% in total) step by step.
With these predictions, we can further augment the previous Planning module and Evalu- ator module with Carrd pred as the additional input, so that we can further propose better
plans considering the opponent's card and estimate the winning rate of each plan, so that can better make the decision. Because the input of Planning module and Evaluator module are highly overlapped and our budgets are limited, we combine these two modules together save the costs:
Make Reasonable Plans: Please plan several stralegies according to actions you can play now to win the whole game step by step. Note that you can say something or keep silent to confuse your opponent.
Sample Prompts for Planning and Evaluator (Incomplete):
Potential opponent's actions and Estimate Winning/Lose/Dratt Rate: From the perspective of the opponent, please infer what the action opponent with probability would do when the opponent holds different cards based on his behaviour pattern, and then calculate the winning/lose/drrau rates when opponent holds different cards step by step. Output in a tree structure:
The sample outputs are shown in Figure and A.7.1.
Planning with Second-Order ToM Modelling: However, elite players in imperfect in- formation games like poker are also adept at dynamically adjusting their strategies, and they may employ "bluffing" weak one to deceive their opponent. Relying solely on a first-order ToM in such situations could lead to incorrect assumptions and potentially costly mistakes. Recognizing this, we introduce a planning method that incorporates a second-orde -odder ToM. In this enhanced model, Suspicion-Agent engages in even more intricate reasoning, where Suspicion-Aye -ggent not only considers what the opponent might do (as in first-order ToM) but also what the opponent believes Suspicion-Agent will do as Figure/ 2 shows. This level of strategic thinking allows Suspicion-Agent to gain an advantage in situations involving tactics like bluffing.
tactic, feigning strong hand when they actually hold
To implement this, Suspicion-Agemt needs not only just consider the current state from its own perspective, but also be capable of role-switching to think his own observation from the opponent's viewpoint. In traditional methods (De Weerd et al., 2013; Tatarchenko al., 2016), they need to iteratively call the first-order ToM function to estimate the action of the opponent. However, we surprisingly find that we can just add the prompts like below, and get the outputs in Sec A.7.2.
Sample Prompts for Second-Order Behaviour Pattern Analysis (Incomplete): From my
perspective, please infer under what circunstances is the opponent likely to be influenced by my actions? Additionally, in what sifuations would the opponent make decisions based solely on their own hand?
From the perspective of the opponent (he cannot observe my card but only action), please infer several beliefs about my game pattern/preference when holding different cards.
With this, LLMs are able to automatically generate insights into whether the opponent's behavior is likely to be reactive to AespicioonAgentt actions, or if they are likely to act independently. Then, we can directly reuse the prompts of the first-order Tom to predict the opponent's cards based on the behavior pattern estimated from the second-order ToM, and we can get sample results in Figure and Section A.7.2. In this way, we can utilize the Planning with second-ordes ToM to make decisions and adapt the strategies accordingly. The concrete algorithms are given in Section A.9. Without mentioning otherwise, we use the second-order ToM and GITT-4-0633 by default.
4 Experiments
We conduct experiments to answer the following questions:
6

---

Published as a conference paper at COLM 2024
Can Suspicion-Agent achieve comparable performance with traditional imperfect informa- tion algorithms without any specialized training? (Section 4.1)
Can Suspicion-Agent adapt its strategies when playing with different opponents? (Section
4.1)
Can Suspicion-Agent play different imperfect information games without any specialized training? (Section A.11.1)
How different orders of ToM improve the performance of Suspicion-Agent (Section 4.3
and A.8)
4.1
Quantitative Evaluation
To quantitatively assess the performance of LLMs in imperfect information games, we chose the RLCard environment (Zha et al., 2019). Due budget limits, our quantitative evaluation focuses on Ledue Hold' em, a simplified version of Limit Texas Hold'em. The game rules of Leduc Hold'em can be found in Appendix A.3. Following (Southey et al., 2012), we also add the opponent's observation into the single game history h after the end of each game, which also conforms with the real-world experience, but we also perform the ablation study about it in Section 4.3 and A.10.
Environments
Competing Methods We have selected a range of methods commonly used in decision- making, such as NESP (Heinrich & Silver, 2016), DQN (Mnih et al., 2015), DMC (Deep Monte Carlo Search for imperfect information games) (Zha et al., 2021) and CFR+ (Zinkevich et al., 2007). Among these, NFSP and DMC specifically designed for imperfect information games and are based on self-play, while CFR+ is grounded in game theory. These algorithms typically show different strategies in the imperfect information games, allowing us to evaluate the adaptability of each method. Note that, our Suspicion-Agent does not have any specialized training when compared with these methods.
Evaluation Methods To ensure the robustness of our evaluation metrics, we meticulously designed dual-method evaluation framework aimed at mitigating the randomness intrinsic to imperfect information games. (1) Variable Random Seeds: Suspicion-Agen play against
different baselines for 100 games utilizing varying random seeds for each game. This tactic is intended to dampen the stochastic variability introduced by the random seed settings. The results are shown in Table 1. (2) Same Cards with Exchange Position: We ran series of 50 games with fixed random seed, thereby keeping the sequence of cards constant across these games. Suspicion-Agen initially played at position 0 for the first 50 games, then we rerun the 50 games but switched the position of Suspicion-Agent and the baseline model. In this way, duspicion-Agent and the baseline should have the same card strength over 100 games, and thus we can better evaluate the performance of each. The results of these experiments are presented in Table 2.
Results Analysis (1) Suspicion-Agent outperforms most baselines: As illustrated in Table 1, it is evident that our GPT-4-based Suspicion-Agent outperforms most other algo- rithms specifically trained on Leduc Hold'em em environments except CFR+. Notably, it not only defeats most of these methods but also secures the highest average chip count in the comparisons. Our approach surpasses the second-best method by an impressive margin of approximately 25%. Even if the current design cannot achieve Jash-Equilibrium these findings compellingly showcase the potential of employing large language models in the realm of imperfect information games, as well as affirm the effectiveness of our proposed framework. (2) The gap between GPT-3.5 and GPT-4 is large: While GPT-4 delivers perfor- mance that either matches outperforms other baselines, agents using GPT-3.5 experience a significant drop in performance. Specifically, the winning probability for agents built on GPT-3.5 stands at just 50%, as opposed to 100% for GPT-4-based agents. Additionally, the average chip payoff for GPT-3.5 agents is negative, underlining the stark performance disparity between the two versions of the language model. The further reason analysis can be found in Appendix A.4. (3) Suspicion-Agent outperforms baselines in both positions: Utilizing identical card sequences for both positions, Suspicion-Ageni exhibits consistent winning pattern against various baselines, as evidenced in Table 2. This robust performance serves as compelling evidence to substantiate the claim that Suspicion-Agent outperforms the baseline models when card strength is held constant.
7

---

Published as a conference paper at COLM 2024
Table 1: The comparison results of Suspicion-Agent when playing with different algorithms trained on Leduc Hold'em em environments. The results are the win/lose chips after 100 games with different seeds, and the number of win/lose chips ranges from 1 to 14.
Opponent Model
Ours
Ours (GPT-4
NFSP
DQN
DMC
CFR+
(GPT-3.5)
Avg.
NESP
-33
-22
-21
-3
-142
66.25
DQN
+33
-55
-8
+200
-4
+22.8
DMC
+22
+55
-15
-49
-24
+4
CHR+
+21
+8
+15
+126
+22
+38.4
Ours (GPT-3.5)
+3
-200
+49
-126
-55
Ours (GPT-4)
+142
+45
+24
-22
+47.25
Table 2: The comparison results of Suspicion-Agent when playing with CFR+ and DMC trained in Leduc Hold' em environments. These results are quantified over 50 games, and pos denote the position of the opponent model. For example, CFR+ (pos 0) denotes the opponent model is in the position 0, and the model is located in the position 1.
Opponent Model
CFR+
CFR+
DMC
DMC
Ours
Ours
(pos 0)
(pos 1)
(pos 0)
(pos 1)
(pos 0) (pos
Avg.
DMC CFR+
-21
-6
-10
+10
-36
4
-1.17
+35
-35
+6
+21
+16
-1
+7
Ours
+1
-16
+11
+36
48
Behaviour Pattern Analysis We illustrate the action percentages of Suspicion-Agent and baselines in Figure 3. We can observe that (1) Suspicion-Agent vs CFR+: CFR+ algorithm (Zinkevich et al., 2007) exhibits a mixed strategy, characterized by a conservative approach where it tends to fold in the face of weak hand, especially when this hand does not align well with the public cards. Our agent, recognizing this pattern, strategically opts to raise more frequently. This tactic effectively applies pressure the CFR+ algorithm, increasing the likelihood of it folding under uncertain conditions. However, Over-reliance on bluffing can lead to significant losses, especially when facing opponents capable of recognizing and exploiting this pattern. Additionally, it's noteworthy that CFR+ itself is not averse to bluffing- Although it typically plays conservatively, CFR+ does occasionally employ bluffing tactics, which can result in losses for our agent. (2) Suspicion-Agent vs DMC: DMC algorithm (Zha et al., 2021) based on the search algorithm DMC employs a more diversified strategy that includes bluffing. It often raises both when it has the weakest and the strongest hands. In response, Suspicion-Agent adapts by raising less frequently and opting call or fold more often based on its own hand and the observed behavior of DMC. (3) Suspicion- Agent vs DQN: DQN appears to have a more aggressive stance, almost always raising with strong mid-level hands and never folding. Suspicion-Agent identifies this and, in turn, minimizes its own raises (the lowest percentage among all matchups), opting more often to call or fold based on the public cards and DQN's actions. (4) Suspicion-Agent vs NFSP: NESP exhibits a follow-through strategy, opting to always call and never fold. Suspicion-Agent responds to this by raising less frequently (compared to matches against CFR+) and choosing to call more (compared to matches against CFR+) based on the public card and NFSP's observed actions. The analysis clearly shows that Suspicion-Agen is highly adaptable and capable of exploiting the weaknesses in the strategies employed by various other algorithms. This speaks volumes about the large language model's capability to reason and adapt in imperfect information games.
4.2 Qualitative Evaluation
In the qualitative evaluation, we assess Suspicion-Agent on three imperfect information games: Coup, Texas Hold'en e'm Limit, and Leduc Hold'em em (Southey et al., 2012). For each game, we provide only the rules and observation rules as described in Section 3.1. Im- portantly, Suspicion-Agent is able to play these games without any additional training or sampling. Qualitative examples from these games are presented in the subsequent sections.
Leduc Hold'em em We present qualitative samples showcasing Suspicion-Agent behaviour under different strategies: Vanilla Planning, Planning with First-Order ToM, and Planning
8

---

Published as a conference paper at COLM 2024
Action fercentage when Paylng
GFR NTSF
GFR
N757
Chick
9.41
Clook
Actan Calagoyy
Adllon Canagery
Figure 3: Left Figure. The illustration of the action percentage of Suspicion-Agent when playing against different algorithms. Right Figure. The illustration of the action percentage of different algorithms when playing against Suspicion-Agent.
actions Call Helee Dork (peen)
Saatin this mond, you are tolding Searts, Tar, SEX NOX pat chips onto pUl, stale you DERWWEG EYC
Batter seessayl cardel foceday chose Teall"
iccerding betavcor panters, ahee
tolae maeee lack,
call
more manederr, Liiely sold gamm
condday' corrast telioves your carde: thet
forst rusd yet.
Accending teadaay
Iaten,
arr Lanes cing 130 1ii, daess 120 Ihi, 2ed (3b.000
with
So, prendottaties
fasse
faudaa talde isey arobabitatel, will toid 1200% prohenatieel, wall sin the pat, sbuce wall
chis fram me. 2 froe bestiayl The surning rale is 50% the touiin rate is ih, and the than rate is
2: Foll
hatde bupes isey aroubsiatell will check 1200% propubatatyl, vott tas sill chiss frem ou, cooddapl. somsirg rate th, the leaing rate 50% and the dras nace
21
which
chis fram ms. 2 froe beadiart The surning rale is th, the toxinn rabe is 30h and the ther rate
I.
21 Saudiay halde tuzes isek anouuaitatel, check 1100% cropubatatell gana procued
sucerd nong casung rate, will checs srsittiin the
canne
this stags.
Saxaay valls 19en
procect the
ropectio cripe un
chils catos
Gatn chuse 58V 2 beauctue chize cast carnet decerns this unage.
piue Selectien:
Figure 4: The qualitative sample of planning with second-order ToM Suspicion-Agent about Strategic Bluffing on Ledue Hold'em em.. More samples are given in Appendix A.11.
with Second ToM in Leduc Ho.d'em. These samples can be viewed in Figure 6, 2, 8, and 9, respectively, and the concrete analysis is given in Appendix A.11.1.
Game Coup and Texas Hold'em Limit As illustrated in Figure 12 and 11 in Appendix, when provided solely with the rules and observation guidelines of Texas Hold'em Limit and Coup, and keeping the agent prompts consistent, Suspicion-Agentic still adept at discerning the opponent's game patterns. It analyzes the strength of its hand and subsequently makes informed decisions to accumulate chips. This is the strong evidence to demonstrate the generalization ability of Suspicion-Agent based on GPT-4 on different imperfect information games, and thus outperforming the algorithms need to be re-trained for every new imperfect information game. Specifically, In the game Coup, without any prior training, Suspicion- Agent rapidly discerns which character the opponent lacks. It then strategically bluffs as that character to block the opponent's actions. This ability to bluff successfully gives Suspicion-Agent consistent advantage throughout multiple rounds.
4.3 Ablation Study and Component Analysis
Ablation Study on Orders of ToM In order to explore how different orders of ToM- aware planning methods affect the behavior of the large language model. We perform the experiments and comparisons on Ledue Hold' em and play against CFR+. We illustrate the action percentages of Suspicion-Agent with planning with different levels of ToM in

---

Published as a conference paper at COLM 2024
Figure 5 and the chips gain results in Table 3. We can observe that (1) Vanilla Planning: Based on Reflexion module, vanilla planning tends to call and check more (the highest call and check percentage when playing against both CFR+ and DMC) during the game, which cannot push the pressure to make the opponent fold and result in many unnecessary losses. In this way, vanilla planning has the lowest chip gain as Table 3 shows. (2) Planning with First-Order ToM: Utilizing First-Order ToM, Suspicion-Agent is capable of making decisions based on its own and estimates of the opponent's 'ss card strength. As a result, it will raise more than vanilla planning but it tends to fold more frequently than other strategies, aiming to minimize unnecessary losses. However, this cautious approach can be exploited by savvy opponent models. For example, DMC often raises when holding the weakest hand, and CFR+ may occasionally raise even with a mid-level hand to exert pressure on Suspicion-Agent. In these instances, Suspicion-Agent (3) Planning with Second-Order ToM: In contrast, Suspicion-Agent excels at identifying and capitalizing on the behavioural patterns of opponent models. Specifically, when DMC will raise
tendency to fold can lead to losses.
checks suggesting its hand doesn't 't align with the public cards SondddSSspiciin--ee as bluff to induce folds from the opponents. As result, Suspicion-Agent exhibits the highest raise rate among the three planning methods evaluated. This aggressive strategy allows Suspicion-Agen to accumulate more chips even when holding weak hand, thereby maximizing its chip gains.
Table 3: The comparison re- sults of Suspicion-Agert when playing with different levels of ToM against CFR+ on Leduc Hold'em environments. The results are quantified after 100 games.
Table 4: The comparison results indicate the impact of including opponent observations in the game history in Ledue Hold' em environments. These results are quantified over 50 games, and pos denote the position of the opponent model.
Opponent Model
CFR+
DMC
Win
Avg Probability
Opponent Model
(pos 1)
(pos 1)
CFR
DMC
DMC CFR+
-6
+10
+2
50%
Ours (vanilla plan)
-89
-72
-35
+21
-14
50%
Ours (w/ First ToM)
-67
-26
Ours (w/o hind_obs)
-45
+18
+17
100%
Ours (w/ Second ToM)
-22
+24
Ours (w/ hind_obs)
-16
+36
+36.5
100%
Ablation Study on the Effect of Hindsight Observation
Following (Southey et al., 2012),
we assume that Suspicion-Agent has access to observations of the opponent after the end of each game, i.e., Hindsight Observation. To assess the impact of it, we conduct an ablation study in which hindsight observations are not incorporated into the current game. Without hindsight observations, we augment the Reflexion module with additional prompts to enable it to infer the opponent's ' cards based on game outcomes and imperfect observations. As demonstrated in Table 5 and 4, Suspicion-Agent retains its performance advantage over the baseline methods without the benefit of hindsight observations. Specifically, we observe that Suspicion-Agent adopts more conservative strategy under the increased uncertainty that comes without hindsight observations. This leads to reduced bluffing, resulting in fewer gains when playing against CFR+. However, it also minimizes the risk of over-bluffing when facing DMC, thus yielding higher chip earnings.
5 Conclusion
In this paper, we introduce Duspicion-Agent the first prompting system designed to en- able large language models to engage in various imperfect information games using only the game rules and observations for interpretation. By incorporating first-order ToM and second-order ToM capabilities, we show that a GPT-4-based Suspicion-Agen can outper- form traditional algorithms such as NESP, even without specialized training or examples. Additionally, we identify and discuss the current limitations of utilizing LLMs in the con- text of imperfect information games. We make all our code and interactive data publicly available to the research community. This will help in better understanding the capabilities of large language models, particularly GPT-4, and we hope our data will encourage the pa development of more efficient models for imperfect information games. In addition, we also present the limitations of Suspicion-Agent in Appendix A.5.
10

---

Published as a conference paper at COLM 2024
References
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, al. Phi-3 technical report: A highly capable language model locally on your phone. ar preprint arXiv:2404.14219, 2024.
Elif Akata, Lion Schulz, Julian Coda-Forno, Scong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models, 2023.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm technical report. Xiv preprint ar arXtt:2305.104m0 104003 2023.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multi- modal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXio preprint (rXio:2302.04023, 2023.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, al. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint MiXiv:2308.09663 2023.
Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418 424, 2018.
Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365 (6456):885-89) 890, 2019.
Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret minimization. In International conference on machine learning, pp- 793-802. PMLR, 2019.
Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong. Combining deep reinforce- ment learning and search for imperfect-iformation games. Advances in Neural Information Processing Systems, 33:17057-17069, 2020a.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020b.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating multi-agent col- laboration and exploring emergent behaviors in agents. arXiv preprint 1088842308.10888, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint ar 22X4:22204.02311, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchro Nakano, et al. Training verifiers to solve math word problems. Xiv preprint S1.i:::1100141666 2021.
Vincent P. Crawford. "fatal attraction" and level-k thinking in games with non-neutral frames. Journal of Economic Behavior and Organization, 156:219-224, 2018. ISSN 0167-2681. doi: https://daooooggg//l00l////22200 ccinnee/arrtceee/ii//99622211111
2018.10..08. URL sctps://wwee http::://wwwww.eeeeeeeeeeeeeeeee com/
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
11

---

Published as a conference paper at COLM 2024
Martin Davies. The mental simulation debate. Philosophical Issues, 5:189-218, 228, 1994.
Harmen De Weerd, Rineke Verbrugge, and Bart Verheij. How much does help to know what she knows you know? agent-based simulation study. Artificial Intelligence, 199: 67-92, 2013.
Jinhao Duan, Remming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel-Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. Gtbench: Uncovering the strategic reasoning limitations of Ilms via game-theoretic evaluations. arXiv preprint arXii:2402.12348, 2024.
Gabriele Farina, Christian Kroer, Noam Brown, and Tuomas Sandholm. Stable-predictive optimistic counterfactual regret minimization. In ICML, 2019a. URL https://arxiv.orrr abs/1902. 04982.
Gabriele Farina, Christian Kroer, and Tuomas Sandholm. Optimistic Regret Minimization for Extensive-Foris Games via Dilated Distanc-Geeneattin Functions. Curran Associates Inc., Red Hook, NY, USA, 2019b.
lan Frank and David Basin. theoretical and empirical investigation search in imperfect information games. Theoretical Computer Science, 252((-2):217-256, 2001.
Chris Frith and Uta Frith. Theory of mind. Current biology, 15(17):R644-R65) 2005.
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilé Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The ca- pacity for moral self-correction in large language models. arXiv preprint arXio:2302.07459, 2023.
Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-Chun Zhu, Demetri Terzopoulos, Li Fei-Fei, and Jianfeng Gao. Mindagent: Emergent gaming interaction, 2023.
Jonathan Gray, Adam Lerer, Anton Bakhtin, and Noam Brown. Human-level performance in no-press diplomacy via equilibrium search. arXiv preprint arXiv:2010.02923, 2020.
Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven CH Hoi. From images to textual prompts: Zero-shot vqa with frozen large language models. arv preprint ar Xix:22212.00844 2022.
Akshat Gupta. Are chatgpt and gpt-4 good poker players? pre-flop analysis, 2023.
John C Harsanyi. Games with incomplete information played by "bayesian" players part ii. bayesian equilibrium points. Management Science, 14(5):320-334, 1968.
Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfecttiinfommation games. arXiv preprint "XXv:1603.00122) 2016.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint ar orXii:2009.033k0 2020.
Bengt Holmström and Roger Myerson. Efficient and durable decision rules with in- complete information. Econometrica: Journal of the Econometric Sociely, pp. 1799-1819, 1983.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. Xiv preprint arXiv:2212.104403, 2022.
Susan Hurley. The shared circuits model (scm): How control, mirroring, and simulation can enable imitation, deliberation, and mindreading. Behavioral and brain sciences, (1(1:11-2, 2008.
12

---

Published as a conference paper at COLM 2024
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. Xiv preprint ar 2rXi::2308.17491, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systemss, 35:1:9992222221922222221 22213,,, 2022.
Michal Kosinski. Theory of mind may have spontaneously emerged in large language models. Xrxiv preprint ar arXiv:2302.02083, 2023.
David M Kreps and Robert Wilson Reputation and imperfect information. Journal of economic theory, 27(2):253 279, 1982.
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Prithviraj Ammanabrolu, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. ArXiv, abs 2305.1799, 2023.
URL https://appp wwwanticscholaaa. orgg/CoppuSDD:200. 2589660133
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Xiv preprint 03::::334008888, 2023a.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms agents. Xiv preprint ar Xiv:2308.0000 00088,, 2023b.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint ar arXi::2301.1363 2023.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang. Ying Nian Wu, Song-Chnn Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. ar Xiv preprint ar Xrv::23044.098 09844, 2023a.
Runyu Lu, Yuanheng Zhu, and Dongbin Zhao. Score-based equilibrium learning in multi- player finite games with imperfect information, 2023b.
Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. Sources of hallucination by large language models on inference tasks. arv preprint arXiv:2305.14557 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.
C. Models. Model card and evaluations for claude models. aattt/ppcc.cmm//ottttttooommanai
https://www-
Soooel-Card-Claude2.ppe 2023.
Nieves Montes, Nardine Osman, and Carles Sierra. Combining theory of mind and abduc- tion for cooperation under imperfect information. In European Conference on Multi-Agent Systems, pp. 294-311. Springer, 2022.
Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337):08-5513 2017.
Y. Nakajima. Babyagi. https://github. comijyheennkajjnn/nubngggi 2023.
Shaun Nichols and Stephen P Stich. Mindreading: An integrated account of pretence, self-awareness, and understanding other minds. 2003.
13

---

Published as a conference paper at COLM 2024
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, C Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
Chong
Xin Ouyang and Ting Zhou. Imperfect-information game ai agent based on reinforcement learning using tree search and deep neural network. Electronics, 12(11), 2023. ISSN 2079-9292, doi: 10.3390/ electromics12112453. URL https: http::/wwww..dpttt com/ wwt.c/oom///////// -079-9292/ 12/11/2453.
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis, 2023.
David Premack and Guy Woodruff. Does the chimpanzee have theory of mind? Behavioral and Brain Sciences, (4):515-526,, 1978. doi: 0010017/014055550006622
Louise Räska-Hardy. Theory theory (simulation theory, theory of mind). 2008.
Tim Roughgarden. Twenty lectures on algorithmic game theory. Cambridge University Press, 2016.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.0476) 2023.
Dale Schuurmans. Memory augmented large language models are computationally univer- sal. arxiv preprint ar arXio:2301.04589, 2023.
Bilgehan Sel, Ahmad Al-Tawaha, Vanshaj Khattar, Lu Wang, Ruoxi Jia, and Ming Jin. Algorithm of thoughts: Enhancing exploration of ideas in large language models. arv preprint xrXiv:2308.103: 10379, 2023.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint ar arXio:2303.1136 2023.
Samuel Sokota, Ryan D'Orazio, Chun Kai Ling, David Wu, Zico Kolter, and Noam Brown. Abstracting imperfect information away from two-player zero-sum games, 2023.
Finnegan Southey, Michael P Bowling, Bryce Larson, Carmelo Piccione, Neil Burch, Darse Billings, and Chris Rayner. Bayes' bluff: Opponent modelling in poker. arXiv preprint arXiv:1207.141, 2012.
Oskari Tammelin. Solving large imperfect information games using cfr+. arXiv preprint arXiv:1407.5044 2014.
Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Multi-view 3d models from single images with a convolutional network. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII 14, pp. 322 322-337.50 Springer, 2016.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXio preprint arXio:2403.08295, 2024.
Meta Fundamental AT Research Diplomacy Team, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 37866624)1066-1177, 2022.
Meta Llama Team. Introducing meta llama 3: The most capable openly available Ilm to date. https://aii meta. com/bloge wwwg:mottgaoooeeoeeeeeeeeaaaaaaa m 11ama-1.ama------- 2024.
14

---

Published as a conference paper at COLM 2024
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint (XXi::2377.09288, 2023.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Xiv preprini afX::2305511221, 2023a.
Ajay
Hongru Wang, Huimin Wang, Lingzhi Wang, Minda Hu, Rui Wang, Boyang Xue, Hongyuan Lu, Fei Mi, and Kam-Fai Wong. Tpe: Towards better compositional reasoning over conceptual tools with multi-persona collaboration, 2023b.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. A survey on large language model-based autonomous agents, 2023c.
Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Avalon's game of thoughts: Battle against deception through recursive contemplation. arXiv preprint arXiv:2310.01320, 2023d.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. Xiv preprint ar Xiv:23000 016600 2023e.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. aiv preprint ar arXi::2109.01633 2021.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. ar preprint (Xii::2006006883 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. thain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24833 2022b.
Yue Wu, So Yeon Min, Shrimai Prabhumoye, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom Mitchell, and Yuanzhi Li. Spring: Gpt-4 out-performs rl algorithms by studying papers and reasoning. arXiv preprint 3xXx:23055.1586 2023.
Michael Wunder, Michael Kaisers, John Robert Yaros, and Michael Littman. Using iter- ated reasoning to predict opponent strategies. In The 10th International Conference on Autonomous Agents and Multiagent Systems Volume 2, AAMAS '11, pp. 593-600, Richland, SC, 2011. International Foundation for Autonomous Agents and Multiagent Systems.
ISBN 0982657161.
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study werewolf, 2023a.
Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Wu. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.18940, 2023b.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and
Yuan Cao. React: Synergizing reasoning and acting in language models. arviv preprint arXiv:2210.03629, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arv preprint ar Xiv:
2023.
Daochen Zha, Kwei-Herng Lai, Yuanpu Cao, Songyi Huang, Ruzhe Wei, Junyu Guo, and Xia Hu. Rlcard: A toolkit for reinforcement learning in card games. ar Xiviv preprint 1rX1::1910.0437, 2019.
15

---

Published as a conference paper at COLM 2024
Daochen Zha, Jingru Xie, Wenye Ma, Sheng Zhang, Xiangru Lian, Xia Hu, and Ji Liu. Douzero: Mastering doudizhu with self-play deep reinforcement learning. international conference machine learning, Pp. 12333-12344 PMLR, 2021.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023.
Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. Advances in neural information processing systemss, 20, 2007.
16

---

Published as a conference paper at COLM 2024
A Appendix
A.1 Related Works
Imperfect Information Game Imperfect information games, like poker, have garnered significant research attention due to the lack of complete information about player states and actions (Brown & Sandholm, 2018; 2019; Moravčík et al., 2017; Southey et al., 2012; Frank & Basin, 2001). Unlike perfect information games, these settings enable strategies involving deception and stochastic approaches (Montes et al., 2022; Kreps & Wilson, 1982). Previous research has explored various aspects, including game theory (e.g., CFR, CFR+) (Lu al., 2023b; Zinkevich et al., 2007; Tammelin, 2014; Farina et al., 20)9b;a), reinforcement learning (Ouyang & Zhou, 2023; Roughgarden, 2016), deep reinforcement learning (e.g,, NESP, DQN, DMC) (Brown et al., 2020a; Heinrich & Silver, 2016; Mnih et al., 2015; Zha et al., 2021), and abstraction techniques (Sokota et al., 2023). However, these methods often require extensive computation and data collection. Recent studies have applied large language models (LLMs) to imperfect information games (Gupta, 2023), showing strong performance in predicting player behaviors and simplifying natural language interactions (Akata et al., 2023; Xu et al., 2023a). GTBench (Duan et al., 2024) evaluates LLM performance in these games, while our work introduces a tailored prompt system for GPT-4, which outperforms traditional algorithms like DMC and NFSP without specialized training. The zero-shot capabilities of LLMs eliminate the need for pre-training and extensive data, offering distinct advantage over traditional methods.
Reasoning and Planning of LLMs LLMs have recently demonstrated exceptional abilities in reasoning and planning across various tasks (Brown et al., 2020b; Chowdhery et al., 2022; Touvron et al., 2023), utilizing evidence, arguments, and logic to make informed judgments (Huang & Chang. 2022). The Chain-of-Thought (CoT) approach, which involves generating intermediate reasoning steps, has significantly improved performance in reasoning tasks (Wei et al., 2022b), while the zero-shot capabilities of LLMs have been enhanced by simple prompt phrases (Kojima et al., 2022). The Treouhhf (ToT) framework further extends CoT by enabling exploration over text units as intermediary steps toward problem-solving (Yao et al., 2023). Inspired by developments like Voyager, BabyAGI, and ReAct (Wang et al., 2023a; Nakajima, 2023; Yao et al., 2022), we propose that the reasoning and planning abilities of LLMs can support agents in imperfect information games using only game rules and observations. With well-crafted prompts, LLM-based agents can autonomously generate text to aid in reflection and planning in these contexts (Schuurmans, 2023; Kim et al., 2023; Wang et al., 2023d; Xu et al., 2023b). Our paper specifically focuses on integrating theory-of-mind (ToM) capabilities into the planning process.
Theory of Mind (ToM) In imperfect information games, classical methodologies often draw from game theory, but they acknowledge that human decision-making typically follows "cognitive hierarchy" of strategies rather than strictly adhering to Nash equilibrium (Wunder et al., 2011). The concept of Theory of Mind (ToM) is essential for understanding human cognitive abilities, as it involves predicting behaviors by attributing mental states like beliefs, desires, and intentions (Premack & Woodruff, 1978; Frith & Frith, 2005; Davies, 1994; Nichols & Sttcch, 2003; Hurley, 2008). In these games, ToM enhances decision-making by anticipating opponents' actions (De Weerd et al., 2013). Level-k thinking within cognitive hierarchy theory posits that players predict others' actions based on varying depths of strategic thinking, closely related to ToM (Crawford, 2018). Simulating ToM involves adopting an opponent's ' perspective to infer potential actions, with higher-order ToM enabling recursive beliefs about others' thoughts and self-reflection on how one's actions influence future interactions (Frith & Frith, 2005). Recent studies have begun evaluating ToM capabilities in LLMs (Frith & Frith, 2005; Kosinski, 2023), but there remains a gap in integrating ToM within LLMs for imperfect information games.
A.2 Background
Prompting in LLMs Recent research has shown that LLMs, notably GPT-4, can function as universal approximators (Schuurmans, 2023; Kim et al., 2023). Given well-crafted prompts, these models can autonomously perform a broad range of tasks using autoregressive text generation (Schuurmans, 2023; Kim et al., 2023). To formalize this, let Fy denote a pre-trained LLM characterized by the parameter f. Consider I = (I[O), [[])...., ... L[M) as sequence
17

---

Published as a conference paper at COLM 2024
of language tokens, where each L[i] is distinct token. The distribution (L(L) can then be expressed as: Fa (L) = I (L, 1)). By framing the input to the language model as specific task instructions or as few-shot input-output examples (Kojima et al., 2022; Wei et al., 2022b), becomes possible
guide Fg to generate meaningful output for various tasks. For convenience, let's denote the prompt as P = P[ .IM))....)..)... The output from the LLM, represented by Y, can then be defined by the following probability distri- bution: PY = - M Fa [Y P[,,,, Y(1,...,, i - 1j), where M is the total number of output tokens. For convenience, the conditional function Fa P y[,,..., 1]) can be abbreviated as Fg, indicating that is the function specific to the given prompt P.
A.3 Game Rules of Leduc Hold'em
In Ledue Hold'em, em, the deck consists only of two Jacks, two Queens, and two Kings. Each player is dealt one of these as a hole card, and single public card is revealed. The winner is determined by matching the rank of the public card with their hole card; if such match exists, the player with the higher-ranked hole card wins. Note that the big blind in our setting is 2, and the payoff of single game ranges from 1 to 14. Following (Southey al., 2012), we also add the opponent's observation into the single game history h after the end of each game, which also conforms with the real-world experience, but we also perform the ablation study about it in Section 4.3 and A.10.
A.4 Gap between GPT-3.5 and GPT-4
Upon closer examination, we've identified several areas where GPT-3.5 falls short: (a) In- struction Comprehension: GPT-3.5 struggles with understanding instructions as effectively as GPT-4 does. (b) Long Prompt Handling: The performance of GPT-3.5 significantly deteri- orates in scenarios involving lengthy prompts. (c) Reasoning and ToM: Even when GPT-3.5 successfully comprehends the instructions, its reasoning and ToM capabilities are markedly inferior to those of GPT-4, resulting in less reasonable outcomes.
A.5 Limitations
Robustness of Results Due to budgetary constraints, our experiments are limited to running 100 games for each comparison with baseline methods. (1) Although this sample size may not be extensive, the superior performance observed over four different baseline algorithms with varying behavioral patterns can still serve as preliminary demonstration of the cognitive capabilities and potential of large language models like GPT-4 in imperfect information games. (2) Given the same game sequences, Suspicion-Agent can outperform baselines in both positions. This consistency highlights the adaptability and robustness of Suspicion-Agent even when faced with varied strategies that give the same card strength. Considering the limited budgets and the experimental results we get, it is safe to claim that Suspicion-Ape Agent based on GPT-4 can potentially outperform previous methods designed for imperfect information games.
Hallucination Problem of Large Language Model Hallucination problem (Zhang et al., 2023; McKenna et al., 2023; Bang al., unfaithful to the provided source content (Ji et al., 2023)-poses significant challenge in LLMs. In our experiments, we found that when given only simple instructions, LLMs can produce outputs that are either meaningful and rigorous or less rigorous and even invalid. This variability compromises the reliability of LLMs,, particularly when they interact with models trained for specialized tasks. In addition, the outputs of LLMs are very sensitive to the prompts. To mitigate this issue, we developed multiple output templates to improve the quality of the outputs of LLMs, the effectiveness of which is empirically demonstrated in our main results (these templates will be made publicly available in our code repository). However, further work is needed to better align LLM-generated outputs with given instruction prompts. Enhancing this alignment is a critical area of research for improving the reliability and real-world applicability of L.MSs.
outputs that are nonsensical or
18

---

Published as a conference paper at COLM 2024
Long Reasoning Problem The limitations of LLMs like GPT-4 manifest in two ways when applied to complex tasks in imperfect information games:
1) Long Prompts Problem: To adapt LLMs for different imperfect information games, it is necessary to input both game rules and game conversion rules. When these are combined with the specialized prompts designed for our Suspicion-Agent the resulting language model prompts become excessively long. We have observed a rapid decline in the quality of the model's output the length of these prompts increases. Limited by the budgets, we implement the planning and evaluator model into a single function, which results in a quite long sequence generation, leading to the performance decline to some extent.
2) Complex Reasoning/ Caatiinn//alllltttii /Calculation Problem: When tasked with conducting intricate calcu- lations-sus such as computing the average win, lose, or draw rate when an opponent holds different cards-GPTa struggles to consistently generate accurate mathematical equations and results.
Expensive Inference Cost and Slow Inference Time As demonstrated in Table 1, only GPT-4 is capable of performing well in the game of Ledue Hold' em. Due to the extensive prompts and inference tokens, the cost per game reaches nearly one dollar. Additionally, the large model size of GPT-4 leads to a longer inference time for Suspicion-Agent, requiring several minutes to complete single game of Leduc Hold'em. These two limitations underscore the importance of developing a specialized local language model for this task, which also serves our motivation for releasing all associated data.
Planning Depth In our paper, we only focus on single-step planning, but note that our planning method is orthogonal to recently proposed approaches that leverage large lan- guage models for planning with depth, such as Tree-of- Thoouhhhss (Yao et al., 2023), Graph- of-Thoughts (Besta et al., 2023), and iggorithm-of-Thougee approaches offer promising directions for future research, they come with high computa- tional costs, and thus we do not incorporate them into our current methods.
(Sel et al., 2023). While these
More Language Model Evaluation In this paper, the evaluation is confined to the per- formance of GPT-3.5 and GPT-4 on imperfect information games, which represent only fraction of the large language models in the contemporary research landscape. For fu- ture work, we aim to expand the scope of our evaluation to include other state-of-the-art large language models, such as PaLM2 (Anil et al., 2023), Claude2 (Models, 2023), and LLaMA2 (Touvron et al., 2023), among others. This broader evaluation will not only offer more comprehensive understanding of the capabilities and limitations of these models in imperfect information games but also facilitate nuanced comparative analysis. Such an approach is expected to yield richer insights into the adaptability and generalizability of large language models in complex, real-world scenarios, thereby contributing to the field's s collective understanding of their potential applications and limitations.
A.6 Future Work
Tool Use As outlined in the section A.5, Suspicion-Agent suffers from hallucination problems and struggles with long context reasoning. This can lead to calculation inaccuracies and sometimes produce responses that deviate from factual information or are out of context. Such issues considerably degrade the performance in final decision-making. A natural solution is break down the problem into multiple sub-problems and employ specialized smaller models or tools (Wang et al., 2023c; Schick et al., 2023; Wang et al., 2023b; Patil et al., 2023; Lu al., 2023a; Patil et al., 2023), for better task completion.
Multi-Modality In the present paper, the analytical scope is limited to text-based imperfect information games. However, is important to recognize that real-world interactions often encompass more than just textual information. For instance, human communication frequently involves a variety of modalities such as facial expressions and vocal tones, which can serve additional cues for interpreting beliefs intentions. Given the increasing advancements in multi-modal large language models-e,2 InstrucBLIP (Dai et al., 2023), LLLVVa (Liu et al., 2023a)- we aim to extend our research to incorporate these multi-modal
19

---

Published as a conference paper at COLM 2024
aspects. By doing so, we aspire to develop AI agents capable of navigating imperfect information games that more closely mimic real-world complexities. Integrating multi- modal observations into our model will not only enrich the agents' understanding of the game environment but also broaden the applicability of our methods. This will potentially lead to more nuanced and comprehensive understanding of the strategic behavior of LLMs in scenarios that more accurately reflect real-world conditions.
Multi-PPlayer Setting In the paper, our focus is restricted to two-player imperfect information games. However, it is worth acknowledging that real-world scenarios often involve multi- player settings, which introduce additional complexities and nuances that are not captured in a two-player framework. Recent developments have given rise to novel multi-player game environments, such as AgentVerse (Chen et al., 2023) and Mind Agent (Gong et al., 2023). These environments present more realistic settings for evaluating the applicability and efficacy of large language models in game theory scenarios. Therefore, a natural extension of our research will be to adapt our methods to these multi-player environments.
A.7 Output Samples
A.7.1 First-Order ToM aware Planning
Opponent's Pattern:
1st Round: When GoodGay holds King, he lends to raise (70%) or call (30%). When GoodGuy holds Queen, he tends to call (60%) raise (40%). When GoodGuy holds Jack, he tends to fold (60%) or call (40%). 2nd Round: When GoodGuy holds King and the public card is Queen Jack, he tends to raise (80%) or check (20%).- When GoodGuy holds a Queen and the public card is a King Jack, he tends to call (70%) or check (30%). When GoodGuy holds Jack and the public card is King or Queen, he tends to fold (60%) or call (40%).
Belief on GoodGuy's cards:
In the 1st round, GoodGuy raised (probability 70%), which according to his behavior pattern, suggests that he might be holding King (70%) or a Queen (30%). In the current round, GoodGuy raised again (probability 80%), which according his behavior pattern and the match with the current public card (Queen), suggests that he night be holding a King (80%) or Queen (20%).
20

---

Published as a conference paper at COLM 2024
Reasonable Plans:
Plan 1: Call - If call, I wiill match GoodGuy's bet of 10 chips, bringing the total pot to 20 chips. This a conservative strategy that allows me stay in the game without risking many chips. However, it also means that I am not putting any pressure on GoodGuy, which could allow him win the game easily if he has higher-ranking card. Plan Raise If raise, will increase the bet by 8 chips, bringing the total pot to 24 chips. This is an aggressive strategy that puts pressure on GoodGuy and could potentially force him to fold if he has low-ranking card. However, it also risks losing more chips if GoodGuy has higher-ranking card and decides to call or raise. Plan 3: Fold If fold, will forfeit the game and lose any chips have already put into the pot. This is defensive strategy that minimizes my losses if I believe that GoodGuy has a frigher-ranking card. However, it also means that I give up any chance of winning the game.
Potential GoodGuy's actions and Estimate Winning/Lose/Draww Rate:
If GoodGuy holds a King (80% probability), he will likely raise (80% probability) or call (20% probability) than my Jack. Therefore, the overall lose rale for this scenario is 80%.- If GoodGuy holds Queen (20% probability), he will likely call (70% probability). If this happens, I will likely lose (100% probability), as a pair of Queens stronger than my single Jack. Therefore, the overall lose rate for this scenario is 20% 70% 14%. Therefore, my overall lose rate is 80% + 14% = 94%. My overall win rate is 0%,, as there is no scenario in which I can win. My overall draw rate is also 6%.
this happens, I will likely lose (100% probability), as King is higher
Estimate Expected Chips Gain for Each Plan: Plan 1: Call Expected Chips Gain = win rate winning payoff- lose rate loss payoff 0% * 10 94% 100 = -9.4 chips. Plan 2: Raise Expected Chips Gain = win rale * winning payoff lose rate lose payoff 0% 12 94% 12 -15.6 chips. Plan 3: Fold- Expected Chips Gain win rate "winnngg payoff-
lose rate lose payoff = 0% * 0 100% == -8 chips.
Plan Selection: Plan 1: Call: -9.4 chips Plan Raise: -15.6 chips- Plan 3: Fold: -8 chips. Based on the estimated expected chips gains, the best plan is to fold, as it results in the smallest loss of chips. This is reasonable decision, as it minimizes my losses in a situation where am likely to lose. It also aligns with my strategy improvement of being more cautious when have low-ranking card.
21

---

Published as a conference paper at COLM 2024
A.7.2 Second-Order ToM aware Planning
Opponent's Pattern:
1. When GoodGuy holds Jack: If the opponent raises in the first round, GoodGay tends to fold (probability 80%) or call (probability 20%). If the opponent checks or calls in the first round, GoodGuy tends call (probability 70%) or raise (probability 30%).-Ir the second round, if the public card is King or Queen, GoodGuy tends to fold (probability 90%) or call (probability 10%). If the public card is a Jack, GoodGny tends raise (probability 80%) or call (probability 20%).
When GoodGuy holds a Queen: - If the opponent raises in the first round, GoodGuy tends to call (probability 60%) or fold (probability 40%). - If the opponent checks or calls in the first round, GoodGuy tends to raise (probability 60%) or call (probubility 40%). -In the second round, if the public card is a King, GoodGuy tends to fold (probability 70%) call (probability 30%). 1If the public card is a Queen or Jack, GoodGuy tends to raise (probability 70%) or call (probability 30%).
When GoodGuy holds a King: If the opponent raises in the first round, GoodGuy tends to call (probability 70%) or raise (probability 30%).. the opponent checks or calls in the first round, GoodGuy tends to raise (probability 80%) or call (probability 20%). - -In the second round, regardless of the public card, GoodGuy tends to raise (probability 70%) or call (probability 30%).
The opponent's guess on Suspicion-Agent's game pattern:
1. When hold Jack: GoodGuy might believe that I tend to raise in the first round (probability 60%) or call (probability 40%). In the second round, if the public card is King or Queen, GoodGuy might believe that I tend to fold (probability 70%) call (probability 30%).-1J the public card a Jack, GoodGuy might believe that I tend to raise (probability 80%) or call (probability 20%).
When hold a Queen: - GoodGuy might believe that tend to raise in the first round (probability 70%) or call (probability 30%). In the second round, if the public card is King, GoodGuy might believe that I tend to fold (probability 60%) or call (probability 40%).
If the public card a Queen or Jack, GoodGuy might believe that I tend to raise (probubility 70%) or call (probability 30%).
3. When hold King: GoodGuy might believe that I tend raise in the first round (probability 80%) or call (probability 20%). In the second round, regardless of the public card, GoodGuy might believe that I tend to raise (probability 70%) or call (probability 30%).
Belief on GoodGuy's Cards:
In the first round, GoodGuy chose to call. According to GoodGuy's 's behavior pattern, when he holds a Jack, he tends to call in the first round with a probability 70%. When he holds a Queen, he tends to raise in the first round with a probability of 60%, and when he holds King, he tends to raise in the first round with a probability of 80%. Since GoodGuy chose to call, more likely that he holds Jack (probability 70%),, less likely a Queen (probability 20%), and least likely a King (probability 10%).
GoodGuy's Belief on My Cards:
In the 1s! round, haven't It taken any action yet. However, based on GoodGuy's 's guess on my game pattern, if choose raise, he might believe that have a King (80% probability) or Queen (20% probability). 1 choose to call, he might believe that I have Queen (60% probability) or a Jack (40% probability). If choose to fold, he might believe that have a Jack (100% probability).
22

---

Published as a conference paper at COLM 2024
Make Reasonable Plans:
Plan 1: Raise- If raise, I can potentially confuse GoodGuy and make him believe that have a higher card. This could lead him to fold, especially if he has a Jack. Plan 2: Fold Folding would be safe option, as I currently hold the lowest rank card. However, this would mean giving up the current game and losing the chips have already put the pla.Plan 3: Check Checking would allow me to the public card without putting more chips in the pot. This could be good strategy if believe that GoodGuy has a higher card.
Estimate Winning/ Siiniing/Lsse/raae Dosw Rate for Each Plan:
Plan 1: Raise- If GoodGuy holds a Jack (70% probability), he will likely fold (80% probability). This would result in win for me.- If GoodGuy holds Queen (20% probability), he will likely call (60% probability) or fold (40% probability). This would result in draw or loss for me. If GoodGuy holds King (10% probability), he will likely call (70% probability) or raise (30% probability). This would result in loss for me.Plan 2: Fold- If fold, I will lose the current game regardless of GoodGuy's card. Plan 3: Check If GoodGuy holds a Jack (70% probability), he will likely call (70% probability) or raise (30% probability). This would result draw or loss for me. - If GoodGuy holds Queen (20% probability), he will likely raise (60% probability) or call (40% probability). This would result in a loss for me. IIf GoodGuy holds a King (10% probability), he will likely raise (80% probability) or call (20% probability). This would result in loss for me. Potential belieoes about the number of winning and lose payoffs for each plan: Plan 1: Raise - After the action, chips in the pot: 6. If win, the winning payoff would be 3. If lose, the lose payoff would be 3. Plan 2: Fold After the action, chips in the pot: 4. If win, the winning payoff would be O (as would have already folded). lose, the lose payoff would be 2. Plan 3: Check After the action, chips in the pot: 4. If win, the winning payoff would be 2. If lose, the lose payoff would be Estimate Expected Chips Gain for Each Plan: Plan Raise Expected chips gain = wimning rate wiiming payoff- lose rate lose payoff = 0.56 *3 0.44 3 = 0.36 Plan 2: Fold Expected chips gain (as would have already folded) Plan 3: Check Expected chips gain = winning rate winning payoff- lose rate loss payoff 0.14 2 0.86 -1.44
Plan Selection:
Based on the estimated expected chips gain, Plan (Raise) would be the best strategy to implement in the current game. This plan has the highest expected chips gain and aligns with the strategy improvement of confusing GoodGuy by making him believe that I have a higher card.
A.8 Ablation Study on Levels of ToM
Comparison of Action Percentage of Agent with Different Planning
Comparison of Action Percentage of Agent with Different Planning
Second-
Secand-Order
60
Resh: Order
Rirst Order Vanilla
Verilla
Percernge
Action Percentnge.. CS0
40-
30-
20-
10-
raise
call
chack
fold
raise
call
chack
fold
Action Category
Action
Figure 5: Left Figure. The illustration of the action percentage of Suspicion-Agent with different levels of ToM when playing against CFR+. Right Figure. he illustration of the action percentage of Suspicion-Agent with different level of ToM when playing against DMC.
23

---

Published as a conference paper at COLM 2024
A.9 Algorithm
Algorithm The inference procedure of our Suspicion-Agent
Initialize the game history D, large language model Fa, Game Rule Rule, Observation
Conversion Rule Ruléøhs do Initialize the current game buffer H
for 1 to Max_Game:
Get the Opponent's Behaviour Pattern and Opponent's Guess on My Pattern, and Reflexion Op O F (Rule,
for = 1 to TaskHorizon do
Get the Private Observation s Pritoy Subb and action of the opponent a} from the environment.
Get the Text Observation Description Obst Egles (Rule, Ruleobs St (ri Spub).
Get the Card Prediction O "aar_ppred" F prard_pred (Rule, lt,, Obs,, 0 hp))
Make Plans Optar ~ Fg plllnn (Rule, hT, Obs', O OpprO)).
plans Evaluate Plans and Make Action dj ~ Fg (Rule, Of O prrdd Op O³)).
Execute Action and Get the Game Result ri Add the Game State and Action into the current game buffer H h U
{st (rj Spuss 4br r'" end for Add the current game buffer Jt into game history D +- DUN end for
A.10 Ablation Study the Effect of Hindsight Observation
In primary results, following (Southey et al., 2012), assume that Suspicion-Agent has access to hindsight observations at the conclusion of each individual game,ie., Hindsight Observation. To assess the impact of this feature, we conduct ablation study in which hindsight observations are not incorporated into the current game. In the absence of these hindsight observations, we augment the Reflexion module with additional prompts to enable it to infer the opponent's 's cards based on game outcomes and Suspicion-Agent' own observations.
As demonstrated in Table 5 and 4, Suspicion-Agent retains its performance advantage over the baseline methods even without the benefit of hindsight observations. Specifically, we observe that Suspicion-Agent adopts more conservative strategy under the increased uncertainty that comes without hindsight observations. This leads to reduced bluffing, resulting in fewer gains when playing against CFR+. However, it also minimizes the risk of over-bluffing when facing DMC, thus yielding higher chip earnings.
Table 5: The comparison results indicate the impact of including opponent observations in the game history when Suspicion-Agent competes against CFR in Leduc Hold'em em envi- ronments. The results are the win/lost chips after 100 games with different seeds, and the number of win /lose chips ranges from 1 to 14.
Opponent Model
CFR+
DMC
Ours (w/o hindsight observation)
-42
+51
Ours (w/ hindsight observation)
-22
+24
24

---

Published as a conference paper at COLM 2024
Table 6: More Suspicion-Agen experimental results using open-sourced large language models.
DMC
DQN
NSFP
CFR+
GPT-4
24
45
142
-22
Mistral-v2 7B Gemma 7B (Team et al., 2024)
23
33
-10
-66
26
10
-72
Llama3-8B Team (2024)
54
-67
-61
-103
Phi3-medium (Abdin et al., 2024)
82
57
36
-14
A.11
Qualitative Samples
A.11.1 Leduc Hold'em
We present qualitative samples showcasing Suspicion-Agent' behaviour under different strategies: Vanilla Planning, Planning with First-Order ToM, and Planning with Second- Order ToM in Leduc Hold'en em. These samples can be viewed in Figure 6, 7, 8, and 9, respectively. In each scenario, Suspicion-Agent holds a Jack, while the opponent has either a Jack or Queen. The opponent's initial choice to call, rather than raise, suggests they have weak hand. Under the Vanilla Planning strategy, Suspicion-Agent chooses to call in order to see the public hand. When this reveals a weak hand, the opponent promptly raises, leaving Suspicion-Agent in a precarious position, given that Jack is the weakest hand. With the First-Order ToM strategy, Suspicion-Agent opts to fold, aiming to minimize losses. This decision is driven by the observation that the opponent typically calls when holding Queen or Jack and has a knack for successfully bluffing Suspicion-Agemt However, these strategies fall short in capitalizing on the inferred weakness of the opponent's 's hand. This shortcoming stems from the fact that they don't consider how Suspicion-Agens might influence the opponent's reactions. In contrast, as demonstrated in Figure 9, simple prompts enable Suspicion-Agent to understand how to influence the opponent's actions. Intentionally choosing to raise places pressure on the opponent, urging them to fold and minimize their losses. As result, Suseicion-Agent manages to win numerous games even when holding hands of similar strength, so that Suspicion-Agen baselines. Moreover, depicted in Figure 10, if the opponent either calls counters the raise made by Suspicion-Agen it suggests the opponent has strong hand. Recognizing this, Suspicion-Agent swiftly adapts its strategy, opting fold to prevent further losses. This demonstrates the impressive strategic flexibility of Suspicion-Agent.
moves
can earn more chips than
0bs:1n this round, you are holdding C
Oba:1s this round, you are solding
Obacin this round, you are holding a lato the you opposent aas pat chips.
white
pet, apponest has put 2 chips.
coposent has pat 2 chips.
Action: Raise
Action: Fold
Vanilla Planning
Planming with Fürst First-Order Order ToM
Planning with Second Order TOM
Figure 6: The qualitative sample of Suspicion-Agent Leduc Hold'em. The whole outputs of Suspicion-Agent are given in Appendix 7, 8 and 9, respectively.
25

---

Published as a conference paper at COLM 2024
Actior: Coll
this round, you hoteing Jack Hearts, rar, you have pat chips into the pot, while your opponont
plan,
night call or raise, loating to Larger par. Plan 2: fold In this plan, 1 will fold hand. This will result in a loss of the current gane, but I will save by chips for fature ganes where I right have better hard, Plan add
plan, cteck,
stans
pot. This allow
Estinate ktmrdagrLess/Dra Rate for Each Plan: Plan Ratze ratse, have 1/3 chance of winntng the the publis Jack, rate is 33%, Plar 2: Fold If fold, will lose this pane, my losing rate 186%. Flan 3: Check If check, the nutcose will depend on the public card. The probabilities are the seme as Pian 35% chance virning, 33%
losing
Queen,
Jacs. bverall, winning
lostng,
Expected Chias Gain for Back Plans Plan 1: Raise The expected chips gain is 33% 3 chips 23% a chips 0.33 chips- Plan 21 rote The expected chips gain -2 chips, becnuse will Lose chips. Plan 3: Check The expected chips gaán comparing the esin of 9.60 chiss, Therefore, I will choose Plan 3: Check.
chias 0.65 chips. chips gate for each plan, Plan Check has the expected
vanálla Planning
Figure 7: The qualitative sample of the vanilla planning of Suspicion-Agent Leduc Hold'em.
Action: Call (ollds Jack too)
tbs:In this round, are holding Jack of Hearts. So far, you have put chips into the pot, while soponent for put chips.
Belliff Goodauy' cards: the let round, Goodiay chose to "call' (probability 1886). According
understanding of Goodbay's betavtor patter,, when te holds King, te tends to raise (praballltty lity 70% or call Ipribability 300), when be holds Queen, he tends to call lprocastiity 688) or raise (probabil 48%) When he (probabillty
iprececl (probability 384) Jack (probabilst 48%) than
(probabilsty Therefore, I estiuste that Goodlay has Dueen (585), King (389), or lack (186).
likely that he holding
Pake Reasonable Plass:
Plan 1: Raise IT raise, will put nore chips into pot, making the total a chips, This could potentially int inicate Gondlny into folding, espectal ially
holding Jork. However, losing wore band. Plan 3: Checkin- If I check, I will not put any more chips into the put. This could be good straing if vant to aue public before saking decision. However, this
a king Ducon, night
even raise,
chips belleoe that Goodlny
loving
chips.
cuuld good strategy
also give Gondhay the appar sumily to ruise,
Potestial GandGay' actions Extimate kisning/Lnww/rra Rate: If Gandlay holds Quwn (600 probab mubabalityle hu will likely call 1806 probabilitty. If the public tard is a Damnn, will libuly lowe (as hold Jacel. public card King Juck, have thame win raise or cluck Goodlay will 18th
love. If fandlay heles King 1ats probabilatyy)
will likely reise (W0 mubability) if raises, it's st's possible that the public card King Oueen, probability).
likely that the publis card is king, and I will Likely lose. ne eight call 130h probubality). If calls,
witl atso Likely lese. Goodlny holds Jock regandless card. in- night call (68% protability). he calls, possible that public
Dieen,
and have chince to win If raise or check and Goodaay folds. Overall, wtening rate is 209 my tostng rate and drww rane Losing Bate
Winning Minning Payoff 786 6 -2. chips plan fold Expected chips Rate Winning Fayoff Lasing Bate los Phyoff = 180v + missinittian 3: Eheckin- Expected chips kinning Rate Minning Payoff iosing Bate Laxing Dayoff
Payoff
Kinning
78t -8.188 chips
gains, "foid Check Raise Therefore. should
Action: fold
Figure 8: The qualitative sample of planning with First-Order ToM Suspicion-Agent Ledue Hold'em em.
26

---

Published as a conference paper at COLM 2024
Action: tall (nolds Jack or Queon!
Cosiin round, you are hotsLng Jack Hoarts, for, you put chips the pot, while your apponent
Bellef on Goodauy's s cards: Goodluy chose "call" in the fürst round, accorsing to hás behovior pattern, when he notes King, he would Like to rass in the first round. Therefare, it unlikely that he tolds King, when ne Hikely
Jack. the ittis arde
(86), Queen (586). Jark 158K).
Potential Goodfay's current halinve about your cards: I have not taken any action in the first round yet. Accerding conmayy garss on perw patterr, night
Leves any
mnie like raise when hold king, Queen, probubility.
aare King 133, 33h1, Suees (33,355), Jock (33.25h).
MMaee Plans: Plan I: baise protact Lityl,
chips (s froe 2 from cooduuy). warning 588, the loxing rate and drew da,
Gocdiuy holds (50% probabilitty), will 12008 probabilityl. the sot, doodbay).
0b,
Plan Patd
probebility
will be chips fros from Gondlay). Ba.
86,
the rate
Gondluy holds (seN probshilitty), will check (188% prababiliry). will lowe cet, which will rate
(2
dooduay).
Plan 3: Check
probabilityl. The gane procecd the - II Goodiary holdss Jack (506 probehilsttl) Lity,, he will check (188K cibebilitt), The game will proceed to the second round. The wiiming rate, lowing rafe, and drew rate carnet deternined this stage.
Gocdluy holds Queen protacilltyy, will. Losing
Extirate Expected thips Gain for tath Plans
- Flan 1: Expected Chips Biain 526 v chips de B chips 1.5 chips chips chips
Expected Chips Gain Flan Expected Gain thips
octermined
Flan Selection: Plan 1: raise
Acttons Ratse
Figure 9: The qualitative sample about Bluffing of planning with Second- Order ToM Suspicion-Agent on Leduc Ho.d'em.
Raise (Blaff)
Action: Call tholds King or Queen)
0bo: You are currestly holding Jack of Hearts (8)) and the public card is Queen of Hearts (HO).
Belief board gare expert's cards: Is first round, board gaarrd_aae__ppppp expert. sav raise and decided to coll. According to his behavior pastern, whes he tolds King, he vould call if he sees raise, lie he holds a dueen, Therefore, likely that beard
queen 158K probabilityy To Suees, he sould ratse, Heweeer, holiding queen ancertain. mand, pane pattern, whin held a King, belioves would raise. when hold bucon, he selioves vould either cail. or raise. vhen hold a Jack. he believes would fels. merefore, froe the perssective board pane expert, tend have King probabilityl or Queen
King (586 probeellity ttyl the second round, the public card is Oueen, If boand game expert holds
sove Therefore, the prababil
your
board oase cipere's guess
pobbabliity)....
Make Plans: Plan sein- hoard. procasilityl will lose king i5 hitter than Jack. in- Ef boars cone, export holds Queon (50h prossnilityl, he will
holds King probabili ry), he will call 13880
sone, queen sigher than Juct. irtnnlan fuldte-
prababilityy.. will
bowrd sxpert holds Buwnn probability), lity) will lase the chips
already 3: checkin- If bcard pane expert holds a King (595 probabilite), he will call 1188% prosooitityl. will lose as king higher
An- qoeed exgert ids Queen (50% probabi lity), will call 13800 lase
procaal
Estimate Expected Chips Gate for Each Plan:
Plan 11 Expected Chips Eain
1885
chips
Plan d: Sepected Gain
188
thaps
Plas selection: Plan has the highesi espected chips gean (-4 chips), so should cheose Plan 2: ruld."
Action: rold (Mininize the losa)
Figure 10: The qualitative sample about Strategic Fold of planning with Recond-Order ToM Suspicion-Agent on Ledue Hold' em.
27

---

Published as a conference paper at COLM 2024
Foreign [Peans
reaaaabee_ text You have two hidden Assassin cards coons. your cards dead
first behavior patsem, if nolds Bike, sould Likely block foreign aid (906) or (30s). if holes Assassin or Captain, he vould pass (1893. Since didn't block the foreign atd, it's iniikely that he has quie. Therefore, the prosasilitie:
Belief
Accending
cards could According to Goodlay 's guess my pare pattere, If I hold a bake, I would Iskely the (705) or take Incone 1300).
(188), Assassin 145%), captain (455)
believes
take
told as desassin, would likely take income (089) or tax (209). 3f hold Captain, sould Likely steal 140%).
Plan Block Aidin- Goodiuy
would
(3085), would
(1805) us I sucressfully block his forcigs aid, If Soodiuy holds Assassin 14551: would Likely challengs (580) or pass (180)
lenges, sould lose (1385) foreign atd- thet thages, sould lase (1826) don't passes, I would win (1004) as sucosstully black his foreign aid.)
don't have Bike to reveal passen, Dake reveal. If
chellange (580)
(508).
overall Himiing/ overiree ratest yo- niminn rate: 100rld00 4540558 458-520 500 Lose rate: 45 455-50h
Plan 2: Poss Foresgn Addin- Goodduy holes bake (185)2 He would likely pass (180%). I would neither vin nor Lose as I dimn it take any action. in- If Goodlay holds assassin (45%)s we wou tikely pass (188%). would neitter as dion't
vin netther
any actioo.
Goodlay
Captadn 14553: world Likely pass 1189). Aavi overail Lese//ras
didn't rate: ewi- Love rate: mM0r- Draw rate: 18ik
Winning
Extinate Expected Chips Gain for Each Plansin- 1: 559
4.t chipstn- Flan 2: repected gein."
strategy
Action: Block_.orciip_.dd 18laff Duke)
Action: Bloc_
60
"readable_
your
Beller on board gane, 'spert's cards: In the first round, boans gare expert atteapted take foreign atd, whach 15 neutral actior that doese't reseal such about cards, Rouever, sher Blocked his foreign ats, he chose probadility
not lenge et, sugpests that sight Since he didn't mnagterge block, Assassis or Captale card. Given his passive play, I would estimate that he his Assasste card (596) and Captain
blacked
paar_ expert's fereign aid. this action sight Load his to balieve that have buke card, as Dake can btack foreign ald, Hewever, could also be blufring. Given his knowlerge of my gare nottem, he eight think that I card 16th) an Assassin card 140%). (1)
Nake Plars:
'income' action board exgurt Buke, night challange (emm) pass An- if
boart expert expert huids Assessir, will (a8te), and will holds. Captais, he will pass (1p00), and will pain cain.
plan Take "tas' actioo (blaffing chnges will
qusee holds Duke arobability), right don't have
leagg (626) passes, will gain three colts, Vi- If boand gare expert holds Assassin, he night challenge 1585) or pass 15881. If te chailennge, will tose ar luence las don" have Dikel, and Coptain, Plan Take "steal" action isluffing Captatnn If (1885),
(185).
inffiencee
passes, will gain three
pass 120%). be holds gain will (1004). will steal too coans free his, vin if board quae, expert holds Assassin, he will pass (1009), and will board_ores, cnpert
gaee.
two coins from his,
captain, sight
blocks,
AakaPlan ds Take "exchange" action toluffing as babassatorisn- If board_qare, oxpert holds buke, he will pass 1i00s), and i will exchange By cards. in- if board_gane, expert holds Assassin, will pass (1805), and will Plan 2 (bluffing tuke taking 'the' action) patential for gaining coles, moweer, it also carries risk of tosing in luence if board gae- expert decides to chall lenge, Plan (taking "nnree"
faxed the above
have the highest
is the safest opt ton, hut the pain mirteal. Plan
of Losing an influence if board_pame pane cxpert holds Captain and decides to block. Plan (bluffiing as Abbascador and taking "axchange" action) doesn't gain any coins but could potentially change the pone dynomics by getting Considering potential Therefore, vould chosse Plan 2 (6 (bluffing as buke and taking 'tax" action) strategs.
would rank
Plan Plan Plan
18tuff
Actiond Pask_Tex (Be bluffed)
Figure 11: The qualitative sample of Suspicion-Agent on Coup, where the GoodGuy is the opponent model and board game expertt Suspicion-Agent.
28

---

Published as a conference paper at COLM 2024
Observation ione (fr di (lhand"; ["00', "i5",, "pubilc_cards"" (109', "Ca", 1er1, "ull_enipo"s 134, 181, "hy_chips"" 18, "Logel Lattions"s Poall", fruiset, "foia", 1, 61b3;
"ratse_russ": Paise
diservation: ("hane";: 1911%, "Ha"],
("por, "t2", "Hr",, "all Hay chips': 16, "Tegal lonel;; ["aall", "ratse', "fold",, "raise muss (2, 2, 2, Ni, run't "rest chino"; 48.0, "opsonert_ress" pereeeetttessteeeeee restr chips"; 52.4),
Readante observation: In this round of Texas Hole' Poker Lisit, you are holding 3.3 of Dianonds and hearis.
comunity cards on the table are
Disannts, king Clubes,
of Clubs,
you ur fold, So far, rounds of betting, and haw not raised in the curren, courd. This is the 5tl gewu wut of iotel of qar in. You have chips left, shile your apponun: has chips Your best cur hand coebinations, dependting snother community card, costd two three-af-a-
hase the aptions to call,
haw raised bvice mach of the first thre
Dt. potentisl if
up.
GoodGay's 's gane patters: Goodlay seens to be conf stayer who not afraid to raise the bet when has decent
sears be good at biuffinn, ss in the first cane where holds pair, he is Likely raise the bet 178) probanality: or call 130% protabilstyl in the first round, regercless of the nsponent act actiio.. 2. If the flup roveals potent ial streight or flush, Gondba is likely to raise the 150 protabilityy or (40h probaiility in the second round, improve
110
rppament
Gondkny
told (50%, arobsbility) GoodGay's guess on gaee patterni GoodSuy night perceive paand_gane_exsee who 15 af ratd to told when the situstion colls for it.
likely check (585 probasility)
coutious player
Round:
or patr, Goostuy night card belov), bet (probability 7883 card (probabilsty 38%) When
when
Goudluy night think that will check (probabilisy 5es) fold lorobability 40s1 Aniunndd hound: in- If I have a potential straight or flush after the flop, Goodlay night think that will raise the bet forohwbil ity 8061 call sprchabilitt 2000-A-- flop, Goodfay night
not have petential straight or flush lprobast thy
(probab: Lity
Balief Goodfay' cards: In the firms round, Gondiny ruised fftrr raised, which according tehee inr pelturn, supasts that be likely has a high card (1b or abows) or a ouit 1809 protebilityl. In the second round, the tlop revealed a potential straight or tlush, and GoodGay rassed again, night
potential straight or Flssh (38% probsbillty). Goodbay to believe that have high card 110 abovel round, the Tlup did not Jasrove my fand, but raised again, which night con fase Goodaay and tead hás bluffing that have strong 158s probabalityl.
supgesting
ruised, which night load (6th probailitt),
the second
believe
current round, strong (399 prolability).
hand (706. lity. free the perspective Gondinyy potential straight (20%, iity, a patential flush (20% tty) or high card (IR.. (ityl
Make Ressonable Plans: Plan 1: call- If 1 call, 1 will natch Goodlay's 's bet. This is safe move that does not risk too nany chaps, can say, VI I. natch your bet, Goodaay. Let's see the rext raise,
het put pressure Gonstayy This Palan Belseln- higger "Paan Fold-
increase
chips. forfeit and lase the chiss have atrezdy bet. This 15 coutious nove that vindnites losses, I car 1"I maik I'll sit this one cut, Goodbay.
potentially
say,
confident, GoodGay.
Plan: Plan call- Goodauy Goodluy's actions and Estiaste /inetog/Looeerraa Rate Likely raise (P0
potential
flush (301,
cibbyiility. I will likely lase (100% probuiility- --l Gondkay has high card (28 abwve) pair (20h probebilaty), bu will libely raise 1580 protubilityl or call (sth probwsslityl. prutubility
likely win (60% probability), drew lostng
probability Drerall, lase draw is is 45.0%
vinniing
likety raise 178% protabt lstyl. will ltkely ty lose (1095 probabilstyy- J. -Ir Goodiay has high card (10 or above) or pair 178% prosabi lityl, he will likely call 150% probabilstty or fald 1985 lase
Raise If Goodiay
potertial straight +tush 138% protabilityy
wir
prdballity).. draw probabelstyy plan
probability). crow rate is 7.0%.
Plan 3: Foldie- If fold, will mntonatically lase the game. The wirning rate for plan is M9, the los ins rate is 1889, the dres rate de.
After seliaves
the payoffs
payoff would be chice, pot: Falsein- After the action, Chips in the pot: 22, If win,
the
lase payoff would be 9 thips, UrArPan 2:
the winsing payoff would be 11 chips. If lose, the lose payort would be 11 crips-AnaPPan 3: poti 16. wis, the virking payorf would be chips.
Foldin-
action. Chips
Estimate Expected Chips Gain for Each Plan: Plan i: Call Expected Chips Gain 42.0v
46.0s -3.6 shaps 2: Raisein- Expected Chips Gain 5th w it 37e 11 - 21 chips Plan 3: roteve- Expected Chips Gain expected chops
a 188% 9 i chips Flan Selection: rank ot every plan tollowsinn). Callin3. Plan relevnnnfnereoors should esecute Plan Ratse, it has highest eest rnsted expected chips gain.");
Action: Raise
Act lonc Fold
Figure 12: The qualitative sample of Suspicion-Agent on Texas Hold'em em Limited, where the GoodGuy is the opponent model and board game.expert is Suspicion-Agent.
29


Reviews:
[{"review_id": "aPtvs9NvZB", "reviewer": "Reviewer_z6gK", "summary": "This paper investigates LLM performance with imperfect information games without additional training or data, and it proposes Suspicion-Agent. To tackle challenges of opponent’s uncertainty, the authors propose theory of mid (ToM) aware planning approach. The authors conduct comprehensive evaluations with GPT4.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in sharing critical feedback. We appreciate your encouraging feedback.\n\n**1. Some discussions and tradeoff would be needed with n-order planning (n≥2).**\n\nA: It is a good point. The necessity of constructing or utilizing third-order (or higher) Theory of Mind (ToM) in game-playing scenarios has been a topic of discussion in previous research [1,2]. These studies suggest that ToM beyond the second-order tends to become less effective. This reduced effectiveness is primarily attributed to the increased complexity involved in calculating third-order (or higher) ToM. This increase in complexity can lead to diminishing returns in terms of strategic advantage, especially in practical gaming scenarios. Players or AI agents attempting to anticipate an opponent's beliefs about another player's beliefs (which is the essence of third-order ToM) often face considerable challenges in accurately processing and utilizing this information effectively. Thus, while higher-order ToM may provide theoretical insights into more complex strategic thinking, its practical application in game-playing contexts is limited by computational and cognitive constraints.\n\n**2. The ToM planning strategy seems to be similar to existing works for LLM game playing**\n\nA: \n\nThank you for the recommendation. Our paper and GTBench are contemporaneous works, and we are happy to include the following discussion in our camera-ready paper:\n\n*Contemporaneous work GTBench proposes a benchmark to systematically evaluate the performance of LLMs on various imperfect information games using existing prompting techniques. In contrast, we have developed a specific prompt system tailored for imperfect information games, which enables GPT-4 to outperform certain learning-based algorithms such as DMC and NFSP, even without specialized training or examples.*\n\n**3. What kind of imperfect applications can the proposed framework generalize to?**\n\nA: Good point. Currently, our Suspicion-Agent focuses on text-based imperfect information games, requiring game rules and game status interpretation rules as input. Therefore, it cannot yet generalize to multi-modal imperfect information games.\n\n[1] De Weerd, etc. Higher-order theory of mind in negotiations under incomplete information. *PRIMA 2013: Principles and Practice of Multi-Agent Systems\n\n[2] De Weerd, etc.  How much does it help to know what she knows you know? An agent-based simulation study. Artificial Intelligence, 199, 67-92.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The angle of imperfect games for LLM evaluation is important and practical to various applications.\n- The proposed approach has certain novelty. The empirical experiments are comprehensive and lead to interesting observations.", "reasons_to_reject": "- The paper only explores first-order and second-order prompts for planning. In practice, applications or games could require much more rounds of planning. Some discussions and tradeoff would be needed with n-order planning (n≥2).\n- The ToM planning strategy seems to be similar to existing works for LLM game playing, like GTBench [1]. The authors need to clarify the differences to similar works and make novelty clear.\n\n[1] GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations", "questions_to_authors": "- What kind of imperfect applications can the proposed framework generalize to?", "extra_scores": {}}, {"review_id": "ef0cObAC3J", "reviewer": "Reviewer_dfyc", "summary": "This paper apply LLM to building a poker playing game. It leverages prompt engineering to baked in the mechanimsm of theory of mind during the planning for each action. The authors conduct experiment on a simplied version of Texas hold'em and shows potential signal of using LLM's reasoning power in the imperfect information games.", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in sharing critical feedback regarding our work. We appreciate your encouraging feedback.\n\n**1. Can you try more open sourced model?**\n\nA: Thanks. We tried more open-sourced models\n\n|               | DMC | DQN | NSFP | CFR+ |\n|---------------|-----|-----|------|------|\n| GPT-4         | 24  | 45  | 142  | -22  |\n| Mistral v2 7B | 23  | 33  | -10  | -66  |\n| Gemma         | 26  | 8   | 10   | -72  |\n| LLama3-8B     | 54  | -67 | -61  | -103 |\n| Phi3-medium-14B   | 82  | 57  | 36   | -14  |\n\nWe indeed find that recent open-sourced models demonstrate strong instruction-following ability and thus can achieve better performance than before.\n\n**2.You idenfiy couple of important ability in the appendix, \"Gap between GPT3.5 and 4\". One of them is Long Prompt Handling.**\n\nA: Thank you very much for the encouraging comment. As a pioneering work that applies LLMs into unseen tasks especially in imperfect information games, we believe our results, exploration and public dataset would inspire more subsequent works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a very solid paper, and applying LLM with planning into poker is novel as far as I am concerned. The proposed method lays a good foundation for this line of research, and I imagine can be used as a solid baseline. The experimental results are comprehensive with different baseline methods. It's also interesting to see the ablation study that improve the understanding.", "reasons_to_reject": "I really like this paper, and I think it would be a good contribution to the conference.", "questions_to_authors": "1. Can you try more open sourced model, e.g., maybe trying to see how scaling helps a model becomes a better poker player?\n2. You idenfiy couple of important ability in the appendix \"Gap between GPT3.5 and 4\". One of them is Long Prompt Handling. As we all know there many recent attempts on increasing context length. It would be interesting to benchmark the performance of LLM playing poker vs their performance on long context handling, e.g., lost in the middle benchmark", "extra_scores": {}}, {"review_id": "7NauiYtQou", "reviewer": "Reviewer_g5Lv", "summary": "This manuscript provides an agent for imperfect information game based on the Theory of Mind. The authors demonstrate the effectiveness of the proposed methods on Leduc Hold’em and make extensive analysis for the proposed method and its interaction with a wide range of baseline methods.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in sharing critical feedback regarding our work.  We provide the following response to address your concerns about results limitations, and thus respectfully hope you can consider the response to the final decision.\n\n**1. Can you provide detailed information on the baseline training?**\n\nA: Thanks. We trained our baselines using the default settings of RLCard (https://github.com/datamllab/rlcard/) and continued training them until convergence (check one by one). For example, we trained CFR+ (used in our experiments) for 2,000,000 epochs and observed convergence at around 1,000,000 epochs. Similarly, we trained DMC for 1,000,000 episodes, noting convergence at approximately 600,000 episodes, and DQN for 100,000 episodes. Therefore, the baselines used in our experiments have already reached their performance limits.\n\n**2. Can you test the exploitability of each method?**\n\nThank you very much for your time and effort.\n\nFirst, we agree with you that randomness and complexity play a significant role in imperfect information game evaluation, and some methods may perform well in some instances, but poorly in others. Because of the limited time and the expensive cost of calling GPT-API, we do not do exploitability evaluation, but we designed two types of experiments to compare Suspicion Agent with baselines and remove game instance randomness:\n\n**Reducing the randomness of Random Seeds:**\n\n**Reducing the randomness of Card Strength/Position: (which is also used in the AIVAT for imperfect information game evaluation [1])**\n\nGiven these two designs, we believe that we can alleviate the effect of randomness in imperfect information games. Given the results in Table 1 and Table2 in the main paper: Suspicion Agent achieves superior performance over learning-based baselines except CFR+ in all 100 random games, 50 games in position 0 and 50 games in position 1. They are the clear evidence to show the advantages of our suspicion agent.\n\nLastly, we would like to highlight that our study is the first to consider the interaction between two agents by introducing the theory of mind using LLM. We respectfully hope that you reconsider your decision in light of these contributions and the innovative aspects of our study.\n\n[1] Burch, Neil, et al. \"Aivat: A new variance reduction technique for agent evaluation in imperfect information games.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The manuscript is easy to follow.\n* The analysis is complete.", "reasons_to_reject": "* It’s unclear how the baseline policy is obtained. Typically the strength of the the policy heavily depends on the number and the diversity of samples they collect during training (even for CFR+), and it is unclear from the current manuscript if the baseline has reached their limit,\n* It seems that the authors have some mis-understanding on the game theory. There is no point to compute the average in Table 2, as the agent can suffer significant loss with other types of players/other game instances. We can only argue that one opponent can beat another opponent in some game instances, and one opponent can beat another opponent if the game instance is sampled from some certain distribution. Similarly for Table 4. Win rate is also not a meaningful criterion. What we really do in game theory is to test the exploitability of the policy, which should be simple for the game with small scale like Leduc Hold’em.", "questions_to_authors": "* Can you provide detailed information on the baseline training?\n* Can you further test the exploitability of each baseline method?", "extra_scores": {}}, {"review_id": "4EPHW6MSoe", "reviewer": "Reviewer_GomY", "summary": "The paper propose Suspicion-Agent, a GPT-4 powered method for playing imperfect information games. A core feature of Suspicion-Agent is that it leverages second order theory of mind (ToM) for planning. Suspicion-Agent is evaluated on the Leduc Hold’em, which is a smaller version of Limit Texas Hold'em, against four typical decision-making methods, including NFSP, DQN, DMC, and CFR+. Experimental results show that Suspicion-Agent outperforms all baselines except CFR+.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments. We provide the following response to address your concerns about paper writing, and we respectfully hope you can consider the response to the final decision.\n\n**1.However, I only find the results on the Leduc Hold’em game.**\n\nA: We will make it clear in our final version that we quantitatively evaluate our method on  Leduc Hold’em game, and qualitatively observe its adaptability across three imperfect information games.\n\n**2. Avalon is also an imperfect information game.**\n\nA: Thanks for the recommendation. The recommendation papers are contemporaneous works, and they are not published, so we do not include them in our original submission, but we are **happy to include the detailed discussion with [1,2] in our camera-ready paper.**\n\n**3. Related Work has to be moved to the main content.**\n\nA: We will include it in our camera-ready version with a 1 page extension.\n\n**4. None of works on playing imperfect information games with LLMs is mentioned in introduction.**\n\nA: In the original submission, our work is indeed the first one, and are contemporaneous works with [1,2]. We would like to delete the claim in our camera-ready version if it makes the paper solid.\n \n**5. Compared with existing work, the method in work is incremental.**\n\nA: As the pioneering work to design the prompting system to enable LLM for poker games, we still believe our proposed methods and the preliminary public experimental study and data are valuable for the community.\n\na. We experimentally demonstrate that even with pre-trained data, **LLM itself cannot play various imperfect information games well as Table 3 shows ((-72 no ToM vs +24 second-order ToM).** Then we introduce the prompting system to enable large language models like GPT-4 to play various imperfect information games using only the game rules and observations without any extra training, this may inspire more LLM-based subsequent works.\n\nb. We comprehensively identify and discuss the current limitations of employing large language models in imperfect information games, contributing valuable insights for further AI-agent research.\n\nc. We are preparing to make all our code and interactive data publicly available. We hope this will catalyze the development of more efficient models in the field of imperfect information games.\n\n\n\n[1] Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation.\n\n[2] Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem of playing imperfect information games with LLMs is interesting and of great value.\n2. The proposed Suspicion-Agent is effective compared with baselines.", "reasons_to_reject": "1. It is stated in abstract that Suspicion-Agent \"demonstrates remarkable adaptability across **a range of imperfect information card games**\" and \"the capabilities of Suspicion-Agent across **three different imperfect information games**\" are qualitatively showcased. However, I only find the results on the Leduc Hold’em game.\n2. Important related works are not discussed [1,2]. Theory of mind is also leveraged in [1] and Avalon is also an imperfect information game. [1] has to be discussed or even used as a baseline.\n3. I think at least the core part of Related Work has to be moved to the main content. Placing the entire Related Work section in appendix is not acceptable. Note that the reviewers are not required to read the appendix section (https://colmweb.org/cfp.html#:~:text=Authors%20may%20use%20as%20many%20pages%20of%20appendices%20(after%20the%20bibliography)%20as%20they%20wish%2C%20but%20reviewers%20are%20not%20required%20to%20read%20the%20appendix). \n4. None of the existing works on playing imperfect information games with LLMs is mentioned in introduction.\n5. Compared with existing work, the method proposed in this work is somewhat incremental.\n\n[1] Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation.\n\n[2] Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper considers LLM agents to play imperfect information games and their ability to display theory-of-mind capability. In addition to the novel ideas, the paper also displays extensive experiments with the game of Poker and show favorable performance of their method vs. other previously considered agents. 
I find the contribution to be novel and important, clear accept.
Decision: Accept