Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization (2024)

Abstract: Large Language Models (LLMs) exhibit robust problem-solving capabilities for diverse tasks. However, most LLM-based agents are designed as specific task solvers with sophisticated prompt engineering, rather than agents capable of learning and evolving through interactions. These task solvers necessitate manually crafted prompts to inform task rules and regulate LLM behaviors, inherently incapacitating to address complex dynamic scenarios e.g., large interactive games. In light of this, we propose Agent-Pro: an LLM-based Agent with Policy-level Reflection and Optimization that can learn a wealth of expertise from interactive experiences and progressively elevate its behavioral policy. Specifically, it involves a dynamic belief generation and reflection process for policy evolution. Rather than action-level reflection, Agent-Pro iteratively reflects on past trajectories and beliefs, "fine-tuning" its irrational beliefs for a better policy. Moreover, a depthfirst search is employed for policy optimization, ensuring continual enhancement in policy payoffs. Agent-Pro is evaluated across two games: Blackjack and Texas Hold’em, outperforming vanilla LLM and specialized models. Our results show Agent-Pro can learn and evolve in complex and dynamic scenes, which also benefits numerous LLM-based applications 1.

heading: 1 Introduction
text: Designing a human-level agent with robust problem-solving abilities has long been a vision in the academic community. This necessitates the agent to possess learning and generalization capabilities across a diverse array of tasks. The advent of Large Language Models (LLMs) (Chowdhery
*The first two authors have equal contributions. This work was done when the first author was an intern at Institute of Software, Chinese Academy of Sciences.
†Corresponding author. 1Code is in https://github.com/zwq2018/Agent-Pro
et al., 2022; Zhang et al., 2022a; Zeng et al., 2023; Touvron et al., 2023a; OpenAI, 2022, 2023; Touvron et al., 2023b) has shed light on this vision, especially they can be rapidly generalized across a wide range of tasks with only a few demonstrations (Wei et al., 2022a,b). Benefiting from this, many systems built upon LLMs have showcased markedly enhanced performance such as questionanswering (Yao et al., 2022; Schick et al., 2023; Shen et al., 2023a; Zhang et al., 2023d; Madaan et al., 2023; Zhang et al., 2022b, 2023e), code generation (Hong et al., 2023; Wu et al., 2023b), and real-world application (Qin et al., 2023b; Zhang et al., 2023b).
Despite these achievements, building a humanlevel agent remains a challenging endeavor. First, most LLM-based agents are designed for specific tasks through sophisticated prompts, including detailed task descriptions and behavioral specifica-
5348
tions. However, numerous real-world tasks, e.g., business, company negotiations, and security, are more intricate with imperfect information, necessitating laborious efforts to design strategic behavior.
Second, most LLM-based agents do not consider interacting with task scenarios, and more critically, cannot learn from past experiences and evolve their behavioral strategies during interactions. In contrast, humans often learn and adjust their behaviors through interaction, especially in novel scenarios. In light of these, a promising yet under-explored topic emerges: Can LLM-based agents learn and elevate behavioral strategies by interacting with the environment like humans? It should be an indispensable ability of a human-level agent.
Recently, numerous studies (Shinn et al., 2023; Wang et al., 2023a; Zhang et al., 2023a; Zhao et al., 2023; Qian et al., 2024) undertake intriguing explorations, e.g., utilizing feedback for self-correction at the action-level. Besides, several efforts also explore deploying LLM in interactive games, including StarCraft (Ma et al., 2023), Minecraft (Wang et al., 2023a), strategy-based gaming (Bakhtin et al., 2022; Guo et al., 2023a; Xu et al., 2023a,b).
Similarly, we first evaluate LLM-based agents with the self-correction strategy in dynamic interactive scenarios, such as multi-player Texas Hold’em, which is a zero-sum game with imperfect information. However, we observe that it loses most of the rounds to its opponents, even the most advanced LLMs. Upon examining its reasoning thoughts and actions, we find that it often adopts irrational behaviors and is unable to deduce effective strategies from long action sequences.
To answer the above question, the Theory of Mind (ToM) (Premack and Woodruff, 1978) may provide some insight. In this framework, each human develops perceptions of himself (self-belief) and the external environment (social-belief) in the social context, and then grounds their decisions on these beliefs, or adjusts incorrect beliefs in response to external feedback. Inspired by this, we advocate Agent-Pro: a LLM-based Agent with Policy-level Reflection and Optimization. AgentPro is endowed with the capacity to learn and evolve within environments, i.e., autonomously reflect on past experiences, calibrate its beliefs about itself and the environment, and optimize its behavior policy without parameter tuning.
Concretely, as shown in Figure 1, an LLM-based agent involves an LLM as the foundational model and some instructions in the prompt to regulate
its behavior (policy). Upon observing partial information from the scenarios, Agent-Pro first updates its self-belief and world-belief, then makes decisions based on these beliefs. After exploring tasks, Agent-Pro performs a policy-level reflection and optimization on past trajectories, beliefs, and results. It autonomously "fine-tunes" its beliefs, searches for useful prompt instructions, and consolidates them into a new behavior policy.
The experiments in two zero-sum games, Blackjack and Texas Hold’em, demonstrate that AgentPro, after evolution, can defeat vanilla LLMs and specialized models, improving the game’s payoffs. It indicates that Agent-Pro enhances its capabilities through interaction and reflection without human guidance. As depicted in Figure 1, the initial prompt is quite simple (Left Bottom), but after learning and evolution, the Agent-Pro generates many practical instructions (Right Bottom). For instance, Agent-Pro records estimations of each opponent’s style in Task Description and adds specific Goals, Strategies in Behavior Policy.
Our Agent-Pro is different from previous strategies, like Reflexion (Shinn et al., 2023). Firstly, Policy-level reflection is designed for policy updating in long-horizon tasks. It is aimed at longhorizon policy updating rather than immediate action correction. The input is a sequence of actions and delayed feedback, while the output is an optimized strategy, rather than a specific action. Therefore, policy-level reflection corrects irrational beliefs and optimizes the old policy into the new one. As introduced in Section 3.2, our policy-level reflection includes belief calibration, policy updates by refining behavioral guidelines and world modeling, and policy verification.
Besides, we innovatively distill long-term memory into Behavioral Guidelines and World Models through prompt optimization. Most previous strategies store historical experience as verbal long-term memory and use it for text-based reasoning. In contrast, we further construct an optimizable policy from long-term interactions, i.e., Behavioral Guidelines and Environmental Models. This includes self-summarized game objectives and rules, effective strategies derived from reflection, and demonstrative trajectories. The contributions of our work are as follows:
• We introduce Agent-Pro, a framework capable of learning and evolving within interactive games, empowering LLM-based agents to effi-
5349
ciently adapt to more complex dynamic tasks.
• We devise a belief-aware decision-making process with self and world-belief, enhancing its capabilities for intricate tasks, i.e., generating more rational actions in interactive scenarios.
• We utilize policy-level reflection and optimization to iteratively update prompt instructions, which empower Agent-Pro to progressively evolve from a novice to a skilled veteran with many strategic behaviors.
• After learning, Agent-Pro is evaluated in multiplayer games and defeats specialized models, gaining notable progress. It develops strategic skills like humans, e.g., actively cutting losses, bluffing, or disguising to influence others.
Not just in card games, similar scenarios abound in the real world as well. Through self-learning and evolution, Agent-Pro can enhance deployment effectiveness in those scenarios, expanding the capability boundaries of LLM-based agents notably.

heading: 2 Problem Definition
text: Our study focuses on multi-player imperfect information interactive games, with two characteristics:
Imperfect Information. Unlike perfect information games (e.g., chess), imperfect information scenarios are characterized by agents only having access to their own states and public information, without knowing the states of others, e.g., in Texas Hold’em, players cannot observe others’ cards, which is dissimilar to many LLM-based tasks.
Dynamic Interaction. There may be multiple agents in the environment, and they may influence each other. That is, the actions of one agent may lead to changes in the environment, which are unpredictable for other agents.
In real-world contexts, such as competition, company negotiations, and security, these scenarios can often be abstracted as multi-agent interactive scenarios with imperfect information. Research on this can offer viable solutions to many real-world problems. We select two games as our testbed: Blackjack and Limit Texas Hold’em with multiplayer. Please refer to Appendix B for details.

heading: 3 Methods
text: To empower agents to learn in interactive contexts, a typical method is reinforcement learning (Zhang et al., 2021, 2022c). This involves
exploring highly rewarding actions through trial and error and solidifying these experiences into model parameters. Nonetheless, the training overhead for LLMs is substantial. Therefore, we employ a gradient-free "exploration-learning" strategy that enables LLM-based agents to learn through in-context learning. Specifically, we convert the policy learning into a prompt optimization process, i.e., LLM autonomously reflects and updates the prompt’s instructions based on its exploration experience, solidifying the high-reward strategies into the prompts. Benefiting from LLM’s generalization capabilities, our agent can summarize rules and learn specialized skills from a small number of samples like humans, making it well-suited for many real-world scenarios.
As shown in Figure 2, Agent-Pro comprises three components: (1) A Belief-Aware DecisionMaking process. It first updates beliefs about the world and itself, rendering more coherent and consistent decisions in dynamic and imperfect game scenarios. (2) A Policy-Level Reflection. Rather than reflecting on a single action, our design empowers LLMs to self-reflect on irrational beliefs from failed experiences. Then, it summarizes these erroneous beliefs into specific prompt instructions, like acting strategy (Behavioral Guideline), descriptions of the task world, and conjectures about other players (World Modeling), etc, which can calibrate its incorrect beliefs, evolving into a better policy. (3) A Prompt Optimization process ensures that the agent’s policy evolves for a higher payoff following a DFS-based search.

heading: 3.1 Belief-aware Decision-Making Process
text: To develop an LLM-based agent better suited for interactive environments, we draw inspiration from the Theory of Mind (ToM) (Premack and Woodruff, 1978; Li et al., 2023b; Guo et al., 2023a). In this framework, human condenses perceptions of themselves (self-belief) and the external environment (social-belief) and then ground their decisions on these beliefs, or adjust incorrect beliefs in response to external feedback. We also design a belief-aware decision-making process for Agent-Pro, simulating human cognitive processes in social contexts.
First, we need to define the policy of an LLMbased agent, which refers to a specific behavioral strategy guiding the agent to interact and complete tasks. It often involves complex prompts designed by experts, covering task rules, strategies, and output formats. In a zero-sum game
5350
with K+1 players (assuming playing order is (op1, our, op2, .., opK)), we denote the policy of our agent as π with some observable information, containing agent’s private information st, public information ot, our own action at, and the actions of all opponents aop1t , a op2 t , .., a opK t , where t means t-th rounds of a game. Therefore a complete game trajectory spanning t rounds:
H0:t = {(s0, o0, aop10 , a0, aop20 , ..., aopK0 ), ...
(st, ot, a op1 t , at, a op2 t , ..., a opK t )}
(1)
As shown in Figure 2, when making a decision, Agent-Pro first generates a dynamic belief
ξ about itself (self-belief) and opponents (worldbelief) in natural language. Then, it predicts an action based on the latest beliefs. For instance, for Texas Hold’em, Agent-Pro’s understanding of its hand cards, plan, and potential risk constitutes its self-belief, while the conjectures about the opponents form its world-belief. These beliefs are updated in each decision-making cycle. Equipped with this, Agent-Pro can generate more coherent and consistent actions:
ξt+1, at+1 ∼ π(H0:t, st+1, ot+1, aop1t+1, ξt) (2) When a game is over, we acquire the observable
state R (e.g., private hand cards after showdown) and the final scores S of all players. The objective is to find an optimal π∗ to maximize S(our).
5351

heading: 3.2 Policy-Level Reflection
text: Equipped with an initial policy (a simple prompt) and a dynamic belief, Agent-Pro already possesses basic capabilities for game exploration. To further enhance Agent-Pro’s capabilities, we design a learning mechanism via a policy-level reflection.
Specifically, many text-based tasks have employed reflection strategies and immediate environmental feedback to correct prior actions. However, in many typical interaction scenarios with longer decision-making processes, action-level reflections are not directly applicable due to delayed feedback. Therefore, for such a long-horizon interaction process, Agent-Pro is instructed to focus on the rationality of beliefs and underlying behavioral policies rather than individual actions.
Belief Calibration As depicted in Figure 2, under the guidance of the current behavior policy, Agent-Pro generates actions based on self-belief and world-belief. If these beliefs are inaccurate, they may lead to irrational actions and eventual failure. Therefore, Agent-Pro examines the rationality of these beliefs based on the final results and reflectss on the reasons for the final failure. Correctness: Whether its beliefs about itself , the game , and its opponents align with the final results. Consistency: Whether each belief and action is self -contradictory. Rationality: Whether the beliefs accurately reflect the underlying intentions behind the opponents. Reasons: Reflect on why it lost to its opponents , which beliefs are problematic , and what the underlying reasons are.
Lastly, to calibrate the incorrect beliefs, AgentPro summarizes these reflections and analyses about itself and the external world into specific instructions: Behavioral Guideline and World Modeling, where the former represents generalized behavioral strategies for this task, and the latter signifies its understanding and conjectures about the game world. For instance, in Texas Hold’em, Agent-Pro summarizes the following contents:

heading: Behavioral Guideline
text: 1-Please summarize a detailed goal based on your reflection on beliefs. {Goal} 2-What strategy helps you build correct belief and win at similar .. {Strategy} 3-Can this game be considered a typical example for future ... {Demonstration} World Modeling 1-Accurately model each player to help build more precise beliefs about them , including action , and style.{Opponent} 2-Describe any game rules or details that are easy to overlook ...{Rule}
Agent-Pro summarizes high-level strategies within the Behavioral Guideline and describes the task and opponents in World Modeling. These instructions can calibrate previous incorrect beliefs and improve policy performance. The entire process can be formalized as follows:
Instructionn+1←LLM (Hn0:T ,{ξn1 , ξn2 ..},Rn,Sn) (3) where Hn0:T denotes a complete trajectory at the n-th match, {ξn1 , ξn2 , ...} denotes the belief sequence, Rn and Sn means the final results and score. Instructionn+1 denotes new generated Behavioral Guideline and World Modeling.
Verification After extracting these instructions, Agent-Pro verifies its efficacy. Agent-Pro incorporates these generated Behavioral Guideline and World Modeling into the prompt and then replays the same game again, i.e., the same opponents and initial conditions. If the final score improves, we retain them in the prompt. Otherwise, we regenerate a new one. If it fails to pass verification after three retries, we discard this trajectoryHn:
πn+1 V erify←−−−−πn ∪ Instructionn+1 (4)
where ∪ means incorporates new instructions into the previous prompt for πn+1. This new policy encompasses more effective instructions, empowering Agent-Pro to establish accurate self- and world beliefs and generate more rational actions.

heading: 3.3 DFS-based Policy Evolution
text: To iteratively update the policy, we devise a policy optimization process based on depth-first search (DFS). It encompasses a policy evaluation process to assess the generalization ability of the new policy in novel game scenarios and a search mechanism to progressively find a better policy.
Policy Evaluation Each time the policy is updated, Agent-Pro is required to evaluate the new strategies. This evaluation process is distinct from the previous Verification step, as the Verification repeatedly utilizes the "training" data for evaluation and can not ensure the generalizability of the new policy. Hence, Agent-Pro conducts a thorough assessment of the new policy in novel trajectories. Besides, it is imperative to eliminate the influence of random factors when policy evaluation, e.g., a poor initial hand due to bad luck or an unfavorable playing order.
Therefore, we first randomly generate a new game for K+1 players. Then we sequentially
5352
swap both the hand cards and the playing order of each player, generating a total of (K+1)2 combinations. To eliminate randomness, we concurrently use these (K+1)2 games to evaluate Agent-Pro’s new policy. We calculate the average score over the (K+1)2 games for each player. Since the influences of hand-card quality and playing order are mitigated, the average score of all combinations can represent the true capabilities of each player. Lastly, we calculate the evaluating metrics:
∆= 1
(K+1)2
(K+1)2∑
j
[ Sj(our)−max
i Sj(opi)
] (5)
where i ∈ {1, . . . ,K} denotes the index of an opponent, and j denotes the index of the games within (K+1)2 combinations. The ∆ assesses its gains relative to the strongest opponent, providing a comprehensive evaluation in multiplayer gaming scenarios.
Policy Search Inevitably, sometimes the new policy does not bring an improvement in ∆ in the new scenario. In such cases, we employ DFS to search for a better policy from other branches (i.e., other candidate policies). As shown in Figure 2, when updating old policy πn, we generate B candidate policies {πn+11 , πn+12 ,..., πn+1B }, forming B branches. Then, we first calculate ∆n+11 for new policy πn+11 and compare it with ∆
n. If ∆n+11 is greater than ∆n, we accept this evolutionary. Otherwise, we reject πn+11 and consider π n+1 2 . If none of the B candidate policies πn+1 enhance AgentPro’s performance, we backtrack to πn and consider its sibling nodes πn2 . Similarly, Agent-Pro explores the environment using πn2 , then also updates B candidate policies and searches in a depth-first manner. Ultimately, we select the policy with the highest ∆ across the entire policy tree.

heading: 4 Game: Blackjack
text: Environment Settings We employ the RLCard (Zha et al., 2019) as our simulators for two games. We train two reinforcement learning agents as opponents: DQN (Mnih et al., 2015), and Deep Monte Carlo Search (DMC) (Zha et al., 2021). Please refer to Appendix A for more details.

heading: 4.1 Results
text: As shown in Table 1, we report the win rates of each agent against the dealer over 900 games. We also provide the results of RL-based models and a human player in Table C3 for reference.
Agent-Pro Significantly Surpasses the Baseline Agents Across most LLMs. The results show that Agent-Pro significantly surpasses most baseline agents with an average advantage of +4%. For example, On Qwen-72B and Llama2-70B, AgentPro significantly surpasses Reflexion with increases of +3.9% and +11%, respectively. For GPT-4, Blackjack is relatively simple, so the win rates of different strategies are quite similar.
What has Agent-Pro learned from evolution? Compared to ReAct and Reflexion, Agent-Pro is more robust. We find that this is due to the effective behavioral guidelines summarized by policy-level reflection. For instance, Agent-Pro summarizes two instructions as follows: 1-When you have achieved a relatively stable total hand value, choosing not to take risks is a good decision. 2-Analyze the dealer cards in World-belief,..., excessive risk-taking can lead to unfavorable outcomes... These self-summarized instructions can alert Agent-Pro to the risks associated with action Hit, thus making more rational decisions.
5353

heading: 4.2 Analyisis
text: Agent-Pro is More Rational than Baselines. We further analyze the Hit rates of the agents under different initial point totals, i.e., the sum of the initial two cards. The hit rate represents whether the agent is willing to take risks to draw cards. At this point, the player needs to consider both their own hand and the dealer’s hand to decide whether to take the risk. However, in Figure 3, we observe that the baseline seems to only focus on its own hand, with no significant difference in behavior when the dealer’s cards are high or low, whereas Agent-Pro is much more reasonable. For instance, for Agent-Pro, areas B1 and B2 show a clear difference. It tends to Stand when the dealer has high cards and Hit when the dealer has low cards. Because it believes the dealer is more likely to bust with high cards, making it not worth the risk for itself. We provide some detailed cases and evolution processes in Figures F2 to F5 to show their difference.

heading: 5 Game: Limit Texas Hold’em
text: Setups In Limit Texas Hold’em, each player has two private cards and chooses from four actions: Fold, Check, Call, Raise. We set up matches among four players: DQN, DMC, GPT-3.5, and X , where X represents the LLM-based agent we aim to evaluate, including Agent-Pro and baselines (Appendix A). The prompts for baselines and AgentPro in Appendices E.3 and E.4. To enable AgentPro to learn within the game, we employ a total of 167 "training" game hands and 20 evaluation hands. Please refer Appendix A.4 for detail.
Metrics Similar to Section 3.3, we sample 100 new game hands and allocate them to players. The players sequentially swap their hands and positions, generating 16 distinct permutations to eliminate the impact of chance and playing order. Lastly, we acquire 1600 games as the test set in total and calculate the average chip counts for four players. We provide detailed statistics in Table B1 regarding "training", evaluation, and test set.

heading: 5.1 Results
text: As shown in Table 2, we report the final chip counts of various LLM-based agents against the other three players (DQN, DMC, GPT-3.5). The results indicate that Agent-Pro consistently outperforms RL-based agents e.g., DMC, and surpasses other LLM-based agents across numerous LLMs.
Agent-Pro Surpasses LLM-based Agents and also Defeats RL-based Agents. We observe that Agent-Pro achieves significant progress on GPT3.5, GPT-4, and Llama2-70B, with an average score increase of +2 points. Besides, it surpasses specialized agents (DMC) on GPT-4, with an advantage of +3.2 points, and outperforms other LLM-based agents by a large margin (larger than 2.0 points). By analyzing the actions of Agent-Pro, we notice that it has learned to use multiple game techniques like humans. For instance, based on the analysis of the opponent’s style in the World Modeling, it may coerce some cautious players into folding by bluffing or sometimes it may disguise itself to entice aggressive opponents to raise their bets.
Belief Enhances Decision-making Capabilities in Dynamic Scenario. Even without the learning process (policy-level reflection), Agent-Pro also can improve Vanilla LLM’s performance by +0.9 points. For instance, on GPT-3.5 and GPT4, it led to improvements of +1 points and +1.3 points, respectively, which already slightly surpasses most LLM-based agents. This improvement stems from the dynamic belief, which enables agents to promptly capture updates in community cards, changes in opponents’ strategies, etc., thereby making more rational decisions. From the perspective of ReAct, our belief can also be seen as a dynamic thought process constructed based on the ToM framework, which endows agents with the ability to actively perceive internal and external beliefs and how they may change over time.
Besides, in Table B2, we explore whether our evolution process could be replaced by few-shot learning, i,e., we add some demonstrations to the prompt of Vanilla LLM, and evaluate its results. We find that failed game trajectories can slightly improve its effectiveness, but not as significantly as our evolution strategy. In Table B2, we also ablate the belief component from Agent-Pro but remain learning process. It shows that directly reflecting on the action sequence is quite unstable, and results in some vague and verbose behavioral instructions.

heading: 5.2 Analysis on Learning Process
text: We analyze the performance of Agent-Pro throughout the whole learning process. As shown in Figure 4, Agent-Pro is evaluated every 10 iterations.
Different LLM-based Agent-Pro Develops Diverse Strategies. We observe that the learning curves of the three Agent-Pros exhibit significant differences. Agent-Pro based on GPT-4 and GPT-
5354
3.5 rapidly improves their performance in the early stages of learning, with a maximum increase of +2.1 and 2.3 chips respectively. In contrast, Llama2-70B exhibits a dissimilar learning process, with performance initially declining in the first half and then improving (+0.6 chips) in the latter half. Analyzing the behaviors of the three agents, we discover that their strategic styles are entirely different. When dealt the same bad hand, the GPT-4-based Agent-Pro is relatively flexible and may bluff to probe opponents. GPT-3.5-based Agent-Pro tends to be cautious and may actively fold in the later stages, whereas the Llama-based Agent-Pro develops a highly conservative, risk-averse strategy. It
concedes at the beginning of the game by opting to Fold, thereby losing only the initial few chips.

heading: 5.3 Analysis on Policy Evolution
text: We manually select 20 challenging games (Details in Table C4). Then, we test three agents on these 20 games: Agent-Pro in the early learning phase (Agent-Pro-Early), Agent-Pro, and Vanilla LLM.
How the Strategy Evolved. We calculate the frequency of the most conservative action (Fold) and the most aggressive action (Raise) during the four stages of the game: PreFlop, Flop, Turn, River. As shown in Figure 5, we discuss how the strategy evolved. 1) The behavior of Vanilla LLM is rather rigid, Folding early in the game (Preflop stage) and ignoring subsequent community cards. 2) As learning progresses, Agent-ProEarly becomes more rational, with a noticeable decrease in Folding frequency during the Preflop stage. It can observe the public cards in subsequent phases before deciding to Folding. Besides, Agent-Pro-Early is more cautious, with a significant decrease in the frequency of Raising. 3) After learning, Agent-Pro exhibits flexible and proactive behavior. Compared to Agent-Pro-Early, its Fold frequency in Preflop continues to decrease, but the frequency of Raising in all four stages has rebounded. This result demonstrates the evolution of the strategy: from irrational to rational, from conservative to flexible. A detailed case study is shown in Appendices F and F.3.
Win More, Lose Less. As shown in Figure 6, we categorize the hands dealt to the agent into three types: strong, medium, and weak hands, and record their performance separately. The results show that Agent-Pro can win more chips with strong hands and lose fewer chips with weak hands compared to Vanilla LLM. Notably, Agent-Pro significantly im-
5355
proves performance (> 80%) with medium-strength hands, which indicates that it learns advanced skills, expanding its capability boundaries.

heading: 6 Discussions
text: Beyond card games, Agent-Pro holds the potential to handle other complex tasks. Although we have only tested Agent-Pro in Blackjack and Texas Hold’em tasks, we believe there is great potential for it to be generalized to many complex tasks.
• Firstly, the information set number for Texas Hold’em can reach up to 10162, far exceeding most tasks. Agent-Pro employs a belief-aware decision-making process to handle these challenges.
• Secondly, unlike most static environments, these strategic games are dynamic, with the game environment often changing in response to the opponent’s actions. Agent-Pro establishes self-belief and world-belief to depict the dynamic environment.
• Lastly, due to the unobservable states of opponents, deceit and bluffing behaviors frequently occur in the game. In such a scenario, AgentPro manages to learn and evolve strategies, surpassing trained RL-based agents. Agentpro adopts a policy-level reflection and optimization process to learn advanced game strategies.

heading: 7 Conclusion
text: We design an LLM-based agent, Agent-Pro, capable of learning and evolution in complex interactive tasks. It first constructs a dynamic belief for decision-making in uncertain scenarios. Then Agent-Pro reflects on its interactive experiences, corrects irrational beliefs, and summarizes its reflections into two instructions: behavioral guidelines and world descriptions for a new policy. Lastly, we evaluate Agent-Pro in two zero-sum
games and observe that its decision-making capabilities significantly improve after learning from historical experiences.

heading: Limitations
text: Agent-Pro has presented a novel paradigm for designing an evolvable LLM-based agent, but we want to highlight that there remain some limitations or improvement spaces: 1) Dependency: the learning process of the Agent-Pro heavily relies on the capability of the foundational model, especially its reasoning and reflection abilities. In Texas Hold’em, the GPT-4-based Agent-Pro approaches the level of human players and surpasses DMC clearly, while GPT-3.5 and Llama2-70Bbased Agent-Pro are still weaker than DMC, despite notable improvements. We plan to continue enhancing the capabilities of Agent-Pro based on weaker LLMs, aiming to achieve greater improvements even on smaller LLM models. 2) Performance: despite indispensable improvements, there may still be a significant gap between Agent-Pro and state-of-the-art algorithms (such as CFR plus) in gaming scenarios. In the future, we will continue to explore this issue and establish a set of benchmarks to evaluate their behaviors more comprehensively.

heading: Acknowledgments
text: This work was supported by the Nanjing Science and Technology Plan under Grants Y23002ZX01, the Key Research and Development Program of Zhejiang Province, China (No. 2024C01034).

heading: Appendix

heading: A Experiment Details

heading: A.1 LLMs
text: We employ the GPT-3.5-Turbo-0613, GPT4-0613, Llama2-Chat-70B (Touvron et al., 2023b) and Qwen-72B (Bai et al., 2023) to construct our agent. To make a fair comparison, we uniformly set the temperature to 1.0 for all experiments. For each test, we repeat it five times and report the average.

heading: A.2 Baselines
text: We compare Agent-Pro with many common LLMbased agent strategies, including Vanilla LLM, ReAct (Yao et al., 2022), Reflexion (Shinn et al., 2023), Self-Consistency (Wang et al., 2022), Multiagent Debate (Du et al., 2023).

heading: A.3 Setups For Blackjack
text: In Blackjack, players must decide to hit or stand based on their own hand, the dealer’s face-up card. We simplify our approach by not incorporating Verification and DFS-based Policy Evolution since Blackjack is simple with a small state space. We collect 50 failed games for policy learning. We evaluate Agent-Pro and baselines on newly sampled 900 games. All prompts are presented in Appendices E.1 and E.2.

heading: A.4 Detailed Setup For Texas Hold’em
text: The whole learning process is as follows: We first randomly allocate 500 game hands for 4 players and then select these failed game hands in which the agent loses a significant number of chips. We collect a total of 167 challenging game hands as the "training" set and 20 game hands as a development set for policy evaluation. Then AgentPro is instructed to conduct a learning process on these "training" instances, containing three phases: Exploration-Reflection-Evolution.
Exploration (§ 3.1): It randomly selects a game from "training" set to play with the latest policy and the belief-aware decision-making process.
Reflection (§ 3.2): If Agent-Pro loses to its opponents, it immediately performs Policy-Level Reflection on this game and then updates to the new policy after passing Verification.
Evolution (§ 3.3): We first sample 2 game hands from the development set to evaluate the new policy and calculate its ∆ with B=8 for DFS. The process
ends when the policy cannot be further improved, or all samples have been explored.
B Introduction of Two Games
We selected the following two games as interactive environments.

heading: B.1 Blackjack
text: Blackjack 2 , also known as 21, is a popular card game that involves a dealer and a player. Players must decide whether to hit or stand based on their own hand, the dealer’s face-up card, and the dealer’s one hidden card. The objective is to beat the dealer without exceeding 21 points. For this game, we observe whether LLM-based agents can make rational decisions under uncertain scenarios.

heading: B.2 Limit Texas Hold’em
text: Limit Texas Hold’em is a popular card game 3
. The game commences with each player being dealt two private cards, which belong exclusively to the player and remain hidden from the others. Five community cards are then dealt face-up in a series of stages: a three-card Flop, followed by a single card on the Turn and another single card on the River. The player can choose from four actions: Fold, Check, Call, Raise. They aim to construct the best five-card poker hand possible using any combination of their private cards and community cards.

heading: B.3 The Challenging of two Games
text: Two games can evaluate the agent’s capabilities from multiple dimensions:
Handling Uncertainty in Environment: Both games are imperfect information games and be used to assess the performance of LLM-based agents in face of uncertainty. For instance, in the game of Blackjack, the card hidden by the dealer introduces significant uncertainty. The agent needs to assess the risk and make decisions accordingly.
Addressing Dynamic Environment: Most realworld scenarios are dynamic rather than static. This requires the agent to capture environmental changes and adapt to them. For instance, in Texas Hold’em, in addition to one’s own hand, the actions of opponents also greatly influence the agent’s decisions. We evaluate whether Agent-Pro can handle such dynamic environments.
2https://en.wikipedia.org/wiki/Blackjack 3https://www.winamax.fr/en/poker-school_rules_
limit-texas-hold--em
5361
Game #Train #Dev #Test Blackjack 50 - 900 Texas Hold’em 167 20 1600
Table B1: The sample sizes of the Training, Development, and Testing sets for the two games, where the training set is utilized for exploration and reflection, the development set for policy evaluation, and the test set for assessing the effectiveness of all methods.
Strategy DQN DMC GPT-3.5 Agent Vanilla LLM -2.2 1.7 -0.9 1.4 - w/ 3 win shots -2.4 2.6 -1.1 0.9 ↓0.5 - w/ 3 lose shots -2.6 1.8 -1.2 2.0 ↑+0.6 - w/ 3 win+3 lose shots -1.9 1.9 -1.5 1.5 ↑+0.1 Agent-Pro -3.9 1.1 -1.5 4.3 ↑2.9 - w/o Belief -3.3 1.5 -0.7 2.5 1.1 - w/o Learning -3 1.5 -1.2 2.7↑1.3
Table B2: Up: We compare the performance of AgentPro with Vanilla LLM with few-shot demonstrations. Each demonstration contains a complete trajectory and final results. Down: We ablate the dynamic belief or learning process from Agent-Pro and evaluate its results.
Addressing Complexity: Blackjack is relatively simple, with an InfoSet number of 1000. In contrast, multi-player Limit Texas Hold’em is very complex, with its Infoset number reaching up to 1014 (Zha et al., 2019). We analyze Agent-Pro’s learning capacity in such intricate scenarios.

heading: C Complementary Experiments
text: To better investigate the performance of Agent-Pro, we design some ablation experiments.

heading: C.1 Whether Few-shot Learning Can Handle Such Complex Interaction
text: First, we compare Agent-Pro with Few-shot Agent: we randomly select some winning and losing game trajectories and their final results as demonstrations in the prompt. Then we evaluate them on the test set. As shown in Table B2, we observe that winning trajectories seem to have no effect (- 0.5), while losing trajectories can slightly improve the final performance (+0.6). This phenomenon is quite intriguing, indicating that these winning demonstrations do not seem to enhance the agent’s decision-making ability in such complex scenarios. This may be because these winning demonstrations are relatively simple for the vanilla agent, which is already capable of winning these games, so including them in the prompt does not provide it with any additional insights. In contrast, those failing
Strategy DQN DMC Human Win-rate ↑(%) 40.0 41.9 37.1
Table C3: We evaluate two RL-based agents, and the human player on the same 900 games.
trajectories instead promote agents to reflect and adjust their behaviors, improving the final results.

heading: C.2 Policy-Level Reflection without Belief
text: We ablate the dynamic belief module, i.e., conducting policy-level reflection directly on the action sequences, state sequences, and final results. Then it also summarizes prompt instructions for policy updates. As shown in Table B2, we observe that after removing, Agent-Pro’s chips drop from 4.3 to 2.5, although there is still an improvement of +1.1 compared to Vanilla LLM. Upon closely examining the Behavioral Guideline and World Modeling it generated, we observe that compared to Agent-Pro, its content is rather vague and verbose, lacking in specificity and conciseness. These results indicate that dynamic belief can enhance both decision-making and policy-level reflection capabilities. Below, we provide two similar instructions, one from Agent-Pro and the other from Agent-Pro without Belief.

heading: A Learned Instruction From Agent-Pro
text: When holding a weak hand , adopting a conservative approach and waiting for the flop can be wise. This strategy allows for the possibility of the community cards improving your hand. However , if the flop doesn ’t enhance your hand ’s strength , folding to minimize losses becomes the prudent choice. During this period , maintaining a low profile and avoiding aggressive actions like raising is advisable.

heading: A Similar Instruction From Agent-Pro w/o Belief
text: In situations where the strength of one ’ s hand isn ’t exactly what one might call robust or particularly promising , it could potentially be somewhat beneficial , or at least not entirely disadvantageous , to entertain the notion of adopting a stance that leans more towards the side of caution .... the unveiling of the community cards .....

heading: C.3 Detailed Analysis Experiments
text: As shown in Table C4, we manually select 20 challenging sets of hands, each with a significant difference in the hands of four players, and then assess the performance of Agent-Pro and Vanilla LLM.
5362
Hand Strength
Hand Community Cards
DQN DMC GPT3.5 Agent Flop Turn River
Strong
H5 S4 D6 DQ S9 C2 CA HA H7 CQ CT H3 S3 DK S5 HK D2 S7 H4 DA HA DQ D9 DT C6 D7 D3 C8 HA HT H5 S9 DQ DJ D4 CK H7 CQ C5 SA H9 C6 S8 S3 SJ HT CK D7 C5 C4 C3 D2
Moderate HJ CQ S7 SA C3 D5 H3 CA C2 H9 S3 D9 C8 H5 C5 DJ H9 S6 D2 HK H2 DA DK SK C3 H8 H5 D6 DT CT C9 C4 S9 C8 S6 S7 HT HQ HJ C3 D5 H3 CA HJ CQ S7 SA C2 H9 S3 D9 C8
Weak
S3 SJ HT CK SA H9 C6 S8 D7 C5 C4 C3 D2 S3 C8 H7 S2 DA CA D7 D5 H8 HJ SJ ST D9 DK D5 SJ C6 D9 S3 H2 C8 CA CQ CT D8 C2 H5 H8 HA S9 C6 D9 C5 H3 D3 C3 HQ S3 DA HA S9 C6 D9 C5 H3 H5 H8 D3 C3 HQ S3 DA H2 C8 DK D5 SJ C6 D9 S3 CA CQ CT D8 C2 H7 S2 DA CA D7 D5 S3 C8 H8 HJ SJ ST D9 HA HT H5 S9 DQ DJ D3 C8 D4 CK H7 CQ C5 DA HA DK S5 HK D2 S7 H4 DQ D9 DT C6 D7 DT CT C9 C4 S9 C8 H5 D6 S6 S7 HT HQ HJ D6 DQ S9 C2 CA HA H5 S4 H7 CQ CT H3 S3 HK H2 H5 C5 DJ H9 S6 D2 DA DK SK C3 H8
Table C4: Each card consists of a letter representing the suit ("S", "H", "D", "C") and a number representing size ("2", "3", "4", "5", "6", "7", "8", "9", "T", "J", "Q", "K", "A"). Among them, "S" represents Spade, "H" represents Heart, "D" represents Diamond, "C" represents Club, and "T" represents 10.

heading: D Related Works

heading: D.1 LLM-based Application
text: Large language models (LLMs), pre-trained on extensive corpora, have demonstrated robust language comprehension and reasoning capabilities. Benefiting from this, researchers have designed a plethora of agent systems built upon LLM, achieving promising results (Xi et al., 2023). Schick et al. (2023); Wu et al. (2023a); Shen et al. (2023a); Wu et al. (2023c, 2024) have harnessed the planning capabilities of LLMs to invoke specialized models and tools for task-solving. Some open-source projects, e.g., AutoGPT4, gentGPT5, BabyAGI6, BMTools7, ChatArena8, LangChain9 have developed an LLM-based assistant. Further, (Qin et al., 2023a,b; Shen et al., 2023b) have empowered LLM to autonomously invoke the APIs for daily life scenarios. Besides, leveraging the code generation capabilities of LLMs, researchers have designed multi-agent collaborative systems (Li et al., 2023a; Chen et al., 2023b; Hong et al., 2023; Wu et al., 2023b) for complex tasks, such as software de-
4https://github.com/Significant-Gravitas/ Auto-GPT
5https://github.com/reworkd/AgentGPT 6https://github.com/yoheinakajima/babyagi 7https://github.com/OpenBMB/BMTools 8https://github.com/chatarena/chatarena 9https://github.com/hwchase17/langchain
velopment. Unlike these task-specific agents that require manually specified behavior protocols, our agents can understand tasks through interaction with the environment. It can optimize its behavioral strategy from past experiences, accomplishing the task more effectively.

heading: D.2 LLMs For Interactive Scenarios
text: Beyond these applications, LLMs have also been utilized in interactive settings (Durante et al., 2024). ReAct (Yao et al., 2022) integrates reasoning, action, and observation into the problem-solving process. Park et al. (2023) introduces generative agents that can simulate human behavior. Fu et al. (2023) show LLMs can improve each other in a negotiation scenario. Zhao et al. (2023); Chen et al. (2023a) propose an experiential learner gathering experiences and extracting from a collection of training tasks. Fan et al. (2023) explored the capability of LLMs to make rational decisions in gametheoretic scenarios. Besides, some studies have designed sophisticated LLM-based agents for largescale games, including StarCraft (Ma et al., 2023), Minecraft (Wang et al., 2023a; Gong et al., 2023), Leduc Hold’em (Guo et al., 2023a), strategy-based gaming (Bakhtin et al., 2022; Xu et al., 2023a; Wang et al., 2023b; Xu et al., 2023b; Lorè and Heydari, 2023) and application for legal contexts (Wu et al., 2020, 2022).
5363
D.3 Improving the Quality of LLM Responses
Enhancing the quality of responses from LLMs has garnered significant attention within the community. We categorize the strategies into two methodologies: 1. Developing superior reasoning architectures. First, Chain-of-Thoughts (Wei et al., 2022b) elicits LLM’s reasoning ability. Works as Least-toMost (Zhou et al., 2022a), Tree of Thoughts (Yao et al., 2023), Graph of Thoughts (Besta et al., 2023) have explored diverse problem-solving procedures and reasoning architectures, significantly enhancing the performance of LLM-based agents. 2. Refining the output of LLMs. Researchers have proposed post-hoc prompting strategies to iteratively refine the outputs of LLMs (Pan et al., 2023), including Reflexion (Shinn et al., 2023), Self-Refine (Madaan et al., 2023; Paul et al., 2023; Huang et al., 2022), Self-Contrast (Zhang et al., 2024), etc. However, these self-correction strategies are performed at the action-level, whereas our agent operates at the policy-level, making it more suited for interactive environments.
Additionally, Supervised Fine-Tuning (SFT) offers another avenue for enhancing LLMs, though it depends on human-annotated data. Recently, Chen et al. (2024) introduced Self-Play fIne-tuNing (SPIN), a novel fine-tuning approach. SPIN’s foundation is a self-play mechanism that enables the LLM to refine its abilities by engaging with its own variations.

heading: D.4 Automatic Prompt Optimization
text: In addition to optimizing the outputs of LLMs, many researchers also enhance the performance of LLMs by searching for a more effective prompt (Zhou et al., 2022b; Hsieh et al., 2023; Guo et al., 2023b; Wang et al., 2023c). APO (Pryzant et al., 2023) emulates the process of gradient optimization. It calculates the "gradients" of the current prompt by analyzing the instances that are inaccurately predicted by this prompt. Furthermore, Yang et al. (2023) and Ye et al. (2023) evaluate each candidate prompt using the training set and iteratively optimize the prompts based on the evaluation results. Cheng et al. (2023) train a Sequence-to-Sequence model to translate an imperfect prompt into a better one. Brooks et al. (2023) and Zhang et al. (2023c) combine reinforcement learning with prompt updating, demonstrating promising results. We extend these prompt optimization techniques to more complex interactive
gaming environments, learning a robust behavioral strategy through policy-level reflection and search. Furthermore, our agent must constantly consider changes in the environment and the styles of opponents, thereby dynamically adjusting the content of the prompts.

heading: D.5 Benchmarking LLM-based Agent
text: To develop an effective evaluation method for LLMs and their capabilities as agents, numerous researchers have concentrated on establishing benchmarks. SmartPlay (Wu et al., 2023d) introduces a benchmark from 6 diverse games with language descriptors for visual observation. Clembench (Chalamalasetti et al., 2023) employs Dialogue Games as testing tools, enabling rapid evaluations across a broad of models. Furthermore, Liu et al. (2023) unveil AGENTBENCH, a comprehensive benchmark that outlines eight distinct environments to assess LLMs. These benchmarks play a crucial role in evaluating both LLMs and LLM-based agents. In our future work, we aim to further evaluate AgentPro utilizing these benchmarks.

heading: E Detailed Prompts
text: We provide detailed prompt designs for two games, including baselines and Agent-Pro in Appendix E.1, E.2, E.3 and E.4.

heading: F Case Study
text: As shown in Figure F4, F5, F2, F3 we provide four cases for Blackjack. We visualize the difference in their solving steps between Agent-Pro and ReAct when using Qwen-72B. Besides, we also provide four cases for Limited Texas Hold’em in Appendix F.3. These cases demonstrate that AgentPro, after learning, has significantly improved in understanding task rules, mastering techniques, and dealing with uncertain environments.
5364

heading: E.1 Baseline’s Prompts For Blackjack
text: Game Rule: ### Game Rules 1. Please try to get your card total to as close to 21 as possible , without going over , and still having a higher total than the dealer. 2. If anyone ’s point total exceeds 21, he or she loses the game. 3. You can only choose one of the following two actions: {"Stand", "Hit"}. If you choose to Stand , you will stop taking cards and wait for the dealer to finish. If you choose to Hit , you can continue to take a card , but there is also the risk of losing the game over 21 points. 4. After all players have completed their hands , the dealer reveals their hidden card. Dealers must hit until their cards total 17 or higher.
Game Information: The dealer ’s face -up card is {Dealer-Card}. The dealer has another hidden card. You don ’t know what it is. Your current cards are {Player-Card}.
---------- Prompt For Vanilla LLM -------------- You are a player in blackjack. Please beat the dealer and win the game. ### {Game Rules} ### {Game Information} ### Please output your action in following format: ###My action is {Your action}, without any other text.
------------- Prompt For Radical LLM -------------- You are an aggressive player of blackjack who likes to take risks to earn high returns. Please beat the dealer and win the game. ### {Game Rules} ### {Game Information} ### Please output your action in following format: ###My action is {Your action}, without any other text.
------------- Prompt For ReAct -------------- You are a player in blackjack. Please beat the dealer and win the game. ### {Game Rules} ### {Game Information} ### Please first think and reason about the current hand and then generate your action as follows: ###My thought is {Your Thought }. My action is {Your action }.
------------- Prompt For Reflexion -------------- You are a player in blackjack. Please beat the dealer and win the game. ### {Game Rules} ### {Game Information} ### Please first think about the current hand and then generate your action in following format: ###My thought is {Your thought }. My action is {Your action }. Assistant: {LLM Response}. My action is {LLM Response} ### Please carefully check the response you just output , and then refine your answer . The final output is also in following format: ###My thought is {Your thought }. My action is {Your action }.
5365

heading: E.2 Agent-Pro’s Prompt For Blackjack
text: Game Rule: ### Game Rules 1. Please try to get your card total to as close to 21 as possible , without going over , and still having a higher total than the dealer. 2. If anyone ’s point total exceeds 21, he or she loses the game. 3. You can only choose one of the following two actions: {"Stand", "Hit"}. If you choose to Stand , you will stop taking cards and wait for the dealer to finish. If you choose to Hit , you can continue to take a card , but there is also the risk of losing the game over 21 points. 4. After all players have completed their hands , the dealer reveals their hidden card. Dealers must hit until their cards total 17 or higher.
Game Information: The dealer ’s face -up card is {Dealer-Card}. The dealer has another hidden card. You don ’t know what it is. Your current cards are {Player-Card}.
------------- Prompt For Agent-Pro -------- You are a player in blackjack. Please beat the dealer and win the game. ### {Game Rules} ### {Game Information} ### {Behavioral Guideline: Goal, Strategy, Demonstration} ### {World Modeling: Rule Description} ### Please read the behavoiral guideline and world modeling carefully. Then you should analyze your own cards and your strategies in Self -belief and then analyze the dealer cards in World -belief. Lastly , please select your action from {" Stand", " Hit "}. ### Output Format: Self -Belief is {Belief about youself }. World -Belief is {Belief about the dealer }. My action is {Your action }. Please output in the given format.
------------- Prompt For Policy-Level Reflection -------------- ### {Game Rules} ### Game Record: {Game Record, Belief Sequences, Final Result} ### You are a seasoned blackjack expert , and you need to carefully reflect on the following record of this losing game:
Correctness: Whether its beliefs about yourself , the game , and the dealer align with the final results. Consistency: Whether each belief and action is self -contradictory. Reasons: Reflect on why you lost to your dealer , which beliefs and actions are problematic , and what the underlying reasons are.
### Output Format: I analyze this game as follows: {Your analysis about the game and belief }.
------------- Prompt For Generating Behavioral Guideline and World Modeling -------------- ### Game Record: {Game Record, Belief Sequences, Final Result} ### Policy -Level Reflection: {Reflection} Following the previous rigorous analysis , you should distill and articulate a set of Behavioral Guidelines and World Modeling. The Behavioral Guideline is about what you consider to be a more reasonable and effective behavioral strategy and suggestions. World Modeling is about the description of the game and the dealer.
Here are some suggestions for you:

heading: Behavioral Guideline
text: 1-Goal: Please summarize the detailed goal based on your reflection ... 2-Strategy: What kind of strategy can lead you to win in similar games ... 3-Demonstration: Can this game be considered a typical example to be preserved for future reference ... World Modeling 1-Rule-Description: Based on the recent reflection , describe any game rules or details that are easy to overlook ...
5366

heading: E.3 Baseline’s Prompts For Limited Texas Hold’em

heading: Game Settings:
text: 1- You are playing the Limit Texas poker game. In this game , there are 4 players from 0 to 3, and your role is player 3. 2- The number of chips every player has is infinite. 3- You just need to win new chips in the competition as much as possible. 4- The actions you can choose are [’call ’, ’raise ’, ’fold ’, ’check ’]
Game Information: Your current hands are {Private Cards}. The current stage: {Stage}. Public cards are {Public Cards}. Number of chips all players have invested are {Inveseted Chip List}. Available actions you can choose are {Available Actions}. Previous actions of all players are: {Preflop: Actions Sequences, Flop: Actions ...}.
---------- Prompt For Vanilla LLM -------------- You are a player in Limited Texas Hold ’em. Beat your opponents and win the game. ### {Game Rules} ### {Game Information} ### Output your action in following format: {" action ": " "} without any other text.
---------- Prompt For Aggressive LLM -------------- You are an aggressive player of limited Texas Hold ’em who likes to take risks to earn high returns. Please beat your opponents and win the game. ### {Game Rules} ### {Game Information} ### Output your action in following format: {" action ": " "} without any other text.
---------- Prompt For Conservative LLM -------------- You are a conservative player of limited Texas Hold ’em who is risk averse and prefers more certainty. Please beat your opponents and win the game. ### {Game Rules} ### {Game Information} ### Output your action in following format: {" action ": " "} without any other text.
---------- Prompt For ReAct -------------- You are a player in Limited Texas Hold ’em. Beat your opponents and win the game. ### {Game Rules} ### {Game Information} ### Please first think and reason about the current state and then generate your action as follows: ###My thought is {Your Thought}, and my action is {" action ": " "}
---------- Prompt For Reflection -------------- You are a player in Limited Texas Hold ’em. Beat your opponents and win the game. ### {Game Rules} ### {Game Information} ### Please first think and reason about the current state and then generate your action as follows: ###My thought is {Your Thought}, and my action is {" action ": " "}.
Assistant: {LLM Response}.
### Please carefully check the thought and the action you just output , and then refine your answer. The final output is also in the same format: ###My revised thought is {Your Thought }. My revised action is {" action ": " "}.
5367

heading: E.4 Agent-Pro’s Prompt For Limited Texas Hold’em

heading: Game Settings:
text: 1- You are playing the Limit Texas poker game. In this game , there are 4 players from 0 to 3, and your role is player 3. 2- The number of chips every player has is infinite. 3- You just need to win new chips in the competition as much as possible. 4- The actions you can choose are [’call ’, ’raise ’, ’fold ’, ’check ’]
Game Information: Your current hands are {Private Cards}. The current stage: {Stage}. Public cards are {Public Cards}. Number of chips all players have invested are {Inveseted Chip List}. Available actions you can choose are {Available Actions}. Previous actions of all players are: {Preflop: Actions Sequences, Flop: Actions ...}.
------------- Prompt For Agent-Pro -------- You are a player in Limited Texas Hold ’em. Beat your opponents and win the game. ### {Game Rules} ### {Game Information} ### {Behavioral Guideline: Goal, Strategy, Demonstration} ### {World Modeling: Rule Description, Opponents Description} ### Please read the behavoiral guideline and world modeling carefully. Then , following their instructions , you should analyze your own cards and your strategies in Self -belief and then analyze and reason about your opponents in World -belief. Lastly , output your action. ### Output Format: Self -Belief is {belief about youself }. World -Belief is {belief about the all opponents }. My action is {" action ": " "}. Please output in the given format.
------------- Prompt For Policy-Level Reflection -------------- ### {Game Rules} ### {Game Information} ### Game Record: {Game Record, Belief Sequences, Final Result} ### You are a seasoned Limited Texas Hold ’em expert , and you need to carefully reflect on the following record of this losing game:
Correctness: Whether its beliefs about yourself , the game , and the opponents align with the final results. Consistency: Whether each belief and action is self -contradictory. Rationality: Whether the beliefs accurately reflect the underlying intentions behind the opponents. Reasons: Reflect on why you lost to your opponents , which beliefs and actions are problematic , and what the underlying reasons are.
### Output Format: I analyze this game as follows: {Your analysis about the game and belief }.
------------- Prompt For Generating Behavioral Guideline and World Modeling -------------- ### Game Record: {Game Record, Belief Sequences, Final Result} ### Policy -Level Reflection: {Reflection} Following the previous rigorous analysis , you should distill and articulate a set of Behavioral Guidelines and World Modeling. The Behavioral Guideline is about what you consider to be a more reasonable and effective behavioral strategy and suggestions. World Modeling is about the description of the game and all opponents.
Here are some suggestions for you:

heading: Behavioral Guideline
text: 1-Goal: Please summarize the detailed goal based on your reflection ... 2-Strategy: What kind of strategy can lead you to win in similar games ... 3-Demonstration: Can this game be considered a typical example to be preserved for future reference ... World Modeling 1-Rule-Description: Based on the recent reflection , describe any game rules or details that are easy to overlook ... 2-Opponent-Modeling: Based on each opponent ’s action and the real hands shown at the end of the game , what do you think their style and strategy are ...
5368

heading: F.1 Cases For Blackjack
text: Agent-Pro is Better at Handling the Uncertainties of the Environment.
State ReAct Vs. Dealer Agent-Pro Vs. Dealer Initial Card The dealer’s face-up card is Spade 4. Your current cards are Heart 5, Club J. Thought My thought is I currently have a total of 15 points
with my two cards, which is not a very strong hand. The dealer’s up-card is a 4, which is a relatively low value card. Based on basic strategy in blackjack, I should hit if my total is less than 17, so I should take a hit to improve my hand. Self-Belief is I currently have a total of 15 points, with a Heart 5 and a Club J. I am slightly below the ideal total of 21, but I am also less likely to go bust if I draw another card. I think my best chance of winning is to stand and hope that the dealer goes bust or ends up with a lower total. World-Belief is The dealer currently has a Spade 4 showing, which means their total is likely to be between 4 and 14 points. Since the dealer must draw more cards if their total is below 17, they are at a higher risk of going bust. However, if the dealer has a strong hidden card, such as a face card or an ace, they could end up with a high total and win the game. Given this uncertainty, standing seems like a safer bet.
Action My action is Hit. My action is Stand. Next Round The final hand is Dealer: Spade 4, Diamond 4.
Your: Heart 5, Club J, Club 8 (23 point) The final hand is Dealer: Spade 4, Diamond 4, Club 8, Spade 7 (23 point). Your: Heart 5, Club J (15 point).
Result Lose Win
Figure F1: Some cases for Blackjack by ReAct and Agent-Pro using Qwen-72B. ReAct behavioral strategies are too aggressive. However, Agent-Pro guesses the dealer’s next move and makes a reasoned decision.

heading: F.2 Learning Process For Blackjack
text: State Before Learning After Learning Using 20 Failed Games World Modeling Empty Blackjack aims to reach a hand value close to 21 withoutexceeding it. Players choose to hit or stand based on
their hand strength relative to the dealer’s visible card. The dealer must hit until reaching 17 or higher.
Behavior Guideline Empty. Maximize winning chances while minimizing risks by as-sessing hand strength and dealer’s visible card. Employ
a cautious approach, balancing potential hand improvement with risks, considering probabilities of busting versus enhancement.
Generated self-belief I currently have a total of 15 points with my twocards, which is not a very strong hand.
I currently have a total of 15 points, with a Heart 5 and a Club J. I am slightly below the ideal total of 21, but I am also less likely to go bust if I draw another card. I think my best chance of winning is to stand and hope that the dealer goes bust or ends up with a lower total.
Generated World-belief The dealer’s visible card is a 4, there’s a chancethey could have a strong hand if their hidden
card is a 10-value card or an Ace.
Since the dealer must draw more cards if their total is below 17, they are at a higher risk of going bust.
Result Lose Win
Figure F2: Initially, both the World modeling and Behavioral Guideline of Agent-Pro are empty. At this point, Agent-Pro generates an ill-considered belief for the same card game. Through interaction and learning, Agent-Pro summarizes accurate World Modeling and Behavioral Guidelines. Faced with the same card game, the beliefs it generates are more reasonable.
5369
ReAct is Aggressive, While Agent-Pro is More Rational. State ReAct Vs. Dealer Agent-Pro Vs. Dealer Initial Card The dealer’s face-up card is Spade 8. Your current cards are Heart 10, Club 8. Thought My thought is The sum of my cards is currently
18. I am in a fairly safe position, not too likely to exceed 21 by hitting. However, since the dealer’s visible card is an 8, there is a good chance that the sum of the dealer’s cards will also be around 18. In order to increase my chances of winning, I need to aim for a higher total. Self-Belief is I currently have a total of 18 points, which is a relatively strong hand. I am confident that I can beat the dealer with this hand. However, I should be cautious not to exceed 21 points by hitting and risking busting. World-Belief is The dealer’s visible card is an 8, which is a moderate hand. It is likely that the dealer may still need to hit to reach a total of 17 or higher. Therefore, I should be careful not to risk busting by hitting and exceeding 21 points.
Action My action is Hit. My action is Stand. Next Round The final hand is
Dealer: Spade 8, Heart 4. (12 points) Your: Heart 10, Club 8, Diamond J (28 points) The final hand is Dealer: Spade 8, Heart 4, Diamond J (22 points). Your: Heart 10, Club 8 (18 points).
Result Lose Win
Figure F3: Some cases for Blackjack by ReAct and Agent-Pro using Qwen-72B. ReAct doesn’t consider the risk of exceeding 21 points.
Agent-Pro Can Better Understand the Rules of the Game. State ReAct Vs. Dealer Agent-Pro Vs. Dealer Initial Card The dealer’s face-up card is Club 6. Your current cards are Club A, Diamond 4. Thought My thought is that I currently have a total of 15
points (Ace counts as 1) and the dealer has 6 points showing. To win the game, I need to get as close to 21 as possible without going over. With the dealer’s current visible card being low, they may have to hit again to reach 17 or higher, increasing their chances of busting. Therefore, I should play conservatively and stand with my current total of 15 to avoid risking busting myself. Self-Belief is I have a total of 15 points, with an Ace that can be either 1 or 11, giving me some flexibility in my strategy. I should aim to get closer to 21 without busting. World-Belief is The dealer’s visible card is a 6, which means they could have a total of anywhere from 16 to 22, and may need to hit again. Given this information, I think my best strategy is to Hit and try to improve my hand.
Action My action is Stand. My action is Hit. Next Round The final hand is Dealer: Club 6, Club 5, Heart
6. Your: Club A, Diamond 4.
The dealer’s face-up card is Club 6. Your current cards are Club A, Diamond 4, Heart 6
Thought Lose Dealer: Club 6 + Club 5 + Heart 6 = 17 You: Club A + Diamond 4 = 5 or 15
Self-Belief is I currently have three cards with a total value of 21. This is a strong hand and I am confident in my chances of winning the game. World-Belief is The dealer currently has a visible card of 6, which is a relatively low value. However, I do not know the value of their hidden card, so I cannot be too complacent. Based on these factors, I believe that standing is the best course of action.
Action Lose My action is Stand. Next Round Lose The final hand Dealer: Club 6, Club 5, Diamond 9 (20). Your: Club A, Diamond 4, Heart 6 (21). Result Lose Win
Figure F4: Some cases for Blackjack conducted by ReAct and Agent-Pro using Qwen-72B. ReAct does not accurately understand the meaning of "A" in the rules of the game.
5370
Agent-Pro can Better Understand the Rules of the Game. State ReAct Vs. Dealer Agent-Pro Vs. Dealer Initial Card The dealer’s face-up card is Spade 3. Your current cards are Heart A, Club 9. Thought My thought is that I currently have a total of 10
points with my Heart A and Club 9. Given that the dealer’s visible card is a Spade 3, it is likely that the dealer has a total of between 13 and 16 points, since they would usually stand on a hard 17 or higher. Therefore, I should aim to get as close to 21 as possible without going over, while considering the possibility that the dealer may have a strong hand. Self-Belief is I have a total of 20 points, which is a strong hand. I am confident that I can beat the dealer with this hand. World-Belief is The dealer’s visible card is a 3, which is a weak hand. However, the dealer may have a hidden card that could potentially change the outcome of the game. Taking unnecessary risks could lead to an unfavorable outcome. Therefore, I should choose to stand and wait for the dealer to reveal their hidden card.
Action My action is Hit. My action is Stand. Next Round The dealer’s face-up card is Spade 3.
Your current cards are Heart A, Club 9, Club 5 The final hand is Dealer: Spade 3, Heart J, Club 5. Your: Heart A, Club 9.
Thought I have a total of 15 points from my two cards (Ace counts as 1), and the dealer shows a 3. Given this situation, there is a good chance that the dealer may bust if they draw more cards to reach at least 17. However, standing with 15 is risky as well since the dealer could still make a strong hand. Win Heart A + Club 9 = 20 Spade 3 + Heart J + Club 5 = 18 Action My action is Hit. Win Action My action is Hit. Win Next Round The dealer’s face-up card is Spade 3, Heart J,
Diamond 8 (21). Your current cards are Heart A, Club 9, Club 5, Spade K (25 or 35)
Win
Result Lose Win
Figure F5: Some cases for Blackjack by ReAct and Agent-Pro using Qwen-72B. Agent-Pro accurately understands the rules of the game and makes rational decisions.
5371

heading: F.3 Cases For Texas Hold’em

heading: Current game State:
text: Now your hand is [’HT’, ’CK ’], and the public cards are []. The number of chips all players have invested is [4, 0, 1, 2]. Currently , Preflop. Action Sequence: Player 0 raises. Player 1 folds. Player 2 folds. The Response of the Agent: Self-Belief: As Player 3, I hold the Ten of Hearts and the King of Clubs in hand , indicating a high card potential. However , no similar suits or consecutive cards suggest a weak hand in terms of potential straight or flush. Nevertheless , the hand still has some promising value given the presence of a King. Even though the community cards have not yet been dealt , it’s worth to consider a strategy of maintaining some aggression to minimize the number of opponents in the later stages. World-Belief: Player 0 seemingly adopted an aggressive strategy and raised already , potentially suggesting a strong hand , although it might also be a bluff. · · · However , whether Player 0 continues his aggressive strategy need to be closely watched. · · · Action: Given the current situation , I am not in a position to take an aggressive action , neither it is wise to fold at this point. Thus , I will choose to simply match the current bet to see the flop. Thus , my action would be Call. ------------------Next Round-------------------------------------------------Current game State: Now your hand is [’HT’, ’CK ’], and the public cards are []. The number of chips all players have invested is [4, 0, 1, 4]. Currently , Preflop. Action Sequence: Player 0 raises. Player 1 folds. Player 2 folds . Player 3 calls. Player 0 checks. The Response of the Agent: Self-Belief: As Player 3, my hand consists of the Ten of Hearts and the King of Clubs. These are not strong cards , and they do not form any promising hands such as a straight or flush. However , the King card is high rank and may provide some advantages in the later stage of the game. A conservative strategy might be the most reasonable at this stage , but a certain level of aggression may also be necessary to exert pressure on my opponent. World-Belief: Player 0 appears to be employing an aggressive strategy , having raised even before any community cards have been dealt. This behavior suggests that he may have a strong hand. However , it could also be possible that Player 0 is bluffing. Since players 1 and 2 have already folded , it appears that the game might turn into a head -on confrontation between Player 0 and me. · · · Action: Given this scenario , it would be wise to use a balanced strategy to counter Player 0’s aggressive play without risking too many chips on my not -so-strong hand. Therefore , I will choose {’action ’: ’check’}. ------------------Next Round--------------------------------------------------Current game State: Now your hand is [’HT’, ’CK ’], and the public cards are [’D7’, ’C5 ’, ’C4 ’]. The number of chips all players have invested is [4, 0, 1, 4]. Currently , Preflop. Player 0 raises. Player 1 folds. Player 2 folds. Player 3 calls. Player 0 checks. Player 3 checks. Flop. Player 0 checks. The Response of the Agent: Self-Belief: As Player 3, the cards in my hand are the Ten of Hearts and the King of Clubs. Without a clear pattern or majority suit , my current hand doesn ’t offer a lot in terms of immediate strength. However , the King is a high -ranking card and could potentially provide a strategic advantage should the game progress towards a showdown. Given the moderateness of my hand and the absence of an apparent competitive advantage , it might be wise to maintain a balanced strategy, toggling between conservative and aggressive plays based on the situation. World-Belief:Now , looking at Player 0, who remains the only player I would engage in this round , following the withdrawal of Players 1 and 2. In the series of events leading up to the Flop , Player 0 has exhibited an aggressive approach , choosing to raise at every opportunity. This could indicate the strength of his hand , or it could be an attempt to bluff , setting up an environment where he may feed on the hesitance of the other players. Given this behavior , it seems logical to assume that he will continue this aggressive strategy going forward.· · · " Action: Given the current circumstances, a cautious but unexpected move on my part could potentially throw off Player 0’s supposed strategy. Thus , I will go with {’action ’: ’raise ’}. This could make him reconsider my playing style and the strength of my hand , potentially inducing unforeseen errors in future rounds. ------------------Next Round-------------------------------------Player 0 fold ------------------Game End--------------------------------------Player0 loses 2 points , Player1 loses 0, Player2 loses 0.5, Agent -Pro wins 2.5 points ------------------Game End --------------------------------------------------- Baseline Action Sequence: Preflop: raises -> calls -> calls -> checks. Flop: checks -> checks Turn: calls -> raises ->raises -> raises -> checks. River: checks -> checks. Baseline loses by 13.0 chip(s). 5372

heading: Current game State:
text: Now your hand is [’HK’, ’H2 ’], and the public cards are []. The number of chips all players have invested is [2, 4, 4, 2]. In Preflop , Action Sequence: Player 0 calls. Player 1 raises. Player 2 calls. The Response of the Agent: Self-Belief: As Player 3, my card combination is currently off -suit high -low hand with King and 2 in Hearts. This is a decent beginning at the Preflop stage , especially considering the high card , King , which carries significant weight. However, it’s still in the early stages of the game and no community cards are revealed, thus full information is not available to make a precise decision. World-Belief: Player 0, Player 1, and Player 2 have all chosen to remain in the game. Player 1 has raised the pot , indicating they could potentially hold a strong hand. Player 0 and Player 2 have both elected to call , suggesting they might be playing more cautiously , possibly because they hold mediocre hands or they ’re waiting to see the upcoming community cards before playing aggressively. · · · Action: Given the present circumstances , I believe the smart play would be to call. While the high King gives my hand some potential, the accompanying 2 substantially weakens my initial stance. Meanwhile, an overly aggressive play could be detrimental if my opponents are holding strong hands, which is possible given Player 1’s raise. Hence , my move would be to call and wait for the flop ’s revelation. Thus , I’d conclude with {’action ’: ’call’}. ------------------Next Round--------------------------------------------------Current game State: Now your hand is [’HK’, ’H2 ’], and the public cards are []. The number of chips all players have invested is [6, 8, 4, 4]. In Preflop , Action Sequence: Player 0 calls. Player 1 raises. Player 2 calls. Player 3 calls. Player 0 raises. Player 1 raises. Player 2 folds. The Response of the Agent: Self-Belief: As Player 3, I currently hold a high card King and a low card 2, both in hearts. This off-suit combination is an okay starting point in the preflop round. I have to keep in mind that luck plays a role in the rounds to come and the community cards to be drawn yet. Moreover , an aggressive approach towards the end can tilt the scales towards me , deploying my king card , if it finally pairs up. World-Belief: · · · Action: Considering these observations and trying to keep up the misleading image , I think the best move for now would be to call , not showing too much aggression , yet staying in the game , waiting for a potentially favorable turn or river. Therefore , I will choose {’action ’: ’call’}. ------------------Next Round--------------------------------------------------Current game State: Now your hand is [’HK’, ’H2 ’], and the public cards are [’DA’, ’DK’, ’SK ’]. The number of chips all players have invested is [8, 8, 4, 8]. In Preflop , Action Sequence: Player 0 calls. Player 1 raises. Player 2 calls. Player 3 calls. Player 0 raises. Player 1 raises. Player 2 folds. Player 3 calls. Player 0 calls. Player 1 checks. The Response of the Agent: Self-Belief: · · · World-Belief: · · · Action: Based on this analysis , I opt for {’action ’: ’raise’} demonstrating a greater level of confidence in my hand and taking advantage of my opponents ’ potential uncertainty. ------------------Next Round--------------------------------------------------Next Action Sequence: Player 0 raises. Player 1 raises. Player 3 raises. Player 0 calls. Player 1 calls. Player 3 checks. Turn , Community Cards: [’DA’, ’DK’, ’SK’, ’C3 ’], Action Sequence: Player 0 raises. Player 1 calls. Player 3 raises. Player 0 raises. Player 1 calls. Player 3 raises. Player 0 calls. Player 1 calls. Player 3 checks. River , Community Cards: [’DA’, ’DK’, ’SK’, ’C3’, ’H8 ’], Action Sequence: Player 0 raises. Player 1 calls. Player 3 raises. Player 0 raises. Player 1 calls. Player 3 raises. Player 0 calls. Player 1 calls. Player 3 checks. ------------------Game End--------------------------------------------------Player0 loses 24.0 points Player1 lost 24.0 points Player2 lose 2.0 points Agent -Pro wins 50.0 points
----------Baseline: Vanilla LLM with the Same Cards and Opponents ---------- Action Sequence: Preflop: Calls -> Folds. Baseline loses by 2.0 points.
5373

heading: Current game State:
text: Now your hand is [’C5’, ’H3 ’], and the public cards are []. The number of chips all players have invested is [2, 4, 1, 2]. In Preflop , Action Sequence: Player 0 calls. Player 1 raises. Player 2 folds. The Response of the Agent: Self-Belief: Currently , I am holding 5 of Clubs and 3 of Hearts , which is a low unsuited hand. At the preflop stage , without any public cards on the table , it ’s a bit risky to invest heavily with such a hand. My feasible strategies include playing it safe and calling , or folding to avoid unnecessary risk. Raising would be a more aggressive move and less rational given my current hand. World-Belief: Looking at my competitors , Player 0 has chosen to call , indicating a possibility of moderate strength in his hand. It seems he is being cautious , potentially having a mediocre hand. Player 1 raises , suggesting a high confidence in his hand , although it could also be a bluff. As Player 2 has already folded , I only have to consider Player 0 and Player 1 in terms of active competition. · · · Action: Given the above analysis , my most reasonable action at this stage would be to call , as raising with a weak hand would be an unnecessary risk , and folding might close potential chances to improve my hand with the public cards to come. Hence , I will choose {’action ’: ’call’}. ------------------Next Round--------------------------------------------------Current game State: Now your hand is [’C5’, ’H3 ’], and the public cards are []. The number of chips all players have invested is [6, 8, 1, 4]. In Preflop , Action Sequence: Player 0 calls. Player 1 raises. Player 2 folds. Player 3 calls. Player 0 raises. Player 1 raises. The Response of the Agent: Self-Belief: Currently , I am holding 5 of Clubs and 3 of Hearts , which is a weak hand , especially in the absence of any public cards. With this hand, my feasible strategies include calling, to maintain the game pace while minimizing potential loss, or folding to completely avoid any risk. Raising, in this scenario, seems irrational given the weak hand. World-Belief: · · · Action: Based on these analyses , the most reasonable action would be to play it safe. Therefore , I will choose {’action ’: ’call’}. ------------------Next Round--------------------------------------------------Current game State: Now your hand is [’C5’, ’H3 ’], and the public cards are []. The number of chips all players have invested is [10, 10, 1, 8]. In Preflop , Action Sequence: Player 0 calls. Player 1 raises. Player 2 folds. Player 3 calls. Player 0 raises. Player 1 raises. Player 3 calls. Player 0 raises. Player 1 calls. The Response of the Agent: Currently , I am holding 5 of Clubs and 3 of Hearts , which is a weak hand , especially in the absence of any public cards. With this hand , my feasible strategies include calling , to maintain the game pace while minimizing potential loss , or folding to completely avoid any risk. Raising , in this scenario , seems irrational given the weak hand. Self-Belief: · · · World-Belief: · · · Action: Given all the above factors , I think the most reasonable action would be to {’action ’: ’call’}. Considering the infinite chips, my main goal is to stay in the game and watch how it unfolds. This strategy is also in line with my conservatism and allows me to stay in the round with minimal risk. ------------------Next Round--------------------------------------------------Next Action Sequence: Player 0 raises. Player 1 calls. Player 3 raises. Player 0 calls. Player 1 calls. Player 3 checks. Turn , Community cards: [’D3’, ’C3’, ’HQ’, ’S3 ’], Action Sequence: Player 0 raises. Player 1 calls. Player 3 raises. Player 0 raises. Player 1 calls. Player 3 raises. Player 0 calls. Player 1 calls. Player 3 checks. River , Community cards: [’D3’, ’C3’, ’HQ’, ’S3’, ’DA ’], Action Sequence: Player 0 raises. Player 1 calls. Player 3 raises. Player 0 raises. Player 1 calls. Player 3 raises. Player 0 calls. Player 1 calls. Player 3 checks. ------------------Game End-------------------------------Player0 loses 25.0 points Player1 loses 25.0 points Player2 loses 0.5 points Agent -Pro wins 50.5 points ----------Baseline: Vanilla LLM with the Same Cards and Opponents ---------- Action Sequence: Preflop: Folds. Baseline loses by 1.0 points..
5374

heading: Current game State:
text: Now your hand is [’HT’, ’CK ’], and the public cards are []. The number of chips all players have invested is [4, 0, 6, 2]. In Preflop , Action Sequence: Player 0 raises. Player 1 folds. Player 2 raises. The Response of the Agent: Self-Belief: My hand cards are heart -ten and club -king which is considered as high cards. · · · My current strategy can be to try and match my King or Ten with the upcoming community cards or use my King card as the high card to potentially outrank my opponents. World-Belief: · · · Their betting strategy could be quite aggressive which can be impactful in trying to potential opponents to fold , allowing them to scoop the pot or they have strong confidence in their hands. · · · Action: · · · I will choose {’action ’: ’call’}. ------------------Next Round--------------------------------------------------Next Action Sequence: Player 0 raises. Player 2 calls. Player 3 calls. Player 0 checks. Flop , community cards is [’D7’, ’C5 ’, ’C4 ’], Action Sequence: Player 2 checks. ------------------Next Round--------------------------------------------------Current game State: Now your hand is [’HT’, ’CK ’], and the public cards are [’D7’, ’C5’, ’C4 ’]. The number of chips all players have invested is [8, 0, 8, 8]. In Preflop , Action Sequence: Player 0 raises. Player 1 folds. Player 2 raises. Player 3 calls. Player 0 raises. Player 2 calls. Player 3 calls. Player 0 checks. In Flop , Action Sequence: Player 2 checks. The Response of the Agent: Self-Belief: · · · Based on the Texas Hold ’em hand ranking , I only have a high card at this moment. Nonetheless , if the Turn and the River bring clubs , I could potentially form a Flush with my King as the highest card. But, the probability of this happening is relatively low, and relying solely on this possibility would be too risky and against the prudent poker strategy. World-Belief: · · · Action: · · · So, {’action ’: ’check’} is my choice. ------------------Next Round--------------------------------------------------Next Action Sequence: Player 0 checks. Player 2 raises. Player 3 calls. Player 0 folds. Player 2 raises. Player 3 calls. Player 2 checks. Player 3 checks. Turn , community cards: [’D7’, ’C5’, ’C4’, ’C3 ’], Action Sequence: Player 2 checks. Player 3 checks. Player 2 checks. Player 3 checks. ------------------Next Round--------------------------------------------------Current game State: Now your hand is [’HT’, ’CK ’], and the public cards are [’D7’, ’C5’, ’C4’, ’C3 ’, ’D2 ’]. The number of chips all players have invested is [8, 0, 16, 12]. In Preflop , Action Sequence: Player 0 raises. Player 1 folds. Player 2 raises. Player 3 calls. Player 0 raises. Player 2 calls. Player 3 calls. Player 0 checks. In Flop , Action Sequence: Player 2 checks. Player 3 checks. Player 0 checks. Player 2 raises. Player 3 calls. Player 0 folds. Player 2 raises. Player 3 calls. Player 2 checks. Player 3 checks. In Turn , Action Sequence: Player 2 checks. Player 3 checks. Player 2 checks. Player 3 checks. In River , Action Sequence: Player 2 raises. The Response of the Agent: Self-Belief: As Player 3, my hand consists of the ten of hearts and king of clubs. The community cards , unfortunately , don ’t create any viable combinations for me except for a potential straight if an Ace or 6 of any suit appears , but that ’s pushing my luck. My current strategy is to proceed with caution, bluff if necessary, and only call when my hand has a good potential. World-Belief: Looking at my opponents , Player 0 and Player 2, they started strong by raising the stakes but have seemed to cool off a bit. This might signify that the community cards didn ’t play into their hand as they would ’ve hoped. · · · Action: Considering all the above , I think it ’s vital to maintain my poker face and keep them guessing. I will {’action ’: ’call’}. I will show confidence while remaining thoughtful of the game situation. ------------------Next Round--------------------------------------------------Next Action Sequence: Player 2 checks. Player 3 checks. ------------------Game End----------------------------------------Player0 loses 4.0 points Player1 loses 0.0 points Player2 wins 12.0 points Agent -Pro loses 8.0 points ----------Baseline: Vanilla LLM with the Same Cards and Opponents ---------- Action Sequence: 1-Preflop: raises -> calls. 2-Flop: checks. 3-Turn: checks -> calls. 4-River: checks -> calls -> calls -> raises -> raises -> checks. Player 3 loses by 15.0 points. 5375

heading: Future Work
text: This could make him reconsider my playing style and the strength of my hand , potentially inducing unforeseen errors in future rounds. ------------------Next Round-------------------------------------Player 0 fold ------------------Game End--------------------------------------Player0 loses 2 points , Player1 loses 0, Player2 loses 0. 5, Agent -Pro wins 2. 5 points ------------------Game End --------------------------------------------------- Baseline Action Sequence: Preflop: raises -> calls -> calls -> checks. Flop: checks -> checks Turn: calls -> raises ->raises -> raises -> checks. River: checks -> checks. Baseline loses by 13. 0 chip(s). 5372