How Far Are LLMs from Professional Poker Players? Revisiting Game-Theoretic Reasoning with Agentic Tool Use
Authors: Minhua Lin
Enyan Dai
Hui Liu
Xianfeng Tang
Yuliang Yan
Zhenwei Dai
Jingying Zeng
Zhiwei Zhang
Fali Wang
Hongcheng Gao
Chen Luo
Xiang Zhang
Qi He
Suhang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Reinforcement Learning
Imperfect Information Game
Strategic Reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: This work systematically studies LLMs in poker, uncovering heuristic, factual, and knowing–doing flaws, and introduces ToolPoker, a tool-integrated framework using external solvers to reach state-of-the-art gameplay and professional-level reasoning.

Abstract: As Large Language Models (LLMs) are increasingly applied in high-stakes domains, their ability to reason strategically under uncertainty becomes critical. Poker provides a rigorous testbed, requiring not only strong actions but also principled, game-theoretic reasoning. In this paper, we conduct a systematic study of LLMs in multiple realistic poker tasks, evaluating both gameplay outcomes and reasoning traces. Our analysis reveals LLMs fail to compete against traditional algorithms and identifies three recurring flaws: reliance on heuristics, factual misunderstandings, and a “knowing–doing” gap where actions diverge from reasoning. An initial attempt with behavior cloning and step-level reinforcement learning improves reasoning style but remains insufficient for accurate game-theoretic play. Motivated by these limitations, we propose ToolPoker, a tool-integrated reasoning framework that combines external solvers for GTO-consistent actions with more precise professional-style explanations. Experiments demonstrate that ToolPoker achieves state-of-the-art gameplay while producing reasoning traces that closely reflect game-theoretic principles.

Published as a conference paper at ICLR 2026
HoW FAR ARE LLMs FROM PROFESSIONAL POKER PLAYERS? REVISITING
THEORETC REASON-
ING WITH AGENTIC TOOL USE
Minhua Lin-
Enyan Dai
Hui Liu
Xianfeng Tang*
Yuliang Yan
Zhenwei Dai
Jingying Zeng'
Zhiwei Zhang
Fali Wang!
Hongcheng Gao Chen Luo²
Xiang Zhang'
Qi He
Suhang Wang'
1 The Pennsylvania State University
HKUST (GZ)
3 AAmazon
Tsinghua University Microsoft
ABSTRACT
As Large Language Models (LLMs) are increasingly applied in high-stakes do- mains, their ability to reason strategically under uncertainty becomes critical. Poker provides a rigorous testbed, requiring not only strong actions but also prin- cipled, game-theoretic reasoning. In this paper, we conduct a systematic study of LLMs multiple realistic poker tasks, evaluating both gameplay outcomes and reasoning traces. Our analysis reveals LLMs fail to compete against traditional al- gorithms and identifies three recurring flaws: reliance on heuristics, factual misun- derstandings, and "knowing "knowing-doing" doing" gap where actions diverge from reasoning. An initial attempt with behavior cloning and step-level reinforcement learning im- proves reasoning style but remains insufficient for accurate game-theoretic play. Motivated by these limitations, we propose ToolPoker, a tool-integrated reasoning framework that combines external solvers for CTO-consistent actions with more precise professional-stye explanations. Experiments demonstrate that ToolPoker achieves state-of-the-art gameplay while producing reasoning traces that closely reflect game-the principles.
INTRODUCTION
Large Language Models (LLMs) are increasingly deployed in high-stakes domains such cyberse- curity (Ameri et al., 202T) and strategic decision-making ạDiang et al. 2023), where success requires not only factual recall but also reasoning under uncertainty and informed decision-making. A natural testbed for these abilities is game-playing, which combines reasoning, planning, and opponent mod- eling. Poker is especially suitable canonical cernplleeeiifoommationn game (Harsanyi 1995), requiring players to act with hidden information, estimate opponents' ranges, and anticipate future outcomes. Importantly, professional players succeed not only by choosing strong actions, but by reasoning in a game-theoretic manner (Brown & Sandholm, 2019), grounding decisions in equilib- rium principles while adapting opponents. Thus, to play like professionals, one optimally but also think strategically. Evaluating LLMs in poker requires going beyond win rate and examining whether their reasoning traces reflect principled strategic thinking.
only act
Motivated by this, we ask: How far are LLMs from professional poker players? Several recent studies have explored LLMs in such game-theoretic games. For instance, GTBench (Duan al. 2024] and PokerBench (Zhuang et al. 2025) focus on gameplay outcomes and show that LLMs struggle to compete. Suspicion-Agent (uoo et al., 2023) uses theory-of of of-mondd prompting in Ledue Hold' em, with GPT-4 surpassing neural baselines such as NFSP (Heinrich & Silver] 2016), but still falls short of equilibrium-basee methods like CFR+ (Zinkevich et al.] (007). GameBot (Lin et al.) 2025c) examines reasoning steps but only measures correctness. While insightful, these works focus narrowly on outcomes, offering limited understandings why LLMs succeed or fail.
To fill this gap, we conduct a systematic study of LLMs in poker, analyzing both gameplay and reasoning traces. Our analysis shows that LLMs consistently underperform traditional baselines, such NESP (Heinrich & Silver 2016) and CFR+ Tammelin (2014), ranging from reinforcement learning (RL) to equilibrium based solvers, due to three key reasoning flaws: (i) Heuristic rea- soning: LLMSs often rely shallow heuristics rather than rigorous game-theoretic principles. (ii)

---

Published as a conference paper at ICLR 2026
(+) User asking LLMss for giving action
[4] Inference tool-intagrate reasoning LLMs (ToolPoker)
Reward
playing Limit
LLM Reasoning
holding @ 3 and 10. The blie
card is .., W hich action shall take? Why?
& Quantities
Final Answer
Game State
LLM Reasoning
Final Answer
Game
CFR Solver
Equitt Calculator
Range Analyzer
spems you are facing weak hand Given this strong hand, I will tend to rise.
LLMs: (1) Situation: ..; (2) Range Estimation:... ..d
Opponent Analysis: . Mix Strategy: Given the above game-theorete etic analysis, my action fold.
[b) Inference of existing text baseb LLMs Figure I: Illustration ToolPoker and advantages over L.Mss using internal policies.
Factual misunderstanding: LLMs sometimes misjudge fundamental aspects the game, such as hand strength, pot odds, opponent range estimation, leading to systematically flawed reasoning and (ii) Knowing-doing gap: even when LLMs articulate sound reasoning, their final actions often deviate from it, exposing a gap between knowledge expression and decision execution.
To investigate whether these flaws can be mitigated internally, we attempt two-stage framework: (i) behavior cloning (BC) expert reasoning traces instill game-theoretic principles, and (ii) RL fine-tuning with step-level rewards. While this improves fluency and expert-like reasoning style, remains insufficient for precise derivations or competitive gameplay, underscoring LLMs' funda- mental limitations in game-theoretic tasks.
Motivated by these limitations, we pursue an alternative direction: leveraging LLMs' strength in tool use. However, achieving this integration in poker is non-trivial and challenging: (i) Multi-tool dependency. Accurate game-theoretic reasoning often requires multiple solvers (e.g., action and equity solvers), and naively teaching LLMs to invoke these tools across multi-turn poker scenarios leads to error propagation and unstable training. (ii) High data cost. Collecting large-scale reasoning traces augmented with solver calls requires expensive LLM annotation and careful domain-specific tool invocation, making prohibitively costly to build.
To address these challenges, we introduce ToolPoker, the first tool-integrated reasoning (TIR) framework for imperfect-informattie games (Fig. [ which teaches LLMs to call external poker solvers to provide game-theoretic optimal (GTO) actions and supporting quantities such as equity and hand ranges for accurate expert-level explanations. (i) We design a unified tool interface that consolidates solver functionalities into single API, returning all quantities in one query sim- plify tool use and stabilize training. (ii) We construct small-scale expert-level reasoning dataset (Sec. [.1p) inspired by the thought process of professional players, and programmatically augment with standardized tool invocation templates and execution outputs, ensuring high-quality and re- ducing annotation cost. This also provides robust foundation for the following RL training TIR. By combining -ur--guarnteeed computation with human-like reasoning, ToolPoker overcomes fun- damental weaknesses of policy-only training and moves LLMs closer to professional- play. Experiments across multiple poker tasks demonstrate that ToolPoker achieves both state-of-the-a gameplay performance and produces reasoning traces that align much more closely with game- theoretic principles.
Our main contributions are summarized as follows: (i) We conduct the first systematic study of LLMMs in poker, revealing fundamental reasoning flaws such as heuristic bias, factual misunder- standing, and snowing-doing gaps. (ii) We make an initial attempt to improve LLMs' internal poli- cies through two-stage RL framework. While effective improving reasoning style, this approach remains insufficient for GTO reasoning and accurate game-theoretic derivation. (iii) We introduce ToolPoker, a tool-integran integraateed reasoning framework that leverages external solvers guarantee GTO- consistent actions while enabling LLMs to generate precise, professionallstyle explanations. (iv) Extensive experiments show that ToolPoker achieves state-of-the-art gameplay performance and produces reasoning traces that align closely with professional game-theoretic principles.
2 BACKGROUNDS AND PRELIMINARIES
Two-Player Imperfect Information Poker Games. In this paper, we explore using LLMs to play poker with imperfect information. Following prior work (Guo et al. 2023 Huang al. 2024
2

---

Published as a conference paper at ICLR 2026
focus three widely studied two-player variants of increasing complexity: Kuhn Poker, Ledue Hold'em, Hold' em, and Limit Texas Hold' em, where their backgrounds and rules are in Appendix B
Game-theoretic Reasoning. In poker, professional players go beyond heuristics or pattern recog- nition by systematically evaluating equity, ranges, and pot odds within a game-theoretic framework, guiding them toward actions that converge to Nash equilibrium. An example of such professional- style reasoning is in Appendix [B6 with further details on Nash equilibrium in Appendix [B.5
Problem Statement. We model two-player poker game partially observable Markov de- cision process (POMDP) (S, AS R, S2 O), where = {s' < < T is the set of true states, T is the maximum turns, is the action space, T is the transition function, R. is the ward function, & denotes the observation space, and O represents the observation function. At "prs(e) st "pri(-a) where snb denotes public information (e.g., community cards, betting), and st pri(()) and spri( ai) are the private cards of player and the op- ponent, respectively. Each player i partially observes of (5 st sprics)) S and conditions on its history i (o), .......,f of) to choose an action af H4p( f(h)), where f is prompt template that converts game states into natural language task descriptions. A full trajectory = , af, a²2 ri, r2...... S" aa. az TT. rr2). The objective for player is to learn a policy Hág that maximizes the cumulative reward Et= rf in the game.
time the state is s' { st
3 ARE LLMs GOOD AT POKER? A PRELIMINARY ANALYSIS
In this section, to understand the capabilities of LLMs in playing poker games, we conduct prelim- inary analysis to provide initial evidence regarding the strengths and weaknesses of LLMs compared to traditional algorithms for imperfect-informann games.
3.1 EXPERIMENTAL SETUP
Tasks, To quantitatively evaluate the performance of LLMs in poker, we consider two widely stud- ied and popular poker games, Ledue Hold'em em and Limit Texas Hold'em (Brown et al. [019 Stein-
berger 2019
Guo et al. 2023). both implemented in the RL.Crr environment (Zha et al. 2021a).
Following (Guo et al. 2023), we consider four traditional baselines for imperfect information games: NFSP (Heinrich & Silver 2016), DQN (Mnih et al. (0155), DMC (Zha) (et al. 2021b), and CFR+ {Tammelin 2014). NFSP and DMC are self-play RL methods tailored to theerrhe guarantee of convergence to the Nash equilibrium. For the more complex Limit Texas Hold' em environment, where CFR+ is computationally prohibitive, extension of CFR+ These baselines cover diverse strategic paradigms, allowing us to assess L.MMs against broad range of opponent types. Details are provided in Appendix C.I
Comparison Methods.
imperfect information games, while CFR+ provides
instead adopt DeepCFR (Brown et al., 2019), scalable neural
Evaluation Protocol. To ensure the robustness of our evaluation metrics, in our experiment, we run series of 50 games with fixed random seeds and fixed player positions We then rerun the 50 games with the same fixed random seeds but switched positions for the compared methods. To evaluate the gameplay performance in poker games, we choose the earned chips the evaluation metric. Specifically, for each individual poker game, each player starts with 100 chips, the small blind is 1 chip, and the big blind is 2 chips.
3.2 COMPARISON WITH TRADITIONAL. METHOD
Setting. We evaluate suite of representative L.LMs spanning a wide range of parameter scales, including Qwen2.5-3B. Qwen2.5-7B. Qwen2.5-72B (Qwen) 2024). Qwen3-8B (Ynng et al.) 2025), Llama3-8B (Grattafiori et al. 2024). GPT-4. (OpenAl 2025). GPT-4o Hurst et al. (2024), )) and 04-mini (OpenAl 2024), where the cerruuctio-followiin versions of these open-source models are adopted. These models are evaluated against the aforementioned traditional baselines.
Results Analysis. Table reports the average chip gain of different LLMs against traditional meth- ods in both Ledue Hold em and Limit Texas Hold' em. From the table, we observe that (i) Most vanilla LLMs, particularly open-source models with smaller scales, underperform relative tra-
3

---

Published as a conference paper at ICLR 2026
Comparison of various vanilla LLMs against different traditional algorithms trained in Leduc Hold" em and Limit Texas Hold' em environments. Each method plays 100 games with vary- ing random seeds and alternated player positions. Results report net chip gains. In Leduc Hold' em, values range from to 14 chips; in Limit Texas Hold' em, they range from to 99 chips. Bold and underline indicate the best and worst performance in each column, respectively. The "Avg." " columns summarize LLMs' mean performance across the four traditional baselines.
Table 1:
Ledue Hald' em
Limit Texms Hold cm
NUSP
DQN
DMC
CIR-
Arg.
NUSP
DQN
DMC
DrepCIR
Qwen2.5
124
135.5
131.5
116
223.5
Q.a12557B
-68.3
-33.5
-188
-144
Qwe? 5-728
+24.5
-18
-25
+6
-53.
-25.5
Qwent sB
72
73
54
66.0
69
73.5
69
73.5
71.2
LLandd 3B
77.5
108.5
90
71
86.7
177.5
205.5
108.5
GPT-4 I-mizi
+41.5
+68.5
-22
-24
+14.0
+43
40
-24
-205.1
-46.7
GPT-4o
+34
+53
-43
-8
+9.0
-40.5
-5.5
-30
-187.0
-21.2
od-mini
+11
-30
-33.5
-8
-2.6
-105
+111
-58
-117.0
-422
ditional methods. This highlights the limited effectiveness of state-of-the-ard LLMs in poker. (ii) CFR+ consistently outperforms all LLMs, including strong closed-source models such as GPT-4o and 04-mini. This is expected, as CFR+ explicitly targets Nash equilibrium strategies, underscor- ing the importance of -heeethe reasoming in cnnerfeectinformattoon games. (ii) Against non- equilibrium baselines (i.e.. NFSP, DQN, DMC), some large-scale and closed-source LLMs demon- strate competitive superior performance. For instance, GPT-4o achieves +-1.5,, +60.5, and -22 chip outcomes against NFSP, DQN, and DMC, respectively. contrast, small open-source L.LMs (e.g., Qwen2.5-3B exhibit severe losses across all baselines (e.g., - 143.5, -161, and -124 chips). These results suggest that while LLMs cannot approximate Nash equilibrium strategies, sufficiently large models can exploit non-equilibrium opponents.
3.3 IN-DEPTH ANALYSIS: DECOMPOSING REASONING FLAWS OF LLMS
To understand why LLMs fail to compete with traditional methods in poker, we conduct an in- depth analysis of their reasoning processes. Specifically, we first present several case studies that highlight three key flaws in LLM reasoning, followed by a quantitative analysis to further validate and interpret these observations.
Case Study of LLMs' Reasoning Flaw. To probe LLMs' decision-making, we examine their rea- soning traces in specific scenarios against baseline opponents. Representative cases from Qwen2.5- 3B and GPT-4o are shown in Table 13 and [4 in Appendix C.2 From these examples, we identify three recurrent flaws: (i) Heuristic Reasoning. LLMs frequently rely on heuristic-driven reasoning, making decisions based on surface-level patterns or intuitive analogies rather than on rigorous game- theoretic principles. In contrast, the Nash-equilibrium algorithm CFR+ consistently achieves the strongest performance, underscoring the value of game-theoretic reasoning in imperfect-information games like poker. The absence of such equilibrium-oriente reasoning substantially constrains the gameplay performance of LLMs. These two findings indicate that while LLMs capable of articu- lating plausible strategic reasoning, their actual decision decision-making remains constrained by executional inconsistencies and heuristic biases. These limitations ultimately hinder their effectiveness in com- plex poker games that require advanced strategic reasoning capabilities. (ii) Factual Misunderstand- ing. LLMs often ground their reasoning in intuitive analogies, making them prone misjudging fundamental aspects of the game, such as hand strength or opponent range estimation. These factual inaccuracies can cascade into flawed reasoning chains and ultimately suboptimal actions. For ex- ample, as shown in Tab. 14 GPT-4o incorrectly judged ( 100 as weak and preferred folding. However, an equity calculator shows this hand has about 60% equity, indicating is relatively strong. (iii) Srowing-Doing Gap. L.MMs often exhibit a mismatch between articulated reasoning and final actions. For instance, in Tab. 13, Qwen2.5-3B correctly reasons that ($3 (210) is not a strong hand and fold is optimal, while it yet proceeds to raise. Such inconsistencies reveal a breakdown between reasoning and execution. Additional case studies are provided in Appendix C.2
Quantitative Analysis of LLMs' Reasoning Flaws. To validate the reasoning flaws observed in case studies, we adopt the LLM LLM-as-a-June Judge framework (Dubois et al. (023). We design three met- rics: heuristic reasoning (HR), factual alignment (FA), and action reasoning consistency (AC), and score each reasoning trace on 0-2 scale using EPT-4.1-mimi as the judge. Metric definitions, judge prompts, and human-L.L.M agreement are in Appendix C.3 and C.5 For each model, sample 20
4

---

Published as a conference paper at ICLR 2026
traces and evaluate Qwen2.5-3B/7Bil GPT-4.1-ming and o4-mini. To ensure reliabilitty of LLM- based judging, we manually curate 20 professional-styl reasoning traces and score them by LLMs. We observe high agreement with human judgement and include it as reference (see Appendix C.5
We report results in Tab. 2 Three key findings are observed: (i) Reasoning flaws persist across all models. Qwen2.5-3B scores only 0.53 HR, 0.18 FA, and 1.53 AC, while 04-mini, the strongest model, reaches 1.80/.56/1.88. still below perfect consistency. This shows systemic heuris- tic, factual, and knowing-doing flaws L.Mss. (ii) Scaling improves but does not eliminate flaws. Larger models (Qwen2.5- 72B, o4-mini) improve all metrics, but sig- nificant FA and AC gaps remain, showing scale alone cannot achieve professional- level reasoning. (iii) Action-reasoning consistency remains imperfect. AC stabilizes around 1.53-1.87, below the professional baseline of 2.0, with 04-mini still exhibiting knowing-doing mismatches. Full details are in Appendix C.4
Table 2: LLM-as-a-Judge score (0-2) evaluating rea- soning traces of various L.LMs in Leduc Hold' em and Limit Texas Hold' em. Bold and underlined numbers indicate the best and worst performance, respectively,
Ledue Hold' em
Limit Texas Hold' 'em
HR
FA
AC
Avg.
HR
FA
AC
Avg.
Professional
2
2
2
2
Qwen2.5-3B
1.53
0.74
0.30
1.60
0.81
Qwen2,5. 7B
0.95
0.86
1.68
1.16
1.00
0.87
1.70
1.19
Qwen2.5.72B
1.78
1.52
1.85
GPT-4,1-mini
0.98
1.73
1.87
1.52
0.95
1.61
1.87
1.47
04-mini
1,80
1.56
1.85
1.73
1.57
1.65
1.88
1.70
Overall, these findings quantitatively reinforce case studies: despite improvements in scale and instruction tuning, current LLMs remain far from professional-level poker reasoning. They con- tinue exhibit heuristic biases, factual misunderstandinnss, and executional inconsistencies that fundamentally limit their game-theoretic reasoning capabilities.
CAN WE IMPROVE LLMS IN POKER? FAILURES AND INSIGHTS
Building on the preliminary analysis of LLM limitations in poker, we next explore how to improve their ability to both act and reason like professional players. A natural starting point is supervised fine-tuning (SFT) on expert gameplay. However, while obtaining expert actions straightforward using established solvers such CFR+, constructing large-scale datasets with high-quality reason- ing traces is extremely costly, making pure SFT impractical at scale. For instance, [Wang et al. (025)) report that mastering even simplified poker games like Ledue Hold' em requires least 400k action-only instances. Adding reasoning traces would multiply both time and financial costs, ren- dering such datasets infeasible construct. To address this, inspired by recent progress in RL for enhancing LLM reasoning (Guo et al. 2025) and by traditional RI. for poker (Heinrich & Silver) 2016), we make an initial attempt to propose a two-stage framework, BC-RIRL, that combines be- havior cloning (BC) with regret-inspired policy optimization (RIRL). In the first stage, BC aims to provide a small but valuable foundation of expert play and reasoning. In the second stage, RIRL refines these policies toward GTO play under Nash buss-equilibrium brusee supervision.
4.1 BEHAVIOR CLONING
We first leverage BC to expose LLMs to professional-style reasoning. Following recent ad- vances in caaoning-auugmeete datasets (Muennighoft et al., 2025) and inspired by professional players's thought process (Appendix [B6)). we curate dataset of professional-level trajectories D3 [(h', a", r')]. where h² is the full interaction history up time t and a' is the corresponding expert response. Expert actions a" are obtained by querying the state-of-the-art CFR+ solver (Tam- melin 2014) with h², ensuring alignment with Nash-equilibrium play. Reasoning traces r" generated using an LLM guided by domain-specific prompt templates covering key concepts such hand equity, pot odds, and opponent ranges, to mimic the explanatory style of professional play- ers. The construction prompts and dataset examples are in Appendix [.3] To ensure dataset quality, we implement an automated pipeline that (i) checks consistency between the annotated actions and CFR+ outputs, and (ii) filters out low-quality samples using our HR/FA/AC metries. After filtering, we obtain compact dataset of approximately 5k reasoning-augmentedd samples, which is then used to fine-tune the LLM policy my via supervised fine-tuning (SFT) to imitate expert responses:
LBC -Eqse ab) (log To (a()((1)))
5

---

Published as a conference paper at ICLR 2026
This imitation phase grounds the LLM in domain knowledge and equips it with basic game-thhoreee reasoning capability. As shown in Sec. 4.3 BC primarily serves as warm start, providing a crucial foundation for the subsequent RL stage.
4.2 REGRET-INSPIRED RL FINE-TUNING
As an initial attempt to refine policies beyond imitation, we attempt a regret-inspired reinforcement learning (RIRL) framework. To overcome the sparse and noisy outcome-based rewards in multi-turn poker games such as Leduc Hold'en em and Texas Hold' em, we experiment with a step-level regret- guided reward that leverages signals from pre-trained CFR solver to guild LLMs minimize cumu- lative regret and convergence to the Nash equilibrium. Full details of RIRL are in Appendix D.1
Regret-guided Reward Design. Motivated by CFR's success in poker playing by approaching Nash equilibrium from Sec. [.2] we optimize LLMs via regret minimization. Our key idea is to compute cumulative regrets from a pre-trained CFR solver and normalize them into fine-grained reward signals that capture each action's relative contribution. For a policy ma as player i, the reward action af is defined as:
- mean(( (ren((()(()))))
R(a!)
Fiomm((r(aa))) )
where Faoem denotes a normalization factor, chosen as the standard deviation in our implementa- tion. raf) the cumulative regret of action a;. indicating how much better or worse it performs compared to the current mixture strategy across time.
Fine-tuning Objective. Based on this signal, we fine-tune LLM policy via PPO (Schulman al. 2017 with the following clipped RL objective:
(o) = EE-D, -0-xx
<o(ul>) A, clip [otd(y)
(b()))) TTta(y)
1_,,1))))1-_11+ee)) ka (TTA (To
(3)
min
where Ty and Wold denote the current and previous policy models, respectively. e is the clipping threshold. Tre is the reference policy that regularizes To update via KL--divergence penalty, mea- sured and weighted by Dr and B, respectively. Generalized Advantage Estimation (GAE) (Schul-) man et al. 2015) is used for advantage estimate A. denotes the input samples drawn from D, which is composed of trajectories generated by the current policy Ty. is the generated outputs via policy LLMs T (-)z). The trajectory collection procedure is introduced in Appendix D.4
4.3 EXPERIMENT ANALYSIS
Experimental Setup. Following the settings in Sec. 3.1 we implement BC-RIRL by fine-tuning LLMs with both BC and RIRL, and compare against traditional algorithms as well as LLM-based approaches. For traditional baselines, we adopt NFSP, DQN, DMC, and CFR+, consistent with Sec. 3.1 For LLM-based baselines, in addition direct prompting without fine-tuning, we con- sider two variants: (i) BC-SPRL, which fine-tunes LLMs through BC and self-play RL with sparse outcome-based rewards. and (ii) RIRL, which fine-tunes LLMs with RIRL alone, without the BC stage. Further details of SPRI. are in Appendix E. Other settings follow these in Sec. 3.1 including the evaluation metrics. The implementation details are Appendix [.5]
Comparison Results. We fine-tune Qwen2.5-7B with BC-RIRL and compare against traditional algorithms and vanilla LLMs. The gameplay and reasoning results are reported in Tab. 3 and Tab.
Gameplay. (i) All RL-based fine-tuning variants improve performance in Kuhn Poker, showing that both outcome- and bagret-based feedback provide useful signals in simple environments. (ii) BC- RIRL outperforms direct prompting and BC-SPRL (e.g.. +17.0 chips vs. GPT-4.1-ming but still trails CFR+ (-34.0 chips) In Ledue Hold' em, indicating dense regret feedback is more effective than sparse outcome rewards in complex poker games, yet insufficient for equilibrium-level play. (iii) Pure RIRL without the BC stage does not yield improvements in Leduc Hold' em (-64.5 chips GPT-4.1-nm 1-mini,, highlighting BC necessary foundation.
Reasoning. (i) RIRL consistently improves HR and AC (e.g., 1.93 HR and 1.90 AC in Leduc Hold' em 1.5 for o4-mini), reducing heuristic flaws and the knowing-doing gap. (ii) RIRI.

---

Published as a conference paper at ICLR 2026
Table 3: Results of comparison fine-tuning methods against various traditional based and vanilla LLMs in Kuhn and Ledue Hold' em environment. Other settings follow these in Tab. Bold and underlined numbers indicate the best and worst performance, respectively.
Traditional Mechods
Vanilla LLMS
Method
NESP
DQN
DMC
CBR
(wen2.5-38
Qwan2.5-7B
GPT-411miii
Kwdn
Qwen? 5-78
-53.0
-16.0
Owen2.5 7Bamm
14.0
-3.0
+10.0
+43.0
+3.0
1.0
11.0
+4.1
Qwend 5-78ac sma..
+6.0
-6.0
+13.0
-14.0
+32.0
+23.0
-22.0
+100
+10.7
Qweed. 5- S-7Bec
+4.0
-8.0
+11.0
-200
+57.0
+27.0
-21.0
+11.
+17,1
Ledor Hold 2m
Owen2.5-78 S-78ee.
-57.5
-92.0
-73.
-40
-90.5
+12.0
Qwen2.5 Sllc--sa
95.5
18.0
Qwend. -ll s191.
-3.5
Table 4: LLM-as-a-Judee score (0-2) evaluating reasoning traces of various LLMs in two realistic poker tasks. Bold and underlined numbers indicate the best and worst performance, respectively.
Leduc Hald'
Limit Texas Flold vm
Avg.
Avr.
Qwan2.5:78
0.95
0:56
1.68
1.16
1,00
0.87
1.70
1,19
(PPT44 1114411-mig I-mini
0.98
1.73
1.87
1.82
0.95
1.61
1.87
1:47
c4 mini
1.50
1.56
1.85
1.75
1.57
1,65
1.85
1.70
Qwen2.5 10ans
0.94
0.59
1.44
1.15
0.98
0.93
1.71
1.20
Qwan25-78ycsn S7
1,50
0.58
1.56
1.47
1.87
0.86
1.64
1.45
5-78ac
1.93
1.06
1,86
1.61
1,88
1.12
1.87
1.59
gains only marginal improvement in FA (1.12, 0.87 and 1.65 for RIRL, Qwen2.5-7B and o4-mini), showing that factual misunderstandings remain the main limitation. Together with the case studies, these results indicate that while BC-RIRI. improves strategic reasoning and action-reasoning align- ment, factual misunderstandings remain a notable challenge. Full analysis are in Appendix D.2 D.2
Takeaway. Our experiments validate that current LLMs are inherently weak at strategic reasoning in game-theoretic tasks. RL fine-tuning with step-level or outcome-based rewards yields modest gameplay gains but still lags behind traditional methods like CFR. Importantly, while our two-stage approach helps LLMs imitate professional reasoning styles, they continue to struggle with precise derivation such as equity and hand ranges. This reveals fundamental limitation: LLMs alone cannot yet achieve both GTO actions and precise reasoning. To bridge this gap, we next explore augmenting L.Mss with tool use, leveraging their natural strength in tool invocation to support GTO- consistent actions and precise game-theoretie reasoning.
5 TOOLPOKER: GAME-THEORETIC REASONING WITH AGENTIC TOOL USE
Building on our analysis in Sec. which highlights the limitations of LLMs in producing GTO actions and precise thee-tte reasoning, we propose ToolPoker, a tool-integrated reasoning (TIR) framework to leverage LLMs' strength in tool use to empower LLMs to leverage external poker solvers to refine their actions and reasoning qualities, which is shown in Fig. [| To make this tool usage stable and effective, we introduce unified tool interface that consolidates multiple poker solvers (e.g., CFR and equity calculators) into single API to simplify this into single-turn tool use. On the training side, we adopt a two-stage strategy: first, behavior cloning on a ouge-au dataset teach the model when and how to call external tools; and second, reinforcement learning with a composite reward to further optimize solver integration and reasoning quality.
5.1 TOOL-INTEGRATED GAME-THEORENT REASONING IN POKER
Rollout Process. To enable GTO-consistent TIR, we design a structured prompt template in Tab. 2T to guide LLM to leverage external poker solvers for game-theoretic reasoning. Con- cretely, given policy LLM my as player i at time t, тт generates reasoning trace enclosed in <think></thinkk tags. To obtain GTO actions and other quantities, To issues a query in <tool></tool> tags, which calls the unified solver interface and returns results wrapped in <output></outpp
tags. These outputs are then incorporated into the reasoning trace before To produces the final action af within <answer><> < answer> tags.
Unified Tool Inference. Obtaining GTO actions and supporting quantities (e.g., equity, pot odds, and range distributions) often requires multiple tool calls, such CFR solver and an equity cal-
7

---

Published as a conference paper at ICLR 2026
Table Comparison of various LLM-basee methods against different traditional algorithms trained in Ledue Hold em and Limit Texas Hold' em environments. Other settings follow these in Tab. Bold and underline indicate the best and worst performance each column, respectively.
Ledue Hold's
Limi Texas Hold
NYSP
DQN
CFR-
Avg.
NESP
DQN
DMC
DeepCFR
Avg.
2.578
-52.5
-91
-73
-68.5
-73.0
-53.5
-188
-144
-101.0
-121.6
Owen2.5 72B
-28.5
18
18
25
9.1
+6
53.5
73.5
57.5
44.6
olvaini
+11
+30
-33.5
-8
-2.6
-105
+111
-58
-117.0
-42.2
Quen22 Swecn2.5-7Baaaaae
-37.0
-64.5
-43.5
-3.01
40.5
-77.5
42.5
-0.5
-70.2
-77.6
Quem? 5
-11.5
+18.0
-1.0
-30
+6.8
+50.5
+63.0
61.5
-5.0
+45.0
Reasoning Score
Qwen2.5-8
Reasoning Score
Qwen225-7
ToolPaker
40
Tacliverr
mini
o4 -iis
Earning Chips
Forning Chhipe
BC RIBL RiBLoveta--
BC KiRLoo
20
2.0
2.0
1.5.
20
40
-40
DON
NFSP
DON
(a) Reasoning Ledue
(b) Reasoning Limit
(c) Ablation
Gameplay (d) Ablation Reasoning
Figure 2: Results for ToolPoker: (a) and (b) present reasoning analysis in Ledue Hold' em and Limit
Texas Hold' em; (c) and (d) show ablation studies on gameplay and reasoning in Ledue Hold' em.
culator. To simplify and stabilize training, we unify these functionalities into a single standardized interface that provides both the solver's actions and auxiliary statistics for game-theoretic reasoning.
5.2 TRAINING ALGORITHM
BC for TIR. To construct high-quality TIR data without incurring prohibitive annotation cost, we build an automated pipeline that programmatically augments the reasoning dataset from Sec. 4.1 with standardized tool invocation templates (e.g., <tool><////// <tool></tool>>> </to>>>) and execution outputs (e.g., <output>< output>), This resulting dataset De is then used to train ToolPoker via SFT, provid- ing foundation for LLMs to know how to invoke tools for game-theoretic reasoning. The realistic example and the details of the automatic pipeline are in Tab. 22 in Appendix G.2
RL Fine-tuning. We train ToolPoker with PPO (Schulman al., 2017), where the objective func- tion is defined in Eq. (8. To better support TIR, we follow ReTool (Feng et al., 2025a) and integrate external poker solvers into the LLM policy Ta, enabling multi-turn real-time tool use that provides cTO--cosistentt actions and supporting quantities from external tools. To guide the training, we design composite reward function. Formally, given player at time step f, the reward defined as
(4) where af is the ground-truth action from the CFR solver, âi is the podel-predicted action, and pf is the generated reasoning trace. Here, Ranswers Riormat and Rooo correspond to the answer reward, format reward, and tool-execution reward, respectively, ensuring that ToolPoker not only outputs GTO-consistent actions but also generates structured reasoning traces with effective tool usage. af and a are the weights to balance the impact of format and tool execution rewards. More details of these reward functions are in Appendix G.3 The fine-tuning algorithm is in Alg. of Appendix G.4 G.4
R(a dt,pp)) = Ranswer (af; at) + Oaz Ritoemaa + 04 (p)),
5.3 EXPERIMENTAL RESULTS
Evaluation Setup. We conduct evaluations on two realistic and complex poker tasks, Ledue Hold' em and Limit Texas Hold" em. We compare ToolPoker with the following baselines: (i) Tra- ditional algorithms: NFSP, DQN, DMC, and CFR: (ii) Vanilla LLMs: Qwen2.5-7B, Qwen2.5-72B. and 04-mini; (iii) Fine-tuning- based baseline: BC-RIRL. Other settings follow these in Sec. 4.3
More Implementation details of ToolPoker are in Appendix G.5
Gameplay Performance. We first explore the gameplay performance of ToolPoker. Qwen2.5-7B is the base model for fine-tuning. We compare ToolPoker with BC-RIRL and three vanilla LLMs. Qwen2.5-7B, Qwen2.5-72B and o4-mini, where the comparison results are reported in Tab. 5 Two key findings emerge: (i) ToolPoker achieve sfate-of-the the- gameplay perfomrance against tradi- tional algorithms. For instance, ToolPoker gains +60.5, +63.0 and +61.5 chips against NFSP, DQN and DMC in Limit Texas Hold' em, while BC-RIRL gains - -77.5, - 82.5 and - -00.5 chips against them. This indicates the effectiveness of ToolPoker in calling CFR solver to obtain GTO-consistent
8

---

Published as a conference paper at ICLR 2026
actions. (ii) ToolPoker slightly underperforms CFR but is still comparable in both poker environ- ments. Specifically, ToolPoker gain -3.0 and -5.0 chips against CFR+ and DeepCFR in both Leduc Hold' em and Limit Texas Hold' em. which are minor. We analyze the reason is that while ToolPoker provides a high success rate in executing the CFR solver to provide GTOO-conssteent action, it is inevitable that occasional errors occur in tool calling.
Reasoning Quality. To assess whether ToolPoker also improves reasoning, we employ the LLM- as-a-Judge framework following the settings in Sec. 4.3 Fig. 2 (a) and (b) summarize the results across three metrics. Two observations emerge: (i) ToolPoker achieves near-perfect across all three scores, outperforming all baselines and approaching professional levels. This indicates that, beyond ffute-of-the-eeee the-hartt gameplay performance, ToolPoker also enables LLMs to generate precise and logically consistent reasoning traces grounded in game-theoretic principles. (ii) Compared with BC-RIRI, ToolPoker yields substantially higher FA scores. This demonstrates the importance of leveraging external solvers: while BC-RIRL can articulate plausible reasoning, it often lacks ac- curate auxiliary quantities (e.g., equities, ranges). In contrast, ToolPoker grounds its reasoning in solver-derived calculations, ensuring rigor and internal consistency.
delivering state-of
Ablation Studies. To understand the impact of each component in ToolPoker, we implement two ab- lated variants: (i) ToolPoker/BC: removes BC and learns tool use only via RL; (ii) ToolPoker/RL: discards RL fine-tuning and relies solely BC. We measure both gameplay performance (against NESP and DQN) and reasoning quality in Leduc Hold' em, with results shown in Fig. 2 (c) and (d). The full ToolPoker achieves the strongest overall performance, while the variants reveal comple- mentary weaknesses. Specifically: (i) ToolPoker/BC suffers from lower HR and weaker gameplay, suggesting can query the solver but fails internalize game-theoretic reasoning patterns; (ii) ToolPoker/RL attains higher HR but performs poorly in gameplay and FA/AC, indicating it imitates reasoning superficially without aligning with GTO-consistent actions. These results highlight that BC provides the foundation for TIR, while RL fine-tuning aligns solver execution with GTO actions and precise derivation. Together, they enable ToolPoker to learn not only how to call the solver, but also how to integrate outputs into coherent, professional-style reasoning traces. More discussions are in Appendix G.6
RELATED WORK
Strategic Reasoning in LLMs. Recent studies have examined LLMs in game-theoretic settings, including poker (Duan et al. 2024 et al. 2024 Zhuang et al. [025 Wang et al. 2025). Unlike prior work that primarily evaluates gameplay outcomes, we also analyze the reasoning process, identifying why LLMs fail to achieve GTO play. Moreover, we introduce the first TIR framework that leverages poker solvers for professional-level gameplay. Further discussion
Appendix A.
Tool Learning on LLMs. TIR equips LLMs with external tools for domains such as math and web search (Gao et al, 2023 Jin al.] (025), which are typically fully observed and single-agent. In contrast, ToolPoker extends TIR to snerfect-information games, integrating poker solvers to ensure GTO actions and rigorous reasoning. Full details on RL and TIR are in Appendix A.2 and A.3
7 CONCLUSIONS AND FUTURE WORKS
In this paper, we revisit strategic reasoning in LLMs through poker with imperfect information. Our analysis shows that current LLMs fall short of professional-level play, exhibiting heuristic biases, factual misunderstandings. and knowing- doing gap between their reasoning and actions. An ini- tial attempt with BC and RIRL partially reduces heuristic flaws but is still not enough for precise ghme-the derivations or competitive gameplay. To address this, we introduce ToolPoker, a TIR framework that leverages LLMs' strength in tool use to incorporate external poker solvers. ToolPoker enables models not only to call solvers for GTO actions but also to ground their rigor- ous and accurate game-theoretic reasoning in solver outputs. Experiments across multiple poker tasks show that ToolPoker achieves sfate-of-thh-att gameplay performance and produces reasoning traces that align closely with professional game-theereeti principles. Our research paves the way for further exploration of TIR in more complex strategic settings, shifting the focus beyond solely improving models' internal policies. Further discussion of future works is provided in Appendix]] []

---

Published as a conference paper at ICLR 2026
8 ETHICS STATEMENT
This paper studies LLMS in the context of poker as a rigorous benchmark for strategic reasoning under uncertainty. While poker involves gambling in practice, experiments conducted en- tirely in simulated environments without any financial transactions or human participants. Thus, this research does not pose risks related to gambling addiction or monetary harm.
Our contributions focus on methodology and evaluation. We study the reasoning capabilities of LLMs. propose new training frameworks, and benchmark them against both traditional algorithms and LLM-based methods. These findings aim to deepen understanding of LLM reasoning in imperfect-informationn games, with potential implications for broader domains such as cybersecu- rity and negotiation. We acknowledge that advanced poker agents could, if misused, be deployed in real-money contexts. To mitigate this risk, we release code and datasets solely for research purposes, emphasizing their use benchmarks for safe and reproducible evaluation.
Finally, we ensured that no personally identifiable or sensitive human data were used in this work. All datasets are synthetically generated using poker solvers or L.MSs. We believe the potential benefits this paper, including advancing understanding the limitations of LLMs' reasoning, im- proving the design of tool-augmented AI, and supporting safer deployment in high-stakes domains, clearly outweigh the minimal risks.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure reproducibility. The details of our proposed methods, including model architectures, training objectives, and hyperparameters, are provided in Sec. H and Sec. [ Experimental setups, including datasets, preprocessing steps, and evaluation protocols, are described
in Sec. 3.1 Sec. 4.3 and Sec. 5.3 with additional details in the Appendix.
ACKNOWLEDGMENT
This material is based upon work supported by, in part by the Army Research Office (ARO) under grant number W91 W911NF-21-1011 and the Department of Homeland Security (DHS) under grant number 17STCIN00001-055 -05-000. The views and conclusions contained in this material those of the authors and should not be interpreted as necessarily representing the official policies, either expressed implied, of the funding agencies.
REFERENCES
Kimia Ameri, Michael Hempel, Hamid Sharif, Juan Lopez. Jr., and Kalyan Perumalla. Cybert: Cy- bersecurity claim classification by fine-tuning the bert language model. Journal of Cybersecurity and Privacy, pp. 615-637, 2021.
Michael Bowling, Neil Burch, Michael Johanson, and Oskari Tammelin. Heads-up limit hold'en poker is solved. Science, 347(6218)) 145-149,,
Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365(6456): 885-890, 2019.
Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret mini- mization. In International conference muchine learning, pp. 793-802 PMLR, 2019.
Tingwei Chen, Jianpeng Chen, and Dawei Zhou. 3d-fum: Benchmarking 3d molecule learning with functional groups. IJCAI, 2024.
Tingwei Chen, Zhenping Guo, Kaiyang Huang, and Kai Sun. Descriptor: Power systems cascading failure analysis synthetic dataset (powercascade). IEEE Data Descriptions, 2025a.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint 1rXiv:2211.12588. 2022.
10

---

Published as a conference paper at ICLR 2026
Yongchao Chen, Yueying Liu, Junwei Zhou, Yilun Hao, Jingquan Wang. Yang Zhang, and Chuchu Fan. Rl-codd-interprette Training Ilms to reason with code via supervised and reinforcement learning. arXiv preprint arXtv:2505.2/668, 2025b.
Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer, and Arjun Yadav. Gamebench: Evaluating strategic reasoning abilities of Ilm agents. arXiv preprint arXiv:2406.066) 2024.
Debrup Das, Debopriyo Banerjee, Somak Aditya, and Ashish Kulkarni. Mathsensei: a tool- augmented large language model for mathematical reasoning. arXiv preprint arXiv:2402.17231, 2024.
Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel- Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. Gtbench: Uncovering the strategic reasoning capabilities of Ilms via game-theeretee evaluations. Advances in Neural Information Processing Systems, 37:28219-2888 28253,, 2024.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:://ppenrstt ww.. net. forum? fwt.f/oorrrmmm
4=
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use llms. arXiv preprint arXiv:2504. 1536, 2025a.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-gre group policy optimization for I1m agent training. arXiv preprint arXiv:2505. 10978, 2025b.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program- aided language models. International Conference on Machine Learning. pp. 10764-10799 PMLR, 2023.
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujuu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. ToRA: A tool-integrated reasoning agent for mathematical problem solving. In The Twelfth International Conference on Learning Representations, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501.12944 2025.
Jiaxian Guo, Bo Yang, Paul Yoo, Bill Yuchen Lin, Yusuke Iwasawa, and Yutaka Matsuo. Suspicion- agent: Playing imperfect information games with theory of mind aware gpt-4. arXiv preprint arXiv:2309.17277 2023.
Akshat Gupta. Are chatgpt and gpt-4 good poker players?-a pre-flop analysis. arXiv preprint arXiv:2308.12466 2023.
John C. Harsanyi. Games with incomplete information. The American Economic Review, pp. 291- 303, 1995.
Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfect- information games. arXiv preprint arXiv:1603.01121, 2016.
Nathan Herr, Fernando Acero, Roberta Raileanu, Maria Perez-Ortiz. and Zhibin Li. Large language models are bad game theoretic reasoners: Evaluating performance and bias in two-player non- zero-sum games. In ICML. 2024 Workshop on L.IMs and Cognition.
11

---

Published as a conference paper at ICLR 2026
Chenghao Huang, Yanbo Cao, Yinlong Wen, Tao Zhou, and Yanru Zhang. Pokergpt: An end-to- end lightweight solver for multi-player texas hold' em via large language model. arXiv preprint arXiv:2401.06781, 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Haitao Jiang, Lin Ge. Yuhe Gao, Jianian Wang, and Rui Song. Large language model for causal decision making, arXiv preprint arXiv:2312.17122, 2023.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.095 2025.
Harold W Kuhn. A simplified two-person poker. Contributions to the Theory of Games, 1:97-103, 2016.
Minhua Lin, Zhengzhang Chen, Yanchi Liu, Xujiang Zhao, Zongyu Wu, Junxiang Wang, Xiang Zhang, Suhang Wang, and Haifeng Chen. Decoding time series with Ilms: A multi-agent frame- work for cross-domain annotation. arXiv preprint arXiv:241011462,, 2024.
Minhua Lin, Hui Liu, Xianfeng Tang, Jingying Zeng. Zhenwei Dai, Chen Luo, Zheng Li, Xiang Zhang, Qi He, and Suhang Wang. How far are lims from real search? comprehensive study efficiency, completeness, and inherent capabilities. arXiv preprint arXiv:2502. 18387, 2025a.
Minhua Lin, Zongyu Wu, Zhichao Xu, Hui Liu, Xianfeng Tang, Qi He, Charu Aggarwal, Xiang Zhang, and Suhang Wang. A comprehensive survey on reinforcement learning-based agen- tic search: Foundations, roles, optimizations, evaluations, and applications. arXiv preprint arXiv:2510.16724, 2025b.
Minhua Lin, Hanqing Lu, Zhan Shi, Bing He, Rui Mao, Zhiwei Zhang, Zongyu Wu, Xianfeng Tang, Hui Liu, Zhenwei Dai, et al. Position: Agentic evolution is the path to evolving lims. arXiv preprint www:2660200359, 2026.
Wenye Lin, Jonathan Roberts, Yunhan Yang, Samuel Albanie, Zongqing Lu, and Kai Han. GAME- BoT: Transparent assessment of LLM reasonnin in games. In Wanxiang Che, Joyce Nabende, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7656-7682, 2025c.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198-124235, 2024.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 51 518(7540):599-33, 2015.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501. 19393,, 2025.
John Nash Jr. Equilibrium points in n-person games. Proceedings of the national academy of sciences, 36(1):48 19:4484911111 1950.
OpenAl Openai o3 and 04-mini system card, 02dc- -789-8000-8-800 4779-9900--1771857722-..
2024.
URL
https://
cdn www.:ooean.ccooooe
03 and 14:d--4--innn wnwaaddo--mi--sssteeeeeeee---eee system pdrd.pdf]
OpenAl Gpt-4.1 system card, April 2025. URL htt.tpppp//paatto..pppaaa..c/oop
models/gpt-4.11
Qwen. Qwen2.5: A party of foundation models, September 2024. URL https: https::/qwen.nn futhubbio/bll htwp:///goggg.....55.5 qwen2.5/
12

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53744 2023.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer. Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to tools. Advances in Neural Information Processing Systems, 36:68539- 68551, 2023.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.022 024388 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint (xXiv:177.006444 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 1199//920 19256, 2024.
Eric Steinberger. Single deep counterfactual regret minimization. arXiv preprint urXiv:1901.0700 2019.
Oskari Tammelin. Solving large imperfect information games using cfr+. arXiv:1407.5044 2014.
arXiv preprint
Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoe Le, et al. Freshllms: Refreshing large language models with search engine augmentation. arXiv preprint arXiv:23100032// 2023.
Wei Wang, Fuqing Bie, Junzhe Chen, Dan Zhang, Shiyu Huang, Exgeny Kharlamov, and Jie Tang. Can large language models master complex card games? arXiv preprint arXiv:2509.01328, 2025.
Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, et al. Webagent-rl: Training web agents via end-to-end multi-turn rein- forcement learning. arXiv preprint arXiv:2505. 2025.
Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, and Vasant G Honavar. Simper: A minimalist approach to preference alignment without hyperparameters. arXiv preprint arXiv:2502.00883 2025.
Zhichao Xu, Zongyu Wu, Yun Zhou, Aosong Feng, Kang Zhou, Sangmin Woo, Kiran Ramnath, Yijun Tian, Xuan Qi, Weikang Qiu, al. Beyond correctness: Rewarding faithful reasoning in retrieval-augmented generation. arXiv preprint arXiv:2510. 13272, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022.
Tarik Zaciragic, Aske Plaat, and K Joost Batenburg. Analysis of bluffing by dqn and cfr in leduc hold' em poker. arXiv preprint arXiv:2509.04125. 2025.
Daochen Zha, Eweii-Hrng Lai, Songyi Huang, Yuanpu Cao, Keerthana Reddy, Juan Vargas, Alex Nguyen, Ruzhe Wei, Junyu Guo, and Xia Hu. Rlcard: platform for reinforcement learning in card games. In Proceedings of the Twenty-iint International Joint Conference on Artificial Intelligence, 2021a.
13

---

Published as a conference paper at ICLR 2026
Daochen Zha, Jingru Xie, Wenye Ma, Sheng Zhang, Xiangru Lian, Xia Hu, and Ji Liu. Douzero: Mastering doudizhu with self-play deep reinforcement learning. In international conference on machine learning, pp. 22333-12344 PMLR, 2021b.
Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. Advances neural information processing systems. 37:110935-11097,, 2024.
Ruize Zhang, Zelai Xu, Chengdong Ma, Chao Yu, Wei-Wei Tu, Wenhao Tang, Shiyu Huang, Deheng Ye, Wenbo Ding, Yaodong Yang, et al. A survey on self-play methods in reinforcement learning. arXiv preprint arXiv:2408.0 000002 2024.
Enmin Zhao, Renye Yan, Jinqiu Li, Kai Li, and Junliang Xing. Alphaholdem: High peghhperformance artificial intelligence for heads-up no-limit poker via end-to-end reinforcement learning. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 36, pp. 4689-4697, 2022.
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in weal-world environ- ments. arXiv preprint (Et::2504.001600 2025.
Richard Zhuang, Akshat Gupta, Richard Yang, Aniket Rahane, Zhengyu Li, and Gopala Anu- manchipalli. Pokerbench: Training large language models to become professional poker players. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 26175-26188
2025.
Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. Advances in neural information processing systems, 20, 2007.
14

---

Published as a conference paper at ICLR 2026
A FULL DETAILS OF RELATED WORKS
A.1 STRATEGIC REASONING IN LLMS
With the rapid progress of LLMs' cognitive capabilities, recent studies have begun to investigate
their potential for strategic reasoning in game- theoreti settings (Duan al.] 2024] Gupta 2023
Huang et al. 2024 Zhuang et al. 2025 Wang et al. 2025). GTBench (Duan et al. (024) introduces comprehensive benchmark covering a variety of games to assess LLMS' ability to follow equilib-
rium principles. Gupta 2023) provide one of the first empirical evaluations of GPT-4 and ChatGPT
in poker, revealing systematic deviations from GTO gameplay. [uoo et al. (2023) explore theory-of- mind (ToM) prompting in Ledue Hold' em, showing that GPT-4 with ToM reasoning can outperform neural baselines such as NFSP (Heinrich & Silver 2016). PokerGPT (Huang et al. (024) fine-tunes LLMs on poker-specific data and observes improvements in gameplay, while PokerBench (Zhuang ct al. 2025) constructs a benchmark on No-Limit Hold'em, More recently, Wang et al. (2025) curate large-scale action-only datasets (more than 400k+ examples) and demonstrate gains card games by fine-tuning LLMs on such data. Additional works (Costarelli et al. 2024 Herr et al.) also inves- tigate gameplay performance and biases of LLMs in other strategic games, such as Tic-Tac-Te and Prisoner's Dilemma. In addition to exploring strategic reasoning in text-based settings, other works study use RL in the multimodal domain (Lin et al., 2024 Zhai et al.) 2024). For instance, /Zhai iet al. (2024) extend this line of work to the multimodal domain by fine-tuning large vision-language models (VLMs) with RL. This paper leverages C-yylt intermediate reasoning to guide VLMs through multi-step decision-making tasks, including poker. This demonstrates that RL can enable VLMs to effectively explore and execute visual -txtual reasoning sequences.
Our work differs in two key aspects: (i) unlike prior works that mainly evaluate or improve LLMs' actions, we further analyze their reasoning process, asking how LLMs think before acting and why they fail to achieve GTO play; and (ii) rather than relying internal policies alone, we propose the first tool-integrated reasoning framework that leverages poker solvers, enabling both equilibrium- consistent actions and professional-style game-theoretic reasoning.
A.2 REINFORCEMENT LEARNING
Reinforcement Learning (RL) has emerged as a powerful mechanism for enhancing the reasoning abilities of LLMs. In context of LLMs, RL was first introduced through Reinforcement Learn- ing from Human Feedback (RLHF) align outputs with human preferences via algorithms such as Proximal Policy Optimization (PPO) (Schulman et al. 2017). Subsequent works proposed more advanced techniques such as Direct Preference Optimization (DPO) (Rafailov et al. 2023), SimPO (Meng et al., 2024), and SimPER (Xiao et al., 2025). which improve the stability and effi ciency of RL training. More recently, researchers have explored both outcome-based rewards (Guo et al., 2025) and step-level rewards (Feng et al., 2025b) to improve problem-solving in domains such as mathematical reasoning (Guo et al. 2025). code generation (Chen et a 2025b), and web re- trieval (Wei et al., 2025 Lin et al. 2025b). In this work, we investigate RL for imperfect-informane games, where sparse outcomes, hidden states, and adversarial dynamics make reward design partic- ularly challenging. Our analysis shows that both mntcome-based and step-level RL signals inef- fective at improving LLMs' internal policies in poker, motivating the use of solver-derived. regret- inspired signals as more reliable feedback.
A.3 TOOL-INTEGRATED REASONING OF LLMS
Tool-integrated reasoning (TIR) has emerged as a promising approach to extend the capabilities of
LLMs. Prior works demonstrate improvements in domains requiring precise computation or external
knowledge, including mathematical calculation (Das et al. 2024), programming (Chen et al. 2022
Lin et al.) 2026), and web search (Vu et al.) 2023). Early studies such as PAL (Gao et al. 2023)
prompt LLMs to generate code for execution, while ToRA (Gou et al. 2024) curate tool-use trajec- tories and apply imitation learning to train tool invocation. More recently, RL has been explored as
an effective framework to improve TIR (Jin et al. 2025] Feng et al., 2025a| Zheng et al. (025). For instance, Search-R1 (Jin et al., 2025) enables search-enghn enginne queries for QA, ReTool (Feng et al.
2025a) improves mathematical reasoning with a code sandbox, and DeepResearcher (Zheng et al.
2025 scales multi-hop retrieval and tool orchestration. Despite these advances, existing TIR
15

---

Published as a conference paper at ICLR 2026
search largely targets fully observed, single-agent tasks. In contrast, poker involves stochasticity, hidden information, and adversarial dynamics, where tools must compute gquilibrium-consisse strategies and counterfactual values rather than deterministic answers. To the best of our knowl- edge, ToolPoker the first TIR framework for imperfect-informatio games. It integrates external poker solvers into L.Ms,, teaching them how to invoke solvers, and grounding their reasoning traces in solver outputs. This ensures rigorous, precise game-theoretic reasoning and GTO-consistent play, bridging prior works on strategic reasoning, RL, and TIR.
B BACKGROUND AND RULES OF POKER
In this section, we introduce the poker variants studied in our work. These games are widely used in the literature benchmarks for imperfect-inie imperfect-information reasoning because they balance tractability with the core challenges of hidden information, sequential decision- making, and stochasticity.
B. KUHN POKER
Kuhn poker (uuhn 2016) is minimalistic poker game designed to capture the essence of imperfect- information Secision-making in tractable form. The game is played with only three cards (c.g., Jack, Queen, King) and two players. Each player antes one chip, and a single betting round follows. Each player receives one private card, and the third card remains hidden.
Players can either check/bet (if no bet has been made) or call/fold (if a bet has been made). Because of its small size-onie only handful of information sets Kuhn poker admits closed-form solutions, including simple Nash equilibrium strategies that mix between bluffing with weak hands and value betting with strong hands. Despite its simplicity, it highlights the central strategic dilemma of poker: balancing deception and value extraction under hidden information.
B.2 LEDUC HOLD' EM
Ledue Hold' em (Zciragic et al., 2025) is a widely studied poker variant that extends Kuhn by introducing multiple betting rounds and public information. The game is played with small deck of six cards consisting of two suits and three ranks. Each player antes one chip and receives a single private card. A first round of betting occurs, after which single public card is revealed. A second round of betting then follows.
The addition of the public card dramatically increases strategic depth: players must update belief's about opponents' ranges as new information is revealed, balance bluffing and value bets across streets, and plan actions that maximize long-term expected value. Although still small enough for exact or approximate equilibrium computation (e.g.. via CFR (Zinkevich et al. (007)). Leduc cap- tures essential poker phenomena such as semi semi-bluffing bluffing, slow-playing, and range narrowing, making it a standard benchmark for algorithmic and M-basedd poker research.
B.3 LIMIT TEXAS HOLD' EM
Limit Texas Hold'em em [Bowling et al., 2015) is a more realistic and complex poker variant that closely related to the full game of Texas Hold'em, em, which is the most popular poker format in practice. The deck consists of 52 standard playing cards. Each player is dealt two private hole cards, and up five public community cards are revealed in stages: the flop (three cards), the turn (one card), and the river (one card). At each stage, players take turns acting in one of several betting rounds.
Unlike No-Limit Hold'em, bet sizes in Limit Hold'en ei are fixed and restricted to small or big bets depending on the round. Each hand therefore unfolds as sequence of structured betting decisions, but the state space remains extremely large compared to Kuhn or Leduc. The presence of multiple streets, large range interactions, and complex pot-odds considerations make Limit Hold' em a sig- nificantly more challenging testbed for LLMs and reinforcement learning algorithms. Professional- level play in this environment demands mastery of equilibrium based reasoning as well as opponent exploitation-skk skills that current L.LMs struggle replicate.
16

---

Published as a conference paper at ICLR 2026
B.4 ADDITIONAL DETAILS OF BACKGROUND AND PRELIMINARY
B.5 GAMB-THBORETIC REASONING
In poker, theerthe reasoning grounded in Nash Equilibrium is essential for professional play. A Nash Equilibrium represents a stable outcome in which each player's strategy is an optimal response to the others. Formally:
Definition B.1 (Nash Equilibrium (Nash Jr 1950)). A Nash Equilibrium is a strategy profile in a game where no player can unilaterally improve their payoff by deviating from their current strategy, assuming the other players' strategies remuin unchanged. Formally, stralegy profile (aj, a,....,;,,) is a Nash Equilibrium if. for every player i:
U((a): o=,) 2 U,(a,,a"i), Yo, E A
(5)
where A; denotes the set of feasible actions for player i, U, the utility function (expected payoff) of player and o=; represents the equilibrium strategies of all players other than i.
Rather than relying solely on heuristics or pattern recognition, professional players systematically evaluate equity, ranges, and pot odds within a game-the framework, thereby providing an opti- mal action. An illustrative example such
reasoning in practice is in Appendix B.6
B.6 PROFESSIONAL PLAYERS POKER
To illustrate how professional poker players think, we provide real example from the blog of a well-known Texas Hold' hold'em em professional playe[] Unlike casual players who rely on intuition, professionals systematically evaluate wide range of factors before acting, including:
Game context: What are the stack sizes, pot size, and stack-to-pot ratio?
Ranges: What range of hands should continue with? What range does my opponent have? How does the board interact with these ranges, and which player benefits most?
Board texture and big hands: Who holds the larger share of strong hands in this spot?
Mixed strategies: What is my optimal mix between actions (e.g., 3-betting vs. calling, check- calling vs. check-raising)?
Bet sizing: How many bet sizes do need here (e.g.. two sizes such as 30% pot and 90% pot)? Which size does my hand prefer relative my overall range?
Randomization: How do I randomize between actions to stay balanced (c.g., using chip marker decide frequencies)?
What is my opponent's likely response my bet? What physical tells, history, or reads do have? At what strategic level are they operating, and what exploits should I consider?
Opponent modeling:
This example shows that professional play is grounded in equilibrium-based reasoning, probabilistic mixing, and careful opponent modeling, far beyond heuristic or surface-level decision making.
Our behavior datasets are designed with these principles in mind, encouraging LLMs to reason through such questions. Details of the text-only BC dataset curation and TIR-enable BC dataset curation are provided in Appendix and Appendix G.2 respectively.
C ADDITIONAL DETAILS OF PRELIMINARY ANALYSIS IN SEC. 3
C.I COMPARISON METHODS
To comprehensively evaluate the performance of LLMSs in playing poker, consider both tradi- tional RL-based baselines and rule-based solver baselines. RL methods serve as learning-based references that have been widely applied imperfect-information games, while rule-based solvers provide near-equilibruu strategies that approximate ground truth. Specifically, we include the fol- lowing methods:
same game- but- i--i ittiisntt innt.nn.. .hm1
17

---

Published as a conference paper at ICLR 2026
NFSP (Heinrich & Silver 2016): Neural Fictitious Self-Play is pioneering framework for learn- ing approximate Nash equilibria in imperfect-inf informmtion games. It combines reinforcement learning to approximate best responses with supervised learning to approximate average strate- gies, enabling agents to learn directly from self-play experience.
DQN Mnih et al. (2013): Deep Q-Network was one of the first breakthroughs in deep RL for sequential decision-mann making. Although originally designed for perfect-informatio environments such as Atari, subsequent works (Zha et al. 2021b] Guo et al. 2023) have adopted as a baseline for imperfect-information games, including poker.
DMC Zha et al. (2021b): The Deep Monte Carlo (DMC) algorithm is originally proposed for the Chinese card game DouDizhu. leverages large-scale self-play with Monte Carlo policy optimization and demonstrates strong performance in complex imperfect-information card games. Following prior works (Zha et al. (022bb) Guo et al. (023). we adapt DMC as baseline for poker.
CFR+ (Tammelin] 2014): Counterfactual Regret Minimization (CFR) (Zinkevich et al. 2007 is a foundational algorithm for solving imperfect-informatie by iteratively minimizing counterfactual regret at each information set. CFR+ enhances CFR with linear regret updates and warm-stan averaging, greatly accelerating convergence. It has become the de facto standard solver in large-scale poker domains and serves as a strong rule-based baseline in evaluation.
games, converging to Nash equilibrium
DeepCFR Brown al. 2019): Building CFR, DeepCFR employs neural function approxima- tion to replace tabular regret tables, thereby generalizing across information sets. While CFR+ is provably effective, its computational cost grows prohibitively in large games such Texas Hold' em. DeepCFR addresses this limitation by learning regret values via neural networks, mak- ing it applicable to larger domains and forming the basis of superhuman agents such as Libra- tus (Brown & Sandholm 2019).
C.2 CASE STUDIES OF LLMs' REASONING FLAWS
We provide the examples from Qwen2.5-3B and GPT-4o in Tab. 13 and 14 to illustrate why LLMs fail in playing poker. From these tables, we consistently observe three limitations of LLMs in playing poker: (1) Heuristic Reasoning; (ii) Factual Misunderstanding: and (iii) Showing-Dooim
Gap. The detailed analysis of these case studies can be found in Sec. 3.3
C.3 EVALUATION METRICS OF
LLM-AS-A-JUDGEE FOR LLMS' REASONING
In the LLM-as-a-Judge approach used in quantitative analysis of LLMs' reasoning traces in Sec. 3.3 we use the following three metries to validate the identified three reasoning flaws:
Heuristic Reasoning Score (HR): The judge prompt template is provided in Tab.
Factual Alignment Score (FA): The judge prompt template is provided in Tab. 16
Action-reasoning Consistency Score (AC): The judge prompt template is provided in Tab. 17
C.4 FULL DETAILS OF QUANTITATIVE ANALYSIS
To further validate the reasoning flaws observed in case studies, we adopt an LLM-as-a-Judge frame- work (Dubois al. 2023). Specifically, we design three metries: heuristic reasoning (HR), factual alignment (FA), and action- reasonnig consistency (AC). Each generated reasoning trace is scored by three independent LLM judges on a 0-2 scale for each metric. GPT-4. 1-mini (OpenAl) 2025) is
used as the judge model. The metric definitions and judge prompts are in Appendix C.3
From the table, we observe that (i) Reasoning flaws persist across all models. All evaluated LLMs demonstrate varying degrees of heuristic reasoning, factual misunderstanding, and knowing-dee doing gaps. For instance, Qwen2.5-3B obtains only 0.53 HR, 0.18 FA, and 1.53 AC, indicating weak fac- tual grounding and limited strategic reasoning. Even the strongest model, 04-mini, while achieving the 1.80 HR, 1.56 FA, and 1.85 AC, still falls short of perfect cetion-reasoon
consistency (1.85). This confirms that these flaws systemic and persist across models. (ii) Scaling improves but does not eliminate reasoning flaws. Large and more powerful models, such as Qwen2.5-77 and 04 mini, generally achieve higher scores across all these metrics compared their lightweight variants. This
18

---

Published as a conference paper at ICLR 2026
suggests that increased scale and instruction tuning enhance the ability of LLMs to approximate game-theoretic reasoning and avoid factual mistakes. Nevertheless, the persistence non-trivial gaps, particularly in FA and AC, indicates that scaling alone is insufficient to reach professional- level game-theoretic reasoning. (iii) Action-reasoning consistency remains imperfect. AC scores stable across models (1.53-1.87) yet below the professional baseline of 2.0. Even the strongest model, 04-mini, reaches 1.85 but still shows knowing-doing gaps where reasoning diverges from action. To directly assess this, we compute mismatch proportions Appendix C.5 which align with AC values and confirm it as both a valid proxy for and evidence of the knowing nhowing-doing gap.
C.5 HUMAN-IN- LOOP EVALUATION FOR LLMs' REASONING
To validate the reliability of LLM-based judging. conduct iemaan-i--the-lopp evaluation. Draw- ing on professional-sty/ee reasoning (Appendix B.6) and our behavior cloning prompt template (Ap- pendix D.3). we use GPT-5 curate 20 reasoning traces and have them scored by LLMs. These traces achieve perfect scores (all achieve maximum 2), showing strong alignment with human judg- ments, which we include reference for our analysis.
C.6 CALIBRATION AND VALIDATION OF OUR LLM-AS- SCORE
In this subsection, we provide the details of how to calibrate and validate our LLM-as aaM--s---JJude Score. Judge calibration. In Appendix C apply the LLM-as-a-Judge approach and use three metries: Heuristic Reasoning (HR), Factual Alignment (FA), and Action-reasoning Consistency (AC) in the scale of 0-2. To calibrate this scale, iteratively refined the HR/FA/AC rubries and judge prompts using a small pilot set of representative hands.
General procedure. We collect a small set of clearly good, medium, and poor reasoning traces for each dimension, manually assign target scores (0/1/2), and refine the textual criteria until the judge consistently reproduces the correct scores.
HR calibration. We anchor the "0/1/2" rubric using examples that are (i) purely heuristic, (ii) partially grounded but inconsistent, and (iii) strongly aligned with game-theoretic principles (e.g., pot odds, range interactions).
FA calibration. We provide objective poker quantities (equities, ranges, pot odds) from external solvers and instruct the judge to only factual correctness.
AC calibration. We explicitly instruct the judge to verify that the reasoning logically implies the same action as the final decision.
Judge validation. Following the protocol in Sec. 3.3 we manually curate 20 professional-style reasoning traces to use them and score them by LLMs. These traces achieve perfect scores (all achieve maximum 2), showing strong alignment with human judgments.
Sensitivity and inter-rater LLM agreement. Our LLM-as-a-Judee results in Tab. 2 Tab. and Fig. 2 are consistent across distinct poker environments (Ledue and Limit Hold' em), indicating that the judge is not domain-sensitvve
To further assess inter-rater agreement, re-evaluate ToolPoker's Limit Hold'en reasoning traces using GPT-5 as the judge (instead of the GPT-4.1 1-mni judge used in the main paper). All settings follow Section 5.3. The results are reported in Tab. 6 From the table, we observe close agreement between the two judge models, validating the robustness of our evaluation and reducing concerns about prompt sensitivity or model-specific bias.
Table 6: Inter-rater agreement: LLM-as-a scores (0-2) on ToolPoker's reasoning traces in Limit Texas Hold'em em. We compare the original judge (GPT-4.1-mini) with another judge (GPT-5).
Method
HR
FA
AC
Avg.
GPT-5
1.94
1.89
1.90
1.91
GPT-4.1-mini
1.93
1.92
1.94
1.94
19

---

where oN denotes the strategy profile at time t, of is the opponents" strategy, u(o,) the expected utility under dg and u(o;"" a) is the utility when player i deviates to action at. The instantaneous regret Taff) measures how much better worse at performs relative to the current mixture strategy, Published as a conference paper at ICLR 2026
D FULL DETAILS OF BC-RIRL
D.1 FULL DETAILS OF REGRET-INSPIRED RL FINE-TUNING
While BC helps LLMs imitate expert play, its limited dataset size and imitation-basee nature make it insufficient for professional-level performance. As an initial attempt to refine policies beyond imitation, we explore regret-inspires reinforcement learning (RIRL) framework. Prior approaches in both traditional RL (Heinrich & Silver, [016 Zhao et al.) 2022) and LLM-based RL (Guo et al.) 2025) typically rely on outcome- rewards (e.g., win/loss). However, poker, especially in multi-round games such as Ledue Hold'em and Texas Hold" em-these sparse and noisy signals fail to capture the contribution of individual actions. To address this, we experiment with a step-level regret-guided reward that leverages signals from a pre-trained CFR solver, aligning fine-tuning with the principle that minimizing cumulative regret drives convergence to the Nash equilibrium.
Regret-guided Reward Design. Inspired by our analysis in Sec. 3.2 which highlights CFR as the state-of-the-an algorithm for approaching Nash equilibrium in imperfect-informatie explore optimizing LLMs through regret minimization. Our key idea is to compute cumulative regrets with CFR and transform them into fine-grained reward signals that estimate each action's contribution. For policy my player the cumulative regret of action of time is defined as:
games,
ra())) = (a=) + L.(a), [a(a)) = .a, u(o.),
(6)
while Re (af)) aggregates this over time. To compare actions within the same decision point, we
normalize regrets into relative reward signal:
R(a)) (a mean ((r(a))))_1
((rm(()((a)))
where From denotes a normalization factor, chosen as the standard deviation in our implementation.
Fine-tuning Objective. Based on this signal, we fine-tune LLM policy via PPO (Schulman et al. 2017) with the following clipped RL objective:
(0)
-EZDD .
<o(ul=) A, clip
To(( (v)a)) 1 +c)+ DKa. (+c))
(8)
min
[Tou(y)=))
where To and Wold denote the current and previous policy models, respectively. is the clipping- related hyperparameter. Tref is the reference policy that regularizes no update via KL-divergence penalty, measured and weighted by Dx and B, respectively. Generalized Advantage Estimation (GAE) (Schulman et al.] (015)) is used as the advantage estimate A. x denotes the input samples drawn from D, which is composed of trajectories generated by the current policy ze- y is the generated outputs via policy LLMs (T(-(z). The procedures of trajectory collection are detailed in Appendix D.4
D.2 FULL DETAILS OF COMPARISON RESULTS
We evaluate whether BC-RIRL improves LLMs' poker performance by fine-tuning Qwen2.5-7B and comparing against both traditional methods and vanilla LLMs. Results in Kuhn and Ledue Hold' em are reported in Tab. 3 We highlight three key findings: (i) All RL-based fine-tuning vari- ants improve performance in Kuhn Poker. This suggests that both outcome-basee and regret-guided feedback provide useful learning signals in simple environments with limited strategy space. (ii) BC-RIRL surpasses direct prompting and BC-SPRL in Leduc Hold'en em, though it still trails tradi- tional algorithms such CFR+. For example, BC-RIRL gains 17.0 chips against GPT-4.1-mini. while still losing 34.0 chips against CFR+. This indicates that regret- guided dense feedback is more effective than sparse outcome-based rewards in complex tasks, but is sufficient to reach equilibrium- level play. (iii Pure RIRL without the BC stage does not yield improvements in Ledue Hold' em. For instance, BC-RIRL and BC-SPRL. gain +7.0 and -64.5 chips against GPT-4. 1-mini, respectively.
20

---

Published as a conference paper at ICLR 2026
This underscores the importance of BC in establishing strong foundation of expert-like reasoning before RL fine-tuning.
To further assess whether BC-RIRL enhances reasoning quality, adopt the LLM-as-a- pro- tocol from Sec. 3.3 and compute three reasoning-trace scores. Results Ledue Hold' em and Limit Texas Hold' em reported in Tab. + with additional case studies provided in Appendix F.T Two findings are observed: (i) RIRL consistently surpasses the baselines on HR and AC. For example, BC-RIRL fine-tuned on Qwen2.5-7B reaches 1.93 HR and 1.90 AC in Leduc Hold'em, em, outperform- ing the strongest vanilla LLM, 04-mini, which achieves 1.80 HR and 1.85 AC. This shows that BC-RIRL effectively mitigates heuristic reasoning flaws and reduces the knowing- RIRL yields only marginal improvements in FA. For instance, in Limit Texas Hold' em, BC-RIRL achieves 1.12 FA, only slightly higher than vanilla Qwen2.5-7B (0.87 FA) and still far behind 04- mini (1.65 FA). Together with the case studies, these results indicate that while BC-RIRL improves strategic reasoning and action-reasoning alignment, factual misunderstandings remain a notable challenge.
gap. (ii)
D.3 ADDITIONAL DETAILS BEHAVIOR CLONING
We provide the BC data construct prompt template, which is shown in Tab. 18 GPT-5-mini is used the target model for annotation. The detailed actions and other auxiliary quantities (e.g., winning probability and hand range) are obtained from pre-trained CFR solver, equity calculator and other tools. These tools are implemented Python.
D.4 TRAJECTORY COLLECTION PROCEDURE
To collect trajectories for RL fine-tuning, we adopt an on-policy setting where the LLM policy competes against a random agent. At each iteration, the LLM plays a batch of N games against the random agent (N 64 in our setting). The LLM's actions from each round stored as individual data samples. Formally, for an LLM policy time step t, a sample represented as (of, h4 a²), where at is the chosen action of player i. After each batch, the collected trajectories are used to fine-tune the L.LM policy no. producing an updated policy nh that is then used for subsequent data collection.
with partial observation of and action history hf at
D.5 IMPLEMENTATION DETAILS OF BC-RIRL
In the behavior cloning stage, construct 5, 000 data samples with both reasoning traces and actions for hehavior cloning. Specifically, to generate actions, we use CFR+ TTammelnn (2014) to compete against random player that randomly selects actions from the action space, and extract the actions from CFR+ as the ground-truth actions. The GPT-5-mini is then used to generate reasoning traces of these actions, where the prompt provided in Appendix D.3] the RL stage, we set Qwen2.5-7B Dwwen..577-Innsrruct as the base model for fine-tuning.
E METHODOLOGY OF SPRL
Inspired by traditional RL in imperfect-informatiie games (Heinrich & Silver 2016 Zhang et al. 2024), we conduct d-uniform self-play by letting a single policy LLM me play both sides. In each round, (i) clone the current policy to obtain fixed opponent TT: (ūi) sample N self-play games between Te(- f(a)) and Tg(. faf)), alternating positions and random seeds, to collect trajec- tories To: (iii) update no with RL on To for 8 steps while keeping a fixed; and (iv) refresh my with the latest xo to start the next cycle.
Fine-tuning Objective. To fine-tune LLMs via RL, then formulate the RL objective function as follows:
max EzwDD. (r v)] 3T 3DD (
where ne is the policy LLM being trained. Fref is the reference LLM (typically the initial pretrained LLM) that regularizes the policy update via KL-divergence penalty, measured and weighted by DeL and B. respectively. x denotes the input samples drawn from D.. which is composed of trajectories generated by the current policy my in self-play setting. y represents the generated
21

---

Published as a conference paper at ICLR 2026
outputs via policy LLMs T=(-) In this paper, choose commonly used Proximal Policy Optimization (PPO) (Schulman et al. (017)) as the backbone RL algorithm, which optimizes LLMs by maximizing the following objective:
(0) = - -BEDD (a
To(y|>)
To (o(a)))
c. Aode ( (10)
min
Aado: clip
Tota(y|z)
Tota(y|z)
where wa and Wold denote the current and previous policy models, respectively. is the clipping- related hyperparameter. The advantage estimate Aadv is computed using Generalized Advantage Estimation (GAE) (Schulman al. 2015). We also investigate the performance of other commonly used RL algorithms, such as GRPO Shao et al. (2024).
Reward Design. Poker is sequential decision-making task with multiple turns. The reward for player at time step f is defined as the discounted cumulative return from until the end of the game:
(11)
R=
where E (0, is the discount factor balancing immediate and long-term outcomes. Because players only observe payoffs after a hand is completed, the task characterized by sparse rewards: intermediate steps yield r
while the terminal step provides rf. We consider two types of terminal signals: (i) binary outcome reward, where r! = 1 if the player wins the hand and r
otherwise; and (ii) normalized earnings reward, where rf Ceam Cmit. with Ceam the final net chip gain (or loss) and Cinit the initial chip count.
F ADDITIONAL DETAILS OF INITIAL ATTEMPT IN SEC.
F.1 CASE STUDIES OF BC-RIRL
We present case studies Qwen2.5-7B fine-tuned with BC-RIRL in Ledue Hold'em (Tab. 19 and Tab. 20). The results show that after fine-tuning, the model can produce reasoning traces that semble those of professional players. However, closer inspection reveals persistent factual misun- derstandings. For example, the model claims that calling is the optimal CFR action, even though the prompt explicitly states that calling is not legal move. This supports our conclusion in Sec. 4.3 while BC-RIRI. improves action-reasoning consistency and professional-style imitation, factual in- accuracies remain a significant challenge, highlighting the limitations of BC-RIRI..
G ADDITIONAL DETAIL OF ToolPoker
G.1 TIR ROLLOUT PROMPT TEMPLATE
The TIR rollout prompt template for poker is provided in Tab. 21
G.2 TIR BC REASONING DATASET CURATION
To construct high-quality TIR data without incurring prohibitive annotation cost, instead of building TIR reasoning-angmenteed dataset from scratch, build an automated pipeline programmati- cally augments the reasoning dataset from Sec. 4.1 with standardized tool invocation templates (e.g.. <t <tool></tol>>> and execution outputs (e.g., cuutputt=/outputtt A detailed example of the appended tool invocation templates is provided in Tab. 22
G.3 REWARD DESIGN
Our hybrid reward function contain the following components:
Answer reward: This reward enforces the alignment of LLMs' final action with the GTO- guarantee action from the CFR solver. Formally, given LLM policy my the player i with
22

---

Published as a conference paper at ICLR 2026
Algorithm Fine-tuning Algorithm of ToolPoker for TIR.
Input:
Policy model Ty. old policy Told, task dataset Dr, masking function M for each training iteration do for each task x in D, do
Sample ground-trute truth GTO action of æ Sample rollout y from Told for Initialize reasoning chain while not end of episode do
Generate next segment: <think> or <tool>
if tool is invoked then
Interact with external poker solvers, obtain <output> Append output to reasoning chain
end if end while
Extract model-predicee action from final response p
Compute the composite reward R(a, p)
Compute GAE advantages Å for Apply loss masking M exclude tool output tokens Compute PPO loss LPPo in Eq. and update T
end for end for
partial observation of and action history h at time step t. the answer reward is denoted as:
Ranswer (a a²) = L
is equivalent otherwise,
(12)
where af and af denote Te'S predicted action and CFR solver's action at time step t. is us.equivalent checks whether the model's final action matches the CFR solver's action as the ground-trune action.
Format reward: (tormaa(!) E {0.1}. which evaluates whether the reasoning trace follows the required structured schema with special tokens in the correct der: reasoning tehink></thinkk tool calling <tool></tool>, feedback output and final action <answer></answer>
Tool execution reward: (p/) Toolsne/Toolyy which measures the fraction of successful tool calls the reasoning trace, encouraging the model to invoke external tools effectively and integrate their outputs into subsequent reasoning.
G.4 RL FANE- ALGORITHM FOR TIR
Alg. summarizes the fine-tuning procedure of ToolPoker for enabling TIR in poker. Given task dataset D., where construction details are in Appendix G.5 the algorithm proceeds follows.
For each task & E D, with a corresponding ground-truth action à from a CFR solver, we first obtain G rollouts from the old policy Told- Each rollout is generated step by step, where the model pro- duces either a <think> segment (internal reasoning) or <tool> call. If tool is invoked, the model interacts with the external poker solver, retrieves the <output>, and appends soning chain. This iterative process continues until the end of the episode. At the end of the rollout, we extract the model-ppreicce
the rea-
action from the final response p. composite reward R(a, à. p) is then computed, combining answer accuracy, reasoning format, and tool-execution quality (see Appendix G.3). Using this reward, we estimate advantages Á with Generalized Advantage Estima- tion (GAE) (Schulman et al. 2015). To ensure tool outputs do not dominate training, we apply masking function M that excludes solver outputs from the loss. Finally, we compute the PPO loss LPPO (Eq. g and update the policy Ty.
Through this iterative process, the model learns not only to query solvers for GTO-consistent actions and other auxiliary quantities but also to integrate solver outputs into coherent reasoning traces, thereby aligning action selection with rigorous game-theoretic principles.
23

---

Published as a conference paper at ICLR 2026
G.5 IMPLEMENTATION DETAILS
We follow existing works Feng et al. 2025a Jin et al. 2025) to train ToolPoker with the VeRL (Sheng et al. 2024) framework. For RL fine-tuiing, based the existing work (Wang al. 2025 we build an automated pipeline to curate an action-only dataset with 400, 000 samples for both Ledue Hold' em and Limit Texas Hold'em Specifically, we use a pretrained CFR solver to compete against random agent and collect the game states and actions of CFR to build such dataset. Note that
Bren2.5--B-Insteee model is the base model. The max response length is set as 8, 192 tokens. The rollout model's temperature 0.7 and top-p is 0.6. For behavior cloning, we curate TIR dataset with 5, 000 samples with both actions and tool-integrated reasoning traces. During RL fine-tuning, the rollout batch size is set to 64, and the mini update size is 16. An AdamW optimizer is utilized with initial learning rate 1e- 6.
G.6 ADDITIONAL DISCUSSION
Generalization without solvers. In realistic settings, external tools may be unavailable or only intermittently accessible. examine this, we ablate ToolPoker by removing RI. fine-tuning and retaining only BC (Sec. 5.3). This variant shows weaker tool-use capability than full ToolPoker, and under intermittent tool access we find that HR and AC remain relatively high while FA degrades first. These results suggest that ToolPoker internalizes core strategic structures (e.g.. range-based reason- ing and mixed strategies), while solvers primarily supply precise numerical quantities supporting view that LLMs provide the reasoning framework whereas external tools ensure the accuracy of game-theoretic computations.
G.7 COMPARISON WITH EXISTING TOOL-USE FRAMEWORK
G.7.1 RELATION TO EXISTING TOOL-USE FRAMEWORK
While ToolPoker follows the general "LLM + tools" paradigm, it is designed specifically for informaation poker games with theorehio principles, whereas prior frameworks focus on general tasks (e.g., math, QA, web search). This difference leads to several important challenges that make existing methods difficult to directly apply.
Task difference: game-theoretic reasoning. Prior TIR methods (Yao et al. 2022 Schick et al., 2023 Feng et al. 2025a) typically aim obtain factual answers or execute deterministic API calls. In contrast, ToolPoker targets strategic reasoning in games where (i) the agent must reason under imperfect information, and optimal play requires Nash-equilibrium (GTO) reasoning, and (ii) expla- nations must reflect gaheethe principles rather than surface-level logic. This setting requires multi-step strategic reasoning that goes substantially beyond previous tool-use scenarios.
Existing frameworks cannot be directly adapted. Unstable interleaved reasoning and tool use. Poker reasoning requires LLMs to generate game-theoretic explanations while coordinating multiple solver calls for diverse quantities (e.g., actions, equities, ranges). Directly applying ReTool-sttyee framework (Feng et al., (025a) to teach L.LMs to invoke multiple tools during reasoning would () force the model to call and integrate several specialized solvers for each hand, (ii) introduce error propagation from tool calls across multi-step game-theoretic reasoning trajectories, and (iii) lead inaccurate explanations and degraded gameplay. High data cost. Toolformer-style ap- proaches (Schick et al. 2023) usually require large-scale reasoning traces augmented with solver calls fine-tune LLMs. For game-theoree theoretic reasoning tasks, generating such traces demands ex- pensive LLM annotation and careful domain-specift tool invocation, making it prohibitively costly to scale to expert- poker play.
ToolPoker: a design specifically addressing these challenges. To overcome these issues, ToolPoker introduces two key design choices. Equilibrium- than asking the LLM to orchestrate multiple tools, ToolPoker consolidates all solver functionalities into a single API call that returns GTO actions as well as auxiliary quantities (e.g., equities, strate- gic ranges, hand distributions). This qqulibrium-oriienee interface stabilizes TIR-RL training and allows the LLM to focus on producing accurate, professional-leved reasoning instead of managing complex tool orchestration. Low-cost, expert-level TIR dataset. Instead of relying large-scale
simplified interface. Rather
24

---

Published as a conference paper at ICLR 2026
reasoning dataset, ToolPoker deliberately constructs small curated expert reasoning dataset aligned with game- theoretic principles, and augments with tool-calling templates and solver outputs. This provides a cost-efficient way to perform behavior cloning from expert-level play, followed by rein- forcement learning fine-tuning.
G.7.2 EMPIRICAL COMPARISON WITH RETOOL
We then empirically compare ToolPoker with ReTool ReToolFeng et (022a) validate the effectiveness of ToolPoker in our poker games with imperfect information. Specifically, we implement ReTool in Ledue Hold' (same solver, same backbone LLM). We modify our BC dataset following the original ReTTool protocol to teach the model to call multiple poker tools during reasoning, and keep RL stage consistent with ReTool.
We compare both methods under the same settings Section 5.3 using Qwen2.5-77 -rren..5-BB-- Instrructt Gameplay and Reasoning results are shown in Tab. and respectively. These results show that while ReTool improves over prompting-onk LLMs, ToolPoker achieves higher gameplay perfor- mance and expert-level reasoning quality, demonstrating the advantages our simple but effective design in ToolPoker for game-theoretic reasoning tasks.
Table 7: Gameplay comparison results of ToolPoker and ReTool in Leduc Hold'em, Qwen2.5-7B. Instruct is the backbone model.
NFSP
DQN
DMC
CFR+
Avg.
Vanilla LLM
55..5
93.00
-73.0
-8.55
-73.0
ReTool ToolPoker
+5.5
+8.5
-4.0
-8.0
+0.5
+11.5
+18.0
+1.0
-3.0
+6.8
Table 8: Reasoning quality comparison results of ToolPoker and ReTool in Leduc Hold' em. Qwen2.5577-IInstruul is the backbone model.
Method
HR
FA
AC
Avg.
Vanilla LLM
0.95
0.86
1.68
1.16
ReTool
1.84
1.65
1.88
1.79
ToolPoker
1.96
1.95
1.91
1.94
G.8 IMPACT OF REWARD COMPONENT IN
To study the contribution of each component in the composite reward, implement three ablative variants of ToolPoker in Leduc Hold' em Swwen2.577B-Instruct backbone), each removing one com- ponent from the composite reward in Eq. 4 All other settings follow Section 5.3 We report both
gameplay performance and reasoning quality in Table 9 and 10 From these tables, we observe:
Ranswer the main driver of improvement. Removing it will make reasoning traces and final decisions less tightly aligned with solvers' outputs (c.g.. GTO-consistent action). leading to worse gameplay performance and reasoning quality (e-g, AC).
Riormat mainly stabilizes format and structure, with a smaller but positive effect on performance. Removing Roonat keeps gameplay competitive.
Rooo benefits reliable tool use. Removing leads to slight drop in gameplay performance and FA/AC scores.
G.9 REWARD VISUALIZATION
We plot the per-component reward trajectories of ToolPoker in Ledue Hold' em in Fig. Qwen2.5- 7B-Instruct is the backbone model. From the figure, we can observe:
Rfformat and Rtoot rapidly approach near 1, indicating that the model can learn to produce correct formats and tool invocation quickly.
25

---

Published as a conference paper at ICLR 2026
Table 9: Gameplay performance of ToolPoker and ablations in Ledue Hold'em em. Qwen2.5-7B-
Instruct is the backbone model.
NFSP
DMC
CFR+
ToolPoker/ R unswer
- 58.5
-72.0
54.5
ToolPoker/ Roormaa
+11.5
+1.0
-4.0
ToolPoket/
+9.0
+0.5
-5.5
ToolPoker
+11.5
-1.0
-3.0
Table 10: Reasoning quality metrics across ablations in Ledue eo.d'em. Qwen2.5-7B-Instmu. the backbone model.
Method
HR
FA
AC
Avg.
ToolPoker/ Ranswer
1.89
1.08
1.45
1.58
ToolPoker/ Riormaa
1.95
1.95
1.91
1.94
ToolPoker/ Ruol
1.95
1.89
1.87
1.90
ToolPoker
1.96
1.95
1.91
1.94
Ranswer gradually increases over training with some variance but no signs of instability or col- lapse.
Ovenell Roward
0.85
0.80
Answer Reword
0.7
0.6
Pormat Roward
1.00
Tool Usage Reward
1.00
0.5
0.98
0.98
0,70
0.96
0.65
0.3
0.94
130
Training Step
Training Step
Training Steg
Training Step
(a)
(b) Ranower
(c) Aiormat
(d) Rial
Figure 3: Reward visualization of ToolPoker in Ledue Hold'em. Qwen225-7B-Instruct the back-
bone model. (a) is the overall composite reward R, (b)-(d) shows the Ranswer, Riormat and Roooss respectively.
H IN-DEPTH ANALYSIS OF TOOLPOKER
H.1 TRANSFERABILIT & SCALABILITY
Extending to Other Imperfect-infoormaioo Games. Although ToolPoker is empirically evaluated on poker the main paper, the framework itself is not poker-specific. We choose poker primary testhed because is a canonical benchmark for imperfect-information. game-theoretic rea- soning: it has mature equilibrium solvers (e.g., CFR+), well-established evaluation protocols, and
is widely used in prior works (Guo et al., 2023 Wang et al. 2025 Zhuang et al. 2025 Duan et al., 2024) to study strategic reasoning.
ToolPoker is architecturally game-agnostic and only requires access to a solver that, given a state description, returns equilibrium quantities (e.g., optimal actions, values, strategy distributions). To instantiate ToolPoker for another
game, the required modifications are mini-
mal:
Build solver API. In new game, collect required solvers for game-theoretic reasoning, and build a unified solver API that returns all supporting quantities from these solvers.
State encoding. The game history, private information, and public observations of the new game must be encoded into text suitable for the I.LM and for the unified solver API.
TIR reasoning dataset construction. Similar poker, create small-scale expert reasoning dataset containing high-quality reasoning traces augmented with solver outputs. This teaches the model how read and interpret solver quantities and how produce game-theoretic explanations.
26

---

Published as a conference paper at ICLR 2026
Two-stage training pipeline. We apply the same training procedure used in Section 5.2, which contains SFT on the solver-augmented reasoning dataset, followed by RL fine-tuning with our composite reward to refine tool-use behavior and action quality.
As an illustrative example, consider extending ToolPoker to an imperfect-information game, Mahjong. We would:
encode each player's private hand, open melds, discards, and round context into text
build unified API such that the LLM query this API to interface with external Mahjong solver to obtain actions (e.g., discard, call) and other supporting quantities (e.g.. shanten count, tile-efficiency metrics, expected value, defensive risk), which similar equities and ranges in poker
build a small colver-augmented reasoning set grounding explanations in strategic principles of Mahjong (e.g., tile efficiency, defense, hand value)
apply the same two-stage training pipeline to finetune LLMs
Empirical Results of ToolPoker in Extending to Three-player Ledue Hold'em em. To further demonstrate scalability, adapted ToolPoker a three-player Limit Texas Hold'em We fol- low the steps above to fine-tune Qwen2.5-7BB In-struutttt using ToolPoker. We choose GPT-4.1-min and vanilla Sive2.5-7B-Inntruue as the opponents, and compare the gameplay performance of the resulting model under the same settings in Section 5.3 The gameplay and reasoning quality results are reported in Tab. 11 and 12
Table 11: Gameplay performance comparison across models in 3-player Leduc Hold'em
Qwen2.5-7B
GPT-4.1-mini
Owen2.5--BTruiPoeer
-36.7
+5.9
+30.8
Table 12: LM-as-a-Judge score (0-2) evaluating reasoning traces of various LLMs in 3-player Leduc Hold'em em.
Method
HR
FA
AC
Avg.
Qwen2.5-7B
0.93
0.88
1.60
1.14
GPT-4.1-min
1.00
1.75
1.83
1.52
1wwen2.5-BBrooPol.e
1.93
1.90
1.88
1.90
From these tables, observe that ToolPoker consistently outperforms vanilla LLM across both gameplay performance and expert-level reasoning scores in this new game, providing empirical evidence that ToolPoker generalizes beyond poker to other imperfect-information domains.
H.2 ERROR ANALYSIS TOOL
In this subsection, provide an in-depth error analysis of ToolPoker.
Error patterns discussion. As shown in Tab. 5 ToolPoker slightly underperforms CFR by 3 chips 100 games, while still achieving comparable overall gameplay. To better understand this phe- nomenon, we conduct an error analysis and observe the following error patterns
State mis-specification. The model may sometimes encode the game state (e.g.. hand card, public card) imperfectly before querying the solver, which can lead to suboptimal actions and quantities from solvers.
Misalignment between solvers' outputs and final actions. In some cases, the LLM may cor- rectly receive solvers' outputs (e.g., action) but does not faithfully follow them in the final answer.
Potential Mitigation. To mitigate these errors, consider several potential methods:
27

---

Published as a conference paper at ICLR 2026
Additional faithfulness reward term: Inspired by recent work faithful agentic search (Xu et al. 2025), we can train reward model to score how faithfully the reasoning aligns with solver outputs, and use this auxiliary reward during RL fine-tune.
Consisteney-awan signal: Similarly, we can add auxiliary reward during RL fine-tuning to encourage correctly querying the external solvers with accurate states.
H.3 ROBUSTNESS OF TOOLPOKER
Robustness to noisy or human-style inputs. A natural question is how ToolPoker handles inputs that deviate from clean CFR-style play, such as suboptimal, noisy, or human-generatee trajectories. Although our main experiments rely on solver-labeled data, we emphasize that ToolPoker is already trained and evaluated in settings that include substantial off-equilibrium and non-expert behavior.
(i) Training already includes noisy, off-equilibrium states. As described Appendix G.5, our RL dataset is constructed by letting a pretrained CFR agent play against a random opponent in both Ledue and Limit Texas Hold'em. We record all states but only use the CFR agent's 's actions as labels. Because the random agent frequently deviates from equilibrium play, the resulting trajectories con- tain diverse and imperfect state distributions far from idealized CFR self-play. Thus, ToolPoker trained on broad range of noisy, non-CFR game patterns rather than purely clean solver trajectories.
(ii) Evaluation already involves diverse, non-expert opponents. In online evaluation, ToolPoker plays against several traditional imperfect-information algorithms (NFSP, DQN, DMC) and LLM- based agents (e.g., prompting-only. BC+RIRL). These opponents generate highly variable and often con-equilibrium strategies. ToolPoker's consistent superiority across these settings demonstrates that it does not overfit to synthetic solver traces and can robustly respond to suboptimal or noisy play.
(iii) Why ToolPoker is expected to generalize to human gameplay. At inference time, ToolPoker does not rely imitation of historical actions. Instead, it queries the unified solver API retrieve cquilibrium-oriented quantities (e.g.. optimal action, equities, ranges) for the current state. Because solver outputs depend only on the observed regac-re of whether the trajectory arose from CFR, heuristics, or human mistakes-TToolPoke can consistently anchor its reasoning to accurate game-theoretic guidance. This design inherently promotes robustness to dfittr-iiittiuutiin human- style inputs.
While we have not yet evaluated ToolPoker on real human gameplay, extending our assessment to human or crowd-sourced datasets is an exciting direction for future work.
I DISCUSSION OF FUTURE WORKS
Our research paves the way for further exploration TIR in more complex strategic settings, shifting the focus beyond solely improving models' internal policies. Future work may explore richer tool ecosystems, multi-agent interactions, and principled frameworks for balancing internal reasoning with external computation, ultimately advancing the development reliable Al systems for high-
stakes decision making (Chen et al. 2024 Lin et al. 2025a Chen et al. 2025a).
J LLM USAGE
We used OpenAILLM (GPT-5) writing and formatting assistant. In particular, it helped refine grammar and phrasing, improve clarity, and suggest edits to figure/table captions and layout (c.g., column alignment, caption length, placement). The LLM did not contribute to research ideation, experimental design, implementation, data analysis, or technical content beyond surface-level edits. All outputs reviewed and edited by the authors, who take full responsibility for the final text and visuals.
28

---

Published as a conference paper at ICLR 2026
Table 13: Realistic Examples of Qwen2.5-3B-Instruc in playing Limit Texas Hold'em em. demon- strates three limitations of LLMs in playing poker: (i) Heuristic reasoning; (ii) Factual Misunder- standing; (iii) Knowing-Doing Gap. Errors identified during reasoning are highlighted red.
Prompt
You are professional poker player playing 2-handed limit-holdem Poker. The following will be game scenario and you need to make the optimal decision.
Here a game summary:
Limit Hold em, each player receives exactly one private card, Small Blind and Big Blind ante and 2 chips, respectively. Everyone started with 100 chips. The player positions involved in this game are Small Blind, Big Blind.
this hand:
Your position is Big Blind, and your holding is Your card: ('Two Clubs', "Ten of Hearts') ].
Community card: Not yet revealed
Current betting round: pre-flop
Current pot: 6 chips
Your admissible actions:
fold
call
raise
Now is your turn make
To remind you, the current pot size 6 chips, and you in position Big Blind, and your holding is Your card: ("Two of Clubs', "Ten of Hearts").
Decide on an action from the admissible actions based on the strength of your hand on this board, your position, and actions before you.
Your optimal action
Model Output by Qwen2.5-3B-Instee
<think> Let's analyze the situation:
Position and Pot Size: You are in the Big Blind position, which is disadvantageous position as you are forced to bet first. The pot size is 6 chips, and you have 50 chips total, so you have a 90% stack to the pot. Community Cards: The community cards not yet revealed, there is no information about the board
yet.
Your Hand: You have the 'Two of Clubs' and 'Ten Hearts', This is suited connector, which a decent hand. However, it is not strong hand by itself, and it is vulnerable to flush draw or straight draw, Opponent's Hand: The opponent in the Small Blind position, which a more advantageous position they can see your bet and respond accordingly. They have 50 chips well, they have a 90% stack to the pot.
5. Betting Rounds: This is the pre-flop round, and the only bet is the small blind. The big blind (you) is the first to act.
Admissible Actions: You can fold, call, raise. Given the above considerations, let's evaluate the options: Fold: Folding would be a safe play if you believe your hand is not strong enough to compete with the opponent's potential hand. However, you have decent hand, and the pot odds favorable if you get raise.
Call: Calling would mean you are matching the small blind's bet. This is neutral play, but it doesn't 't give you any advantage or disadvantage.
Raise: Raising would give you more information about your opponent's hand. If they call, you can see their hand and decide whether to continue or fold. If they fold, you win the pot with your hand. Considering the strength of your hand and pot odds, raise the optimal action. can raise to chips. which a reasonable amount to get your opponent to consider folding or calling. Therefore, the optimal action is to raise. </think>
aanswerbraseed answer>
29

---

Published as a conference paper at ICLR 2026
Table 14: Realistic Examples of GPT-4o in playing Limit Texas Hold'em. It demonstrates three limitations of LLMs in playing poker: (i) Heuristic reasoning; (ii) Factual Misunderstanding Errors identified during reasoning are highlighted in red.
Prompt
You are professional poker player playing 2-handed limit-holdent Poker. The following will be game scenario and you need to make optimal decision.
Here a game summary:
Limit Hold' em, each player receives exactly one private card, Small Blind and Big Blind ante chips, respectively. Everyone started with 100 chips. The player positions involved this game are Small Blind, Big Blind.
this hand:
Your position Small Blind, and your holding Your card: ['King Spades', "Ten Clubs').
Community card: Not yet revealed
Current betting round: pre-flop
Current pot: 3 chips
Your admissible actions:
fold
call
raise
Now is your turn make
To remind you, the current pot size 3 chips, and you in position Small Blind, and your holding Your card: ['King of Spades', , 'Ten Clubs'].
Decide on an action from the admissible actions based on the strength your hand on this board, your position, and actions before you.
Your optimal action is:
Model Output by GI'T-40
<think> heads-up Limit Hold'en em pre-flop, hand values are generally stronger than full-ring games because the range of hands played is wider and aggression is more frequent. The hand King-Ten offsuit is marginal hand. is not a premium hand, but it is playable in heads-up situations, especially from the Small Blind position where you have the advantage of acting first post-flop.
Given that the pot is chips (small blind chip big blind 2 chips), and it is pre-flop with no bets yet, might consider calling to see a flop. However, KT offsuit is not strong enough to profitably continue, and defending wide risks being exploited.
Raising here not justified, the hand lacks sufficient value and fold equity. Folding avoids marginal spots and preserves chips for stronger hands.
Therefore, the optimal GTO decision is to call and see the flop.s/think>> wanwer>fold</assseee
30

---

Published as a conference paper at ICLR 2026
Table 15: Heuristic Reasoning (HR) Judge Prompt.
HR Judge Prompt Template
You judge evaluating the factual correctness of an LLM's reasoning in poker decision.
You will be given:
(1) The game context (hand, position, pot size, betting round, etc.)
(2) The LLM's reasoning text
(3) The LLM's final action
(4) Objective equity analysis from an external tool (hand strength, win prohability, Monte Carlo equity, etc.)
Your task:
Identify if the reasoning shows reliance on "heuristics* (e.g.. "I feel strong." "I have face "ard,""opponn"""" "opponent probably bluffing") rather than "stralegic/game--hherreeee principles* (e.g., pot odds, equity calculation, Nash equilibrium considerations).
Score the reasoning on a 0-2 scale:
0 = Purcly heuristic-driven (surface-level intuitive analogies, rigorous reasoning)
= Mixed (some hcuristic reasoning, some strategic/game-theomee thhoorccccic reasoning)
2 Principled (reasoning grounded mainly in sound game- theorete probabilistic principles)
Output format (JSON only):
"heuristic_reasnn www.oorinascoooooooo. _ccore:: 0 1 2,
exxplaaation": "Brief explanation citing specific parts of reasoning."
Table 16: Factual Alignment (FA) Judge Prompt Template.
FA Judge Prompt
You are a judge evaluating factual correctness an LLM's reasoning in poker decision.
You will be given:
(1) The game context (hand, position, pot size, betting round, etc.)
(2) The LLM's reasoning text
(3) The LLM's final action
(4) Objective equity analysis from external tool (win probability, hand range of both you and the opponent.
etc.)
Your task:
Compare the LLM'' reasoning with the objective ground truth.
Identify whether the reasoning contains factual misunderstandings, such as:
Incorrect classification of hand strength (c.g., calling AA "a weak hand")
Misstating probabilities or equity
Incorrect statements about positions, betting order, pot size, or available actions
Misinterpreting community cards, hole cards, or ranges
Do not penalize the L.M for strategic differences (e.g., preferring raise vs. call), only for factual inaccura-
cies.
Scoring (0-2 scale):
0= Major factual errors (core aspects wrong, c.g., misclassifying AA as weak, misstating betting order)
Minor factual emrrr (some inaccuracies but overall interpretation mostly correct)
2 = Factually correct (no significant inaccuracies; reasoning aligns with objective equity and context)
Output format (JSON only):
{
"factual_come correctness_scors 2,
"explanation": "Brief' explanation citing specific parts of the reasoning."
31

---

Published as a conference paper at ICLR 2026
Table 17: Action-reasoning Consistency Judge Prompt Template.
AC Judge Prompt
You judge evaluating the consistency hetween L.M'' articulated reasoning and its final decision in
a poker hand.
You will given:
(I) The poker state description (public cards, private cards, pot size, etc.)
(2) The LLM's step-by-step reasoning text
(3) The LLM's final action (fold, call, raise, etc.)
Your task:
Determine whether the LLM's 's reasoning logically implies the same action as its final decision.
the reasoning suggests one action (e.g.. fold) but the final action differs (e.g., call or raise), this a "knowing- doing inconsistency."
If the reasoning and action align, mark as "consistent."
Scoring (0-2 scale):
The reasoning clearly points to one action, but the final decision is different. Partially consistent The reasoning mixed, ambiguous, suggests multiple options, with the final action aligning with only part of the reasoning.
0 = Inconsistent -
2 Fully consistent The reasoning unambiguously supports final action and contradictions are present.
Output format (JSON only):
coattual.correetessssoomm 0 0 2.
"explanation": "Brief explanation of why reasoning matches, partially matches, mismatches the final action."
32

---

Published as a conference paper at ICLR 2026
Table 18: Behavior Cloning Dataset Construction Template.
Judge Prompt
You are an poker expert working on {game_name) .name) explaining optimal CFR-based play using rigorous game theory. Your task is generale reasoning traces explain the action from CFR from the game-theoretic perspective like professional poker players.
You will be given the following information:
Game: [game_name)
Street: (statteenaaaee name);
Position: (posit ion_mme):
Stack (remaining chips):
my stack: (my-stack),
opponent stack: Copponent.stacee
Pot (betted chips): (total.pot), my pot: (my-pot), opponent pot: (epponent.pot))
Blinds/Antes: (antes]:
Board/Public Card: {public card];
Action History: faction history);
My Winning Probability: (my.equity)
Opponent's Winning Probability: (epponent.equityy)
My Hand Histogram: (my-hand_his) Opponent's Hand Histogram: (opponent.hand.his))
Mix Action Strategy from CFR: (actiondist)
<think>
1) Situation Summarize current situation from my perspective, e.g., position, stack,
2) Range Estimatation - Explain the range estimation of my hand and opponent's hand based on the hand histogram, respectively. And compare them what range of hand do continue with?
3) Board Fit - Explain how the board texture interacts with both ranges, and whether future cards are more favorable me the opponent.
4) Hand Quality Judge my hand's strength given its equity compared to opponent's hand, and also utilize the hand histogram estimate who holds more the strongest possible combos.
5) Strategy Mix -
We are given the action probabilities from CFR, explain the strategy that which action is more likely taken to achieve equilibrium.
6) Stack Sizes/ Pot Size Analyze the chips remaining and the pot size to see how much room there is to maneuver.
Opponent strategy Analyze the action history predict what my opponent's strategy facing my bet.
8) Final Action Based on the above analysis, explain the final action that should take.
</think>
Be concrete, mathematical, and educational. Reference specific probabilities and regret values. Write clear numbered steps (1-8)
Format your response as:
<think>
[Detailed step-hy-step strategic analysis with specific numbers]
</think>
<answer> fexpert action] </ answer>>
33

---

Published as a conference paper at ICLR 2026
Case Study BC-RIRL fine-tuned Qwen2.5-7B in Leduc Hold'em em (Part Errors iden- tified during reasoning are highlighted in red.
Table 19:
Prompt
You are an poker expert working on Leduc Hold'en em cards: 2J, 2Q. two suits) explaining optimal CFR-based play using rigorous game theory. Your task is to generate reasoning traces to explain the action from CFR from the game-theorctic perspective like professional poker players.
Ledue Hold'em a simple two-player zero-sum imperfoct-inforration poker game, The deek contains six cards: two Jacks (J), two Queens (Q), and two Kings (K). One private card is dcalt to each player, and one card is placed aside unseen. There are two betting rounds: Pre-flop and Flop. In the Pre-flop stage, the community card is unseen. And in the Flop stage, single community card is then revealed. In each round, the first player must bet or check. If bet is made, the other player must call or fold or raise (up to one raise per round). If one player raises, the other player can only call the raise fold. All bets of fixed size. At showdown, a pair beats any high card; between high cards, K > Q J.
Here is a game summary:
In unknown, each player receives exactly one private card, and small blind antes chips and big blind antes 2 chips. Everyone started with 100 chips. The player positions involved in this game Small Blind, Big Blind.
In this hand:
Your position is Small Blind, and your holding is Your card: Queen of Heart.
Community card: King of Spade
Current betting round: flop
Current pot: 4 chips
This is the historical action of the game: Small Blind call; Big Blind check;
Your admissible actions:
fold
check
raise
Now it your turn to make move.
To remind you, the current pot size is 4 chips, and you are in position Small Blind, and your holding is Your
card: Queen Heart.
Decide on an action from the admissible actions based on the strength your hand on this board, your position, and actions before you.
Your optimal action is:
Model Output by BC-RIRL fine-tuned Qwen2.5-7B-Instrree (Part I)
<think> Situation & Game State:
Position: Small Blind (SB)
Current street: Flop
Pot size: chips (SB + BB 2 previous actions)
Action history: SB called, BB checked
Effective stacks: 98 chips behind for both players
Your private card: Q
Board card: K
Hand Quality and Board Interaction:
You hold QV while the board shows K
Your hand strength is currently just high card (Queen), with potential to improve QQ
Because one King is already on the board, the chance opponent holds KK is reduced.
Relative the hoard, your equity moderate not strong.
Strategy Mixing & CFR Output:
CFR cquilibrium prescribes calling here with probability 1.0.
Calling is unique optimal action.
Folding or raising has negative expected value.
continue in Tab. 20
34

---

Published as a conference paper at ICLR 2026
Table 20: Case study of BC-RIRL fine-tuned Qwen2.5-7B Leduc Hold' em (Part II), which is continued from Tab. 19 Errors identified during reasoning are highlighted in red.
Model Output by BC-RIRI. fine-tuned Qwen2.5-7B-Instruee (Part
Stack Sizes Future Commitment:
Pot-to-stack ratio is small (4 vs 98).
A call costs very little and preserves flexibility.
Raising would overcommit without justification.
Opponent Range Estimation:
BB checked, capping their range to many weak hands and some traps. Against this range, equilibrium prefers passive call.
Information Set & Regret Minimization:
CFR regret tables show positive cumulative regret only for calling. Folding/raising have negative regret, thus dominated.
Conclusion:
The optimal action to call.
This preserves equity and follows CFR equilihrium.
</think>
<answer>eall </ answer>
Table 21: TIR rollout structured prompt template.
TIR Rollout Prompt
You are helpful AI Assistant that provides expert poker analysis. You are given a partial observation from player and admissible actions, you will need provide the GTO action win the game. You now have the ability to invoke external poker solvers to obtain the GTO action and other auxiliary quantitics (c.g., equity, all players' possible hand ranges, and the action distribution from CFR) during the reasoning. The tool calling will be wrapped in <cco>></tooll The returned results will be wrapped in <output> </ </output> to help provide precise theortte reasoning with GTO final actions.
The reasoning process is wrapped in <think> </think> tags. The final action wrapped in <answer> </ /answr>>.
Note that:
To invoke poker solvers, please write JSON command inside <tool> </tool> tags. Each command should have name and args key, with args being a dictionary.
params ams-valuvell ...) </tool> Tools expect specific JSON input formats Do not make tools or arguments that are not listed.
Example: co steol>so >>solvee (params-name)
the tools. you will see the tool outputs inside <output> </output> tags in the same order from the system.
After you have
For example, can respond in the following format:
<thi /hhnno//Reaaoning processk</thikk <tool>/Poker solver exccution atguments/</tool> <output>(Output of poker solver\</output>> <think>\Reasoning process|<<ttinks Lanswer>[Find [aooon)//nsswr>....
35

---

Published as a conference paper at ICLR 2026
Table 22: Example of tool-integrated behavior cloning data
Structured Prompt
(omit the prompt description here)
Structured Output
<think>
Situation
You are on the turn in Limit Hold' em acting from the Small Blind with stacks remaining 90 cach and the pot currently 20 (you and opponent have each contributed 10). The board S3, SJ, HQ, SK (three spades: A J., K ; also J-Q-K high cards).
The Big Blind checked to you on the turn.
Let's call poker solver get GTO actions, equities hand ranges. (ardd 'J' ' HQ' 'K' 1, 6, gr_rrasse_num-i
<\think>
<tocl>sciver (player_card=[""_ (C"' my_pet=6, opponent_ legal_actions- raise', feld' "check'
<think> (omit reasoning process here) </think>
<answer>calk/anssee
36


Reviews:
[{"review_id": "CIzY0ILvW2", "reviewer": "Reviewer_3gDV", "summary": "The paper uses poker to investigate game-theoretic reasoning in LLMs. It first compares vanilla LLM performance against several traditional baselines for game-theoretic reasoning, finding that LLMs noticeably underperform these baselines. An approach for improving LLM performance on poker is then introduced, with two main components: (1) behavior cloning, which consists of fine-tuning the LLM on expert-level trajectories augmented with reasoning traces and (2) RL fine tuning. This approach fails on its own, but these two high-level components are incorporated into the paper's last main contribution, ToolPoker, which allows the LLM to incorporate calls to external poker solvers into its reasoning trace (and is trained on a similar combination of behavior cloning and RL fine-tuning).", "questions": "1. What sets ToolPoker apart from other tool-use approaches?\n2. Can ToolPoker be generalized to other imperfect-information games? What changes might have to be made (other than the tools called)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is very clear in its presentation and generally high in quality: each newly introduced approach built on the last in a way that made the paper particularly easy to follow and made the motivation for the different components of the ToolPoker system apparent. The main points of significance and originality for the paper are that (1) it evaluates how LLMs reason about poker and presents qualitative and quantitative analyses of particular types of shortcomings in the reasoning process of vanilla LLMs and (2) applies a tool use framework to the poker setting.", "weaknesses": "The primary weakness of this paper lies in the novelty of the approach: while the paper does a very good job analyzing LLM performance on poker and explaining why the ToolPoker approach was developed, it is not clear if there is anything that sets ToolPoker apart from other tool-use frameworks, other than the task setting. In particular, explicitly comparing the strengths of ToolPoker with other approaches like ReTool (mentioned in the paper) would be helpful in evaluating the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RS9N1UorP7", "reviewer": "Reviewer_whik", "summary": "This paper investigates the strategic reasoning capabilities of LLMs in the domain of imperfect-information games, using poker as a rigorous and interpretable benchmark. The authors systematically evaluate several LLMs across realistic poker environments, including Leduc Hold’em and Limit Texas Hold’em, assessing both gameplay outcomes and the quality of reasoning traces. The results show that existing LLMs exhibit three recurring flaws. To address these deficiencies, the authors first test a two-stage internal improvement pipeline—behavior cloning followed by reinforcement learning with step-level rewards. Although this approach yields more coherent, human-like reasoning, it remains insufficient for accurate game-theoretic play. Motivated by these limits, the paper introduces ToolPoker, a TIR framework that enables LLMs to call external poker solvers for GTO actions and quantitative support such as equity and hand ranges. ToolPoker unifies solver interfaces through a single API, constructs an expert-level reasoning dataset augmented with solver outputs, and trains models using a combination of supervised fine-tuning and PPO-based reinforcement learning", "questions": "1. Whether ToolPoker has any formal link to equilibrium convergence or regret minimization theory? Specifically, under what assumptions does the interaction between the LLM’s policy and the external solver guarantee or approximate GTO consistency?\n\n2. How sensitive is the model’s performance to the weighting of the three reward components-answer, format, tool execution? Were any instabilities observed when tuning these hyperparameters?\n\n3. Does ToolPoker include any mechanism to regulate when the solver should be called, or does it always invoke the solver at each decision step? If so, could you analyze cases where solver calls lead to redundant or conflicting information, and whether the model learns adaptive tool-use behavior over time?\n\n4. The paper cites ReTool (Feng et al., 2025) but does not present a direct comparison. Given the conceptual similarity—both integrate external APIs into LLM reasoning—could the authors discuss key differences in design philosophy or experimental setup?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The first systematic study analyzing LLM reasoning and action alignment in poker, identifying fundamental weaknesses in heuristic dependence, factual errors, and knowing–doing gaps.\n\n- A detailed investigation of whether behavior cloning and step-level RL can internally mitigate these flaws, revealing their limited capacity to achieve GTO-consistent reasoning.\n\n- ToolPoker integrates external solvers into LLM reasoning for imperfect-information games, with a unified API and solver-augmented training corpus.", "weaknesses": "1. The composite reward (Eq. 4) combines R_answer, R_format, and R_tool with tunable weights.  How each component quantitatively contributes to tool-learning behavior. Providing ablation or sensitivity analyses—e.g., varying α_f and α_t, or visualizing reward trajectories—would improve transparency and reproducibility. Moreover, discussing how the model avoids reward hacking, e.g., overusing the solver or formatting cues without deeper reasoning, would strengthen the credibility of the RL setup.\n\n2. ToolPoker’s underlying design—LLMs invoking structured external APIs and fine-tuned with PPO—is conceptually close to frameworks like ReAct (Yao et al., 2023), Toolformer (Schick et al., 2023), and ReTool (Feng et al., 2025). The paper would benefit from explicitly contrasting how ToolPoker extends these approaches to imperfect-information and equilibrium-seeking contexts, possibly through a comparative discussion table or controlled ablation with ReTool baselines. Without this, the contribution risks being perceived as an application-specific adaptation rather than a fundamentally new paradigm. \n\n3. To reach the stated goal of establishing a “principled, general framework for tool-integrated strategic reasoning,” it would benefit from stronger theoretical justification. Why or when the hybrid architecture (LLM reasoning + solver calls) should converge toward a game-theoretic equilibrium, nor how tool-use uncertainty affects strategic optimality.  for instance, linking solver-invocation frequency to bounded rationality or expected regret", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eInVfE92ea", "reviewer": "Reviewer_4Bxf", "summary": "This paper investigates the ability of LLMs to perform game-theoretic reasoning in the domain of two-player poker, a challenging incomplete-information game. The authors identify three fundamental reasoning flaws in current LLMs: reliance on shallow heuristics, factual misjudgments, and a marked \"knowing-doing\" gap between articulated reasoning and actions taken. Following attempts to mitigate these flaws using behavior cloning (BC) and regret-inspired reinforcement learning (RIRL), the authors propose ToolPoker in which LLMs interface with external solvers (e.g., CFR and equity calculators) via a unified tool API. ToolPoker combines imitation and RL to ensure both game-theoretic optimal (GTO) play and reasoning traces aligned with professional-level principles.", "questions": "1. Can the authors provide more details on how the LLM-as-a-Judge scores were calibrated and validated? Specifically, are there estimates of both inter-rater LLM agreement and alignment with human judges beyond the 20-trace reference set? How sensitive are reasoning scores in Tables 2 and 4 to the prompt, model, or domain?\n\n2. How does ToolPoker handle non-standard, noisy, or human-style inputs that diverge from solver-generated play? Has the method been evaluated on datasets that originate from human-expert games or crowd-sourced play, and how robust are the results in such settings?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and well-organized. The proposed method is simple and easy to follow.\n2. This paper presents extensive experimental results with detailed analysis on the ablation study as well as limitations.", "weaknesses": "1. The novelty of the paper is limited. It mainly applies reinforcement learning to a large language model using a classic game-theoretic solver (e.g., CFR+) as the reward signal or direct PPO, without introducing any fundamentally new algorithmic contributions or insights. Essentially, the work repackages standard solver outputs within an RL fine-tuning framework, resulting in incremental rather than conceptual advancement. Also, while the related work section is decent, it omits several works that would be natural inclusions, such as [1]\n\n2. While the core analysis and ToolPoker formulation are well-motivated by Leduc Hold'em and Limit Texas Hold'em, the current instantiation is restricted exclusively to these benchmarks. There is little empirical or conceptual discussion about scalability to larger, multi-player, or variable-rule settings. \n\n3. The system occasionally produces factual misunderstandings when tool outputs are unavailable or misinterpreted, revealing a continuing challenge for robust factual alignment.\n\n4. Most experiments are performed with synthetic datasets or CFR-solver-based action labels, raising questions about how well ToolPoker would generalize to noisy or inherently human gameplay traces.\n\nReference:\n\n[1] Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1bQrOSUZTQ", "reviewer": "Reviewer_4eQy", "summary": "This paper tackles poker as a domain for strategic reasoning and analyzes the shortcomings of LLMs on poker. \nThe authors examine what features of reasoning correlate with poor performance, finding gaps between action and knowledge and flawed or heuristic reasoning. \nModels are compared to a range of baseline solvers.\nTo improve models, they propose a training approach consisting of behavior cloning on curated data followed by RL using PPO. They find that this improves performance over the untrained model, but that it still lags behind solvers like CFT. \nTo further improve the model, they propose ToolPoker, a method which enables LLMs to recruit external solvers and tools to improve.\nThey train ToolPoker also via a combination of BC and PPO and find that it further improves and closes much of the gap between models and CFP.", "questions": "Minor comments:\n- Tables 1, 3, 4 are unreadably small", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper compares to standard baselines\n- games are run across multiple runs \n- both open and closed-source models are considered\n- qualitative analysis of reasoning is backed by quantitative results\n- The LLM judge is externally validated\n- BC+RL results show improvements, with ToolPoker showing further improvements", "weaknesses": "- ToolPoker improvements are not that surprising to me. Basically the gains here seems to boil down to externalizing the parts of the task that are more difficult for the model to external solvers. Implementationally, the training method might be useful, but I don't see what research question it is addressing that hasn't already been addressed by the BC+RL experiments. It seems like in the end, ToolPoker is largely introducing an engineered system that would be of limited use, since it offloads most of the strategic reasoning the model would do to other tools (so no longer getting at the motivating point of the paper on strategic reasoning) while also being costlier and less effective than baseline poker-playing solutions. \n- L083 the claim is made that ToolPoker is for imperfect-information games, but it is only evaluated on poker, and seems very explicitly engineered for poker with limited transfer to other games. \n- The reasoning analysis does not take into account that reasoning might not be faithful. Plenty of prior work has called into question whether reasoning traces from LLMs are faithful explanations of their behavior (see https://aigi.ox.ac.uk/wp-content/uploads/2025/07/Cot_Is_Not_Explainability.pdf for references). The analysis seems to hinge on reasoning being a causal explanation of the model's actions -- if that's not the case, it would explain the knowledge-action gap. \n- if ToolPoker uses CFR solver as a tool, why does it not outperform CFR in Table 5? L436 indicates it's a result of tool-calling errors, is there a way to reduce these? \n- The comparison of tool calling reasoning feels spurious. In this case, the tools are providing a lot of information that was not available to the model without tools.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)