Code World Models for General Game Playing
Authors: Wolfgang Lehrach
Daniel Hennes
Miguel Lazaro-Gredilla
Xinghua Lou
Carter Wendelken
Zun Li
Antoine Dedieu
Marc Lanctot
Atil Iscen
John Schultz
Marcus Chiam
Ian Gemp
Piotr Zielinski
Satinder Singh
Kevin Patrick Murphy
Venue: ICLR 2026 Poster (2026)
Keywords: large language models
code world models
code generation
information set MCTS
planning
partial observability
two-player games
imperfect information games
Primary area: foundation or frontier models, including LLMs
TL;DR: Instead of using LLMs-as-a-policy to play games, we use LLMs to implement an explicit code world model and combine it with a planner to play games, including imperfect information ones.

Abstract: Large Language Models (LLMs) reasoning abilities are increasingly being applied to classical board and card games, but the dominant approach---involving prompting for direct move generation---has significant drawbacks. It relies on the model's implicit fragile pattern-matching capabilities, leading to frequent illegal moves and strategically shallow play. Here we introduce an alternative approach: We use the LLM to translate natural language rules and game trajectories into a formal, executable world model  represented as Python code. This generated model---comprising functions for state transition, legal move enumeration, and termination checks---serves as a verifiable simulation engine for high-performance planning algorithms like Monte Carlo tree search (MCTS). In addition, we prompt the LLM to generate heuristic value functions (to make MCTS more efficient), and inference functions (to estimate hidden states in imperfect information games). Our method offers three distinct advantages compared to directly using the LLM as a policy: (1) Verifiability: The generated CWM serves as a formal specification of the game's rules, allowing planners to algorithmically enumerate valid actions and avoid illegal moves, contingent on the correctness of the synthesized model; (2) Strategic Depth: We combine LLM semantic understanding with the deep search power of classical planners; and (3) Generalization: We direct the LLM to focus on the meta-task of data-to-code translation, enabling it to adapt to new games more easily. We evaluate our agent on 10 different games,  of which 4 are novel and created for this paper. 5 of the games are  fully observed (perfect information), and 5 are partially observed (imperfect information). We find that our method outperforms or matches Gemini 2.5 Pro in 9 out of the 10 considered games.

Published as a conference paper at ICLR 2026
CODE WORLD MODELS FOR GENERAL GAME PLAYING
Wolfgang Lehrach' Daniel Hennes Miguel Lázaro-Gredilla" Xinghua Lou
Carter Wendelken
Zun Antoine Dedieu John Schultz
Marc Lanctot Atil Iscen
Marcus Chiam
Ian Gemp Piotr Zielinski Google DeepMind
Satinder Singh
Kevin P. Murphy
[wpl, hennes, lazarogredilla, xinghua, owendelken, lizun, adedieu, lanctot, atil, jhtschultz, marcuschiam, imgemp, zielinski, baveja, kpmurphy) @google.com
ABSTRACT
Large Language Models (LLMs) reasoning abilities are increasingly being applied to classical board and card games, but the dominant approach involling prompt- ing for direct move generation has significant trakw relies on the model's s implicit fragile pattern-matching capabilities, leading to frequent illegal moves and strategically shallow play. Here we introduce an alternative approach: We use the L.LM to translate natural language rules and game trajectories into for- mal, executable world model represented Python code. This generated model- comprising functions for state transition, legal move enumeration, and termination checks-serves serves verifiable simulation engine for tigh-performanc planning algorithms like Monte Carlo tree search (MCTS). In addition, we prompt the LLM generate heuristic value functions (to make MCTS more efficient), and infer- ence functions (to estimate hidden states in imperfect information games). Our method offers three distinct advantages compared directly using the LLM as a policy: (1) Verifiability: The generated CWM serves as a formal specification of the game's 's rules, allowing planners to algorithmically enumerate valid actions and avoid illegal moves, contingent on the correctness of the synthesized model; (2) Strategic Depth: We combine LLM semantic understanding with the deep search power of classical planners; and (3) Generalization: We direct the LLM to focus on the meta-task data-to-code translation, enabling it to adapt to new games more easily. We evaluate agent on 10 different games, of which 4 are novel and created for this paper. 5 of the games are fully observed (perfect information), and 5 are partially observed (imperfect information). We find that our method outperforms or matches Gemini 2.5 Pro in 9 out of the 10 considered games.
INTRODUCTION
Large Language Models (LLMs) have shown impressive abilities at solving various reasoning tasks, and recently have been applied as "agents" which can play classical (often multi-player) games, like Chess, Go, and even complex imperfect information games like Poker and Bridge. The standard approach is to treat the L.M policy, by asking it pick move each step using prompting strategy based on the trajectory of observations and actions seen so far, plus optional text meta data about the game. This method treats the LLM an end-to-end "intuitive player", leveraging its vast training data to recognize patterns and select moves that seem promising. However, strategic mastery often requires deep multi-step lookahead, characteristic of "System 2" deliberation (Kahneman, 2003). While strong play can be achieved through training specialist models (Ruoss et al. 2024]] Schultz et al.) 2025), direct play from generalist L.MSs often lacks deep tactical foresight, despite recent advances in "thinking" (Liao et al. 2025). as we show empirically in this paper. In addition,
"Equal contribution.
Note that Jordi Grau-Moya (jordigrau @ google.com) is the 8th author but due to purely administrative over- sight was left off ICL submission. Jord is present the contemporary arxiv submission.

---

Published as a conference paper at ICLR 2026
the LLM as policy approach does not work very well on novel games that are not part of the LLM's training set, as we will also show.
We propose use LLMs in a different way, namely as induction engines that can leverage their prior knowledge to map a small amount of observed trajectory (game play) data, plus a textual game de- scription, into plausible world models, represented Python code, using iterative code refinement methods as in Tang et al. (2024a). We call the result of this process a "Code World Model" (CWM). In the context of game playing. a CWM consists of definition of the (possibly latent) state, a func- tion that specifies which moves are legal at each step, a state transition function, an observation function (for latent states), a reward function, and a function that checks for termination. Further- more, for the challenging case of partially observable games, we introduce novel paradigm that effectively tasks the L.M with synthesizing a regularized autoeneoder: an inference function (the encoder) maps observations to plausible latent histories, and the CWM (the decoder) reconstructs observations from them, with the game's rules and API serving as a strong structural regularizer.
Although there is prior work that uses LLMs to learn symbolic world models (see See. B and then leverage them for planning, we differ three main ways. First, we handle the case of partially observed and stochastic worlds (such as Poker), whereas all prior work (to the best of our knowl- edge) either assumes fully observed and deterministic environments, or (in the case of Curtis et al. (225 assumes post-hoc observability; both cases make model learning much easier. Second, in addition to learning a CWM, we ask the LLM to generate heuristic value functions, which signifi- cantly improves the performance of our search-based policies, such MCTS and Information Set MCTS (Cowling et al., 2012). Third, we demonstrate that our approach outperforms a state-of-the- art "thinking" LLM across various two-player games, including novel (or "OOD") ones which we create, to avoid contamination issues with the training set of the LLM.
2 BACKGROUND
Interactions in multiplayer games can be described using the formalism of extensive-form games (Kuhn 1953; Shoham & Leyton-Brewt Browwn 2009 Albrecht et al. 2024 Murphy 2025): y there is a set N 11,2.... ................................ n of n players that take discrete actions a E A. Sequences of actions called histories h H; all games start at the initial empty history, and end at terminal histories Z C H. There is a special player called chance (also sometimes called nature), c, which plays with known, fixed (stochastic) policy-the chance outcome distribution-e.3 representing dice rolls and card draws. Due to chance events being explicitly represented by the game environment, each history h can be thought of as a unique transcription of a game (either finished or in progress) and as a "ground truth" state known only to the environment. At every history h, there is a player to act 7 rh) E . (c), and a set of legal actions A(h) S A. Formally defining states in partially- observable (imperfect information) requires several definitions; thus, we defer this to Appendix C[ couple it with the description of the search method (policy generation). Agents encode policies to take actions (h)) A(A), where A(-) represents a discrete probability distribution. For each agent i, the goal is to find a policy that maximize its own cumulative reward Σ (hz). However, in the multiagent setting each individual objective jointly depends on choices of other agents.
Our game environments are based on OpenSpiel (Lanctot et al., 2019): each implementation pro- vides logic to determine legal actions, transitions from one ground truth state to the next, rewards, and player observations in a general way. However, the agent does not know the true environment model. Instead, it must learn the code world model by using an LLM applied to text description of the game, together with example game play data, as described detail in Sec. 4 Given the learned CWM. we pick the best move by using existing game solvers: for perfect information games, we use MCTS, and for imperfect information games, we use Information Set MCTS (see Appendix C In both cases, we optionally augment the search algorithm with a learned value function, and in the case of ISMCTS, we augment the search algorithm with a hidden state estimator. We also tried learning policy using PPO applied to the (partially observed) CWM: Appendix B for details.
2

---

Published as a conference paper at ICLR 2026
3 RELATED WORK
There is growing interest in evaluating the abilities of LLMs to play games, as exemplified by the recent release of Kaggle Game Areng as well as other recent work (Costarelli et al. 2024 Duan
et al.
2024
Verma et al. 2025 Hu et al. 2025a Sun et al. 2025
Apolina-Kun et al. 2025 Hu
al.
2025b Guertlrr al. 2025 Mishra al. (025). Similar to these papers, our aim is to design
LLM-based agents that play text-based games. Fürthermore, like ggbench (Verma et al. 2025), ). we
assess the generality of our agents using novel games, that are (by construction) out-of cuttof-ddisribution (OOD) for the LLM. However, rather than using the LLM directly as a policy, we focus using the
LLM to generate a CWM, to which we then apply standard solvers, such as (IS)MCTS or PPO.
There are few other papers that also use model-based approach, similar to ours. "WorldCoder" generates a set of CWM hypotheses from trajectory data using LLM-poweree code synthesis, stores each hypothesis (candidate model) in a tree, and uses Thompson sampling to decide which hypoth- esis to ask the LLM to improve, see (Tang et al. 2024a). Given the learned CWM, WorldCoder uses ReAct-style methods Yao al. 2022) for decision-making GIF-MCTS (Dainese al.) 2024) developed a similar method, but uses MCTS for agent decision-making. Our work extends this past work by considering strategic multiagent environments, synthesizing value functions (to speed up (IS)MCTS), and synthesizing and refining inference functions (to handle imperfect information games).
Imperfect information games can be considered special kind of (multi-agent) partially observable Markov decision process (POMDP). Learning such models from observational data notoriously difficult. In very recent work, Curtis et al. (2025) introduce "POMDP Coder", which learns par- tially observed CWM. However, unlike us, they assume the hidden states are observed in hindsight (at the end of the trajectory). By contrast, we also consider a "closed deck" scenario, in which the hidden states are never observed. In addition, Curtis al. (2025) use a determinized belief space planner (related to the POMCP method of Silver & Veness 2010 whereas we use ISMCTS (see Appendix(C] C or PPO (Appendix E).
There are other many other ways to use LLMs for reasoning in games and multiagent systems. A recent line of work focuses on using LLMs to construct game-theoretic models of arbitrary scenarios in order to derive and deploy intelligent, strategic policies. Gemp et al. (2024) treats LLM as an environment transition operator, controllable via instruction sets. An extensive- form game tree is explicitly constructed in OpenSpiel and an equilibrium over instruction sets is computed. Daskalakis let al. 2024) demonstrates how to design a game tree for Romeo and Juliet with the assistance of an LLM, subsequently modifying the tree so that the classic story lies in the support of its Nash equilibrium. Xu et al. (2025) embeds several observed Werewolf dialogues in a latent space, clusters the messages to form finite action space and resulting game tree, and then runs counterfactual regret minimization on this discrete latent representation to derive policy. Mensfelt et al. (2024a) proposed an approach to automatically translate natural language descriptions of small bimatrix games to logic representations (similarly in Mensfelt et al. (2024b)). Most closely related to this work, Deng et al.] (2025) automated the construction of explicit (imperfect-information extensive- form game trees from natural language descriptions of games, including debugging module to ensure the resulting Gambit (Savani & Turocy/ 2024) representation was valid. In contrast to this work, they only conditioned on game descriptions (rules) not observed trajectories and applied their pipelines to games with game trees containing at most 25 decision nodes (Kuhn Poker); code-word models offer the potential to scale to much larger game instances in some cases due their more efficient encoding of repeat transitions.
4 METHODS
At high level, when confronted with a new game, our general game playing agent follows these steps: First, it plays a few games to completion using random policy. The data collected during each game forms a trajectory, which consists of observations, rewards, legal actions, and states at each timestep. Second, it uses a textual description of the rules of the game, plus the generated
Seee httph:t/////// https://www..aggle..c ctm waw.oom//aamaeee arena
3

---

Published as a conference paper at ICLR 2026
trajectories, to learn CWM Finally, the agent plays the game in arena against other opponents, using an MCTS policy built on top of the synthetic CWM. For imperfect information games (IIGs) use ISMCTS instead of MCTS. If all the synthesized elements are correct, the amount of play-time compute increases, the playing behavior of our agent gets closer to optimal. Thus, in contrast with L.L--as-a-policy agents, shift the burden the L.M from producing a good policy to producing good world model, which in turn enables planning methods to turn compute into playing performance.
4.1 SYNTHESIZING THE CODE WORLD MODEL.
A CWM is playable, approximate copy of a target game, It contains functions providing logic to update the game state when an action is taken (transition function, which includes a termination), the legal actions given state, the observation given state (observations and state differ in the case of IIGs), the distribution for chance nodes, and the reward function for a state. All these functions are deterministic, with randomness entering the game only through the actions of the chance player. To synthesize a new CWM, we provide the LLM with the game's rules and offline trajectories, and demand that it creates CWM following the OpenSpiel API (Lanctot et al. 2019) format. See Appendix H for prompt details.
A single-shot generation of the CWM will often be insufficient to produce a correct implementation of the game unless we add some kind of corrective feedback. Thus we subject the initial CWM to iterative refinement (Dainese et al. 2024 Tang et al., 2024b) to improve its quality. For refinement, series of unit tests are automatically generated from the offline trajectories. For each transition in an offline trajectory, unit tests are generated in order check the correctness of the CWM predictions as compared with the original trajectory (states, observations, rewards, legality of actions), and the absence of execution errors.
In the case of IIGs, this process requires that the offline trajectories contain not only the observations of the game and the actions of the players, but also the hidden states and the actions of all other players (including chance). The post-hoc availability of hidden states, an assumption also used in concurrent work (Curtis al., (2255,, can sometimes be unrealistic. See 4.4 introduces novel approach to handle CWM W learning from partially observed trajectories.
Unit tests are binary, so we can measure the transition accuracy as the rate of correctness of such tests. We refine the CWM until perfect transition accuracy (1.0) is achieved or our refinement budget runs out. We feed back the stack traces from failed unit tests to the LLM help the refinement. We consider two separate approaches to refinement:
Conversation (sequential refinement). This a serial "chat mode" approach, in which the stack trace of a newly failed unit test is appended to our previous interactions with the LLM to create the new prompt, and a new CWM addressing the unit test failure is requested. Failed unit tests derived from the offline trajectories
submitted to the LLM until all pass.
Tree search. Just like in the REx approach (Tang et al. 2024b a), we maintain multiple CWMs in a refinement tree structure, and use Thompson sampling to choose which CWM refine next, favor- ing those that either have high transition accuracy or have been refined few times. Each LLM call consists of fresh prompt that contains the CWM chosen be refined, the refinement instructions, and the stack trace of a failed unit test for that CWM. The prompts and hyperparameters used during synthesis are presented in Appendix
4.2 SYNTHESIZING INFERENCE FUNCTIONS FOR IIGs
One of the novelties of our work is the synthesis of inference functions to enable the use of ISMCTS planning with the learned CWM at play time in imperfect information games (IIGs). To see why this is necessary, note that ISMCTS requires that at each game step t the agent can estimate the hidden state of the game st as explained in Appendix C More precisely, at play time, agent must
We could potentially update the CWM after each step of game play, as we acquire new data, but in this paper, we learn the model up-front, given the initial offline trajectories and game description, for reasons of efficiency.
Note:
4

---

Published as a conference paper at ICLR 2026
be able to sample from its belief state PMr (s (1s) where M is the estimated CWM Since exact inference incurs an exponential cost in the worst case, we ask the LLM to synthesize code to approximately sample from the posterior, utilizing only agent i's actions ai and observations ofr so far from the offline trajectory. We consider two alternative approaches to achieve this goal: hidden history inference and hidden state inference. We describe these below.
Hidden history inference. Since all the functions in the CWM are deterministic, the posterior over the hidden state 8t can be obtained from the posterior over the action history ht, which includes the actions of the chance player. In this approach, the agent controlling player i asks the LLM to create a function that samples he ~ Par (he (o d. The CWM can then be used to execute he and recreate history of hidden states $1a and observations Ti. A unit test created for each time step in which player acts, verifying that the sampled values match the run time evidence (i.e., o of and a; = a²). This allows refinement (on the offline trajectories) to be applied to the inference function.
Once the refined inference function passes all unit tests (i.e., inference accuracy is 1.0), we can claim that the sampled hy belongs to the support of PM (he lo (of di:)). and therefore, the & gener- ated by this process belongs to the support of pas par(s) (8t jojr: a).)) Although this does not guarantee that 5, is correctly distributed, the correct support is already very informative, given the extremely sparse support of state posteriors in games. Furthermore, this approach guarantees that the sampled posterior state & is a valid CWM state. Note that at play time the (test) inference accuracy can drop below 1.0 (depending on how well the synthesized inference code generalizes to novel obser- vations), meaning that the approximate posterior samples might not always belong the support of the actual posterior. However, & is still guaranteed by construction to be a valid hidden state in the CWM.
Hidden state inference. Rather than obtaining a state posterior sample indirectly through the action history, it also possible to ask the LLM to create code that directly samples 6, ~ pM (s a²). Then, the CWM can be used to obtain or from &t. Correctness of the inference function can be partially validated by a unit test at each time step that verifies that the sampled values match the actual observations. or = or. CWM refinement can then be used to improve the synthesized infer- ence function. State inference potentially much simpler than full history inference, but it cannot guarantee that the produced sample & belongs to the support of the posterior, nor that it constitutes a valid CWM hidden state, because it ignores the dependency between consecutive states.
4.3 SYNTHESIZING VALUE FUNCTIONS
Another novelty of our work is the synthesis of value functions to speed up and improve value estimation in MCTS and ISMCTS. This can be faster (and potentially more accurate) than estimating the value of new leaf node through random rollouts. To synthesize deterministic value function (ss) to estimate the value of the (potentially hidden) state at leaf nodes, we can prompt the LLM to generate code, just there is no ground truth to compare to. Instead, multiple functions are generated and the best one is selected through a tournament.
did for learning the CWM. However, value functions not refined, since
4.4 OPEN DECK VS CLOSED DECK DURING TRAINING
So far we assumed that the offline trajectories (used to train the CWM) contained hidden state information even for IIGs. Concurrent work Curtis et al. (2025) also assumes the ability to peek at hidden states. We refer to this setup as open deck synthesi This setup is justified in several
practical scenarios, such in cooperative training environment where players share information to learn the mechanics of the game, during the design phase of a new game where developers have full
'For players other than we assume uniform prior on the legal actions defined by the CWM. Only the support of this prior affects our approach, as we will focus on posterior support, see below.
"Unlike the CWM functions, inference functions stochastic (samplers). Thus, their unit tests poten- tially stochastic, but for correct inference functions they will deterministically pass.
'e want to emphasize that in our open deck setting, hidden state information is only available in the offiine trajectories to aid CWM synthesis, and not during actual game play. Thus the players only ever see observations, but the CWM learner may hidden states (in the open deck setting).
5

---

Published as a conference paper at ICLR 2026
access to the state, or when human expert provides fully annotated "open-book" demonstrations to bootstrap an agent's understanding.
However, there are scenarios in which the agent only ever access its own observations and actions, so that the open deck assumption is violated. This would be the case, e.g., the agent plays novel game online. We refer to this scenario closed deck synthesis; to the best of our knowledge, this scenario has not been addressed in prior CWM work.
To handle this scenario, we propose to combine the pieces introduced in the previous sections to build a regularized CWM "autoencoder". The idea is as follows: we ask the LLM to generate a CWM and hidden history inference function, just like above, but drop all the unit tests that not verifiable without access to the hidden information (i.e., those checking the transition accuracy between consecutive hidden states), and we just keep the ones that we can verify (i.e., checking the result of mapping observations to hidden states and back to observations). We additionally add unit tests few iterations of random play ensuring that there are no execution errors. In other words, we refine based on the inference accuracy and lack of execution errors. This generates kind of autoencoder, where the inference function acts as an encoder, producing a hidden sequence of actions hn from oi aa and the CWM acts as a decoder, recreating the observations and actions from the latent h2. Instead of a bottleneck, or a regularization term, the game rules and the required OpenSpiel API (used in the unit tests) introduced in the context of the LLM act regularizers to prevent trivial latent spaces from being discovered. Valid posterior histories hy (i.e., those that pass all unit tests) can be used to obtain a lower bound on the likelihood of the CWM, as follows: Pas (oi.) = Mi, PM (of. (prr))))()))))))))))))( S (mr(oi:hhe)) (he) = PM (he). (The last equality follows because Par (or (h+)) = when all unit test pass.) This lower bound is tightest when hg is the maximum a posteriori, but is valid for any sample.
EXPERIMENTS
Following the approach described in Sec. 4 we build agent, which we call CWM-(IS)MCTS,, which performs CWM synthesis (using either open or closed deck trajectories), and then plays using MCTS or ISMCTS. (We also tried learning policy using PPO; see Appendix E for details.) We measure the playing abilities of our agent on multiple games against three other agents: A random legal action executor called Random; an (IS)MCTS agent that has access to the game's ground truth (GT) code, including inference functions but not value functions, which we call GCT( and an LLM as a policy, which we call Gemini 5Pro (we use "dynamic thinking", rather than specifying thinking budget). All methods have access to the same data: the rules of the game text and 5 offline trajectories. (IS)MCTS approaches always run 1,000 simulations before taking an action, using either the value function 10 rollouts (in which all players act randomly) determine the initial value of a new leaf node. A sketch of the information flow for each agent is given in Appendix G
To validate the generality of our approach we use both perfect and imperfect information games, as well as well-known and OOD games. The perfect information games are: Tic-tac-toe, Connect four, Backgammon, Generalized tic-tac-toe (OOD), and Generalized chess (OOD). The imperfect information games are: Leduc poker. Bargaining, Gin rummy, Quadranto (OOD), and Hand of war (OOD). The out- -itto--diitribttion (OOD) games are not part of the LLM's training set, and have been created by us for these experiments. See Appendix [oor the rules of each game.
5.1 SYNTHESIS ACCURACY
The CWM agent operates by synthesizing CWM of the game (and potentially other auxiliary functions) prior to playing the game, see Sec. for details. We use Gemini 2.5 Pro for synthesis. For the concrete prompts used during synthesis, see Appendix For examples of synthesized code, Appendix
Refinement attempts to increase the fraction of units tests that pass, iterating until all pass or the budget for LLM calls is exhausted. The fraction of unit tests that the CWM passes is the training transition accuracy, and the fraction of tests that the inference function passes is the training infer- ence accuracy. To check for overfitting the offline trajectories, after synthesis, we measure the

---

Published as a conference paper at ICLR 2026
accuracy separate test set of 10,000 transitions, randomly sampled from 100 games where each player is randomly assigned a random policy or MCTS on the ground truth game code. This yields the test transition accuracy and test inference accuracy. The test set is never used to train instead is used to estimate the accuracy of the learned CWMs. Finally, at play time against the L.M as policy, online transitions observed, and again used to assess the accuracy of the CWM and inference functions.
5.1.1 PERFECT INFORMATION GAMES
For perfect information games, find that learn correct CWM for all the games, and that the resulting learned models have high test (generalization) accuracy. Both conversation and tree search work very well this setting. Appendix D contains precise numbers (Tables and 7 respectively). and shows the quick convergence of the CWM with the number of LLM calls (Fig. We will stick with tree search for the remainder of this paper, since ability to backtrack confers additional resilience in harder settings.
5.1.2 IMPERFECT INFORMATION GAMES, OPEN DECK
In the case of imperfect information games (open deck learning), we find that the transition accuracy of the learned CWMs is very high, except for Gin rummy, where the training accuracy is just 84% and the test accuracy is 79%. See Table for details. We hypothesize this is due to its high degree of logical and procedural complexity. Unlike games with more uniform rules, Gin rummy involves a multi-stage scoring phase (knocking, laying off melds, calculating deadwood, and checking for undercuts) that difficult for the LLM to capture perfectly in code from small number of trajecto- ries. This highlights& a key frontier for CWM synthesis: mastering games with intricate, multi-step procedural subroutines.
We also measure the inference accuracy obtained by the synthetic inference functions. We tried both hidden history and hidden state inference (see Sec (42) Results with hidden history infer- ence (shown in Table [ and Fig are slightly better, so this will be the method of choice for the CWM-ISMCTS agent. (The results with hidden state inference are provided and Fig. Z Results for 3 of the 5 games are good, but once again we see that results for Gin rummy are quite poor (inference accuracy is only about 52%), and to lesser extent Hand of war (inference accuracy is about 94%), even though CWM accuracy for Hand of war is good (about 98%). This suggests that hidden history inference is harder than learning the transition dynamics from fully observed sequence of trajectories.
Appendix [ Table
Table 1: Imperfect info. games, CWM refinement via tree search, hidden history inference.
transition accuracy
inference accuracy
Game
OOD
LLM calls
train
test
online
train
test
online
Bargaining
1.0000
0.9827
1.0000
1.0000
1.0000
1,0000
23.0
Ledue poker
X
1.0000
0.9977
0.9942
1.0000
1.0000
1,0000
4.4
Gin rummy
0.7816
0.7455
0.9044
0.5857
0.5376
0.9678
500.0
Quadranto
1.0000
1.0000
1.0000
1.0000
0.9864
0.9916
6.0
Hand of war
1.0000
0.9814
0.9868
1.0000
0.9357
1.0000
144.0
5.1.3 IMPERFECT INFORMATION GAMES, CLOSED DECK
Finally, we consider CWM synthesis with refinement in the novel closed deck setup in which no hidden information is available, not even post-hoc. The results in Table 2 show degradation the synthesis quality with respect to the open deck setting of Table [ Despite this, game play performance does not degrade significantly, we show in the next section.
7

---

Published as a conference paper at ICLR 2026
transition (train)
transiion acc. (test)
inference acc. (train)
inference (tess)
1.-
Dataset Bargining Gin nummy
2066
0.4
Hand of war
Leduc poker Quadrants
0.0-
12 Numberol cata
14
8 10
10 12
36
20
10 16
Mum
Narber
Number LUN
Figure Evolution of the transition and inference accuracy with the number of LLM calls for imperfect games with refinement via tree search and hidden history inference.
Table 2: Imperfect information games, hidden history inference, closed deck.
inference accuracy
Game
OOD
# LLM calls
train
online test
Bargaining
X
1,00000
0.67359
0.76000
88.2
Ledue poker
X
1.00000
0.97080
0.96585
9.0
Gin rummy
X
0.05538
0.09523
0.53953
500.0
Quadranto
1.00000
0.95183
0.96085
99.0
Hand of war
0.86250
0.82130
0.94835
338.2
5.2 ARENA: GAME PLAY PERFORMANCE
In this section, we test how the previous synthesis results translate into playing performance against other opponents in our game arena. Since the CWM synthesis process is stochastic, we repeat times. automatically rejecting bad samples (see Appendix F)., and pick random CWM for each match. Results correspond to the average of 100 matches.
5.2. PERFECT INFORMATION GAMES
All of our perfect information games are ternary-outcome games, are limited to win, lose, or draw (W/L/D). Fig. 2 shows the performance our CWM-MCTS agent, when acting as Player 0 or Player 1, against three different competitors. A player forfeits when it fails provide valid action in the allotted time. The middle pair of bars each panel show CWM-MCTS playing against GT-MCTS, an upper bound for performance that uses the ground truth (GT) code of the game for planning. Both agents are similarly good, without either of them clearly winning in any of the games. This highlights the quality of our code synthesis. CWM-MCTS able to beat Gemini 2. 5Pr (which is used as policy) in all the considered games. For detailed numerical results, see Table in Appendix D We used a synthetic value function for Gen. dic-tac-toe see Fig. 9 for the ablation without value function.
Backgammon
Connect GTMCTTS
chess
Mers
Mors
Player
Figure 2: W/L/D rates for game play between CWM-MCTS and three opponents. CWMs are refined via tree search and hidden history inference.
8

---

Published as a conference paper at ICLR 2026
5.2.2 IMPERFECT INFORMATION GAMES, OPEN DECK
Our imperfect information games contain a mixture ternary-outcome games, zero-sum games and general-sum games (see Table 3 for summary of all the games' characteristics). Win/loss/draw rates and payoff distributions are shown Fig. 3 Except for Hand of war, CWM-ISMCTS beats or matches Gemini 2rro in all imperfect information games. In the case of Gin rummy, this should be interpreted as Gemini 2. 5Pro being a very weak player, you can check its forfeit rate Table I6 For Leduc poker, although our average performance is superior, Bargaining we used a synthetic value function, which results in a significant improvement when CWM-ISMCTS acts player (see Fig. Appendix D for the corresponding ablation). We did not observe an improvement or degradation in performance when value functions were applied to the other games.
also observe high variance. For
Bargaining
Leduc poker
Gin rummy
Quadranto
Hand of war
Nardon
109
Geriv OT JONTTS
handor
Gomii 2.he ST. C7.
Nardom
Gante's 2 She oT Cer
handor
53
-38
Maya Noten
Opporent NoMer
CWAEBCC'S S Plye
CWP SMCTS de "layee
mm
II Wiiby rorict
18 Brow
Los
Uso by harlok
Figure 3: W/LD rates and payoff distributions for game play between CWM-ISMCTS and three oppo- nents. CWMs are refined via tree search and hidden history inference, open deck.
5.2.3 IMPERFECT INFORMATION GAMES, CLOSED DECK
Finally, we consider the closed deck setting, in which games strictly partially observable, and no hidden state information or actions from other players are available the offline trajectories. Results degrade w.r.t. the open deck setting, but CWM-1SMCTS-Closee continues to beat or match Gemini 2Po (with high variance in the case of Leduc poker). We hypothesize that the non- intuitive improvement of CWM- IWI-II-CMMSSSsssee due to the freedom to synthesize simpler state spaces when playing closed deck. Refer to Tables [4 and T5 in Appendix Doo for detailed results.
Hand of war w.r.t. the open deck setting could be
Bargaining
Leduc poker
Gin rummy
Quadrania anto
Hand of war
Gwwi otjeners
aadon
Saw ot araees
Garin Sa
nandon
Payyr Tebar
Opporent Sebur
O 1SMCTS Payer
CWM SNCTS as 7layer
Wh
I1 WWiiy Torfeit
I1 Drew
Lom
Loes by Tortel
Figure 4: W/LD rates and payoff distributions for game play between CWM-ISMCTS and three oppo- nents. CWMs refined via tree search and hidden history inference, closed deck.
DISCUSSION
In this work we extend the existing CWM framework by considering two-player games, performing value function code synthesis to improve player performance, introducing the concept of "inference as code" to enable state estimation in imperfect information games, and providing a learning algo- rithm (based on code-based autoencoders) to enable learning in the novel closed deck (strict partial observability) setting. Our results show the superiority of this approach with respect to LLMs as policies multiple perfect and imperfect information games, including newly created ones.

---

Published as a conference paper at ICLR 2026
However, also notice that method struggles learn the rules of Gin rummy, imperfect information game with intricate logic, especially in the very challenging closed deck setting. In future work, we hope to extend method to enable active and online learning of the world model, so the agent can more effectively discover the true hidden causal mechanisms underlying each game (c.f., (Geng al., (025)). In addition, would like to extend the technique to handle open-wordh world games with free- form text and/or visual interfaces, so as to evaluate it on larger sets of novel games,
see (Ying al. 2025).
ACKNOWLEDGMENTS
Jordi Grau-Moya is the 8th author but due to a purely administrative oversight was left off ICLR submission. Jordi is present in the contemporary arxiv submission and has been involved in this project since its inception, contributing significantly to both the foundational ideas and the codebase.
REFERENCES
Stefano V. Albrecht, Filippos Christianos, and Lukas Schäfer. Multi-Agent Reinforcementt Learning: Foundations and Modern Approaches. MIT Press, 2024. URL
http://wwwwwllrrmmmm com
Lucia Cipolina-Kun, Marianna Nezhurina, and Jenia Jitsev. Game reasoning arena: A framework and benchmark for assessing reasoning capabilities of large language models via game play, 2025. URL httts:t///a/iiv..oto http://o/g//b////a/eeese 03368
Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer, and Arjun Yadav. Gamebench: Evaluating strategic reasoning abilities of Ilm agents, 2024. URL
06613
Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In H. Jaap van den Herik, Paolo Ciancarini, and H. H. L.. M. (Jeroen) Donkers (eds.), Computers and Games, pp. 72-83, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg. ISBN 978-3-540-75538-8
Peter I. Cowling, Edward Powley, and Daniel Whitchouse. Information set Monte Carlo tree search. IEEE Transactions Computational Intelligence and Al in Games, 4:120-143, 2012.
Aidan Curtis, Hao Tang, Thiago Veloso, Kevin Ellis, Joshua Tenenbaum, Tomás Lozano-Péres and Leslie Pack Kaelbling. LLM-guided probabilistic program induction for POMDP model estimation. arXiv preprint mrxiv:2505.022/6. 2025.
Nicola Dainese, Matteo Merler, Minttu Alakuijala, and Pekka Marttinen. Generating code world models with large language models guided by monte carlo tree search. In A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Ad- vances in Neural Information Processing Systems, volume 37, pp. (0429-60474) Curran As- sociates, Inc., 2024. URL httpps://proceeding..eet neurips. wwweeeriss.cccaaaappeeeee hiapperffiles/ppprrree .dd]
Constantinos Daskalakis, Ian Gemp, Yanchen Jiang, Renato Paes Leme, Christos Papadimitrion, and Georgios Piliouras. Charting the shapes of stories with game theory. In NeurIPS Creative Al Track, 2024.
Shilong Deng, Yongzhao Wang, and Rahul Savani. From natural language extensive-form game representations. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, pp. 593-601, 2025.
Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel-Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. Gtbench: Uncovering the strategic reasoning capabilities of Ilms via game-theoretic evaluations. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37. pp. 28219-28255 Curran Asso- ciates, Inc., 2024. URL
neur httppp.cccppppppe pdf
10

---

Published as a conference paper at ICLR 2026
Ian Gemp, Roma Patel, Yoram Bachrach, Mare Lanctot, Vibhavari Dasagi, Luke Marris, Georgios Piliouras, Siqi Liu, and Karl Tuyls. Steering language models with game-theoretic solvers. In Agentic Markets Workshop ICML 2024, 2024.
Jiayi Geng, Howard Chen, Dilip Arumugam, and Thomas L Griffiths. Are large language models reliable AI scientists? assessing reverse-engimeering of black-box systems. arXiv [cs.LG], May 2025. URL httphtp//aaxxw..or//aa//////////
17968
Leon Guertler, Bobby Cheng, Simon Yu, Bo Liu, Leshem Choshen, and Cheston Tan. Textarena, 2025. URL /tt.pp/://arxiv.ogg/ab///////// 11442
Lanxiang Hu, Mingjia Huo, Yuxuan Zhang, Haoyang Yu, Eric P. Xing, lon Stoica, Tajana Rosing, Haojian Jin, and Hao Zhang. Imgame-hencie bench: How good are llms at playing games?, 2025a. URL
hwwggg/bb/255000 15146
Sihao Hu, Tiansheng Huang, Gaowen Liu. Ramana Rao Kompella, Fatih Ilhan, Selim Furkan Tekin, Yichang Xu, Zachary Yahn, and Ling Liu. A survey on large language model-basee game agents, 2025b. URL httpt:: /ttpp:://axivve org. abwpg///b//24...... 24040003994
Daniel Kahneman. A perspective on judgement and choice. American Psychologist, 58:697-720, 2003.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6998 2014.
Levente Koesis and Csaba Szepesvári. Bandit based monte-carlo planning. In Proceedings of the 17th European Conference on Machine Learning, ECML''6, pp. 282-293, Berlin, Heidelherg, 2006. Springer-Verlar ISBN 354045375X. doi: 1..1007/11871842 29. URL htttps://doi. org/10. 1877/187784222
H. W. Kuhn. Extensive games and the problem information. Annals of Mathematics Studies, 28:
193-216, 1953.
Mare Lanctot, Edward Lockhart, Jean- Bapt-ste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Saeta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schrittwieser, Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan-Davis. Open- Spiel: A framework for reinforcement learning in games. CoRR, abs/1908.09453, 2019, URL 09453
Yi Liao, Yu Gu, Yuan Sui, Zining Zhu, Yifan Lu, Guohua Tang, Zhongqian Sun, and Wei Yang. Think in games: Learning to reason in games via reinforcement learning with large language models, 2025. URL https://arxx wwwpiivoorgg//// otggabs/ 21365
Agnieszka Mensfelt, Kostas Stathis, and Vince Trencsenyi. Autoformalization of game descriptions using large language models. arXiv preprint M4XX::2409.123000 2024a.
Agnieszka Mensfelt, Kostas Stathis, and Vince Trencsenyi. Autoformalizing and simulating game- theoretic scenarios using llm-augmented agents. arXiv preprint arXiv:2412.08805, 2024b.
Prakamya Mishra, Jiang Liu, Jialian Wu, Xiaodong Yu, Zicheng Liu, and Emad Barsoum. Tit-bench: A benchmark for evaluating reasoning ability with simple and novel tic-tac-toe-style games, 2025.
URL httpt:tt////xi...ttttt wttp:/iwww.r/////////e/ 10209
Kevin Murphy,
Reinforcement learning: An overview, 2025. URL https://arxiv.org/ab//
2412. 2526555
Martin J. Osborne and Ariel Rubinstein. A Course in Game Theory. MIT Press, 1994.
11

---

Published as a conference paper at ICLR 2026
Anian Ruoss, Grégoire Delétang, Sourabh Medapati, Jordi Grau-Moya, Li Kevin Wenliang, Elliot Catt, John Reid, Cannada A. Lewis, Joel Veness, and Tim Genewein. Amor- tized planning with large-scale transformers: A case study on chess. In A. Glober- son, I.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Ad- vances in Neural Information Processing Systems, volume 37, pp. 65765-65790 Curran As- sociates, Inc., 2024. URL htttts:ttttt//rees //ppoe/ed/s/sseeesgg/s/s neerrrrerrreeenneeeeiiesresssese neurips. cc/paper Files/paper//20/0 pdf
Rahul Savani and Theodore L.. Turocy. Gambit: The package for computation game theory, version 16.2.0 edition, 2024. https://wwwt https://ww.ggambitt www.gammbbtprojjct.orr
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. arXiv preprint arXiv:
John Schultz, Jakub Adamek, Matej Jusup, Marc Lanctot, Michael Kaisers, Sarah Perrin, Daniel Hennes, Jeremy Shar, Cannada Lewis, Anian Ruoss, Tom Zahavy, Petar Veličković, Laurel Prince, Satinder Singh, Eric Malmi, and Nenad Tomašev. Mastering board games by external and internal planning with language models. In Proceedings of the Forty-Secondl International Conference on Machine Learning (ICML), 2025, URL wwt.:httts://aattee http://///akii.oog///////1111 2412 12119
Y. Shoham and K. Leyton-Brown, Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations. Cambridge University Press, 2009.
David Silver and Joel Veness. Monte-carlo planning in large pomdps. Advances in neural informa- fion processing systems, 23, 2010.
Haoran Sun, Yusen Wu, Peng Wang, Wei Chen, Yukun Cheng, Xiaotie Deng, and Xu Chu. Game theory meets large language models: A systematic survey with taxonomy and new frontiers, 2025.
URL http:://aaxxxii..ttt http://orgg/bb///////oe/s http:/2552.099900 09053
Hao Tang. Keya Hu, Jin Peng Zhou, Si Cheng Zhong, Wei-Long Zheng, Xujie Si, and Kevin Ellis. Code repair with LLMs gives exploration-cxplottaiinn tradcoff. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, November 2024a. URL
https::/openreview.neettt/ wttp://.eee//ff/////ee iddoo633664)
Hao Tang, Darren Key, and Kevin Ellis. Worldcoder, a model-based I1m agent: Building world models by writing code and interacting with the environment, 2024b. URL https://arxiv.
org/abs/2402.. 12275
Vivek Verma, David Huang, William Chen, Dan Klein, and Nicholas Tomlin. Measuring general intelligence with generated games, 2025. http:/ttss///arii.orrr/////// www.orgg//bbb.oeeeeeeeeeee 2500.. 07215
Zelai Xu, Wanjun Gu, Chao Yu, Yi Wu, and Yu Wang. Learning strategic language agents in the werewolf game with iterative latent space policy optimization. In Forty-second International Conference on Machine Learning, 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models., 2022. URL httpss///a/a org/ abs/2210. 03629
Lance Ying, Katherine Collins, Prafull Sharma, Cedric Colas, Kaiya Ivy Zhao, Adrian Weller, Zenna Tavares, Phillip Isola. Samuel J Gershman, Jacob D Andreas, Thomas L Griffiths, Francois Chollet, Kelsey R Allen, and Joshua Tenenbaum. Assessing adaptive world models in machines with novel games. arXiv [cs.Al], July 2025. URL http:t://aaaxii.or///aaaaa
12821
12

---

Published as a conference paper at ICLR 2026
A INFORMATION ON THE GAMES
summary of the games that we use in our experiments is given in Table 3
Table 3: Details of the games that we use. The columns have the following meaning: OOD: whether the game novel (no source code on the internet); Observability: Full means perfect information game, partial means imperfect information game; Payoff: W/L/D means Win/Lose/Draww General means general sum; Actions: number of possible actions; Obs. dim.: dimensionality of the obser- vation tensor IS dim.: dimensionality of the information set (i.e., game's s hidden state).
Name
OOD
Observability
Payoff
# Actions
Obs. dim.
IS dim.
Backgammon
X
Full
W/L/D
1352
200
Connect four
Full
W/L/D
7
126
Tic-tac-toe
Full
W/L/D
9
27
Gen. tic-tac-toe
Full
W/L/D
36
108
Gen. chess
Full
W/L/D
5555
250
Bargaining
Partial
General
121
93
309
Ledue poker
Partial
Zero-sum
3
16
30
Gin rummy
Partial
Zero-sum
241
644
655
Quadranto
Partial
W/L/D
5
7
Hand of war
Partial
W/L/D
16
27
73
B SYNTHESIS RESULTS USING GEMMA27B
Table 4: Perfect information games, refinement via conversation using Gemma27B.
transition accuracy
Game
OOD
# LLM calls
train
test
Backgammon
X
0.17462
0.16742
500.0
Connect four
X
0.07826
0.10005
500.0
Tic-tac-toe
X
0.98049
0.98441
193.0
Gen. tic-tac-toe
0.94833
0.90307
500.0
Gen. chess
0.49281
0.51603
500.0
Table Imperfect information games, hidden history inference, closed deck using Gemma27B..
inference accuracy
Game
OOD
LLM calls
train
test
Bargaining
X
0.09412
0,14296
500.0
Ledue poker
X
0.13333
0.11799
500.0
Gin rummy
X
0.00797
0.01393
500.0
Quadranto
0.09412
0.10343
500.0
Hand of war
0.07812
0.07949
500.0
13

---

Published as a conference paper at ICLR 2026
C INFORMATION SET MONTE CARLO TREE SEARCH
Definition (Extensive-Fon Forrn Game with Imperfect Infomation (Osborne & Rubinstein 1994)). A finite extensive-form game with imperfect information is a tuple (N A. Z, T, Ser Uz), where:
..) = (1,2,3, .. u {c} is a finite set of n players and a special player called chance.
A is a finite set of actions.
H. is a set of histories (sequences of acctioosss C H. the set of terminal histories (marking the full play game from start finish). Every game starts the empty history hg V. At each non-terminal history h E H Z, let A(h) C A denote the set of legal actions available at h.
player function T: H- N that identifies which player is act at every nonterminal history.
For each player i E N. partition I, of of - = for all I It and h h E T: A(h) A(N') and -())= r(²)= ((h)==+ = i. I, called player i's information partition and each E I, is called an information state.
with the property that
A fuucction Se that assigns probability distribution over actions at every h E H - 2 where T Th) c. Here, Schh) E ((A) the chance outcome distribution at chance event
A utility function u N + [U_, .U+] C R, where U- and U+ are upper and lower bounds the utility and (2) is the utility to player i at terminal history 2.
Recall that a history encodes the sequence of actions taken by all players, including chance. But in imperfect information game, not all aspects of the history are observable. For instance, in game of poker. contains information about the cards held by all players (as chosen by the dealers actions), but some of this information is private and hence not known by some players. After an
action is executed and added to the history (h+-1, az) = h4, each player i E A perceives individual observations of (h,). The state (from the perspective of an agent i) is then function of of e.g., just the last observation.
To choose actions in an IIG, we can use the Information Set MCTS method of (Cowling et al. 2012). which we now describe. First, recall that in classical MCTS, there is root node corresponding the current state of the game which all simulations start from, and non-root nodes which correspond to states that occur after the root state. At each node, statistics such as average values state-action values, (s,, a). and simulation counts are maintained. The main differences in ISMCTS are: (i) the simulations start at a distribu- tion of possible ground truth states and (ii) statistics maintained and aggregated across information states with respect to the current player.
Update uolu est estimates storee
ha
Figure 3 contains an example with a simplified poker game with a deck of three cards (Jack, Queen, King). In this example, the cur- rent player has received the King as a private card and no actions have yet been taken, there are only two ground truth states: the opponent could have either the Queen or the Jack. An iteration first samples the Queen and continues with this ground truth state hng sampling actions, and generating histories f1, f2, hg, and so on then added
Figure 5: ISMCTS. A search tree is built over possible ground truth histories (e.g. h1, h2, ...). Because the player cannot distinguish be- tween certain histories, statis- ties are aggregated at the level of information sets (dotted boxes), which group all his- tories that appear identical to the player.
until the first node not in the tree is encountered. to the tree, and a random rollout policy takes over until terminal state. The dotted boxes are the analogs of nodes stored in a tree (or lookup table) and correspond to information states. Return es- timates (i.e., Q-value statistics) and visit counts are maintained these nodes in classical MCTS [Coulom] 2007} (aggregated over different samplings of ground truth states), and UCB is used to se- lect actions in the standard way (Koesis & Szepesvári 2006).
14

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTAL RESULTS
In the interest of space, some additional experimental results are included this section.
D.1 SYNTHESIS
D.1.1 ACCURACY OF LEARNED TRANSITION AND INFERENCE FUNCTIONS
Comparing Table 6 and 7 it is apparent that though both options work reasonably well, tree search has the edge, both in terms of accuracy (higher) and number of L.M calls (lower).
Table 6: Perfect information games, refinement via tree search.
transition accuracy
Game
OOD
# LLM calls
train
online test
Backgammon
X
1.00000
0.99932
1.00000
16.8
Connect four
X
1.00000
1.00000
1.00000
2.0
Tic-tac-toe
X
1,00000
1.00000
1.00000
2.0
Gen. tic-tac-toe
1,00000
1.00000
1,00000
2.4
Gen. chess
1.00000
1.00000
1.00000
5.2
Table 7: Perfect information games, refinement via conversation.
transition accuracy
Game
OOD
LLM calls
train
test
online
Backgammon
X
1.00000
0.99944
1.00000
13.2
Connect four
X
1,00000
1.00000
1.00000
3.2
Tic-tac-toe
X
1.00000
1.00000
1.00000
2.0
Gen. tic-tac-toe
1.00000
1.00000
1.00000
2.4
Gen. chess
1.00000
1.00000
1.00000
4.2
Table 8: Imperfect info. games, refinement via tree search, hidden state inference.
transition accuracy
inference accuracy
Game
OOD
# LLM calls
train
test
online
train
test
online
Bargaining
X
1.0000
0.9482
0.8712
1.0000
1.0000
1.0000
32.8
Ledue poker
1.0000
0.9854
0.9942
1.0000
1.0000
1,0000
4.2
Gin rummy
X
0.8943
0.8243
0.8909
1.0000
0.9513
0.9738
500.0
Quadranto
1.0000
1.0000
0.9991
1.0000
0.9911
0.9876
7.4
Hand of war
1.0000
0.9782
0.9806
1.0000
1.0000
1.0000
28.0
15

---

Published as a conference paper at ICLR 2026
D.1.2 ACCURACY OF LEARNED TRANSITION AND INFERENCE FUNCTIONS VS NUMBER OF LLM CALLS
transition acc. (train)
transition acc. (test)
1.0-
Dataset Backgammon Connect tour
0.6-
tic-tac-toe Tic: tacitattto
0.2
6
1012 1416181 Number cals
16 20
Number al..M
Figure 6: Evolution of the transition accuracy of the best generated CWM with the number of LLM calls for perfect games (with CWM refinement via tree search).
(train)
transition acc. (test)
acc. (train)
inference (test)
0.8-
Dataset Bargaining
Gin
3.44
Luduc poler
0.2
Quadranto
hamberal Manberd LLMids
Narba
Munder
Figure 7: Evolution of the transition and inference accuracy with the number of LLM calls for imperfect games with refinement via tree search and hidden state inference.
inference acc. (train)
inference acc. (test)
1.0
0.8
Dataset
Accuracy
0.6
Bargaining
rummy
0.4
Hand of
Leduc poker
Quadranto
0.0
20
40
100 120 140 160 180 200 Number of LLM calls
60
80
20 40 60
100 120 140 160 180 200
Number of LLM calls
Figure 8: Evolution of the inference accuracy with the number of LLM calls for imperfect games with refinement tree search with closed deck.
16

---

Published as a conference paper at ICLR 2026
D.1.3 TREE SEARCH SETTINGS
We the following settings for treesearch throughout our experiments.
heuristic_weighte 0: Weight on the heuristic value (higher means more exploitation).
The heuristic weight C' adjusts the parameters a and / of the Beta prior on each arm Tang et al. (2024b). In particular, we set a + x h and 3 + (1 C) x h, where the heuristic value h is the average pass rate of the unit tests.
num_retries=500: Number of retries for tree search.
num_tests_on_init-ss Number of tests of each type to include on the first synthesis.
num_tests_on__eroorit. Number failed tests of each type to include during code re- finement.
itteerristicvvlle_oooiiiieee 01: Minimum heuristic value to consider node for expansion on initialization.
wtn._herristic_alle_ggiiii 01: Minimum heuristic value gain consider node for expansion.
17

---

Published as a conference paper at ICLR 2026
D.2 DETAILED PER-GAME ARENA RESULTS
For games in which the outcomes are win, lose or draw, we show the frequency of these outcomes in different columns, for each agent. For games with arbitrary payoft (Bargaining. LeDuc-Poker, Gin Rummy), we show the payoff to each player in 2 different columns. We consider the case when our agent acts as Player 0 or Player I, and show these in different rows, account for first-mover advantage.
For imperfect information games, we show results for hidden history inference (open deck learning), hidden state inference (open deck learning), and hidden history inference (closed deck learning).
For games in which the outcomes are win, lose draw, we also report (in small font) the number of games with outcome that were forfeited vs the total number of games with that outcome. (A forfeit means the agent has either thrown an exception or tried to execute an illegal action, since our game arena API does not allow the agent to see which actions are legal at given point in the game.)
D.2.1 PERFECT INFORMATION GAMES
Table 9:
Win rates using CWM refinement via tree search against multiple opponents. For each game, results in the first (second) correspond to agent going first (second).
Gemini 2.5 Pro
GT-MCTS
Random
Game
Win (exteive)
Loss (oxteros)
Draw m
Win
Loss
Draw
Win
Draw
1.00 (1O0/D0))
0.00 (m))
0.00
0.08
0.92
0.00
0.98
0.02
0.00
Backgammon
1.00 (100110))
0.00
0.00
0.07
0.93
0.00
0.98
0.02
0.00
1.00 (2000)
0.00
0.00
0.69
0.31
0.00
1.00
0.00
0.00
Connect four
1.00 (1/00))
0.00
0.00
0.28
0.72
0.00
1.00
0.00
0.00
0.05 (05)
0.00 gum)
0.95 (99)
0.00
0.00
1.00
0.97
0.00
0.03
Tic-tac-toe
0.00 (040)
0.00 (00))
1.00 (10m)
0.00
0.00
1.00
0.75
0.00
0.25
0.89 (D00))
0.10 ao1o)
0.01 m
0.88
0.12
0.00
1.00
0.00
0.00
Gen. tic-tac-toe
0.93 (093))
0.07 (07)
0.00 (
0.37
0.63
0.00
1.00
0.00
0.00
1.00 (12/10m)
0.00 (00)
0.00
0.18
0.43
0.39
1.00
0.00
0.00
Gen. chess
1.00 (97710m)
0.00 (DX))
0.00
0.49
0.17
0.34
1.00
0.00
0.00
18

---

Published as a conference paper at ICLR 2026
D.2.2 HIDDEN HISTORY INFERENCE
Table 10: Payoffs using CWM refinement via search and hidden history inference against mul- tiple opponents. For each game, results in the first (second) row correspond to our agent going first (second).
Gemini 2.5 Pro
GT-ISMCTS
Random
Game
P
Us
Them
Us
Them
Us
Them
8.90
3.31
8.01
5.26
8.21
2.41
Bargaining
8.80
4.73
7.51
5.44
8.12
2.81
-0.03
0.03
-0.65
0.65
0.86
-0.66
Ledue poker
1.55
-1.55
0.24
-0.24
1.09
-1.09
120.54
-120.54
-15.62
115.62
-4.92
4.92
Gin rummy
123.00
-123.00
-115.62
115.62
-15.99
15.99
Table 11: Win rates using CWM refinement via tree search and hidden history inference against multiple opponents. For each game, results in the first (second) row correspond to our agent going first (second).
Gemini 2.5 Pro
GT-ISMCTS
Random
Game
Win (oofeeth)
Loss (feffech)
Draw
Win
Loss
Draw
Win
Loss
Draw
0.91 (1691)
0.08 (03)
0.01 (
0.06
0.02
0.92
0.27
0.03
0.70
Quadranto
0.75 (o755)
0.18 (018)
0.07
0.11
0.08
0.81
0.31
0.06
0.63
0.35
0.56 (11/56)
0.09
0.20
0.72
0.08
0.33
0.57
0.10
Hand of war
0.33 (053)
0.62 (39/52)
0.05 (5)
0.31
0.61
0.08
0.33
0.62
0.05
19

---

Published as a conference paper at ICLR 2026
D.2.3 HIDDEN STATE INFERENCE
Tabble 12: Payoffs using CWM refinement via search and hidden state inference against multiple opponents. For each game, results the first (second) row correspond to our agent going first (second).
Gemini 2.5 Pro
GT-ISMCTS
Random
Game
P
Us
Them
Us
Them
Us
Them
8.48
4.32
7.46
4.17
7.70
2.72
Bargaining
7.98
6.42
7.25
6.04
7.78
3.47
1.75
-1.75
0.16
-0.16
1.19
-1.19
Ledue poker
0.37
-0.37
0.41
-0.41
1.12
-1.12
66.42
-66.42
-114.39
114.39
-28.29
28.29
Gin rummy
121.77
-121.77
-121.77
121.77
-4.92
4.92
Table 13:
Win rates using CWM refinement via tree search and hidden state inference against mul- tiple opponents. For each game, results in the first (second) row correspond to our agent going first (second).
Gemini 2.5 Pro
GT-ISMCTS
Random
Game
Win (fofeech))
Loss (feefech)
Draw
Win
Loss
Draw
Win
Loss
Draw
0.58 (16258)
0.40 (040)
0.02 (3)
0.13
0.02
0.85
0.19
0.04
0.77
Quadranto
0.37 (057)
0.54 (054)
0.09
0.14
0.01
0.85
0.26
0.07
0.67
0.41 (16441)
0.49
0.10 (10)
0.25
0.61
0.14
0.59
0.28
0.13
Hand of war
0.44 (0144)
0.40 (0o4))
0.16 (36)
0.42
0.41
0.17
0.63
0.24
0.13
20

---

Published as a conference paper at ICLR 2026
D.2.4 HIDDEN HISTORY INFERENCE WITH CLOSED DECK LEARNING
Table 14: Payoffs using CWM refinement via search with closed deck against multiple oppo- nents. For each game, results in the first (second) row correspond to our agent going first (second).
Gemini 2.5 Pro
GT-ISMCTS
Random
Game
P
Us
Them
Us
Them
Us
Them
1I
7.03
5.91
7.01
5,98
7.37
3.76
Bargaining
7.07
7.02
6.83
6.01
7.39
3.68
0.57
-0.57
-0.56
0.56
0.86
-0.86
Ledue poker
0.49
-0.49
-0.21
0.21
1.71
-1.71
29.52
-29.52
-11..39
114.39
-19.331
119.31
Gin rummy
-63.96
63.96
-11.331
119.31
-121.77
121.77
Table 15:
Win rates using CWM refinement via tree search with closed deck against multiple oppo- nents. For each game, results in the first (second) row correspond our agent going first (second).
Gemini 2.5 Pro
GT-ISMCTS
Random
Game
Win (Gunfiid))
Loss (fedfanh)
Draw
Win
Loss
Draw
Win
Loss
Draw
0.69 (0W9)
0.26 (016)
0.05 (5)
0.08
0.05
0.87
0.23
0.05
0.72
Quadranto
0.71 (271)
0.22 (022)
0.07 m
0.13
0.09
0.78
0.27
0.05
0.68
0.41
0.42
0.17
0.31
0.57
0.12
0.61
0.24
0.15
Hand of war
0.54 (0544)
0.25 (W25)
0.21 (n))
0.47
0.40
0.13
0.62
0.26
0.12
21

---

Published as a conference paper at ICLR 2026
D.3 FORFEIT RATES FOR NON-TERNARY-OUTER OUTCOME GAMES
Forfeit rates for non-temaary-outcome games using CWM refinement via tree search and hidden history inference against multiple opponents. This is the rate which each agent forfeits the game by failing to execute a legal action. For each game, results in the first (second) row correspond our agent going first (second).
Table 16:
Gemini 2.5 Pro
GT-ISMCTS
Random
Game
Us
Them
Us
Them
Us
Them
0.00
0.00
0.00
0.00
0.00
0.00
Bargaining
0.00
0.01
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Leduc poker
0.00
0.00
0.00
0.00
0.00
0.00
0.01
0.99
0.94
0.00
0.04
0.00
Gin rummy
0.00
1.00
0.94
0.00
0.13
0.00
Table 17: Forfeit rates for non-ternayy-outooee games using CWM refinement via tree search and hidden state inference against multiple opponents. This is the rate at which each agent forfeits the game by failing execute legal action. For each game, results in the first (second) row correspond to our agent going first (second).
Gemini 2.5 Pro
GT-ISMCTS
Random
Game
P
Us
Them
Us
Them
Us
Them
0.00
0.00
0.00
0.00
0.00
0.00
Bargaining
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.16
0.00
0.00
0.00
0.00
Ledue poker
0.00
0.00
0.00
0.00
0.00
0.00
0.23
0.77
0.93
0.00
0.23
0.00
Gin rummy
0.00
0.99
0.99
0.00
0.04
0.00
Table 18: Forfeit rates for non-temary-outtcome games using CWM refinement via tree search with closed deck against multiple opponents. This is the rate at which each agent forfeits the game by failing to execute legal action. For each game, results in the first (second) row correspond to agent going first (second).
Gemini 2.5 Pro
GT-ISMCTS
Random
Game
P
Us
Them
Us
Them
Us
Them
0.00
0.00
0.00
0.00
0.00
0.00
Bargaining
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Ledue poker
0.00
0.00
0.00
0.00
0.00
0.00
0.38
0.62
0.93
0.00
0.97
0.00
Gin rummy
0.76
0.24
0.97
0.00
0.99
0.00
22

---

Published as a conference paper at ICLR 2026
D.4 VALUE FUNCTION ABLATIONS
As explained in the main text, the purpose of value functions is to speed up (IS)MCTS by providing better value initialization for new leaf nodes. This can also result in higher quality selections for a fixed budget. Synthetic value functions are generated by the LLM in one-shot, and its usefulness assessed a tournament ran on top of the synthesized CWM. Agents using different value func- tions (or potentially no value function) compete against each other the synthesized CWM to evaluate performance.
The use of value function only delivered improvements in the case of Gen. tic-tac-toe and Bargaining, so our agent only used value functions when playing those games. Note that the choice to use value functions not can be assessed before actual online game play, by having the agent play locally (with and without using a value function) on its own synthetic CWM as proxy, and assessing which option is most beneficial.
Fig. 2 shows the ablation corresponding to not using a value function in Gen. tic-tac-toe and Bargaining.
Gen. tic-tac-toe STARTTS
with Value Function
25hr
+5
GT-NCTS
15
Win
Win by Forfeit
Draw
Loss
O CWMMMTS 25 Player 0
MCTS Player
Bargaining history inf.
with Value Function
Cemeri2.57e
Ransim
2.50a
Random
Player Retur
Osponent
CWM-ISMCTS Player
Player
Bargaining closed deck
with Value Function
R. GT. GNETS
Randon
Randon
0
0
Player Rebur
Opponien Return
O CWM.ISMCTS as Player
CWM-ISMCTS as Player
Figure 9: Ablation for Gen. tic-tac-toe and Bargaining. Effect of using synthesized value functions (right column) vs not (left column) to improve planning in CWMs.
23

---

Published as a conference paper at ICLR 2026
E PLANNING WITH PPO INSTEAD OF (IS)MCTS
E. TRAINING PPO AGENT ON TOP OF A CWM
The CWM agent discussed in the main paper relies on (IS)MCTS to take actions within its learned CWM. While effective, this online planning process can be slow. We investigate an alternative ap- proach: amortizing the planning computation into reactive policy, trained with the PPO algorithm Schulman et al. 2017
We entirely learn this PPO policy within the learned CWM environment. For each game, we train PPO-CWM M agent (acting either as Player Player 1) to maximize its rewards against an opponent that uniformly picks legal action.
Mapping JSON observations to 1D tensors: The CWM represents observations in the JSON format provided by OpenSpiel, whereas the actor-critic networks we use (which are based on MLPs and RNNs) requires fixed-size ID arrays as input [ JSON into flat tensor representation. We generate this mapping programmatically by prompting a LLM shown below, providing the CWM training sequences examples.
Consequently,
need procedure map each
are expert reinforcement learning researcher and Python programmer.
Your task is to implement the following two functions which form a bijective pair:
def observatiio_ -> np. ndarray: # 1D
def censor_to_observationnteet -> ndarray: # 1D
An example input dataset is as follows: (example)
First reason about the problem and possible corner cases. Finally output only the resulting two functions without any placeholders.
Architecture. Our PPO agent uses an actor-critic architecture. For perfect information games, the actor and critic networks share common feature extractor consisting of two 256-unit fully- connected layers with tanh activations. The actor head is final linear layer that outputs logits for each action, which are then masked to ensure only legal moves are considered. The critic head is separate linear layer that outputs single scalar value.
For imperfect information games, we augment this architecture with a recurrent neural network to process historical information. An input observation is first passed through 256-unit linear layer (with tanh activation). The result fed into an RNN along with the previous hidden state ht- to produce an output vector. This output is concatenated with the original input 2a and passed through a final 256-unit hidden layer (with tanh activation) before being fed to the actor and critic heads as described above.
PPO training. The PPO-CWM agent is trained for total of 10M agent steps inside the CWM, using the hyperparameters in Table 19 From the two-player trajectories collected, we extract the single-agent sequence of observations, actions, and rewards corresponding to the PPO-CWM agent. This filtered data is used to compute the advantages and the final PPO loss objective.
For each game, and each player, we train 5 PPO-CWM agents with different seeds, and select the one with the highest win rate against the random opponent for final evaluation. This agent is then benchmarked in the Arena, as described in Sec. We include matches against our CWM MCTS agent to compare both approaches for leveraging the CWM.
We could use transformers, which can handle ISON strings, for the actor and critic, but such models would he much slower train.
24

---

Published as a conference paper at ICLR 2026
Table 19: PPO hyperpameters.
Module
Hyperparameter
Value
Environment
Number of environments
50
Rollout horizon in environment
100
Advantage
X
0.99
0.95
Loss
clipping
0.2
Value loss coefficient
0.5
Entropy loss coefficient start
Entropy loss coefficient end
0.01
Entropy loss coefficient schedule
Linear
Learning
Optimizer
Adam (Kingma & Ba 2014)
Learning rate
0.0003
Max. gradient norm
0.5
Learning rate annealing
False
Number of minibatches (MFRL)
10
Number of epochs (MFRI.)
E.2 RESULTS
Arena results are presented in Tables 20 to 24 due to the poor performance of the CWM on that game.
Note that PPO-CWM was not trained on Gin Rummy
PPO-CWM vs. Random. PPO-CWM outperforms the random agent for all the games.
PPO-CWM vs. Gemini 2.5 Pro. Our PPO-CWM agent outperforms or matches Gemini in all the games. For perfect information games, PPO-CWM wins in Backgammon, Generalized Chess and Tic-Tac-Toe;; and exhibits mixed results (winning one player and losing the other) in Connect Four and Generalized Tic-Tac- Toe. For imperfect information games, for both open deck and closed deck, PPO-CWM wins in Bargaining and Quadranto, and ties in Hand of War and Ledue Poker.
PPO-CWM vs. CWM MCTS. For perfect information games, where the learned CWM a near- perfect replica of the environment, PPO-CWM is outperformed by CWM-MCTS agent, The only exception Generalized Tic-Tac-Too Toe when PPO-CWM acts as Player 0. For imperfect infor- mation games, PPO-CWM wins in two games (Hand of War and Bargaining) and loses in the other two games (Ledue poker and Quadranto).
E.2.1 GAMES WITH PERFECT INFORMATION
Table 20: PPO-CWM win rates using CWM refinement via tree search against multiple opponents. For each game, results in the first (second) row correspond to our agent going first (second).
CWM MCTS
Gemini 2.5 Pro
GT-MCTS
Random
Draw
Win
Win
0.01 an:
0.99 amn
0.00 m
1.00 caooen
0.00 pm
0.00 m
0.02
0.98
0.00
0.92
0.08
0.00
Backgammon
0.03
0.97 dso
0.00
1.000 pavras
0.00
0.00
0.01
0.99
0.00
0.94
0.06
0.00
0.00 ywr
1,00 a00i
0.00
0.92
0.08
0.00
0.00
1.00
0.00
1.00
0.00
0.00
ar)
0.00
0.00
0.00
1.00
0.00
0.000
1.00
0.00
0.00
1.00
1.00
0.00
0.00
Tic-tac-tre
0.00 yarr
1,00 doves
0.00 m
0.87 avers
0.12 an2
0.01 m
0.00
1.00
0.00
0.1
0.01
0.08
Gen. Lip-tac-toc
0.04 vae
0.96 ..9
0.55 ass
0.00 m
0.00
0.91 amo
0.38
0.62
om
0.00 m
0.00
0.05
0.54
0.95
0.46
0.00
0.00
0.99
1.00
0.00
0.01
0.00
0.00
8
0.00 pi
1.00 gones
0.00 m
0.94 osem
0.05
0.00
0.00
1.00
0.00
1.00
0.00
0.00
0.08
1.00
0.0
25

---

Published as a conference paper at ICLR 2026
E.2.2 HIDDEN HISTORY INFERENCE
Table 21: PPO-CWM win rates using CWM refinement via tree search and hidden history inference against multiple opponents. For each game, results in the first (second) row correspond to our agent going first (second).
CWM MCTS
Gemini 2.5 Pro
GT-ISMCTS
Random
Game
Win susaas
Lass devealy
Draw
Win deviak)
Loss deeiaao
Draw
Win
Loss
Drw
Win
Loss
Draw
0.01
0.64
0.35
0.61
0.19
0.20
0.00
0.62
0.38
0.51
0.18
0.31
Qundranto
0.02 om
0.30 pon
0.68 on
0.64 use
0.25 aas
0.11
0.01
0.46
0.53
0.43
0.13
0.44
0.75 aus
0.1 are
0.
0.30 asn
0.47 awe
0.23
0.32
0.58
0.10
0.66
0.19
0.15
Hand
0.64
0.29
0.07
0.54
0.29
0.17
0.58
0.56
0.06
0.66
0.19
0.15
Table 22: PPO-CWM payoft's using CWM refinement via tree search and hidden history inference against multiple opponents. For each game, results in the first (second) row correspond to our agent going first (second).
CWM MCTS
Gemini 2.5 Pro
GT-ISMCTS
Random
Game
P
Us
Them
Us
Them
Us
Them
Us
Them
EB
7.92
4.53
8.69
4.22
7.92
4.31
7.77
2.72
Bargaining
7.57
4.53
8.37
5.11
8.04
4.88
7.87
3.24
-0.60
0.60
-0.82
0.82
-0.54
0.54
1.25
-1.25
Ledue poker
-1.58
1.58
0.95
-0.95
-2.26
2.26
1.99
-1.99
E.2.3
HIDDEN STATE INFERENCE
Table 23: PPO-CWM win rates using CWM refinement via tree search and hidden state inference against multiple opponents. For each game, results in the first (second) row correspond agent going first (second).
CWM MCTS
Gemini 2.5 Pro
GT-ISMCTS
Random
Game
Win Ieeeans
Loss eveeaii
Draw
Win oveeani
Loss decesto
Draw
Win
Loss
Loss
Draw
0.01 aru
0.54 asn
0.45 ms
0.75 am
0.18 ann
0.07
0.03
0.52
0.45
0.59
0.11
0.30
Quadranto
0.18
0.48
0.34
0.74
0.25
0.01
0.00
0.72
0.28
0.57
0.14
0.29
0.40
0.47
0.13
0.35
0.55
0.10
0.27
0.63
0.10
0.60
0.26
0.14
Hand war
0.53 usts
0.35 mmn
0.12
0.57 ass
0.24 ose
0.19 cm
0.50
0.44
0.06
0.53
0.30
0.17
Table 24: PPO-CWM payoffs using CWM refinement via tree search and hidden state inference against multiple opponents. For each game, results in the first (second) row correspond our agent going first (second).
CWM MCTS
Gemini 2.5
Pro
GT-ISMCTS
Random
Game
P
Us
Them
Us
Them
Us
Them
Us
Them
8.45
3.93
8.32
4.76
8.53
4.83
8.00
2.64
Bargaining
6.44
3,80
7.98
4.66
7.82
4.60
8.21
2.67
-1.93
1.93
0.31
-0.31
-1.56
1.56
1.58
-1.58
Ledue poker
-0.51
0.51
-0.57
0.57
-1.29
1.29
1.99
-1.99
26

---

Published as a conference paper at ICLR 2026
E.2.4 HIDDEN HISTORY INFERENCE WITH CLOSED DECK LEARNING
Table 25: PPO-CWM win rates using CWM refinement via tree search with closed deck against multiple opponents. For each game, results in the first (second) row correspond to our agent going first (second).
CWM MCTS
Gemini 2.5 Pro
GT-ISMCTS
Random
Game
Win susaas
Lass deieaa
Draw
Win deviak)
Loss deciaao
Draw
Win
Loss
Drw
Win
Loss
Draw
0.01
0.59
0.40
0.74
0.24
0.02
0.01
0.60
0.39
0.51
0.11
0.38
Qundranto
0.02 om
0.47 pwty
0.51 ov
0.70 am
0.27 aay
0.03
0.00
0.37
0.63
0.58
0.16
0.26
0.50 nee
0.41 ose
0.09 m
0.32 am
0.49 own
0.19
0.27
0.64
0.09
0,58
0.26
0.16
Hand
0.51
0.38
0.11
0.47
0.30
0.23
0.46
0.45
0.09
0.70
0,20
0.10
Table 26: PPO-CWM payofts using CWM refinement via tree search with closed deck against multiple opponents. For each game, results in the first (second) row correspond to our agent going first (second).
CWM MCTS
Gemini 2.5
Pro
GT-ISMCTS
Random
Game
P
Us
Them
Us
Them
Us
Them
Us
Them
EB
5.96
5.90
7.15
4.96
6.35
5.21
7.07
3.21
Bargaining
6.70
5.49
7.73
5.10
7.40
5.34
8.07
2.79
-1.69
1.69
-0.63
0.63
-1.05
1.05
1.72
-1.72
Ledue poker
-0.08
0.08
1.61
-1.61
-1.78
1.78
2.25
-2.25
27

---

Published as a conference paper at ICLR 2026
F AUTOMATIC REJECTION OF BAD CWM SAMPLES
The CWM refinement process can occasionally produce a low-quality CWM. This is rarely the case for perfect information games, where more information available for refinement and unit tests are more strict, but we have observed this happening in the case of imperfect information games. To reduce this effect, in the case of imperfect information games, we sample 5 CWMs, create CWM- ISMCTS agent from each one, and make those agents compete against each other. Agents are then ranked according to the average payoff obtained those competitions. Agents that are worse than the best scoring agent by more than 10% of the observed utility range are rejected.
Since we do not have access the ground truth game for these competitions, the agents use the CWM of one of them as a stand-in for the actual game. We call the CWM used to play the game the host. This means that we have 2 possible hosts x 5 agents acting as Player 0 x 5 agents acting as Player 1. This results in a total of 50 possible matches. Since the outcome of match is stochastic, we repeat each match 50 times. Execution failures or the execution of illegal actions during these games result in both players losing the game.
G SKETCH OF INFORMATION FLOW OF EACH AGENT
Here provide sketch of the information flow for each the agents. Of course, many details omitted, and the prompts are highly simplified, see Appendix H for the actual prompts.
def _lm_agent gggentggeneraton (LLM, rules, traj): prompt Yfu are playing game with these rules: (rules). An" f"Example trajectories: (traj}.An)
def Ailcyaactionoos_hiitrryy return LLM(prompt f"AAtion-observatioo history: (action_ode obs obs__sstttrr......o "Pick the next best action.")
return policy
def htapntteeereet_iinoeeeeeeeeeteee iLduce_cwmLLLM, rules, traj) not GT else ground_truth_M Induce_.allu__fn(LLN rules, traj, M)
rules, traj, GT=False):
def policylactionnoos. return return policy
M, V)
def wtw.aaat__imeeeettinffoooooooooo (LLM, rules, traj, GT=False):
(M, I) poddue__cww__oomppLLL rules, traj) if not else truthh_ V anduce_value_fn(LL) rules, traj, M)
def snlicy(attin__os_hhhttrrrr return ISMCTS(action_obs hhittoorr history, M, V, I)
return policy
def induce_cwn__ero__hhe corrossshttLLi rules, traj):
prompt = Yf"You are playing game with these rules: (rules). An" f"Generate python code that matches this API: (fn_ssinaauree)nn f"The code should pass these unit tests: (make_tests(traj)))
return LLM(prompt)
28

---

Published as a conference paper at ICLR 2026
H SYSTEM AND AGENT PROMPTS
H.
TREE SEARCH
Our tree search prompt is:
You expert python programmer who is building the game of (game_name). Here is a description of the game: {game_desc)
The goal is implement python function with the following signature. START FUNCTION SIGNATURE
sounctionsiinaatiee i END FUNCTION SIGNATURE
original implementation as follow. Please try to refine the original code. # START CODE BLOCK
{orig_code)
# END CODE BLOCK
Your code should satisfy the following unit tests. Your code should fix the TODO errors in the comments of the unit tests, if any. START UNIT TESTS
(test_code)
END UNIT TESTS
Do not repeat the unit tests, only return functions.
Do not leave placeholders.
Do not repeat the function signature. Do not copy the unit tests.
Only produce code that is compact.
Do write comments explaining what the code does. Do use helper functions to reduce code duplication.
Start by reasoning about the game and the unit tests. Also reason about the errors and possible fixes.
Finally, try write [num_targets] versions of the code. Make sure each code is a different code blocks starting with "python.
pracctionssinature contains function definition for the LLM to out, while test_code defines the properties (expressed unit tests) that the resulting code needs to satisfy.
conction_signatee and tedt_code both depend on if the game perfect or imperfect infor- mation game, whether it is being learnt in an open closed deck fashion, and if the inference is perform via hidden history or hidden state inference. These variations are defined in the following sections.
Finally, orig_code the code being refined at each iteration. On the first iteration, this paragraph is not present.
H.2 PERFECT INFORMATION GAMES
function_signaturel is defined as follows:
Action: str
State: dict[str, Any]
PlayerObservation: dict[str, Any]
29

---

Published as a conference paper at ICLR 2026
def astly_eetionstttee: State, action: Action) -> State: "Returnn the new state after an action has been taken. "**
def get www.uuurrrrttii puurent_player(sttte: State) -> int:
"Returns current player, with -1 for chance and -4 for terminal.
def dee__plyyr__nae(plaae ". "Returns the name of the player, with 'chance' for -1, and 'terminal' for -4.
int) -> str:
def (eeardseestttttee:: State) list[float]: "*"Returns the rewards per player from their last action, "**
def get Laegaalaciiossssateee State) -> list[Action]:
"Returns legal actions that can be taken in current state. """
def get_ ctt.obeeraaiiinsstttee State) ListpPaayeroserratiinn] "*"tuun the observation for player. ""*
test_code tests the transition between two states, testing each of the API calls defined in conction_signatur.. Here an example transition unit test for tic tac toe, where the board is provided as flat 1D array:
class iestrranstioon2unitteet... (ests
def
state = ('board': [None, None, None, None, 'x', None, 'o', None, None], currenttplayer_mark! 'x'}
ret_current__layerrffff
self.assertEqual ('O',, get_player_name (0))
self self.assertEqual 0.0], get_rewards(state))
self (elf..sssert board': [None, None, None, None, 'x', None, 'o', None, None],
current_ppaaer__arkk 'x' ('board': [None, None, None, None, 'x', None, 'o', None, None], paareent__ply__maa_______ Ses.errteet
'x'}], get_.bbervations x(0,1)', 'x(0,2)', 'x(1,0)', 'x(1,2)', 'x(2,1)', 'x(,,2) I]), Leagelleeallaaaaeeeeeaeeeaaallee self aa..aaasstttuuaaaccc (('oard'' 'o',, apply_action(state, 'x(1,0)')
board': [None, None, None, 'x', 'x', None, 'o', None, None], jurrent_player_mare
If this test has failed, the LLM is provided with the python error message in the form of comment before the test. The use of self asser tEqual style functions ensures that the L.M is provided with rich description of how the expected and actual data structures vary.
H.3 HIDDEN HISTORY INFERENCE FUNCTION SYNTHESIS, OPEN DECK
function_signaturr starts with the version from Section H.1 then adds the inference definition:
resample_historyC
ListEtupplePPlayerobssrwaaat Action None]],
player_id: int
-> list[Action]:
"""Stochastically sample one many potential history of actions for players( including 'chance' and 'terminal')
This is given only single player's observations and actions, and needs to recreate the player_id's observations
def
Iisttupple(Playeebbssrvvttto Action None]], player_id: int) list[Action]:
"w """Stochasiccally sample one many potential history of actions for all players( including 'chance' and 'terminal''
This is given only single player's observations and actions, and needs to recreate the player_id's observations
30

---

Published as a conference paper at ICLR 2026
unit_text again starts with the definition from Section H.1 then adds the following test for added inference function:
state INITIAL_STATE
jos_action_history
itttonnnccton = (obr(obsacttio hiittrottooooooorr)
current_player__ot current_player_action = next(obs_and_actionnitt
player_id (player_id)
for action in
player_id):
pfint(f"In state ((state])") if get_current__plaerrtstee
player_id:
self ae...assertEquall (currntttt _b_s, www.serrttionssstttttllllyrrrrrr
ofint( observation ([current_player_obb))
ass..assett (curreett (current__paayr__ctiil action)
current.player. current perrent_player_acttio = mext(obs_and__acionniit
print(f"Taking action ((action))") state apply_action(state,, action) try:
looss_an__acton__t
raise ValueError cacrrorrrFFFllee F'alledd to iterate through all observations.
except Stoplteration:
pass
self. asserttqual (player_id, get_current.pllaer((ee.
where INITIAL_ STATE is provided at the beginning of the unit tests and the static first state of the obs_action in_acttion___ hi story is the history of observations and actions for player player_id for which we want to resample the history of actions that lead to the current observations.
game. obs
Note the presence of print statements inside the unit test. The last ten lines of standard output provided to the LLM in addition to the eтror message.
H.4 HIDDEN STATE INFERENCE FUNCTION SYNTHESIS
conction_signttee again starts with the version from Section H.l then adds the inference fune- tion definition:
def (osepple_ttteeoss_______ celoobssattoo_tiiteeeeeee hiitory: List[tupleePPlayeebbberraal
Action None]],
player_iid _id: int) >> list[int]:
"""Stohhsticallly sample of the reachable statess for player given the observation and action history that recreates the player's observation. ".*
unit unit_test again starts with the definition from Section H.I then adds the following test for added inference function above:
iissaccion_hhstory Cobs_action_history player_id = {player_id)
resampled_state resample resappeestateeosss.cciiohhhhtooo player_id)
self. asserttuual aaserrEuuul(bssatttiooo..tttoot
})
H.5 HIDDEN HISTORY INFERENCE FUNCTION SYNTHESIS, CLOSED DECK
cnnction_signatur is similar to that in Section H.2
def (obbraooobsaction histooo:: list[tuple[ (aaaoleeeeaaeerreervaaiioo Action | None]],
player_idd int, cast_is_terminall bool) -> list[Action]: """Stochastically sample one of many potential histories of actions for all players( including "chance" and 'terminal')
given only single player's observations and actions.
It needs to recreate the player_ide _i''s observations. jast_is_terminal indicates if the last player observation is from end of game when player_id is -4. nu*
31

---

Published as a conference paper at ICLR 2026
Note the extra argument cest_iistteriinal. This indicates that the final observation in wwtaaacnnnistttteee is of the terminal state. This allows adding tests that resample the entire game from the beginning to the terminal state, testing the ability of the LLM to predict the final reward of the player. In open deck, the transition tests cover this. For simplicity, we assumed that the rewards are terminal but this is easy relax.
The corresponding uni t_sss for the inference function is:
state = INITIAL_STATE
obs_action_history fobs__acion_history
player_id {player_id}
last_is_terminal.
obs_and_action_iter = http:/oo__actoo_hiitiieiee
current_player__ob,, curreet__laaer_actioon = (att(obsand_action__..
for action in sisampleehisttrrrdbsaatttto
player_id, test_is_terminal):
print state ((state))")
if det_curreet__layer(sttccc = player_id:
aslffaasserEqualll (curreent cetuureet_pplaer_.ooo
pfint( observation [[current.player_obs]]]
self. (cureent current_player__oe current.player_action mext(obs_and_accionn_l
action)
(rint action ((action))") state apply_actionsstate, action)
try:
next
raise ValueError (' FFailed to iterate through all observations.
except StopIteration:
pass
Again, this is very similar to unit_test in Section H.2 but also covers the terminal state of the game and it's associated reward.
Note that no transition unit tests are added as we not have access to state. However, just testing the inference function is not enough ensure that the resulting closed deck game is playable. Instead, a random play test is added to unit_test:
state = (initial_state)
rg for it in range(1000) upper bound gane length current_player get_current__layersttar rewards (staree
RandomStaate(sseed)
assert (enwwraddss ==
print (f"State is {[state)), current player is ([current.player]), rewards are {( rewards})")
if current_player = -4: # Game over break
if current_player in [o, 1]: Real players
(f for current player ((Ge..obrrreroooo(tteeeerrrrrrrr }}}")
else:
assert current.player ==
legao_actions get
chosen_action = rg. choice Legaa__cctions)
pfint(f"Taking action ((repr (chossnnactiioon))) from ([len(legalacctionn)) options, first 10 are (([*legal_actin "ctleaal_octtoons[1111)"""""""""
state apply_action chosen_action else:
raise ValueError Gf"Game did not end after 1000 steps.")
This tests that if every player randomly picks valid move, the game will correctly play and termi- nate. Note that we assume access to the static and deterministic initial state of the game, before any chance nodes have taken place. This could also be synthesized by the LLM instead.
32

---

Published as a conference paper at ICLR 2026
H.6 RESAMPLING THE STATE GAME PLAYING TIME FOR IMPERFECT INFORMATION GAMES
When playing the game, we allow the system up 10 tries to get valid state that produces the current observations:
for retry in range(10): json_state (start_state)
try:
actions = resample historyyobss.ctiionniissoee player_id)
for action in actions:
json_state = apply_ [__actionjjon_sttttee action)
state_log (jsonssjjso_sstte)
except Exception as e: Running generated code, could raise anything. continue
recreated_obs = gel.odseration((jsenntttet))) [plyyeeetttpplle_idlll
if recreated_ode obss == obs irs_acttionhisttrr-111199 return json_state
Additionally, the ISMCTS process fails due to, e.g., poor understanding of the game termination criteria in the CWM, we fall back to resampling the state and then return uniformly sampled legal action from that state.
H.7 VALUE FUNCTION SYNTHESIS
Our value function synthesis function prompt is
are an expert python programner. You are playing game (game), and to synthesize value function for monte carlo tree search.
[same_descripttion
For reference, game implemented as follow
(code)
The function you need to write is: (value_function)
It should return the reward at terminal states, and otherwise estimate of the value for each non-terminal states.
It should always be a float: (player_tests)
Terminal states should match rewards: (terminal_tests)
To write good value function first reason about the game and produce a heuristic value that is informative, and do not just output zeros everywhere other than terminal states.
Finally ONLY output the new value_function, do not output any other text, code,
explanations or placeholders.
The response code must be single CODE BLOCK that uses this format: The opening fence: ""python
The closing fence:
Where value_function
def ioaluuefunction(staaee dict[str, Any], player_id: int) >> float:
"*"Reuurns the value estimate for player_id in state.
33

---

Published as a conference paper at ICLR 2026
For terminal states the function returns the true return. For ongoing play the function should return a value estimate that reflect the winning potential of the player with given player_id.
player_tests and terminal_tests the list of example
courrent_players
self. att.arresssssssteeeeeeeeeeeeeeee finellus_uucciiosstate,,e (current_player)), float)
if (current_player) == yyspie..Playerid TERMINAL::
rewards = (stare
for player in range(len
(stetinnnnni,,,ttee,, player))
34

---

Published as a conference paper at ICLR 2026
1 GAME RULES
1.1 BACKGAMMON
Backgammon two-player board game that combines strategy and luck. The object to move all of your checkers off the board before your opponent does. Here a breakdown of the rules:
The Board and Setup:**
** The #oard:** The board consists of 24 narrow triangles called *xpoints**, These points are grouped into four quadrants of six points each:
#IInner/Home Board:** The quadrant closest each player's starting position.
**Outer #oard:** The quadrant further from each player's s starting position.
**The Bar:** The area in the middle of the board, separating the two sides.
**The Bear-Off Area:** The area off the board where checkers are moved once they reach the player's home board.
**Checkers:** Each player has 15 checkers of color (typically black and white). are to determine movement.
**Dice:** Two
#DDubbiig Cube (Optional but common) *** cube with the numbers 2, 4, 8, 16, 32, and 64, used to increase the stakes of the game.
**Initial Setup:**
Each player's 15 checkers are set up in specific configuration on the points:
2 checkers the opponent's 24-point.
checkers on the opponent's 13-point.
3 checkers on their own 8-point.
* 5 checkers on their own 6-point.
x*Gameplay:**
**Starting the Game:** * Each player rolls one die. The player with the higher roll goes first. If the rolls are the same, they roll again until one player rolls higher. The player who goes first uses the numbers rolled *both* dice to make their first move.
2. **Rolling the Dice:** On subsequent turns, each player rolls two dice.
x*Moving Checkers:* After rolling the dice, the player must their checkers according to the numbers rolled.
**Separated Moves: Each die represents a separate move. You can move one checker the distance of one die's roll and another checker the distance of the other die's roll.
**Combined Move:** You can move one checker combined distance both dice rolls, but Ronly if* the point you would land on for the first die's 's roll not blocked (see "Blocked Points" below).
**Mandatory Moves: You must move your checkers if possible. If you can only make of the two moves indicated by the dice, you must make that If you make both, you must make both.
**No Legal *o If you cannot make any legal moves based on the dice roll, your turn ends.
**Point Direction:**
always move your checkers from your opponent's s inner board towards your own home board. The points are numbered to 24, where 24 is the latest point in your opponent's inner board. Each move makes the checker move to smaller numbered points.
**Landing on Point:**
**Empty Point:** You can land on an empty point.
**Point Occupied by Your Own Checkers:** You can land on a point occupied by any number of your own checkers.
*xPoint Occupied Opponent's Checkers:
35

---

Published as a conference paper at ICLR 2026
x*Blots:** If point is occupied by *only one* of your opponent's 's checker's, it called "blot." " If you land on blot, you "hit" the opponent's checker. The hit checker is placed on the **bar**, **Blocked ** If a point is occupied by #two or more* of your opponent checkers, it is "blocked." You *cannotx Land on blocked point.
exEntering from the Bar: a player has checkers on the bar, they must re-enter them onto the board before making any other moves.
**Re-entry ** You can re-enter checker from the bar onto a point in your die. For
opponent's 's home board that corresponds to the number rolled example, if you roll 3, you can re-enter checker onto your opponent's point.
**Blocked Re-entry:** If the corresponding point in your opponent's 's home board is cannot re-enter using
blocked by two or more of your opponent ss checkers, that die roll.
x*Priority:** You must use any available die rolls to re-enter checkers from the bar. If you can re-enter one checker but not the other based on your dice roll, you must still re-enter the one you can. If you cannot re-enter any checkers, your turn ends.
**Doubles:** If you roll doubles (e.g., two 45), you can use each number *four times * So, two 4s means you have four moves of 4. You can use these moves in any combination, as long as they are legal. Each turn allows you to make two moves only So if player rolls double, they take extra turn, making most two moves each of the turns.
8. **Baaring *ff If you don't have any checkers outside of your home board or at the bar, you can "bear off" chekers (moving them off the board) that are at your home board.
**Bearing Off Rolls:** bear off a checker, you must roll the exact number that the checker is on to move it off the board from its current point. For example, if a checker is on your 4-point, you need to roll 4 to bear it off. **Higher Rolls:** If you roll number higher than the highest point occupied by your pieces, you can still bear off a piece. However, you must bear off a piece from the highest occupied point. For example, if your highest occupied point is the 4-point, and you roll 6, you can bear off piece only from the 4-point, cannot bear off piece from lower points.
*sLower Rolls:** If you roll number lower than the point your checker is on, you can still move a checker from a higher point the distance of the roll (if legal ), or you must move a checker from lower point the distance of the roll if possible. You cannot bear off checker if you have checkers on higher points in your home board that can be moved by the dice roll.
*BBocked Bear Off:** You cannot bear off a checker any your checkers are still on the bar or outside of your home board. You must bring all your checkers into your home board before bearing off.
exMoving Pieces Within Home Board:** Instead of bearing off, you can also move your checkers within your home board using the dice rolls.
9. *cttion Notation:** Player moves are typically represented using a specific notation Each turn consists of at most two moves. Each move isrepresented by a string of the form "Move checker at X using Y roll", where "x" is the pasition of the checker is done based the dice roll with the
being moved, and "Y" indicates if the higher or lower number.
The first position is always a number between 1 and 24 or "Bar", and it is presented in each player's perspective, where 24 is the latest point in the opponent's 's inner board.
For double roll (e.g., , a 3-3, granting four moves
as per rule "Doubles")
It is assumed these four moves are made by the player in two stages: First, providing two moves, then the player gets second turn then providing the subsequent moves. For instance, player might move two checkers, each making two 3-point moves. Next, the player would have a second turn, providing two more moves of 3.
10. **Board Notation:** The board is represented with arrays of 24 numbers,
36

---

Published as a conference paper at ICLR 2026
where each number is either (empty) number between and 15 (indicating number of checkers of that color on that point).
The first array is for the first player, and the second is for the second. The board ordering is from second player's persective. Starts from first player's home base and ends at the second player's home base. The first index is the latest point in the second player's inner board, meaning position for the first player and for second player.
**Winning the Game:**
The first player bear off all of their checkers wins the game.
**Optional Rules (Commonly Used) ***
** Doubling Cube:**
x*Offering a Double:** At start of their turn, *before* rolling the dice, player can offer to "double" the stakes the game. **Accepting Double:** The opponent can either accept or decline the double. If they decline, they lose the game immediately and the current stake is paid. If they accept, the stakes cube, meaning they are the only one who can offer the next double.
doubled, and the opponent
"owns" the doubling
**Subsequent Doubles: RM The owner of the cube can offer to redouble at the start of their turn. The stakes continue to double with each accepted redouble (2, 4, 8, 16, etc.).
**Gammon Backgammon: These are ways to win with higher stakes.
**Gammon: ** a player bears off all their checkers before the opponent has borne off *any* checkers, the winner wins "gammon," which is typically worth double the value of the doubling cube.
x*Backgammon; If player bears off all their checkers before the opponent has borne off *any* checkers
the opponent still has one or more checkers on the bar or in the winner's home board, the winner wins a "backgammon," which is typically worth triple the value of the doubling cube.
Key Concepts Strategy:**
**Hitting Blots:** Hitting your opponent's s checkers puts them on the bar and disrupts their progress.
**Making Points:** Occupying points with two
of your checkers creates "blocks" that prevent your opponent from moving past. Strategic point-making crucial.
**Prime:** Creating a "prime" (six consecutive blocked points) can severely hinder your opponent's movement.
** Moving your checkers quickly towards your home board. **Posi cossitioninn:** Carefully considering where to move your checkers to maximize your options and limit your opponent's s.
**Runnin::**
* *xRisk vs. Reward: Balancing the risk of leaving blots with the potential for making good moves.
Backgammon game with layers strategy that unfold as you play, While the dice introduce an element of chance, skillful play, understanding probability, and strategic decision-making significantly influence the outcome. Enjoy the game!
1.2 CONNECT FOUR
### Rules of Connect Four
** Connect Four is played on a 6-row by 7-column vertical grid, which starts completely empty.
* x*Players and Marks:*x There are two players: Player 0 uses the 'x' mark and Player the 'o' mark.
**Turn:*** Player ('x') always goes first, and turns alternate between players. **Making a *ove:** On your turn, you choose a column to drop your mark into. The mark will fall to the lowest unoccupied square within that chosen column. Attempting to drop mark into column that is already full is an invalid move; you must choase column with at least empty square complete your turn.
37

---

Published as a conference paper at ICLR 2026
*wWinning the Game:** The winner is the first player to get four of their marks in (horizontally, vertically, as a winning line is formed.
diagonally). The game ends immediately soon
**Drawing the Game; ** If all 42 squares on the grid are filled and neither player has won, the game ends in draw.
*xEnd the Game: The game only concludes upon win draw. A player must make move on their turn as long as there is at least one valid move available on board.
**Move Notation:** Use the move notation '[mark][col]', where col is the 0-indexed column you are dropping your mark into. For example, 'x3' means Player ('x') drops their mark into the fourth co from the left (column index 3).
1.3 TIC-TAC-TOE
#4B Rules Tic-Tac-Toe
**Setup:** Tic-Tac-Toe is played on 3x3 grid, which starts completely empty.
**Players and Marks;** There are two players: Player 0 uses the 'x' mark and Player uses the 'o' mark.
**Turns*** Player ('x') always goes first, and turns alternate between players.
* wAMaking Move:** ** On your turn, you must place your mark in a single, unoccupied square. Attempting to place mark in already occupied square is an invalid move you must choose empty square complete your turn.
**Winning the Game:** The winner is the first player to get three their marks row (horizontally, vertically, or diagonally). The game ends immediately as soon as a winning line is formed.
**Drawing the Game; ** If all nine squares the grid are filled and neither player has won, the game ends in draw.
**End the Game: The game only concludes upon win or draw. A player must make move on their turn as long as there is at least one valid move available on the board.
*MMove Notation:*x Use the notation 'mark(row, col)', where and col e- indexed. For example, 'x(0,0)' means Player ('x') places their mark in the top- left square.
1.4 GEN. TIC-TAC-TOR
Generalized Tic-Tac-Toe (6x6, Win Length 4, 2 Players)
1. Overview: This is two-player strategy game played 6x6 grid. The goal is to the first player form continuous 1ine of four of your own marks. This game is a specific configuration of generalized Tic-Tac-Toe framework.
2. Game Setup:
Board: 6x6 grid of cells (35 cells total), with rows columns numbered
0 to 5.
Players: Two players. Conventionally, one player uses 'x' and the other uses 'o'.
Starting State: The board is initially empty.
3. Gameplay:
Players take turns placing their mark unoccupied cell the board, designated player (e.g., Player 'x') makes the first move. The game continues with players alternating turns.
4. Winning Condition:
player wins if they are the first to place four their marks in an unbroken straight line.
This line can be:
38

---

Published as a conference paper at ICLR 2026
Hor Norizontal: Four marks in the Vertical: Four marks the sane column.
Diagonal: Four marks along any the board's diagonal lines (both directions).
5. Draw Condition:
If all cells on the board are filled with marks, neither player has achieved a line of four of their marks, the game is draw.
6. End of Game:
game concludes immediately when either:
One player achieves a winning line of four marks (that player is the winner). All cells are filled, and winning 1ine exists (the game is a draw).
Key Parameters ters for this Specific Variant:
Number of Rows: 6
Number of Columns:
Winning Line Length:
Number of Players:
1.5 GEN. CHESS
The game of generalized chess is a two player game where each player controls a collection of pieces and wins by capturing the target piece from the other player. Each kind of game piece has specific pattern of movements that it can execute. piece stays the board and doesn't land on another of that player's s pieces. If the piece lands on an opponent piece, it captures the opponent piece and removes from the board. Allowed piece movements are not the same as in standard chess.
execute any of its available moves long that
board, rows labeled A-E from top to bottom, and columns are labeled 1-5 from left to right. move from starting square to a destination square is written as 'start_to_end" for example, "A2_to_C2" means move the piece from square A2 to square C2.
Actions described using board coordinates. For
Passing turn is specified 'PASS"
This 'army5x5a' variant of generalized chess is played on 5x5 board.
It includes the following pieces, with their correspanding set of allowed general: [(1, 0), (-1, 0), (0, 1), (8, -1), (0, -2), (0, 2)]
infantry: [(1, 0), (2, 0), (1, -1), (1, 1), (-1, 0)]
cavalry: [(0, 3), (1, 2), (2, 1), (3, 0)]
Game pieces depicted with the following symbals: 'general': : 'x', 'infantry's 'I' cavalry': : 'V'. Player pieces are upper-case while Player 1 pieces are lower-case.
The 'general' is the target piece. Capturing this piece wins the game.
1.6 BARGAINING
The rules of "bargaining" aren't fixed and formal like a board game with a rulebook. Instead, it's a dynamic social process of negotiation where two or more parties attempt to reach mutually agreeable outcome on price terms for product, service, or agreement. Here's a breakdown of core principles and common "rules" of bargaining, understood more as strategies and expectations:
**Core Principles of Bargaining:**
39

---

Published as a conference paper at ICLR 2026
*xMutual Desire for Agreement: Both parties generally want reach deal, their initial positions are far apart.
IIfoormation Asymmetry:** One party often has more information than the other, which can influence the negotiation.
*xIterative Process: Bargaining usually involves a series of offers and counter- offers.
**Focus on Value: Bargaining about perceived value what each party believes the item or service is worth. **Potential for Compromise:** Both parties are usually expected to give little to reach an agreement.
**Implicit "Rules" or Common Strategies:**
These are not hard-and-fast rules, but rather common practices and expectations that guide the negotiation:
**Know Your Limits (Kalk-Away Point) *** Before starting, each party should have a clear idea of the maximum (for buyer) or minimum (for a seller) price they are willing to accept. This is your "reservation point."
**Start with an Anchor (Opening Offer) The first offer sets an "anchor" for negotiation. This is usually a price lower than what the seller expects (for a buyer) or higher than what the buyer expects (for seller).
**Seller's Perspective:*x Start higher than your desired price.
**Buyer's Perspective:** Start lower than what you" re willing to pay.
3. **Justffy Your Offers:** Simply stating price is less effective than explaining * why* you' re offering that price. Reference market value, condition of the item, your budget, etc.
4. xxMake Concessions Incrementally:** Don't tt jump straight to your walk-away point. Make small concessions with each counter-offer. This signals a willingness to negotiate while still trying get the best possible deal.
**Signal Willingness to Walk Away (But Don't Bluff Too Much) ;** Letting the other party know you' re willing to walk away if you don't 't get satisfactory price can be powerful tactic. However, repeated unbelievable threats can undermine your credibility.
6. *LListen Actively and Ask Questions:** Pay attention to the other party's offers, reasoning, and potential underlying needs. Asking questions can reveal information and build rapport.
7. **Be *atient:** Bargaining takes time. Don't t rush the process.
8. x*Maintain Respectful Tone:** Even if the negotiation becomes difficult, try to maintain polite and respectful demeanor. Aggression shut down the conversation.
9. **Consider Non-Price *actors:** While price is central, bargaining can also involve other terms like delivery time, payment method, warranties, or additional items included.
10. *nKnow When to Stop:** If it's clear you won't reach an agreement that meets your needs, it's okay to respectfully end the negotiation.
**Be Prepared to Walk Away:** If you can't reach an agreement within your limits, you must be prepared to walk away. This is crucial for maintaining your boundaries.
12. **The Final Offer:** Often, one party will indicate their "final offer. This suggests they unwilling to make further concessions. However, this isn't always truly final and can be tested with counter-offer.
13. **The Art of the Counter-Offer:** Respond to offers with counter-offer that is concession from your previous position, but still moves you closer to your goal.
40

---

Published as a conference paper at ICLR 2026
**Situational Differences:**
The "rules" of bargaining can vary depending on the context:
**Cul Norms: Bargaining is much common and expected cul tureu (e.g bazaars many parts the world) than others (e.g., retail stores most Western countries).
**pp of Item/Service:** Bargaining for car is different than bargaining for a small trinket market.
**Power Dynamics:** Who has leverage in the negotiation can significantly impact the process.
**In summary, the "rules" of bargaining are less about strict regulations and more about strategic communication, understanding the other party's perspective, and being prepared to make concessions reach mutually acceptable agreement. It's negotiation dance where both parties are trying to get the best possible outcome within their own limits.**
1.7 LEDUC POKER
Leduc Poker is simplified two-player poker game, ideal for research, that small deck to focus on core poker concepts like betting strategy and imperfect.
information.
Here is detailed breakdown of the rules clarify legal moves. Note that in this implementation, the "Check" action is not available; players must use "Call" instead. call may be zero-cost if there is no outstanding bet to match.
*+1. Setup & Preliminaries**
**Players:**
**Deck;** 6 cards (two Jacks, two Queens, two Kings).
wMBlinds: Before cards are dealt, mandatory bets are posted:
Player (P1) posts **Small Blind** of 1 unit.
Player (P2) posts **Big Blind** of 2 units.
**The :eal:** Each player receives one private card, face down.
*+2. Core Betting Rules**
**Raise Sizing:** The amount to raise is fixed. **Round 1;** The raise amount is **2 units** **Round 2:*x The raise amount is **4 units*x.
* **Total Betting Cap The total betting cap for each round is maximum of x* two raises**,
**Acting First:** Player (the small blind) acts first in both betting rounds (pre- flop and post-flop).
Pre-Flop Betting** This round occurs before the public card is revealed.
*M3. Round 1:
**P1's 's First *c must act on P2's 2-unit Big Blind.
**Fold:** Forfeit the 1-unit blind. P2 wins the pot. **Call:** Match the 2 units by putting in more unit.
**Raise:** Make 2-unit raise, for total of units (P1 puts in units). The total betting cap has been reached. **P2's *ction;**
If P1 *xcalledkx, P2 can **Call** (a zero-cost action, as bets are equal) to end the round, **Raises* (by putting in 2 more units to make it 4 total). If *araisedxx, P2 only **Call** (by putting in The betting cap been reached.
units) #*Foldex.
**+P1ss Second Action (if necessary):** If P1 called and P2 then raised, the action returns to P1. P1 can only **Call** (by putting in more units) or **Fold**,
*+4. The Flop: Public Card**
41

---

Published as a conference paper at ICLR 2026
After Round betting concludes, public card is dealt face-up. This card is shared by both players.
*+5. Round 2: Post-Flop Betting**
This round occurs after the flop. There are blinds.
sMP1's First Action:*
**Call:** Make a zero-cost call to pass the turn (as there is no outstanding bet)
**Raise:** Make 4-unit raise.
*AP2's Action:**
**called** (at zero-cost), P2 can also **Call** (at zero-cost, ending the round) or **Raise** 4 units.
If P1 **raised**, P2 can **Call** (matching the 4 units), **Raisex* (by putting in another 4 units, for total bet of 8), **Fold**. The total betting cap has been reached.
*SSusequeent Actions:**
If P2 **raised** (after P1's initial zoro-cost call), the action returns to P1, who can **Cal1** (the 4 unit bet), **Raise** (to 8 total), or **Fold**, The total betting cap has been reached.
a player k*raises*R, the other player can only **Call** or **Faldee, as the betting cap has been reached.
**6 Showdown Hand Ranking**
If neither player folds, showdown occurs after Round 2 betting.
**Hand:** ** A player's hand is their private card combined with the public card. **Hand Ranks (best to ;
1. **Pain:** Two cards of the same rank (e.g., J-J). Higher pairs beat lower pairs.
2. *xHiihh Card:** If no one has pair, the player wi the highest card wins (K
> +J
**Ties:** If both players have the same hand rank (e.g., both have King-high), the pot is split.
**7. Winning**
A player wins the pot either by being the only one left after the other folds, or by having the best hand at showdown.
I.8 GIN RUMMY
The Game of Gin Rummy
Gin Rummy is a two-player card game played with a standard 52-card deck. The primary objective is to form "melds" in your hand, which are either sets of three four cards of the cards the same suit sequence (e.g., 4h 5h 6h). Cards not part any meld are referred to as "deadwood." " value of deadwood cards corresponds to their rank (Aces are 1 point, face cards are 10, and number cards are their face value). The ultimate goal is to minimize the point value of your deadwood.
rank (e.g., 7h 7c 7d) or runs of three
Rummy concludes when player "knocks." " A player can choose knock on their turn if the total point value of their deadwood is less than or equal to a predetermined "knock card" value. Announcing "gin" is special type of knock where player has deadwood at all.
round
Player Hand Information: This section provides details about your own hand.
Deadwood:
This calculates the current point total of the cards in your hand that are not part valid meld (a set or run). Minimizing this value is the primary goal.
The Card Grid: This is visual representation of the cards currently hold. It is organized logically for easy parsing:
42

---

Published as a conference paper at ICLR 2026
Rows: Each of the four rows corresponds to suit, in the order of Spades (top row), Clubs, Diamonds, and Hearts (bottom row).
Columns: The columns represent the rank of the cards, ordered from Ace on the far left to King the far right.
Here are also some example moves:
Player: 8 Action: Pass
Player: Action: Draw upcard
Player:
Action: Ic
Player: 8 Action: 3d
Player: 1 Action: Draw stock
Action Legality is Dictated by Game Phase: Before selecting a move, you must first check the phase.
If the phase is Draw, the only valid actions Draw upcard Draw stock.
If the phase is Discard, the only valid actions are to discard specific fic card from your hand (e.g., Action: 4c) or to Knock.
player cannot discard a card until after they have successfully drawn one.
# Special Case: The First Turn of the Round
very first turn of round a unique rule. The non-dealer has first option on the initial upcard.
The non-dealer either take the upcard (Draw upcard) or Pass.
If the non-dealer passes, the dealer then has the same choice: take the upcard or pass.
If both players pass the initial upcard, the non-dealer must then start their turn by drawing from the stock pile. After this initial sequence, play continues with the standard draw/discard phases.
Knocking:
When player knocks in Gin Rummy, the round immediately ends and a specific sequence of scoring, known as the "layoff," begins. Here is a detailed breakdown of what happens.
1. The Knock Laying Down Hands
First, the player who is knocking (the "knocker") lays their hand face up on the table, organising their cards into melds (sets and runs) and separating their unmelded cards, known as "deadwood.""
What the Player Needs to Do After Knocking
After sending Action: Knock, the player must follow strict, multi-step process to lay down their hand for scoring.
Step 1: Declare
Melds
The player must now explicitly declare their melds to the game, one by one. For exampple, if the agent's 's hand contains two valid runs:
of clubs: 7c8c9CTc
A run of diamonds: 9dTdJdQd
Correct First Move in the Knock Phase:
43

---

Published as a conference paper at ICLR 2026
Player: Action: 7cBc9cTe
or
Player: Action: 9dTdJdQd
Step 2: Declare Subsequent Melds
After the agent declares its first meld, it will receive observation. The game will still be Phase: Knock. The Valid actions will now include any remaining melds that can be made from the cards left in the hand.
Note: The value of the knocker's s deadwood must be points less (or the value of the designated knock card for that round). Face cards are worth 10 points, aces are 1 point, and all other cards are their numerical value.
2. The Opponent's Turn: Laying Off
Next, the defending opponent lays down their own hand, also separating their melds from their deadwood. Crucially, the opponent then gets the opportunity to
"lay off" any of their own deadwood cards by adding them to the knocker's melds.
For example:
If the knacker has a meld of three Kings (Ks Ks Ks), and the opponent has the fourth King (Ks) deadwood, they can add it to the knocker's 's set, thus eliminating those 10 points from their deadwood count.
If the knocker has a run of 5h 6h 7h, the opponent can lay off 4h or an 8h to extend the run.
The knocker is not allowed to lay off any of their deadwood the opponent 's melds.
3. Scoring the Hand After the opponent has finished laying off their cards, both players calculate the final value of their remaining deadwood. The scoring for the hand is then determined in one of three ways:
a) Successful Knack
If the knocker's s deadwood count is lower than the opponent's deadwood count, the knocker scores the difference between the counts.
Example: The knocker has points of deadwood. The opponent initially has 35 points, but after laying off 18-point card, their deadwood is reduced to 25. The knocker scores 18 points (25 7).
b) An Undercut
If the opponent, after laying off their cards, has a deadwood count that is equal to or less than the knocker's count, they have "undercut" the knocker. In this scenario, the opponent scores the difference in points (if any) plus bonus, which typically points.
The knocker has points. The opponent has points after layoffs. The opponent scores points (8 - 6) plus 25-point bonus, for total of 27 points.
Example:
c) Going Gin
If the knocker has deadwood count of zero, this is called "going gin." The knocker receives bonus (typically 25 points) in addition the full value of the opponent's s entire deadwood count. When player goes gin, the opponent not allowed to lay off any of their cards.
Example: A player gin. Their opponent has 42 points of deadwood. The ginning player points.
42 points plus 25-point gin bonus, for total of 67
What if the Stock Pile Runs Out? If the stock pile is reduced to its last two cards and the player who drew the
44

---

Published as a conference paper at ICLR 2026
third last card discards wi thout knocking, the hand is declared draw. points are awarded either player, and the deal passes the next player for a new round.
If see Phase: Wall in observation, it means:
Stock Pile Exhausted: The round has concluded because there are no more cards to be drawn from the stock.
No Player Has Knocked: Neither you your opponent were able knock by the time the last card was drawn.
The Hand is Draw: No points are awarded to either player for this round. The hand is over
terminal state for the current round. The only thing to do is to acknowledge the result and wait for the next hand to be dealt. The deal will typically pass to the player who didn't tt deal the drawn hand. Thus "Player: X Action: Pass" must be provided action.
No Action Required: The game
1.9 QUADRANTO
Quadranto a partially observable game which two players try catch each other in a 4 by matrix.
The 4 4 matrix is divided in quadrants, At the beginning, player is randomly placed in the top left quadrant and player is randomly placed in the bottom right quadrant.
During their turn, each player can choose to move in each of the four cardinal directions, "Left", "Right", "Up", "Down". Or they choose to "Stay", , which means they remain where they are. When player moves, it lands on the same location where the other player is, it wins and the game ends.
The observation tells the player where it is located and in which *quadrant* the opponent player is located. Therefore, neither player knows exactly where the other player is located until the very moment which one player catches the other.
If the players perform total of 20 without catching each other, the game ends draw, both players get 0 points. If one catches the other, the winning player gets +1 points and the losing player gets -1 points.
I.10 HAND WAR
a*Hand of Wark* is a strategic card game where choosing your cards wisely is key to victory. You't 11 manage hand of cards, adding layer of tactical decision-making to every round you aim to capture all of your opponent's cards.
**bbeectiv::**
The goal of Hand of is capture as many of your opponent's cards.
**Setup:**
**Shuffle and Deal: Thoroughly shuffle the deck. Deal the entire deck evenly between two players, face down. **Form #ands:** Each player draws the top three cards from their draw pile.
**Gameplay (The "Battle"):*s
45

---

Published as a conference paper at ICLR 2026
**Choose Card: Simultaneouslyy both players select card from their
hand and place face down.
**Reveal and * Both players flip their chosen cards. **Higher Card Wins:** The player with the higher-ranking card wins the battle and takes both cards, placing them at the bottom of their win pile. **Card Ranking:** Ace (High), K, Q, J.
*nDraw New Cards:** After battle, players draw from their draw pile replenish their hand to three cards.
*** "Showdown" (When Cards :
*#Declaration If cards are the same rank, a "Showdown" accurs. **Face-Down Cards: Each player places card from their draw pile face
down.
**Choose Battle Card:** Players choose card from their hand and place it face up.
**Determine Showdown Winner:** Higher battle card wins all cards in the
Showdown.
**Another Tie:** Repeat Showdown process (burn 1, choose card).
**Draw After Showdown: Players replenish their hand three cards.
*^Game End :onditions:**
The game end in of two ways:
**Winning by Capturing All Cards:** The game ends immediately if one player possesses all 16 cards. That player is the winner.
2. **Winnigg Win Pile Count (Draw Pile Depleted) :** If player's draw pile becomes empty and they are required to perform an action they cannot complete (such as drawing a card to replenish their hand after battle, or burning a card during Showdown), the game ends immediately. In this scenario:
The number of cards in each player's win pile is counted.
The player with more cards their win pile wins both players have an equal number of cards in their win piles, the game results in draw.
game.
Note: The win pile is never reshuffled into the draw pile during play.
**Notation**
Player always moves first, followed by player 1. Thus if the previous move was player 0 move, then the next move is player move.
convenience, list of played cards is provided PubliclyRevealedCards, but not playable anymore.
46

---

Published as a conference paper at ICLR 2026
J SAMPLE SYNTHESIZED CWMS AND INFERENCE FUNCTIONS
We show the resulting CWM after tree search refinement on perfect and imperfect information games, and for the latter we include both open and closed deck inference. This includes inference and value functions when they exist.
J.1 TIC-TAC-TOE (PERFECT INFORMATION)
import collections
from typing import Mny cast
clarity defined problem.
State dict[str, Any] Playersbervation dictCstr, Any]
constants napping players, karks, defining winning PLAYER : 6, lo' 13
JATANTNEI TNFS
[
(9, 1, 2), (,, 4, 5), (6, 7,
Columns
3,
1), (G,
Diagonala
(9, 4, 8), (2, 4,
]
Helper
def wwweck..ineerreeeee. (board: lististr None]) -> str None:
""Checks if there
the board, returning the winnen's mark."" mee
pl, p2, pd Tine
y Check if all three cells in line are the same and not empty.
board[pl]] and board[pi] board[p2] en board[p]]:
return board[p1]
return None
def (baadb list(str None]) -> bool:
"Ceeck gane has either by
return wineck.iinee (baad)) not None or all(ceel is not None cell in board)
Functions
def alcllacctiionssttt State, action: Action) >> State:
"*"Rsturns state afteer action
action[e]
row int(accton[2))
col = =nnt(action[]]
Create copy board nodify.
new board statel board [[:] index row x col
nox_board[index)
I A gore is over if there a winner or the board is full.
if (ww.aaa_ooerrew_bbbbbo next_player,
Alternate turns betwoen 'x' and 'o".
next_player saak to" if rark == 'x" else 'x"
return ('board": new_board, "current_piayer_waar next_player_rarte
def patrrrnnt_playrstttttr State) +> int: " "Rtturrss current statel'cuu player_rart parrk
chance For terninal. m*.
if nark is None:
return -4 Terninal state
AYFRDwark]
def get www.ppaaarrrrrr Laaver__avejplayer.. id: int) +> str: "*"Raturns the name of the player, with "chance" for -1, and "tarminna" For player_id zz
"terrinal" player_id =1:
return "chance" str(player_idf)
def get (stardsstssttee State) lsst[float]: "Returns the rewards per player fros their last action, ane
Rowards
tersinal
get net__cureet_pll pureent_playerrtttate
return Ea.e, 8.8]
_check_wimer board winnee_juakk
47

---

Published as a conference paper at ICLR 2026
return L1.0, -1.0] Player
winner_nee _aakk to "o":
return [-1.8, 1.6] if Player wins
def vet_legal. actaonststato: State) -> List[Action):
""Returns legal actions be in current state."! !!.
Logal actions the gane get current, player (state) -4;
return []
actions []
tark state current, player waakk ]
# action is logal for any easty (None) square.
for cell enmerateessttee 'baadd)))
rox, col divmod actions.
def get dbsersationssstate: State) 1ist[Payerddservattaa]: "*"Rsurns the observation for player. ***
abservable (state, state)
state.
48

---

Published as a conference paper at ICLR 2026
J.2 BARGAINING (IMPERFECT INFORMATION, OPEN DECK)
Import copy
Ieport itartools
randon
typing import AP,, Dict, List, Optional, Tuple
3 Type Allases For clarity
Dict[str, Anyl
PlayerObservatiion Dict[str, Any]
Gare Constants
sortoc(['X', 'Y', (2'))
MIMPII AYERS =
MAX_TURNS = 10
outcoses, likely deterministle testing or analysis.
49

---

Published as a conference paper at ICLR 2026
1,1,1:21111111...
1,3.1222222200000
1.11332.3.0:0041.
11,3114.8.63,.122
1.223.0111::7...11
2,1.2,2,,,,22222
1.3.2:4,2.0.8,,,22"
2.1113221111.0110000
1.2.1:1.2.333.1.00
14.12.1....11..2222
1.33.:0.1..:4....0
11..50.2.112.0000
11 11.3112.2222112233
11,.119,,,11811135.
2.2.1.0.2.,,4111.00
3.2.1:2.1.2:133.11
11.3:15.1.2:3,8,77
"62.111....0..3.4
22..1:11,,,2,5,000
1,113:6.111.0,11333
22.323,,4,,332,."
4.1.2.1.4...22.1.11
112.118...18.111.66
22,22.1,,,44223338
22.12233,,,0:12223333
2.1.1222..22222.
1 1,222.4.0.3222133""
22.22222.1.22222211
22.22.21111,:,,..00
15.1..1.2..:1...77
11.13:322.1:022.08
3,2.1122 1,1,2,2,,,,2111 1,2,1
2.1.4:1...003,,0.1
1.2.2.661111.88....
22.1222222,2.11,,,8
1.4.111.1.1122,,,88
"3..1.2.....3......
"2..2.2222211881111
11:1.3225511:66,,8
1.3.113.1.4:1110.0
1...111.0.221..6.1.
15:1111..111111144
1...1.1....33330.11
3.1.1.1,,,::22443
2,221.21,,4223333
11,,224,,,,1,,,,,222
22,2111..,,,211.3
33.11113.3.7:111..8
2,
1.3.1.2...50002211
4,2.1.1,,,,000,.0
1,114.0.6.111...11
0..13:39...0:3322.11
2,,
22..1.2..122222236.
2,,122241,,0,,,2222 1..02.1..111........
1.1.1.1111.322..00
9.11113.1.,22220.4
11142.1.111111111111
2. 4,,
1.1.231112,22222222
11.12.8..0111133333
1...1...0.
1.3.1.4.1.333.2211
1,1,111 2,,2,,1,,,,,,,1,22
3.3.1.2.1111..111
1,1.3.2
22.1,3:2.0,22331111
2,3.2.1,221,1,,,3
14,12,2..81111122222
10,1,30
2,2.110,,,622211.0
1...2.6....88...11
21.2..1..222221.2222
22..1.1...22332.00
2..13:2.22.2331.222
3...1.1..466:22001
1,,,11111,,,,,,,,
3.1.122,,44411..00
2.012.,,
13.1221111:2222.3
12.22211..2331144.000
1,.14,5,111,2,44.
5,,,3,5,,,,,,1,0.
3,3.1.2,,,,,,0,,1
2,.12.5,..8:22....1
11,1,3:9,1,,:6611
1.1.311...224,6,,0
33.2111.2..3322220000
6,,2111.4,66221,,3
2:722.1
2.122.222,24442233
1.1.5...111...5...
"1,233.0,,,0:8.1.11
1.,41...1.6:9.001..
1,21,,2223 1,7,
1,,,,.2.1,,,,,3111
2,2,222,
22,41,3,,,4333111
15,1,1:8,,,4:1333
1,,7,133, 1,8,1,8'
22,113;4,2,0,220022
1.2.1.6.1.113...44
1,660.
1,,2,.4
1,11,,1,,,,33 3,3,,,,
1,1,3,,,,11113332,
4,,1111
22221110,,,4,,,,24
"5,111:0.,,,1122,33
3,
1,10,0000
3,111,,,,,111161111
42,2222,,,,11220...
2,22,228,1,
3.1.1...........22
5,,,,11,,, 11,22.661111,,44.
2.122,,,.1113,,..2.
11,,,2,,,,,488..
4,,,4,3,,,,, 2,2,2,11
411,2116,,,2266.111
1,,2,2,,22,22...2
122.1.1.11..111..8.
22.22112...22222,44
41,7,:20,,,04441,1"
1,,1,61
5,
3,1,11,,,,11,,,,2
1,755.1
1,,22:2,0,11141122
2,
3,3,11
2,,0
1,1144,2,11,1122
22,1223,,,,1.5.008"
1,,1,,3,,,111,,,
1,3,119.8,11,,,117
2,,1,,22.222114,,22
1,4.2221..22222..0
2...222.....222....
1.3.133...:10.1.77
112,422,0,22224400
1,,1,,,,,,,,,,,,,,
1,,1,79
0,,,,,1,,,
12.22212.2.110.22..
22.22.1.2.222233000
2...22............
2,,22.22,,,
"1,42,,,,,,444111
1,,1,,2,,,,,,8,,,4
2,222,3,
11.51::3.,,18331122
1,14:3,3,112,,2
2,2,2:411,022.0.3
4,122:9.0.112222338
"1,13::,0,,2004430
2,221121
4,1,1:1,4,229,3337
3,2,122,..,1122,3
2,,3,,,,,,,,34,,,2
2,00
2,111:2,,,00223311
221.222..1.1.......
22.12222.411:226666
11,13:22,,1117.33
2.251.2
1..1..6.....11....
122,,,
2,,2,22
2,2,,,,,,,
2,1.1:200000331111
222.2200,.331188..1
4,1.1.1,,,,,,2,,88
11,1,3:6,1111
4,1,21166.0311.4
1,33:440,,2:4222.0
2.12221..333222000
4.122.2..
1::32.1.
"1,14:2.0,0:300..11
22.12222222::400.111
10,
1,2,422..11110.00
22,1,23,,4,000000
11,4,22,1,11144,
11.2.200.1.1:2220.1
22221.1..
7,11
1,1.1:1...111....1
1,,.2.3.1.2.7.1..
2,,22244
",,1.333,4,000..22
4,1,111,,,661111.55
2,221133220011.0.0
1.111110.10,:11..4.
13..111...0:222222
2,
2,1,4:2.2.151.00
21,1,5:0.2.9:1....11
5,11119,,,4411100
2,,1,,2,,,,2661111
1.222.00..222233.11
",3.1.3.0.2.77111
1,,23.11.0,334,,3,0
2,222233
2.222...
21,4,131.,,11,,11 1.0
11,2.133.,,10:13...
3,9,3,
2,1.2.3.2.133.0..
11.2.200.1.0:003..
11.3.11..1.,332..1
1.3.1.7.1.000,0.0
1..33:2.5.0:3.1.22
2.2.1..2....411..22
12.12.20...1111.222
"1,41::6004:002222
2,6,2.0
3,1.3114..122.4.0
11,15.2...11666..0
22.12:1...::3...00
1,2,2:4.0.3300...2
8,0,11.8
1.222.2.1..22234.03
1..22..
1..22.10.,,2.330.1
1,2,3.2
22.3.:1..,22112.22
"1,1333,7.0:881111
22,0.11122,0000..11
1,3.7,33,,3311..
1,22220.3.224440333
22.13.34.2.,::155.11
3.1.1..7.3311.000
21,42222.2.01441..
122.6,0
1.:0..1....0000..1"
1,
2,222,2..
3.3.1.14.3.11..222
11.11.111.:1111.
5,
1,
1,3,1,7.1,,44,1133
1,2,4.1
2...1...2
1..2..6.....0..1.222
6,
2.1.1.1..11...111.
11....11112::110333
221.2....0,332.11
1..12:1...0::4.0222
1,,,,:3,1,,,11, 2,111,6
4,1.111,,,,2266668
1.3.1.0.0,,1,,.1.2.
12.2222..111.222..222
50
1,2.1,1,,,44,2211
1,,,0,4"
1..11.1..1.911...7.
2.222.23.0222111.2
2,,4111,11661122.30
5,1.111,,,511.1.555 1.
22,2222,,233,222333
22:12:4222.::222223
11,411:4
1.5
1.3.115,0.551,1..0
3,,11,,,,4,6,111,
18.12222.4.00022244
"...2.2.2..00.6..1
13.2.110.2.8222133,,
11,3111,,,,,,112,33
11,18,4,,.2,,55..9
12,,,.11,,,,,2,221111
22,22233,,,,,002233,
1,,,3,,8,,,,,7,..,

---

Published as a conference paper at ICLR 2026
3 -.. Helper Functions
def qusttitiestessstti
Dictistr, intJ: "1,2,0" into dictionary.
"FFarses quantity return (iten: int(e) for iten, a in zip(ITENS, q,str, (atn.split(")))
def -oorrat.quar tustttieesssetnttteeee Dict(str, intl) "1,2,8"." "*Forrats quantity dictionnrry return Join(stt (wantrrtauutttssgeetttte 83) for 1ton ITEM))
def cecrenttttttt
offering_player- offered quantstees: Dict[str, intl) List(Dict[atr, int]l: agreement structure shares IO. ())
"*Crrettes
accepted offer.
shares(arTering_piae player] = orfrre__uantitie
offering. priiing.aayerr....
player gets the
pool.
Marresothher_pappe
iten: statel pool' [[tter] offered quantities (et(n,en, 8) for ites in ITENS
shares
-.. Core API Functions
aooly astaons State, action: Action)
"Ratuums the new state after an action has been taken."
new_state copy deepcopy(state) player_id get_ourrent_playercee
if player_Ie =: +1: Chance player sets up the game.
pool_str, ve,atr, vl_str action aplin((:)) : nea_statel (potel '] parse_quanittess
nex_statel player_ S_values'] pppaaaeeeeeeeee garse_quntiiiiesree _st_str)
nea_statel paaaeeppppe___ae_ea walluess]] sarre..uuntiiiiis(tLssrr
nex_statel" "ustatte"crrret_ppoe "urreett_paayer"
If "agrees" In action:
player agrees the 1ast offer, anding the "ast_aafrrppplyer player"), J, Last_ofrar( quantities last_offer new_statel'ofFee historyy [[-1]] new "agrtertgreeeeet" nea_statel mma_statt[cuurrett. cuureett_paayer" None Mark as a terminal state.
Last_affer[
"offers"
player nakes offer.
nex_state (ouuttunuuuuua ]
quantities www..qunniiieeeattiiiiii split(( offers ()(:3)
nea_offer
"nun turn's nev_statel trun_turss'] '].
player' : player_id,
quancities" quantittes
nea_statel offer_history" 1. .ppent(nev_offer)
If turn limit reached, this offer becores forced, zero-revard new_state currsnttppaayer" None
new,statef" agneementt' ] _creeteeaggeee prrsst..greeeenteeew.sttt player_id, qunticies) else:
statet current, player' str(1 player_id)
player.
return new_state
def www.soreent_llyeeetttteete State)
"Beturs current player, with chance and for terninal player state get(" current._paaeeeeeee
player "chance",
player is None:
return -4
(nt(player)
def www.ayeeeerrrellayrrr... int) -o str: "Retumm the name of the player, with "chance' for -1, and 'terminal' for -4, if player_id return 'chance'
player -d: return 'teminal' (trapeeee__1d)
def get_renands State) list[flost]:
"Retums the revards per player fron their last action
Rewards
for voluntary
forced apreement
for 1f not scate.gete agresnent') or statel nur, curns' TURNS: return [0.0] NUN
turn (WRL)U results
resords
for in rongeoOM. player_values values state "ppayeer_ [[]_vall
player_share share statef "spreement
player.
51

---

Published as a conference paper at ICLR 2026
rexard han(player_share get(Aton, 0) player_values, get(iton, 6) ITEN5) rewares, (pptenr))) return rewords
get Lepal, actions(state: State) List[Action];
"Retums all legal actions for the current player." player, id get_currentt.or player(state)
player_ie Lo
Chance player
return ACTCEEACTIOOS
if player_ 10 return (]
Terninal state
[]
pool statel pool"]
Generate possible "effer" actions iterating through iten conbinations. 1TEYS]
ranges trange(pocl gat 8) for carbo in itertools product(vranges).
quantities (item: count for 4ten, count in ZiD(ITENS, conbo)) actions appond(!" plaayer (player_id) offers [_forma__qantitiesses (tt..autttiieesquniiitee))))
"agree" is legal move if at least one offer has been mode by the opponent.
if statel offer_history actions append(("playee (olayer_id] agrees?)
return actions
def www._cbeeraaiinn(ttin
MisttPaayerdseervattoo
"Ratums observation for each player, containingg public private informttion."" base obs
'current_player' statel "current_player",,
statel' paaal",
turns': statel turns "..
'agroonent's statel agromment" 'J,
Lentenniall poL_ourrent_plaeer(ttass
observations U
Deternine pravious_offar based
porninal_prevoos_.tot
If fa_terninal:
In terminal state, the "pravious affer" is the one that on the table second-to-last accepted offer history.
correspords
if lan(statel offer Jisttrrrttrrr" 1:
prrnisl_previoos_oFFe statel state oiferriisttrry []=2]
range PLAYERS):
abs base_obs, copy()
absl values 1 absß ry..paaye_idd'1
if terrini::::
obs[' previeus.offer") hrminoll.revioos.offe
active gane, previous offer made by opponent.. opponent_.id = obat prevei oo next((
reversed[stacel offfer eiferry }> offer[ (playee"""""" apzonent_id
), None)
observations, append(obs)
def reserple history(obs, action, history: Jist/ubleePlyyedderrvttt Action None]], player id: int) >> list[Action]:
** "*Stochostieally samples of potential histories given single player's perspective,"
abk_acttonuhittrrEEee
Opponent's is values are private and must be sorpled randonly to create possible history,
ooponent id player_id
poonent_vvllues (iten: randint(e, NOCITELVALLE
ITEMS)
values ((), ())
values(player. id]] first_obs['vvlurs values[sponent_idg] ssponent_valuus
Reconstruct the 'chance' action that started the gane.
fornas_.aantitiessffire pooossoooosllcs st..forrt.ggattiiieevllllee wtl.forrat.gunntttieevaaaaa
Collect all know offers Cown and opponent s) fron the abservation history.
crown_offers []
set()
obs, action obs acttion history:
Opponent's offers are seen in the "previous.offer" field.
prev.otfer ots, get( previous_orffer"
if offer and offeri nun.turn") turn'] J tumms offers aspendCorev.offer
52

---

Published as a conference paper at ICLR 2026
corns add(prev (ottirreeoooerrnnoee tunnnt
Own offers are reconstructed fron the actions taken.
"offers' action increments
for anexee state's observation.
The offer itself is recorded with this new turn number.
turn obsl" nun, turns ++ if
quantities spparssesoe tissssouantttessccttionn splitç' offers ')113> knswn_offers. append(t turtur'': turn, "player's player_id, 'quntiicie'' quantities))
seen_turns (de(tur))
Reconstruct sequence actions in chronological order.
known, offers laey x: x( nuununtturn")) resarpled_actee actions ceannce_action]
resarpled,acttenn acctions extends
("player player")) affers pttaappuuttttesss""pttititt }>}" for in offwrs
action
an (e.8-r "agrees").
final action action, hist] if rinal_action and 'affers' not in rssaalaction:
resarpled, appen((rsnalacction)
return restrpledLactions
value, fusction(state: dictfstr, AnyJ, player_id: inc) float:
"""sturns the value estinate for player_id in state.
tersinal
Function
play
Function should of the player with given player_id.
a value estinate that reflect vinning potencial
Handle Tenainal get_ourrent_playe pplayer(statte))
if player_le 3 or player_Id >o NUM PLAYERS: For non-players Tike "terminal" (-4), return 8.8
active players, return (statt)(ttaa]pppaa
actual reward achioved.
Kon-Terninal
# 2. Basic Information
vyvvaluss state get(F'player Sesbyer_(plyyr_.dd...ve This happen initial "chance" values gned. my_vals return 0.0
opporent..id player_id pool statel'pool" offer_histon stateC Cofferofferrreee Saaeoooff____ttrrrre _historyy]
purrent.tum.playe pec.corren..laaer(ssttte)
Caloulate Total Potential Value
The raximum value this player could get if they received all ittes. total total_ry. vy..ulue sun(poo), get(iten, 8) my.values, (ette 9) for iten in ITEMS)
total_.y.vvluu return 0.0
nothing the pool is valumble expected outcome in 8.
Define Baseline "Tair" Expectation sixple assueption fair_value, www.waleessttatt....... vy- 2.8.
player
about
5. Core Heuristic Logic based current negotiation status heuriatic_value fair_value_eetinnn Befault split expectation
if not offer_histor::
First offers The best estimate fair solit. heuristic.value fair_value_estinett
else:
offer offer history[-1]
current.turn.player
player_id:
ny turn
if last offer player'] apconent_id id:
Opponent mode the last offer. can agree or counter.
Calculate value their
their, propor_dopseddee ty.there.iif.c aa_gree
Last_offer[ cvantsties']
sten: pool, get(iten, 3) their_prososed__e share srapoooo_saaregeetttteeee. 3)
iten ITEMS
value on table sunt
my share if, agree get(iter, 3) values (iten, 8)
iten ITEMS
My position's 's value the better of what can get now versus general expectation fron contirued negotiation. heuristic value naxivaluee table, fair, fair.value. estinate) player_id
Last_offer['
53

---

Published as a conference paper at ICLR 2026
offer was Ny turn again reans ny previous offer was implicitly rejected.
opponent didn't agres.
fall back to the baseline expectation.
heuristic_value
current, turnuplayer osporent_idd
's oxponent's 5 turn. They are considering my last offer. if ofst_ player:] player_1e id:
ry.propeeed share last_offerl "sstooferr(auunticiees)
state
hoping they accept.
value offfoooeeer = sung
my ww..propoeeddeo share get(iten, 3) my.values get(inen, 8)
iten ITEMS
heuristic value volue_of.mm_pff my.
Pressure Discount
out,
getting from agreement Increases.
This discounts the potential ruture value accordingly.
if statel nun, turns" ] se NXX TURNS:
be Forced
action.
turns turnsleffe left MAX, TURNS statel mun, turns']
sort Factor nakes the discount less severa In early turns. (turns_ieft
return (heur pressura_Factor)
54

---

Published as a conference paper at ICLR 2026
J.3 BARGAINING (IMPERFECT INFORMATION, CLOSED DECK)
import copy
Seport randon
from typing import Any, Bict, list, Tuple, Optional
definitions
State Dictlstr, AnyJ
Playerbbervation DietCstr, Anyl
Bane constants
MUN_PLAYERS AYERS =22
ITENS sortod(E'X', 'Y', (2'3)
PODI LOILXALUFS range(),
ITRNLVLLES range(8,
possible gies sceariis
_CHANCE_DITCOO
.chance_rng randon. Randon(e)
for in (2n):
pool (iten: _chance_mm. choice(PoOL VALUES)
LTEMS)
po.values (iten: _charce_rng choice((TEX, VALUE) for iten ITE6)
pl_values (item: _chenc_rrng rng. choioe([TFX UFS) for itwn in TTPHS)
pool_str (o)(("(()((V))
k, sorted iters)))
pl.sr (()(())))=(()" for k, In sortad(pi_values (ttea()))
polatr "oin
(k)=((" for k. in sorted(pt.xalues. iters())
"pool=(pool_str) (ot__oollsssppsstr))) (p9_str):p1 (ww.strr)pp..vbbess0......
def pooree_eee offer action (action:::o Action) is Tuplefint, Diet[str, int]]:
"Prres an offer action string into player 10 and quntities."
action. split()
player_id int(ports(1])
quintities (isen: int(e) for iten, in Zip(ITEMS, parta[i] aplitC ">)
return player id. quantities
wtt.aallaaaeeerrrrtttttttee resard(tundle: Diccistr, int], values: Dict(str, int])
"*"alculates the total value of a bundle of itens for a player. return sun(bundle get(iten, 8) * values get(iten, for iten in ITES)
def reconstruct, off.eetttfffeeeeeeeeee actionnfti:: Dictfstr, Anyl) Action:
"Reconstructs an offer action string from an offer dictionary. cuantities, str "_cdi((strooffer coofferr'quanttt quantities g.getiieen, 8)) for iten in ITEMS)
plpaayer (offerl player 1} offers (quantities_str)
def aoply (state; State, action: Action) >> State: "RRetunns the new state ofter action has been taken, nev_state copy. despcay((state)
state get( current.ppayee >> 'chance': :
Initialize the gane state froe the chance rode action.
action. oction.split(": split(":')
(p.split('="((a) new statttt"paasseee a.values") 1] (p. politt"=')ee) (nt (o for p parts()) splitt spptss)) (aautt":"00 (DD1) splitt -
nex, stale("pool"
instp <p
parts(e) pautert(( (olit(".)))
nex, statel "zaate(playerr.. pplayer, veluees") ] = (p. co.splii("" [00]: int solit('=)) for parts[2]. split( (1)()) split(.
"raate"curront_playee
return state
'sgrees" in action:
reached.
terrinal.
][-1] offerer_id id Last_afferl "player")
new.statel'offen
offerer, bundle offtet wunttities)
accaptor_
pool'][ltees offferer_budlle gat(iton, for ITEMS)
agreement (O. ( sgrasonttorrorrr.tn id] (iferer_bundla offerer_ie]
new aagrreennnn""t agreerent www.sttat"'cureenn "offers' "cuureet__paayer"" None
is nade. Incresent turn
player.
player id, quantities urrre_offer_actionssctta
rwn_stateE'nin turts
turns"], 'player's player_id, 'quantities's cuantities) offer histtrrrhhstry' appen(ooferr)
nes, www.state""ofrrr
If sen_state (runtturss]] >e MAX_TURNS:
cuurrentpplayer"
gase limit reached.
purrent nww.sttae(cuureet..laag
new nov_statel
player_id
def get_curreet__lle paart_ppayerrsttteee State) >> fnt:
"**tuuurns current, player, with -1 for chance and -4 for terninal."
get( current_player" "chance';
return
55

---

Published as a conference paper at ICLR 2026
state. get(teuureeen "urrent_playeer) return -4
statel agreement") statel "erteennnuutuaaee turns") MAX_
return state!" "turen"cureent. "current, player).
get pltyer res_player_nareellayer
int) str:
"Returns the name of the player, with "chance" for -1, and "terminal" for -4, return (-1: "chance', -4: "terminna'' ). -ett(ppayye__i,, str (player_1d))
ret_renardsstttte Stase) 1sstErloat]:
"*"Rsturns rewards. Rexards are 0 if the gare ends due the turn linit." if statel 'un, turns >e NX. TURNS or not statel agresment"): return NLM.PLAYERS
po. po.reward _calcullte..rrrcc... www.eeeeeeeeeee. (ewar(ssttte"" aewerdsttaeeaaarreeenn agreerent [[0], statel player 3 voluuess
pl pl_reward _cc.lcllateeeccccc roward(statel" 'agreement ][],, statel player_1.values
[Ploattpa_ roward)))) [leat(pl_reaar))]
def get Legal, actions(state: State) -> List[Action]:
"""eturns
in
player ce__current__payee (state)
player = -4:
return []
return
actions []
(gpand(["ppaaee (playor) agrees")
# Generate all possible offer conbinations based the itex pool. "ppoo""
quantity_ rarges (ranse(poc-.get(tte 03
STEMS)
for quantitles in startools, (-quettttuutttyyyarree
e.str 5." "'Jooi(saa(sstr quantitias))
append(f player (player) offera (g_str)")
return actions
def in...bbervatians(ttat::
Iist(Paaerrbaervattinn
"etturns abservation For player.
abservations []
player_at_turr eet_current_paayrrttat.
(player_at__ie)
for In range (AALLLVVVEE):::
previous_offer None
agraimint, "previous offer" is accepted lan(statel' innssaaat"oofrr_jhiste offrrrrhttttrr"
is_terninal statel's agrosment
previcus_offer = statel offer ferttrrttrrr"" ( 1[-23
alif Saftel""ffe__hstteee 3:
previcus.offer statel loffer historiistrrrr [[-13
obs
ho.pplaer_idd;
valuss's state[f" "lauer_
turns's statel hunntturs'' [.
"ggreesent's statel agreeeent
previous.offer previous
pllyeettppayeer' str(player.at_.urr)) if player_at, be else Mone,
observations append(obs) sppeellcssssspeemlllss abservations
def reserple history(obs,attion action, history: List/uble(Plyyedderrvttn Action None]], player_id: int, temminalt bool) >> players.
IisttAttion]:
"* SStochstteeally
man patantial histories
Reconstruct and yield the chance action.
first_obs obs.action.histrr
uosonent.valuee ('X:: 1V's 12's
fixed sosoment values for reprooucibility.
p.vals [O, 03
p.vals[player (i] firstt (values)] p.vals(1 player_id] opponent_vvluus
stol "_sooin(""(K)=t" for k, v sorted(firstt ebsl "oel").itens())
pa.str "oin(f" (k)+()) for k, in sortedip valsfee. ittes())) sortedip vals[]]]l]sooeeee (pol_spoolssrrso [btp::rrrr.e.vvlessspp..... wal.ee:cpl..sstt)e). str)pppppppp.....ppppp valuusss. (pt_sr"""
Jonn for
# 2. Reconstruct the interleaved gare actions fron the player's perspective. opstoeppenttt_uu tut.riidttttt lystsst.action
for obs, action in ots_action_hiseee history:
if action:
if obs "et""rrrious.offe""
offer peee'ppoooou_orf _offerr
Only vield offers that haven't been ylelded aveid duplication.
offert "arclpp player_id offer(' mux,tuun I Last.opponent tieldee
56

---

Published as a conference paper at ICLR 2026
yield wwwconstructoofffee..... acterraactoo
Last wwwpooeennttuunnncileioo effer["nus, ('nunturrn)
action:
action
if "grees" in action:
return
Deduce
game ended agresment initiated by player.
last_ 1s, terninal:
last, obs, dast_action = obs, www..cction.histrrri.
and last_obsl aagssstsrreeeett
agreerent Last_ obsl" agreenent
my qusttttuasities _pwpse.ofeeeraaeeee prrseoofer_.ccion(ss sttrratttoony.sss..ttttt last.acioon
Case Opponent agreed
last offer. bundle
ratches last offer.
wy_lass_guantities agraasentpllaerr_
yield Ipplayer (1 player_so 1d) agrees"
Case B: Opponent rade counter-offer, which I would have implicitly agreed to.
opponent_
player_1a
opponent, bundle = gareenen:[ogpoeeen id] quantitles_se
je-jn(strr(pppennttt (oppanent_bunnae got(:ter, 93) for ster in ITENS)
yield I'player [opponent_id) ofters (Quantities_str"""
yield f"player (player_idj sgrees"
typing import Any, Bict,
def valuee_ Futctionsstttte: diet[str, AnyJ, player _id: int) Float:
"*"Roturns value
player_id
For tersinal states the function returns the true raturn, For ongaing play the Function should return value estinate that reflect the winning potencial player given player_id.
a Gane constants and helper functions defiied in local scope for self-containment. ITRSS sO'ted(['X", 'Y', (2'3) 18 KULALLYEESS
def _ww.culateeeacccoo rewoulatt_rrvrreraaoao (bundll Dict[str, int], values: Dict[str, Int]) +> float:
** "Calouultes total value return sun(bundee, get(iten, 8) values, get(iten, for TTFMS)
player," w*e
"* ""ettorniness Salayer_intennnlsatttee
def curreett___oppee
Diet[str, Ary]) int: terninal.
currant_player state_dict. ww..ditt.ge got(' sut""curret_pplaeee
agreesent state_dict get ('agreement") turns state_dict (atlicttteettaaaaaaaaaee tot'nnn_turn'',
current_player is None or (agreement and Isinstance(agreeeeet list) and Jen (agreement)
iF 14_terninal:
puayent.player 'chance':
return -1 Chance node code
int(current_plaaee
DietEstr, Any]) List[flont:: "Calculatee reverds for all players in cerminal state. ***
Internal _tt.__eerdd..ineeeaattttttttt
state_dict pet('agreeeent')
get (et(aaatturns'' 8)
No reward if the gase ends due to turn linit or no agremeet is made, (isinstance(arreeeenn,, list) and len(sgremment)
TUCN NUMEPLAYERS
rbured calculate reward(ssreementio)) state state,dictf player 8 values)))
pl_reward www.ullteeeeeeeeeeeeeeeeeeeeeeee cewpartteeeerrddrreeeetttii siate,dict] playyr vvaluess
return [fleat(po_rr reward)) float(pl_reward)
Yain Function Logic begins
current.player_cod. wtt.corrent.plaerrinteeee///
landle Terminal States: Return
firal revard.
surrent.player_com 4::
if player 9: NCTS ray query the value for the tersinal node itself.
return 8.8
2. Handle Non-Terminal States: Return heuristic-based value estinste.
values state[f'player (player valuess state pool""
57

---

Published as a conference paper at ICLR 2026
offer vistory state. gec( hfttter. (3)
if leuuisstcc for the start of the gare (no offers yet).
assumption
player achieve half their value.
history:
my pocl valoe colluulate_reann(ppoo)) my.values)
return ny total value / 2.0
offt_ 11sto_
if Case At It's the opponent's is turn This neans rade the last offer. estirate
offer
potential, rerlects aspiration.
current, player_coce player_id:
my bundle in,ry, lest. offer last. offer" quantities')
my_value_of.eo offfrr _calculate, reward www.buuddee__neeeeee lastt ey.values)
return Float(ry www.vaallffrrpooeeee _t oy
a Case Bi It's my turn. The opponent made the last offer.
if My potential lies between what they offered and what I last asked for.
to re. bundle bundle to opponent last_affer] quantities ponlled_bundle_too toure
Calculate
achieve accepting.
pool gen, iten
offeres_.unnde, topoppeent
}
value_of. whwir_offerrto..oooo _oure __c.culaae_rewardccc.... reward[Lvplied bundie_too..ooo to vy_vale
my last
gauge ny aspiration
my aappirrtiion -1.0
for offer in reversed(offer
offarl'player') player_id:
ry burbuddeee_dn last offer offer['auant ['uantsttiss]]
ry sypasplration sealculaserraarddyyyymmu hindil__nnyyllosttofooo ny_values)
break
haven't offer yet, ay aspiration
baseline.
if ny vy_asotration tw__tatl__pol____ _oool_vuuee reterlatt_rwwrddpooll sy_values)
aypaspiiation ry_total_pool_yall
The heuristie is the niopaint between their offer and ay aspiration, representing a likely comprosise point. houristic_value value (value_of__teel tt__tetr_afre_tttt pefeertttreeeee sy.aspiration) 2.0
58


Reviews:
[{"review_id": "EXimSu5IF8", "reviewer": "Reviewer_Liqi", "summary": "This paper proposes a new method for gameplay agents, to use LLMs to generate a code world model, which serves as a verifiable simulation engine for the planning algorithms. The experiments evaluate the gameplay performance on 10 different games, outperforming or rivaling Gemini 2.5 Pro.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. To use the LLM to generate the code world model is a novel idea to represent a specific game in a verifiable manner.\n\n2. The work is solid. I also read the appendix of the paper. It provides all necessary details and examples.", "weaknesses": "1. The experiments are made on 10 distinct games, generalizing 4 to the other 6 games. It is hard for readers to assess the OOD generalizability of the method.\n\n2. One concern is what kind of games (e.g. poker-like games) can benefit from the proposed method. Can the method work for any types of games? The complexity to generate a code world for card games is somewhat low, for example, the code cannot be very long, so what about more complex games?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pLDCSMKrOx", "reviewer": "Reviewer_4ELJ", "summary": "The authors extend the Code World Model (CWM) framework by considering two-player games, performing value function code synthesis to improve player performance, introducing the concept of \"inference as code\" to enable state estimation in imperfect information games, and providing a learning algorithm (based on code-based autoencoders) to enable learning in the novel closed deck (strict partial observability) setting. Their results shows the superiority of this approach with respect to LLMs as policies on multiple perfect and imperfect information games, including newly created ones.", "questions": "There are many who have shown \"Code as policies\", working in different settings. Why is it better than just creating actions through LLMs?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Well written paper.\n- Very good related works section\n- Good amount of environments/games tested", "weaknesses": "- Less ablation studies performed\n- Very few baselines\n- Only Gemini 2.5 tested. Other LLMs would bring the variance that is needed to be demonstrated", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y4GdpBI6kA", "reviewer": "Reviewer_jLRx", "summary": "This paper proposes Code World Models (CWMs), using an LLM to translate natural-language rules plus a few example trajectories into executable Python code that implements a game’s world model (state transitions, legal moves, termination), along with synthesized heuristic value functions and inference functions for imperfect-information games. Experiments cover 5 perfect information games and 5 imperfect information games, and showing promising results while using Gemini 2.5 Pro as the LLM.", "questions": "1. In closed-deck learning, can you quantify how the learned state-space size correlates with performance? This corresponds to your hypothesis at Line 455 - 457.\n2. Have you tried iterative online refinement (updating the CWM during play), and if so, does it reduce reliance on high-quality initial trajectories? (You note it’s possible but skipped for efficiency.)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. \"data to code\" fashion creates a verifiable simulation of games, and both perfect and imperfect information games can be applied in this fashion.\n2. Two ways of synthesizing inference functions for imperfect information games to avoid the exponential cost is a valuable contribution. Both hidden history inference and hidden state inference are straightforward yet effective.", "weaknesses": "1. Synthesize quality relies on generated test cases over a limited amount of trajectories. When LLM failed to parse game rules, it might not build the code world model effectively. \n2. It seems like CWM performs worse than Random in the game of Gin rummy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YtjARRuqB3", "reviewer": "Reviewer_2crY", "summary": "This paper introduces a very interesting .", "questions": "In addition from my questions above, I'd like to know if it is possible to know what would be the behaviour of a reasoning model alone in this agentic workflow scenario. This _is_ important since a comparison between a Markov-like optimised prompt/workflow versus the effect of a (behind-an-API) reasoning model _versus_ a RLM using the Markov-optimised approaches would allow an ablation on whether it is the prompt, the workflow, or the model that provides the contributions.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "This is a thorough, well-founded work. It is well-motivated and it starts with good rigour and a clever approach to what will eventually become, in my opinion, a large open problem in agentic workflows.", "weaknesses": "While the motivations/contributions are strong, the paper's methodology is not there yet. It has some weaknesses that (unfortunately) do not make it, in my _personal_ opinion, ready for publication. \n\n1. The paper's writing is inconsistent across the work. Tonal shifts occur often (e.g., L431 'you can check it in...')\n2. Only one (closed-source) baseline.\n    - While not strictly needed, an ablation on prompts/models would be helpful to ascertain _why_ the results are the way they are.\n3. Paper structuring and presentation needs work. This is different from the paper's writing: overall the presentation / scientific methodology needs to be improved.\n\nHere's how the paper can be improved:\n1. Make the writing consistent--proof-read and ensure that it adjusts to the expected tonality of a scientific article. \n    - Ensure descriptions of tables/figures are well-done. E.g., what is '# LLM Calls' in Table 2? If it is 'number of LLM calls', why is it a float? Is it an average then?\n    - The related work section needs _a bit_ (not a lot) of work. Some known works in both applying Markov processes or getting LLMs to play videogames are missing. I would also suggest adding a small explanation on what they did.\n    - Make sure your paper is proofread. For example, L308 introduces 'OOD' before defining what it is. The sentence in L86 does not need parentheses. This indicates a lack of proofreading.\n    - Definitions are important. Gemini is not indicated as a reasoning model: since the behaviour of an LLM (next-token predictor) and an RLM (baked-in CoT) are quite distinct, referring to Gemini (an RLM) as an LLM does require some clarification.\n2. More open baselines/agents will be beneficial for the robustness, soundness, and longevity of the work. These three are _musts_ for contributions to any conference, let alone ICLR. \n    - Related: the code would be better put in a repository. This will avoid presentation issues like those in App. I.2\n3. On scientific writing:\n    1. Scientific writing follows a very specific template:\n        1. Results/Experiments contain the outcomes of your evaluation of the hypothesis. These should be supported with numerical evidence. Opinions and interpretations of the results (like 5.1.1) are for the discussion.\n        2. Discussion contains the discussion _of the results_, not of your paper. \n        3. Conclusion (which is missing) allows you to draw a conclusion (or interpretation) of your hypothesis based on data.\n    2. Significant digits must be consistent and should make sense (why is accuracy in T2 reported to five significant digits? Is such precision truly needed?).\n    3. Skipping the definition of imperfect info games because they are 'tricky' _would_ make sense if it weren't for the fact that (a) the experimental work does rely on imperfect information; and (b) the appendix to which the reader is referred does not contain a formal definition.\n\nI genuinely think this could be a very strong contribution, but needs more work than might be feasible for this submission.\n\nMinor, and not something that influenced my review: the lack of a reproducibility statement plus (1) the fact that all the code is in an appendix; and (2) the baselines are closed-source, do not indicate good, open science practices. I would encourage the authors to rethink this and add such a statement. Again, not something that can/will impact my assessment of the paper, but always nice to have.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)