Look-ahead Reasoning with a Learned Model in Imperfect Information Games
Authors: Ondrej Kubicek
Viliam Lisý
Venue: ICLR 2026 Poster (2026)
Keywords: Imperfect Information Games
Two-player Zero-sum Games
Reinforcement Learning
Learned Game Models
Game Abstraction
Look-ahead Search
Value Function
Continual Resolving
MuZero
Primary area: reinforcement learning

Abstract: Test-time reasoning significantly enhances pre-trained AI agents’ performance. However, it requires an explicit environment model, often unavailable or overly complex in real-world scenarios. While MuZero enables effective model learning for search in perfect information games, extending this paradigm to imperfect information games presents substantial challenges due to more nuanced look-ahead reasoning techniques and large number of states relevant for individual decisions. This paper introduces an algorithm LAMIR that learns an abstracted model of an imperfect information game directly from the agent-environment interaction. During test time, this trained model is used to perform look-ahead reasoning. The learned abstraction limits the size of each subgame to a manageable size, making theoretically principled look-ahead reasoning tractable even in games where previous methods could not scale. We empirically demonstrate that with sufficient
capacity, LAMIR learns the exact underlying game structure, and with limited capacity, it still learns a valuable abstraction, which improves game playing performance of the pre-trained agents even in large games.

Published as a conference paper at ICLR 2026
LOOK-AHEAD REASONING WITH A LEARNED MODEL IN IMPERFECT INFORMATION GAMES
Ondřej Kubíčck
Viliam Lisý
Artificial Intelligence Center, FEE
Artificial Intelligence Center, FEE
Czech Technical University in Prague
Czech Technical University in Prague
Czech Republic
Czech Republic vi viliam. wiw.m..aaaaggtss......ommmm
kubicon3e fel. cvut.cz
ABSTRACT
Test-time reasoning significantly enhances pre-trained AI agents' performance. However, requires an explicit environment model, often unavailable or overly complex in real-world scenarios. While MuZero enables effective model learning for search in perfect information games, extending this paradigm to imperfect in- formation games presents substantial challenges due to more nuanced look-ahead reasoning techniques and large number of states relevant for individual decisions. This paper introduces algorithm LAMIR that learns an abstracted model of an imperfect information game directly from the agent-enviromment interaction. During test time, this trained model is used to perform look-ahead reasoning. The learned abstraction limits the size each subgame to a manageable size, making theoretically principled look-ahead reasoning tractable even games where pre- vious methods could not scale. We empirically demonstrate that with sufficient capacity, LAMIR learns the exact underlying game structure, and with limited capacity, it still learns a valuable abstraction, which improves game playing per- formance of the pre-trained agents even in large games.
INTRODUCTION
Strategic reasoning and planning are key components of human intelligence, encompassing our ability to reason about possible outcomes of actions complex situations, often with incomplete in- formation and uncertain consequences. Although humans navigate such decision- maaiiong naturally, replicating this process in artificial intelligence remains a fundamental challenge. Games, with their well-defined rules and yet complex strategic landscapes, evaluating AI planning and reasoning methods (Mnih et al.) 2015 Silver et al.) 2018 Perolat et al., 2022 Schultz al. 2025).
ideal testbeds for developing and
In perfect information games like Chess, Go or Shogi, look-ahead search algorithms as Minimax and Monte Carlo Tree Search (MCTS) have achieved superhuman performance by systematically exploring future states (Russell & Norvig 2003) [iiver et al. 2018). These methods traditionally rely on access game rules implement state transitions the search. MuZero demonstrated that an agent can learn model of the environment dynamics implicitly through interaction and use this learned model to perform MCTS planning, removing the dependency on explicit implementation of
the rules (Schrittwieser et al. 2020
Not requiring explicit pre-pie programmed representation of the environment greatly expands applica- bility of Al methods. A method that does not require explicit rules representation can be applied, for example, to create an opponent in proprietary video game without access its source code; to create AI opponents for large database of games for online game playing platform, where programming a suitable representation for each of them would be prohibitively expensive; or a game design setting, where the game is repetitively modified without the need for programmer to reflect the changes in the implementation.
However, extending the model learning paradigm imperfect information games such as Poker or Stratego presents fundamental difficulties. Since players the game, theoretically sound look-ahead reasoning methods need to reason about the distribution
lack complete knowledge the state of

---

Published as a conference paper at ICLR 2026
of all possible hidden states consistent with shared knowledge (Kovañık et al. 2023). which differs from the MCTS used by MuZero.
Our work aims to enable look-ahead reasoning in two-player imperfect information games us- ing a learned abstract model, thereby eliminating the need for explicit game rules and also enabling look-ahead reasoning in parts of the game intractable without abstraction. Following the approach of |Schrittwieser et al. (2020), we focus only on games without chance events, which is a large class that includes commonly used benchmarks like Dark Chess, Stratego, Battleship Im- perfect Information Goofspiel. This allows us to tackle the unique difficulties of learning an effective abstraction for imperfect information without conflating it with the separate challenges introduced by chance events. Our contributions are: We identify the necessary components that a learned model must capture to facilitate look-ahead reasoning under imperfect information. We introduce a training procedure to learn these components from sampled game trajectories. We demonstrate how tractably small, domain-independent abstractions can be learned concurrently with the model. Finally, we introduce way to conduct look-ahead reasoning with the learned model.
Our empirical evaluation shows that small games, given sufficient capacity, the strategies pro- duced by look-ahead reasoning are less exploitable than those of concurrently trained Regularized Nash Dynamics (RNaD). Furthermore, in large games with shared knowledge consistent with over 10 states for some decisions, the proposed ahead reasoning is still applicable in all decision points and significantly improves over RNaD, reaching up to 80% win rate in head-to-head play.
1.1 RELATED WORK
Direct Policy Optimization One approach for computing strategies in large imperfect informa- tion games stores the strategy implicitly in neural network weights and directly optimizes this policy based on self-play traces. These methods include: policy-gradient gradient algorithms with reward regular-
ization, like Regularized Nash Dynamics (Perolat et al., [02T]] 2022, Sokota et al., 2023] Masaka
et al., 2025 training networks to approximate CFR results, like Deep CFR or DREAM (Brown
eet al., 2019 2020 Steinberger et al. 22200]; or iteratively training best responses to growing strategy
pools, like PSRO (Lanctot et al.) (017;] Vinyals et al. (019)) Critically, these approaches rely solely on the trained actor during gameplay and cannot refine decisions with additional test-time com- putation. Pre-trained agents without test-time reasoning usually very exploitable (Wang et al, 2023
Lisy & Bowling 2017), and adding test-time reasoning greatly improves their capabilities in games (Silvrr et al.) 2016]] Kubíček al., 2024), and beyond (Snell et al. 2024). This paper enables adding test-time reasoning to policies created by direct policy optimization algorithms.
Look-ahead reasoning Reasoning algorithms in imperfect information games, such as Counter- factual Regret Minimization (Zinkevich et al. 2007). iteratively improve player's policies by sys- tematically iterating over all possible future action sequences in all possible (unobserved) states of the game. In large games, this requires either domain-specific abstractions, like Libratus (Brown) & Sandholm, 2018) or depth-limited reasoning, like DeepStack, ReBeL, Student of Games, SePoT (Moravčik et al. 2017, Brown et al. 2020 Schmiid et al.) 2023 Kubiček et al., 2024). In either case, all these algorithms require explicit implementation of game rules construct game trees and manage belief states. It limits their applicability when exact rules are unavailable, computation- ally prohibitive or the amount of possible hidden states is too huge. Knowledge-limite limited subgame solving (Zhang& & Sandholm) 2021; 2026 Liu et al. (2023) can reduce the complexity, but even this reduced state space remains intractable in the games we study here. In contrast, our approach does not require explicit game rules and automatically learns tractably small abstraction of the game just from full traces of game play.
Model learning In single-player settings, Dreamer, TD-MPC and subsequent works showed that learning models and generating artificial traces can match purely model-free approaches (Hafner
ct al. 2020 2021 2025 Hansen et al. 2022 2024). MuZero demonstrated similar results in perfect information games, using learned models for reasoning during gameplay (Schrittwieser et al. 2020 Antonoglou et al. 2022). Our work extends these approaches to imperfect information games with- out chance. The model we learn is similar to TD-MPC (Hansen et al. (002)). but it trains additional components to account for the abstractions and more nuanced imperfect information search.
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND
We define two-player zero-sum simultancous move game as 9 = (N, W, w A, 0) (Kovařík a 2 where N = [1, 2] are the players, W the set of world states in the game and " E W is the initial world state. A = A, is the set of joint actions.. We use A(w) S A to denote the set of joint legal actions in world state w. T : W x A + W is the transition function and R W x A + I is the reward function, which corresponds to the reward of player 1. We use Rí a) = R. R(w, a) and R2(w,a) a) = - Rww, a) as rewards for player and respectively. O w x A x W + O is the observation function. O = Oo 01 Og is the set of joint public and private observations. O can be factored as O = (Oo, O O2), where OD is a public part of observation and 11, O2 are private parts of observations for each player. Even though we focus on simultaneous-move games, this does not limit the generality, since any sequential-move game can be converted into a simultaneous-mon move game by adding fictitious moves for the non-acting player in each decision node.
History h wo aa" .... a' E (W.))'W is finite sequence of world states and actions, which starts in the initial world state w = INI and for each timestep E {O.... ,/-13 holds a E (ww)) and w=(1 T(w", a'). H is the set of all possible histories within the game. We use h E h to denote that h' contains h as a prefix. We will use INI = w wNNT to denote the initial history. Each history h ends with some world state w'. We will sometimes use history h in the game functions instead of the world state w'. For example A(h) := A(w") corresponds to the set of joint legal actions in the final world state of the history. Each player does not observe the whole world state at each timestep, but only observes public observations On and its private observations O, This means that the player may not be able to distinguish between several different histories. We will use s, E S, to denote the set of all histories consistent with the observations of the player i, which we will call the information set. S, is the set of all the information sets of the player i. (,)) is the information set that corresponds to history h. Similarly, the public state So E So is information of an external player, which does not have private observations, so it contains all the histories consistent with public observations. Each public state contains or more information sets for each player.
Strategy of player i is a function T; : S: + AA; that maps each information set to a proba- bility distribution over the actions. We will sometimes use (i(si, az) as probability that player will play a, in information set 8; if it follows the strategy πद. π = (T,, T2) is a joint strat- egy profile of both players. If h _ h', then the reach probability of reaching history h' from history h under strategy profile π is (*(('(h) = II>" HEN (s; a.). We also use p Any reach probability can be factored into the contribution by each player Px (h) = IlEE Pt (h(l'). Expected utility of a history h if all players follow strategy profile π is uz (h) = EngN' PK (M(h), a) ITEN m((s((h'), a:).
pa (h) == pa (MI)NNIT))
Best response against strategy Ti is strategy π =BR E BR-,, which maximizes the opponent's
utility u u_i (f. (IANIT)= 2 u-i ( (NNNTT) for any T'zG. We use a -i here to symbolize the other player than i, which is standard notation in games. If all players play best response to each other, the resulting strategy profile is known as Nash Equilibrium " (Nash) 1950; Kuhn & Tucker, 1951.. In two-player zero-sum games, this is usually the sought after solution concept. As metric
to evaluate quality of strategy, we use exploitability {(;,) u (f,,"".)) (1
(I, - uu (h which is how much can opponent gain, when it plays best response as compared to the Nash equilibrium. In two-player zero-sum games, the exploitability is always nonnegative and is zero if and only if the Ty is a part of Nash equilibrium.
3 LEARNING THE GAME MODEL
In perfect information games, players possess complete knowledge of the current game state rep- resented by a history h. Consequently, search algorithms initiate from a single, known root state, simplifying the search. In contrast, imperfect information games (IIGs) grant players only par- tial observability through an information set 8;, which typically corresponds to multiple possible underlying world states. As established in prior works, approximating optimal strategies via look- ahead reasoning in IIGs requires a more sophisticated approach than in perfect information settings (Kovařík et al. 2023 Moravčík et al. 2017.
3

---

Published as a conference paper at ICLR 2026
Specifically, it is insufficient to restrict the reasoning to only those histories consistent with the player's 's information set. Instead, the reasoning must encompass all histories that share the pub- lic state Sp- This necessity arises because sound reasoning algorithms compute strategies for all players simultaneously, aiming for mutual best responses characteristic of an equilibrium. Consider this situation in two-player Poker: if player holds two Kings, their information set 8; includes all histories consistent with this hand but with varying opponent hands. A reasoning restricted only to those histories would implicitly grant the opponent knowledge of i's hand when computing oppo- nent's 's strategy, leading to suboptimal strategies. Therefore, the look-ahead reasoning must operate over the broader set of histories consistent with public state to compute valid equilibrium strategies
(Moravčík et al. 2017 Schmid et al. 2023 Kovarik et al. 2023 Milec et al. 2024).
We adopt reinforcement learning paradigm where an agent learns from interaction with the envi- ronment. During training, we assume access to a game simulator capable of generating the whole game trajectories. During testing (gameplay), the agent receives only its own information set 8, at each step and must rely entirely its learned model to plan, without access to the simulator or explicit rules. This means that the agent does not use any domain-specific knowledge. at any point.
We propose model inspired by MuZero (Schrittwieser et al., 2020) but adapted for the IIG set- ting, which requires additional structures necessary to model the imperfect information. Our model comprises three core learnable functions, parameterized by 0:
Representation function A/ : S; + S. Maps a player i's potentially high-dimensional information set 8 to a fixed- size latent representation s; E S.
Dynamics function Y S x S x A1 x A2 3 x representations for all players (51, 32) and the joint action taken (a1, a2), this function pre- dicts the resulting next latent representations for both players, the immediate reward r (c.g., for player 1), and binary termination flag This models the joint evolution of the game across possible hidden states. We use B = [00,1] 13.
Given the latent
Legal actions function Ie S, - BlA:l. Predicts the mask of legal actions A, available to player from their latent representation 57. This is crucial for constraining the look-ahead reasoning only to the feasible parts of the game.
Each training episode a game trajectory is sampled. This trajectory is h wNIT a .a d'-' where a' = (at, ab) is the joint action at step t. For any sub-history h [ h, the simulator provides the true information sets (h²), legal actions (5()),, and the reward r². The model is trained to predict these quantities through recurrent application of its components.
Starting from initial latent state 8; the actual actions from the trajectory:
Af(()) the dynamics function is unrolled for k steps using
L.k 1,++1 ck+1 prckt F.A+T& = To(s) 1.k .22 aatth .02 altk) 1
$2
Here, si is the predicted latent state after k unrolls from step t, and pf, and [t,F are the predicted reward and termination logit. The legal actions function predicts logits A To(s from the initial latent state.
The model parameters 0 are optimized by minimizing a combined loss function over the trajectory:
cy
A9)
1=0
SEN
Logal Action Prodiction
(1)
cb (II-k (I==,,,t+k=]]
CB (rt.k 1(k
Ec ( Ablo)th))
SENN
Termination Prodiction
Reward Prediction
Latent State Prediction
where C/ and CJ are binary cross- cross-entropy losses, while LB and CP are mean squared errors. The target for the dynamics loss is the latent representation of the actual subsequent information set.
4

---

Published as a conference paper at ICLR 2026
Minimizing L trains the functions A/. To, Lo collectively act learned simulator. This learned model captures the necessary dynamics within the game, enabling test-time look-ahead rea- soning algorithms (discussed in Section 55) to effectively plan over the required set of public states without recourse to the original game rules or simulator.
LEARNING THE ABSTRACT MODEL
A primary limitation of sound look-ahead reasoning in imperfect information games is the potential size of the public states, as the number of information sets consistent with public information may be exponential in the history length, making the look-ahead reasoning intractable (Moravčík al., 2017 Schmid et al. (023).
Although traditional abstraction techniques often rely on domain expertise or require offline enumer- ation and analysis of all information sets (Cermák al. 2020; Kroer & Sandholm, 2018: Ganzfried & Sandholm 2014 Brown & Sandholm 2015 Bard et 22114JJoaanso et al. (003). we aim to learn domain- -ineaiiienndeeeeeent abstraction directly from the game experience during training. Our goal is to partition the vast space of information sets sharing a public state into a manageable number L of abstract information sets, enabling tractable reasoning. Such an abstraction may contain imperfect recall as discussed in Section
Consider Texas Hold' aald'emmee Poker as an example. player might hold any of the 1326 private hands, each corresponding to a different information set. Our abstraction aims to represent this multitude using only I. representatives, learned based on similarity within the training process rather than predefined rules.
We adapt the model from Section 3 but we will use mechanisms inspired by online clustering to limit the amount of information sets in each public state. We hypothesize that information sets behaving similarly, e.g. having similar optimal strategies or leading to similar future states, should be grouped. We formalize this using a function K S RK which maps any information set to K <-dimensional space, in which the clustering will be performed.
To satisfy the condition that each public state consists of most L. information sets of each player, we split the representation function into two parts as shown in Figure [ The public state represen- tation Ar, So + S maps a public state so to L latent abstract information sets for a player The information set representation Ai,0 S - AS; maps a real information set to the probability distributions on the abstract information sets provided by Ag,e- Crucially, despite Ao providing a probability distribution, we enforce the many-to-one mapping, so we represent any real information set by the single abstract information set, corresponding to the highest probability. This represen- tative will then be used for training dynamics Yo and legal actions l-g- We opted to this, so that the dynamics function constructs the search tree, which is compatible with look-ahead reasoning algorithms like Counterfactual Regret Minimization.
In order to perform the clustering, we require the same function x as for the real information sets. We introduce Kę S, + IR which will be trained to predict this clustering property for each abstract information set. The dynamics function Tg and legal actions function I i are defined as in Section
3
To learn the abstraction that approximates the proposed clustering based on K, we train Ar, Afe and Kæ jointly. Each training step, the simulator provides trajectory h = "..ara...aa"ae.... w'' st sf represents the public state and information set of player at time ł in the same trajectory.
The training involves two additional losses, the first one L4 updates the Aio9 and Kg using a soft clustering objective similar to fuzzy c-means (Bezdek et al.) 1984). minimizes mean squared error between real and abstract properties weighted by the softmax. The second loss LS updates the A/, by minimizing the cross entropy loss between the predicted probability distribution over the abstract information sets and the one-hot encoded vector of the abstract information set that is the nearest neighbor of the real information set based the is.
c = ΣΣ Σ
(1mo(5))
x2s()112
(2)
EA. (==1llo(=5)==((())
EEN (()
5

---

Published as a conference paper at ICLR 2026
S0
A,o
Figure 1: The public state and information set representations functions of player i. First the A,, predicts abstract information and then A predicts the probability distribution over those abstractions.
arg min [Ko(s)) n($!)1??
Ca = -1=Σ (
(3)
Here, x sg(x) is an identity operator that stops the gradient flow through x. The 7 controls the softness of the clustering and as Y oo the clustering becomes hard. The gradients from C propagate through both Ko and Ai,o, but the gradients from LG are only propagated through A
C3 L3 LB +
The overall loss also includes the model learning loss from Section 3 Importantly, C3 is computed using the dynamics based on the selected abstract information set. Furthermore, the gradients from F. are not backpropagated through Ar,,, Ai. or Kg. This decouples the learning of the abstraction structure from the learning of the model dynamics.
DEPTH-LIMITED SOLVING
While the learned model and abstraction allow reconstructing the whole game tree of the abstracted game, practical applications in large imperfect information games rely on depth- limttelimited reasoning combined with a learned value function estimate payoffs beyond the reasoning horizon. Defining and training optimal value function for imperfect information games is challenging, as they theoret- ically depend on belief states (Brown al. 2020; Kovařík al. 2023). The belief states are public states with corresponding probability distribution of reaching each history. Such value function is often trained by repeatedly sampling varied belief states and performing depth-limited limited reasoning in each of them Moravčík et al. (2017); Schmid et al. (2023). which is generally intractable.
In this section we present an algorithm Learned Abstract Model for Imperfect-informattie Reason- ing (LAMIR), which learns the abstracted model from Section along with value function that enables depth-limit reasoning. We first show what additional components are necessary to learn the value function, then summarize the whole training procedure and lastly we show how these trained components for depth-limited reasoning.
LAMIR value function based the value- states (Brown al. 2018 Kubiček al. 2024 Milec et al. 2025). Training of this value function just from samples requires these additional components:
Strategy function no S, + AA, which for given information set from the original game returns the strategy trained with some policy-gradiem algorithm, like RNaD (Perolat al., 2022).
Transformations function To : S, + ||A.I representing T characteristic directions in strategy space explored by the policy-gradient algorithm during training. For sin- gle transformation X E Th the transformed strategy is computed locally as (a,, a,) = n, 7,(8,, a,) x(*,, a.). The resulting strategy ς. (8t) is clipped non-negative numbers and then normalized (Kubiček et al. 2024).
Value function U9 S x S. + | which approximates the expected value of each combination of transformed strategies between players (Kubičck et al. 2024).
We train To and To using real information sets from sampled trajectories. However, og is trained using the corresponding joint abstract information sets instead of histories. It is possible that several

---

1.NEW are Published as a conference paper at ICLR 2026
real histories map to the same abstract state with varying reach probabilities. This may introduce bias to the trained values. While importance sampling could correct this, we found in our experiments did not affect the results in any significant way. This is most likely due to transformations being just a heuristic approach to approximate different parts of the strategy space.
We have used Regularized Nash Dynamics (RNaD) the policy-gradient algorithm, which includes Neural Replicator Dynamics (NeuRD) loss for strategy training and mean squared error for the associated baseline value function Cg (Hennes et al. 2020 Perolat al. 2021 2022). Note, that the value function from the RNaD cannot be used as a value function for look-ahead reasoning, because represents value for specific belief based on the networks strategy and not for arbitrary belief. We use mean squared eπror for the training of the value function Lg and the targets are computed by the V-trace (Espeholt al. 2018, Kubiček al. 2024), which estimates the value of different policies in an off-policy setting.
Following Kubíček et al. (2024), transformations represent the strategy changes during training. For each player we compute the difference vector 8f R; concatenated strategies along the whole trajectory before and after the training step. Instead of the hard clustering proposed originally, we use the soft clustering from Section
N,N 1,OLD where 1,LLD and n
=;
E --
4 Σ
Σ IEN XIET
IIx:
8211 -221?
(5)
Exer e -iceme--llx--ee -
5.1 TRAINING
A single training iteration of LAMIR begins by sampling a trajectory h = w INT cording to a sampling strategy μ.. In the on-policy setting, this strategy the current policy, μ = mp- While our approach is not limited to on-policy learning. off-policy settings would require scaling the policy and value function updates with importance sampling.
a² a 'w" ac-
For any sub-history ht [ h. the simulator provides the true public state representation s (h"), true information set representation sht)). legal actions (st). and the reward r². Our method uses a two-step update per episode. This is necessary because the transformation loss C depends on strategy changes induced by the policy-gradient step. First step trains the strategy function we using a policy-gradient loss L. of the underlying policy-gradient algorithm. In the specific case of RNaD, this loss is NeuRD (Hennes et al. 2020). Second step computes the transformation loss L and value function loss C/ (Kubíček et 2024). The second step also includes all losses related to learning the game abstraction.
At each timestep the abstraction network Ag, predicts a set of I. possible abstract information sets for each player: $L_5 = As,)(s)). The goal is to cluster information sets in a K -ddimensional space. A trainable function Kig (s),)) maps the abstract information sets into this space. Similarly x(s{) maps the true information set. g could be separate provided function either by environment or user, but similarly it could be the trained strategy K = kq- Lg is computed using the K -ddimensional representations. It updates both Ai,0 and Kg.
Concurrently, separate network Ai, predicts probability distribution over the L possible abstract sets sf given the current real information set s². The L is a cross-entropy loss that trains A/. to increase the likelihood of selecting the closest abstraction, 5/ in the K-dimensional space. Finally, the LM is computed as described in Section 3 The dynamics model Ta is trained to predict the most likely future abstraction, $
k+ after k steps. The final losses for the two-step update
C/ CPG
C-= + Cg + CJ
5.2 LOOK-AHEAD REASONING
At test time, LAMIR employs continual resolving (Moravšík et al,, 2017: Schmid et al.) 2023) within the learned abstract game. The process begins at the initial history INI,, which defines
7

---

Published as a conference paper at ICLR 2026
the starting public state su (h and information sets $1 ( (I The root of the initial abstract subgame is generated using A,,;, A1,8-
The following process then repeats until the terminal state reached. The algorithm constructs the depth-limited game tree using the learned dynamics Yg and legal actions To. At the depth limit, a final "decision layer" is added for non-terminal states. This layer consists of T artificial actions where both players choose their strategy for the remainder of the game. Each combination leads to
a terminal state with a reward defined by the learned value function dg (Brown et al. 2018 Kubiček et al. 2024).
In this newly created abstracted game LAMIR runs Counterfactual Regret Minimization+ (CFR+)
or some of it's variant (Zinkevich et al. 2007 Tammelin 2014 Farina et al. 2019) to get strategy at each decision node. The actual current information set of the acting player 8; is mapped to its
corresponding abstract information set क7 using A4.0- The strategy corresponding to 87 is used to sample an action that advances the real game to the new state.
Out of the previous game tree, all the histories consistent with sq are used to create a new gadget game. Each of those histories is associated with joint abstract information set. Thanks to Ar,,, there are at most L unique combinations. Any two histories that share the same joint abstract information set are merged so that the new subgame root contains at most L nodes. Counterfactual values and the acting player's reaches are reused from the previous subtree to provide necessary statistics to the gadget game. The algorithm repeats this process until it reaches the terminal state (Burch et al. 2014 Moravčík et al. (0117).
EXPERIMENTS
6.1 EXPLOOTABIILIYY IN SMALLER GAMES
To ensure that the strategies found by LAMIR approximate Nash equilibria, we applied it in games small enough to compute exact exploitability, which serves as distance from the Nash equilibrium in two-player zero-sum games. For various abstraction sizes L. and different properties for clustering s we trained LAMIR with 10 different random seeds for 100,000 episodes. In all of our experiments used Regularized Nash Dynamics (RNaD) to train the strategy for multi-valued states value function. The rules of the games used in experiments are in Appendix C
Starting at episode 80,000 we have computed exploitability every 1000 episodes. In each public state, the algorithm uses the trained functions to construct the depth- subgame with depth limit 1. Then this subgame is solved using CFR+ and the strategy from abstract infosets is mapped to real ones. Then we compute the exploitability of the final composed strategy in the original game. The results for different x and I are displayed in Figure 2
Compared to RNaD, the LAMIR has a more profound training. As a result single training step of LAMIR takes more time. In our implementation, the training roughly 2-2.5 times slower per- iteration depending on the size of the abstraction L. We further discuss this in Appendix [.2 Still, the exploitability of RNaD reported in Figure does not improve even with increased training time. The main likely cause are network approximation errors. Similarly in the original results, RNaD could not decrease the exploitability in Ledue Poker beyond a certain point (Perolat et al.) (02TT.
We used three different K, which served as a basis for clustering, first the legal actions, second the legal actions with the current RNaD strategy, and third the legal actions, RNaD strategy, and the player's action history. Each choice of x is capable of outperforming the concurrently trained RNaD with the same amount training episodes given sufficient L. When using the action history as k, each information set is uniquely defined in given public state, which means that, with sufficient capacity, the dynamics network should model the underlying game. This was evaluated in II Goof- spiel 5 with L 30, which indeed has mapping one-to-one for each abstract and information set. The main reason why the exploitability is greater than 0 arises from the discrepancies in the rewards produced by the dynamics function.
8

---

Published as a conference paper at ICLR 2026
N..0 (010)
Lose
Lecal
Actioa Fre0 Sastep ACOC atory Lagal acteet Strsing
100
Acticn litayy Legal Intiot Strstayy
006
MEP
23.20
2000
1000
3500
2373
405 kgy
1300
000
Alstraction Link
Atstoaziion Lanii
(a) Imperfect Information Goofspiel 5
(b) Imperfect Information Oshi Oshi-Zuma Zumo 3,5
Exploitabilityy of LAMIR in a different games by using continual resolving with depth- limit in each subgame with different choice of abstraction limit L and k. The largest public state in II Goofspiel 5 contains 30 infosets and in Oshi-Zumo 3,5 it contains 625 information sets.
Figure 2:
Algorithm
II Guofspiel 10
II Goofspiel 13
II Goofspiel 15
LAMIR к Legal actions
54.47 = 0.25 %
60.68 + 0.34 %
80.49 + 0.26 %
LAMIR x = RNaD strategy
61.60 = 0.29 %
58.33 + 0.27 %
61.80 + 0.36 %
Table 1: Average win rate with 2-sigma error bars of LAMIR against RNaD in different games.
6.2 HEAD-TO-HEAD IN LARGE GAMES
The main usage of LAMIR is in very large games, where the exact exploitability cannot be com- puted. This experiment evaluates LAMIR in this exact setting, where we compare it with Regular- ized Nash Dynamics (RNaD) in head-to-head play. For each game and k, we have trained LAMIR with 3 different random seeds for 3 million episodes. Similarly, we have trained RNaD with 6 differ- ent random seeds for the same number of episodes. The hyperparameter settings remained the same for each game. Then we matched each trained seed of LAMIR with each trained seed of RNaD and played more than 100,000 matches. Note that when we use k = RNaD strategy, is the strategy that is learned concurrently for the value function and not the RNaD strategy against which the algorithm is compared later. The resulting win rates with 2-sigma error bars are in Table T
LAMIR consistently outperforms RNaD in each of the tested games. Imperfect Information Goof- spiel is known for its large public states, so continual resolving without abstractions is not applica- ble. SePoT (uubčček et al. 2024 was also evaluated in such large domains, but we did not compare against it, as it only uses CFR if the subgame is small enough. The authors showed that in II Goof- spiel 13, SePoT has win rate of only 52% which is likely caused by not resolving almost any subgame due to the limit on the size of the subgame. Furthermore, we show that even in larger games than those tested with SePoT, LAMIR improves over RNaD even more.
LIMITATIONS AND FUTURE WORK
LAMIR advances the scalability of the continual resolving paradigm to larger games by integrating leamed models and abstractions. However, it presents several interesting avenues for future research.
The computational complexity of the look-ahead reasoning, when using CFR is linear in the amount of information sets in the game. This complexity still remains, as LAMIR only reduces the size of the game in each public state but uses CFR in the abstract game. Each subgame LAMIR construct with depth D contains at most (O.......)) L [A]2d unique nodes and at most L | A1 terminal histories, where T is the amount of transformations. Most of the subgames would be smaller, but this is the main limitation of LAMIR as it limits how large L can be.
Currently, LAMIR's is dynamics network Yp does not explicitly model chance nodes within the game. LAMIR could still be applied in games with chance events, but this is not an intended setting as the absence of chance nodes will result in poor abstraction, regardless of the abstraction capacity L. Although algorithms like Stochastic MuZero (Antonoglou et al.) 2022) demonstrate that modeling chance is feasible in learned models for perfect information games, integrating chance nodes into
9

---

Published as a conference paper at ICLR 2026
our framework, particularly in conjunction with learned abstractions, requires careful consideration and is a key area for future work.
The effectiveness of the learned abstraction depends the chosen property function ε for clus- tering. In games without chance, if this property function could perfeetly distinguish two different information sets, L is greater than the size of the largest public state and neural networks have suffi- cient capacity, then LAMIR learns a pear-perfect model of the game. When L is reduced to achieve scalability, the learned abstraction may not capture all crucial strategic nuances, which can affect the strength of the derived strategy. In experiments, we have used simple proxies present in any game and it still yielded strong performance. Ideally, would also be learned during the training process.
Reducing L necessarily introduces imperfect recall, meaning that the player in the abstract game may "forget" information it previously knew. Algorithms like Counterfactual Regret Minimization (CFR) guarantee convergence to Nash equilibrium strategy only in perfect recall games. While CFR's convergence is not generally guaranteed in imperfect recall settings, it has been shown for subclasses such as A-loss recall games (Cermák al., 2020). Our abstractions are A-loss recall games if the public observations in the original game depend only on prior public information and the actions taken in the current round. Games like Imperfect Information Goofspiel or Oshi-Zumn satisfy these conditions, but many others, including Battleships, Dark Chess, or Stratego, do not. Thus, for games that do not fall into the A-loss recall category after abstraction, theoretical conver- gence guarantees for CFR-based methods within LAMIR are not assured.
LAMIR focuses on abstracting information sets but does not inherently abstract action spaces. In games with very large or continuous action spaces (e.g.. bet sizing in Poker), the sheer number of actions can remain a bottleneck for the look-ahead reasoning, regardless of the information set abstraction. While action abstractions have been extensively studied (Brown & Sandholm [014] 2018 Li et al.] (024), integrating them with LAMIR is out of the scope of single paper, but presents another direction for future improvement.
CONCLUSION
In this paper, we have introduced LAMIR, an algorithm designed to enable look-ahead reasoning by learning the model of the game with a suitable abstraction directly from experience, without the need for any domain-specific knowledge.
Our core contributions are fourfold. First, we have proposed a method for learning the fundamental components of model of dynamics of imperfect information games without chance. Second, we developed a technique for automatically learning an abstraction by clustering information sets, ef- fectively reducing the size of the game. Third, we integrated these components with learning of value function in the abstracted game that enables depth- limtted look-ahead reasoning. Lastly, we demonstrate how LAMIR facilitates the continual resolving paradigm by performing a depth-limites look-ahead reasoning in each decision node encountered.
We empirically verify that, when given sufficient capacity, LAMIR learns a nearly perfect model. Still, the game-playing capabilities degrade gracefully when the abstraction capacity is reduced. We have also shown that LAMIR manages to perform look-ahead reasoning even in intractably large public states. Thanks to that, it achieved up to 80% win rate in large games compared to RNaD, strong baseline that was successfully used to create a human-expert level player in Stratego.
The primary impact of LAMIR lies in scaling look-ahead reasoning techniques to larger games by learning abstraction and its model directly from experience. This overcomes the most notable limitations of the continual resolving paradigm, which requires each subgame to be tractable and to have explicit access to the game model. LAMIR has several limitations, which can be addressed in future work. Still, LAMIR is, the best of our knowledge, the first algorithm that enables Kook-ahead reasoning in large-scale games like Imperfect Information Goofspiel 15 without any domain-specific knowledge, substantially outperforming the model-free policy-gradient algorithm.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research is supported by Czech Science Foundation (GA25-18353S) and Grant Agency of the CTU in Prague (SGS23/184/0) 34/2H33//00H33333/1/333 Computational resources were supplied by (e-INFRA CZ LM2018 840)) supported by the Ministry of Education, Youth and Sports of the Czech Republic and also (C2.02.1.01/0000.0//66 0190000076)))
REFERENCES
loannis Antonoglou, Julian Schrittwieser, Sherjil Ozair, Thomas K Hubert, and David Silver. Plan- ning stochastic environments with a learned model. In International Conference on Learning Representations, 2022. URL htlLttts:
Nolan Bard, Michael Johanson, and Michael Bowling. Asymmetric abstractions for adversarial settings. In Proceedings of the 2014 international conference on Autonomous agents and multi- agent systems, pp. 501-508, 2014.
James C Bezdek, Robert Ehrlich, and William Full. Fem: The fuzzy c-means clustering algorithm. Computers & geosciences, 10(2-3)1191-00, 1984.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miine and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: //qithub. com/ jax-m1/jax
Noam Brown and Tuomas Sandholm. Regret transfer and parameter optimization. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, AAAI'14, pp. 594 691-601. AAAI Press, 2014.
Noam Brown and Tuomas Sandholm. Simultaneous abstraction and equilibrium finding in games. In International Joint Conference on Artificial Intelligence, 2015. URL https://app.
Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418 424, 2018. doi: 00.1126/science.a017733 URL
https: wwwww www.nce.eccc..... org/ doi/ abs 10. wwt.////iiiiiee aao1733333..
Noam Brown, Tuomas Sandholm, and Brandon Amos. Depth-limited solving for imperfect- information games. In S. Bengio, H. Wallach, H. Larochelle, Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 31, 2018 URL htttps://proceediiii/ wwtp.pssccc/pppppprrr...... wttpppppee.lleepppppppeeeeeeeeee 2018 fi
Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret mini- mization. In International conference on machine learning, pp. 793-802. PMLR, 2019.
Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong. Combining deep reinforcement learning and search for imperfect-information games. Advances in neural information processing systems, 33:1 7067-11069. 2020.
Neil Burch, Michael Johanson, and Michael Bowling, Solving imperfect information games using decomposition. In Proceedings of the Twenty-Eihtth AAAI Conference on Artificial Intelligence, AAAI'14, pp. 602-608. AAAI Press, 2014.
DeepMind, Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Antoine Dedieu, Clau- dio Fantacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, lurii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, George Papamakarios, John Quan, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Laurent Sartran, Rosalia Schneider, Eren Sezener. Stephen Spencer, Srivatsan Srinivasan, Miloš Stanojević, Wojeiech Stokowiec. Luyu Wang, Guangyao Zhou, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL http:
7 //github.ccc comeb..comeee e www.oooow...............

---

Published as a conference paper at ICLR 2026
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International conference machine learning, pp. 1407-1416. PMLR, 2018.
Gabriele Farina, Christian Kroer, Noam Brown, and Tuomas Sandholm. Stable-predictive optimistic counterfactual regret minimization. In International conference on machine learning, pp. 1853- 1862. PMLR, 2019.
Sam Ganzfried and Tuomas Sandholm. Potential-aware imperfect-recall abstraction with earth mover's distance in imperfect-information games. Proceedings of the AAAI Conference on Artifi- cial Intelligence. 28(1), Jun. 2014. doi: 10. 10.1609/aaii. 22811.8816 URL BtLtttts:///sssaat orr/ php/ AAAI/ wwt.AA//atiieeevvveeee 8816
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
URL http::tt///pptt htttps://cpenrevieeee n woree//oormmiiit watpmmmmids1111oo00
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with
discrete world models. In International Conference on Learning Representations, 2021. URL
https://cpenrew// wwt.iiewweeeeeeeeeeeeeeee ftr.m/ormmmmmmo meet:/oorrmmiido0abrrryr
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, pp. 1-7, 2025.
Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for contin- control. In The Twelfth International Conference on Learning Representations, 2024. URL /tttps://opentee
ew net. forum?id= wtw.rrmmmmidxxxhhtttttt.o
Nicklas A Hansen, Hao Su. and Xiaolong Wang. Temporal difference learning for model predictive control. In International Conference on Machine Learning, pp. 8387-8406. PMLR, 2022.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2024. URL
com/ gwtp://ooolllllaeooo
Daniel Hennes, Dustin Morrill, Shayegan Omidshafiei, Rémi Munos, Julien Perolat, Marc Lanetot, Audrunas Gruslys, Jean- -aatttap Lespiau, Paavo Parmas, Edgar Duèñez Guzmán, and Karl Tuyls. Neural replicator dynamics: Multiagent learning via hedging policy gradients. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgemt Systems, AAMAS '20, pp. 492-501, Richland, SC, 2020. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450375184
Michael Johanson, Neil Burch, Richard Valenzano, and Michael Bowling. Evaluating state-space abstractions in extensive-form games. In Proceedings of the 2013 International Conference on Autonomous Agents and Multi-Agent Systems, AAMAS 13, pp. 1pp271-22 271-278, Richland, SC, 2013. In- tematiionl Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781999811 1771145011935.
Vojtéch Kovařík, Martin Schmid, Neil Burch, Michael Bowling, and Viliam Lisý. Rethinking formal models of partially observable multiagent decision making. Artificial Intelligence, 303: 103645, 2022.
Vojtëch Kovařík, Dominik Seitz, Viliam Lisý, Jan Rudolf, Shuo Sun, and Karel Ha. Value functions for depth-limine solving in zero-sum imperfect-infee innirmaatioooooooonnnn games. Artificial Intelligence, 314: 103805, 2023.
Christian Kroer and Tuomas Sandholm. A unified framework for extensive-form game ab- straction with bounds. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 31, 2018. URL ht
http:///aaaaeeeeeeeeeeeaaaaaa/ files/papez/ 0ttp:///
12

---

Published as a conference paper at ICLR 2026
Ondiej Kubíček, Neil Burch, and Viliam Lisý. Look-ahead search on top of policy networks in
imperfect information games. Kate Larson (ed.), Proceedings of the Thirty-Third Interna-
tional Joint Conference on Artificial Intelligence, IJCAI-24, pp. 4344-4352, International Joint URL
Conferences on Artificial Intelligence Organization, 8 2024. doi:
httts:://do .rgg www.org0000 wtl.gg/1...99.....
Main Track.
Harold W. Kuhn and Albert William Tucker (eds.). Contributions to the Theory of Games, Volume Princeton University Press, Princeton, 1951. ISBN 9781400881727. doi: doi: 10.1515/9784400888
URL httthttp:///dee org/oogg//ooooo 10. 1515/978140088172
Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Per- olat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent rein- forcement learning. In Guyon, U. Von Luxburg. S. Bengio, H. Wallach, Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- URL httppss/pproceeediingnge//////ee http://cceeeiin.....eeeeeeeeeeee neurips cttp:ppp.cccpppppppppepepeppppp wwtpccccppppppeeeececeeeeeeeeeee 2017 h00/:f/le2222eeeee55500ffffffff3 http:///nnn.raaorrre//////p/p//s
ume 30, 2017.
pdf
Mare Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Saeta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schrittwieser, Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan-Davis. Open- Spiel: A framework for reinforcement learning in games. CoRR, 19111998999533 2019, URL http:/ /tt//axxvvv.// org/ abs/ 1908 09453
Boning Li. Zhixuan Fang, and Longbo Huang, RI-efr: improving action abstraction for imper- fect information extensive-form games with reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org. 2024.
V. Lisý and Michael Bowling. Eqilibrium approximation quality of current no-limit poker bots. The AAAI-17 Workshop on Computer Poker and Imperfect Information Games, WS-17-06, 2017. URL https: / /tpppaapppa///////eaa wtw.:eeeaaaaniiiiiooeeooo
org/ CorpusID: 6206144
Weiming Liu, Haobo Fu, Qiang Fu, and Yang Wei. Opponent-limited online search for imperfect information games. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Pruceedings of Machine Learning Research, pp. 21567-21585. PMLR, 23-29 Jul 2023. URL Ctlhttps:: hatpps://proceedingg: mlr press mtt.....eeeeerr22//////.....a
Wataru Masaka, Mitsuki Sakamoto, Kenshi Abe, Kaito Ariu, Tuomas Sandholm, and Atsushi Iwasaki. The power of perturbation under sampling in solving extensive-form games. arXiv preprint arXiv:2501. 16600, 2025.
David Milec, Ondřej Kubiček, and Viliam Lisy. Continual depth-limited responses for comput- ing counter-strategies in sequential games. In Proceedings of the 23rd International Confer- ence on Autonomous Agents and Multiagent Systems, AAMAS '24, pp. 2393-2395, Richland, SC, 2024. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9798400704864.
David Milec, Vojtëch Kovařík, and Viliam Lisy. Adapting beyond the depth limit: Counter strate- wwtpp/wwwwwpppppp////oo////////.
gies in large imperfect information games, 2025. URL
10464
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 8(7540) 2015.
Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artifi- cial intelligence in heads-up no- no-limit poker. Science, 356(6337):508 513, 2017. doi: 10.1
science.ce science.aam6996 aam6960. URL https: www. www.sscieeeee.e.ee org /oii aamm 9.00
13

---

Published as a conference paper at ICLR 2026
John F. Nash. Equilibrium points in jignj/it- games. Proceedings of the National Academy of Sciences, 36(1):48 49, 1950. doi: 10.1073/pnas.3..144 URL httpt:tpps//mmm /www pnas orq/ Coi/abs/ 10. 10..11033/pa0
36.1.48
Julien Perolat, Remi Munos, Jean-Baptiste Lespiau, Shayegan Omidshafiei, Mark Rowland, Pedro Ortega, Neil Burch, Thomas Anthony, David Balduzzi, Bart De Vylder, et al. From poincaré re- currence to convergence in imperfect information games: Finding equilibrium via regularization. In International Conference on Machine Learning, pp. 8525-8535, PMLR, 2021.
Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T Connor, Neil Burch, Thomas Anthony, et al. Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 378(6623):990-99) 9960099999-0000 2022.
Stuart J. Russell and Peter Norvig. Artificial Intelligence: Modern Approach. Pearson Education, 2 edition, 2003. ISBN 0137903952.
Martin Schmid, Neil Burch, Mare Lanctot, Matej Moravcik, Rudolf Kadlec, and Michael Bowling. Variance reduction in monte carlo counterfactual regret minimization (vr-meefr) for extensive form games using baselines. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 2157-2164, 2019.
Martin Schmid, Matej Moravčík, Neil Burch, Rudolf Kadlec, Josh Davidson, Kevin Waugh, Nolan Bard, Finbarr Timbers, Mare Lanctot, G. Zacharias Holland, Elnaz Davoodi, Alden Christianson, and Michael Bowling. Student of games: A unified learning algorithm for both perfect and imper- fect information games. Science Advances, (46))ead3335 2023. doi: 10.1 1 126/sciadv.adg33222.. URL lttpps:/ http::/wwwwwscinc..eec. science, wwg//oo.io.o.ooooo doi abs 112. wtw.:////iciiioo adg 3256
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature, 588(7839):604-605 609, 2020. ISSN 0028-0836. doi: 10. 01068/44156600000000511--11
John Schultz, Jakub Adamek, Matej Jusup, Marc Lanctot, Michael Kaisers, Sarah Perrin, Daniel Hennes, Jeremy Shar, Cannada Lewis, Anian Ruoss, Tom Zahavy, Petar Veličkovic, Laurel Prince, Satinder Singh, Eric Malmi, and Nenad Tomašev. Mastering board games by external and internal planning with language models, 2025.
12119
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):4844889 2016.
David Silver, Thomas Hubert, Julian Schritwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanetot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that mas- ters chess, shogi, and go through self-play. Science, 36266419) 2018 doi: 10.1 science.aar64004. URL https::////. wwtp::/wwwwwwww..../...ccc www.orggg/o.... http:///d..aaab/.......... 1ttp///1111111///eeeeeeeee [aar6404]
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar, Scaling IIm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 2rXiv:2408.03314 2024.
Samuel Sokota, Ryan D'Orazio, Chun Kai Ling, David J. Wu, J. Zico Kolter, and Noam Brown. Abstracting imperfect information away from two-player zero-sum games. In International Con- ference Machine Learning (ICMI). 2023.
Eric Steinberger, Adam Lerer, and Noam Brown. Dream: Deep regret minimization with advantage baselines and model-free learning. arXiv preprint arXiv:2006.104 10410, 2020.
Oskari Tammelin. Solving large imperfect information games using cfr+. arXiv preprint 1rXiv:1407.5042, 2014.
Guido Van Rossum and Fred L. Drake. Python Reference Manual. CreateSpace, Scotts Valley, CA, 2009. ISBN 1441412697.
14

---

Published as a conference paper at ICLR 2026
Jiní Čermák, Viliam Lisý, and Branislav Bošanský. Automated construction of bounded-low
imperfect-recall abstractions extensive-form games. Artificial Intelligence. 282:103248, 2020.
ISSN 0004-3702. doi: http:///o..og////...o////aaamm0p 103248. URL httpp:: www.
ecieenedion wwt.irrttttttoeeeeee. httpa//arr.....aaeeeeaaaaaaaaa/a
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning, nature, 575(7782):350-354 2019.
Tony Tong Wang, Adam Gleave, Tom Tseng, Kellin Pelrine, Nora Belrose, Joseph Miller, Michael D Dennis, Yawen Duan, Viktor Pogrebniak, Sergey Levine, al. Adversarial policies beat superhu- go ais. In International Conference on Machine Learning, pp. 357699.-379.... PMLR, 2023.
Brian Hu Zhang and Tuomas Sandholm. Subgame solving without common knowledge. In Pro- ceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. ISBN 9781713845393.
Brian Hu Zhang and Tuomas Sandholm. General search techniques without common knowledge for imperfect-information information games, and application to superhuman fog of war chess. In Proceedings of the International Conference on Learning Representations (ICLR), 2026. URL https://
arxiv. www..org/aae. abs /506 01242 Early version arXiv, 2025.
Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural Information Processing Systems, volume 20, 2007. URL httpt:ttt////ttt http://eeeeese.eee.e.o....o/ooer Ei220008eSscccaaaaaaaaa0003333ee
15


Reviews:
[{"review_id": "btbbahimQi", "reviewer": "Reviewer_nqxi", "summary": "This paper presents a new method for building an agent that plays two-player zero-sum imperfect-information games, using a lookahead search technique similar to MuZero.\n\nThe main Dynamics function maps a joint latent infostate and a joint action to a new joint latent infostate. It is learned through sampled trajectories via a simulator. It can then be used at test-time with depth-limited solving to play well.\n\nAdditionally, the method uses an abstract model to deal with large public states.\n\nThe paper includes experiments computing exact exploitability in small Goofspiel and Oshi-Zumo games, and head-to-heads results against RNaD in large Goofspiel.", "questions": "1. Apologies if I missed it in the paper, but is it true that this method only works on games with no chance?\n2. While playing the game, is it possible that the root public state of the game tree contains more than $L$ information states? \n3. I'm a bit confused on Section 4, especially what it is used for. I think Section 4 describes how to train $\\Lambda_{i,\\theta}$, $\\kappa_{\\theta}$, and $\\Lambda^I_{i, \\theta}$.\n\nHowever, the description of the second loss says that it updates $\\Lambda^I_\\theta$. Is this a typo? Should it be $\\Lambda^I_{i, \\theta}$? But the equation describing the second loss only includes $\\Lambda_{i, \\theta}. \n\nIn the description of LAMIR at the end of Section 5, these are only used to map the real current infoset $s_i$ to the abstracted one $\\bar{s_i}$. Is this all it is used for? And why can't we just use $\\Lambda^I_\\theta$ to do this? \n4. In Section 4, why do we need to map latent infostates and infostates to a K-dimensional space? Why can't we just do everything in the latent space?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The research direction and idea are sensible. The method is explained well. The experiments are solid.", "weaknesses": "I don't quite understand the Abstract Model (see questions).\n\nI think more could be said about the motivation/application of this work. It's practically useful only if we (A) don't already have a perfect simulator but (B) have access to trajectories that include both players' infostates and actions. When could this be useful?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lAbjQyDtAs", "reviewer": "Reviewer_eWxn", "summary": "The authors propose abstract look-ahead search at test time for imperfect-information games. While training RNaD policy, they learn a world model over latent information sets. They then learn a per-public-state abstraction that clusters many real infosets into latent prototypes, yielding a depth-limited latent subgame that's smaller than the original game or world model. At the depth limit they attach a policy-transformation leaf gadget (from Kubíček et al., 2024) and solve the subgame with CFR. Test-time look-ahead achieves lower exploitability than vanilla RNaD in small games and beats RNaD in larger games.", "questions": "a) This method seems like it would be extendable to other self-play methods like magnetic mirror descent. Would you expect clear integration issues or challenges in using this lookahead method with population-based methods like PSRO?\n\nb) Is any regularization or activation applied to shape or constrain the space of latent infostates $\\bar{S}_i$?\n\nc) Without any loss of novelty, the base world model dynamics and encoder model seem mechanically very similar to the one used in TD-MPC [1] and TD-MPC2 [2] (A key distinction is that LAMIR operates over information sets under partial observability.)\n Would the world model stability improvements introduced in TD-MPC2 potentially be applicable to this work?\ne.g. \n1. Only training the initial state encoding $\\Lambda_\\theta^I(s_i^{t})$ using dynamics loss and stopping the gradients through dynamics targets $\\Lambda_\\theta^I(s_i^{t+k})$\n2. Bounding and regularizing the latent infostate space with SimNorm\n\n[1] Hansen, N., Wang, X., & Su, H. (2022). Temporal Difference Learning for Model Predictive Control.ICML 2022 (PMLR).\n\n[2] Hansen, N., Su, H., & Wang, X. (2024). TD-MPC2: Scalable, Robust World Models for Continuous Control.ICLR 2024.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This paper makes important and large contributions towards using latent-representation world models effectively in a 2p0s imperfect information self-play setting.\n- The proposed method is relatively straightforward.\n- The experimental results are sufficient and convincing.\n- The limitation to games without chance and lack of convergence grantees for games without A-loss recall are well explained.", "weaknesses": "- The proposed feature encoding for latent infostate clustering is a heuristic.\n- Code is not provided.\n- Minimal ablations on the world model architecture design.\n- Missing related works: TD-MPC and TD-MPC2", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bOWbPA6pqd", "reviewer": "Reviewer_q3UX", "summary": "This paper introduces LAMIR (Learned Abstract Model for Imperfect-information Reasoning), an algorithm designed to enable look-ahead reasoning in large-scale imperfect information games by an abstract model, which can directly learn from experience without any domain-specific knowledge. LAMIR achieves up to an 80% win rate in large imperfect information games compared to RNaD, and also shows robustness in some smaller games.", "questions": "Please address each concern raised in the weaknesses.\n\n### Regarding the training process:\n1. How is $\\Lambda_{i,\\theta}$ (the green component in Figure 1) specifically trained? Is it implemented using a clustering algorithm, or is it also modeled as a neural network? If it is a neural network, what are its input and output representations? Does it use the clustering result as the training target? It seems that the training loss in equations (2) and (3) is related only to $\\Lambda^{I}_{i,\\theta}$.\n\n2. In equation (2), during training, does it require enumerating all $\\overline{s^t_i}$? Or is it simply using sampling? If it is sampling, how many samples are required? Is this sampling process required to be performed for every public state? What is the computational cost of this sampling process? If the number of samples is too small, is the clustering into $L$ groups still effective? How is the value of $L$ determined in practice? Does the number of samples increase as $L$ increases? If so, how does the computational cost scale with $L$? Overall, what is the additional computational cost of training LAMIR compared to RNaD?\n\n3. What kind of training data is used? Is it based on a pre-defined dataset or trajectories obtained from its self-play? \n\n### Regarding test time:\n1. Section 5 presents the overall algorithm that integrates all components discussed in the previous sections. However, the description in Section 5 is unclear, particularly regarding how the components are used in the tree search. It would be helpful to include a figure illustrating the test-time architecture, along with a more detailed explanation of each component.\n\n2. During the search tree, is it still required to enumerate all $\\overline{s^t_i}$ or sample $\\overline{s^t_i}$ to obtain the latent abstract information sets?\n\n3. It seems that $\\Lambda_{i,\\theta}^{I}$ is used only at the root node of the game tree, where a specific abstraction set is chosen. However, this abstraction set only represents a single category. After performing a sequence of actions during the search tree, can we guarantee that this category still adequately covers all possible scenarios at the leaf nodes? Or is $\\Lambda_{i,\\theta}^{I}$ repeatedly used during the search at each internal node?\n\n4. The inference time is not reported. Currently, the depth is set to 1. Why not use a larger depth? It would be helpful to include an ablation study analyzing different depth settings, along with a discussion of the additional computational cost and potential performance gains as the depth increases.\n\n### Regarding the experiments:\n\n1. How are the different bases for clustering ($\\kappa$) implemented?\n2. Regarding training and test time, are LAMIR with different $L$ values and RNaD all trained under the same training time limits?\n3 . Based on the results in Table 1, why does using Legal Actions for clustering lead to a higher win rate than the RNaD strategy in the Goofspiel 13 and Goofspiel 15 environments? Additionally, would it be fair to use the RNaD strategy for clustering within LAMIR and then compare it directly with RNaD itself, given that LAMIR would already know the RNaD strategy?\n\n### Other questions and suggestions\n1. The reference formatting in the main text is incorrect. Most citations are missing brackets.\n2. How are the public state and the player's information set generated during training, especially in large-scale games? Additionally, what is the approximate number of public states in the \"large games\" used in the experiments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper provides a novel approach that combines model learning, abstraction techniques, and value function learning to address scalability challenges in large-scale imperfect-information games.\n\n2. The comparison to the baseline, RNaD, is persuasive, as it was previously used in DeepNash in Stratego.\n\n3. The experimental figures and methodology descriptions are clear and well-structured, providing a comprehensive understanding.\n\n4. The future work proposed by the authors is sound. For instance, integrating abstract action spaces could further enhance the applicability of LAMIR to various types of games, particularly those with large or continuous action spaces.", "weaknesses": "1. The description in Section 5 is insufficient. In particular, the explanations of the training and testing processes should be clarified, as they are somewhat difficult to follow in the current version. This is especially important because Section 5 presents the overall algorithm that integrates all components discussed in the previous sections. Including a flowchart or pseudocode would help readers better understand the steps. \n\n2. Section 4 uses a significant amount of space to describe the clustering process and explains how to train the clustering of the public state. However, it seems that this approach is applied only to the root node of the test-time search tree. The tree search process requires a more detailed description, including how each component introduced in previous sections is used.\n\n3. Since LAMIR requires clustering during training, the overall training time is expected to increase. The paper should clearly report the additional training cost of LAMIR and compare it with previous approaches, such as RNaD. It would also be helpful to add an additional experiment discussing how the computational cost increases as the number of clusters $L$ increases, as well as how much data is required to train both the clustering and abstraction processes.\n\n4. As mentioned in Section 7, the proposed approach seems to be constrained by the limitations of the CFR algorithm, where it is not applicable to many other games like Dark Chess and Stratego. However, the abstraction method itself should not be affected by this. Have there been any attempts to apply the core ideas of abstraction to these games?\n\n5. The experiments are conducted on relatively simple imperfect information games. Are there any experiments on more complex games, such as poker? Including such results would make the work more convincing by better demonstrating the generality and effectiveness of LAMIR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XYcxwnp09X", "reviewer": "Reviewer_MHic", "summary": "The authors proposed a novel algorithm that enables subgame search in imperfect-information games with a large number of public states. Specifically, they trained a clustering network to group similar information sets and solved the resulting abstracted subgames using CFR+. In addition, to constrain the size of the subgame, they trained a value network to predict state values, enabling depth-limited subgame search.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The performance of the algorithm looks good. It outperforms RNaD consistently.\n- The writing of the paper is clear.\n- The paper enables subgame search in games with a large public states.", "weaknesses": "- The techniques presented in this paper are not particularly novel. It appears to combine several existing approaches -- subgame search, game modeling, and abstraction. That said, integrating all of these components simultaneously is a non-trivial task.\n- The games evaluated in this paper do not seem to require model learning, as used in MuZero. I would prefer to see the method tested on games where developing an accurate simulator is intractable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)