Look-ahead Reasoning with a Learned Model in Imperfect Information Games

## Abstract

Abstract Test-time reasoning significantly enhances pre-trained AI agents’ performance. However, it requires an explicit environment model, often unavailable or overly complex in real-world scenarios. While MuZero enables effective model learning for search in perfect information games, extending this paradigm to imperfect information games presents substantial challenges due to more nuanced look-ahead reasoning techniques and large number of states relevant for individual decisions. This paper introduces an algorithm LAMIR that learns an abstracted model of an imperfect information game directly from the agent-environment interaction. During test time, this trained model is used to perform look-ahead reasoning. The learned abstraction limits the size of each subgame to a manageable size, making theoretically principled look-ahead reasoning tractable even in games where previous methods could not scale. We empirically demonstrate that with sufficient capacity, LAMIR learns the exact underlying game structure, and with limited capacity, it still learns a valuable abstraction, which improves game playing performance of the pre-trained agents even in large games.

## 1 Introduction

Strategic reasoning and planning are key components of human intelligence, encompassing our ability to reason about possible outcomes of actions in complex situations, often with incomplete information and uncertain consequences. Although humans navigate such decision-making naturally, replicating this process in artificial intelligence remains a fundamental challenge. Games, with their well-defined rules and yet complex strategic landscapes, serve as ideal testbeds for developing and evaluating AI planning and reasoning methods (Mnih et al., 2015; Silver et al., 2018; Perolat et al., 2022; Schultz et al., 2025).

In perfect information games like Chess, Go or Shogi, look-ahead search algorithms as Minimax and Monte Carlo Tree Search (MCTS) have achieved superhuman performance by systematically exploring future states (Russell & Norvig, 2016; Silver et al., 2018). These methods traditionally rely on access to game rules to implement state transitions in the search. MuZero demonstrated that an agent can learn a model of the environment dynamics implicitly through interaction and use this learned model to perform MCTS planning, removing the dependency on explicit implementation of the rules (Schrittwieser et al., 2020).

Not requiring explicit pre-programmed representation of the environment greatly expands applicability of AI methods. A method that does not require explicit rules representation can be applied, for example, to create an AI opponent in a proprietary video game without access to its source code; to create AI opponents for a large database of games for an online game playing platform, where programming a suitable representation for each of them would be prohibitively expensive; or in a game design setting, where the game is repetitively modified without the need for a programmer to reflect the changes in the implementation.

However, extending the model learning paradigm to imperfect information games such as Poker or Stratego presents fundamental difficulties. Since players lack complete knowledge of the state of the game, theoretically sound look-ahead reasoning methods need to reason about the distribution of all possible hidden states consistent with shared knowledge (Kovařík et al., 2023), which differs from the MCTS used by MuZero.

Our work aims to enable look-ahead reasoning in two-player imperfect information games using a learned abstract model, thereby eliminating the need for explicit game rules and also enabling look-ahead reasoning in parts of the game intractable without abstraction. Following the approach of Schrittwieser et al. (2020), we focus only on games without chance events, which is a large class that includes commonly used benchmarks like Dark Chess, Stratego, Battleship or Imperfect Information Goofspiel. This allows us to tackle the unique difficulties of learning an effective abstraction for imperfect information without conflating it with the separate challenges introduced by chance events. Our contributions are: We identify the necessary components that a learned model must capture to facilitate look-ahead reasoning under imperfect information. We introduce a training procedure to learn these components from sampled game trajectories. We demonstrate how tractably small, domain-independent abstractions can be learned concurrently with the model. Finally, we introduce a way to conduct look-ahead reasoning with the learned model.

Our empirical evaluation shows that in small games, given sufficient capacity, the strategies produced by look-ahead reasoning are less exploitable than those of concurrently trained Regularized Nash Dynamics (RNaD). Furthermore, in large games with shared knowledge consistent with over $10^{18}$ states for some decisions, the proposed look-ahead reasoning is still applicable in all decision points and significantly improves over RNaD, reaching up to 80% win rate in head-to-head play.

### 1.1 Related Work

#### Direct Policy Optimization

One approach for computing strategies in large imperfect information games stores the strategy implicitly in neural network weights and directly optimizes this policy based on self-play traces. These methods include: policy-gradient algorithms with reward regularization, like Regularized Nash Dynamics (Perolat et al., 2021; 2022; Sokota et al., 2023; Masaka et al., 2025); training networks to approximate CFR results, like Deep CFR or DREAM (Brown et al., 2019; 2020; Steinberger et al., 2020); or iteratively training best responses to growing strategy pools, like PSRO (Lanctot et al., 2017; Vinyals et al., 2019). Critically, these approaches rely solely on the trained actor during gameplay and cannot refine decisions with additional test-time computation. Pre-trained agents without test-time reasoning are usually very exploitable (Wang et al., 2023; Lisy & Bowling, 2016), and adding test-time reasoning greatly improves their capabilities in games (Silver et al., 2016; Kubíček et al., 2024), and beyond (Snell et al., 2024).
This paper enables adding test-time reasoning to policies created by direct policy optimization algorithms.

#### Look-ahead reasoning

Reasoning algorithms in imperfect information games, such as Counterfactual Regret Minimization (Zinkevich et al., 2007), iteratively improve player’s policies by systematically iterating over all possible future action sequences in all possible (unobserved) states of the game. In large games, this requires either domain-specific abstractions, like Libratus (Brown & Sandholm, 2018) or depth-limited reasoning, like DeepStack, ReBeL, Student of Games, SePoT (Moravčík et al., 2017; Brown et al., 2020; Schmid et al., 2023; Kubíček et al., 2024). In either case, all these algorithms require explicit implementation of game rules to construct game trees and manage belief states. It limits their applicability when exact rules are unavailable, computationally prohibitive or if the amount of possible hidden states is too huge. Knowledge-limited subgame solving (Zhang & Sandholm, 2021; 2025; Liu et al., 2023) can reduce the complexity, but even this reduced state space remains intractable in the games we study here. In contrast, our approach does not require explicit game rules and automatically learns a tractably small abstraction of the game just from full traces of game play.

#### Model learning

In single-player settings, Dreamer and subsequent works showed that learning models and generating artificial traces can match purely model-free approaches (Hafner et al., 2020; 2021; 2025). MuZero demonstrated similar results in perfect information games, using learned models for reasoning during gameplay (Schrittwieser et al., 2020; Antonoglou et al., 2022). Our work extends these approaches to imperfect information games without chance.

## 2 Background

We define two-player zero-sum simultaneous move game as $\mathcal{G}=(\mathcal{N},\mathcal{W},w^{\text{INIT}},\mathcal{A},\mathcal{T},\mathcal{R},\mathcal{O})$ Kovařík et al. (2022), where $\mathcal{N}=\{1,2\}$ are the players, $\mathcal{W}$ is the set of world states in the game and $w^{\text{INIT}}\in\mathcal{W}$ is the initial world state. $\mathcal{A}=\Pi_{i\in\mathcal{N}}\mathcal{A}_{i}$ is the set of joint actions,. We use $\mathcal{A}(w)\subseteq\mathcal{A}$ to denote the set of joint legal actions in world state $w$. $\mathcal{T}:\mathcal{W}\times\mathcal{A}\to\mathcal{W}$ is the transition function and $\mathcal{R}:\mathcal{W}\times\mathcal{A}\to\mathbb{R}$ is the reward function, which corresponds to the reward of player 1. We use $\mathcal{R}_{1}(w,a)=\mathcal{R}(w,a)$ and $\mathcal{R}_{2}(w,a)=-\mathcal{R}(w,a)$ as rewards for player 1 and 2 respectively. $\mathcal{O}:w\times\mathcal{A}\times\mathcal{W}\to\mathbb{O}$ is the observation function. $\mathbb{O}=\mathbb{O}_{0}\times\mathbb{O}_{1}\times\mathbb{O}_{2}$ is the set of joint public and private observations. $\mathcal{O}$ can be factored as $\mathcal{O}=(\mathcal{O}_{0},\mathcal{O}_{1},\mathcal{O}_{2})$, where $\mathcal{O}_{0}$ is a public part of observation and $\mathcal{O}_{1},\mathcal{O}_{2}$ are private parts of observations for each player. Even though we focus on simultaneous-move games, this does not limit the generality, since any sequential-move game can be converted into a simultaneous-move game by adding fictitious moves for the non-acting player in each decision node.

History $h=w^{0},a^{0}\dots a^{l-1}w^{l}\in(\mathcal{W}\mathcal{A})^{*}\mathcal{W}$ is a finite sequence of world states and actions, which starts in the initial world state $w^{0}=w^{\text{INIT}}$ and for each timestep $t\in\{0,\cdots,l-1\}$ holds $a^{t}\in\mathcal{A}(w^{t})$ and $w^{t+1}=\mathcal{T}(w^{t},a^{t})$. $\mathcal{H}$ is the set of all possible histories within the game. We use $h\sqsubseteq h^{\prime}$ to denote that $h^{\prime}$ contains $h$ as a prefix. We will use $h^{\text{INIT}}=w^{\text{INIT}}$ to denote the initial history. Each history $h$ ends with some world state $w^{l}$. We will sometimes use history $h$ in the game functions instead of the world state $w^{l}$. For example $\mathcal{A}(h):=\mathcal{A}(w^{l})$ corresponds to the set of joint legal actions in the final world state of the history. Each player $i$ does not observe the whole world state at each timestep, but only observes public observations $\mathcal{O}_{0}$ and its private observations $\mathcal{O}_{i}$. This means that the player may not be able to distinguish between several different histories. We will use $s_{i}\in\mathcal{S}_{i}$ to denote the set of all histories consistent with the observations of the player $i$, which we will call the information set. $\mathcal{S}_{i}$ is the set of all the information sets of the player $i$. $s_{i}(h)$ is the information set that corresponds to history $h$. Similarly, the public state $s_{0}\in\mathcal{S}_{0}$ is an information of an external player, which does not have private observations, so it contains all the histories consistent with public observations. Each public state contains one or more information sets for each player.

Strategy of player $i$ is a function $\pi_{i}:\mathcal{S}_{i}\to\Delta\mathcal{A}_{i}$ that maps each information set to a probability distribution over the actions. We will sometimes use $\pi_{i}(s_{i},a_{i})$ as a probability that player will play $a_{i}$ in information set $s_{i}$ if it follows the strategy $\pi_{i}$. $\pi=(\pi_{1},\pi_{2})$ is a joint strategy profile of both players. If $h\sqsubseteq h^{\prime}$, then the reach probability of reaching history $h^{\prime}$ from history $h$ under strategy profile $\pi$ is $P^{\pi}(h^{\prime}|h)=\prod_{h^{\prime\prime}aw\sqsubseteq h^{\prime}}\prod_{i\in\mathcal{N}}\pi_{i}(s_{i}(h^{\prime\prime}),a_{i})$. We also use $P^{\pi}(h):=P^{\pi}(h|h^{\text{INIT}})$. Any reach probability can be factored into the contribution by each player $P^{\pi}(h|h^{\prime})=\prod_{i\in\mathcal{N}}P^{\pi}_{i}(h|h^{\prime})$. Expected utility of a history $h$ if all players follow strategy profile $\pi$ is $u_{i}^{\pi}(h)=\sum_{h\sqsubseteq h^{\prime}a}P^{\pi}(h^{\prime}|h)\mathcal{R}_{i}(h^{\prime},a)\prod_{i\in\mathcal{N}}\pi_{i}(s_{i}(h^{\prime}),a_{i})$.

Best response against a strategy $\pi_{i}$ is a strategy $\pi_{-i}^{BR}\in BR_{-i}$, which maximizes the opponent’s utility $u_{-i}^{(\pi_{i},\pi_{-i}^{BR})}(h^{\text{INIT}})\geq u_{-i}^{(\pi_{i},\pi_{-i}^{\prime})}(h^{\text{INIT}})$ for any $\pi_{-i}^{\prime}$. We use a $-i$ here to symbolize the other player than $i$, which is a standard notation in games. If all players play a best response to each other, the resulting strategy profile is known as Nash Equilibrium $\pi^{*}$ Nash (1950); Kuhn & Tucker (1951). In two-player zero-sum games, this is usually the sought after solution concept. As a metric to evaluate quality of a strategy, we use exploitability $\mathcal{E}(\pi_{i})=u_{-i}^{(\pi_{i},\pi_{-i}^{BR})}(h^{\text{INIT}})-u_{-i}^{\pi^{*}}(h^{\text{INIT}})$, which is how much can opponent gain, when it plays best response as compared to the Nash equilibrium. In two-player zero-sum games, the exploitability is always nonnegative and is zero if and only if the $\pi_{i}$ is a part of Nash equilibrium.

## 3 Learning the Game Model

In perfect information games, players possess complete knowledge of the current game state represented by a history $h$. Consequently, search algorithms initiate from a single, known root state, simplifying the search. In contrast, imperfect information games (IIGs) grant players only partial observability through an information set $s_{i}$, which typically corresponds to multiple possible underlying world states. As established in prior works, approximating optimal strategies via look-ahead reasoning in IIGs requires a more sophisticated approach than in perfect information settings Kovařík et al. (2023); Moravčík et al. (2017).

Specifically, it is insufficient to restrict the reasoning to only those histories consistent with the player’s $i$ information set. Instead, the reasoning must encompass all histories that share the public state $s_{0}$. This necessity arises because sound reasoning algorithms compute strategies for all players simultaneously, aiming for mutual best responses characteristic of an equilibrium. Consider this situation in two-player Poker: if player $i$ holds two Kings, their information set $s_{i}$ includes all histories consistent with this hand but with varying opponent hands. A reasoning restricted only to those histories would implicitly grant the opponent knowledge of $i$’s hand when computing opponent’s strategy, leading to suboptimal strategies. Therefore, the look-ahead reasoning must operate over the broader set of histories consistent with public state to compute valid equilibrium strategies Moravčík et al. (2017); Schmid et al. (2023); Kovařík et al. (2023); Milec et al. (2024).

We adopt a reinforcement learning paradigm where an agent learns from interaction with the environment. During training, we assume access to a game simulator capable of generating the whole game trajectories. During testing (gameplay), the agent receives only its own information set $s_{i}$ at each step and must rely entirely on its learned model to plan, without access to the simulator or explicit rules. This means that the agent does not use any domain-specific knowledge. at any point.

We propose a model inspired by MuZero Schrittwieser et al. (2020) but adapted for the IIG setting, which requires additional structures necessary to model the imperfect information. Our model comprises three core learnable functions, parameterized by $\theta$:

- •
Representation function $\Lambda^{I}_{\theta}:\mathcal{S}_{i}\to\overline{\mathcal{S}_{i}}$. Maps a player $i$’s potentially high-dimensional information set $s_{i}$ to a fixed-size latent representation $\overline{s_{i}}\in\overline{\mathcal{S}_{i}}$.
- •
Dynamics function $\Upsilon_{\theta}:\overline{\mathcal{S}_{1}}\times\overline{\mathcal{S}_{2}}\times\mathcal{A}_{1}\times\mathcal{A}_{2}\to\overline{\mathcal{S}_{1}}\times\overline{\mathcal{S}_{2}}\times\mathbb{R}\times\mathbb{B}$. Given the latent representations for all players ($\overline{s_{1}},\overline{s_{2}}$) and the joint action taken ($a_{1},a_{2}$), this function predicts the resulting next latent representations for both players, the immediate reward $r$ (e.g., for player 1), and a binary termination flag $l$. This models the joint evolution of the game across possible hidden states. We use $\mathbb{B}=\{0,1\}$.
- •
Legal actions function $\Gamma_{\theta}:\overline{\mathcal{S}_{i}}\to\mathbb{B}^{|\mathcal{A}_{i}|}$. Predicts the mask of legal actions $\mathcal{A}_{i}$ available to player $i$ from their latent representation $\overline{s_{i}}$. This is crucial for constraining the look-ahead reasoning only to the feasible parts of the game.

Each training episode a game trajectory is sampled. This trajectory is $h=w^{\text{INIT}}a^{0}\dots a^{l-1}w^{l}$, where $a^{t}=(a_{1}^{t},a_{2}^{t})$ is the joint action at step $t$. For any sub-history $h^{t}\sqsubseteq h$, the simulator provides the true information sets $s_{i}^{t}(h^{t})$, legal actions $\mathcal{A}_{i}^{t}(s_{i}^{t})$, and the reward $r^{t}$. The model is trained to predict these quantities through recurrent application of its components.

Starting from an initial latent state $\overline{s_{i}^{t,0}}=\Lambda^{I}_{\theta}(s_{i}^{t})$, the dynamics function is unrolled for $k$ steps using the actual actions from the trajectory:

$$ $\overline{s_{1}^{t,k+1}},\overline{s_{2}^{t,k+1}},\overline{r^{t,k+1}},\overline{l{}^{t,k+1}}=\Upsilon_{\theta}(\overline{s_{1}^{t,k}},\overline{s_{2}^{t,k}},a_{1}^{t+k},a_{2}^{t+k})$ $$

Here, $\overline{s_{i}^{t,k}}$ is the predicted latent state after $k$ unrolls from step $t$, and $\overline{r^{t,k}}$ and $\overline{l{}^{t,k}}$ are the predicted reward and termination logit. The legal actions function predicts logits $\overline{A_{i}^{t}}=\Gamma_{\theta}(\overline{s_{i}^{t,0}})$ from the initial latent state.

The model parameters $\theta$ are optimized by minimizing a combined loss function over the trajectory:

$$ $\displaystyle\begin{split}\mathcal{L}_{\theta}^{M}=\sum_{t=0}^{l-1}\bigg[&\sum_{i\in\mathcal{N}}\underbrace{\mathcal{L}_{\theta}^{L}(\overline{A_{i}^{t}},\mathcal{A}_{i}^{t})}_{\text{Legal Action Prediction}}\\ +&\sum_{k=1}^{l-t}\Big(\underbrace{\mathcal{L}_{\theta}^{T}(\overline{l^{t,k}},\mathbb{I}[t+k=l])}_{\text{Termination Prediction}}+\underbrace{\mathcal{L}_{\theta}^{R}(\overline{r^{t,k}},r^{t+k})}_{\text{Reward Prediction}}+\sum_{i\in\mathcal{N}}\underbrace{\mathcal{L}_{\theta}^{D}(\overline{s_{i}^{t,k}},\Lambda^{I}_{\theta}(s_{i}^{t+k}))}_{\text{Latent State Prediction}}\Big)\bigg]\end{split}$ (1) $$

where $\mathcal{L}_{\theta}^{L}$ and $\mathcal{L}_{\theta}^{T}$ are binary cross-entropy losses, while $\mathcal{L}_{\theta}^{R}$ and $\mathcal{L}_{\theta}^{D}$ are mean squared errors. The target for the dynamics loss is the latent representation of the actual subsequent information set.

Minimizing $\mathcal{L}_{\theta}^{M}$ trains the functions $\Lambda^{I}_{\theta},\Upsilon_{\theta},\Gamma_{\theta}$ to collectively act as a learned simulator. This learned model captures the necessary dynamics within the game, enabling test-time look-ahead reasoning algorithms (discussed in [Section 5](https://arxiv.org/html/2510.05048v1#S5)) to effectively plan over the required set of public states without recourse to the original game rules or simulator.

## 4 Learning the Abstract Model

A primary limitation of sound look-ahead reasoning in imperfect information games is the potential size of the public states, as the number of information sets consistent with public information may be exponential in the history length, making the look-ahead reasoning intractable Moravčík et al. (2017); Schmid et al. (2023).

Although traditional abstraction techniques often rely on domain expertise or require offline enumeration and analysis of all information sets Čermák et al. (2020); Kroer & Sandholm (2018); Ganzfried & Sandholm (2014); Brown & Sandholm (2015); Bard et al. (2014); Johanson et al. (2013), we aim to learn domain-independent abstraction directly from the game experience during training. Our goal is to partition the vast space of information sets sharing a public state into a manageable number $L$ of abstract information sets, enabling tractable reasoning. Such an abstraction may contain imperfect recall as discussed in [Section 7](https://arxiv.org/html/2510.05048v1#S7).

Consider Texas Hold’em Poker as an example. A player might hold any of the $1326$ private hands, each corresponding to a different information set. Our abstraction aims to represent this multitude using only $L$ representatives, learned based on similarity within the training process rather than predefined rules.

We adapt the model from [Section 3](https://arxiv.org/html/2510.05048v1#S3), but we will use mechanisms inspired by online clustering to limit the amount of information sets in each public state. We hypothesize that information sets behaving similarly, e.g. having similar optimal strategies or leading to similar future states, should be grouped. We formalize this using a function $\kappa:\mathcal{S}_{i}\to\mathbb{R}^{K}$, which maps any information set to a $K$-dimensional space, in which the clustering will be performed.

To satisfy the condition that each public state consists of at most $L$ information sets of each player, we split the representation function into two parts as shown in [Figure 1](https://arxiv.org/html/2510.05048v1#S4.F1). The public state representation $\Lambda_{i,\theta}:\mathcal{S}_{0}\to\overline{\mathcal{S}_{i}}^{L}$ maps a public state $s_{0}$ to $L$ latent abstract information sets for a player $i$. The information set representation $\Lambda^{I}_{i,\theta}:\mathcal{S}_{i}\to\Delta\overline{\mathcal{S}_{i}}$ maps a real information set to the probability distributions on the abstract information sets provided by $\Lambda_{i,\theta}$. Crucially, despite $\Lambda^{I}_{i,\theta}$ providing a probability distribution, we enforce the many-to-one mapping, so we represent any real information set by the single abstract information set, corresponding to the highest probability. This representative will then be used for training dynamics $\Upsilon_{\theta}$ and legal actions $\Gamma_{\theta}$. We opted to this, so that the dynamics function constructs the search tree, which is compatible with look-ahead reasoning algorithms like Counterfactual Regret Minimization.

In order to perform the clustering, we require the same function $\kappa$ as for the real information sets. We introduce $\kappa_{\theta}:\overline{\mathcal{S}_{i}}\to\mathbb{R}^{K}$, which will be trained to predict this clustering property for each abstract information set. The dynamics function $\Upsilon_{\theta}$ and legal actions function $\Gamma_{i,\theta}$ are defined as in [Section 3](https://arxiv.org/html/2510.05048v1#S3).

To learn the abstraction that approximates the proposed clustering based on $\kappa$, we train $\Lambda_{i,\theta},\Lambda^{I}_{i,\theta}$ and $\kappa_{\theta}$ jointly. Each training step, the simulator provides trajectory $h=w^{\text{INIT}}a^{0}\dots a^{l-1}w^{l}$. $s_{0}^{t}$ $s_{i}^{t}$ represents the public state and information set of player $i$ at time $t$ in the same trajectory.

The training involves two additional losses, the first one $\mathcal{L}_{\theta}^{A}$ updates the $\Lambda_{i,\theta}$ and $\kappa_{\theta}$ using a soft clustering objective similar to fuzzy c-means Bezdek et al. (1984). It minimizes a mean squared error between real and abstract properties weighted by the softmax. The second loss $\mathcal{L}_{\theta}^{S}$ updates the $\Lambda^{I}_{\theta}$ by minimizing the cross entropy loss between the predicted probability distribution over the abstract information sets and the one-hot encoded vector of the abstract information set that is the nearest neighbor of the real information set based on the $\kappa$.

$$ $\mathcal{L}_{\theta}^{A}=\sum_{t}^{l-1}\sum_{i\in\mathcal{N}}\sum_{\overline{s_{i}^{t}}\in\Lambda_{i,\theta}(s_{0}^{t})}||\kappa_{\theta}(\overline{s_{i}^{t}})-\kappa(s_{i}^{t})||^{2}\frac{e^{-\gamma||\kappa_{\theta}(\overline{s_{i}^{t}})-\kappa(s_{i}^{t})||^{2}}}{\sum_{\overline{s_{i}^{t}}^{\prime}\in\Lambda_{i,\theta}(s_{0}^{t})}e^{-\gamma||\kappa_{\theta}(\overline{s_{i}^{t}}^{\prime})-\kappa(s_{i}^{t})||^{2}}}$ (2) $$

$$ $\mathcal{L}_{\theta}^{S}=\sum_{t}^{l-1}\sum_{i\in\mathcal{N}}\sum_{\overline{s_{i}^{t}}\in\Lambda_{i,\theta}(s_{0}^{t})}\mathcal{L}_{\theta}^{CE}(\overline{s_{i}^{t}},s_{i}^{t})$ (3) $$

The $\gamma$ controls the softness of the clustering and as $\gamma\to\infty$ the clustering becomes hard. The gradients from $\mathcal{L}_{\theta}^{A}$ propagate through both $\kappa_{\theta}$ and $\Lambda_{\theta}$, but the gradients from $\mathcal{L}_{\theta}^{S}$ are only propagated through $\Lambda^{I}_{\theta}$.

$$ $\mathcal{L}_{\theta}^{MA}=\mathcal{L}_{\theta}^{M}+\mathcal{L}_{\theta}^{A}+\mathcal{L}_{\theta}^{S}$ (4) $$

The overall loss also includes the model learning loss from [Section 3](https://arxiv.org/html/2510.05048v1#S3). Importantly, $\mathcal{L}_{\theta}^{M}$ is computed using the dynamics based on the selected abstract information set. Furthermore, the gradients from $\mathcal{L}_{\theta}^{M}$ are not backpropagated through $\Lambda_{\theta}$, $\Lambda^{I}_{\theta}$ or $\kappa_{\theta}$. This decouples the learning of the abstraction structure from the learning of the model dynamics.

Figure: Figure 1: The public state and information set representations functions of player $i$. First the $\Lambda_{i,\theta}$ predicts 4 abstract information sets and then $\Lambda^{I}_{i,\theta}$ predicts the probability distribution over those abstractions.
Refer to caption: x1.png

## 5 Depth-Limited Solving

While the learned model and abstraction allow reconstructing the whole game tree of the abstracted game, practical applications in large imperfect information games rely on depth-limited reasoning combined with a learned value function to estimate payoffs beyond the reasoning horizon. Defining and training optimal value function for imperfect information games is challenging, as they theoretically depend on belief states Kovařík et al. (2023); Schmid et al. (2023). The belief states are public states with corresponding probability distribution of reaching each history. Such value function is often trained by repeatedly sampling varied belief states and performing depth-limited reasoning in each of them. Such value function may be trained even in our model, but it would require two-phase training process, where first phase trains the abstraction and second the value function.

To enable single-phase training we integrate an approximate value function based on the multi-valued states approach Brown et al. (2018); Kubíček et al. (2024); Milec et al. (2025), which results in an algorithm we call Learned Abstract Model for Imperfect-information Reasoning (LAMIR), which consists of the parts described in Sections [3](https://arxiv.org/html/2510.05048v1#S3) and [4](https://arxiv.org/html/2510.05048v1#S4) and these additional components:

- •
Strategy function $\pi_{\theta}:\mathcal{S}_{i}\to\Delta\mathcal{A}$, which for given information set from the original game returns the strategy trained with some policy-gradient algorithm, like RNaD Perolat et al. (2022).
- •
Transformations function $\tau_{\theta}:\mathcal{S}_{i}\to\mathbb{R}^{|\mathcal{A}_{i}|\times|T|}$, representing $T$ characteristic directions in strategy space explored by the policy-gradient algorithm during training. For a single transformation $\chi\in\tau_{\theta}$ the transformed strategy is computed locally as $\pi_{i}^{\chi}(s_{i},a_{i})=\pi_{i}(s_{i},a_{i})+\chi(s_{i},a_{i})$. The resulting strategy $\pi_{i}^{\chi}(s_{i})$ is then normalized Kubíček et al. (2024).
- •
Value function $v_{\theta}:\overline{\mathcal{S}_{i}}\to\mathbb{R}^{|T|\times|T|}$, which approximates the expected value of each combination of transformed strategies between players Kubíček et al. (2024).

We train $\pi_{\theta}$ and $\tau_{\theta}$ using real information sets from sampled trajectories. However, $v_{\theta}$ is trained using the corresponding joint abstract information sets. Multiple real histories can map to the same abstract state with varying reach probabilities under the sampling strategy. This introduces bias if the value function aims to represent the expected value under the target strategy. While importance sampling could correct this, we found in our experiments it did not affect the results in any significant way. This is most likely due to transformations being just a heuristic approach to approximate different parts of the strategy space.

We used Regularized Nash Dynamics (RNaD) as the policy-gradient algorithm, which includes Neural Replicator Dynamics (NeuRD) loss for strategy training and mean squared error for the associated baseline value function $\mathcal{L}_{\theta}^{PG}$ Hennes et al. (2020); Perolat et al. (2021; 2022). Note, that the value function from the RNaD cannot be used as a value function for look-ahead reasoning, because it does not work with the belief states. We use mean squared error for the training of the value function $\mathcal{L}_{\theta}^{V}$ and the targets are computed by the V-trace Espeholt et al. (2018); Kubíček et al. (2024), which estimates the value of different policies in an off-policy setting.

Following Kubíček et al. (2024), transformations represent the strategy changes during training. For each player $i$ we compute the difference vector $\delta_{i}^{t}=\pi_{i}^{t,\text{NEW}}-\pi_{i}^{t,\text{OLD}}$, where $\pi_{i}^{t,\text{OLD}}$ and $\pi_{i}^{t,\text{NEW}}$ are concatenated strategies along the whole trajectory before and after the training step. Instead of the hard clustering proposed originally, we use the soft clustering from [Section 4](https://arxiv.org/html/2510.05048v1#S4).

$$ $\mathcal{L}_{\theta}^{T}=\sum_{i\in\mathcal{N}}\sum_{\chi_{i}\in T}||\chi_{i}-\delta_{i}^{t}||^{2}\frac{e^{-\gamma||\chi_{i}-\delta_{i}^{t}||^{2}}}{\sum_{\chi_{i}^{\prime}\in T}e^{-\gamma||\chi_{i}^{\prime}-\delta_{i}^{t}||^{2}}}$ (5) $$

Since transformation loss depend on strategy changes after the step of policy-gradient training, we use a two-step update per episode. First we update the Policy-gradient algorithm and then all the other components of the model.

$$ $\mathcal{L}_{\theta}^{1}=\mathcal{L}_{\theta}^{PG}$ (6) $$

$$ $\mathcal{L}_{\theta}^{2}=\mathcal{L}_{\theta}^{MA}+\mathcal{L}_{\theta}^{V}+\mathcal{L}_{\theta}^{T}$ (7) $$

At test time, LAMIR employs continual resolving Moravčík et al. (2017); Schmid et al. (2023) for the acting player $i$. It starts in the initial public state with a single joint abstract information set. The following process then repeats until the terminal state is reached. The algorithm constructs the depth-limited game tree using the trained dynamics and legal actions functions $\Upsilon_{\theta},\Gamma_{\theta}$. At the depth-limit in non-terminal states it adds additional decision layer with $T$ actions for both players. Each combination of actions corresponds to joint transformations and it leads to a terminal state with rewards corresponding to $v_{\theta}$. In this tree, the algorithm performs look-ahead reasoning with Counterfactual Regret Minimization+ (CFR+) Zinkevich et al. (2007); Tammelin (2014). This results in a strategy in each abstract information set in current public state $s_{0}$. The real current information set $s_{i}$ is mapped to the abstracted one $\overline{s_{i}}$ with $\Lambda_{i,\theta}$ and $\Lambda^{I}_{i,\theta}$. The CFR strategy in $\overline{s_{i}}$ is then used to sample the action and move into a new public state $s_{0}$ and ${s_{i}}$. Out of the previous game tree, all the abstracted information sets in $s_{0}$ are used to create a new gadget game Burch et al. (2014). Both counterfactual values and the reaches of the resolving player $i$ are reused from the previous subtree. The algorithm repeats this process until it reaches the terminal state.

## 6 Experiments

### 6.1 Exploitability in Smaller Games

To ensure that the strategies found by LAMIR approximate Nash equilibria, we applied it in games small enough to compute exact exploitability, which serves as a distance from the Nash equilibrium in two-player zero-sum games. For various abstraction sizes $L$ and different properties for clustering $\kappa$ we trained LAMIR with 10 different random seeds for 100,000 episodes. In all of our experiments we used Regularized Nash Dynamics (RNaD) to train the strategy for multi-valued states value function. The rules of the games used in experiments are in Appendix [C](https://arxiv.org/html/2510.05048v1#A3).

Starting at episode 80,000 we have computed exploitability every 1000 episodes. In each public state, the algorithm uses the trained functions to construct the depth-limited subgame with depth limit 1. Then this subgame is solved using CFR+ and the strategy from abstract infosets is mapped to real ones. Then we compute the exploitability of the final composed strategy in the original game. The results for different $\kappa$ and $L$ are displayed in [Figure 2](https://arxiv.org/html/2510.05048v1#S6.F2).

Figure: (a) Imperfect Information Goofspiel 5
Refer to caption: img/goofspiel_5/evaluation_Each_subgame_bar.png

We used three different $\kappa$, which served as a basis for clustering, first the legal actions, second the legal actions with the current RNaD strategy, and third the legal actions, RNaD strategy, and the player’s action history. Each choice of $\kappa$ is capable of outperforming the concurrently trained RNaD with the same amount of training episodes given sufficient $L$. When using the action history as $\kappa$, each information set is uniquely defined in a given public state, which means that, with sufficient capacity, the dynamics network should model the underlying game. This was evaluated in II Goofspiel 5 with $L=30$, which indeed has mapping one-to-one for each abstract and information set. The main reason why the exploitability is greater than 0 arises from the discrepancies in the rewards produced by the dynamics function.

### 6.2 Head-to-head in Large Games

The main usage of LAMIR is in very large games, where the exact exploitability cannot be computed. This experiment evaluates LAMIR in this exact setting, where we compare it with Regularized Nash Dynamics (RNaD) in head-to-head play. For each game and $\kappa$, we have trained LAMIR with 3 different random seeds for 3 million episodes. Similarly, we have trained RNaD with 6 different random seeds for the same number of episodes. The hyperparameter settings remained the same for each game. Then we matched each trained seed of LAMIR with each trained seed of RNaD and played more than 100,000 matches. Note that when we use $\kappa=$ RNaD strategy, it is the strategy that is learned concurrently for the value function and not the RNaD strategy against which the algorithm is compared later. The resulting win rates with 2-sigma error bars are in [Table 1](https://arxiv.org/html/2510.05048v1#S6.T1).

**Table 1: Average win rate with 2-sigma error bars of LAMIR against RNaD in different games.**
| Algorithm | II Goofspiel 10 | II Goofspiel 13 | II Goofspiel 15 |
| --- | --- | --- | --- |
| LAMIR $\kappa=$ Legal actions | 54.47 $\pm$ 0.25 % | 60.68 $\pm$ 0.34 % | 80.49 $\pm$ 0.26 % |
| LAMIR $\kappa=$ RNaD strategy | 61.60 $\pm$ 0.29 % | 58.33 $\pm$ 0.27 % | 61.80 $\pm$ 0.36 % |

LAMIR consistently outperforms RNaD in each of the tested games. Imperfect Information Goofspiel is known for its large public states, so continual resolving without abstractions is not applicable. SePoT Kubíček et al. (2024) was also evaluated in such large domains, but we did not compare against it, as it only uses CFR if the subgame is small enough. The authors showed that in II Goofspiel 13, SePoT has a win rate of only 52%, which is likely caused by not resolving almost any subgame due to the limit on the size of the subgame. Furthermore, we show that even in larger games than those tested with SePoT, LAMIR improves over RNaD even more.

## 7 Limitations and Future Work

LAMIR advances the scalability of the continual resolving paradigm to larger games by integrating learned models and abstractions. However, it presents several interesting avenues for future research.

The computational complexity of the look-ahead reasoning, when using CFR is linear in the amount of information sets in the game. This complexity still remains, as LAMIR only reduces the size of the game in each public state but uses CFR in the abstract game. Each subgame LAMIR construct with depth $D$ contains at most $\sum_{d\in\{0,\dots,D\}}L^{2}|\mathcal{A}|^{2d}$ unique nodes and at most $L^{2}|\mathcal{A}|^{2D}T^{2}$ terminal histories, where $T$ is the amount of transformations. Most of the subgames would be smaller, but this is the main limitation of LAMIR as it limits how large $L$ can be.

Currently, LAMIR’s dynamics network $\Upsilon_{\theta}$ does not explicitly model chance nodes within the game. LAMIR could still be applied in games with chance events, but this is not an intended setting as the absence of chance nodes will result in poor abstraction, regardless of the abstraction capacity $L$. Although algorithms like Stochastic MuZero Antonoglou et al. (2022) demonstrate that modeling chance is feasible in learned models for perfect information games, integrating chance nodes into our framework, particularly in conjunction with learned abstractions, requires careful consideration and is a key area for future work.

The effectiveness of the learned abstraction depends on the chosen property function $\kappa$ for clustering. In games without chance, if this property function could perfectly distinguish two different information sets, $L$ is greater than the size of the largest public state and neural networks have sufficient capacity, then LAMIR learns a near-perfect model of the game. When $L$ is reduced to achieve scalability, the learned abstraction may not capture all crucial strategic nuances, which can affect the strength of the derived strategy. In experiments, we have used simple proxies present in any game and it still yielded strong performance. Ideally, $\kappa$ would also be learned during the training process.

Reducing $L$ necessarily introduces imperfect recall, meaning that the player in the abstract game may ”forget” information it previously knew. Algorithms like Counterfactual Regret Minimization (CFR) guarantee convergence to a Nash equilibrium strategy only in perfect recall games. While CFR’s convergence is not generally guaranteed in imperfect recall settings, it has been shown for subclasses such as A-loss recall games Čermák et al. (2020). Our abstractions are A-loss recall games if the public observations in the original game depend only on prior public information and the actions taken in the current round. Games like Imperfect Information Goofspiel or Oshi-Zumo satisfy these conditions, but many others, including Battleships, Dark Chess, or Stratego, do not. Thus, for games that do not fall into the A-loss recall category after abstraction, theoretical convergence guarantees for CFR-based methods within LAMIR are not assured.

LAMIR focuses on abstracting information sets but does not inherently abstract action spaces. In games with very large or continuous action spaces (e.g., bet sizing in Poker), the sheer number of actions can remain a bottleneck for the look-ahead reasoning, regardless of the information set abstraction. While action abstractions have been extensively studied Brown & Sandholm (2014; 2018); Li et al. (2024), integrating them with LAMIR is out of the scope of single paper, but it presents another direction for future improvement.

## 8 Conclusion

In this paper, we have introduced LAMIR, an algorithm designed to enable look-ahead reasoning by learning the model of the game with a suitable abstraction directly from experience, without the need for any domain-specific knowledge.

Our core contributions are fourfold. First, we have proposed a method for learning the fundamental components of a model of dynamics of imperfect information games without chance. Second, we developed a technique for automatically learning an abstraction by clustering information sets, effectively reducing the size of the game. Third, we integrated these components with a learning of value function in the abstracted game that enables depth-limited look-ahead reasoning. Lastly, we demonstrate how LAMIR facilitates the continual resolving paradigm by performing a depth-limited look-ahead reasoning in each decision node encountered.

We empirically verify that, when given sufficient capacity, LAMIR learns a nearly perfect model. Still, the game-playing capabilities degrade gracefully when the abstraction capacity is reduced. We have also shown that LAMIR manages to perform look-ahead reasoning even in intractably large public states. Thanks to that, it achieved up to 80% win rate in large games compared to RNaD, a strong baseline that was successfully used to create a human-expert level player in Stratego.

The primary impact of LAMIR lies in scaling look-ahead reasoning techniques to larger games by learning abstraction and its model directly from experience. This overcomes the most notable limitations of the continual resolving paradigm, which requires each subgame to be tractable and to have explicit access to the game model. LAMIR has several limitations, which can be addressed in future work. Still, LAMIR is, to the best of our knowledge, the first algorithm that enables look-ahead reasoning in large-scale games like Imperfect Information Goofspiel 15 without any domain-specific knowledge, substantially outperforming the model-free policy-gradient algorithm.

## Acknowledgments

This research is supported by Czech Science Foundation
(GA25-18353S) and Grant Agency of the CTU in Prague
(SGS23/184/OHK3/3T/13). Computational resources were supplied
by (e-INFRA CZ LM2018140) supported by the Ministry of
Education, Youth and Sports of the Czech Republic and also
(CZ.02.1.01/0.0/0.0/16 019/0000765)

## References

- Antonoglou et al. (2022)
Ioannis Antonoglou, Julian Schrittwieser, Sherjil Ozair, Thomas K Hubert, and David Silver.
Planning in stochastic environments with a learned model.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=X6D9bAHhBQ1](https://openreview.net/forum?id=X6D9bAHhBQ1).
- Bard et al. (2014)
Nolan Bard, Michael Johanson, and Michael Bowling.
Asymmetric abstractions for adversarial settings.
In *Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems*, pp. 501–508, 2014.
- Bezdek et al. (1984)
James C Bezdek, Robert Ehrlich, and William Full.
Fcm: The fuzzy c-means clustering algorithm.
*Computers & geosciences*, 10(2-3):191–203, 1984.
- Bradbury et al. (2018)
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.
JAX: composable transformations of Python+NumPy programs, 2018.
URL [http://github.com/jax-ml/jax](http://github.com/jax-ml/jax).
- Brown & Sandholm (2014)
Noam Brown and Tuomas Sandholm.
Regret transfer and parameter optimization.
In *Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence*, AAAI’14, pp. 594–601. AAAI Press, 2014.
- Brown & Sandholm (2015)
Noam Brown and Tuomas Sandholm.
Simultaneous abstraction and equilibrium finding in games.
In *International Joint Conference on Artificial Intelligence*, 2015.
URL [https://api.semanticscholar.org/CorpusID:12321994](https://api.semanticscholar.org/CorpusID:12321994).
- Brown & Sandholm (2018)
Noam Brown and Tuomas Sandholm.
Superhuman ai for heads-up no-limit poker: Libratus beats top professionals.
*Science*, 359(6374):418–424, 2018.
doi: 10.1126/science.aao1733.
URL [https://www.science.org/doi/abs/10.1126/science.aao1733](https://www.science.org/doi/abs/10.1126/science.aao1733).
- Brown et al. (2018)
Noam Brown, Tuomas Sandholm, and Brandon Amos.
Depth-limited solving for imperfect-information games.
In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), *Advances in Neural Information Processing Systems*, volume 31. Curran Associates, Inc., 2018.
URL [https://proceedings.neurips.cc/paper_files/paper/2018/file/34306d99c63613fad5b2a140398c0420-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2018/file/34306d99c63613fad5b2a140398c0420-Paper.pdf).
- Brown et al. (2019)
Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm.
Deep counterfactual regret minimization.
In *International conference on machine learning*, pp. 793–802. PMLR, 2019.
- Brown et al. (2020)
Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong.
Combining deep reinforcement learning and search for imperfect-information games.
*Advances in neural information processing systems*, 33:17057–17069, 2020.
- Burch et al. (2014)
Neil Burch, Michael Johanson, and Michael Bowling.
Solving imperfect information games using decomposition.
In *Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence*, AAAI’14, pp. 602–608. AAAI Press, 2014.
- DeepMind et al. (2020)
DeepMind, Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Antoine Dedieu, Claudio Fantacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, Iurii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, George Papamakarios, John Quan, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Laurent Sartran, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Miloš Stanojević, Wojciech Stokowiec, Luyu Wang, Guangyao Zhou, and Fabio Viola.
The DeepMind JAX Ecosystem, 2020.
URL [http://github.com/google-deepmind](http://github.com/google-deepmind).
- Espeholt et al. (2018)
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al.
Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.
In *International conference on machine learning*, pp. 1407–1416. PMLR, 2018.
- Ganzfried & Sandholm (2014)
Sam Ganzfried and Tuomas Sandholm.
Potential-aware imperfect-recall abstraction with earth mover’s distance in imperfect-information games.
*Proceedings of the AAAI Conference on Artificial Intelligence*, 28(1), Jun. 2014.
doi: 10.1609/aaai.v28i1.8816.
URL [https://ojs.aaai.org/index.php/AAAI/article/view/8816](https://ojs.aaai.org/index.php/AAAI/article/view/8816).
- Hafner et al. (2020)
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.
Dream to control: Learning behaviors by latent imagination.
In *International Conference on Learning Representations*, 2020.
URL [https://openreview.net/forum?id=S1lOTC4tDS](https://openreview.net/forum?id=S1lOTC4tDS).
- Hafner et al. (2021)
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba.
Mastering atari with discrete world models.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=0oabwyZbOu](https://openreview.net/forum?id=0oabwyZbOu).
- Hafner et al. (2025)
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap.
Mastering diverse control tasks through world models.
*Nature*, pp. 1–7, 2025.
- Heek et al. (2024)
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee.
Flax: A neural network library and ecosystem for JAX, 2024.
URL [http://github.com/google/flax](http://github.com/google/flax).
- Hennes et al. (2020)
Daniel Hennes, Dustin Morrill, Shayegan Omidshafiei, Rémi Munos, Julien Perolat, Marc Lanctot, Audrunas Gruslys, Jean-Baptiste Lespiau, Paavo Parmas, Edgar Duèñez Guzmán, and Karl Tuyls.
Neural replicator dynamics: Multiagent learning via hedging policy gradients.
In *Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems*, AAMAS ’20, pp. 492–501, Richland, SC, 2020. International Foundation for Autonomous Agents and Multiagent Systems.
ISBN 9781450375184.
- Johanson et al. (2013)
Michael Johanson, Neil Burch, Richard Valenzano, and Michael Bowling.
Evaluating state-space abstractions in extensive-form games.
In *Proceedings of the 2013 International Conference on Autonomous Agents and Multi-Agent Systems*, AAMAS ’13, pp. 271–278, Richland, SC, 2013. International Foundation for Autonomous Agents and Multiagent Systems.
ISBN 9781450319935.
- Kovařík et al. (2022)
Vojtěch Kovařík, Martin Schmid, Neil Burch, Michael Bowling, and Viliam Lisỳ.
Rethinking formal models of partially observable multiagent decision making.
*Artificial Intelligence*, 303:103645, 2022.
- Kovařík et al. (2023)
Vojtěch Kovařík, Dominik Seitz, Viliam Lisý, Jan Rudolf, Shuo Sun, and Karel Ha.
Value functions for depth-limited solving in zero-sum imperfect-information games.
*Artificial Intelligence*, 314:103805, 2023.
- Kroer & Sandholm (2018)
Christian Kroer and Tuomas Sandholm.
A unified framework for extensive-form game abstraction with bounds.
In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), *Advances in Neural Information Processing Systems*, volume 31. Curran Associates, Inc., 2018.
URL [https://proceedings.neurips.cc/paper_files/paper/2018/file/aa942ab2bfa6ebda4840e7360ce6e7ef-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2018/file/aa942ab2bfa6ebda4840e7360ce6e7ef-Paper.pdf).
- Kubíček et al. (2024)
Ondřej Kubíček, Neil Burch, and Viliam Lisý.
Look-ahead search on top of policy networks in imperfect information games.
In Kate Larson (ed.), *Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24*, pp. 4344–4352. International Joint Conferences on Artificial Intelligence Organization, 8 2024.
doi: 10.24963/ijcai.2024/480.
URL [https://doi.org/10.24963/ijcai.2024/480](https://doi.org/10.24963/ijcai.2024/480).
Main Track.
- Kuhn & Tucker (1951)
Harold W. Kuhn and Albert William Tucker (eds.).
*Contributions to the Theory of Games, Volume I*.
Princeton University Press, Princeton, 1951.
ISBN 9781400881727.
doi: doi:10.1515/9781400881727.
URL [https://doi.org/10.1515/9781400881727](https://doi.org/10.1515/9781400881727).
- Lanctot et al. (2017)
Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Perolat, David Silver, and Thore Graepel.
A unified game-theoretic approach to multiagent reinforcement learning.
In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), *Advances in Neural Information Processing Systems*, volume 30. Curran Associates, Inc., 2017.
URL [https://proceedings.neurips.cc/paper_files/paper/2017/file/3323fe11e9595c09af38fe67567a9394-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2017/file/3323fe11e9595c09af38fe67567a9394-Paper.pdf).
- Lanctot et al. (2019)
Marc Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Saeta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schrittwieser, Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan-Davis.
OpenSpiel: A framework for reinforcement learning in games.
*CoRR*, abs/1908.09453, 2019.
URL [http://arxiv.org/abs/1908.09453](http://arxiv.org/abs/1908.09453).
- Li et al. (2024)
Boning Li, Zhixuan Fang, and Longbo Huang.
Rl-cfr: improving action abstraction for imperfect information extensive-form games with reinforcement learning.
In *Proceedings of the 41st International Conference on Machine Learning*, ICML’24. JMLR.org, 2024.
- Lisy & Bowling (2016)
Viliam Lisy and Michael Bowling.
Equilibrium approximation quality of current no-limit poker bots.
*arXiv preprint arXiv:1612.07547*, 2016.
- Liu et al. (2023)
Weiming Liu, Haobo Fu, Qiang Fu, and Wei Yang.
Opponent-limited online search for imperfect information games.
In *Proceedings of the 40th International Conference on Machine Learning*, ICML’23. JMLR.org, 2023.
- Masaka et al. (2025)
Wataru Masaka, Mitsuki Sakamoto, Kenshi Abe, Kaito Ariu, Tuomas Sandholm, and Atsushi Iwasaki.
The power of perturbation under sampling in solving extensive-form games.
*arXiv preprint arXiv:2501.16600*, 2025.
- Milec et al. (2024)
David Milec, Ondřej Kubíček, and Viliam Lisý.
Continual depth-limited responses for computing counter-strategies in sequential games.
In *Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems*, AAMAS ’24, pp. 2393–2395, Richland, SC, 2024. International Foundation for Autonomous Agents and Multiagent Systems.
ISBN 9798400704864.
- Milec et al. (2025)
David Milec, Vojtěch Kovařík, and Viliam Lisý.
Adapting beyond the depth limit: Counter strategies in large imperfect information games, 2025.
URL [https://arxiv.org/abs/2501.10464](https://arxiv.org/abs/2501.10464).
- Mnih et al. (2015)
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.
Human-level control through deep reinforcement learning.
*nature*, 518(7540):529–533, 2015.
- Moravčík et al. (2017)
Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling.
Deepstack: Expert-level artificial intelligence in heads-up no-limit poker.
*Science*, 356(6337):508–513, 2017.
doi: 10.1126/science.aam6960.
URL [https://www.science.org/doi/abs/10.1126/science.aam6960](https://www.science.org/doi/abs/10.1126/science.aam6960).
- Nash (1950)
John F. Nash.
Equilibrium points in ¡i¿n¡/i¿-person games.
*Proceedings of the National Academy of Sciences*, 36(1):48–49, 1950.
doi: 10.1073/pnas.36.1.48.
URL [https://www.pnas.org/doi/abs/10.1073/pnas.36.1.48](https://www.pnas.org/doi/abs/10.1073/pnas.36.1.48).
- Perolat et al. (2021)
Julien Perolat, Remi Munos, Jean-Baptiste Lespiau, Shayegan Omidshafiei, Mark Rowland, Pedro Ortega, Neil Burch, Thomas Anthony, David Balduzzi, Bart De Vylder, et al.
From poincaré recurrence to convergence in imperfect information games: Finding equilibrium via regularization.
In *International Conference on Machine Learning*, pp. 8525–8535. PMLR, 2021.
- Perolat et al. (2022)
Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T Connor, Neil Burch, Thomas Anthony, et al.
Mastering the game of stratego with model-free multiagent reinforcement learning.
*Science*, 378(6623):990–996, 2022.
- Russell & Norvig (2016)
Stuart J Russell and Peter Norvig.
*Artificial intelligence: a modern approach*.
pearson, 2016.
- Schmid et al. (2019)
Martin Schmid, Neil Burch, Marc Lanctot, Matej Moravcik, Rudolf Kadlec, and Michael Bowling.
Variance reduction in monte carlo counterfactual regret minimization (vr-mccfr) for extensive form games using baselines.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 33, pp. 2157–2164, 2019.
- Schmid et al. (2023)
Martin Schmid, Matej Moravčík, Neil Burch, Rudolf Kadlec, Josh Davidson, Kevin Waugh, Nolan Bard, Finbarr Timbers, Marc Lanctot, G. Zacharias Holland, Elnaz Davoodi, Alden Christianson, and Michael Bowling.
Student of games: A unified learning algorithm for both perfect and imperfect information games.
*Science Advances*, 9(46):eadg3256, 2023.
doi: 10.1126/sciadv.adg3256.
URL [https://www.science.org/doi/abs/10.1126/sciadv.adg3256](https://www.science.org/doi/abs/10.1126/sciadv.adg3256).
- Schrittwieser et al. (2020)
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver.
Mastering Atari, Go, chess and shogi by planning with a learned model.
*Nature*, 588(7839):604–609, 2020.
ISSN 0028-0836.
doi: 10.1038/s41586-020-03051-4.
- Schultz et al. (2025)
John Schultz, Jakub Adamek, Matej Jusup, Marc Lanctot, Michael Kaisers, Sarah Perrin, Daniel Hennes, Jeremy Shar, Cannada Lewis, Anian Ruoss, Tom Zahavy, Petar Veličkovic, Laurel Prince, Satinder Singh, Eric Malmi, and Nenad Tomašev.
Mastering board games by external and internal planning with language models, 2025.
URL [https://arxiv.org/abs/2412.12119](https://arxiv.org/abs/2412.12119).
- Silver et al. (2016)
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.
Mastering the game of go with deep neural networks and tree search.
*nature*, 529(7587):484–489, 2016.
- Silver et al. (2018)
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis.
A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.
*Science*, 362(6419):1140–1144, 2018.
doi: 10.1126/science.aar6404.
URL [https://www.science.org/doi/abs/10.1126/science.aar6404](https://www.science.org/doi/abs/10.1126/science.aar6404).
- Snell et al. (2024)
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar.
Scaling llm test-time compute optimally can be more effective than scaling model parameters.
*arXiv preprint arXiv:2408.03314*, 2024.
- Sokota et al. (2023)
Samuel Sokota, Ryan D’Orazio, Chun Kai Ling, David J. Wu, J. Zico Kolter, and Noam Brown.
Abstracting imperfect information away from two-player zero-sum games.
In *International Conference on Machine Learning (ICML)*, 2023.
- Steinberger et al. (2020)
Eric Steinberger, Adam Lerer, and Noam Brown.
Dream: Deep regret minimization with advantage baselines and model-free learning.
*arXiv preprint arXiv:2006.10410*, 2020.
- Tammelin (2014)
Oskari Tammelin.
Solving large imperfect information games using cfr+.
*arXiv preprint arXiv:1407.5042*, 2014.
- Van Rossum & Drake (2009)
Guido Van Rossum and Fred L. Drake.
*Python 3 Reference Manual*.
CreateSpace, Scotts Valley, CA, 2009.
ISBN 1441412697.
- Čermák et al. (2020)
Jiří Čermák, Viliam Lisý, and Branislav Bošanský.
Automated construction of bounded-loss imperfect-recall abstractions in extensive-form games.
*Artificial Intelligence*, 282:103248, 2020.
ISSN 0004-3702.
doi: https://doi.org/10.1016/j.artint.2020.103248.
URL [https://www.sciencedirect.com/science/article/pii/S0004370220300126](https://www.sciencedirect.com/science/article/pii/S0004370220300126).
- Vinyals et al. (2019)
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al.
Grandmaster level in starcraft ii using multi-agent reinforcement learning.
*nature*, 575(7782):350–354, 2019.
- Wang et al. (2023)
Tony Tong Wang, Adam Gleave, Tom Tseng, Kellin Pelrine, Nora Belrose, Joseph Miller, Michael D Dennis, Yawen Duan, Viktor Pogrebniak, Sergey Levine, et al.
Adversarial policies beat superhuman go ais.
In *International Conference on Machine Learning*, pp. 35655–35739. PMLR, 2023.
- Zhang & Sandholm (2021)
Brian Hu Zhang and Tuomas Sandholm.
Subgame solving without common knowledge.
In *Proceedings of the 35th International Conference on Neural Information Processing Systems*, NIPS ’21, Red Hook, NY, USA, 2021. Curran Associates Inc.
ISBN 9781713845393.
- Zhang & Sandholm (2025)
Brian Hu Zhang and Tuomas Sandholm.
General search techniques without common knowledge for imperfect-information games, and application to superhuman fog of war chess, 2025.
URL [https://arxiv.org/abs/2506.01242](https://arxiv.org/abs/2506.01242).
- Zinkevich et al. (2007)
Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione.
Regret minimization in games with incomplete information.
In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), *Advances in Neural Information Processing Systems*, volume 20. Curran Associates, Inc., 2007.
URL [https://proceedings.neurips.cc/paper_files/paper/2007/file/08d98638c6fcd194a4b1e6992063e944-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2007/file/08d98638c6fcd194a4b1e6992063e944-Paper.pdf).

## Appendix A Experimental details

Our implementation of LAMIR was done in Python 3.12.3 Van Rossum & Drake (2009) with libraries from JAX ecosystem Bradbury et al. (2018); DeepMind et al. (2020); Heek et al. (2024) for automatic differentiation and GPU acceleration. Our implementation of RNaD is derived out of the implementation in OpenSpiel Lanctot et al. (2019). All experiments were run in cluster with several GPUs Nvidia Tesla A100 and CPUs AMD EPYC 7543 with 1TB memory. Each experiment always used only one CPU core and one GPU at most. The training used at most 8GBs of memory per training. Gameplay evaluation and exploitability computation took at most 64GBs of memory. The hyperparameters used are in [Tables 2](https://arxiv.org/html/2510.05048v1#A1.T2) and [3](https://arxiv.org/html/2510.05048v1#A1.T3)

### A.1 Soft clustering changes

When we have used the soft clustering as described in [Section 4](https://arxiv.org/html/2510.05048v1#S4). However, we have encountered that in some instances when the abstraction limit $L$ was greater than the amount of information sets in a public state, each cluster center. This caused a problem with the dynamics network, which sometimes mapped to two different continuations and therefore revealed some information to the player. Since some information sets are sampled more often than others, it may happen that the soft clustering pulls several clusters more to this sample than to others. To avoid these problems, we have introduced three changes to the clustering loss. First, we added an additional loss $\mathcal{L}_{\theta}^{\text{Rep}}$ that repels the clusters that are closer than $d_{r}$. Second, we changed $\mathcal{L}_{\theta}^{A}$ so that if some cluster is closer than $d_{h}$, we changed the soft clustering to hard, e.g. we move only the closest cluster, for that particular sample. Third, we added Gaussian noise to each sampled point with mean 0 and scale $\sigma=0.02$.

$$ $\mathcal{L}_{\theta}^{\text{Rep}}=\sum_{t}^{l-1}\sum_{i\in\mathcal{N}}\sum_{\overline{s_{i}^{t}}\in\Lambda_{i,\theta}(s_{0}^{t})}\sum_{\overline{s_{i}^{\prime t}}\in\Lambda_{i,\theta}(s_{0}^{t})}\max\{0,d_{r}-||\kappa_{\theta}(\overline{s_{i}^{t}})-\kappa_{\theta}(\overline{s_{i}^{\prime t}})||^{2}\}$ (8) $$

### A.2 Public state decoder

With the abstraction loss $\mathcal{L}_{\theta}^{A}$ as we defined it in [Section 4](https://arxiv.org/html/2510.05048v1#S4), it may happen that two different public states will produce the same abstract information set. This may not cause any issues, but we decided to avoid it by training a decoder $\Lambda^{-1}_{i}:\overline{\mathcal{S}_{i}}\to\mathcal{S}_{0}$ with loss of L2, which is also propagated through $\Lambda_{i}$. As a result in each $\overline{s_{i}}$ is embedded the information to which public state it belongs. In other words, it also means that information about the public state is also used within $\kappa$, but since it is the same for each information set with the same public state, it does not affect the clustering.

### A.3 Off-policy Regularized Nash Dynamics

The Regularized Nash Dynamics (RNaD) algorithm, which builds upon the Follow the Regularized Leader paradigm, typically updates a player’s strategy based on an advantage function Perolat et al. (2021). In the successful Stratego implementation (DeepNash), the authors adapted this by training an information set value function, which returns a scalar value for an information set. This function approximates the expected game outcome if all players follow the current network strategy in the whole game (both before and after this decision point). This value was then used to derive an advantage function for policy updates, maintaining convergence guarantees within their on-policy training framework.

A strictly on-policy approach may suffer from insufficient exploration, potentially leaving some parts of the game state unvisited. As LAMIR aims to find an abstraction in each part of the game, we employed an off-policy sampling strategy. Specifically, we used $\epsilon$-on-policy sampling: at each decision point, an action was chosen uniformly at random with probability $\epsilon$, or sampled from the current network strategy with probability $1-\epsilon$. This ensures that all parts of the game can be visited with non-zero probability.

However, this off-policy sampling introduces a challenge: a value function trained naively would estimate the value of the $\epsilon$-on-policy sampling strategy, rather than the target network strategy. To obtain an unbiased estimator of the network strategy we trained a history value function. This value function was trained using targets derived from the V-trace off-policy estimator Espeholt et al. (2018), which corrects for the discrepancy between the sampling policy and the target policy. The advantage for taking an action at information set was then computed similarly to how RNaD derives advantages from an information set value function Schmid et al. (2019); Perolat et al. (2022). Specifically, the advantage relies on counterfactual values - the expected outcome if a specific action is taken and then the network strategy is followed, weighted by the probability opponent have played to this decision point. To ensure unbiased estimates of these counterfactual values under our off-policy sampling scheme, the estimation of this value uses a importance sampling correction for each opponent decision preceding this value Schmid et al. (2019); Masaka et al. (2025).

### A.4 Dynamics network

n our experiments, we worked with games exhibiting a specific property: the public observation (and thus the public state) at the next step depends only on the previous public state and the joint action taken by all players. In other words, the transition of the public state is independent of the private information distinguishing different information sets within the same current public state.

We leveraged this property to refine our dynamics network ($\Upsilon_{\theta}$). Instead of directly predicting the subsequent abstract information set representation for each player, the modified dynamics function performs a two-stage prediction:

- 1.
Predicts the next public state identifier ($s_{0}^{\prime}$) within the original game.
- 2.
For each player $i$, it predicts a probability distribution over the $L$ abstract information set within that predicted next public state $s_{0}^{\prime}$.

To determine the actual subsequent abstract information set representation for player $i$, we first use public state representation function $\Lambda_{i,\theta}(s_{0}^{\prime})$ to get the $L$ abstract information sets. Then we select the $\overline{s_{i}}$, which corresponds to the highest probability from the dynamics network (argmax selection). This modification effectively separates the prediction of the public state transition from the prediction of the players’ abstracted private states within that future public context. It can simplify the learning task for the dynamics network when the underlying game structure supports this decomposition.

**Table 2: Common hyperparameters**
| Group | Parameter | Value |
| --- | --- | --- |
| RNaD | Regularization $\eta$ | 0.2 |
| NeuRD threshold $\beta$ | 2 |  |
| NeuRD clipping $c$ | 10000 |  |
| V-trace clipping $c$ | 1.0 |  |
| V-trace clipping $\rho$ | $\infty$ |  |
| Target network update $\gamma_{a}$ | $10^{-3}$ |  |
| Off-policy sampling $\epsilon$ | 0.5 |  |
| Amount of transformations $T$ | 10 |  |
| Training | Learning rate | $3\cdot 10^{-4}$ |
| Architecture | MLP |  |
| Activation functions | ReLU |  |
| Optimizer | ADAM (ADAMW for $\kappa_{\theta}$) |  |
| Adam decay rate $\beta_{1}$ | 0.9 (0 for $\pi_{\theta}$) |  |
| Adam decay rate $\beta_{2}$ | 0.999 |  |
| Weight decay of ADAMW | $10^{-5}$ |  |
| Gradient clipping | 100 (1 for $\kappa_{\theta},\tau_{\theta}$) |  |
| Abstractions | Sample noise scale $\sigma$ | 0.02 |
| Soft clustering | Softmax temperature $\gamma$ | 1 |
| Hard clustering threshold $d_{h}$ | 0.3 |  |
| Closest cluster distance $d_{r}$ | 0.5 |  |
| Look-ahead reasoning | CFR+ iterations | 1000 |
| Depth limit | 1 |  |

**Table 3: Specific hyperparameters for each experiment.**
| Parameter | Exploitability | Leduc | Head-to-head |
| --- | --- | --- | --- |
| Architecture Parameters |  |  |  |
| $\pi_{\theta}$ MLP layers | 256, 256 | 256, 256 | 512, 512 |
| $v_{\theta}$ MLP layers | 512, 512 | 512, 512 | 4096, 4096 |
| $\Lambda^{I}_{\theta}$ MLP layers | 256, 256 | 256, 256 | 512, 512 |
| $\Lambda_{\theta}$ MLP layers | 1024, 1024 | 256, 256 | 4096, 4096 |
| $\Gamma_{\theta}$ MLP layers | 128, 128 | 128, 128 | 512, 512 |
| $\Upsilon_{\theta}$ MLP layers | 256, 256 | 256, 256 | 2048, 2048 |
| $\kappa_{\theta}$ MLP layers | 256, 256 | 256, 256 | 1024, 1024 |
| $\tau_{\theta}$ MLP layers | 512, 512 | 512, 512 | 4096, 4096 |
| $\Lambda^{-1}_{\theta}$ MLP layers | 128, 128 | 128, 128 | 512, 512 |
| Training Parameters |  |  |  |
| Batch size | 64 | 128 | 128 |
| Episodes | $10^{5}$ | $10^{5}$ | $3\cdot 10^{6}$ |
| Other Parameters |  |  |  |
| Abstract information set dimension | 64 | 64 | 256 |
| Regularization policy change each | 1000 | 1000 | 20000 |

## Appendix B Additional experimental details

### B.1 Tabular K-means

In this experiment, we evaluate abstraction performance separately to show that tabular clustering of the real information sets in each public state to a limited number of abstract ones leads to graceful degradation of the quality of the strategies. As a clustering property $\kappa$, we have used either legal action or strategy computed by 4,000 iterations of Counterfactual Regret Minimization. We performed the tabular K-means as a clustering algorithm and then we constructed the original game tree, while changing the information structure to use the abstraction. This new abstract game was solved using CFR and then the exploitability of this final strategy was computed in the original game. The results are displayed in [Figure 3](https://arxiv.org/html/2510.05048v1#A2.F3).

Figure: (a) Goofspiel 4
Refer to caption: img/goofspiel_4/kmeans_exploitability.png

The maximal amount of information sets per player in public state is 7, 30 and 168 for Goofspiel 4, 5 and 6 respectively. Increasing the abstraction limit beyond 10 in Goofspiel 5 and 20 in Goofspiel 6 increases the performance only slightly, so it suggests that this size is sufficient in those games. We have used this knowledge in our large experiments and we set the abstraction limit $L=20$. Also, it is important to note that legal actions do not provide sufficient information to create an optimal abstraction in larger Goofspiels. However, in large games, as tested in [Section 6.2](https://arxiv.org/html/2510.05048v1#S6.SS2), it still outperformed the RNaDs strategy as $\kappa$.

#### Computational resources:

For each value of $K$ in the experiment, we have used a single core of AMD EPYC 7543 for all 10 seeds. For each $K$ in Goofspiel 4, the computation took less than a minute, in Goofspiel 5 it took less than hour and in Goofspiel 6 it was less than 8 hours. However, all of those were ran in parallel. Approximately the resulting computational time was at most 1375 CPU hours.

### B.2 Exploitability in Smaller Games

We provide additional experimental results for [Section 6.1](https://arxiv.org/html/2510.05048v1#S6.SS1), which were omitted from the main body of the paper due to limited space. We show a performance of different $\kappa$ from those in the main body and also different ablation studies when computing exploitability from the same training runs.

[Figure 4](https://arxiv.org/html/2510.05048v1#A2.F4) contains the exploitability of the abstraction in three different settings. First, we construct the original game tree, and then we map each infoset with public state representation function $\Lambda_{\theta}$ and information set representation $\Lambda^{I}_{\theta}$ to the abstract one. Then, we solved this changed game with CFR+ and computed the exploitability of this new strategy in the original one. The Second was to use a dynamics network to construct the whole game, which is equivalent to using $\infty$ depth-limit and then using CFR+ in this game. This setting does not use a value function, so it evaluates the quality of the abstraction $\Lambda_{\theta}$ and the dynamics $\Upsilon_{\theta}$. Third, we used the LAMIR, which is the same setting as in [Section 4](https://arxiv.org/html/2510.05048v1#S4).

In Goofspiel 5, the experiments have shown that using just the RNaD strategy $\pi_{\theta}$, which was trained for the $v_{\theta}$ for clustering $\kappa$ performs worse than the RNaD itself. This is partially due to changes to the clustering to avoid collapsing clusters together, which we discussed in [Section A.1](https://arxiv.org/html/2510.05048v1#A1.SS1) and also because the strategy itself was not stationary during the training, as it changed from the RNaD dynamics.

Using just the abstraction and then mapping it to the original game shows that with sufficient capacity, the learned abstraction mirrors the game’s underlying structure, as evidenced by $L=30$ in Goofspiel. In Oshi-Zumo, even with $L\geq 5$ it is enough to model the game perfectly when using the legal actions for clustering $\kappa$. This suggests that even if Oshi-Zumo is quite a large game, as the largest public state contains 625 information sets, it is not important to distinguish between them.

Constructing the whole game tree sometimes produces worse results than using the continual resolving. This mainly happens if the abstraction is worse than the value function, which may fix some mistakes that the poor abstraction caused only further in the game. This mainly happened in Oshi-Zumo. In goofspiel it occurred only when using only the action history as $\kappa$.

Figure: (a) Goofspiel 5
Refer to caption: img/goofspiel_5/evaluation_Only_abstraction_bar_full.png

#### Computational resources:

Training of each seed, abstraction limit $L$, clustering property $\kappa$ and game was ran on a single GPU Nvidia Tesla A100 for less than an hour. Then each checkpoint made during this training was evaluated sequentially on a single CPU AMD EPYC 7543 for each ablation study. So in total this experiment cost 390 GPU hours and 6240 CPU hours.

### B.3 Leduc Hold’em

We have also evaluated LAMIR in a small version of Poker, Leduc Hold’em, which is a popular benchmark. LAMIR cannot model the chance nodes and is not intended to be used for such games. However, with some domain-specific workarounds, we were able to use LAMIR even for Leduc. These workarounds were only used in the test part to unroll the chance nodes out of game rules of Leduc. We have used the same training setting as in [Section 6.1](https://arxiv.org/html/2510.05048v1#S6.SS1) with 100,000 episodes and evaluated the last 21 checkpoints each 1000 episodes. We evaluated the same trained models in two settings. In both we solved separately part of the game before dealing a public card, by using a value function at the depth-limit and after dealing a public card. In one setting, we have created the subgame from the rules of the game and only mapped the information structure from the abstraction; in the second, we only used the rules of the game to construct states after the chance nodes and then used the dynamics network to construct the rest of the subgames. The results are displayed in [Figure 5](https://arxiv.org/html/2510.05048v1#A2.F5).

Figure: Figure 5: Exploitability in Leduc Hold’em either by constructing the subgame from the rules of the game, or from the dynamics network
Refer to caption: img/leduc/mean_results.png

In Leduc Hold’em, each public state has 6 information sets for each player. So for $L\geq 6$, LAMIR learns the underlying game. However, even with $L=5$, it is still capable of outperforming RNaD. Decreasing the size further degrades the performance, even if the abstraction $L=3$ should be sufficient due to the invariance in the card suits. This suggests that improving $\kappa$ and the clustering may produce better results.

#### Computational resources:

We have again used for a training of a single seed with abstraciton limit just a single GPU Nvidia Tesla A100 for at most hour. Each checkpoint was then evaluated on a single CPU AMD EPYC 7543 for each of those 2 ablation studies in less than 2 hours. In total, the computational cost was 60 GPU hours and 120 CPU hours.

### B.4 Head-to-head in Larger Games

#### Computational resources:

The training of each seed and clustering function $\kappa$ for both RNaD and LAMIR was done in parallel each on a single GPU Nvidia Tesla A100. The training took 24 hours. Heads-to-heads was done performed in parallel for each final saved model from training. The evaluation of each pair took 192 hours on a CPU AMD EPYC 7543. The total computational cost was 864 GPU hours and 20736 CPU hours.

## Appendix C Game rules

### C.1 Leduc Hold’em

Leduc Hold’em is a simplified version of Texas Hold’em poker. It is played by two players with a deck of 6 cards in two suits: Spades and Hearts. Each suit has three ranks: Jack, Queen, and King. At the beginning of the game, each player performs a mandatory bet of 1 coin to the pot, and the dealer deals privately one card to each player. The game then proceeds to the first betting round, where players take turns by either folding, calling, or raising. If some player folds, the game immediately ends, and the other player receives all the coins in the pot. If any player calls, it puts as many coins into the pot so that the total amount of coins put in by both players is the same, and the game proceeds to the next round. If a player raises, it puts the same amount of coins and two more coins into the pot as during the call. The players can raise only twice during a single round. After the first round ends, one of the remaining four cards is revealed as public. Then, the players proceed with the betting round. The only difference is that the raise now adds four more coins to the pot. At the end of this betting round, both players reveal the following rule that decides their card and the winner: If a player’s private card matches the rank of the public card (e.g., the player has a King and the public card is a King), they have a pair and win. Otherwise, the player holding a higher value card wins, and the order from the highest rank is King, Queen, Jack. If both players hold the same rank, it is a draw, and they will split the money in the pot.

### C.2 Imperfect Information Goofspiel

Imperfect information Goofspiel $N$ is a game played by two players, where each player receives cards valued from 1 to $N$. The dealer has the same cards. Then each turn, the dealer reveals a single card from its deck to both players. Each player then secretly places one of its cards as a bet. The dealer looks at both cards and gives the points corresponding to the public card to the player that had the highest bid. In case of a draw, the dealer discards its card. The players can only play the same card once. We have used a version where the dealer has predefined order of cards, so it always shows from the highest card to the lowest one.

### C.3 Imperfect Information Oshi-Zumo

Imperfect information Oshi-Zumo $K,N$ is played by two players on a board of size $2K+1$ with a fighter in the middle of the board. Each player starts with $N$ coins. Then each turn, players secretly place bids from 0 to the amount of coins they are still holding. The player that had the higher bet pushes the fighter closer to the opponents edge. The game ends either when neither of the players has any coins, when the fighter is on the edge of the board (positions 0 and $2K$) or after the maximum number of rounds. We have used $N$ as the number of rounds. The reward of player 1 is then $\mathcal{R}_{1}=\frac{P-K}{K}$, where $P$ is the position of the fighter at the end of the game.

## Appendix D Large Language Models usage

During the writing of this paper, the Large Language Models (LLMs) were used to refine the writing, both by polishing the text and to better communicate the main contributions of the paper. Specifically, we have used Gemini 2.0 Flash, ChatGPT, and Claude Sonnet 4.

We have also used the LLM coding assistant Cursor with Claude Sonnet 3.5 and later Claude Sonnet 4 as the underlying model for the experimental evaluation.

The authors always double-checked all of the LLM outputs to ensure their correctness.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Apologies if I missed it in the paper, but is it true that this method only works on games with no chance?
2. While playing the game, is it possible that the root public state of the game tree contains more than $L$ information states? 
3. I'm a bit confused on Section 4, especially what it is used for. I think Section 4 describes how to train $\Lambda_{i,\theta}$, $\kappa_{\theta}$, and $\Lambda^I_{i, \theta}$.

However, the description of the second loss says that it updates $\Lambda^I_\theta$. Is this a typo? Should it be $\Lambda^I_{i, \theta}$? But the equation describing the second loss only includes $\Lambda_{i, \theta}. 

In the description of LAMIR at the end of Section 5, these are only used to map the real current infoset $s_i$ to the abstracted one $\bar{s_i}$. Is this all it is used for? And why can't we just use $\Lambda^I_\theta$ to do this? 
4. In Section 4, why do we need to map latent infostates and infostates to a K-dimensional space? Why can't we just do everything in the latent space?
soundness: 3
strengths: The research direction and idea are sensible. The method is explained well. The experiments are solid.
summary: This paper presents a new method for building an agent that plays two-player zero-sum imperfect-information games, using a lookahead search technique similar to MuZero.

The main Dynamics function maps a joint latent infostate and a joint action to a new joint latent infostate. It is learned through sampled trajectories via a simulator. It can then be used at test-time with depth-limited solving to play well.

Additionally, the method uses an abstract model to deal with large public states.

The paper includes experiments computing exact exploitability in small Goofspiel and Oshi-Zumo games, and head-to-heads results against RNaD in large Goofspiel.
weaknesses: I don't quite understand the Abstract Model (see questions).

I think more could be said about the motivation/application of this work. It's practically useful only if we (A) don't already have a perfect simulator but (B) have access to trajectories that include both players' infostates and actions. When could this be useful?

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: a) This method seems like it would be extendable to other self-play methods like magnetic mirror descent. Would you expect clear integration issues or challenges in using this lookahead method with population-based methods like PSRO?

b) Is any regularization or activation applied to shape or constrain the space of latent infostates $\bar{S}_i$?

c) Without any loss of novelty, the base world model dynamics and encoder model seem mechanically very similar to the one used in TD-MPC [1] and TD-MPC2 [2] (A key distinction is that LAMIR operates over information sets under partial observability.)
 Would the world model stability improvements introduced in TD-MPC2 potentially be applicable to this work?
e.g. 
1. Only training the initial state encoding $\Lambda_\theta^I(s_i^{t})$ using dynamics loss and stopping the gradients through dynamics targets $\Lambda_\theta^I(s_i^{t+k})$
2. Bounding and regularizing the latent infostate space with SimNorm

[1] Hansen, N., Wang, X., & Su, H. (2022). Temporal Difference Learning for Model Predictive Control.ICML 2022 (PMLR).

[2] Hansen, N., Su, H., & Wang, X. (2024). TD-MPC2: Scalable, Robust World Models for Continuous Control.ICLR 2024.
soundness: 3
strengths: - This paper makes important and large contributions towards using latent-representation world models effectively in a 2p0s imperfect information self-play setting.
- The proposed method is relatively straightforward.
- The experimental results are sufficient and convincing.
- The limitation to games without chance and lack of convergence grantees for games without A-loss recall are well explained.
summary: The authors propose abstract look-ahead search at test time for imperfect-information games. While training RNaD policy, they learn a world model over latent information sets. They then learn a per-public-state abstraction that clusters many real infosets into latent prototypes, yielding a depth-limited latent subgame that's smaller than the original game or world model. At the depth limit they attach a policy-transformation leaf gadget (from Kubíček et al., 2024) and solve the subgame with CFR. Test-time look-ahead achieves lower exploitability than vanilla RNaD in small games and beats RNaD in larger games.
weaknesses: - The proposed feature encoding for latent infostate clustering is a heuristic.
- Code is not provided.
- Minimal ablations on the world model architecture design.
- Missing related works: TD-MPC and TD-MPC2

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Please address each concern raised in the weaknesses.

### Regarding the training process:
1. How is $\Lambda_{i,\theta}$ (the green component in Figure 1) specifically trained? Is it implemented using a clustering algorithm, or is it also modeled as a neural network? If it is a neural network, what are its input and output representations? Does it use the clustering result as the training target? It seems that the training loss in equations (2) and (3) is related only to $\Lambda^{I}_{i,\theta}$.

2. In equation (2), during training, does it require enumerating all $\overline{s^t_i}$? Or is it simply using sampling? If it is sampling, how many samples are required? Is this sampling process required to be performed for every public state? What is the computational cost of this sampling process? If the number of samples is too small, is the clustering into $L$ groups still effective? How is the value of $L$ determined in practice? Does the number of samples increase as $L$ increases? If so, how does the computational cost scale with $L$? Overall, what is the additional computational cost of training LAMIR compared to RNaD?

3. What kind of training data is used? Is it based on a pre-defined dataset or trajectories obtained from its self-play? 

### Regarding test time:
1. Section 5 presents the overall algorithm that integrates all components discussed in the previous sections. However, the description in Section 5 is unclear, particularly regarding how the components are used in the tree search. It would be helpful to include a figure illustrating the test-time architecture, along with a more detailed explanation of each component.

2. During the search tree, is it still required to enumerate all $\overline{s^t_i}$ or sample $\overline{s^t_i}$ to obtain the latent abstract information sets?

3. It seems that $\Lambda_{i,\theta}^{I}$ is used only at the root node of the game tree, where a specific abstraction set is chosen. However, this abstraction set only represents a single category. After performing a sequence of actions during the search tree, can we guarantee that this category still adequately covers all possible scenarios at the leaf nodes? Or is $\Lambda_{i,\theta}^{I}$ repeatedly used during the search at each internal node?

4. The inference time is not reported. Currently, the depth is set to 1. Why not use a larger depth? It would be helpful to include an ablation study analyzing different depth settings, along with a discussion of the additional computational cost and potential performance gains as the depth increases.

### Regarding the experiments:

1. How are the different bases for clustering ($\kappa$) implemented?
2. Regarding training and test time, are LAMIR with different $L$ values and RNaD all trained under the same training time limits?
3 . Based on the results in Table 1, why does using Legal Actions for clustering lead to a higher win rate than the RNaD strategy in the Goofspiel 13 and Goofspiel 15 environments? Additionally, would it be fair to use the RNaD strategy for clustering within LAMIR and then compare it directly with RNaD itself, given that LAMIR would already know the RNaD strategy?

### Other questions and suggestions
1. The reference formatting in the main text is incorrect. Most citations are missing brackets.
2. How are the public state and the player's information set generated during training, especially in large-scale games? Additionally, what is the approximate number of public states in the "large games" used in the experiments?
soundness: 3
strengths: 1. This paper provides a novel approach that combines model learning, abstraction techniques, and value function learning to address scalability challenges in large-scale imperfect-information games.

2. The comparison to the baseline, RNaD, is persuasive, as it was previously used in DeepNash in Stratego.

3. The experimental figures and methodology descriptions are clear and well-structured, providing a comprehensive understanding.

4. The future work proposed by the authors is sound. For instance, integrating abstract action spaces could further enhance the applicability of LAMIR to various types of games, particularly those with large or continuous action spaces.
summary: This paper introduces LAMIR (Learned Abstract Model for Imperfect-information Reasoning), an algorithm designed to enable look-ahead reasoning in large-scale imperfect information games by an abstract model, which can directly learn from experience without any domain-specific knowledge. LAMIR achieves up to an 80% win rate in large imperfect information games compared to RNaD, and also shows robustness in some smaller games.
weaknesses: 1. The description in Section 5 is insufficient. In particular, the explanations of the training and testing processes should be clarified, as they are somewhat difficult to follow in the current version. This is especially important because Section 5 presents the overall algorithm that integrates all components discussed in the previous sections. Including a flowchart or pseudocode would help readers better understand the steps. 

2. Section 4 uses a significant amount of space to describe the clustering process and explains how to train the clustering of the public state. However, it seems that this approach is applied only to the root node of the test-time search tree. The tree search process requires a more detailed description, including how each component introduced in previous sections is used.

3. Since LAMIR requires clustering during training, the overall training time is expected to increase. The paper should clearly report the additional training cost of LAMIR and compare it with previous approaches, such as RNaD. It would also be helpful to add an additional experiment discussing how the computational cost increases as the number of clusters $L$ increases, as well as how much data is required to train both the clustering and abstraction processes.

4. As mentioned in Section 7, the proposed approach seems to be constrained by the limitations of the CFR algorithm, where it is not applicable to many other games like Dark Chess and Stratego. However, the abstraction method itself should not be affected by this. Have there been any attempts to apply the core ideas of abstraction to these games?

5. The experiments are conducted on relatively simple imperfect information games. Are there any experiments on more complex games, such as poker? Including such results would make the work more convincing by better demonstrating the generality and effectiveness of LAMIR.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses.
soundness: 3
strengths: - The performance of the algorithm looks good. It outperforms RNaD consistently.
- The writing of the paper is clear.
- The paper enables subgame search in games with a large public states.
summary: The authors proposed a novel algorithm that enables subgame search in imperfect-information games with a large number of public states. Specifically, they trained a clustering network to group similar information sets and solved the resulting abstracted subgames using CFR+. In addition, to constrain the size of the subgame, they trained a value network to predict state values, enabling depth-limited subgame search.
weaknesses: - The techniques presented in this paper are not particularly novel. It appears to combine several existing approaches -- subgame search, game modeling, and abstraction. That said, integrating all of these components simultaneously is a non-trivial task.
- The games evaluated in this paper do not seem to require model learning, as used in MuZero. I would prefer to see the method tested on games where developing an accurate simulator is intractable.


Decision: Accept (Poster)