SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Christopher Solinas<sup>1</sup>, Douglas Rebstock<sup>1</sup>, Nathan R. Sturtevant<sup>1,2</sup>, Michael Buro<sup>1</sup>  
<sup>1</sup>Department of Computing Science, University of Alberta  
<sup>2</sup>Alberta Machine Intelligence Institute (Amii)  
{solinas,dreibstoc,nathanst,mburo}@ualberta.ca
Historically applied exclusively to perfect information games, depth-limited search with value functions has been key to recent advances in AI for imperfect information games. Most prominent approaches with strong theoretical guarantees require subgame decomposition—a process in which a subgame is computed from public information and player beliefs. However, subgame decomposition can itself require non-trivial computations, and its tractability depends on the existence of efficient algorithms for either full enumeration or generation of the histories that form the root of the subgame. Despite this, no formal analysis of the tractability of such computations has been established in prior work, and application domains have often consisted of games, such as poker, for which enumeration is trivial on modern hardware.
Applying these ideas to more complex domains requires understanding their cost. In this work, we introduce and analyze the computational aspects and tractability of filtering histories for subgame decomposition. We show that constructing a single history from the root of the subgame is generally intractable, and then provide a necessary and sufficient condition for efficient enumeration. We also introduce a novel Markov Chain Monte Carlo-based generation algorithm for trick-taking card games—a domain where enumeration is often prohibitively expensive. Our experiments demonstrate its improved scalability in the trick-taking card game  $Oh$  Hell. These contributions clarify when and how depth-limited search via subgame decomposition can be an effective tool for sequential decision-making in imperfect information settings.
Games are a standard model for sequential decision-making. As the number of sequential decisions needed to play increases, the size of the game's state space can grow exponentially—quickly becoming too large to search exhaustively. Depth-limited search navigates this issue by replacing decision points below a certain depth with a value function that captures or approximates the value of playing the subgame from that position onward. This can massively boost scalability and has been key to several famous results in perfect information games including Checkers (Schaeffer and Lake [1996]), Chess (Campbell et al. [2002]) and Go (Silver et al. [2017]).
The same idea has recently been successfully applied to certain imperfect information games such as poker (Brown and Sandholm [2019]; Moravčík et al. [2017])—achieving similarly impressive results by defeating human experts. In contrast to the perfect information setting, for which computing the value function only requires evaluating the current history, most successful depth-limited search algorithms in imperfect information games require evaluating whole sets of histories and their reach probabilities. This information is used to generalize the concept of a subgame and its value
by decomposing the game tree into public belief states, which are analogous to belief states in Markov systems such as Hidden Markov Models (HMMs) and Partially Observable Markov Decision Processes (POMDPs). This approach is commonly referred to as subgame decomposition; we describe the associated history and reach probability computations as history filtering. Although current theory establishes the necessary information for theoretically sound depth-limited search in imperfect information games (Kovářík et al. [2020]), it is unclear how, or if, this information can be computed efficiently. Understanding these computations provides insight into scaling search to larger, more complex imperfect information games.
In this work, we define variants of history filtering for subgame decomposition that are useful for search—called enumeration and generation—while also developing a suitable notion for efficiently solving them. Efficient solutions should take at most polynomially many steps in the length of the input observation sequence. We show that, in general, such algorithms only exist if  $\mathsf{P} = \mathbb{NP}$ . From there, we investigate methods for efficient enumeration and generation. First, by identifying a structural property of the game tree that is both a necessary and sufficient condition for efficient enumeration, and then by introducing a novel, unbiased generation algorithm for trick-taking card games, based on Markov Chain Monte Carlo. Through experiments in Oh Hell, we validate its improved scalability and highlight the potential advantages of this approach. Our contributions advance the theory of depth-limited search in imperfect information domains.
In this section, we summarize the concepts and algorithms related to history filtering and public belief states.
2.1 Factored Observation Stochastic Games
The recent introduction of Factored Observation Stochastic Games (FOSGs) (Kovarik et al. [2019]) has helped clarify fundamental concepts about decomposing public and private observations and information in multi-agent, partially-observable sequential decision problems.
An FOSG is a tuple  $G = \langle \mathcal{N}, \mathcal{W}, \mathcal{P}, w^0, \mathcal{A}, \mathcal{T}, \mathcal{R}, \mathcal{O} \rangle$ .  $\mathcal{N} = \{1, \dots, N\}$  represents the set of players,  $\mathcal{W}$  is the set of world states, and  $\mathcal{A}$  is the set of joint actions.  $\mathcal{P}: \mathcal{W} \to 2^{\mathcal{N}}$  is the player function, which describes which players act in which world states.  $\mathcal{T}: \mathcal{W} \times \mathcal{A} \to \Delta \mathcal{W}$  is the state transition function, where  $\Delta \mathcal{W}$  represents the set of probability distributions over  $\mathcal{W}$ .  $\mathcal{R}: \mathcal{W} \times \mathcal{A} \to \mathbb{R}^N$  assigns a reward to each player, and  $\mathcal{O}: \mathcal{W} \times \mathcal{A} \times \mathcal{W} \to \mathbb{O}^{N+1}$  is the observation function—which maps transitions (world state-action-world state triples) to private observations for each player and a public observation common to all players.
Games start at the initial world state  $w^0$ . In any world state  $w \in \mathcal{W}$ , player  $i$  acts when  $i \in \mathcal{P}(w)$ . The joint action set  $\mathcal{A} := \prod_{i \in \mathcal{N}} \mathcal{A}_i$  is defined as the product of each player's individual action sets across all  $w \in \mathcal{W}$ .  $\mathcal{A}_i(w) \subset \mathcal{A}_i$  denotes the legal actions for  $i$  in  $w$ , and  $\mathcal{A}(w) := \prod_{i \in \mathcal{P}(w)} \mathcal{A}_i(w)$  is the set of legal joint actions in  $w$ . Play proceeds when each  $i \in \mathcal{P}(w)$  chooses an action  $a_i \in \mathcal{A}_i(w)$  — resulting in joint action  $a := (a_i)_{i \in \mathcal{P}(w)}, a \in \mathcal{A}(w)$ . The next state  $w'$  is sampled from  $\mathcal{T}(w, a)$ , while the reward is determined by evaluating  $\mathcal{R}(w, a)$ . Finally,  $\mathcal{O}(w, a, w')$  is factored into public and private observations as  $(\mathcal{O}_{\mathrm{priv}(1)}(w, a, w'), \dots, \mathcal{O}_{\mathrm{priv}(N)}(w, a, w'), \mathcal{O}_{\mathrm{pub}}(w, a, w'))$ .
2.2 Policies, Reach Probabilities, and Beliefs
A history is a sequence  $h := (w^0, a^0, w^1, a^1, \dots, w^t)$  of world states and actions for which  $w^k \in \mathcal{W}$ ,  $a^k \in \mathcal{A}(w^k)$ , and  $\mathbb{P}[\mathcal{T}(w^k, a^k) = w^{k+1}] > 0$  for  $k, 0 \leq k \leq t-1$ . We refer to  $|h| := t$  as the length of  $h$ . The set of all legal histories is denoted as  $\mathcal{H}$ . We use the standard notation  $h' \subseteq h$  to denote that  $h'$  is a prefix history of  $h$  (i.e.,  $h'$  is a subsequence of  $h$  starting at  $w^0$  and ending in a world state). A terminal history  $z \in \mathcal{Z}$  signifies that play has reached a world state where the game ends. The utility for player  $i$ ,  $u_i : \mathcal{Z} \to \mathbb{R}$  is the sum of all rewards accumulated by  $i$  over the world state and action sequence.
A history  $h$  produces a public state, which is the sequence of public observations  $s_{pub}(h) := (O_{pub}^{1}, O_{pub}^{2}, \dots, O_{pub}^{t})$  produced along the trajectory with  $O_{pub}^{k} := O_{pub}(w^{k-1}, a^{k-1}, w^{k})$ . We
Figure 1: The same history viewed in the world state tree (left) and the public tree (right). Grey boxes are public states, which can be consistent with many histories from the world state tree.
Figure 2: (Left) An example public belief state in a two-player game with ranges  $r_1$  and  $r_2$ .  $S$  is consistent with a set of 5 histories. Player 1's range contains two private information states. (Right) A player's beliefs (top) and a joint range (bottom). Player 1's beliefs, given the first private infostate, consist of reach probabilities for the individual histories in the infostate. The joint range consists of reach probabilities over all histories in  $S$ .
denote a public state generated by an unknown history as  $S \in S$ , where  $S$  is the set of all public states (and the vertex set of the public tree, see Figure 1). Likewise, player  $i$ 's private information state  $S_{i} \in S_{i}$  similarly captures the sequence of observations seen only by that player:  $s_{i}(h) := (O_{i}^{1}, O_{i}^{2}, \dots, O_{i}^{t})$ , where  $O_{i}^{k} := \mathcal{O}_{\text{priv}(i)}(w^{k-1}, a^{k-1}, w^{k})$ . Taken together,  $(s_{pub}(h), s_{i}(h))$  represents all information available to player  $i$  at history  $h$ , and is referred to as the player's information state or infostate.  $S_{i}(S)$  denotes the set of all infostates consistent with public state  $S$  and is a partition of  $S$ . We use  $\mathcal{H}_{S} := \{h \in \mathcal{H} : s_{pub}(h) = S\}$  to denote the set of histories consistent with public state  $S$ . Since all players receive (possibly empty) observations at every world state transition, this formalism avoids non-timeability and thick infostates present in the extensive form. See Kovařík et al. [2020] for details.
A player plays according to a policy  $\pi_i: S_i \to \Delta(\mathcal{A})$  which maps player  $i$ 's infostates to the set of probability distributions over action set  $\mathcal{A}$ . A joint policy  $\pi = (\pi_1, \dots, \pi_n)$  is a tuple consisting of every player's policy. The reach probability of a history under  $\pi$  is  $P^{\pi}(h) = P_c(h)P_1^{\pi}(h)P_2^{\pi}(h)\dots P_N^{\pi}(h)$  where each  $P_i^{\pi}(h)$  is a product of action probabilities taken by player  $i$  to reach  $h$  from  $w^0$ , and  $P_c(h)$  is the product of all probabilities from chance transitions taken according to the stochastic transition function  $\mathcal{T}$ . The reach probability of an infostate (or public state)  $S_i$  under  $\pi$  can be expressed as  $P^{\pi}(S_i) := \sum_{h \in S_i} P^{\pi}(h)$ . Infostate reach probabilities can also be decomposed into  $P^{\pi}(S_i) = P_i^{\pi}(S_i)P_{-i}^{\pi}(S_i)$  where  $-i$  denotes all players except  $i$ . Since all  $h \in S_i$  are indistinguishable to  $i$ ,  $P_i^{\pi}(S_i) = P_i^{\pi}(h)$ , whereas  $P_{-i}^{\pi}(S_i) = \sum_{h \in \mathcal{H}_{S_i}} P_{-i}^{\pi}(h)$  (Kovarík et al. [2020]).
Kovarik et al. [2020] define a belief under joint policy  $\pi$  as the probability of reaching a history  $h$  given the current infostate  $S_{i}$ : for any  $h\in \mathcal{H}_{S_i}$ ,  $P^{\pi}(h|S_i) = P^{\pi}(h) / \sum_{h'\in \mathcal{H}_{S_i}}P^{\pi}(h')$ . The range for a joint policy  $\pi$  at public state  $S$ ,  $r^{\pi}(S)\coloneqq ((P_j^\pi (S_i))_{S_i\in S_j(s_{pub})})_{j = 1,2,\dots,N}$  contains each player's reach probabilities for their infostate partition of  $S$ . The (normalized) joint range  $P^{\pi}(h|S)$  contains the normalized reach probabilities of all  $h\in S$ . These concepts are visualized in an abstract game in Figure 2. Our analysis focuses on history filtering with respect to the joint range—assuming the joint policy is constant and known by all players.
2.3 Depth-Limited Search, Subgames, and Belief State Computation
Several depth-limited search algorithms, such as Information Set Monte Carlo Tree Search (ISMCTS) (Cowling et al. [2012]), Player of Games (Schmid et al. [2021]), and ReBel (Brown et al. [2020]), require methods for simulating actions and state transitions. In these algorithms, search starts from a ground truth history—a plausible state of the game—and proceeds until a depth limit; value functions represent the players' expected reward in the subgame below.
Subgames and their value functions are intuitive in the perfect information setting because they can be rooted at any non-terminal history. However, the necessary information for computing value functions in the imperfect information setting is structured differently. Subgames are instead rooted at public belief states (PBS), which are tuples  $\beta \coloneqq (S,r^{\pi}(S))$  composed of a public state and a range. Searching from  $\beta$  requires computing histories from  $\mathcal{H}_S$  because the actions and transitions necessary for forward simulation are defined for world states, not public states, and computing optimal value functions at the subgame leaves requires ranges defined over all histories in  $\mathcal{H}_S$  (Kovarik et al. [2020]). Player of Games (Schmid et al. [2021]) opts for the Monte Carlo approach, which generates  $h\in \mathcal{H}_S$  for player  $i$  by sampling an infostate from  $S_{i}\in S_{i}(S)$  according to  $i$ 's component of  $r^\pi (S)$ , and then sampling according to their beliefs  $P^{\pi}(h|S_i)$  (see Figure 2 for the distinction). ISMCTS (Cowling et al. [2012]) is similar, but instead samples directly from the normalized joint range  $P^{\pi}(h|S)$ . Either case depends on solving a common computational problem: histories from  $\mathcal{H}_S$  must be sampled according to  $r^{\pi}(S)$ , given only  $S$  and  $\pi$ .
Belief state computation is an important topic in general Markov systems with partial observability. As such, it appears frequently in areas such as stochastic control (Nayyar et al. [2013]), learning in decentralized POMDPs (Dibangoye et al. [2016]; Oliehoek [2013]), and multi-agent reinforcement learning (Fickinger et al. [2021]). In FOSGs, which are a generalization of POMDPs, search techniques that use public belief states have also been crucial to the development of superhuman poker AI (Brown and Sandholm [2019]; Brown et al. [2020]; Moravcik et al. [2017]; Schmid et al. [2021]). In card game AI, Perfect Information Monte Carlo (PIMC) search was successfully applied to Contract Bridge (Levy [1989]; Ginsberg [2001]) and later Skat (Buro et al. [2009]). Both applications rely on game-specific implementations for history filtering. Richards and Amir [2012] described the problem as information set generation and provided the first generic, but exponential-time, algorithm for solving it. Seitz et al. [2021] provided an algorithm for approximating information set generation based on deep neural networks. Šustr et al. [2021] identified that exponential-sized  $\mathcal{H}_S$  cause memory issues in methods that explicitly represent the range, and that efficient search requires the range to be represented compactly.
Recently, the idea of fine-tuning has helped push the scalability of planning in POMDPs via reinforcement learning (Fickinger et al. [2021]) and search in FOSGs (Sokota et al. [2021]) by eliminating the need to explicitly represent the range. In reinforcement learning, fine-tuning consists of online updates to a parameterized model of a blueprint policy or Q-value function using sample trajectories with a fixed horizon. These local improvements to the model helped train policies which achieved a new state-of-the-art in self-play Hanabi (Fickinger et al. [2021]). Belief fine-tuning (BFT) (Sokota et al. [2021]) extended the idea to PBS approximation without pre-training the generative model as a function of the joint policy  $\pi$ .
In some domains, it is unclear how to structure a dynamics model for fine-tuning such that only legal sample histories are produced. For example, in trick-taking card games, the public observation sequence could indicate that certain players cannot possibly have certain cards in their hands; all histories output by the model should satisfy this constraint. Our approach uses only the dynamics model defined by the game. It is parameter-free and instead calls for simple domain-specific algorithms for history construction and neighbor generation in a Markov chain. This results in unbiased history generation and guarantees that any sample history is legal according to the game rules.
3 History Computation in Public Belief States
Our first contribution is to introduce a family of history filtering relations  $\mathrm{FILTER}(G, \pi)$  which formalize this computation for a given FOSG  $G$  and joint policy  $\pi$ . For simplicity, we limit our
analysis to FOSGs with finite world state sets  $\mathcal{W}$  and action sets  $\mathcal{A}$ , and joint policies that can be evaluated in polynomial time with respect to history length.
3.1 Family of Filtering Relations
Computing histories from a given public state is a special case of particle filtering, so we name our family of relations FILTER.
Definition 1. (FILTER) For any FOSG  $G$  with finite  $\mathcal{W}$ ,  $\mathcal{A}$  and joint policy  $\pi$ , let  $FILTER(G, \pi) := \{(S, h) \in \Sigma^* \times \Sigma^* : s_{pub}(h) = S, P^\pi(h|S) > 0\}$ .
An FOSG  $G$  and a joint policy parameterize the relation—which pairs public states  $S \in S$  and histories consistent with  $S$ .  $S$  is treated as the problem input (coded using alphabet  $\Sigma$ ) and reachable histories from  $\mathcal{H}_S$  are valid outputs.
Our definition treats the FOSG  $G$  and the joint policy  $\pi$  as fixed and assumes that the FOSG functions  $(\mathcal{T},\mathcal{A},\mathrm{etc.})$  and the policy can be evaluated in polynomial time with respect to their inputs, and are not encoded as inputs to the problem. Thus, the input and output sizes of a problem instance are the lengths of the encoded observation sequence and history, respectively.
A relation  $R \subset \Sigma^{*} \times \Sigma^{*}$  over alphabet  $\Sigma$  is polynomially balanced if there exists a polynomial  $p$  such that for all  $(x, y) \in R$ ,  $|y| \leq p(|x|)$  (i.e., the length of output  $y$  is at most polynomial in the input length) (Jerrum et al. [1986]).  $R$  is polynomial-time verifiable if the predicate  $(x, y) \in R$  ( $xRy$  for short) can be tested in polynomial time. Lemma 1 states that this is the case for  $\mathsf{FILTER}(G, \pi)$  with finite  $\mathcal{W}$  and  $\mathcal{A}$ . Finite  $\mathcal{W}$  and  $\mathcal{A}$  imply polynomial balance and polynomial-time verification is done using  $|h|$  evaluations of the observation function and policy to check if  $S$  is produced and the reach probability is nonzero. All proofs are in the appendix.
Lemma 1. For any FOSG  $G$  with finite  $\mathcal{W}$ ,  $\mathcal{A}$  and arbitrary joint policy  $\pi$ ,  $FILTER(G, \pi)$  is polynomially balanced and polynomial-time verifiable.
3.2 Computational Problem Variants
For a binary relation  $R$ , there are several naturally associated computational problems. Given a problem instance  $x \in \Sigma^{*}$ , some of these are:
1. Existence: Is there a  $y \in \Sigma^{*}$  such that  $xRy$ ?
2. Construction: Return a  $y \in \Sigma^{*}$  such that  $xRy$  if one exists.
3. Generation: Generate a  $y \in \Sigma^{*}$  such that  $xRy$  according to some predetermined distribution over the solution set  $\{y \in \Sigma^{*} : xRy\}$  if one exists.
4. Counting: Compute  $|\{y \in \Sigma^* : xRy\} |$ .
5. Enumeration: Return all  $y \in \Sigma^{*}$  such that  $xRy$ .
Of these, generation and enumeration are clearly relevant to history filtering. Prior work (Schmid et al. [2021]; Brown et al. [2020]; Moravčík et al. [2017]; Brown and Sandholm [2019]) has generally relied on enumerative methods, i.e. filtering histories by explicitly representing the entire PBS. Generative methods for history filtering potentially have the advantage of avoiding explicit PBS representation. In the next section, we analyze the computational complexity of problem variants 1-5.
4 Complexity of Filtering Histories
Efficient (polynomial-time and space) algorithms for history filtering enable scalable depth-limited search in imperfect information games. In this section, we provide an FOSG instance where the construction variant of FILTER is intractable and explain when efficient enumeration is possible.
4.1 FNP-Completeness of Construction
Consider the following two-player FOSG, based on Functional Boolean Satisfiability (FSAT):
Definition 2. (3-FSAT-GAME) For a given integer  $m$ , world states are encoded as  $m$ -variable truth assignments. Starting at initial state  $w^0$ , player 1 chooses an action  $a^0 = (y_1, y_2, \dots, y_m)$  that represents a truth assignment. This is followed by a transition to some  $w = (y_1, y_2, \dots, y_m)$  that encodes the same assignment.  $O_{pub}(w^0, a^0, w)$  reveals no public information about the transition to  $w$  except that the action and transition occurred.  $w$  has joint action set  $A(w) := \{a\}$  and  $T(w, a) := w$  for all  $w \in \mathcal{W}$ . Rewards are arbitrary. When action  $a$  is taken at time  $t$ , the public observation function  $O_{pub}(w, a, w)$  generates, at random, a 3-CNF clause  $c_t$  that is satisfied by  $w$ .
Player 1 chooses an  $m$ -variable truth assignment, and then player 2 repeatedly takes arbitrary actions—outputting a 3-CNF clause satisfied by the assignment as a public observation each time. The truth assignment is unknown to player 2, so solving a 3-CNF Boolean formula is necessary to construct a history consistent with the public observations. Additionally, a sequence of observations equivalent to any satisfiable 3-CNF formula over  $m$  variables can be generated by playing this game.
Theorem 1. There exists a joint policy  $\pi$  for which the construction problem associated with FILTER(3-FSAT-GAME,  $\pi$ ) is FNP-complete.
Theorem 1 implies that unless  $\mathsf{P} = \mathsf{NP}$ , computing even a single history corresponding to a given public state in the 3-FSAT-GAME is intractable in the worst case. It follows that the same applies to more complex computations such as generation and enumeration. However, there are several examples where these computations have been successfully performed in practical examples of games such as poker—we discuss where efficient enumeration is feasible next.
4.2 Efficient Enumeration in Games with Sparse Public States
Prior work has often limited application domains to games where the public state is trivially enumerated and beliefs can be represented explicitly in physical memory; the efficiency of these algorithms depends on a structural property of the game tree that we call sparsity.
Definition 3. The public tree  $S$  of an FOSG  $G$  is sparse if and only if all public states  $S = (o^1, o^2, \ldots, o^t)$  in  $S$  satisfy  $|\mathcal{H}_S| \leq p(t)$  for some polynomial  $p$ . Public trees that do not satisfy this property are dense.
Public states in games with sparse public trees can be enumerated in polynomial time using a simple breadth-first search that makes at most  $|\mathcal{A} \times \mathcal{W}|p(k)$  calls to the observation function at depth  $k$  (see proof of Theorem 2 in the appendix for more details).
Theorem 2. For any FOSG  $G$  with finite  $\mathcal{W}, \mathcal{A}$ , the enumeration problem associated with  $FILTER(G, \pi)$  can be solved in polynomial time if and only if  $G$ 's public tree is sparse.