SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As an example, consider the following variants of two-player poker. In Texas Hold'em, there is a 52-card deck and two cards are dealt to each player; public states in this game are of constant size. With  $n$  cards in the deck where each player is dealt 2 cards, the number of histories per public state is polynomial in  $n$ . Both of these games have sparse public trees. However, with  $n$  cards in the deck and  $k$  cards dealt to each player, the number of histories is exponential in  $k$ , so the public tree is dense.
Sparsity itself does not guarantee that enumeration is feasible on modern hardware. For instance, physical memory and time constraints still prevent enumeration in large trick-taking card games with constant-sized public states such as Skat and Hearts. In the next section, we propose an algorithm for history generation in these games and validate it empirically.
5 MCMC History Generation in Trick-Taking Card Games
Trick-taking card games (TTCGs) like Contract Bridge, Skat, and Hearts are played by millions of people worldwide and have historically been of significant interest to the AI community. However, superhuman-level computer play in these games has yet to be achieved—in part because of their large public belief states. Here, we devise a Gibbs sampler (Geman and Geman [1984]) for history generation in TTCGs that treats histories as states in a Markov chain and uses local computations to generate histories without explicit belief representation.
Figure 3: A flow network constructed from a history in a TTCG with three suits and two players. The suits have three, one, and two unknown cards remaining, respectively, and each player must be dealt three more cards. The first player is void in the third suit, and the second player is void in the second suit.
5.1 Challenges of Filtering in TTCGs
Depending on the size of the deck and the number of cards dealt to each player, public states in TTCGs may be large early in the game: up to  $n! / (n - k)!$  for decks with  $n$  cards and  $k$  cards dealt in total. They shrink as information is revealed because players must be dealt the cards they play, and void suits (when a player reveals they cannot have a certain suit through play) imply certain cards cannot be dealt to certain players. These observations lead to two possible constraints on a card being dealt to a player: either it must have been dealt to a player, or it cannot have been dealt to a player. Both can be checked efficiently through a single pass over the observation sequence.
5.2 History Construction in TTCGs
The solution to history construction for some  $S \in S$  is a history  $h \in \mathcal{H}_S$ . This history can serve as an initial state for the Markov process we have designed to solve generation in TTCGs. As we now describe, construction can be solved in polynomial time using an algorithm for integer maximum flow such as Edmonds-Karp (Edmonds and Karp [1972]) along with some simple pre- and post-processing steps.
Given a history  $h$  with the subsequence of private actions that represent the deal,  $\sigma \sqsubseteq h$ , we create a flow network that captures the constraints of the cards in  $\sigma$ . Cards revealed through play must be dealt to the player that played them, so we can ignore them when solving for deals that satisfy the other constraint: where a player cannot have any more cards in a suit. The source vertex is connected to  $k$  suit vertices via directed edges with a capacity that corresponds to the number of unknown cards remaining in that suit. Each suit vertex is connected to a player vertex if it is possible for that player to hold cards of that suit in their hand. The edges connecting the player and suit vertices have capacity equal to the number of total unknown cards remaining in all suits. Finally, the player vertices are connected to the sink via edges that correspond to the number of cards remaining in that player's hand. See Figure 3 for an example.
Lemma 2. For TTCG  $G$  and joint policy  $\pi$  with full support,  $FILTER(G,\pi)$  can be solved in polynomial-time using a maximum flow computation.
Applying a maximum flow algorithm based on augmenting paths will construct integral flows that represent assignments of the number of unknown cards in each suit dealt to each player—which we call a suit length assignment. We can select one of possibly many histories that satisfy the suit length assignment (and the cards that have been explicitly revealed through play) and construct a history from  $\mathcal{H}_S$ .
5.3 TTCG Gibbs Sampler
The TTCG Gibbs sampler is Markov Chain Monte Carlo method (see Haggström and others [2002] for an overview) for generative history filtering in TTCGs. It is based on two concepts: Markov chain states are histories from  $\mathcal{H}_S$ , and transitions use unnormalized reach probabilities and involve local modifications to the suit length assignment of the current state.
Algorithm 1: RingSwap
input :S - public state,  $\sigma$  deal consistent with  $S$  output:  $\Omega_{\sigma}$  set of neighbors of  $\sigma$    
1 let  $A_{n\times m}$  be the suit assignment matrix for  $\sigma$    
2  $\Omega_{\sigma}\gets \{\}$    
3 for Row i in rows(A) do   
4 for Columns  $j,k,j\neq k$  in cols(A) do   
5 if void(S,i,j) or void(S,i,k) or  $a_{i,k} = 0$  , continue   
6  $C\gets A$    
7  $c_{i,j}\gets c_{i,j} + 1;c_{i,k}\gets c_{i,k} - 1$    
8  $\Omega_{\sigma}\gets \Omega_{\sigma}\bigcup \mathrm{BFS}(C,n)$  to find all ways to make C a valid suit length assignment   
9 end   
10 end   
11 return  $\Omega_{\sigma}$
We start by describing the neighbor generation algorithm, RingSwap (Algorithm 1), which operates on suit length assignment matrices. Constraints on suit length assignments can be encoded using a matrix with row and column sums equal to the number of unknown cards in each suit and player hand. Void suits are represented as entries fixed to zero. For example, the following is a suit length assignment matrix that satisfies the max flow network in Figure 3:
$$
\left[ \begin{array}{c c c} 2 & 1 & 0 \\ 1 & 0 & 2 \end{array} \right]
$$
The rows sum to 3 because each player has 3 unknown cards, and the columns sum to the number of unknown cards in the corresponding suit.
With suit assignment matrix  $A$  corresponding to  $\sigma$ , RingSwap repeats the following for all players  $i$ . For every pair of non-void suits  $j$  and  $k$ , perform a swap by adding a card to  $A_{i,j}$  and removing one from  $A_{i,k}$ . The column sums of the matrix are now incorrect ( $j$  has too many cards and  $k$  has one too few), and must be corrected via a sequence of swaps in other rows. All sequences of swaps of length  $< n$  that lead to valid suit length assignments are then computed via BFS and a valid assignment is selected proportionally to the number of histories it corresponds to.
We can now describe the TTCG Gibbs sampler. For public state  $S$  at time  $t$ , given  $X_{t} = h$ , deal  $\sigma \sqsubseteq h$ , and a joint policy  $\pi$  with full support at all infostates, consider the following Markov chain:
At time  $t + 1$ :
1. Compute  $\Omega_{\sigma}$ , the set of all neighbors of  $\sigma$  using the procedure RingSwap  $(S, \sigma)$
2. Sample  $\sigma^{\prime}$  uniformly from  $\Omega_{\sigma}$
3. Compute  $\Omega_{\sigma'}$  and  $h'$  such that  $\sigma' \subseteq h'$  by replacing  $\sigma$  with  $\sigma'$  in  $h$  to form  $h'$ .
4. Let  $z = \min \{1, \frac{\bar{P}^{\pi}(h') |\Omega_{\sigma}|}{\bar{P}^{\pi}(h) |\Omega_{\sigma'|}}\}$ .
5. With probability  $z$ ,  $X_{t + 1} = h'$ , otherwise  $X_{t + 1} = h$
State transitions are done according to the Metropolis-Hastings algorithm (Metropolis et al. [1953]; Hastings [1970])— with unnormalized reach probabilities  $\bar{P}^{\pi}$  as  $\mu^{*}$  and uniform selection over the neighboring states. All computations are local to the current history at time  $t$ , and take at most polynomial time in the history length. The following theoretical details state that the chain is suitable for unbiased history generation.
Theorem 3. The TTCG Gibbs sampler is aperiodic and irreducible.
Theorem 3 implies that the TTCG Gibbs Sampler converges to some stationary distribution; the following theorem ensures that its stationary distribution is the desired  $P^{\pi}$ .
Theorem 4. The stationary distribution of the TTCG Gibbs sampler with input  $\pi$  is  $P^{\pi}$ .
Given an initial history obtained by solving the construction problem for the game and a policy with full support at all infostates, the TTCG Gibbs sampler correctly generates histories from  $P^{\pi}(\cdot | S)$  in
(a) 192 Histories
(b) 12,960 Histories
(c) 544,320 Histories
Figure 4: Value estimation error of TTCG Gibbs Sampler with specified burn-in and baselines on PBS of various sizes. Error bars show one standard error of the mean over 100 runs. See the appendix for full game parameters.
(b) 12,960 Histories
(c) 544,320 Histories
Figure 5: The effect of the number of samples burned on the value estimation error of the TTCG Gibbs Sampler. Burning fewer samples can result in a lower value error after an equal number of transitions. Error bars show one standard error of the mean over 100 runs.
the limit. The next section validates the efficiency and approximation quality of the TTCG Gibbs sampler empirically in the domain of Oh Hell (Parlett [2008]); we leave the theoretical analysis of its mixing time to future work.
5.4 Experiments
The TTCG Gibbs sampler we just described runs in polynomial-time with respect to history length and removes dependencies on explicit belief representation. Thus, the algorithm's scalability depends on the mixing time of the underlying Markov chain. The following empirical results suggest that the chains mix rapidly in this domain.
We evaluate the TTCG Gibbs sampler using a value estimation task in the trick-taking card game Oh Hell. Oh Hell is an  $N$ -player game with  $n$  cards in the deck, where each player is dealt between 1 and  $\lfloor (n - 1) / N \rfloor$  cards. We control the size of randomly generated  $\mathcal{H}_S$ —starting small and then scaling to  $\mathcal{H}_S$  several orders of magnitude larger—by varying the number of cards in the deck and the number of cards dealt to players. The task is to estimate the expected value of  $S$  under  $\pi$ ,  $V^{\pi}(S) = \sum_{h \in S} P^{\pi}(h) V^{\pi}(h)$ , where the value of a history  $h$ ,  $V^{\pi}(h) = \sum_{z \in \mathcal{Z}: h \sqsubseteq z} P^{\pi}(h, z) u(z)$  is the expected utility of the terminal histories in  $\mathcal{Z}$  reachable from  $h$ . The initial state is chosen uniformly from  $\mathcal{H}_S$ , and ranges are constructed using policies learned via independent Q-learning (Claus and Boutilier [1998])—hyperparameters and other details can be found in the supplementary material. Strong performance in this task does not necessarily imply improved search performance; instead, it demonstrates the sampler's ability to generate histories from the correct public belief state.
Figure 4 shows the value error curves of the TTCG Gibbs sampler with a specified burn-in compared to two Monte Carlo baselines. True samples from the exact joint range, and Importance performs (biased) importance sampling with a uniform proposal and unnormalized reach probabilities. The latter estimates the PBS value by correcting the weight of samples drawn uniformly from  $\mathcal{H}_S$ . In practice—especially at larger scales than these experiments—importance sampling may not be feasible without a generation algorithm that approximates a known proposal distribution in polynomial time. We see that the TTCG Gibbs sampler outperforms importance sampling and closely
approximates sampling from the joint range in all cases while using a burn-in orders of magnitude smaller than the size of the public belief states.
Figure 5 shows the effect of burning samples on value estimation error. On one hand, our results show that burning fewer samples can result in a better value estimate with fewer state transitions. This is unsurprising because larger sample sizes generally lead to better estimates. However, evaluating a sample could be several orders of magnitude more costly than performing a state transition in the Markov chain in practice. Comparing Figures 5 and 4 shows that, if the estimation task is constrained by time or computational resources, fewer high-quality samples may produce a better estimate. The appropriate burn-in depends both on the task and the available resources.
Our experiments show that the TTCG Gibbs Sampler leads to increased scalability over the enumerative approach in this value estimation task. With a small burn-in of 20 state transitions, a good approximation of the PBS value is achieved after only 400 samples and therefore only 8,000 total state transitions—compared to the enumerative approach which must construct and calculate the reach probability for 544,320 histories. State transitions are computed locally to the current state (history) and do not require knowledge of the rest of the PBS, so memory requirements are dramatically reduced. These effects increase along with public state size; in Oh Hell with a 52-card deck, public states can have over  $10^{61}$  histories.
In this paper, we analyzed the computational complexity of history filtering for subgame decomposition and depth-limited search in imperfect information games. Although even the simplest form of the computation is likely intractable in general, we have shown that depth-limited search remains a viable option in certain classes of imperfect information games. Efficient enumeration is achievable in games with polynomially-sized public states; many application domains from prior work seem to have this property. However, generative methods for history filtering may not require explicit belief representation, and are therefore more scalable than enumeration. To this end, we have introduced a novel generation algorithm for trick-taking card games for asymptotically correct and efficient history filtering that fits seamlessly with Monte Carlo-style search algorithms.
Our TTCG Gibbs sampler needs a method that constructs a valid history from the public state but is otherwise flexible. It does not depend on knowing the size of the public state and does not require normalized reach probabilities to produce samples with the correct probability. Unlike the enumerative approach which front-loads all of its computation, our algorithm applies to the setting where the game-playing algorithm must return an action within a predefined time budget. It is also easy to parallelize; multiple samples can be generated simultaneously by starting the process multiple times from the same initial state and running each for the desired burn-in time. We demonstrate its effectiveness empirically—though future work should prove rapid mixing analytically.
This work was funded by the Canada CIFAR AI Chairs Program. We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), Prairies DRI, and the Digital Research Alliance of Canada.
Noam Brown and Tuomas Sandholm. Superhuman AI for multiplayer poker. Science, 365(6456):885-890, 2019.
Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong. Combining deep reinforcement learning and search for imperfect-information games. Advances in Neural Information Processing Systems, 33:17057-17069, 2020.
Michael Buro, Jeffrey Richard Long, Timothy Furtak, and Nathan R Sturtevant. Improving state evaluation, inference, and search in trick-based card games. In *IJCAI*, pages 1407-1413, 2009.
Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu. Deep Blue. Artificial Intelligence, 134(1-2):57-83, 2002.
Martin Schmid, Matej Moravcik, Neil Burch, Rudolf Kadlec, Josh Davidson, Kevin Waugh, Nolan Bard, Finbarr Timbers, Marc Lanctot, Zach Holland, et al. Player of games. arXiv preprint arXiv:2112.03178, 2021.
Dominik Seitz, Nikita Milyukov, and Viliam Lisy. Learning to guess opponent's information in large partially observable games. In Proc. AAAI Workshop Reinforcement Learning in Games, 2021.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. Nature, 550(7676):354-359, 2017.
Samuel Sokota, Hengyuan Hu, David J Wu, J Zico Kolter, Jakob Nicolaus Foerster, and Noam Brown. A fine-tuning approach to belief state modeling. In International Conference on Learning Representations, 2021.
Michal Šustr, Vojtech Kovarík, and Viliam Lissy. Particle value functions in imperfect information games. In AAMAS Adaptive and Learning Agents Workshop, 2021.