SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhenxing Ge
Nanjing, Jiangsu, China
zhenxingge@smail.nju.edu.cn
xuzhengcs@smail.nju.edu.cn
Tianyu Ding
Redmond, Washington, USA
tianyuding@microsoft.com
Wenbin Li*
liwenbin@nju.edu.cn
Yang Gao*
gaoy@nju.edu.cn
Subgame solving is an essential technique in addressing large imperfect information games, with various approaches developed to enhance the performance of refined strategies in the abstraction of the target subgame. However, directly applying existing subgame solving techniques may be difficult, due to the intricate nature and substantial size of many real-world games. To overcome this issue, recent subgame solving methods allow for subgame solving on limited knowledge order subgames, increasing their applicability in large games; yet this may still face obstacles due to extensive information set sizes. To address this challenge, we propose a generative subgame solving (GS2) framework, which utilizes a generation function to identify a subset of the earliest-reached nodes, reducing the size of the subgame. Our method is supported by a theoretical analysis and employs a diversity-based generation function to enhance safety. Experiments conducted on research games as well as the challenging large game of GuanDan demonstrate a significant improvement over the blueprint.
Subgame solving is a standard technique commonly adopted by superhuman AIs to play in games [Campbell et al., 2002, Silver et al., 2016]. This approach takes advantage of the fact that for perfect information games (where both players have complete knowledge about the state of the game), finding optimal strategies at any given decision node only requires knowledge of the subgames rooted at that node [Brown and Sandholm, 2017]. Therefore, AI agents can make use of this fact to improve their own strategies while playing with others, which proves significantly beneficial for large games, e.g., Chess [Campbell et al., 2002] and Go [Silver et al., 2017].
Recent advancements in subgame-solving techniques have proven to be highly successful in imperfect information games, such as no-limit Texas hold'em poker [Moravčík et al., 2017, Brown and Sandholm, 2018, 2019], Mahjoog [Li et al., 2020], and dark chess [Zhang and Sandholm, 2021]. However, state-of-the-art subgame solving algorithms suffer from the efficiency of constructing and solving large games. This limitation hinders their practicality for performing subgame-solving on large subgame trees, thus restricting their application in games such as DouDiZhu [Zha et al., 2021, Jiang et al., 2019] and GuanDan [Lu et al., 2022], which have average information set sizes
up to  $10^{23}$  and  $10^{30}$ , respectively. The substantial size of these games dramatically increases the magnitude of players'  $|\mathcal{I}_i|$  in subgame, rendering it impractical to find desirable strategies in real time, as convergence within the limited time frame cannot be guaranteed. To address this computational challenge, techniques such as Abstraction [Sandholm, 2015, Brown et al., 2015] are commonly used in Texas Hold 'em. However, it has been observed that these techniques are inadequate for games like GuanDan [Zha et al., 2021]. A recent work [Zhang and Sandholm, 2021] has partially alleviated this problem by partitioning the subgame tree according to players' knowledge order and discarding nodes that are "far away" from the current infoset. This approach can enable safe strategy refinement under specific conditions of usage in a much smaller subgame, effectively reducing the number of subgame-solving player's information sets  $|\mathcal{I}_i|$ . Nonetheless, the large number of states within other players' infosets  $|\mathcal{I}_{-i}|$  remains unmanageably large and cannot be further reduced, making it difficult to apply these approaches to large games. Moreover, the players' infoset might be complex, and enumerating all states within these sets during subgame construction can be a non-trivial task, adding to the complexity of subgame development.
To address the intricate nature of large games, we present a novel approach called generative subgame solving (GS2). Real-time refinement of these games requires the utilization of a reduced subgame that encompasses fewer players' infosets, which can be efficiently constructed and solved in an online manner while maintaining certain safety guarantees. Our proposed GS2 framework employs a generation function that generates a subset of the other player's infoset, effectively reducing the number of information states  $|\mathcal{I}_{-i}|$ . Subsequently, a subgame is constructed using nodes rooted in these states. The proposed GS2 framework enables players to construct a more manageable subgame, even in the context of large games. However, the exploitability of the refined strategy may potentially increase compared to the previous knowledge-limited subgame solving, as there is information loss during the construction of the subgame tree. To mitigate this, we begin by providing a theoretical analysis of GS2. We then propose a diversity-based generation function that produces "diverse" subsets, as determined by the analysis, to reduce the exploitability of the refined strategy. Experiments show that GS2, when combined with a diversity-based generation function, significantly enhances performance, making it comparable to the knowledge-limited subgame solving method, while requiring much less computation time.
In this paper, we make the following specific contributions:
- We propose a novel subgame solving method—generative subgame solving (GS2). Unlike traditional techniques that require the full information set, GS2 innovatively generates a subset of the opponent's info set, thereby reducing the number of information states. This allows for substantial subgame size reduction, making GS2 particularly beneficial for large games, which have historically been challenging due to their vast information set sizes.
- We provide the theoretical analysis of GS2 with arbitrary generation functions, a feature that has been largely overlooked in the past. We leverage the theoretical understanding to introduce a diversity-based generation function for GS2, leading to a significant reduction in the potential exploitability of the refined strategy, a common issue encountered in previous knowledge-limited subgame-solving techniques.
- We conduct extensive experiments for evaluating GS2 on both research games and the notoriously complex large poker game, GuanDan. The experiments underscore the practical value of our method, as GS2 successfully reduces exploitability and increases the expected payoff with appropriate generation functions. In comparison to the knowledge-limited subgame-solving method, GS2 delivers comparable performance while significantly reducing computation time. Moreover, in relatively large games with time limits, GS2 outperforms the knowledge-limited subgame solving, thus showcasing its effectiveness and efficiency.
The concept of refining a coarse strategy (also known as a blueprint) in imperfect information games has been studied for a long time [Gilpin and Sandholm, 2006]. A crucial component of modern refinement techniques is determining the new strategy, which can be adapted from the blueprint or computed from scratch.
The adaptation-based method samples the outcomes of the current games and adapts the current strategy. Online outcome sampling (OOS) method [Lisy et al., 2015] is a simulation-based algorithm that relies on Monte Carlo Counterfactual Regret Minimization [Lanctot et al., 2009], which conducts
targeted outcome sampling and updates the regret of each action, resulting in an adapted strategy. Monte Carlo Continual Resolving method [Šustr et al., 2019] combines OOS with the Continual Resolving method [Moravčík et al., 2017] to enhance sampling efficiency while minimizing memory costs associated with OOS. Parametric Monte-Carlo Policy Adaptation method [Li et al., 2020] improves this by performing Monte Carlo tree search for sampling, as well as gradient updates using the sampled trajectories to adjust the blueprint. Despite being theoretically sound, most Adaptation-based Methods are limited in application due to requiring online adaptation to be performed using the same or similar method that computes the blueprint.
On the other hand, methods that compute the new strategy from scratch have no such requirement. Endgame subgame solving methods [Ganzfried and Sandholm, 2015, Gilpin and Sandholm, 2006] first construct the subgame and assume that all the players conform to the pre-defined blueprint before reaching the subgame. The new strategy is then computed in the subgame and the player that performs endgame solving proceeds to implement the new strategy instead of the blueprint in the subgame. Conversely, safe subgame solving methods [Burch et al., 2014, Moravcik et al., 2016, Brown and Sandholm, 2017, Brown et al., 2018, Zhang and Sandholm, 2021] use an evaluation function to construct an augmented subgame, ensuring that the new strategy devised in the augmented subgame is less exploitable than the blueprint.
Search algorithms are also applied for obtaining better joint policy within teammates in collaborative games such as Hanabi [Lerer et al., 2020, Hu et al., 2021] and the bidding phases of contract Bridge [Tian et al., 2020]. For example, SPARTA [Lerer et al., 2020] utilizes exact belief updates for single-agent search and multi-agent search and retrospective belief updates for multi-agent search to handle the large belief range. Subsequently, the strategy is improved through Monte Carlo rollouts. On the other hand, instead of maintaining an explicit belief distribution in single-agent search, Learned Belief Search [Hu et al., 2021] method uses a learned belief model to sample states from the belief distribution, allowing for the application in games with large belief space. Rather than simply improve the strategy via rollouts, Joint Policy Search [Tian et al., 2020] method first decomposes the global changes of the game value into local changes at each info set, and then iteratively improves the strategy based on the decomposition. Although these approaches primarily focus on improving strategy in collaborative settings and could not be directly used in games with adversaries, the underlying idea could be helpful when developing a new technique that searches within teammates in games like GuanDan.
Our method follows the scheme of subgame solving. Different from the existing methods, our method focuses on expanding the applicability of subgame solving to large games that are intractable for existing techniques. This is achieved by generating a smaller subgame, thereby enabling the integration of techniques that enhance the safety and performance of the refined subgame strategy, such as those presented in [Burch et al., 2014, Brown and Sandholm, 2017].
It is worth noting that previous techniques, such as [Brown and Sandholm, 2015, Brown et al., 2015], also explored the real-time reduction of the entire subgame. These techniques specifically consider the subgame tree that follows a rational opponent's action and disregards the remaining parts. While these techniques appear promising, they lack a solid theoretical guarantee. Furthermore, they are only applicable to action abstraction, as the chance player's actions are determined by a fixed probability distribution, and one cannot assume the rationality of the chance player. Thus, these techniques are not suitable for real-world games that involve enormous chance outcomes, such as GuanDan.
3.1 Extensive-form Games
An extensive-form game with imperfect information can be formulated as  $G = \{N, H, P, \mathcal{I}, \{u_i\}\}$ , where  $N = \{1, 2, 3, \ldots\} \cup \{c\}$  is the set of players,  $c$  denotes the chance player who chooses actions with a fixed probability,  $H$  is the set of all possible history sequences  $a^1 a^2 \cdots a^k$  and each  $a^i$  is an action. For each history  $h \in H$ , player function  $P: H \to N$  gives the acting player. Additionally, the action set  $A(h)$  contains legal actions that player  $P(h)$  can choose, which is formally expressed as  $A(h) = \{a | h \cdot a \in H\}$ . The set of leaf nodes, or terminal nodes, is denoted by  $Z$ . For each player  $i \in N$ , the payoff function  $u_i: Z \to R$  is the payoff that player  $i$  will receive upon reaching a terminal history  $z \in Z$ . Imperfect information is represented by information sets (infosets).
Infoset  $\mathcal{I}$  is the set of information partitions of  $H$ . For any infographic  $I_i \in \mathcal{I}_i$ , histories  $h, h' \in I_i$  are indistinguishable to player  $i$ . As such, the acting player and the corresponding legal action set must be the same for all nodes in the same information set. Hence, by denoting the acting player and action set at信息发布  $I_i$  as  $P(I_i)$  and  $A(I_i)$ , we have  $P(I_i) = P(h)$ ,  $A(I_i) = A(h), \forall h \in I_i$ . We also use  $\mathcal{I}_i(h)$  to denote the信息发布 that contains  $h$  for player  $i$ .
A strategy  $\sigma_{i}(I_{i})$  is a probability distribution over the actions  $A(I_{i})$  for each info set  $I_{i}$ , and its probability of a specific action  $a$  is denoted by  $\sigma_{i}(I_{i},a)$ . Note that  $P(I_{i}) = i$ . Let  $-i$  denote all the players in the game except player  $i$ ,  $\sigma_{i} \in \Sigma_{i}$  and  $\sigma_{-i} \in \Sigma_{-i}$  denote player  $i$  and  $-i$ 's strategies, respectively. Player  $i$ 's expect payoff when all the players adhere to the profile  $(\sigma_{i},\sigma_{-i})$  is denoted by  $u_{i}(\sigma_{i},\sigma_{-i})$ . A Nash Equilibrium is a self-enforcing strategy profile  $\sigma^{*}$  that no one has an incentive to unilaterally change its strategy. Therefore,  $\forall i \in N$ ,  $u_{i}(\sigma_{i}^{*},\sigma_{-i}^{*}) \geq \max_{\sigma_{i}^{\prime} \in \Sigma_{i}} u(\sigma_{i}^{\prime},\sigma_{-i}^{*})$ . A best response strategy  $BR(\sigma_{-i}) = \max_{\sigma_{i}^{\prime}} u_{i}(\sigma_{i}^{\prime},\sigma_{-i})$  is a strategy that maximizes player  $i$ 's own payoff against  $\sigma_{-i}$ . The exploitability of strategy  $\sigma_{i}$  in a two-player zero-sum game is denoted by  $\exp (\sigma_i)$ , representing how much payoff player  $i$  can lose by changing their strategy from  $\sigma_{i}^{*}$  to  $\sigma_{i}$  against their opponent's best response strategy. Formally,  $\exp (\sigma_i) = u_i(\sigma_i^*,\sigma_{-i}^*) - u_i(\sigma_i,BR(\sigma_i))$ , where  $\sigma^{*}$  is the Nash Equilibrium strategy.
For two history  $h, h'$ , we write  $h \sqsubset h'$  if there is an action sequence leading from  $h$  to  $h'$ . The probability of reaching history  $h$  when all players act according to  $\sigma$  is represented by  $\pi^{\sigma}(h) = \Pi_{h' \cdot a \sqsubseteq h} \sigma_{P(h')} (h', a)$ , and  $\pi_i^{\sigma}(h) = \Pi_{h' \cdot a \sqsubseteq P(h') = i} \sigma_i(h', a)$  is player  $i$ 's contribution to the probability. The expected value of player  $i$  at history  $h$  when players play according to  $\sigma$  is denoted by  $v_i^{\sigma}(h) = \sum_{z \in Z} \pi^{\sigma}(h, z) u_i(z)$ , where  $\pi^{\sigma}(h, z)$  is the probability of reaching terminal node  $z$  from  $h$ , or 0 if  $h \sqsubset z$  does not hold. The value of an infoset  $v_i^{\sigma}(I)$  is the sum of the expected values of all nodes in the infoset, and each one is weighted by the player's counterfactual probability.
This can be formally expressed as  $v_{i}^{\sigma}(I) = \frac{\sum_{h\in I}\left(\pi_{-i}^{\sigma}(h)v_{i}^{\sigma}(h)\right)}{\sum_{h\in I}\pi_{-i}^{\sigma}(h)}$ . The counterfactual best response value  $u_{i}^{*}(\sigma_{-i}|I)$  to strategy  $\sigma_{-i}$  upon reaching info set  $I$  is the best value for player  $i$  against  $\sigma_{-i}$ . This can be defined as  $u_{i}^{*}(\sigma_{-i}|I) = v_{i}^{\langle BR(\sigma_{-i}),\sigma_{-i}\rangle}(I)$  and  $u_{i}^{*}(\sigma_{-i}|Ia) = v_{i}^{\langle BR(\sigma_{-i}),\sigma_{-i}\rangle}(I,a)$ .
An imperfect information subgame (hereafter subgame) is a set of nodes in a game tree, in which information is not divided over infosets. Formally, a subgame  $S$  is a set of nodes  $h \in H$ , such that  $\forall h' \in H$ , if  $\exists h \in S$  and  $h' \sqsupseteq h$ , then  $h' \in S$ . Moreover,  $S$  is closed under infoset:  $\forall h \in S$ , if  $h \in I_i$  and  $h' \in I_i$  for some player  $i$ , then  $h' \in S$ . We define  $S_{top}$  as the set of earliest-reachable nodes in  $S$ , then  $h \in S_{top}$  if and only if  $h \in S$  and  $h' \notin S$  for all  $h' \sqsubset h$ .
3.2 Prior Subgame Solving Methods
In this section, we review the existing methods of subgame solving in imperfect information games. Without loss of generality, we assume that Player 1 (P1) is the responsible party for subgame solving. Within this context, P1 is provided with an initial blueprint strategy and aims to optimize it. During a playthrough, P1 reaches an infoSet  $I$  and carries out subgame solving to refine the blueprint for the remainder of the game.
As an initial step, standard subgame-solving methods enumerate the top node of the subgame whenever a player reaches a particular information set, allowing them to have direct access to the entire subgame tree. Once the subgame  $S$  is constructed, the augmented subgame or gadget game is formed and solved by an appropriate equilibrium finding algorithm. Most subgame solving methods differ mainly in the methodology of constructing the augmented subgame.
The augmented subgame  $G'$  in Endgame solving [Ganzfried and Sandholm, 2015] begins with a chance node that leads to all nodes in  $S_{top}$ . The probability of reaching node  $h \in S_{top}$  is proportional to the reach probability of node  $h$  in the original game  $G$ , provided that the players are playing according to the blueprint  $\sigma$ , that is, reaching  $h$  with probability  $\frac{\pi^{\sigma}(h)}{\sum_{h' \in S_{top}} \pi^{\sigma}(h)}$ . The augmented subgame  $G'$  is then solved, producing a new strategy  $\sigma_1^S$  for P1. For the remainder of the game, P1 is expected to replace the blueprint with  $\sigma_1^S$ .
Subgame resolving [Burch et al., 2014] differs from endgame solving in the structure of augmented subgames. The initial node of the augmented subgame is a chance node, with Player 2 (P2)'s alternative node, denoted by  $h_r$ , added between each node  $h \in S_{top}$  and the initial node. Each  $h_r$  is reached with a probability that is in proportion to P2's probability of reaching  $h$ , if it tries
to do so—that is  $\pi_{-2}^{\sigma}(h)$ . The set of  $h_r$  is denoted by  $S_r$ , with the效果图 in  $S_r$  being determined according to P2's效果图 in  $S_{top}$ . P2 is presented with two choices for each alternative node: entering  $h_{top}$ , or terminating the game and receiving an alternative payoff at  $\mathcal{I}_2(h_{top})$ , which is equal to the counterfactual best response of the blueprint, or  $u_2^* (\sigma |\mathcal{I}_2(h_{top}))$ .
Subgame resolving develops a strategy that will not be worse than the blueprint. Conversely, Maxmargin subgame solving [Moravecik et al., 2016] seeks to optimize the strategy to ensure that P2 will receive a lower payoff if it chooses to enter the subgame. Formally, the margin of a strategy  $\sigma_1^S$  in subgame  $S$  is defined as  $M(\sigma^S |I_2) = u_2^*(\sigma_1|I_2) - u_2^*(\sigma_1^S |I_2)$ . Maxmargin subgame solving hence attempts to find a strategy  $\sigma_1^S$  that maximizes  $\min_{I_2 \in S_{top}} M(\sigma^S |I_2)$ .
On the other hand, knowledge-limited subgame solving (KLSS) [Zhang and Sandholm, 2021] aims to construct a small augmented subgame while providing safety guarantees. KLSS partitions  $S_{top}$  into different knowledge orders based on the distance from  $I$  in the info set hypergraph of the subgame. The  $k$ -th order knowledge limited subgame solving constructs the subgame using nodes at most  $k - 1$  units away from the current history  $h$  in the original  $S_{top}$ . KLSS takes advantage of the fact that nodes outside the  $k$ -th order and their descendants can be discarded if P1's strategy at these nodes is fixed after the payoff is correctly scaled by nodes inside the  $(k + 1)$ -th order. Hence, the  $S_{top}$  of the subgame in the 1st order KLSS (1-KLSS) can be as small as the player's current info set and its descendants, which significantly reduces the subgame size.
4 Generative Subgame Solving (GS2)
4.1 General GS2 Framework
In Section 3.2, all methods except KLSS perform subgame solving on the entire subgame tree, which is intractable for large-size games that are difficult to abstract. KLSS attempts to address the problem by limiting the set  $S_{top}$  to a lower knowledge order, which reduces the subgame size but is still not feasible for games with a large info set size. For example, in heads-up Texas Hold'em, where one player only holds two cards, the info set size is no more than  $\binom{52}{2} = 1326$ , making it easy to construct for 1-KLSS. On the contrary, in games such as GuanDan, with four players each holding 27 cards at the start, the info set size can be as large as  $10^{30}$ . Consequently, constructing and storing the corresponding subgame on a computer becomes impractical.
Even if the subgame is somehow constructed, current equilibrium finding [Lanctot et al., 2009, Farina et al., 2021] algorithm that bound player  $i$ 's regret by  $O\left(\frac{\sqrt{|\mathcal{I}_i|}}{T}\right)$  is not guaranteed to converge in real-time. This is due to the presence of up to  $10^{20}$  other players'infosets in a single player  $i$ 's infoset in GuanDan. Furthermore, the subgame cannot be abstracted using the techniques in Texas Hold'em [Zha et al., 2021]. Thus, simply applying 1-KLSS would exceed the time limit, necessitating a further reduction in subgames.
Motivated by this, we propose a Generative Subgame Solving (GS2) method to reduce the memory and time cost by solving only subgames within a subset of the entire 1-KLSS subgame tree. Specifically, let  $\mathcal{Q} = \{Q_1, Q_2, \ldots, Q_R\}$  be the set of all subsets of  $S_{top}$ , such that if  $h \in Q_r$ ,  $\forall h' \in \mathcal{I}_2(h)$ ,  $h' \in Q_r$ . Here,  $Q_r$  is referred to as a block of  $S_{top}$ . GS2 then introduces a generation function with probability distribution  $f: \mathcal{I}_1 \to \Delta^{|Q|}$ , where  $\Delta^{|Q|}$  is the  $|\mathcal{Q}|$ -simplex. Upon reaching infoSet  $I$ , GS2 first generates  $Q_r \sim f(I)$  and then constructs the generative subgame  $S$  such that,  $\forall h \in Q_r$ ,  $h \in S$  and  $\forall h' \supset h, h' \in S$ .
Let  $f(I, Q_r)$  be the probability of  $Q_r$  and  $\omega(h) = \sum_{r: h \in Q_r} f(\mathcal{I}_1(h), Q_r)$  be the probability that  $h$  is considered.
Proposition 4.1. Given a strategy  $\sigma_{1}$  in a two-player zero-sum game  $G$ , reached infoset  $I_{1}$ , and a generation function  $f$ , the refined subgame strategy  $\sigma_{1}^{S}$  is computed by GS2 with safe subgame solving techniques. Let  $\sigma_{1}^{\prime}$  be the strategy that plays according to  $\sigma_{1}^{S}$  in subgame  $S$  and  $\sigma_{1}$  elsewhere. Let  $\delta(\sigma_{1}, h) = \max_{\sigma_{1}^{\prime}} u_{2}^{*}(\sigma_{1}^{\prime}|\mathcal{I}_{2}(h)) - u_{2}^{*}(\sigma_{1}|\mathcal{I}_{2}(h))$ . Then the exploitability of the expected strategy  $\exp(\mathbb{E}[\sigma_{1}^{\prime}])$  is bounded by
$$
\exp \left(\mathbb {E} \left[ \sigma_ {1} ^ {\prime} \right]\right) \leq \exp \left(\sigma_ {1}\right) + \max  _ {h \in I _ {1}} (1 - \omega (h)) \pi_ {- 2} ^ {\sigma_ {1} ^ {\prime}} (\mathcal {I} _ {2} (h)) \delta \left(\sigma_ {1}, h\right). \tag {1}
$$