SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 4.1 gives the upper bound for GS2 with arbitrary generation functions.  $\delta$  can be regarded as a metric for evaluating the effectiveness of the blueprint strategy, measuring its improvement compared to the worst strategy. While GS2 may increase exploitability, it can efficiently reduce the size of the subgame. In balanced games where each unobserved move of other players and chance events have roughly the same size of potential outcomes, only  $\frac{|Q_r|}{|S_{top}|}$  of the other player's info set is considered, thus, the approximate equilibrium can be efficiently computed. Decreasing  $|Q_r|$  reduces the computational requirement, but it also leads to lower  $\omega(h)$  and a potential increase in exploitability. The choice of the generation function  $f$  can be a trade-off between potential improvement and safety.
Intuitively, Proposition 4.1 implies that the new strategies solved by GS2 will be less exploitable if histories  $h \in S_{top}$  are more frequently considered in the sampled  $Q_r$  (resulting in lower  $1 - \omega(h)$ ) or the blueprint does not perform well (leading to lower  $\delta(\sigma_1, h)$ ). As such, it is suitable to apply GS2 in games with a coarse blueprint, unlike previous unsafe subgame solving methods [Ganzfried and Sandholm, 2015, Gilpin and Sandholm, 2006] that necessitate the blueprint to be close to a Nash equilibrium to construct accurate belief distributions. To further enhance safety, one can apply GS2 in large subgames and switch to other safe subgame-solving methods as the subgame size decreases, since GS2 is intentionally designed to be effective in handling large subgames.
For large-scale games, sampling a block from a non-trivial generation function  $f$  is challenging due to the numerous blocks involved. However, randomly generating a single state  $h$  in current info set  $I$  is feasible (e.g., randomly choosing a valid opponent hand in poker), and the block can be obtained by repeating the random generation process and adding all  $h'$  that is in the same player 2's info set of  $h$ . This approach is referred to as the random generation function. The generated size  $k$  is defined as the size of a randomly generated set of  $h$ . When using this function, the upper bound of exploitability increment for the expected strategy will decrease in proportion to the size of the generated block. According to Proposition 4.1, we have
$$
\exp \left(\mathbb {E} \left[ \sigma_ {1} ^ {\prime} \right]\right) \leq \exp \left(\sigma_ {1}\right) + \left(1 - \frac {k}{\left| I _ {1} \right|}\right) \max  _ {h \in I _ {1})} \pi_ {- 2} ^ {\sigma_ {1} ^ {\prime}} \left(\mathcal {I} _ {2} (h)\right) \delta \left(\sigma_ {1}, h\right), \tag {2}
$$
for the expected strategy  $\mathbb{E}[\sigma_1^{\prime}]$
To enhance the safety of GS2, it is suggested by Equation (2) to generate as many states as time permits. While increasing the number of states would improve the guarantee, it would also increase the time and resource requirements for solving the subgame. However, the generated size can be controlled and kept relatively small compared to the original subgame. Moreover, techniques such as Abstraction, which traverse the entire效果图, can be combined to further reduce the subgame size while maintaining the theoretical guarantee. It is preferable to have larger generated blocks, which require simpler techniques to operate on them. Previous techniques [Ganzfried and Sandholm, 2014] that involve comparing all potential outcomes would be time-consuming in large games. In contrast, GS2 reduces the subgame size by considering the counterfactual value of the states according to a diversity-based generation function. In general, a diversity-based generation function generates a block  $Q'$  such that the maximum earth mover's distance between any player 2's效果图 and  $Q'$  is minimized:
$$
Q ^ {\prime} = \arg \min  _ {Q} \max  _ {I _ {2} \subset Q _ {r} \backslash Q} d i s t (I _ {2}, Q), \tag {3}
$$
where  $dist(I_2, Q) = \sum_i^N \min_{I_2' \in Q} |u_i^*(\sigma_1|I_2') - u_i^*(\sigma_1|I_2)|$ . Let  $\mathcal{H}(I_2')$  be the set of infoset that is not vastly dissimilar from  $I_2'$ , specifically defined as  $\{I_2 \in Q_r \backslash Q'|I_2' = \arg \min_{I_2 \in Q_r \backslash Q'} \sum_i^N |u_i^*(\sigma_1|I_2') - u_i^*(\sigma_1|I_2)|\}$ . The counterfactual reach probability of infoset  $I_2'$  is scaled up by  $\sum_{I_2 \in \mathcal{H}(I_2')} \pi_{-2}^{\sigma_1}(I_2)$ . In two-player zero-sum games, it is equal to finding a subset such that the alternative values of infosets are likely to be uniformly distributed in range  $[\min_{I_2 \subset Q_r} u_2^*(\sigma_1|I_2), \max_{I_2 \subset Q_r} u_2^*(\sigma_1|I_2)]$ , which can be efficiently obtained through techniques such as dynamic programming.
Intuitively, the diversity-based generation function generates a block that is the most "diverse" and is able to encompass a wide range of situations (including unobserved others' or chance behaviors) with a small number of infosets. The pseudo-code for GS2 utilizing the diversity-based generation function is presented in Appendix C.8.
(a) Part of the Liar's Dice game tree.
(b) 1-KLSS subgame tree of info set  $I_{1}$ .
(c) GS2 subgame tree of info set  $I_{1}$ .
Figure 1: Example of GS2 in Liar's Dice. The white node is a chance node, representing exogenous events. Nodes are colored in blue if the acting player is player 1 and red for player 2. (a) Part of the Liar's Dice game tree.  $h_{i,j}$  is the history that the two players' dice landed on  $i$  and  $j$ , respectively. For simplicity, descendants of  $h_{ij}$  that would not be changed in the standard implementation of 1-KLSS and GS2 are omitted. (b) The 1-KLSS subgame tree of infoSet  $I_1$ . (c) The GS2 subgame tree of infoSet  $I_1$ . GS2 only constructs the subgame with a subset of  $I_1$ . The subset shown is for illustration purposes and would be generated by a generation function in practice.
4.2 Example of How GS2 Works
To further clarify the concept of GS2, we present a simple example. Figure 1 illustrates an example of GS2 applied in Liar's Dice with 8-sided dice. In this example,  $c$  (the white node) is the chance node, representing exogenous events. Nodes are colored in blue if the acting player is player 1 and red for player 2. Figure 1(a) shows a portion of the Liar's Dice game tree. Each player dice at the beginning. For simplicity, the remainder of the game is omitted and replaced by the gray area, as the nodes rooted at each history  $h$  will not be altered in the subgame-solving process.
If player 1 wishes to refine its strategy after determining that info set  $I_{1}$  has been reached, the subgame constructed by 1-KLSS is shown in Figure 1(b). The chance nodes are redundant since they only have one chance outcome but are retained for consistency.  $\hat{u}_{2,I_m}^*$  is the scaled alternative value of  $\hat{u}_2^* (\sigma_1|\mathcal{I}_2(h_{1m}))$  for player 1's blueprint  $\sigma_{1}$ . Formally,  $\hat{u}_{2,I_m}^* = \frac{\hat{u}_2^*(\sigma_1|\mathcal{I}_2(h_{1m}))\sum_{h'\in\mathcal{I}_2(h_{1m})}\pi_{-2}^{\sigma_1}(h')}{\pi_{-2}^{\sigma_1}(h_{1n})}$ . The payoff for player 2 is then subtracted by  $\hat{u}_{2,I_m}^*$  upon reaching the corresponding node. Additional payoffs should also be taken into consideration, accounting for the omitted subtree.
GS2 further reduces the subgame size, as shown in Figure 1(c). It only constructs a subgame for a subset of  $I_{1}$ , which is generated by a diversity-based generation function. This not only increases the computational efficiency of the subgame solving in large games but also refines the blueprint by selectively focusing on the most typical portions of the game tree.
5.1 Evaluation on Research Games
To evaluate the practical performance of GS2, experiments are conducted on a range of commonly used research games with varying效果图 sizes.
(f)
Figure 2: Exploitability of GS2 with random generation function (GS2-R) and diversity generation function (GS2-D) in different games (Lower is better). GS2 and 1-KLSS-limited are required to refine the strategy in 3 seconds, while 1-KLSS-unlimited and Unsafe subgame solving method are not. (a) 3-rank limit Leduc poker. (b) Liar's Dice with 6-sided dice. (c) Liar's Dice 7 with 7-sided dice. (d) 5-card Goofspiel, random order with 3 rounds. (e) 10-card Goofspiel, ascending order with 3 rounds. (f) 12-card Goofspiel, ascending order with 3 rounds.
Implementation details. All games are implemented using the open-source library Open-Spiel [Lanctot et al., 2019]. In each game, a blueprint is first computed via the CFR algorithm, after which GS2 is applied at each infographic after both players have taken one action to refine the blueprint. The subgame is solved in a nested fashion, utilizing Maxmargin solving with linear programming whenever feasible, and otherwise employing MCCFR [Lanctot et al., 2009]. The unsafe subgame solving method is implemented based on Ganzfried and Sandholm [2015], Brown and Sandholm [2017]. The computation time is limited for 1-KLSS [Zhang and Sandholm, 2021] and GS2. In these methods, the player is allotted a maximum of 3 seconds to compute an approximate Nash Equilibrium after constructing a subgame. In order to provide a comprehensive evaluation of GS2's performance, we also present the results of 1-KLSS and unsafe subgame solving, where there are no time constraints imposed on the computation. It is important to note that GS2 is not expected to surpass the performance of these two methods, as obtaining an exact Nash Equilibrium may require significantly more time. For instance, when solving the subgame of Liar's Dice with 7 dice sides, the computation time for the strategy in unsafe subgame solving can reach up to 122 seconds, and 41 seconds for 1-KLSS, rendering them impractical for real-time decision-making. In contrast, GS2 with  $k = 3$  only necessitates 10 seconds of computation time at the same info set<sup>2</sup>, showcasing its efficiency.
Results. The effect of varying the value  $k$  (i.e., the number of other player's infosets in the generated block) in the diversity-based generation function is evaluated, and the exploitability of blueprints and refined strategies is presented in Figure 2. Experimental results demonstrate that GS2 is capable of effectively reducing exploitability in practice for a coarse blueprint in limited computation time, despite the absence of safety guarantees. As the size of the game increases, GS2 with diversity generation exhibits superior performance compared to 1-KLSS with limited computation time, as evident from the results presented in Figure 2(e) and Figure 2(f). However, in the case of 10-card Goofspiel and 12-card Goofspiel, where the infoset size is extremely large, GS2 with random generation is comparatively inferior to that of other methods. This discrepancy arises due to the inherent limitation of randomly generating a block, as it fails to yield a desirable subgame within the confines of an infoset size reaching up to 1110. Consequently, this inability to generate a good
2The experiment was conducted on an Intel(R) Xeon(R) Gold 6242R CPU @ 3.10GHz
subgame leads to an increase in exploitability. For 5-card Goofspiel and Liar's Dice, GS2 with diversity generation performs closely to 1-KLSS-unlimited but with significantly less computation time, showing the effectiveness of GS2. Additionally, the results suggest that as  $k$  increases, the exploitability of the refined strategy decreases, which aligns with previous theoretical analysis.
For other baselines, the unsafe subgame solving demonstrates excellent performance in 3-rank limit Leduc poker, but it performs inferior in Liar's Dice due to the lack of a safety guarantee. While 1-KLSS without time limits exhibits the best performance in most games, it requires significantly more computation time than GS2, as discussed previously.
5.2 Evaluation on GuanDan
We use the techniques in Appendix C.1 to create an agent for the game of GuanDan. The choice to test GS2 on GuanDan is driven by the game's complexity, which surpasses that of other games such as DouDizhu, and the availability of several benchmark agents due to a GuanDan AI competition. GS2 is tested against the two champions of the competition to evaluate its performance:
- The first champion is a rule-based agent and is tested against top human professionals. Despite never having won in a GuanDan tournament, the agent has achieved victory for certain decks in the tournament and has proven to be challenging for experts who are not familiar with it.
- The second champion is a DMC [Zha et al., 2021] agent with warm-starting and post-processing that has a  $90\%$  winning percentage against the first champion in the GuanDan tournament. We believe it to be an expert-level agent, although it has not been tested against human experts.
Implementation details. The state-action value function of GuanDan is created by DMC and the subgame is further simplified as outlined in C.2. The approximation is implemented based on the concept of not just sampling from opponent infosets but also within the infoset itself to manage complexity. This approach is like applying a specific generation function to the opponent's infoset. It leads to a conservative strategy refinement by assuming opponents are aware of the player's exact private information. Such an assumption prevents a drastic rise in exploitability at the present infoset. This method is influenced by patterns observed in the game of GuanDan. In this game, as actions unfold, uncertainty diminishes, thus human players are expected to guess the others' cards, especially near the end of the game, which is crucial for expert players. Therefore, considering the worst case that the opponents know the player's cards would be practical. Even though this idea seems somewhat domain-specific, we believe it is also applicable to similar games.
During the entire game, only one player performs subgame solving, while the teammate continues to play according to the state-action value function. In other words, the teammate will select the action with the highest value among all legal actions. The opponent team consists of two identical agents who do not communicate during the game. To better evaluate performance, we conduct tournaments with a fixed game level. In each tournament, a deck is played four times to minimize variance. The player who performs GS2 rotates their seat to the next player after each round of play. At the end of each game, players will receive a score according to the order in which the player and its teammate empty their hand, which is equal to the value of levels they would prompt in the original GuanDan game for the winning team and the negative value for the losing team. A more detailed description of GuanDan is given in Appendix B. The tournament is then repeated 100 times and the average score of GS2 in each position is calculated.
Results. We first present the results when playing against the second champion. Different settings of  $k$  are applied in the tournament. Despite the remarkably small value of  $k$  relative to the size of the infoset, the results shown in Figure 3(a) indicate that the application of GS2 leads to a significant performance gain. It also shows that as more states are generated by GS2, the better its performance becomes. The results of GS2 against the first champion are given in Figure 3(b). Compared to the first champion, GS2 has less improvement in the average score. We believe that this is attributed to the outstanding performance of the strategy constructed by the value function, which results in the improvement not being fully reflected in the average score. Overall, the experiment results suggest that GS2 is able to improve the performance of a given blueprint in large games with desirable subgame sizes.
(a) Performance against the second champion agent.
(b) Performance against the first champion agent.
Figure 3: The average score of GS2 against two prior agents at each position in GuanDan (Higher is better). The score of the blueprint (blue) gives the baseline value that agents could achieve without subgame solving at each position.
We propose a novel subgame-solving method, named GS2, which is designed to address the challenges posed by games with imperfect information and large info set sizes. GS2 only constructs a partial subgame tree, thereby reducing the memory and time costs associated with conventional subgame solving. Our method is able to construct robust strategies for large games with unmanageable info set sizes. We theoretically analyze the exploitability bound for GS2 and propose a diversity-based generation function to mitigate the issue of unsafeness. Through experimental evaluations in research games and a complex poker game-GuanDan, we demonstrate that GS2 indeed decreases exploitability and improves the performance of the given blueprint in practice.
While GS2 shows promising performance with a simple diversity-based generation function, it is important to note that further research is needed to address the limitations of the method, particularly when alternative values are not adequate for describing the state. Additionally, it remains an open question as to whether a safe generation function exists that can further expand the applicability of GS2.
We thank the anonymous reviewers for their constructive comments. This work was supported in part by Science and Technology Innovation 2030 New Generation Artificial Intelligence Major Project(No.2018AAA0100905), the National Natural Science Foundation of China(No.62192783, No.62106100, No.62276142), Primary Research & Development Plan of Jiangsu Province (No.BE2021028), Jiangsu Natural Science Foundation (BK20221441), Jiangsu Provincial Double-Innovation Doctor Program (JSSCBS20210021), and in part by the Collaborative Innovation Center of Novel Software Technology and Industrialization.
Noam Brown and Tuomas Sandholm. Safe and nested subgame solving for imperfect-information games. Advances in neural information processing systems, 30:689-699, 2017.
Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418-424, 2018.
Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365(6456): 885-890, 2019.
Noam Brown and Tuomas W Sandholm. Simultaneous abstraction and equilibrium finding in games. 2015.
Noam Brown, Sam Ganzfried, and Tuomas Sandholm. Hierarchical abstraction, distributed equilibrium computation, and post-processing, with application to a champion no-limit texas hold'em
Tuomas Sandholm. Abstraction for solving large incomplete-information games. In Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 4127-4131, 2015.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.
Michal Šustr, Vojtěch Kovářík, and Viliam Lisý. Monte carlo continual resolving for online strategy computation in imperfect information games. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 224-232, 2019.
Yuandong Tian, Qucheng Gong, and Yu Jiang. Joint policy search for multi-agent collaboration with imperfect information. Advances in Neural Information Processing Systems, 33:19931-19942, 2020.
Daochen Zha, Jingru Xie, Wenye Ma, Sheng Zhang, Xiangru Lian, Xia Hu, and Ji Liu. Douzero: Mastering doudizhu with self-play deep reinforcement learning. In International Conference on Machine Learning, pages 12333-12344. PMLR, 2021.
Brian Zhang and Tuomas Sandholm. Subgame solving without common knowledge. Advances in Neural Information Processing Systems, 34:23993-24004, 2021.
Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. Advances in neural information processing systems, 20, 2007.