SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1: Build a full binary tree  $T_{-n}$  with the height  $\lceil \log_2(n - 1) \rceil$  for  $-n$  with the set of internal nodes  $\mathcal{N}_{T_{-n}}$  and  $|\mathcal{N}_{T_{-n}}| = n - 2$
2: for each  $i$  in  $\{1, \ldots, n - 1\}$  do
3: Search  $T_{-n}$  to replace  $i$  with  $n$  in each node including  $i$  to form a binary tree  $T_{-i}$  with the set of internal nodes  $\mathcal{N}_{T_{-i}}$
4: end for
5:  $\underline{\mathcal{N}}\gets \cup_{i\in N}\mathcal{N}_{T_{-i}}$
where we could add  $|N''| > 1$  to ensure that Eq.(11) and Eq.(12) do not generate the same constraints. Appendix A shows some examples of these constraints. Eq.(11) represents the relation between the correlation plan  $x_{N'}$  and the mixed strategy  $x_i$  (i.e., the special correlation plan  $x_{\{i\}}$ ) for each  $i \in N'$ , and Eq.(12) represents the relation between the correlation plan  $x_{N'}$  and the correlation plan  $x_{N'}$  or  $x_{N'}$ . Equivalently, Eqs.(2), (11), and (12) represent the marginalization constraints that independent probability distributions ought to obey, where  $x_{N'}$  is the joint distribution (represented by Eq.(2)) of independent distributions  $x_i$  for all  $i \in N'$  (represented by Eq.(11)) or independent distributions  $x_{N'}$  and  $x_{N'}$  (represented by Eq.(12)).
We now show the effectiveness of our correlation plans by showing our method strictly reduces the feasible solution space after the relaxation. Reducing the feasible solution space will make the program efficiently solvable, as shown in the experiments. Let  $\mathcal{M}$  be the original feasible solution space for the original multilinear program that is transformed into a bilinear program according to  $\mathcal{N}$ , i.e.,  $\mathcal{M}$  is particularly constrained by Eqs.(1b), (3a), and (3b). We define the convex relaxation space  $\mathcal{R}$  as using Eq.(6) to represent each bilinear term in Eq.(3a), i.e.,  $\mathcal{R}$  is particularly constrained by Eqs.(1b), (6), and (3b). We define our tight relaxation space  $\mathcal{T}$  based on our correlation plans, i.e.,  $\mathcal{T}$  is particularly constrained by Eqs.(1b), (2), (11), (12), and (3b). (Proofs are in Appendix B.)
Theorem 2.  $\mathcal{M} \subset \mathcal{T} \subset \mathcal{R}$ , i.e.,  $\mathcal{T}$  is strictly smaller than  $\mathcal{R}$  but still includes  $\mathcal{M}$ .
The property  $\mathcal{M} \subset \mathcal{T} \subset \mathcal{R}$  means that: i) we can use  $\mathcal{T}$  to strictly reduce the feasible solution space after the relaxation, and ii) restricting the feasible solution space to  $\mathcal{T}$  does not reduce the space of NEs and then guarantees optimality for the original program. We now explicitly restrict the feasible solution space to  $\mathcal{T}$  by adding Eqs.(2), (11), and (12) to Program (5) for any binary collection  $\mathcal{N}$ :
$$
\max  _ {x} g (x) \tag {13a}
$$
$$
\text {s . t . E q s .} (1 b) - (1 g), (2), (3), (4), (1 1), (1 2). \tag {13b}
$$
Theorem 3. The optimal solution of Program (13) maximizes  $g(x)$  over the space of NEs.
Using the bilinear constraint Eq.(3a) in Program (13) is necessary for computing an optimal NE by solving Program (13) because  $\mathcal{M} \neq \mathcal{T}$ . Appendix C shows that, after removing Eq.(3a) in Program (13), the inefficiency can be arbitrarily large, and the resulting strategy profile may not be an NE.
3.3 Minimum-Height Binary Trees
In Program (13), we need to add a set of linear constraints and bilinear constraints for each correlation plan corresponding to each element in any binary collection  $\mathcal{N}$ . The size of the vanilla binary collection  $\overline{\mathcal{N}}$  is  $2^n - (n + 2)$ , which grows exponentially with the number of players. In this section, we propose building minimum-height binary trees to obtain a minimum binary collection. Our binary collection gives us a minimum set of correlation plans, which requires significantly fewer bilinear terms than  $\overline{\mathcal{N}}$ .
There are different ways to divide a subset of players, which determines different binary collections. For example,  $N' = \{1, 2, 3, 4\}$  can be divided into  $\{1, 2\}$  and  $\{3, 4\}$  or  $\{1\}$  and  $\{2, 3, 4\}$ , which will lead to different binary collections. Therefore, for obtaining a minimum binary collection  $\mathcal{N}$ , the challenge is how to effectively divide each element in  $\mathcal{N}$ .
Algorithm 2 CRM
1: Input: A game  $G = (N, A, u)$  and an objective function  $g(x)$
2: A binary collection  $\underline{\mathcal{N}}\gets$  The output of Algorithm 1
3: Create Eqs.(1b)-(1g)
4: Create Eqs.(3), (2), (4), (11), and (12) according to  $\mathcal{N}$
5:  $x^{*} \gets$  An optimal solution by solving Program (13) based on  $\underline{\mathcal{N}}$ , i.e.,  $\max_x g(x)$  s.t. Eqs.(1b) - (1g), (2), (3), (4), (11), (12)
6: Output: An optimal NE  $x^{*}$ .
To overcome this challenge, we propose building a minimum-height binary tree for each element in  $\{-i\mid i\in N\}$  and ensuring that the number of internal nodes in these binary trees is the minimum. The binary division for each element in a binary collection  $\mathcal{N}$  creates a binary tree for each
Figure 1: Binary trees for  $-5$  and  $-3$
element in  $\{-i\mid i\in N\}$ . For example, Figure 1(a) is a binary tree for  $-5 = \{1,2,3,4\}$ , and Figure 1(b) is a binary tree for  $-3 = \{1,2,5,4\}$  in five-player games. Each binary tree for  $-i$  is a full binary tree, i.e., each internal node has two children, with  $n - 2$  internal nodes and  $n - 1$  leaf nodes, where the height is the number of internal nodes on the longest path from the root to a leaf (e.g., the height in Figure 1(a) is 2). Details for these binary trees are shown in Appendix D. We can then build a full binary tree  $T_{-n}$  with the minimum height  $\lceil \log_2(n - 1)\rceil$  for  $-n$  and then replace  $i$  with  $n$  in the nodes of  $T_{-n}$  to obtain  $T_{-i}$  for each  $i\in -n = \{1,\dots ,n - 1\}$ . That creates  $n$  full binary trees for  $\{-i\mid i\in N\}$ . This procedure is shown in Algorithm 1 (details are shown in Appendix D), generating our minimum binary collection  $\underline{\mathcal{N}}$  including all internal nodes in these trees. For example, Figure 1(a) builds a binary tree  $T_{-5}$ , and Figure 1(b) obtains  $T_{-3}$  by replacing 3 with 5 in  $T_{-5}$ . Generally, we only need to create at most  $\lceil \log_2(n - 1)\rceil$  new internal nodes to build a minimum-height binary tree for each  $-i$  with  $i\in -n$ . Then  $|\underline{\mathcal{N}} |$  is at most  $n - 2 + (n - 1)\lceil \log_2(n - 1)\rceil$ , i.e.,  $O(n\log n)$ .
Theorem 4.  $\underline{\mathcal{N}}$  generated by Algorithm 1 is a binary collection, and  $O(n\log n)$  for the size of  $\underline{\mathcal{N}}$  is the minimum size of all binary collections of a game  $G$ . (Proofs are in Appendix B.)
$\underline{|\mathcal{N}|}$  only grows sub-quadratically with  $n$  and is much smaller than  $|\overline{\mathcal{N}}| = 2^n - (n + 2)$  for  $\overline{\mathcal{N}}$ . Then  $\underline{\mathcal{N}}$  requires fewer bilinear terms than  $\overline{\mathcal{N}}$  when  $n > 3$ . For example, in a seven-player game with two actions for each player, by using  $\underline{\mathcal{N}}$  with  $|\underline{\mathcal{N}}| = 21$  correlation plans, the number of bilinear terms is 564, which is much smaller than 2044 by using  $\overline{\mathcal{N}}$  with  $|\overline{\mathcal{N}}| = 119$  correlation plans. Table 3 of Appendix G shows more examples. Note that Algorithm 1 cannot reduce the number of internal nodes when  $n = 3$  because each element in  $\{-i \mid i \in N\}$  includes only two players in three-player games. Our algorithm, CRM, is solving Program (13) based on  $\underline{\mathcal{N}}$ , which is shown in Algorithm 2 and is illustrated in Appendix A.
3.4 Complexity
The problem of finding an optimal NE is NP-hard [9], and our algorithm, CRM, i.e., Program (13) based on  $\mathcal{N}$  generated by Algorithm 1, is a mixed-integer bilinear program, whose scalability is mainly affected by the number of bilinear terms and integer variables. Generally, the problem of solving a linear integer program is NP-hard, and the time complexity is  $O(I^2(EC^2)^{2E+3})$  [32], where  $I$  is the number of integer variables,  $E$  is the number of constraints containing integer variables, and  $C$  is the maximum value among constants and the range of integer variables in these constraints. Theoretically, each bilinear term can be represented by a mixed-integer linear program by introducing a new set of constraints and binary integer variables [21]. Suppose each bilinear term introduces  $I'$  integer variables and  $E'$  constraints, and each player has  $m$  actions. Program (5) based on  $\overline{\mathcal{N}}$  has  $mn$  binary integer variables with  $mn$  constraints and the following number of bilinear terms:
$$
\sum_ {N ^ {\prime} \in \overline {{\mathcal {N}}}} \prod_ {i \in N ^ {\prime}} | A _ {i} | \leq (2 ^ {n} - n - 2) m ^ {n - 1} \leq 2 ^ {n} m ^ {n - 1}.
$$
Then the time complexity for solving the Program (5) based on  $\overline{\mathcal{N}}$  is  $O(I_1^2 (E_1C^2)^{2E_1 + 3})$  where  $I_{1} = 2^{n}m^{n - 1}I^{\prime} + mn$  and  $E_{1} = 2^{n}m^{n - 1}E^{\prime} + mn$ . Program (13) based on  $\underline{\mathcal{N}}$  has  $mn$  binary
Table 1: Part of experimental results (more results are in Tables 4 and 5 of Appendix H). The format is: Average Runtime  $\pm$ $95\%$  Confidence Interval (Percentage of Games not Solved within the Time Limit) (Utility Gap). Note that the unit of the runtime is second, the case that all games have been solved with the time limit should be  $(0\%)$  and is omitted, we only need to care about the utility gap (a larger gap means losing more) for EXCLUSION, and the utility gap  $\infty$  represents EXCLUSION cannot return a solution within the time limit.
Random Game	Runtime ± 95% Confidence Interval (Percentage of Games not Solved) (Utility Gap)
Vary	(n,m)	CRM	MIBP	ENUMPOLY	EXCLUSION
n	(3,2)	0.01 ± 0	0.02 ± 0	0.03 ± 0.01	31 ± 41 (gap:15%)
(5,2)	0.2 ± 0.1	0.5 ± 0.4	11 ± 4	753 ± 148 (73%) (gap:64%)
(7,2)	25 ± 17	429 ± 131 (20%)	1000 ± 0 (97%)	835 ± 119 (80%) (gap:53%)
m	(3,5)	0.2 ± 0.03	0.3 ± 0.1	1000 ± 0 (100%)	1000 ± 0 (100%) (gap:67%)
(3,8)	4 ± 3	247 ± 140 (17%)	1000 ± 0 (100%)	1000 ± 0 (100%) (gap:∞)
(3,10)	9 ± 9	334 ± 167 (30%)	1000 ± 0 (100%)	1000 ± 0 (100%) (gap:∞)
(3,13)	38 ± 21	342 ± 151 (27%)	1000 ± 0 (100%)	1000 ± 0 (100%) (gap:∞)
GAMUT Game	CRM	MIBP	ENUMPOLY	EXCLUSION
Random LEG	2 ± 1	1000 ± 0 (100%)	1000 ± 0 (100%)	986 ± 27 (97%) (gap:11%)
Random graphical	0.1 ± 0.1	803 ± 140 (83%)	50 ± 30	971 ± 55 (97%) (gap:32%)
Uniform LEG	2.2 ± 1	1000 ± 0 (100%)	1000 ± 0 (100%)	986 ± 26 (97%) (gap:11%)
Table 2: Ablation study (more results are in Table 6 of Appendix H). Note that  $\overline{\mathcal{N}}$  (in CR and C) and  $\underline{\mathcal{N}}$  (in CRM, CM, and M) result in the same bilinear terms in three-player games because each element in  $\{-i\mid i\in \{1,2,3\}\}$  includes only two players such that Algorithm 1 cannot reduce the number of internal nodes to reduce the number of bilinear terms, and then CR and CRM (or C and CM) have the same performance. The unit of the runtime is second.
Runtime ± 95% Confidence Interval (Percentage of Games not Solved)
Game	CRM	CR	CM	C	M
(8, 2)	156± 83 (3%)	612± 129 (33%)	190 ± 102 (7%)	763 ± 120 (60%)	1000 ± 0 (100%)
(7, 2)	25 ± 17	89 ± 51	36 ± 28	408 ± 157 (30%)	488 ± 111 (10%)
(3, 15)	167± 86 (3%)	167 ± 86 (3%)	317 ± 137 (17%)	317 ± 137 (17%)	558 ± 150 (40%)
(3, 17)	231± 122 (10%)	231 ± 122 (10%)	326 ± 134 (20%)	326 ± 134 (20%)	784 ± 102 (53%)
Random graphical	0.1 ± 0.1	0.4 ± 0.1	0.2 ± 0.1	0.6 ± 0.4	814 ± 134 (80%)
Uniform LEG	2.2 ± 1	5 ± 4	2.5 ± 2	5 ± 5	999 ± 2 (97%)
integer variables with  $mn$  constraints and the following number of bilinear terms:
$$
\sum_ {N ^ {\prime} \in \underline {{\mathcal {N}}}} \prod_ {i \in N ^ {\prime}} | A _ {i} | \leq | \underline {{\mathcal {N}}} | m ^ {n - 1},
$$
i.e.,  $O((n\log n)m^{n - 1})$  bilinear terms (this size is the minimum because  $O(n\log n)$  is the minimum size of binary collections by Theorem 4) and. Then the time complexity for solving Program (13) based on  $\underline{\mathcal{N}}$  is  $O(I_2^2 (E_2C^2)^{2E_2 + 3})$  where  $I_{2} = (n\log n)m^{n - 1}I^{\prime} + mn$  and  $E_{2} = (n\log n)m^{n - 1}E^{\prime} + mn$ , and thus  $O(n\log n)$  of Algorithm 1 can be ignored. Therefore, CRM dramatically reduces the time complexity (i.e., the term  $2^{n}$  in  $I_{1}$  and  $E_{1}$  is changed to the term  $n\log n$  in  $I_{2}$  and  $E_{2}$ ).
Following prior work for NEs [34, 4, 13], we evaluate our approach on two sets of games: randomly generated games (i.e.,  $(n,m)$  with  $n$  players and  $m$  actions for each player) and six-player three-action games that are generated by GAMUT [31]. Payoffs are generated from the interval between 0 and 100 (other ranges (e.g.,  $[0,1]$ ) do not affect the result). Details are shown in Appendix F. We show the game size in terms of the number of bilinear terms and integer variables in Appendix G, e.g., the number of bilinear terms in the game  $(9,2)$  is 19152 based on  $\overline{\mathcal{N}}$  but is 2512 based on  $\underline{\mathcal{N}}$ .
Baselines: We compare our  $\mathbf{CRM}^4$  shown in Algorithm 2, i.e., solving Program (13) based on our  $\mathcal{N}$ , to the state-of-the-art algorithms: i) MIBP [34, 13]: the equivalent of solving Program (5) based on  $\overline{\mathcal{N}}$ ; ii) EXCLUSION [4]: the first implemented algorithm guarantees to converge to an NE by using a tree-search based method by splitting the continuous probability space of the solution; and iii) ENUMPOLY [28]: an algorithm in the well-known game-solving package Gambit which
<sup>4</sup>Codes are available at https://github.com/Youzhi333/optimalNE.
tries to find all NEs by enumerating all the supports which could be the support of an NE and then searching for an equilibrium on that support. They represent approaches to solving a nonlinear program, finding an NE, and enumerating all Nash equilibria, respectively. There are some other algorithms in Gambit [28] for finding an NE in a multiplayer game, including: i) GNM [16]: a global Newton method approach; ii) IPA [17]: an iterated polymatrix approximation approach; iii) LIAP: a function minimization approach; iv) SIMPDIV [41]: a simplicial subdivision approach; and v) LOGIT [29, 40]: a quantal response method. However, they cannot guarantee finding an NE [4]. Therefore, they are not suitable for finding an optimal NE. In fact, we show in Appendix I that all of them fail to solve many games and even run significantly slower than CRM in many games. Note that these Gambit algorithms only achieve some NE if the game is solved, which may not be optimal.
Algorithm Setting and Metric: We set a time limit of 1000 seconds for each case unless stated otherwise. Our optimality gap for EXCLUSION is significantly smaller than 0.001 in [4] (we verified that, with the same optimality gap, our result for EXCLUSION is almost the same as the one in [4]). We mainly use the runtime and the percentage of games that are not solved within the time limit to measure the performance of our approach. Details are shown in Appendix F (also caption of Table 1).
Result: Part of results are shown in Table 1, and more results are in Tables 4 and 5 of Appendix H. They show that the runtime of our CRM steadily increases with the game size. Note that the runtime of CRM includes the runtime for our Algorithm 1, which is extremely small (see Appendix E). Moreover, CRM is significantly faster than the baselines and is two or three orders of magnitude faster than the state-of-art baselines MIBP, ENUMPOLY, and EXCLUSION in most games. The reasons are that: 1) MIBP with too many bilinear terms and large feasible solution space after the relaxation cannot perform well without CRM's novel techniques in Section 3, where each of these techniques significantly boosts the performance (see the ablation study); and 2) the exponentially many NEs and the large search space caused by splitting the continuous probability space make ENUMPOLY and EXCLUSION, respectively, hard to scale up. EXCLUSION always has large utility gaps, which means that we will lose large utilities if we use EXCLUSION for our problem. The result that CRM runs significantly faster than EXCLUSION means that CRM is a faster algorithm not only for computing an optimal NE but also for just computing an NE. Furthermore, the gap between CRM and any of the baselines increases with the number of players or actions. In games with a large gap between CRM and baselines, the real gap should be larger because these baselines have not solved all of them within the time limit, while CRM solved all of them. Overall, CRM significantly overcomes the limitation of baselines.
Ablation Study: We evaluate each component of CRM by using the following variants: i) CR: solving Program (13) based on  $\overline{\mathcal{N}}$ ; ii) CM: solving Program (13) based on  $\underline{\mathcal{N}}$  without the relation constraints Eqs.(11) and (12); iii) C: solving Program (13) based on  $\overline{\mathcal{N}}$  without the relation constraints Eqs.(11) and (12); and iv) M: solving Program (5) based on  $\underline{\mathcal{N}}$ . Part of results are in Table 2, and more results are in Table 6 of Appendix H. We can see that each component of our approach significantly boosts its performance.
Existing works define a correlation plan as a probability distribution over the joint action space of all players, and use it to formulate constraints for a correlated equilibrium [37, 1]. However, the constraints for the space of correlated equilibria cannot be used in our program due to the following two reasons. First, there are no correlation plans for coordinating all players in our program after the convex relaxation because our formulation based on [34] has reduced the degree of the multilinear program for the space of NEs in order to significantly reduce the number of bilinear terms. Second, our correlation plans are different from the correlation plan for correlated equilibria because our correlation plans are only for subsets of the players. Recently, the correlation plan [49] based on a decomposition of the extensive-form game into public states has been used to compute correlated equilibria. However, their approach is not suitable for our problem because our game is not extensive-form and then does not have the property of their problem. Then our approach exploiting the relations of correlation plans and minimizing the number of correlation plans is novel.
Several recent efforts have developed relatively efficient algorithms to find an NE that maximizes the utility of a team of players in zero-sum games [45, 50, 51, 52, 11, 24, 53]. However, these algorithms cannot be used in games where team members have different utility functions. Existing
works transforming multilinear terms into bilinear terms only focus on special cases. For example, the transformation in [13] is equivalent to our transformation based on  $\overline{\mathcal{N}}$ , which is only a special case of our transformation framework. They [13] then directly solves the bilinear program based on this special transformation for finding an NE, which is equivalent to our baseline MIBP. Experiments show that our approach with novel techniques in Section 3 significantly outperforms [13]. Similar to the formulation in [34], there are other formulations [2, 3] for finding an optimal NE for two players under the problem of computing a leader-follower (Stackelberg) equilibrium for a single leader and two followers after a mixed strategy is committed by the leader. These formulations are different from ours because of the difference between the NE and the Stackelberg equilibrium. For example, the leader will commit a strategy to the followers in a Stackelberg equilibrium, i.e., the followers know the leader's strategy, but this cannot happen in an NE as all players move simultaneously. Moreover, after dropping the dependences of the followers to the leader's strategies in these bilinear programming formulations, the problem boils down to computing an optimal NE in two-player games because they only consider two followers in their formulations, which results in the same two-player formulation of [34].
For the existing general optimization techniques, e.g., Reformulation-Linearization Technique (RLT) [35, 25, 36], they add linear constraints by multiplying linear constraints with a single variable to reduce the feasible solution space of the convex relaxation and the number of bilinear terms if they can be represented by linear constraints (i.e., variants of original linear constraints). However, these operations are not very effective for our problem because the bilinear terms cannot be represented by those linear constraints (i.e., variants of original linear constraints), and simply multiplying linear constraints with a single variable cannot effectively represent the relation between auxiliary variables and nonlinear terms. Indeed, RLT is implemented in Gurobi [18, 19], but its performance (see MIBP in Table 1) is not good enough for large games in experiments. Moreover, our approach significantly outperforms the state-of-the-art optimization solver Gurobi (see results for CRM versus MIBP in Table 1) in experiments.
Similarly to the previous literature [34, 4, 13], to efficiently evaluate the algorithms, we set a time limit of 1000 seconds for each case unless stated otherwise. It means that we may need 30,000 seconds (almost 8 hours) to run an algorithm for each game setting (e.g., the game (6, 3)) with 30 cases. We totally run 13 algorithms for 21 different game settings, whose total runtime is more than 2,000 hours if each algorithm needs 30,000 seconds for each game setting. A higher time limit means more runtime. For example, if the time limit is 10,000s, we may need 20,000 hours (more than 800 days), which is not reasonable for a personal computer. Increasing the game size will cause a similar problem as well. Our goal is only to show that our proposed algorithm runs faster than baselines. Therefore, as a proof of concept, our time limit and game size are reasonable and practical.
Our algorithm CRM is significantly faster than the state-of-the-art baseline, and it can solve many real-world games: e.g., (1) multiplayer hand games using only the hands of the players (https://en.m.wikipedia.org/wiki/Hand_game), including the rock-paper-scissors games, Morra games, and their variants; and (2) the matching pennies game with several players and only two actions for each player. However, we cannot handle extremely large games now because we are handling a very hard problem, and then it is unrealistic to expect that our exact algorithm CRM could run very fast in large games. Our algorithm is an attempt to make this computation of optimal NEs feasible, and our algorithm framework can be built on by further innovative heuristics to improve the computation of optimal NEs. That is, for games with more players or actions, we can exploit the auxiliary speed-up techniques: the multiagent learning framework—Policy-Spaced Response Oracles (PSRO) [22, 54, 55, 51, 46, 23], the abstraction techniques [47], or only considering approximate NEs. Specifically, our algorithm CRM could be used as the meta-solver in PSRO.
This paper proposes a novel algorithm (CRM) for computing optimal NEs based on our transformation framework. CRM uses correlation plans with their relations to strictly reduce the feasible solution space after the convex relaxation while minimizing the number of correlation plans to significantly reduce the number of bilinear terms. Experiments show that CRM significantly outperforms baselines.
This research is supported by the InnoHK Fund, ONR grants N00014-18-1-2670, and N00014-20-1-2407. Bo An is supported by the National Research Foundation, Singapore under its Industry Alignment Fund – Pre-positioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.