SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Corollary 3.4 (Improved rates via optimism). There exist regret minimization algorithms that guarantee that the average strategy of the mediator  $\bar{\mu} \coloneqq \frac{1}{T}\sum_{t=1}^{T}\pmb{\mu}^{(t)}$  converges to the set of optimal equilibria at a rate of  $T^{-1/2}$ . The per-iteration complexity is analogous to Corollary 3.3.
While this rate is slower than the (near)  $T^{-1}$  rates known for converging to some of those equilibria [24, 34, 85, 3], Corollaries 3.3 and 3.4 additionally guarantee convergence to optimal equilibria; improving the  $T^{-1/2}$  rate of Corollary 3.4 is an interesting direction for future research.
Last-iterate convergence The convergence results we have stated thus far apply for the average strategy of the mediator—a typical feature of traditional guarantees in the no-regret framework. Nevertheless, an important advantage of our mediator-augmented formulation is that we can also guarantee last-iterate convergence to optimal equilibria in general games. Indeed, this follows readily from our reduction to two-player zero-sum games, leading to the following guarantee.
Theorem 3.5 (Last-iterate convergence to optimal equilibria in general games). There exist algorithms that guarantee that the last strategy of the mediator  $\pmb{\mu}^{(T)}$  converges to the set of optimal equilibria at a rate of  $T^{-1/4}$ . The per-iteration complexity is analogous to Corollaries 3.3 and 3.4.
As such, our mediator-augmented paradigm bypasses known hardness results in the traditional learning paradigm (Proposition D.2) since iterate convergence is no longer tied to Nash equilibria.
3.2 Thresholding and binary search
A significant weakness of the above Lagrangian is that the multiplier  $\lambda^{*}$  can be large. This means that, in practice, the zero-sum game that needs to be solved to compute an optimal equilibrium could have a large reward range. While this is not a problem for most tabular methods that can achieve high precision, more scalable methods based on reinforcement learning tend to be unable to solve games to the required precision. In this section, we will introduce another Lagrangian-based method for solving the program (G) that will not require solving games with large reward ranges.
Specifically, let  $\tau \in \mathbb{R}$  be a fixed threshold value, and consider the bilinear saddle-point problem
$$
\max  _ {\boldsymbol {\mu} \in \Xi} \min  _ {\substack {\boldsymbol {\lambda} \in \Delta^ {n + 1}, \\ \boldsymbol {x} _ {i} \in X _ {i}: i \in [ n ]}} \quad \boldsymbol {\lambda} _ {0} (\boldsymbol {c} ^ {\top} \boldsymbol {\mu} - \tau) - \sum_ {i = 1} ^ {n} \boldsymbol {\lambda} _ {i} \boldsymbol {\mu} ^ {\top} \mathbf {A} _ {i} \boldsymbol {x} _ {i}, \tag{L2}
$$
where  $\Delta^k \coloneqq \{\pmb{\lambda} \in \mathbb{R}_{\geq 0}^k : \mathbf{1}^\top \pmb{\lambda} = 1\}$  is the probability simplex on  $k$  items. This Lagrangian was also stated—but not analyzed—by Farina et al. [30], in the special case of correlated equilibrium concepts (NFCCE, EFCCE, EFCE). Compared to that paper, ours contains a more complete analysis, and is general to more notions of equilibrium.
Like (L1), this Lagrangian is also a zero-sum game, but unlike (L1), the reward range in this Lagrangian is bounded by an absolute constant:
Proposition 3.6. Let  $\Gamma$  be a (mediator-augmented) game in which the reward for all agents is bounded in  $[0,1]$ . For any fixed  $\tau \in [0,1]$ , the saddle-point problem (L2) can be expressed as a zero-sum extensive-form game whose reward is bounded in  $[-2,2]$ .
1. The deviator picks an index  $i \in \llbracket n\rrbracket \cup \{0\}$ .
2. If  $i \neq 0$ , nature picks whether Player  $i$  can deviate, uniformly at random.
3. The game  $\Gamma$  is played. All players, except  $i$  if  $i \neq 0$  and nature selected that  $i$  can deviate, are constrained to play according to  $d_i$ . The deviator plays on behalf of Player  $i$ .
4. Upon reaching terminal node  $z$ , there are three cases. If nature picked  $i = 0$ , the utility is  $u_0(z) - \tau$ . Otherwise, if nature picked that Player  $i \neq 0$  can deviate, the utility is  $-2u_i(z)$ . Finally, if nature picked that Player  $i \neq 0$  cannot deviate, the utility is  $2u_i(z)$ .
The mediator's expected utility in this game is exactly
$$
\lambda_ {0} u _ {0} (\boldsymbol {\mu}, \boldsymbol {d}) - \sum_ {i = 1} ^ {n} \lambda_ {i} [ u _ {i} (\boldsymbol {\mu}, \boldsymbol {x} _ {i}, \boldsymbol {d} _ {- i}) - u _ {i} (\boldsymbol {\mu}, \boldsymbol {d}) ]
$$
where  $\lambda \in \Delta^{n + 1}$  is the deviator's mixed strategy in the first step.
The above observations suggest a binary-search-like algorithm for computing optimal equilibria; the pseudocode is given as Algorithm 1. The algorithm solves  $O(\log(1/\varepsilon))$  zero-sum games, each to precision  $\varepsilon$ . Let  $v^*$  be the optimal value of (G). If  $\tau \leq v^*$ , the value of (L2) is 0, and we will therefore never branch low, in turn implying that  $u \geq v^*$  and  $\ell \geq v^* - \varepsilon$ . As a result, we have proven:
Theorem 3.7. Algorithm 1 returns an  $\varepsilon$ -approximate equilibrium  $\mu$  whose value to the mediator is at least  $v^{*} - 2\varepsilon$ . If the underlying game solver used to solve (L2) runs in time  $f(\Gamma, \varepsilon)$ , then Algorithm 1 runs in time  $O(f(\Gamma, \varepsilon) \log(1 / \varepsilon))$ .
ALGORITHM 1: Pseudocode for binary search-based algorithm
1 input: game  $\Gamma$  with mediator reward range [0, 1], target precision  $\varepsilon > 0$   
2  $\ell \gets 0$ ,  $u \gets 1$   
3 while  $u - \ell > \varepsilon$  do  
4  $\tau \gets (\ell + u) / 2$   
5 run an algorithm to solve game (L2) until either  
6 (1) it finds a  $\mu$  achieving value  $\geq -\varepsilon$  in (L2), or  
7 (2) it proves that the value of (L2) is  $< 0$   
8 if case (1) happened then  $\ell \gets \tau$   
9 else  $u \gets \tau$   
10 return the last  $\mu$  found
The differences between the two Lagrangian formulations can be summarized as follows:
1. Using (L1) requires only a single game solve, whereas using (L2) requires  $O(\log(1 / \varepsilon))$  game solves.
2. Using (L2) requires only an  $O(\varepsilon)$ -approximate game solver to guarantee value  $v^{*} - \varepsilon$ , whereas using (L1) would require an  $O(\varepsilon / \lambda^{*})$ -approximate game solver to guarantee the same, even assuming that the critical Lagrange multiplier  $\lambda^{*}$  in (L1) is known.
Which is preferred will therefore depend on the application. In practice, if the games are too large to be solved using tabular methods, one can use approximate game solvers based on deep reinforcement learning. In this setting, since reinforcement learning tends to be unable to achieve the high precision required to use (L1), using (L2) should generally be preferred. In Section 4, we back up these claims with concrete experiments.
In this section, we demonstrate the practical scalability and flexibility of our approach, both for computing optimal equilibria in extensive-form games, and for designing optimal mechanisms in large-scale sequential auction design problems.
4.1 Optimal equilibria in extensive-form games
We first extensively evaluate the empirical performance of our two-player zero-sum reduction (Section 3.1) for computing seven equilibrium solution concepts across 23 game instances; the results using the method of Section 3.2 are slightly inferior, and are included in Appendix H. The game instances we use are described in detail in Appendix F, and belong to following eight different classes of established parametric benchmark games, each identified with an alphabetical mnemonic: B - Battleship [30], D - Liar's dice [68], GL - Goofspiel [88], K - Kuhn poker [61], L - Leduc poker [93], RS - ridesharing game [101], S - Sheriff [30], TP - double dummy bridge game [101].
For each of the 23 games, we compare the runtime required by the linear programming method of Zhang and Sandholm [100] ('LP') and the runtime required by our learning dynamics in Section 3.1 ('Ours') for computing  $\varepsilon$ -optimal equilibrium points.
Table 1 shows experimental results for the case in which the threshold  $\varepsilon$  is set to be  $1\%$  of the payoff range of the game, and the objective function is set to be the maximum social welfare (sum of player utilities) for general-sum games, and the utility of Player 1 in zero-sum games. Each row corresponds to a game, whose identifier begins with the alphabetical mnemonic of the game class, and whose size in terms of number of nodes in the game trees is reported in the second column. The remaining columns compare, for each solution concept, the runtimes necessary to approximate the optimum equilibrium point according to that solution concept. Due to space constraints, only five out of the seven solution concepts (namely, NFCCE, EFCCE, EFCE, COMM, and CERT) are shown; data for the two remaining concepts (NFCCERT and CCERT) is given in Appendix G.
We remark that in Table 1, the column 'Ours' reports the minimum across the runtime across the different hyperparameters tried for the learning dynamics. Furthermore, for each run of the algorithms, the timeout was set at one hour. More details about the experimental setup are available in Appendix G, together with finer breakdowns of the runtimes.
We observe that our learning-based approach is faster—often by more than an order of magnitude—and more scalable than the linear program. Our additional experiments with different objective functions and values of  $\varepsilon$ , available in Appendix G, confirm the finding. This shows the promise of our computational approach, and reinforces the conclusion that learning dynamics are by far the most scalable technique available today to compute equilibrium points in large games.
4.2 Exact sequential auction design
Next, we use our approach to derive the optimal mechanism for a sequential auction design problem. In particular, we consider a two-round auction with two bidders, each starting with a budget of 1. The valuation for each item for each bidder is sampled uniformly at random from the set  $\{0,1/4,1/2,3/4,1\}$ . We consider a mediator-augmented game in which the principal chooses an outcome (allocation and payment for each player) given their reports (bids). We use  $\mathrm{CFR} + [95]$  as learning algorithm and a fixed Lagrange multiplier  $\lambda \coloneqq 25$  to compute the optimal communication
Table 1: Experimental comparison between our learning-based approach ('Ours', Section 3.1) and the linear-programming-based method ('LP') of Zhang and Sandholm [100]. Within each pair of cells corresponding to 'LP' vs 'Ours,' the faster algorithm is shaded blue while the hue of the slower algorithm depends on how much slower it is. If both algorithms timed out, they are both shaded gray.
Game	# Nodes	NFCCE	EFCCE	EFCE	COMM	CERT
LP	Ours	LP	Ours	LP	Ours	LP	Ours	LP	Ours
B2222	1573	0.00s	0.00s	0.00s	0.01s	0.00s	0.02s	2.00s	1.49s	0.00s	0.02s
B2322	23839	0.00s	0.01s	3.00s	0.69s	9.00s	1.60s	timeout	4m 41s	2.00s	1.24s
B2323	254239	6.00s	0.33s	1m 21s	14.23s	3m 40s	44.87s	timeout	timeout	37.00s	40.45s
B2324	1420639	38.00s	2.73s	timeout	3m 1s	timeout	10m 48s	timeout	timeout	timeout	6m 14s
D32	1017	0.00s	0.01s	0.00s	0.02s	12.00s	0.40s	0.00s	0.06s	0.00s	0.01s
D33	27622	2m 17s	12.93s	timeout	1m 46s	timeout	timeout	timeout	4m 37s	4.00s	3.14s
GL3	7735	0.00s	0.01s	1.00s	0.02s	0.00s	0.01s	timeout	7.72s	0.00s	0.02s
K35	1501	49.00s	0.76s	46.00s	0.67s	57.00s	0.55s	1.00s	0.03s	0.00s	0.01s
L3132	8917	26.00s	0.59s	8m 43s	5.13s	8m 18s	6.10s	8.00s	3.46s	1.00s	0.10s
L3133	12688	38.00s	0.94s	20m 26s	8.88s	21m 25s	6.84s	12.00s	3.40s	1.00s	0.22s
L3151	19981	timeout	15.12s	timeout	timeout	timeout	timeout	timeout	16.73s	2.00s	0.21s
L3223	15659	4.00s	0.44s	1m 10s	2.94s	2m 2s	5.52s	19.00s	18.19s	1.00s	0.61s
L3523	1299005	timeout	1m 7s	timeout	timeout	timeout	timeout	timeout	timeout	timeout	2m 58s
S2122	705	0.00s	0.00s	0.00s	0.01s	0.00s	0.02s	2.00s	0.35s	0.00s	0.02s
S2123	4269	0.00s	0.01s	1.00s	0.06s	1.00s	0.15s	1m 33s	59.63s	1.00s	0.15s
S2133	9648	1.00s	0.02s	3.00s	0.11s	3.00s	0.49s	timeout	12m 11s	2.00s	0.92s
S2254	712552	1m 58s	7.43s	timeout	22.01s	timeout	3m 34s	timeout	timeout	timeout	2m 42s
S2264	1303177	3m 43s	11.74s	timeout	39.23s	timeout	timeout	timeout	timeout	timeout	timeout
TP3	910737	1m 38s	7.44s	timeout	13.76s	timeout	13.46s	timeout	timeout	timeout	26.70s
RS212	598	0.00s	0.00s	0.00s	0.00s	0.00s	0.00s	2.00s	0.01s	0.00s	0.00s
RS222	734	0.00s	0.00s	0.00s	0.00s	0.00s	0.00s	3.00s	0.01s	0.00s	0.00s
RS213	6274	timeout	14.68s	timeout	15.54s	timeout	23.37s	6m 25s	8.74s	0.00s	0.02s
RS223	6238	timeout	timeout	timeout	timeout	timeout	timeout	8m 54s	4.00s	1.00s	0.01s
equilibrium that corresponds to the optimal mechanism. We terminated the learning procedure after 10000 iterations, at a duality gap for (L1) of approximately  $4.2 \times 10^{-4}$ . Figure 1 (left) summarizes our results. On the y-axis we show how exploitable (that is, how incentive-incompatible) each of the considered mechanisms are, confirming that for this type of sequential settings, second-price auctions (SP) with or without reserve price, as well as the first-price auction (FP), are typically incentive-incompatible. On the x-axis, we report the hypothetical revenue that the mechanism would extract assuming truthful bidding. Our mechanism is provably incentive-compatible and extracts a larger revenue than all considered second-price mechanisms. It also would extract less revenue than the hypothetical first-price auction if the bidders behaved truthfully (of course, real bidders would not behave honestly in the first-price auction but rather would shade their bids downward, so the shown revenue benchmark in Figure 1 is actually not achievable). Intriguingly, we observed that  $8\%$  of the time the mechanism gives an item away for free. Despite appearing irrational, this behavior can incentivize bidders to use their budget earlier in order to encourage competitive bidding, and has been independently discovered in manual mechanism design recently [25, 75].
4.3 Scalable sequential auction design via deep reinforcement learning
We also combine our framework with deep-learning-based algorithms for scalable equilibrium computation in two-player zero-sum games to compute optimal mechanisms in two sequential auction settings. To compute an optimal mechanism using our framework, we use the PSRO algorithm [63], a deep reinforcement learning method based on the double oracle algorithm that has empirically scaled to large games such as Starcraft [96] and Stratego [72], as the game solver in Algorithm 1. To train the best responses, we use proximal policy optimization (PPO) [90].
<sup>7</sup>We also tested PSRO on the Lagrangian (L1), but this proved to be incompatible with deep learning due to the large reward range induced by the multiplier  $\lambda$ .
Figure 1: Exploitability is measured by summing the best response for both bidders to the mechanism. Zero exploitability corresponds to incentive compatibility. In a sequential auction with budgets, our method is able to achieve higher revenue than second-price auctions and better incentive compatibility than a first-price auction.
First, to verify that the deep learning method is effective, we replicate the results of the tabular experiments in Section 4.2. We find that PSRO achieves the same best response values and optimal equilibrium value computed by the tabular experiment, up to a small error. These results give us confidence that our method is correct.
Second, to demonstrate scalability, we run our deep learning-based algorithm on a larger auction environment that would be too big to solve with tabular methods. In this environment, there are four rounds, and in each round the valuation of each player is sampled uniformly from  $\{0, 0.1, 0.2, 0.3, 0.4, 0.5\}$ . The starting budget of each player is, again, 1. We find that, like the smaller setting, the optimal revenue of the mediator is  $\approx 1.1$  (right-side of Figure 1). This revenue exceeds the revenue of every second-price auction (none of which have revenue greater than 1).<sup>8</sup>
We proposed a new paradigm of learning in games. It applies to mechanism design, information design, and solution concepts in multi-player extensive-form games such as correlated, communication, and certification equilibria. Leveraging a Lagrangian relaxation, our paradigm reduces the problem of computing optimal equilibria to determining minimax equilibria in zero-sum extensive-form games. We also demonstrated the scalability of our approach for computing optimal equilibria by attaining state-of-the-art performance in benchmark tabular games, and by solving a sequential auction design problem using deep reinforcement learning.
<sup>8</sup>We are inherently limited in this setting by the inexactness of best responses based on deep reinforcement learning; as such, it is possible that these values are not exact. However, because of the success of above tabular experiment replications, we believe that our results should be reasonably accurate.
We are grateful to the anonymous NeurIPS reviewers for many helpful comments that helped improve the presentation of this paper. Tuomas Sandholm's work is supported by the Vannevar Bush Faculty Fellowship ONR N00014-23-1-2876, National Science Foundation grants RI-2312342 and RI-1901403, ARO award W911NF2210266, and NIH award A240108S001. McAleer is funded by NSF grant #2127309 to the Computing Research Association for the CIFellows 2021 Project. The work of Prof. Gatti's research group is funded by the FAIR (Future Artificial Intelligence Research) project, funded by the NextGenerationEU program within the PNRR-PE-AI scheme (M4C2, Investment 1.3, Line on Artificial Intelligence). Conitzer thanks the Cooperative AI Foundation and Polaris Ventures (formerly the Center for Emerging Risk Research) for funding the Foundations of Cooperative AI Lab (FOCAL). Andy Haupt was supported by Effective Giving. We thank Dylan Hadfield-Menell for helpful conversations.
[1] Kenshi Abe, Mitsuki Sakamoto, and Atsushi Iwasaki. Mutation-driven follow the regularized leader for last-iterate convergence in zero-sum games. In Conference on Uncertainty in Artificial Intelligence (UAI), volume 180, pages 1–10. PMLR, 2022.
[2] Ilan Adler. The equivalence of linear programs and zero-sum games. Int. J. Game Theory, 42 (1):165-177, 2013.
[3] Ioannis Anagnostides, Constantinos Daskalakis, Gabriele Farina, Maxwell Fishelson, Noah Golowich, and Tuomas Sandholm. Near-optimal no-regret learning for correlated equilibria in multi-player general-sum games. In Symposium on Theory of Computing (STOC), pages 736-749. ACM, 2021.
[4] Ioannis Anagnostides, Ioannis Panageas, Gabriele Farina, and Tuomas Sandholm. On last-iterate convergence beyond zero-sum games. In International Conference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pages 536-581. PMLR, 2022.
[5] Robert Aumann. Subjectivity and correlation in randomized strategies. Journal of Mathematical Economics, 1:67-96, 1974.
[6] Waiss Azizian, Franck Iutzeler, Jérôme Malick, and Panayotis Mertikopoulos. The last-iterate convergence rate of optimistic mirror descent in stochastic variational inequalities. In Conference on Learning Theory (COLT), volume 134 of Proceedings of Machine Learning Research, pages 326–358. PMLR, 2021.
[7] Yakov Babichenko and Aviad Rubinstein. Communication complexity of approximate nash equilibria. Games and Economic Behavior, 134:376-398, 2022.
[8] Yu Bai, Chi Jin, Huan Wang, and Caiming Xiong. Sample-efficient learning of stackelberg equilibria in general-sum games. In Conference on Neural Information Processing Systems (NeurIPS), pages 25799-25811, 2021.
[9] Siddharth Barman and Katrina Ligett. Finding any nontrivial coarse correlated equilibrium is hard. In ACM Conference on Economics and Computation (EC), pages 815-816. ACM, 2015.
[10] Michael Bowling, Neil Burch, Michael Johanson, and Oskari Tammelin. Heads-up limit hold'em poker is solved. Science, 347(6218), January 2015.
[11] Noam Brown and Tuomas Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418-424, 2018.
[12] Noam Brown and Tuomas Sandholm. Solving imperfect-information games via discounted regret minimization. In AAAI Conference on Artificial Intelligence (AAAI), 2019.
[83] Christos Papadimitriou and Tim Roughgarden. Computing equilibria in multi-player games. In Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 82-91. SIAM, 2005.
[84] Binghui Peng, Weiran Shen, Pingzhong Tang, and Song Zuo. Learning optimal strategies to commit to. In AAAI Conference on Artificial Intelligence (AAAI), pages 2149-2156. AAAI Press, 2019.
[85] Georgios Piliouras, Ryann Sim, and Stratis Skoulakis. Beyond time-average convergence: Near-optimal uncoupled online learning via clairvoyant multiplicative weights update. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
[86] Jad Rahme, Samy Jelassi, and S. Matthew Weinberg. Auction learning as a two-player game. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.
[87] Sasha Rakhlin and Karthik Sridharan. Optimization, learning, and games with predictable sequences. In Conference on Neural Information Processing Systems (NeurIPS), pages 3066-3074, 2013.
[88] Sheldon M Ross. Goofspiel—the game of pure strategy. Journal of Applied Probability, 8 (3):621-625, 1971.
[89] Tim Roughgarden and Omri Weinstein. On the communication complexity of approximate fixed points. In Annual Symposium on Foundations of Computer Science (FOCS), pages 229-238, 2016.
[90] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[91] Shai Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends® in Machine Learning, 4(2), 2012.
[92] Maurice Sion. On general minimax theorems. Pacific J. Math., 8(4):171-176, 1958.
[93] Finnegan Southey, Michael Bowling, Bryce Larson, Carmelo Piccione, Neil Burch, Darse Billings, and Chris Rayner. Bayes' bluff: Opponent modelling in poker. In Conference on Uncertainty in Artificial Intelligence (UAI), July 2005.
[94] Vasilis Syrgkanis, Alekh Agarwal, Haipeng Luo, and Robert E Schapire. Fast convergence of regularized learning in games. In Conference on Neural Information Processing Systems (NeurIPS), pages 2989-2997, 2015.
[95] Oskari Tammelin, Neil Burch, Michael Johanson, and Michael Bowling. Solving heads-up limit Texas hold'em. In International Joint Conference on Artificial Intelligence (IJCAI), 2015.
[96] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michael Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782): 350-354, 2019.
[97] John von Neumann. Zur Theorie der Gesellschaftsspiele. Mathematische Annalen, 100:295-320, 1928.
[98] Bernhard von Stengel and Françoise Forges. Extensive-form correlated equilibrium: Definition and computational complexity. Mathematics of Operations Research, 33(4):1002-1022, 2008.
[99] Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.