SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Brian Hu Zhang*
bhzhang@cs.cmu.edu
Gabriele Farina*
gfarina@mit.edu
Ioannis Anagnostides
ianagnos@cs.cmu.edu
Federico Cacciamani
DEIB, Politecnico di Milano
federico.cacciamani@polimi.it
Stephen McAleer
smcaleer@cs.cmu.edu
Andreas Haupt
haupt@mit.edu
Andrea Celli
andrea.celli2@unibocconi.it
Vincent Conitzer
conitzer@cs.cmu.edu
Nicola Gatti
nicola.gatti@polimi.it
We introduce a new approach for computing optimal equilibria and mechanisms via learning in games. It applies to extensive-form settings with any number of players, including mechanism design, information design, and solution concepts such as correlated, communication, and certification equilibria. We observe that optimal equilibria are minimax equilibrium strategies of a player in an extensive-form zero-sum game. This reformulation allows us to apply techniques for learning in zero-sum games, yielding the first learning dynamics that converge to optimal equilibria, not only in empirical averages, but also in iterates. We demonstrate the practical scalability and flexibility of our approach by attaining state-of-the-art performance in benchmark tabular games, and by computing an optimal mechanism for a sequential auction design problem using deep reinforcement learning.
What does it mean to solve a game? This is one of the central questions addressed in game theory, leading to a variety of different solution concepts. Perhaps first and foremost, there are various notions of equilibrium, strategically stable points from which no rational individual would be inclined to deviate. But is it enough to compute, or indeed learn, just any one equilibrium of a game? In two-player zero-sum games, one can make a convincing argument that a single equilibrium in fact constitutes a complete solution to the game, based on the celebrated minimax theorem of von Neumann [97]. Indeed, approaches based on computing minimax equilibria in
two-player zero-sum games have enjoyed a remarkable success in solving major AI challenges, exemplified by the recent development of superhuman poker AI agents [10, 11].
However, in general-sum games it becomes harder to argue that any equilibrium constitutes a complete solution. Indeed, one equilibrium can offer vastly different payoffs to the players than another. Further, if a player acts according to one equilibrium and another player according to a different one, the result may not be an equilibrium at all, resulting in a true equilibrium selection problem. In this paper, therefore, we focus on computing an optimal equilibrium, that is, one that maximizes a given linear objective within the space of equilibria. There are various advantages to this approach. First, in many contexts, we would simply prefer to have an equilibrium that maximizes, say, the sum of the players' utilities—and by computing such an equilibrium we also automatically avoid Pareto-dominated equilibria. Second, it can mitigate the equilibrium selection problem: if there is a convention that we always pursue an equilibrium that maximizes social welfare, this reduces the risk that players end up playing according to different equilibria. Third, if one has little control over how the game will be played but cares about its outcomes, one may like to understand the space of all equilibria. In general, a complete picture of this space can be elusive, in part because a game can have exponentially many equilibria; but computing extreme equilibria in many directions—say, one that maximizes Player 1's utility—can provide meaningful information about the space of equilibria.
That being said, many techniques that have been successful at computing a single equilibrium do not lend themselves well to computing optimal equilibria. Most notably, while no-regret learning dynamics are known to converge to different notions of correlated equilibria [49, 38, 39, 47], little is known about the properties of the equilibrium reached. In this paper, therefore, we introduce a new paradigm of learning in games for computing optimal equilibria. It applies to extensive-form settings with any number of players, including information design, and solution concepts such as correlated, communication, and certification equilibria. Further, our framework is general enough to also capture optimal mechanism design and optimal incentive design problems in sequential settings.
Summary of Our Results A key insight that underpins our results is that computing optimal equilibria in multi-player extensive-form games can be cast via a Lagrangian relaxation as a two-player zero-sum extensive-form game. This unlocks a rich technology, both theoretical and experimental, developed for computing minimax equilibria for the more challenging—and much less understood—problem of computing optimal equilibria. In particular, building on the framework of Zhang and Sandholm [100], our reduction lends itself to mechanism design and information design, as well as an entire hierarchy of equilibrium concepts, including normal-form coarse correlated equilibria (NFCCE) [79], extensive-form coarse correlated equilibria (EFCCE) [31], extensive-form correlated equilibria (EFCE) [98], communication equilibria (COMM) [36], and certification equilibria (CERT) [37]. In fact, for communication and certification equilibria, our framework leads to the first learning-based algorithms for computing them, addressing a question left open by Zhang and Sandholm [100] (cf. [40], discussed in Appendix B).
We thus focus on computing an optimal equilibrium by employing regret minimization techniques in order to solve the induced bilinear saddle-point problem. Such considerations are motivated in part by the remarkable success of no-regret algorithms for computing minimax equilibria in large two-player zero-sum games (e.g., see [10, 11]), which we endeavor to transfer to the problem of computing optimal equilibria in multi-player games.
In this context, we show that employing standard regret minimizers, such as online mirror descent [91] or counterfactual regret minimization [106], leads to a rate of convergence of  $T^{-1/4}$  to optimal equilibria by appropriately tuning the magnitude of the Lagrange multipliers (Corollary 3.3). We also leverage the technique of optimism, pioneered by Chiang et al. [18], Rakhlin and Sridharan [87] and Syrgkanis et al. [94], to obtain an accelerated  $T^{-1/2}$  rate of convergence (Corollary 3.4). These are the first learning dynamics that (provably) converge to optimal equilibria. Our bilinear formulation also allows us to obtain last-iterate convergence to optimal equilibria via optimistic gradient descent/ascent (Theorem 3.5), instead of the time-average guarantees traditionally derived within the no-regret framework. As such, we bypass known barriers in the traditional learning paradigm by incorporating an additional player, a mediator, into the learning process. Furthermore, we also study an alternative Lagrangian relaxation which, unlike our earlier approach, consists of solving a sequence of zero-sum games (cf. [30]). While the latter approach is less natural, we find that it is preferable when used in conjunction with deep RL solvers since it obviates the need for solving games with large reward ranges—a byproduct of employing the natural Lagrangian relaxation.
Experimental results We demonstrate the practical scalability of our approach for computing optimal equilibria and mechanisms. First, we obtain state-of-the-art performance in a suite of 23 different benchmark game instances for seven different equilibrium concepts. Our algorithm significantly outperforms existing LP-based methods, typically by more than one order of magnitude. We also use our algorithm to derive an optimal mechanism for a sequential auction design problem, and we demonstrate that our approach is naturally amenable to modern deep RL techniques.
In this subsection, we highlight prior research that closely relates to our work. Additional related work is included in Appendix B.
A key reference point is the recent paper of Zhang and Sandholm [100], which presented a unifying framework that enables the computation via linear programming of various mediator-based equilibrium concepts in extensive-form games, including NFCCE, EFCCE, EFCE, COMM, and CERT. Perhaps surprisingly, Zhang et al. [101] demonstrated that computing optimal communication and certification equilibria is possible in time polynomial in the description of the game, establishing a stark dichotomy between the other equilibrium concepts—namely, NFCCE, EFCE, and EFCCE—for which the corresponding problem is NP-hard [98]. In particular, for the latter notions intractability turns out to be driven by the imperfect recall of the mediator [101]. Although imperfect recall induces a computationally hard problem in general from the side of the mediator [19, 59], positive parameterized results have been documented recently in the literature [103].
Our work significantly departs from the framework of Zhang and Sandholm [100] in that we follow a learning-based approach, which has proven to be a particularly favorable avenue in practice; e.g., we refer to [26, 16, 78, 77, 104] for such approaches in the context of computing EFCE. Further, beyond the tabular setting, learning-based frameworks are amenable to modern deep reinforcement learning methods (see [70, 71, 62, 69, 51, 76, 57, 13, 52, 73, 72, 86, 105, 42], and references therein). Most of those techniques have been developed to solve two-player zero-sum games, which provides another crucial motivation for our main reduction. We demonstrate this experimentally in large games in Section 4. For multi-player games, Marris et al. [70] developed a scalable algorithm based on policy space response oracles (PSRO) [62] (a deep-reinforcement-learning-based double-oracle technique) that converges to NFC(C)E, but it does not find an optimal equilibrium.
Our research also relates to computational approaches to static auction and mechanism design through deep learning [27, 86]. In particular, similarly to the present paper, Dütting et al. [27] study a Lagrangian relaxation of mechanism design problems. Our approach is significantly more general in that we cover both static and sequential auctions, as well as general extensive-form games. Further, as a follow-up, Rahme et al. [86] frame the Lagrangian relaxation as a two-player game, which, however, is not zero-sum, thereby not enabling leveraging the tools known for solving zero-sum games. Finally, in a companion paper [102], we show how the framework developed in this work can be used to steer no-regret learners to optimal equilibria via nonnegative vanishing payments.
We adopt the general framework of mediator-augmented games of Zhang and Sandholm [100] to define our class of instances. At a high level, a mediator-augmented game explicitly incorporates an additional player, the mediator, who can exchange messages with the players and issue action recommendations; different assumptions on the power of the mediator and the players' strategy sets induce different equilibrium concepts, as we clarify for completeness in Appendix A.
Definition 2.1. A mediator-augmented, extensive-form game  $\Gamma$  has the following components:
1. a set of players, identified with the set of integers  $\llbracket n\rrbracket \coloneqq \{1,\dots ,n\}$ . We will use  $-i$ , for  $i\in \llbracket n\rrbracket$ , to denote all players except  $i$ ;
2. a directed tree  $H$  of histories or nodes, whose root is denoted  $\varnothing$ . The edges of  $H$  are labeled with actions. The set of actions legal at  $h$  is denoted  $A_{h}$ . Leaf nodes of  $H$  are called terminal, and the set of such leaves is denoted by  $Z$ ;
${}^{2}$  Notably missing from this list is the normal-form correlated equilibrium (NFCE),the complexity status of which (in extensive-form games) is a long-standing open problem.
3. a partition  $H \setminus Z = H_{\mathbf{C}} \sqcup H_0 \sqcup H_1 \sqcup \dots \sqcup H_n$ , where  $H_i$  is the set of nodes at which  $i$  takes an action, and  $\mathbf{C}$  and 0 denote chance and the mediator, respectively;
4. for each agent  $i \in [[n]] \cup \{0\}$ , a partition  $\mathcal{I}_i$  of  $i$ 's decision nodes  $H_i$  into information sets. Every node in a given information set  $I$  must have the same set of legal actions, denoted by  $A_I$ ;
5. for each agent  $i$ , a utility function  $u_{i}: Z \to \mathbb{R}$ ; and
6. for each chance node  $h \in H_{\mathbf{C}}$ , a fixed probability distribution  $c(\cdot |h)$  over  $A_h$ .
To further clarify this definition, in Appendix A we provide two concrete illustrative examples: a single-item auction and a welfare-optimal correlated equilibrium in normal-form games.
At a node  $h \in H$ , the sequence  $\sigma_i(h)$  of an agent  $i$  is the set of all information sets encountered by agent  $i$ , and the actions played at such information sets, along the  $\varnothing \to h$  path, excluding at  $h$  itself. An agent has perfect recall if  $\sigma_i(h) = \sigma_i(h')$  for all  $h, h'$  in the same infoset. We will use  $\Sigma_i := \{\sigma_i(z) : z \in Z\}$  to denote the set of all sequences of player  $i$  that correspond to terminal nodes. We will assume that all players have perfect recall, though the mediator may not.<sup>4</sup>
A pure strategy of agent  $i$  is a choice of one action in  $A_{I}$  for each information set  $I \in \mathcal{I}_i$ . The sequence form of a pure strategy is the vector  $\pmb{x}_i \in \{0,1\}^{\Sigma_i}$  given by  $\pmb{x}_i[\sigma] = 1$  if and only if  $i$  plays every action on the path from the root to sequence  $\sigma \in \Sigma_i$ . We will use the shorthand  $\pmb{x}_i[z] = \pmb{x}_i[\sigma_i(z)]$ . A mixed strategy is a distribution over pure strategies, and the sequence form of a mixed strategy is the corresponding convex combination  $\pmb{x}_i \in [0,1]^{\Sigma_i}$ . We will use  $X_i$  to denote the polytope of sequence-form mixed strategies of player  $i$ , and use  $\Xi$  to denote the polytope of sequence-form mixed strategies of the mediator.
For a fixed  $\pmb{\mu} \in \Xi$ , we will say that  $(\pmb{\mu}, \pmb{x})$  is an equilibrium of  $\Gamma$  if, for each player  $i$ ,  $\pmb{x}_i$  is a best response to  $(\pmb{\mu}, \pmb{x}_{-i})$ , that is,  $\max_{\pmb{x}_i' \in X_i} u_i(\pmb{\mu}, \pmb{x}_i', \pmb{x}_{-i}) \leq u_i(\pmb{\mu}, \pmb{x}_i, \pmb{x}_{-i})$ . We do not require that the mediator's strategy  $\pmb{\mu}$  is a best response. As such, the mediator has the power to commit to its strategy. The goal in this paper will generally be to reach an optimal (Stackelberg) equilibrium, that is, an equilibrium  $(\pmb{\mu}, \pmb{x})$  maximizing the mediator utility  $u_0(\pmb{\mu}, \pmb{x})$ . We will use  $u_0^*$  to denote the value for the mediator in an optimal equilibrium.
**Revelation principle** The revelation principle allows us, without loss of generality, to restrict our attention to equilibria where each player is playing some fixed pure strategy  $d_{i} \in X_{i}$ .
Definition 2.2. The game  $\Gamma$  satisfies the revelation principle if there exists a direct pure strategy profile  $d = (d_{1},\ldots ,d_{n})$  for the players such that, for all strategy profiles  $(\mu ,x)$  for all players including the mediator, there exists a mediator strategy  $\pmb{\mu}^{\prime}\in \Xi$  and functions  $f_{i}:X_{i}\to X_{i}$  for each player  $i$  such that:
1.  $f_{i}(\pmb{d}_{i}) = \pmb{x}_{i}$ , and
2.  $u_{j}(\pmb{\mu}', \pmb{x}_{i}', \pmb{d}_{-i}) = u_{j}(\pmb{\mu}, f_{i}(\pmb{x}_{i}'), \pmb{x}_{-i})$  for all  $\pmb{x}_{i}' \in X_{i}$ , and agents  $j \in [[n]] \cup \{0\}$ .
The function  $f_{i}$  in the definition of the revelation principle can be seen as a simulator for Player  $i$ : it tells Player  $i$  that playing  $x_{i}'$  if other players play  $(\pmb{\mu}, \pmb{d}_{-i})$  would be equivalent, in terms of all the payoffs to all agents (including the mediator), to playing  $f(x_{i}')$  if other agents play  $(\pmb{\mu}, \pmb{x}_{-i})$ . It follows immediately from the definition that if  $(\pmb{\mu}, \pmb{x})$  is an  $\varepsilon$ -equilibrium, then so is  $(\pmb{\mu}', \pmb{d})$ —that is, every equilibrium is payoff-equivalent to a direct equilibrium.
The revelation principle applies and covers many cases of interest in economics and game theory. For example, in (single-stage or dynamic) mechanism design, the direct strategy  $d_{i}$  of each player is to report all information truthfully, and the revelation principle guarantees that for all non-truthful mechanisms  $(\mu, x)$  there exists a truthful mechanism  $(\mu', d)$  with the same utilities for all players. For correlated equilibrium, the direct strategy  $d_{i}$  consists of obeying all (potentially randomized) recommendations that the mediator gives, and the revelation principle states that we can, without loss of generality, consider only correlated equilibria where the signals given to the players are what actions they should play. In both these cases (and indeed in general for the notions we consider in this
3We will use agent to mean either a player or the mediator.
4Following the framework of Zhang and Sandholm [100], allowing the mediator to have imperfect recall will allow us to automatically capture optimal correlation.
In a mechanism design context, a strategy for the mediator  $\mu$  induces a mechanism; here we slightly abuse terminology by referring to  $(\mu, d)$  also as a mechanism.
paper), it is therefore trivial to specify the direct strategies  $\pmb{d}$  without any computational overhead. Indeed, we will assume throughout the paper that the direct strategies  $\pmb{d}$  are given. Further examples and discussion of this definition can be found in Appendix A.
Although the revelation principle is a very useful characterization of optimal equilibria, as long as we are given  $d$ , all of the results in this paper actually apply regardless of whether the revelation principle is satisfied: when it fails, our algorithms will simply yield an optimal direct equilibrium which may not be an optimal equilibrium. Under the revelation principle, the problem of computing an optimal equilibrium can be expressed as follows:
$$
\max  _ {\boldsymbol {\mu} \in \Xi} u _ {0} (\boldsymbol {\mu}, \boldsymbol {d}) \quad \text {s . t .} \quad \max  _ {\boldsymbol {x} _ {i} \in X _ {i}} u _ {i} (\boldsymbol {\mu}, \boldsymbol {x} _ {i}, \boldsymbol {d} _ {- i}) \leq u _ {i} (\boldsymbol {\mu}, \boldsymbol {d}) \forall i \in [   [ n ]   ].
$$
The objective  $u_0(\pmb{\mu}, \pmb{d})$  can be expressed as a linear expression  $c^\top \pmb{\mu}$ , and  $u_i(\pmb{\mu}, \pmb{x}_i, \pmb{d}_{-i}) - u_i(\pmb{\mu}, \pmb{d})$  can be expressed as a bilinear expression  $\pmb{\mu}^\top \mathbf{A}_i \pmb{x}_i$ . Thus, the above program can be rewritten as
$$
\max  _ {\boldsymbol {\mu} \in \Xi} \quad \boldsymbol {c} ^ {\top} \boldsymbol {\mu} \quad \text {s . t .} \quad \max  _ {\boldsymbol {x} _ {i} \in X _ {i}} \boldsymbol {\mu} ^ {\top} \mathbf {A} _ {i} \boldsymbol {x} _ {i} \leq 0 \forall i \in [   [ n ]   ]. \tag {G}
$$
Zhang and Sandholm [100] now proceed by taking the dual linear program of the inner maximization, which suffices to show that (G) can be solved using linear programming.
Finally, although our main focus in this paper is on games with discrete action sets, it is worth pointing out that some of our results readily apply to continuous games as well using, for example, the discretization approach of Kroer and Sandholm [60].
3 Lagrangian relaxations and a reduction to a zero-sum game
Our approach in this paper relies on Lagrangian relaxations of the linear program (G). In particular, in this section we introduce two different Lagrangian relaxations. The first one (Section 3.1) reduces computing an optimal equilibrium to solving a single zero-sum game. We find that this approach performs exceptionally well in benchmark extensive-form games in the tabular regime, but it may struggle when used in conjunction with deep RL solvers since it increases significantly the range of the rewards. This shortcoming is addressed by our second method, introduced in Section 3.2, which instead solves a sequence of suitable zero-sum games.
3.1 "Direct" Lagrangian
Directly taking a Lagrangian relaxation of the LP (G) gives the following saddle-point problem:
$$
\max  _ {\boldsymbol {\mu} \in \Xi} \min  _ {\substack {\lambda \in \mathbb {R} _ {\geq 0}, \\ \boldsymbol {x} _ {i} \in X _ {i}: i \in [ n ]}} \quad \boldsymbol {c} ^ {\top} \boldsymbol {\mu} - \lambda \sum_ {i = 1} ^ {n} \boldsymbol {\mu} ^ {\top} \mathbf {A} _ {i} \boldsymbol {x} _ {i}. \tag{L1}
$$
We first point out that the above saddle-point optimization problem admits a solution  $(\pmb{\mu}^{*},\pmb{x}^{*},\lambda^{*})$
Proposition 3.1. The problem (L1) admits a finite saddle-point solution  $(\pmb{\mu}^{*},\pmb{x}^{*},\lambda^{*})$ . Moreover, for all fixed  $\lambda >\lambda^{*}$ , the problems (L1) and (G) have the same value and same set of optimal solutions.
The proof is in Appendix C. We will call the smallest possible  $\lambda^{*}$  the critical Lagrange multiplier.
Proposition 3.2. For any fixed value  $\lambda$ , the saddle-point problem (L1) can be expressed as a zero-sum extensive-form game.
Proof. Consider the zero-sum extensive-form game  $\hat{\Gamma}$  between two players, the mediator and the deviator, with the following structure:
1. Nature picks, with uniform probability, whether or not there is a deviator. If nature picks that there should be a deviator, then nature samples, also uniformly, a deviator  $i \in [[n]]$ . Nature's actions are revealed to the deviator, but kept private from the mediator.
Computing optimal equilibria can be phrased as a linear program, and so in principle Adler's reduction could also lead to an equivalent zero-sum game [2]. However, that reduction does not yield an extensive-form zero-sum game, which is crucial for our purposes; see Section 3.
2. The game  $\Gamma$  is played. All players, except  $i$  if nature picked a deviator, are constrained to according to  $d_{i}$ . The deviator plays on behalf of Player  $i$ .
3. Upon reaching terminal node  $z$ , there are two cases. If nature picked a deviator  $i$ , the utility is  $-2\lambda n \cdot u_i(z)$ . If nature did not pick a deviator, the utility is  $2u_0(z) + 2\lambda \sum_{i=1}^{n} u_i(z)$ .
The mediator's expected utility in this game is
$$
u _ {0} (\boldsymbol {\mu}, \boldsymbol {d}) - \lambda \sum_ {i = 1} ^ {n} \left[ u _ {i} (\boldsymbol {\mu}, \boldsymbol {x} _ {i}, \boldsymbol {d} _ {- i}) - u _ {i} (\boldsymbol {\mu}, \boldsymbol {d}) \right].
$$
This characterization enables us to exploit technology used for extensive-form zero-sum game solving to compute optimal equilibria for an entire hierarchy of equilibrium concepts (Appendix A).
We will next focus on the computational aspects of solving the induced saddle-point problem (L1) using regret minimization techniques. All of the omitted proofs are deferred to Appendices D and E.
The first challenge that arises in the solution of (L1) is that the domain of the minimizing player is unbounded—the Lagrange multiplier is allowed to take any nonnegative value. Nevertheless, we show in Theorem D.1 that it suffices to set the Lagrange multiplier to a fixed value (that may depend on the time horizon); appropriately setting that value will allow us to trade off between the equilibrium gap and the optimality gap. We combine this theorem with standard regret minimizers (such as variants of CFR employed in Section 4.1) to guarantee fast convergence to optimal equilibria.
Corollary 3.3. There exist regret minimization algorithms such that when employed in the saddlepoint problem (L1), the average strategy of the mediator  $\bar{\pmb{\mu}}\coloneqq \frac{1}{T}\sum_{t = 1}^{T}\pmb{\mu}^{(t)}$  converges to the set of optimal equilibria at a rate of  $T^{-1 / 4}$ . Moreover, the per-iteration complexity is polynomial for communication and certification equilibria (under the nested range condition [100]), while for NFCCE, EFCCE and EFCE, implementing each iteration admits a fixed-parameter tractable algorithm.
Furthermore, we leverage the technique of optimism, pioneered by Chiang et al. [18], Rakhlin and Sridharan [87], Syrgkanis et al. [94], to obtain a faster rate of convergence.