Title: Computing Optimal Equilibria and Mechanisms via Learning in Zero-Sum Extensive-Form Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: extensive-form games

deep reinforcement learning

mechanism design

correlated equilibria
Abstract: We introduce a new approach for computing optimal equilibria via learning in games. It applies to extensive-form settings with any number of players, including mechanism design, information design, and solution concepts such as correlated, communication, and certification equilibria. We observe that optimal equilibria are minimax equilibrium strategies of a player in an extensive-form zero-sum game. This reformulation allows to apply techniques for learning in zero-sum games, yielding the first learning dynamics that converge to optimal equilibria, not only in empirical averages, but also in iterates. We demonstrate the practical scalability and flexibility of our approach by attaining state-of-the-art performance in benchmark tabular games, and by computing an optimal mechanism for a sequential auction design problem using deep reinforcement learning.

Computing Optimal Equilibria and Mechanisms via Learning in Zero-Sum Extensive-Form Games

Abstract

We introduce a new approach for computing optimal equilibria and mechanisms via learning in games. It applies to extensive-form settings with any number of players, including mechanism design, information design, and solution concepts such as correlated, communication, and certification equilibria. We observe that optimal equilibria are minimax equilibrium strategies of a player in an extensiveform zero-sum game. This reformulation allows us to apply techniques for learning in zero-sum games, yielding the first learning dynamics that converge to optimal equilibria, not only in empirical averages, but also in iterates. We demonstrate the practical scalability and flexibility of our approach by attaining state-of-the-art performance in benchmark tabular games, and by computing an optimal mechanism for a sequential auction design problem using deep reinforcement learning.

1 Introduction

What does it mean to solve a game? This is one of the central questions addressed in game theory, leading to a variety of different solution concepts. Perhaps first and foremost, there are various notions of equilibrium , strategically stable points from which no rational individual would be inclined to deviate. But is it enough to compute, or indeed learn , just any one equilibrium of a game? In two-player zero-sum games, one can make a convincing argument that a single equilibrium in fact constitutes a complete solution to the game, based on the celebrated minimax theorem of von Neumann [97]. Indeed, approaches based on computing minimax equilibria in

two-player zero-sum games have enjoyed a remarkable success in solving major AI challenges, exemplified by the recent development of superhuman poker AI agents [10, 11].

However, in general-sum games it becomes harder to argue that any equilibrium constitutes a complete solution. Indeed, one equilibrium can offer vastly different payoffs to the players than another. Further, if a player acts according to one equilibrium and another player according to a different one, the result may not be an equilibrium at all, resulting in a true equilibrium selection problem . In this paper, therefore, we focus on computing an optimal equilibrium, that is, one that maximizes a given linear objective within the space of equilibria. There are various advantages to this approach. First, in many contexts, we would simply prefer to have an equilibrium that maximizes, say, the sum of the players' utilities-and by computing such an equilibrium we also automatically avoid Pareto-dominated equilibria. Second, it can mitigate the equilibrium selection problem: if there is a convention that we always pursue an equilibrium that maximizes social welfare, this reduces the risk that players end up playing according to different equilibria. Third, if one has little control over how the game will be played but cares about its outcomes, one may like to understand the space of all equilibria. In general, a complete picture of this space can be elusive, in part because a game can have exponentially many equilibria; but computing extreme equilibria in many directions-say, one that maximizes Player 1's utility-can provide meaningful information about the space of equilibria.

That being said, many techniques that have been successful at computing a single equilibrium do not lend themselves well to computing optimal equilibria. Most notably, while no-regret learning dynamics are known to converge to different notions of correlated equilibria [49, 38, 39, 47], little is known about the properties of the equilibrium reached. In this paper, therefore, we introduce a new paradigm of learning in games for computing optimal equilibria. It applies to extensive-form settings with any number of players, including information design, and solution concepts such as correlated, communication, and certification equilibria. Further, our framework is general enough to also capture optimal mechanism design and optimal incentive design problems in sequential settings.

Summary of Our Results A key insight that underpins our results is that computing optimal equilibria in multi-player extensive-form games can be cast via a Lagrangian relaxation as a two-player zero-sum extensive-form game. This unlocks a rich technology, both theoretical and experimental, developed for computing minimax equilibria for the more challenging-and much less understood-problem of computing optimal equilibria. In particular, building on the framework of Zhang and Sandholm [100], our reduction lends itself to mechanism design and information design, as well as an entire hierarchy of equilibrium concepts, including normal-form coarse correlated equilibria (NFCCE) [79], extensive-form coarse correlated equilibria (EFCCE) [31], extensive-form correlated equilibria (EFCE) [98], communication equilibria (COMM) [36], and certification equilibria (CERT) [37]. In fact, for communication and certification equilibria, our framework leads to the first learning-based algorithms for computing them, addressing a question left open by Zhang and Sandholm [100] ( cf. [40], discussed in Appendix B).

We thus focus on computing an optimal equilibrium by employing regret minimization techniques in order to solve the induced bilinear saddle-point problem. Such considerations are motivated in part by the remarkable success of no-regret algorithms for computing minimax equilibria in large two-player zero-sum games ( e.g. , see [10, 11]), which we endeavor to transfer to the problem of computing optimal equilibria in multi-player games.

In this context, we show that employing standard regret minimizers, such as online mirror descent [91] or counterfactual regret minimization [106], leads to a rate of convergence of T -1 / 4 to optimal equilibria by appropriately tuning the magnitude of the Lagrange multipliers (Corollary 3.3). We also leverage the technique of optimism , pioneered by Chiang et al. [18], Rakhlin and Sridharan [87] and Syrgkanis et al. [94], to obtain an accelerated T -1 / 2 rate of convergence (Corollary 3.4). These are the first learning dynamics that (provably) converge to optimal equilibria. Our bilinear formulation also allows us to obtain last-iterate convergence to optimal equilibria via optimistic gradient descent/ascent (Theorem 3.5), instead of the time-average guarantees traditionally derived within the no-regret framework. As such, we bypass known barriers in the traditional learning paradigm by incorporating an additional player, a mediator , into the learning process. Furthermore, we also study an alternative Lagrangian relaxation which, unlike our earlier approach, consists of solving a sequence of zero-sum games ( cf . [30]). While the latter approach is less natural, we find that it is preferable when used in conjunction with deep RL solvers since it obviates the need for solving games with large reward ranges-a byproduct of employing the natural Lagrangian relaxation.

Experimental results We demonstrate the practical scalability of our approach for computing optimal equilibria and mechanisms. First, we obtain state-of-the-art performance in a suite of 23 different benchmark game instances for seven different equilibrium concepts. Our algorithm significantly outperforms existing LP-based methods, typically by more than one order of magnitude. We also use our algorithm to derive an optimal mechanism for a sequential auction design problem, and we demonstrate that our approach is naturally amenable to modern deep RL techniques.

1.1 Related work

In this subsection, we highlight prior research that closely relates to our work. Additional related work is included in Appendix B.

A key reference point is the recent paper of Zhang and Sandholm [100], which presented a unifying framework that enables the computation via linear programming of various mediator-based equilibrium concepts in extensive-form games, including NFCCE, EFCCE, EFCE, COMM, and CERT. 2 Perhaps surprisingly, Zhang et al. [101] demonstrated that computing optimal communication and certification equilibria is possible in time polynomial in the description of the game, establishing a stark dichotomy between the other equilibrium concepts-namely, NFCCE, EFCE, and EFCCE-for which the corresponding problem is NP -hard [98]. In particular, for the latter notions intractability turns out to be driven by the imperfect recall of the mediator [101]. Although imperfect recall induces a computationally hard problem in general from the side of the mediator [19, 59], positive parameterized results have been documented recently in the literature [103].

Our work significantly departs from the framework of Zhang and Sandholm [100] in that we follow a learning-based approach, which has proven to be a particularly favorable avenue in practice; e.g. , we refer to [26, 16, 78, 77, 104] for such approaches in the context of computing EFCE. Further, beyond the tabular setting, learning-based frameworks are amenable to modern deep reinforcement learning methods (see [70, 71, 62, 69, 51, 76, 57, 13, 52, 73, 72, 86, 105, 42], and references therein). Most of those techniques have been developed to solve two-player zero-sum games, which provides another crucial motivation for our main reduction. We demonstrate this experimentally in large games in Section 4. For multi-player games, Marris et al. [70] developed a scalable algorithm based on policy space response oracles (PSRO) [62] (a deep-reinforcement-learning-based double-oracle technique) that converges to NFC(C)E, but it does not find an optimal equilibrium.

Our research also relates to computational approaches to static auction and mechanism design through deep learning [27, 86]. In particular, similarly to the present paper, D¨ utting et al. [27] study a Lagrangian relaxation of mechanism design problems. Our approach is significantly more general in that we cover both static and sequential auctions, as well as general extensive-form games. Further, as a follow-up, Rahme et al. [86] frame the Lagrangian relaxation as a two-player game, which, however, is not zero-sum, thereby not enabling leveraging the tools known for solving zero-sum games. Finally, in a companion paper [102], we show how the framework developed in this work can be used to steer no-regret learners to optimal equilibria via nonnegative vanishing payments.

2 Preliminaries

We adopt the general framework of mediator-augmented games of Zhang and Sandholm [100] to define our class of instances. At a high level, a mediator-augmented game explicitly incorporates an additional player, the mediator , who can exchange messages with the players and issue action recommendations; different assumptions on the power of the mediator and the players' strategy sets induce different equilibrium concepts, as we clarify for completeness in Appendix A.

Definition 2.1. A mediator-augmented, extensive-form game Γ has the following components:

a set of players, identified with the set of integers J n K := { 1 , . . . , n } . We will use -i , for i ∈ n , to denote all players except i ;

J K 2. a directed tree H of histories or nodes , whose root is denoted ∅ . The edges of H are labeled with actions . The set of actions legal at h is denoted A h . Leaf nodes of H are called terminal , and the set of such leaves is denoted by Z ;

a partition H \ Z = H C ⊔ H 0 ⊔ H 1 ⊔··· ⊔ H n , where H i is the set of nodes at which i takes an action, and C and 0 denote chance and the mediator, respectively;

for each agent 3 i ∈ J n K ∪ { 0 } , a partition I i of i 's decision nodes H i into information sets . Every node in a given information set I must have the same set of legal actions, denoted by A I ;

for each agent i , a utility function u i : Z → R ; and

for each chance node h ∈ H C , a fixed probability distribution c ( · | h ) over A h .

To further clarify this definition, in Appendix A we provide two concrete illustrative examples: a single-item auction and a welfare-optimal correlated equilibrium in normal-form games.

At a node h ∈ H , the sequence σ i ( h ) of an agent i is the set of all information sets encountered by agent i , and the actions played at such information sets, along the ∅ → h path, excluding at h itself. An agent has perfect recall if σ i ( h ) = σ i ( h ′ ) for all h, h ′ in the same infoset. We will use Σ i := { σ i ( z ) : z ∈ Z } to denote the set of all sequences of player i that correspond to terminal nodes. We will assume that all players have perfect recall, though the mediator may not. 4

A pure strategy of agent i is a choice of one action in A I for each information set I ∈ I i . The sequence form of a pure strategy is the vector x i ∈ { 0 , 1 } Σ i given by x i [ σ ] = 1 if and only if i plays every action on the path from the root to sequence σ ∈ Σ i . We will use the shorthand x i [ z ] = x i [ σ i ( z )] . A mixed strategy is a distribution over pure strategies, and the sequence form of a mixed strategy is the corresponding convex combination x i ∈ [0 , 1] Σ i . We will use X i to denote the polytope of sequence-form mixed strategies of player i , and use Ξ to denote the polytope of sequence-form mixed strategies of the mediator.

For a fixed µ ∈ Ξ , we will say that ( µ , x ) is an equilibrium of Γ if, for each player i , x i is a best response to ( µ , x -i ) , that is, max x ′ i ∈ X i u i ( µ , x ′ i , x -i ) ≤ u i ( µ , x i , x -i ) . We do not require that the mediator's strategy µ is a best response. As such, the mediator has the power to commit to its strategy. The goal in this paper will generally be to reach an optimal (Stackelberg) equilibrium , that is, an equilibrium ( µ , x ) maximizing the mediator utility u 0 ( µ , x ) . We will use u ∗ 0 to denote the value for the mediator in an optimal equilibrium.

Revelation principle The revelation principle allows us, without loss of generality, to restrict our attention to equilibria where each player is playing some fixed pure strategy d i ∈ X i .

Definition 2.2. The game Γ satisfies the revelation principle if there exists a direct pure strategy profile d = ( d 1 , . . . , d n ) for the players such that, for all strategy profiles ( µ , x ) for all players including the mediator, there exists a mediator strategy µ ′ ∈ Ξ and functions f i : X i → X i for each player i such that:

The function f i in the definition of the revelation principle can be seen as a simulator for Player i : it tells Player i that playing x ′ i if other players play ( µ , d -i ) would be equivalent, in terms of all the payoffs to all agents (including the mediator), to playing f ( x ′ i ) if other agents play ( µ , x -i ) . It follows immediately from the definition that if ( µ , x ) is an ε -equilibrium, then so is ( µ ′ , d ) -that is, every equilibrium is payoff-equivalent to a direct equilibrium.

The revelation principle applies and covers many cases of interest in economics and game theory. For example, in (single-stage or dynamic) mechanism design, the direct strategy d i of each player is to report all information truthfully, and the revelation principle guarantees that for all non-truthful mechanisms ( µ , x ) there exists a truthful mechanism ( µ ′ , d ) with the same utilities for all players. 5 For correlated equilibrium, the direct strategy d i consists of obeying all (potentially randomized) recommendations that the mediator gives, and the revelation principle states that we can, without loss of generality, consider only correlated equilibria where the signals given to the players are what actions they should play. In both these cases (and indeed in general for the notions we consider in this

paper), it is therefore trivial to specify the direct strategies d without any computational overhead. Indeed, we will assume throughout the paper that the direct strategies d are given. Further examples and discussion of this definition can be found in Appendix A.

Although the revelation principle is a very useful characterization of optimal equilibria, as long as we are given d , all of the results in this paper actually apply regardless of whether the revelation principle is satisfied: when it fails, our algorithms will simply yield an optimal direct equilibrium which may not be an optimal equilibrium. Under the revelation principle, the problem of computing an optimal equilibrium can be expressed as follows:

The objective u 0 ( µ , d ) can be expressed as a linear expression c ⊤ µ , and u i ( µ , x i , d -i ) -u i ( µ , d ) can be expressed as a bilinear expression µ ⊤ A i x i . Thus, the above program can be rewritten as

Zhang and Sandholm [100] now proceed by taking the dual linear program of the inner maximization, which suffices to show that (G) can be solved using linear programming. 6

Finally, although our main focus in this paper is on games with discrete action sets, it is worth pointing out that some of our results readily apply to continuous games as well using, for example, the discretization approach of Kroer and Sandholm [60].

3 Lagrangian relaxations and a reduction to a zero-sum game

Our approach in this paper relies on Lagrangian relaxations of the linear program (G). In particular, in this section we introduce two different Lagrangian relaxations. The first one (Section 3.1) reduces computing an optimal equilibrium to solving a single zero-sum game. We find that this approach performs exceptionally well in benchmark extensive-form games in the tabular regime, but it may struggle when used in conjunction with deep RL solvers since it increases significantly the range of the rewards. This shortcoming is addressed by our second method, introduced in Section 3.2, which instead solves a sequence of suitable zero-sum games.

3.1 'Direct' Lagrangian

Directly taking a Lagrangian relaxation of the LP (G) gives the following saddle-point problem:

We first point out that the above saddle-point optimization problem admits a solution ( µ ∗ , x ∗ , λ ∗ ) :

Proposition 3.1. The problem (L1) admits a finite saddle-point solution ( µ ∗ , x ∗ , λ ∗ ) . Moreover, for all fixed λ > λ ∗ , the problems (L1) and (G) have the same value and same set of optimal solutions.

The proof is in Appendix C. We will call the smallest possible λ ∗ the critical Lagrange multiplier .

Proposition 3.2. For any fixed value λ , the saddle-point problem (L1) can be expressed as a zerosum extensive-form game.

Proof. Consider the zero-sum extensive-form game ˆ Γ between two players, the mediator and the deviator , with the following structure:

Nature picks, with uniform probability, whether or not there is a deviator. If nature picks that there should be a deviator, then nature samples, also uniformly, a deviator i ∈ J n K . Nature's actions are revealed to the deviator, but kept private from the mediator.

The game Γ is played. All players, except i if nature picked a deviator, are constrained to according to d i . The deviator plays on behalf of Player i .

Upon reaching terminal node z , there are two cases. If nature picked a deviator i , the utility is -2 λn · u i ( z ) . If nature did not pick a deviator, the utility is 2 u 0 ( z ) + 2 λ ∑ n i =1 u i ( z ) .

The mediator's expected utility in this game is

This characterization enables us to exploit technology used for extensive-form zero-sum game solving to compute optimal equilibria for an entire hierarchy of equilibrium concepts (Appendix A).

We will next focus on the computational aspects of solving the induced saddle-point problem (L1) using regret minimization techniques. All of the omitted proofs are deferred to Appendices D and E.

The first challenge that arises in the solution of (L1) is that the domain of the minimizing player is unbounded-the Lagrange multiplier is allowed to take any nonnegative value. Nevertheless, we show in Theorem D.1 that it suffices to set the Lagrange multiplier to a fixed value (that may depend on the time horizon); appropriately setting that value will allow us to trade off between the equilibrium gap and the optimality gap. We combine this theorem with standard regret minimizers (such as variants of CFR employed in Section 4.1) to guarantee fast convergence to optimal equilibria.

Corollary 3.3. There exist regret minimization algorithms such that when employed in the saddlepoint problem (L1) , the average strategy of the mediator ¯ µ := 1 T ∑ T t =1 µ ( t ) converges to the set of optimal equilibria at a rate of T -1 / 4 . Moreover, the per-iteration complexity is polynomial for communication and certification equilibria (under the nested range condition [100]), while for NFCCE, EFCCE and EFCE, implementing each iteration admits a fixed-parameter tractable algorithm.

Furthermore, we leverage the technique of optimism , pioneered by Chiang et al. [18], Rakhlin and Sridharan [87], Syrgkanis et al. [94], to obtain a faster rate of convergence.

Corollary 3.4 (Improved rates via optimism) . There exist regret minimization algorithms that guarantee that the average strategy of the mediator ¯ µ := 1 T ∑ T t =1 µ ( t ) converges to the set of optimal equilibria at a rate of T -1 / 2 . The per-iteration complexity is analogous to Corollary 3.3.

While this rate is slower than the (near) T -1 rates known for converging to some of those equilibria [24, 34, 85, 3], Corollaries 3.3 and 3.4 additionally guarantee convergence to optimal equilibria; improving the T -1 / 2 rate of Corollary 3.4 is an interesting direction for future research.

Last-iterate convergence The convergence results we have stated thus far apply for the average strategy of the mediator-a typical feature of traditional guarantees in the no-regret framework. Nevertheless, an important advantage of our mediator-augmented formulation is that we can also guarantee last-iterate convergence to optimal equilibria in general games. Indeed, this follows readily from our reduction to two-player zero-sum games, leading to the following guarantee.

Theorem 3.5 (Last-iterate convergence to optimal equilibria in general games) . There exist algorithms that guarantee that the last strategy of the mediator µ ( T ) converges to the set of optimal equilibria at a rate of T -1 / 4 . The per-iteration complexity is analogous to Corollaries 3.3 and 3.4.

As such, our mediator-augmented paradigm bypasses known hardness results in the traditional learning paradigm (Proposition D.2) since iterate convergence is no longer tied to Nash equilibria.

3.2 Thresholding and binary search

A significant weakness of the above Lagrangian is that the multiplier λ ∗ can be large. This means that, in practice, the zero-sum game that needs to be solved to compute an optimal equilibrium could have a large reward range. While this is not a problem for most tabular methods that can achieve high precision, more scalable methods based on reinforcement learning tend to be unable to solve games to the required precision. In this section, we will introduce another Lagrangian-based method for solving the program (G) that will not require solving games with large reward ranges.

Specifically, let τ ∈ R be a fixed threshold value, and consider the bilinear saddle-point problem

where ∆ k := { λ ∈ R k ≥ 0 : 1 ⊤ λ = 1 } is the probability simplex on k items. This Lagrangian was also stated-but not analyzed-by Farina et al. [30], in the special case of correlated equilibrium concepts (NFCCE, EFCCE, EFCE). Compared to that paper, ours contains a more complete analysis, and is general to more notions of equilibrium.

Like (L1), this Lagrangian is also a zero-sum game, but unlike (L1), the reward range in this Lagrangian is bounded by an absolute constant:

Proposition 3.6. Let Γ be a (mediator-augmented) game in which the reward for all agents is bounded in [0 , 1] . For any fixed τ ∈ [0 , 1] , the saddle-point problem (L2) can be expressed as a zero-sum extensive-form game whose reward is bounded in [ -2 , 2] .

Proof. Consider the zero-sum extensive-form game ˆ Γ between two players, the mediator and the deviator , with the following structure:

The deviator picks an index i ∈ n ∪ { 0 } .

̸

J K 2. If i = 0 , nature picks whether Player i can deviate, uniformly at random.

̸

The game Γ is played. All players, except i if i = 0 and nature selected that i can deviate, are constrained to play according to d i . The deviator plays on behalf of Player i .

̸

Upon reaching terminal node z , there are three cases. If nature picked i = 0 , the utility is u 0 ( z ) -τ . Otherwise, if nature picked that Player i = 0 can deviate, the utility is -2 u i ( z ) . Finally, if nature picked that Player i = 0 cannot deviate, the utility is 2 u i ( z ) .

̸

The mediator's expected utility in this game is exactly

where λ ∈ ∆ n +1 is the deviator's mixed strategy in the first step.

The above observations suggest a binary-search-like algorithm for computing optimal equilibria; the pseudocode is given as Algorithm 1. The algorithm solves O (log(1 /ε )) zero-sum games, each to precision ε . Let v ∗ be the optimal value of (G). If τ ≤ v ∗ , the value of (L2) is 0 , and we will therefore never branch low, in turn implying that u ≥ v ∗ and ℓ ≥ v ∗ -ε . As a result, we have proven:

Theorem 3.7. Algorithm 1 returns an ε -approximate equilibrium µ whose value to the mediator is at least v ∗ -2 ε . If the underlying game solver used to solve (L2) runs in time f (Γ , ε ) , then Algorithm 1 runs in time O ( f (Γ , ε ) log(1 /ε )) .

ALGORITHM 1: Pseudocode for binary search-based algorithm

The differences between the two Lagrangian formulations can be summarized as follows:

Using (L1) requires only a single game solve, whereas using (L2) requires O (log(1 /ε )) game solves.

Using (L2) requires only an O ( ε ) -approximate game solver to guarantee value v ∗ -ε , whereas using (L1) would require an O ( ε/λ ∗ ) -approximate game solver to guarantee the same, even assuming that the critical Lagrange multiplier λ ∗ in (L1) is known.

Which is preferred will therefore depend on the application. In practice, if the games are too large to be solved using tabular methods, one can use approximate game solvers based on deep reinforcement learning. In this setting, since reinforcement learning tends to be unable to achieve the high precision required to use (L1), using (L2) should generally be preferred. In Section 4, we back up these claims with concrete experiments.

4 Experimental evaluation

In this section, we demonstrate the practical scalability and flexibility of our approach, both for computing optimal equilibria in extensive-form games, and for designing optimal mechanisms in large-scale sequential auction design problems.

4.1 Optimal equilibria in extensive-form games

We first extensively evaluate the empirical performance of our two-player zero-sum reduction (Section 3.1) for computing seven equilibrium solution concepts across 23 game instances; the results using the method of Section 3.2 are slightly inferior, and are included in Appendix H. The game instances we use are described in detail in Appendix F, and belong to following eight different classes of established parametric benchmark games, each identified with an alphabetical mnemonic: B - Battleship [30], D - Liar's dice [68], GL - Goofspiel [88], K - Kuhn poker [61], L - Leduc poker [93], RS - ridesharing game [101], S - Sheriff [30], TP - double dummy bridge game [101].

For each of the 23 games, we compare the runtime required by the linear programming method of Zhang and Sandholm [100] ('LP') and the runtime required by our learning dynamics in Section 3.1 ('Ours') for computing ε -optimal equilibrium points.

Table 1 shows experimental results for the case in which the threshold ε is set to be 1% of the payoff range of the game, and the objective function is set to be the maximum social welfare (sum of player utilities) for general-sum games, and the utility of Player 1 in zero-sum games. Each row corresponds to a game, whose identifier begins with the alphabetical mnemonic of the game class, and whose size in terms of number of nodes in the game trees is reported in the second column. The remaining columns compare, for each solution concept, the runtimes necessary to approximate the optimum equilibrium point according to that solution concept. Due to space constraints, only five out of the seven solution concepts (namely, NFCCE, EFCCE, EFCE, COMM, and CERT) are shown; data for the two remaining concepts (NFCCERT and CCERT) is given in Appendix G.

We remark that in Table 1, the column 'Ours' reports the minimum across the runtime across the different hyperparameters tried for the learning dynamics. Furthermore, for each run of the algorithms, the timeout was set at one hour. More details about the experimental setup are available in Appendix G, together with finer breakdowns of the runtimes.

We observe that our learning-based approach is faster-often by more than an order of magnitudeand more scalable than the linear program. Our additional experiments with different objective functions and values of ε , available in Appendix G, confirm the finding. This shows the promise of our computational approach, and reinforces the conclusion that learning dynamics are by far the most scalable technique available today to compute equilibrium points in large games .

4.2 Exact sequential auction design

Next, we use our approach to derive the optimal mechanism for a sequential auction design problem. In particular, we consider a two-round auction with two bidders, each starting with a budget of 1 . The valuation for each item for each bidder is sampled uniformly at random from the set { 0 , 1 / 4 , 1 / 2 , 3 / 4 , 1 } . We consider a mediator-augmented game in which the principal chooses an outcome (allocation and payment for each player) given their reports (bids). We use CFR+ [95] as learning algorithm and a fixed Lagrange multiplier λ := 25 to compute the optimal communication

equilibrium that corresponds to the optimal mechanism. We terminated the learning procedure after 10000 iterations, at a duality gap for (L1) of approximately 4 . 2 × 10 -4 . Figure 1 (left) summarizes our results. On the y-axis we show how exploitable (that is, how incentive-incompatible) each of the considered mechanisms are, confirming that for this type of sequential settings, second-price auctions (SP) with or without reserve price, as well as the first-price auction (FP), are typically incentive-incompatible. On the x-axis, we report the hypothetical revenue that the mechanism would extract assuming truthful bidding. Our mechanism is provably incentive-compatible and extracts a larger revenue than all considered second-price mechanisms. It also would extract less revenue than the hypothetical first-price auction if the bidders behaved truthfully (of course, real bidders would not behave honestly in the first-price auction but rather would shade their bids downward, so the shown revenue benchmark in Figure 1 is actually not achievable). Intriguingly, we observed that 8% of the time the mechanism gives an item away for free. Despite appearing irrational, this behavior can incentivize bidders to use their budget earlier in order to encourage competitive bidding, and has been independently discovered in manual mechanism design recently [25, 75].

4.3 Scalable sequential auction design via deep reinforcement learning

We also combine our framework with deep-learning-based algorithms for scalable equilibrium computation in two-player zero-sum games to compute optimal mechanisms in two sequential auction settings. To compute an optimal mechanism using our framework, we use the PSRO algorithm [63], a deep reinforcement learning method based on the double oracle algorithm that has empirically scaled to large games such as Starcraft [96] and Stratego [72], as the game solver in Algorithm 1. 7 To train the best responses, we use proximal policy optimization (PPO) [90].

0

.

4

0

.

6

0

.

8

1

1

.

2

1

.

4

0

0

.

2

0

.

4

0

.

6

Ours

(optimal)

FP

SP

R

1

/

4

R

1

/

2

R

3

/

4

Revenue (if bidders were truthful)

Exploitability

Small sequential auction (Section 4.2)

0

.

4

0

.

6

0

.

8

1

1

.

2

1

.

4

0

5

·

10

-

2

0

.

1

0

.

15

Ours

FP

SP

R

1

/

10

R

1

/

5

R

3

/

10

R

2

/

5

Revenue (if bidders were truthful)

Exploitability

Large sequential auction (Section 4.3)

FP: First-price auction

SP: Second-price auction

R

p

: Second-price action with reserve price

p

First, to verify that the deep learning method is effective, we replicate the results of the tabular experiments in Section 4.2. We find that PSRO achieves the same best response values and optimal equilibrium value computed by the tabular experiment, up to a small error. These results give us confidence that our method is correct.

Second, to demonstrate scalability, we run our deep learning-based algorithm on a larger auction environment that would be too big to solve with tabular methods. In this environment, there are four rounds, and in each round the valuation of each player is sampled uniformly from { 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } . The starting budget of each player is, again, 1 . We find that, like the smaller setting, the optimal revenue of the mediator is ≈ 1 . 1 (right-side of Figure 1). This revenue exceeds the revenue of every second-price auction (none of which have revenue greater than 1 ). 8

5 Conclusions

We proposed a new paradigm of learning in games. It applies to mechanism design, information design, and solution concepts in multi-player extensive-form games such as correlated, communication, and certification equilibria. Leveraging a Lagrangian relaxation, our paradigm reduces the problem of computing optimal equilibria to determining minimax equilibria in zero-sum extensiveform games. We also demonstrated the scalability of our approach for computing optimal equilibria by attaining state-of-the-art performance in benchmark tabular games, and by solving a sequential auction design problem using deep reinforcement learning.

References

Kenshi Abe, Mitsuki Sakamoto, and Atsushi Iwasaki. Mutation-driven follow the regularized leader for last-iterate convergence in zero-sum games. In Conference on Uncertainty in Artificial Intelligence (UAI) , volume 180, pages 1-10. PMLR, 2022.

Ilan Adler. The equivalence of linear programs and zero-sum games. Int. J. Game Theory , 42 (1):165-177, 2013.

Ioannis Anagnostides, Constantinos Daskalakis, Gabriele Farina, Maxwell Fishelson, Noah Golowich, and Tuomas Sandholm. Near-optimal no-regret learning for correlated equilibria in multi-player general-sum games. In Symposium on Theory of Computing (STOC) , pages 736-749. ACM, 2021.

Ioannis Anagnostides, Ioannis Panageas, Gabriele Farina, and Tuomas Sandholm. On lastiterate convergence beyond zero-sum games. In International Conference on Machine Learning (ICML) , volume 162 of Proceedings of Machine Learning Research , pages 536-581. PMLR, 2022.

Robert Aumann. Subjectivity and correlation in randomized strategies. Journal of Mathematical Economics , 1:67-96, 1974.

Wa¨ ıss Azizian, Franck Iutzeler, J´ erˆ ome Malick, and Panayotis Mertikopoulos. The last-iterate convergence rate of optimistic mirror descent in stochastic variational inequalities. In Conference on Learning Theory (COLT) , volume 134 of Proceedings of Machine Learning Research , pages 326-358. PMLR, 2021.

Yakov Babichenko and Aviad Rubinstein. Communication complexity of approximate nash equilibria. Games and Economic Behavior , 134:376-398, 2022.

Yu Bai, Chi Jin, Huan Wang, and Caiming Xiong. Sample-efficient learning of stackelberg equilibria in general-sum games. In Conference on Neural Information Processing Systems (NeurIPS) , pages 25799-25811, 2021.

Siddharth Barman and Katrina Ligett. Finding any nontrivial coarse correlated equilibrium is hard. In ACM Conference on Economics and Computation (EC) , pages 815-816. ACM, 2015.

Michael Bowling, Neil Burch, Michael Johanson, and Oskari Tammelin. Heads-up limit hold'em poker is solved. Science , 347(6218), January 2015.

Noam Brown and Tuomas Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science , 359(6374):418-424, 2018.

Noam Brown and Tuomas Sandholm. Solving imperfect-information games via discounted regret minimization. In AAAI Conference on Artificial Intelligence (AAAI) , 2019.

Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong. Combining deep reinforcement learning and search for imperfect-information games. In Conference on Neural Information Processing Systems (NeurIPS) , 2020.

Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Finite-time last-iterate convergence for learning in multi-player games. In Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Modibo K. Camara, Jason D. Hartline, and Aleck C. Johnsen. Mechanisms for a no-regret agent: Beyond the common prior. In Annual Symposium on Foundations of Computer Science (FOCS) , pages 259-270. IEEE, 2020.

Andrea Celli, Alberto Marchesi, Gabriele Farina, and Nicola Gatti. No-regret learning dynamics for extensive-form correlated equilibrium. Conference on Neural Information Processing Systems (NeurIPS) , 33:7722-7732, 2020.

Xi Chen, Xiaotie Deng, and Shang-Hua Teng. Settling the complexity of computing twoplayer Nash equilibria. Journal of the ACM , 2009.

Chao-Kai Chiang, Tianbao Yang, Chia-Jung Lee, Mehrdad Mahdavi, Chi-Jen Lu, Rong Jin, and Shenghuo Zhu. Online optimization with gradual variations. In Conference on Learning Theory (COLT) , pages 6-1, 2012.

Francis Chu and Joseph Halpern. On the NP-completeness of finding an optimal strategy in games with common payoffs. International Journal of Game Theory , 2001.

Vincent Conitzer and Dmytro Korzhyk. Commitment to correlated strategies. In AAAI Conference on Artificial Intelligence (AAAI) . AAAI Press, 2011.

Vincent Conitzer and Tuomas Sandholm. New complexity results about Nash equilibria. Games and Economic Behavior , 63(2):621-641, 2008. Early version in IJCAI-03.

Constantinos Daskalakis and Ioannis Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization. In 10th Innovations in Theoretical Computer Science Conference, ITCS 2019 , volume 124 of LIPIcs , pages 27:1-27:18. Schloss Dagstuhl Leibniz-Zentrum f¨ ur Informatik, 2019.

Constantinos Daskalakis, Paul W Goldberg, and Christos H Papadimitriou. The complexity of computing a nash equilibrium. SIAM Journal on Computing , 39(1), 2009.

Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. Near-optimal no-regret learning in general games. In Conference on Neural Information Processing Systems (NeurIPS) , pages 27604-27616, 2021.

Yuan Deng, Vahab Mirrokni, and Song Zuo. Non-clairvoyant dynamic mechanism design with budget constraints and beyond. Available at SSRN 3383231 , 2019.

Miroslav Dud´ ık and Geoffrey J Gordon. A sampling-based approach to computing equilibria in succinct extensive-form games. In Conference on Uncertainty in Artificial Intelligence (UAI) , pages 151-160, 2009.

Paul D¨ utting, Zhe Feng, Harikrishna Narasimhan, David Parkes, and Sai Srivatsa Ravindranath. Optimal auctions through deep learning. In International Conference on Machine Learning , pages 1706-1715. PMLR, 2019.

Gabriele Farina and Tuomas Sandholm. Polynomial-time computation of optimal correlated equilibria in two-player extensive-form games with public chance moves and beyond. In Conference on Neural Information Processing Systems (NeurIPS) , 2020.

Gabriele Farina, Christian Kroer, and Tuomas Sandholm. Regret circuits: Composability of regret minimizers. In International Conference on Machine Learning , pages 1863-1872, 2019.

Gabriele Farina, Chun Kai Ling, Fei Fang, and Tuomas Sandholm. Correlation in extensiveform games: Saddle-point formulation and benchmarks. In Conference on Neural Information Processing Systems (NeurIPS) , 2019.

Gabriele Farina, Tommaso Bianchi, and Tuomas Sandholm. Coarse correlation in extensiveform games. In AAAI Conference on Artificial Intelligence , 2020.

Gabriele Farina, Christian Kroer, and Tuomas Sandholm. Better regularization for sequential decision spaces: Fast convergence rates for nash, correlated, and team equilibria. In ACM Conference on Economics and Computation (EC) , page 432, 2021.

Gabriele Farina, Christian Kroer, and Tuomas Sandholm. Faster game solving via predictive Blackwell approachability: Connecting regret matching and mirror descent. In AAAI Conference on Artificial Intelligence (AAAI) , 2021.

Gabriele Farina, Ioannis Anagnostides, Haipeng Luo, Chung-Wei Lee, Christian Kroer, and Tuomas Sandholm. Near-optimal no-regret learning dynamics for general convex games. In Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Tanner Fiez, Benjamin Chasnov, and Lillian J. Ratliff. Convergence of learning dynamics in stackelberg games. CoRR , abs/1906.01217, 2019.

Francoise Forges. An approach to communication equilibria. Econometrica: Journal of the Econometric Society , pages 1375-1385, 1986.

Franc ¸oise Forges and Fr´ ed´ eric Koessler. Communication equilibria with partially verifiable types. Journal of Mathematical Economics , 41(7):793-811, 2005.

Yoav Freund and Robert Schapire. Adaptive game playing using multiplicative weights. Games and Economic Behavior , 29:79-103, 1999.

Drew Fudenberg and David Levine. The Theory of Learning in Games . MIT Press, 1998.

Kaito Fujii. Bayes correlated equilibria and no-regret dynamics. CoRR , abs/2304.05005, 2023.

Itzhak Gilboa and Eitan Zemel. Nash and correlated equilibria: Some complexity considerations. Games and Economic Behavior , 1:80-93, 1989.

Denizalp Goktas, David C. Parkes, Ian Gemp, Luke Marris, Georgios Piliouras, Romuald Elie, Guy Lever, and Andrea Tacchetti. Generative adversarial equilibrium solvers. CoRR , abs/2302.06607, 2023.

Noah Golowich, Sarath Pattathil, and Constantinos Daskalakis. Tight last-iterate convergence rates for no-regret learning in multi-player games. In Conference on Neural Information Processing Systems (NeurIPS) , 2020.

Eduard Gorbunov, Hugo Berard, Gauthier Gidel, and Nicolas Loizou. Stochastic extragradient: General analysis and improved rates. In International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 151 of Proceedings of Machine Learning Research , pages 7865-7901. PMLR, 2022.

Eduard Gorbunov, Nicolas Loizou, and Gauthier Gidel. Extragradient method: O (1 /K ) lastiterate convergence for monotone variational inequalities and connections with cocoercivity. In International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 151 of Proceedings of Machine Learning Research , pages 366-402. PMLR, 2022.

J Green and J-J Laffont. Characterization of satisfactory mechanisms for the revelation of preferences for public goods. Econometrica , 45:427-438, 1977.

Amy Greenwald and Keith Hall. Correlated Q-learning. In International Conference on Machine Learning (ICML) , pages 242-249, Washington, DC, USA, 2003.

Sergiu Hart and Yishay Mansour. How long to equilibrium? the communication complexity of uncoupled equilibrium procedures. Games and Economic Behavior , 69(1):107-126, 2010.

Sergiu Hart and Andreu Mas-Colell. A simple adaptive procedure leading to correlated equilibrium. Econometrica , 68:1127-1150, 2000.

Sergiu Hart and Andreu Mas-Colell. Uncoupled dynamics do not lead to Nash equilibrium. American Economic Review , 93:1830-1836, 2003.

Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfect-information games. arXiv preprint arXiv:1603.01121 , 2016.

Johannes Heinrich, Marc Lanctot, and David Silver. Fictitious self-play in extensive-form games. In International Conference on Machine Learning (ICML) , volume 37 of JMLR Workshop and Conference Proceedings , pages 805-813. JMLR.org, 2015.

Michael D. Hirsch, Christos H. Papadimitriou, and Stephen A. Vavasis. Exponential lower bounds for finding brouwer fix points. J. Complex. , 5(4):379-416, 1989.

Wan Huang and Bernhard von Stengel. Computing an extensive-form correlated equilibrium in polynomial time. In International Workshop on Internet and Network Economics , pages 506-513. Springer, 2008.

Dmitry Ivanov, Ilya Zisman, and Kirill Chernyshev. Mediated multi-agent reinforcement learning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems , pages 49-57, 2023.

Albert Jiang and Kevin Leyton-Brown. Polynomial-time computation of exact correlated equilibrium in compact games. In ACM Conference on Electronic Commerce (EC) , 2011.

Peter H Jin, Sergey Levine, and Kurt Keutzer. Regret minimization for partially observable deep reinforcement learning. In International Conference on Machine Learning (ICML) , 2018.

Emir Kamenica and Matthew Gentzkow. Bayesian persuasion. American Economic Review , 101(6):2590-2615, 2011.

Daphne Koller and Nimrod Megiddo. The complexity of two-person zero-sum games in extensive form. Games and Economic Behavior , 4(4):528-552, October 1992.

Christian Kroer and Tuomas Sandholm. Discretization of continuous action spaces in extensive-form games. In Autonomous Agents and Multi-Agent Systems , pages 47-56. ACM, 2015.

H. W. Kuhn. A simplified two-person poker. In H. W. Kuhn and A. W. Tucker, editors, Contributions to the Theory of Games , volume 1 of Annals of Mathematics Studies, 24 , pages 97-103. Princeton University Press, Princeton, New Jersey, 1950.

Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien P´ erolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In Conference on Neural Information Processing Systems (NeurIPS) , pages 4190-4203, 2017.

Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien P´ erolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. Advances in neural information processing systems , 30, 2017.

Chung-Wei Lee, Christian Kroer, and Haipeng Luo. Last-iterate convergence in extensiveform games. In Conference on Neural Information Processing Systems (NeurIPS) , pages 14293-14305, 2021.

Joshua Letchford and Vincent Conitzer. Computing optimal strategies to commit to in extensive-form games. In ACM Conference on Electronic Commerce (EC) , 2010.

Joshua Letchford, Vincent Conitzer, and Kamesh Munagala. Learning and approximating the optimal strategy to commit to. In Algorithmic Game Theory, Second International Symposium, SAGT 2009 , volume 5814 of Lecture Notes in Computer Science , pages 250-262. Springer, 2009.

Tianyi Lin, Zhengyuan Zhou, Panayotis Mertikopoulos, and Michael I. Jordan. Finite-time last-iterate convergence for multi-agent learning in games. In International Conference on Machine Learning (ICML) , volume 119 of Proceedings of Machine Learning Research , pages 6161-6171. PMLR, 2020.

Viliam Lis` y, Marc Lanctot, and Michael H Bowling. Online monte carlo counterfactual regret minimization for search in imperfect information games. In Autonomous Agents and MultiAgent Systems , 2015.

Siqi Liu, Marc Lanctot, Luke Marris, and Nicolas Heess. Simplex neural population learning: Any-mixture bayes-optimality in symmetric zero-sum games. In International Conference on Machine Learning (ICML) , volume 162 of Proceedings of Machine Learning Research , pages 13793-13806. PMLR, 2022.

Luke Marris, Paul Muller, Marc Lanctot, Karl Tuyls, and Thore Graepel. Multi-agent training beyond zero-sum with correlated equilibrium meta-solvers. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021 , volume 139 of Proceedings of Machine Learning Research , pages 7480-7491. PMLR, 2021.

Luke Marris, Ian Gemp, Thomas Anthony, Andrea Tacchetti, Siqi Liu, and Karl Tuyls. Turbocharging solution concepts: Solving NEs, CEs and CCEs with neural equilibrium solvers. In Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Stephen McAleer, John B. Lanier, Roy Fox, and Pierre Baldi. Pipeline PSRO: A scalable approach for finding approximate nash equilibria in large games. In Conference on Neural Information Processing Systems (NeurIPS) , 2020.

Stephen McAleer, John B. Lanier, Kevin A. Wang, Pierre Baldi, and Roy Fox. XDO: A double oracle algorithm for extensive-form games. In Conference on Neural Information Processing Systems (NeurIPS) , pages 23128-23139, 2021.

Jason Milionis, Christos Papadimitriou, Georgios Piliouras, and Kelly Spendlove. An impossibility theorem in game dynamics. Proceedings of the National Academy of Sciences , 120 (41), 2023.

Vahab Mirrokni, Renato Paes Leme, Pingzhong Tang, and Song Zuo. Non-clairvoyant dynamic mechanism design. Econometrica , 88(5):1939-1963, 2020.

Matej Moravˇ c´ ık, Martin Schmid, Neil Burch, Viliam Lis´ y, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science , May 2017.

Dustin Morrill, Ryan D'Orazio, Marc Lanctot, James R. Wright, Michael Bowling, and Amy R. Greenwald. Efficient deviation types and learning for hindsight rationality in extensive-form games. In International Conference on Machine Learning (ICML) , volume 139 of Proceedings of Machine Learning Research , pages 7818-7828. PMLR, 2021.

Dustin Morrill, Ryan D'Orazio, Reca Sarfati, Marc Lanctot, James R Wright, Amy R Greenwald, and Michael Bowling. Hindsight and sequential rationality of correlated play. In AAAI Conference on Artificial Intelligence (AAAI) , volume 35, pages 5584-5594, 2021.

H. Moulin and J.-P. Vial. Strategically zero-sum games: The class of games whose completely mixed equilibria cannot be improved upon. International Journal of Game Theory , 7(3-4): 201-221, 1978.

Roger Myerson. Optimal auction design. Mathematics of Operation Research , 6:58-73, 1981.

Roger B Myerson. Multistage games with communication. Econometrica: Journal of the Econometric Society , pages 323-358, 1986.

Christos Papadimitriou. Algorithms, games and the Internet. In Symposium on Theory of Computing (STOC) , pages 749-753, 2001.

Christos Papadimitriou and Tim Roughgarden. Computing equilibria in multi-player games. In Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 82-91. SIAM, 2005.

Binghui Peng, Weiran Shen, Pingzhong Tang, and Song Zuo. Learning optimal strategies to commit to. In AAAI Conference on Artificial Intelligence (AAAI) , pages 2149-2156. AAAI Press, 2019.

Georgios Piliouras, Ryann Sim, and Stratis Skoulakis. Beyond time-average convergence: Near-optimal uncoupled online learning via clairvoyant multiplicative weights update. In Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Jad Rahme, Samy Jelassi, and S. Matthew Weinberg. Auction learning as a two-player game. In 9th International Conference on Learning Representations, ICLR 2021 . OpenReview.net, 2021.

Sasha Rakhlin and Karthik Sridharan. Optimization, learning, and games with predictable sequences. In Conference on Neural Information Processing Systems (NeurIPS) , pages 30663074, 2013.

Sheldon M Ross. Goofspiel-the game of pure strategy. Journal of Applied Probability , 8 (3):621-625, 1971.

Tim Roughgarden and Omri Weinstein. On the communication complexity of approximate fixed points. In Annual Symposium on Foundations of Computer Science (FOCS) , pages 229-238, 2016.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Shai Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends® in Machine Learning , 4(2), 2012.

Maurice Sion. On general minimax theorems. Pacific J. Math. , 8(4):171-176, 1958.

Finnegan Southey, Michael Bowling, Bryce Larson, Carmelo Piccione, Neil Burch, Darse Billings, and Chris Rayner. Bayes' bluff: Opponent modelling in poker. In Conference on Uncertainty in Artificial Intelligence (UAI) , July 2005.

Vasilis Syrgkanis, Alekh Agarwal, Haipeng Luo, and Robert E Schapire. Fast convergence of regularized learning in games. In Conference on Neural Information Processing Systems (NeurIPS) , pages 2989-2997, 2015.

Oskari Tammelin, Neil Burch, Michael Johanson, and Michael Bowling. Solving heads-up limit Texas hold'em. In International Joint Conference on Artificial Intelligence (IJCAI) , 2015.

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Micha¨ el Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature , 575(7782): 350-354, 2019.

John von Neumann. Zur Theorie der Gesellschaftsspiele. Mathematische Annalen , 100:295320, 1928.

Bernhard von Stengel and Franc ¸oise Forges. Extensive-form correlated equilibrium: Definition and computational complexity. Mathematics of Operations Research , 33(4):1002-1022, 2008.

Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In 9th International Conference on Learning Representations, ICLR 2021 . OpenReview.net, 2021.

Brian Hu Zhang and Tuomas Sandholm. Polynomial-time optimal equilibria with a mediator in extensive-form games. In Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Brian Hu Zhang, Gabriele Farina, Andrea Celli, and Tuomas Sandholm. Optimal correlated equilibria in general-sum extensive-form games: Fixed-parameter algorithms, hardness, and two-sided column-generation. In ACM Conference on Economics and Computation (EC) , pages 1119-1120. ACM, 2022.

Brian Hu Zhang, Gabriele Farina, Ioannis Anagnostides, Federico Cacciamani, Stephen Marcus McAleer, Andreas Alexander Haupt, Andrea Celli, Nicola Gatti, Vincent Conitzer, and Tuomas Sandholm. Steering no-regret learners to optimal equilibria. CoRR , abs/2306.05221, 2023.

Brian Hu Zhang, Gabriele Farina, and Tuomas Sandholm. Team belief DAG: generalizing the sequence form to team games for fast computation of correlated team max-min equilibria via regret minimization. In International Conference on Machine Learning (ICML) , volume 202 of Proceedings of Machine Learning Research , pages 40996-41018. PMLR, 2023.

Hugh Zhang. A simple adaptive procedure converging to forgiving correlated equilibria. CoRR , abs/2207.06548, 2022.

Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C. Parkes, and Richard Socher. The ai economist: Taxation policy design via two-level deep multiagent reinforcement learning. Science Advances , 8(18):eabk2607, 2022.

Martin Zinkevich, Michael Bowling, Michael Johanson, and Carmelo Piccione. Regret minimization in games with incomplete information. In Conference on Neural Information Processing Systems (NeurIPS) , 2007.

A Illustrative examples of mediator-augmented games

In this section, we further clarify the framework of mediator-augmented games we operate in through a couple of examples. We begin by noting that the family of solution concepts for extensive-form games captured by this framework includes, but is not limited to, the following:

normal-form coarse correlated equilibrium * [5, 79],

extensive-form coarse correlated equilibrium * [31],

extensive-form correlated equilibrium * [98],

certification (under the nested range condition [46, 37]) [37, 100],

communication equilibrium [81, 36],

mechanism design for sequential settings, and

information design/Bayesian persuasion for sequential settings [58].

We refer the interested reader to Zhang and Sandholm [100, Appendix G] for additional interesting concepts not mentioned above.

Example A.1 (Single-item auction) . Consider the single-good monopolist problem studied by Myerson [80]. Each player i ∈ J n K has a valuation v i ∈ V i . Agent valuations may be correlated, and distributed according to F ∈ ∆( V ) , where V := V 1 × V 2 × · · · × V n . The mechanism selects a (potentially random) payment p and a winner i ∗ ∈ J n K . The agents' utilities are quasilinear: u i ( i ∗ , p ; v i ) = v i -p if i ∗ = i and 0 otherwise. The seller wishes to maximize expected payment from the agents. This has the following timeline.

The mechanism commits to a (potentially randomized) mapping ϕ : v = ( v 1 , . . . , v n ) ↦→ ( i ∗ , p ) .

Nature samples valuations v = ( v 1 , . . . , v n ) ∼ F .

Each player i ∈ J n K privately observes her valuation v i , and then decides what valuation v ′ i to report to the mediator.

The winner and payment are selected according to ϕ ( v ′ ) .

Player i ∗ gets utility v i ∗ -p , while all other players get 0 . The mediator obtains utility u 0 = p .

In this extensive-form game, the primitives from our paper are:

a (pure) mediator strategy µ ∈ Ξ is a mapping from valuation reports v ′ = ( v ′ 1 , . . . , v ′ n ) to outcomes ( i ∗ , p ) -that is, mediator strategies are mechanisms, and mixed strategies are randomized mechanisms;

a (pure) player strategy x i ∈ X i for each player i ∈ J n K is a mapping from V i to V i indicating what valuation Player i reports as a function of its valuation v i ∈ V i ;

the direct (in mechanism design language, truthful) strategy d i for each player i is the identity map from V i to V i . (Hence, in particular, d i ∈ X i is a strategy of Player i , so it makes sense, for example, to call ( µ , d ) = ( µ , d 1 , . . . , d n ) a strategy profile.)

In particular, if profile ( µ , d 1 , . . . , d n ) is such that each player i is playing a best response, then µ is a truthful mechanism.

The conversion in Proposition 3.2 creates a zero-sum extensive-form game ˆ Γ , whose equilibria for the mediator (for sufficiently large λ ) are precisely the revenue-maximizing mechanisms. ˆ Γ has the following timeline:

Nature picks, with equal probability, whether there is a deviator. If nature picks that there is a deviator, nature also selects which player i ∈ J n K is represented by the deviator.

Nature samples valuations v = ( v 1 , . . . , v n ) ∼ F .

If nature selected that there is a deviator, the deviator observes i and its valuation v i ∈ V i , and selects a deviation v ′ i ∈ V i .

The mediator observes ( v ′ i , v -i ) ( i.e. , all other players are assumed to have reported honestly) and selects a winner i ∗ and payment p , as before.

There are now two cases. If nature selected at the root that there was to be a deviator, the utility for the mediator is -2 λnu i ( i ∗ , p ; v i ) . If nature selected at the root that there was to be no deviator, the utility for the mediator is 2 p +2 λ ∑ n i =1 u i ( i ∗ , p ; v i ) = 2 p +2 λ ( v i ∗ -p ) .

As our second example, we show how the problem of computing a social-welfare-maximizing correlated equilbrium (CE) in a normal-form game can be captured using mediator-augmented games. Example A.2 (Social welfare-optimal correlated equilibria in normal-form games) . Let A i be the action set for each player i ∈ J n K in the game, and let utility functions u i : A → R , where A := A 1 × A 2 ×···× A n . The social welfare is the function u 0 : A → R given by u 0 ( a ) := ∑ n i =1 u i ( a ) . In the traditional formulation, a CE is a correlated distribution µ over A . The elements ( a 1 , . . . , a n ) sampled from µ can be thought of as profiles of action recommendations for the players such that no player has any incentive to not follow the recommendation (obedience). This has the following well-known timeline.

At the beginning of the game, the mediator player chooses a profile of recommendations a = ( a 1 , . . . , a n ) ∈ A .

Each player observes its recommendation a i and chooses an action a ′ i .

Each player gets utility u i ( a ′ ) , and the mediator gets utility u 0 ( a ′ ) = ∑ n i =1 u i ( a ′ ) .

In this game:

mixed strategies µ for the mediator are distributions over A , that is, they are correlated profiles;

a (pure) strategy for player i ∈ J n K is a mapping from A i to A i , encoding the action player i ∈ J n K will take upon receiving each recommendation;

the direct strategy d i is again the identity map ( i.e. , each player selects as action what the mediator recommended to him/her).

In particular, if profile ( µ , d 1 , . . . , d n ) is such that each player i is playing a best response, then µ is a CE.

Proposition 3.2 yields the following zero-sum game whose mediator equilibrium strategies (for sufficiently large λ ) are precisely the welfare-optimal equilibria:

Nature picks, with equal probability, whether there is a deviator. If nature picks that there is a deviator, nature also selects which player i ∈ J n K is represented by the deviator.

The mediator picks a pure strategy profile a = ( a 1 , . . . , a n ) ∈ A .

If there is a deviator, the deviator observes i ∈ J n K and the recommendation a i and picks an action a ′ i .

There are now two cases. If nature selected at the root that there was to be a deviator, the utility for the mediator is -2 λnu i ( a ′ i , a -i ) . If nature selected at the root that there was to be no deviator, the utility for the mediator is 2 u 0 ( a ) + 2 λ ∑ n i =1 u i ( a ) = 2(1 + λ ) u 0 ( a ) .

B Further related work

In this section, we provide additional related work omitted from the main body.

We first elaborate further on prior work regarding the complexity of computing equilibria in games. Much attention has been focused on the complexity of computing just any one Nash equilibrium. This has been motivated in part by the idea that if even this is hard to compute, then this casts doubt on the concept of Nash equilibrium as a whole [23]; but the interest also stemmed from the fact that the complexity of the problem was open for a long time [82], and ended up being complete for an exotic complexity class [23, 17], whereas computing a Nash equilibrium that reaches a certain objective value is 'simply' NP -complete [41, 21]. None of this, however, justifies settling for just any one equilibrium in practice. Moreover, for correlated equilibria and related concepts, the complexity considerations are different. While one (extensive-form) correlated equilibrium can be computed in polynomial time even for multi-player succinct games [83, 56, 54] (under the polynomial expectation property), computing one that maximizes some objective function is typically NP -hard [83, 98]. To make matters worse, even finding one that is strictly better-in terms of social welfare-than the worst one is also computationally intractable [9]. Of course, our results do not contradict those lower bounds. For example, in multi-player normal-form games the strategy space of the mediator has an exponential description, thereby rendering all our algorithms exponential in the number of players. We stress again that while there exist algorithms that avoid this exponential dependence, they are not guaranteed to compute an optimal equilibrium, which is the main focus of this paper.

Moreover, in our formulation the mediator has the power to commit to a strategy. As such, our results also relate to the literature on learning and computing Stackelberg equilibria [8, 35, 66, 84, 20], as well as the work of Camara et al. [15] which casts mechanism design as a repeated interaction between a principal and an agent. Stackelberg equilibria in extensive-form games are, however, hard to find in general [65]. Our Stackelberg game has a much nicer form than general Stackelberg games-in particular, we know in advance what the equilibrium strategies will be for the followers (namely, the direct strategies, ). This observation is what allows the reduction to zero-sum games, sidestepping the need to use Stackleberg-specific technology or solvers and resulting in efficient algorithms.

In an independent and concurrent work, Fujii [40] provided independent learning dynamics converging to the set of communication equilibria in Bayesian games, but unlike our algorithm there are no guarantees for finding an optimal one. Also in independent and concurrent work, Ivanov et al. [55] develop similar Lagrangian-based dynamics for the equilibrium notion that, in the language of this paper and Zhang and Sandholm [100], is coarse full-certification equilibrium . Differing from ours, their paper does not present any theoretical guarantees (instead focusing on practical results).

C Proof of Proposition 3.1

In this section, we provide the proof of Proposition 3.1, the statement of which is recalled below.

Proposition 3.1. The problem (L1) admits a finite saddle-point solution ( µ ∗ , x ∗ , λ ∗ ) . Moreover, for all fixed λ > λ ∗ , the problems (L1) and (G) have the same value and same set of optimal solutions.

Proof. Let v be the optimal value of (G). The Lagrangian of (G) is

Now, making the change of variables ¯ x i := λ i x i , the above problem is equivalent to

where ¯ X i is the conic hull of X i : ¯ X i := { λ i x i : x i ∈ X i } . Note that, when X i is a polytope of the form X i := { F i x i = f i , x i ≥ 0 } , its conic hull can be expressed as ¯ X i = { F i x i = λ i f i , x i ≥ 0 , λ i ≥ 0 } . Thus, (1) is a bilinear saddle-point problem, where Ξ is compact and convex and ¯ X i is

convex. Thus, Sion's minimax theorem [92] applies, and we have that the value of (1) is equal to the value of the problem

Since this is a linear program 9 with a finite value, its optimum value must be achieved by some ¯ x := (¯ x 1 , . . . , ¯ x n ) := ( λ 1 x 1 , . . . , λ n x n ) . Let λ ∗ := max i λ i . Using the fact that µ ⊤ A i d i = 0 for all µ , the profile

is also an optimal solution in (2). Therefore, for any λ ≥ λ ∗ , x ′ := ( x ′ 1 , . . . , x ′ n ) is an optimal solution for the minimizer in (L1) that achieves the value of (G), so (G) and (L1) have the same value.

Now take λ > λ ∗ , and suppose for contradiction that (L1) admits some optimal µ ∈ Ξ that is not optimal in (G). Then, either c ⊤ µ < v , or µ violates some constraint max x i µ ⊤ A i x i ≤ 0 . The first case is impossible because then setting x i = d i for all i yields value less than v in (L1). In the second case, since we know that (L1) and (G) have the same value when λ = λ ∗ , we have

D Fast computation of optimal equilibria via regret minimization

In this section, we focus on the computational aspects of solving the induced saddle-point problem (L1) using regret minimization techniques. In particular, this section serves to elaborate on our results presented earlier in Section 3.1. All of the omitted proofs are deferred to Appendix E for the sake of exposition.

As we explained in Section 3.1, the first challenge that arises in the solution of (L1) is that the domain of Player min is unbounded-the Lagrange multiplier is allowed to take any nonnegative value. Nevertheless, we show in the theorem below that it suffices to set the Lagrange multiplier to a fixed value (that may depend on the time horizon); we reiterate that appropriately setting that value will allow us to trade off between the equilibrium gap and the optimality gap. Before we proceed, we remark that the problem of Player min in (L1) can be decomposed into the subproblems faced by each player separately, so that the regret of Player min can be cast as the sum of the players' regrets (see Corollary E.2); this justifies the notation ∑ n i =1 Reg T X i used for the regret of Player min below.

Theorem D.1. Suppose that Player max in the saddle-point problem (L1) incurs regret Reg T Ξ and Player min incurs regret ∑ n i =1 Reg T X i after T ∈ N repetitions, for a fixed λ = λ ( T ) > 0 . Then, the average mediator strategy Ξ ∋ ¯ µ := 1 T ∑ T t =1 µ ( t ) satisfies the following:

For any strategy µ ∗ ∈ Ξ such that max i ∈ J n K max x ∗ i ∈ X i ( µ ∗ ) ⊤ A i x ∗ i ≤ 0 ,

The equilibrium gap of ¯ µ decays with a rate of λ -1 :

As a result, if we can simultaneously guarantee that λ ( T ) → + ∞ and 1 T ( Reg T Ξ + ∑ n i =1 Reg T X i ) → 0 , as T → + ∞ , Theorem D.1 shows that both the optimality gap (Item 1) and the equilibrium gap

(Item 2) converge to 0 . We show that this is indeed possible in the sequel (Corollaries 3.3 and 3.4), obtaining favorable rates of convergence as well.

It is important to stress that while there exists a bounded critical Lagrange multiplier for our problem (Proposition 3.1), thereby obviating the need for truncating its value, such a bound is not necessarily polynomial. For example, halving the players' utilities while maintaining the utility of the mediator would require doubling the magnitude of the critical Lagrange multiplier.

Next, we combine Theorem D.1 with suitable regret minimization algorithms in order to guarantee fast convergence to optimal equilibria. Let us first focus on the side of Player min in (L1), which, as pointed out earlier, can be decomposed into subproblems corresponding to each player separately (Corollary E.2). Minimizing regret over the sequence-form polytope can be performed efficiently with a variety of techniques, which can be classified into two basic approaches. The first one is based on the standard online mirror descent algorithm (see, e.g. , [91]), endowed with appropriate distance generating functions (DGFs) [32]. The alternative approach is based on regret decomposition, in the style of CFR [106, 29]. In particular, given that the players' observed utilities have range O ( λ ) , the regret of each player under suitable learning algorithms will grow as O ( λ √ T ) (see Proposition E.1). Furthermore, efficiently minimizing regret from the side of the mediator depends on the equilibrium concept at hand. For NFCCE, EFCCE and EFCE, the imperfect recall of the mediator [101] induces a computationally hard problem [19], which nevertheless admits fixed-parameter tractable algorithms [103] (Proposition E.3). In contrast, for communication and certification equilibria the perfect recall of the mediator enables efficient computation for any extensive-form game. As a result, selecting a bound of λ := T 1 / 4 on the Lagrange multiplier, so as to optimally trade off Items 1 and 2 of Theorem D.1, leads to the following conclusion.

Corollary 3.3. There exist regret minimization algorithms such that when employed in the saddlepoint problem (L1) , the average strategy of the mediator ¯ µ := 1 T ∑ T t =1 µ ( t ) converges to the set of optimal equilibria at a rate of T -1 / 4 . Moreover, the per-iteration complexity is polynomial for communication and certification equilibria (under the nested range condition [100]), while for NFCCE, EFCCE and EFCE, implementing each iteration admits a fixed-parameter tractable algorithm.

Furthermore, we leverage the technique of optimism , pioneered by Chiang et al. [18], Rakhlin and Sridharan [87], Syrgkanis et al. [94] in the context of learning in games, in order to obtain faster rates. In particular, using optimistic mirror descent we can guarantee that the sum of the agents' regrets in the saddle-point problem (L1) will now grow as O ( λ ) (Proposition E.4), instead of the previous bound O ( λ √ T ) obtained using vanilla mirror descent. Thus, letting λ = T 1 / 2 leads to the following improved rate of convergence.

Corollary 3.4 (Improved rates via optimism) . There exist regret minimization algorithms that guarantee that the average strategy of the mediator ¯ µ := 1 T ∑ T t =1 µ ( t ) converges to the set of optimal equilibria at a rate of T -1 / 2 . The per-iteration complexity is analogous to Corollary 3.3.

We reiterate that while this rate is slower than the (near) T -1 rates known for converging to some of those equilibria [24, 34, 85, 3], Corollaries 3.3 and 3.4 additionally guarantee convergence to optimal equilibria; improving the T -1 / 2 rate of Corollary 3.4 is an interesting direction for the future.

Last-iterate convergence The results we have stated thus far apply for the average strategy of the mediator-a typical feature of traditional guarantees in the no-regret framework. In contrast, there is a recent line of work that endeavors to recover last-iterate guarantees as well [22, 45, 1, 14, 6, 99, 64, 43, 67, 44]. Yet, despite many efforts, the known last-iterate guarantees of no-regret learning algorithms apply only for restricted classes of games, such as two-player zero-sum games. There is an inherent reason for the limited scope of those results: last-iterate convergence is inherently tied to Nash equilibria, which in turn are hard to compute in general games [23, 17]-let alone computing an optimal one [41, 21]. Indeed, any given joint strategy profile of the players induces a product distribution, so iterate convergence requires-essentially by definition-at the very least computing an approximate Nash equilibrium.

Proposition D.2 (Informal) . Any independent learning dynamics (without a mediator) require superpolynomial time to guarantee ε -last-iterate convergence, for a sufficiently small ε = O ( m -c ) , even for two-player m -action normal-form games, unless PPAD ⊆ P .

There are also unconditional exponential communication-complexity lower bounds for uncoupled methods [7, 53, 89, 48], as well as other pertinent impossibility results [50, 74] that document the inherent persistence of limit cycles in general-sum games. In contrast, an important advantage of our mediator-augmented formulation is that we can guarantee last-iterate convergence to optimal equilibria in general games. Indeed, this follows readily from our reduction to two-player zero-sum games, for which the known bound of O ( λ/ √ T ) for the iterate gap of (online) optimistic gradient descent can be employed (see Appendix E).

Theorem 3.5 (Last-iterate convergence to optimal equilibria in general games) . There exist algorithms that guarantee that the last strategy of the mediator µ ( T ) converges to the set of optimal equilibria at a rate of T -1 / 4 . The per-iteration complexity is analogous to Corollaries 3.3 and 3.4.

As such, our mediator-augmented learning paradigm bypasses the hardness of Proposition D.2 since last-iterate convergence is no longer tied to convergence to Nash equilibria.

E Omitted proofs from Section D

In this section, we provide the omitted proofs from Appendix D, which concerns the solution of the saddle-point problem described in (L1) using regret minization. Appendix E.1 then presents a slightly different approach for solving (L1) using regret minimization over conic hulls, which is used in our experiments.

We begin with the proof of Theorem D.1, the statement of which is recalled below. In the following proof, we will denote by L : Ξ × X ∋ ( µ , ( x i ) n i =1 ) ↦→ c ⊤ µ -λ ∑ n i =1 µ ⊤ A i x i the induced Lagrangian, for a fixed λ > 0 .

Theorem D.1. Suppose that Player max in the saddle-point problem (L1) incurs regret Reg T Ξ and Player min incurs regret ∑ n i =1 Reg T X i after T ∈ N repetitions, for a fixed λ = λ ( T ) > 0 . Then, the average mediator strategy Ξ ∋ ¯ µ := 1 T ∑ T t =1 µ ( t ) satisfies the following:

For any strategy µ ∗ ∈ Ξ such that max i ∈ J n K max x ∗ i ∈ X i ( µ ∗ ) ⊤ A i x ∗ i ≤ 0 ,

The equilibrium gap of ¯ µ decays with a rate of λ -1 :

Proof. Let ¯ µ ∈ Ξ be the average strategy of the mediator and ¯ x i ∈ X i be the average strategy of each player i ∈ J n K over the T iterations. We first argue about the approximate optimality of ¯ µ . In particular, we have that

where (3) follows from the fact that

in turn implying (3) since ∑ n i =1 max x ∗ i ∈ X i ¯ µ ⊤ A i x ∗ i ≥ ∑ n i =1 ¯ µ ⊤ A i d i = 0 ; (4) uses the notation µ ∗ to represent any equilibrium strategy optimizing the objective c ⊤ µ ; and (5) follows from the fact that, by assumption, µ ∗ satisfies the equilibrium constraint: max x ∗ i ∈ X i ( µ ∗ ) ⊤ A i x ∗ i ≤ 0 for any player i ∈ J n K , as well as the nonnegativity of the Lagrange multiplier. This establishes Item 1 of the statement.

Next, we analyze the equilibrium gap of ¯ µ . Consider any mediator strategy µ ∈ Ξ such that µ ⊤ A i x i ≤ 0 for any x i ∈ X i and player i ∈ J n K . By (6),

But, by the equilibrium constraint for µ , it follows that µ ⊤ A i x i ≤ 0 for any x i ∈ X i and player i ∈ J n K , in turn implying that ∑ n i =1 µ ⊤ A i x i ≤ 0 . So, combining with (7),

Finally, given that max x ∗ i ′ ∈ X i ′ ¯ µ ⊤ A i ′ x ∗ i ′ ≥ ¯ µ ⊤ A i ′ d i ′ = 0 for any player i ′ , it follows that

and (8) implies Item 2 of the statement.

Bounding the regret of the players To instantiate Theorem D.1 for our problem, we first bound the regret of Player min in (L1) in terms of the magnitude of the Lagrange multiplier. As we explained in Appendix D, the regret minimization problem faced by Player min can be decomposed into subproblems over the sequence-form polytope, one for each player. To keep the exposition selfcontained, let us first recall the standard regret guarantee of CFR under the sequence-form polytope.

Proposition E.1 ([106]) . Let Reg T X i be the regret cumulated by CFR [106] over the sequence-form polytope X i . Then, for any T ∈ N ,

where D i > 0 is the range of utilities observed by player i ∈ J n K and C > 0 is an absolute constant.

An analogous regret guarantee holds for online mirror descent [91]. As a result, given that the range of observed utilities for each player is O ( λ ) , for a fixed Lagrange multiplier λ , we arrive at the following result.

Corollary E.2. If all players employ CFR, the regret of Player min in (L1) can be bounded as

Here, we used the simple fact that regret over a Cartesian product can be expressed as the sum of the regrets over each individual set [29].

Bounding the regret of the mediator We next turn our attention to the regret minimization problem faced by the mediator. The complexity of this problem depends on the underlying notion of equilibrium at hand. In particular, for the correlated equilibrium concepts studied in this papernamely, NFCCE, EFCCE and EFCE, we employ the framework of DAG-form sequential decision problem (DFSDP) [103]. In particular, DFSDP is a sequential decision process over a DAG. We will denote by E the set of edges of the DAG, and by S the set of its nodes; we refer to Zhang et al. [103] for the precise definitions. A crucial structural observation is that a DFSDP can be derived from the probability simplex after repeated Cartesian products and scaled-extensions operations; suitable DGFs arising from such operations have been documented [32]. As such, we can use the following guarantee shown by Zhang et al. [103].

Proposition E.3 ([103]) . Let Reg T Ξ be the regret cumulated by the regret minimization algorithm R Ξ used by the mediator (Player max in (L1) ) up to time T ∈ N . Then, if R Ξ is instantiated using CFR, or suitable variants thereof, Reg T Ξ = O ( |S| √ TD ) , where D is the range of the utilities observed by the mediator. Further, the iteration complexity is O ( | E | ) .

As a result, combining Theorem D.1 with Proposition E.3 and Corollary E.2, and setting the Lagrange multiplier λ := T 1 / 4 , we establish the statement of Corollary 3.3.

Faster rates through optimism Next, to obtain Corollary 3.4, let us parameterize the regret of optimistic gradient descent in terms of the maximum utility, which can be directly extracted from the work of Rakhlin and Sridharan [87].

Proposition E.4. If both agents in the saddle-point problem (L1) employ optimistic gradient descent with a sufficiently small learning rate η > 0 , then the sum of their regrets is bounded by O ( λ ) , for any fixed λ > 0 .

Proof Sketch. By the RVU bound [94, 87], the sum of the agents' regrets can be bounded as (diam 2 Ξ +diam 2 X ) /η , for a sufficiently small η = O (1 /λ ) , where diam Ξ and diam X denote the ℓ 2 -diameter of Ξ and X , respectively. Thus, taking η = Θ(1 /λ ) to be sufficiently small implies the statement.

As a result, taking λ := T 1 / 2 and applying Theorem D.1 leads to the bound claimed in Corollary 3.4.

Last-iterate convergence Finally, let us explain how known guarantees can be applied to establish Theorem 3.5. By applying [4], it follows that for a sufficiently small learning rate η = O (1 /λ ) there is an iterate of optimistic gradient descent with O ( 1 √ ) duality gap. Thus, setting η =

We further note that once the desirable duality gap O η √ T has been reached, one can fix the players' strategies to obtain a last-iterate guarantee as well.

η T Θ(1 /λ ) to be sufficiently small we get that the duality gap is bounded by O ( λ √ T ) . As a result, for λ := T 1 / 4 Theorem D.1 implies a rate of T -1 / 4 , as claimed in Theorem 3.5. We remark that while the guarantee of Theorem D.1 has been expressed in terms of the sum of the agents' regrets, the conclusion readily applies for any pair of strategies ( ¯ µ , ¯ x ) ∈ Ξ × X by replacing the term Reg T Ξ + ∑ n i =1 Reg X i with the duality gap of ( ¯ µ , ¯ x ) with respect to (L1) (for the fixed value of λ ). ( 1 )

E.1 An alternative approach

In this subsection, we highlight an alternative approach for solving the saddle-point (L1) using regret minimization. In particular, we first observe that it can expressed as the saddle-point problem

where ¯ X i := { λ i x i : λ i ∈ [0 , K ] , x i ∈ X i } is the conic hull of X i truncated to a sufficiently large parameter K > 0 . Analogously to our approach in Appendix D, suitably tuning the value of K will allow us trade off between the optimality gap and the equilibrium gap. In this context, we point out below that how to construct a regret minimizer over a conic hull.

Regret minimization over conic hulls Suppose that R X i is a regret minimizer over X i and R + is a regret minimizer over the interval [0 , K ] . Based on those two regret minimizers, Algorithm 2 shows how to construct a regret minimizer over the conic hull ¯ X i . More precisely, Algorithm 2 follows the convention that a generic regret minizer R interacts with its environment via the following two subroutines:

R . NEXTSTRATEGY: R returns the next strategy based on its internal state; and

R . OBSERVEUTILITY ( u ( t ) ) : R receives as input from the environment a (compatible) utility vector u ( t ) at time t ∈ N .

The formal statement regarding the cumulated regret of Algorithm 2 below is cast in the framework of regret circuits [29].

Proposition E.5 (Regret circuit for the conic hull) . Suppose that Reg T X i and Reg T + is the cumulative regret incurred by R X i and R + , respectively, up to a time horizon T ∈ N . Then, the regret Reg T ¯ X i of R ¯ X i constructed based on Algorithm 2 can be bounded as

Proof. By construction, we have that Reg T ¯ X i is equal to

where the last derivation uses that λ ∗ i ∈ [0 , K ] .

As a result, by suitable instantiating R X i and R + ( e.g. , using Proposition E.1), the regret circuit of Proposition E.5 enables us to construct a regret minimizer over ¯ X i with regret bounded as O ( K √ T ) . In turn, this directly leads to a regret minimizer for Player min in (9) with regret bounded by O ( K √ T ) . We further remark that Theorem D.1 can be readily cast in terms of the saddle-point problem (9) as well, parameterized now by K instead of λ . As a result, convergence bounds such as Corollary 3.3 also apply to regret minimizers constructed via conic hulls.

F Description of game instances

In this section, we provide a detailed description of the game instances used in our experiments in Section 4.1.

F.1 Liar's dice ( D ), Goofspiel ( GL ), Kuhn poker ( K ), and Leduc poker ( L )

Liar's dice At the start of the game, each of the three players rolls a fair k -sided die privately. Then, the players take turns making claims about the outcome of their roll. The first player starts by stating any number from 1 to k and the minimum number of dice they believe are showing that value among all players. On their turn, each player has the option to make a higher claim or challenge the previous claim by calling the previous player a 'liar.' A claim is higher if the number rolled is higher or the number of dice showing that number is higher. If a player challenges the previous claim and the claim is found to be false, the challenger is rewarded +1 and the last bidder receives a penalty of -1. If the claim is true, the last bidder is rewarded +1, and the challenger receives -1. All other players receive 0 reward. We consider two instances of the game, one with k = 2 ( D32 ) and one with k = 3 ( D33 ).

Goofspiel This is a variant of Goofspiel with limited information. In this variation, in each turn the players do not reveal the cards that they have played. Instead, players show their cards to a neutral umpire, who then decides the winner of the round by determining which card is the highest. In the event of a tie, the umpire directs the players to divide the prize equally among the tied players, similar to the Goofspiel game. The instance GL3 which we employ has 3 players, 3 ranks, and imperfect information.

Kuhn poker Three-player Kuhn Poker, an extension of the original two-player version proposed by Kuhn [61], is played with three players and r cards. Each player begins by paying one chip to the pot and receiving a single private card. The first player can check or bet ( i.e. , putting an additional chip in the pot). Then, the second player can check or bet after a first player's check, or fold/call the first player's bet. The third player can either check or bet if no previous bet was made, otherwise they must fold or call. At the showdown, the player with the highest card who has not folded wins all the chips in the pot. We use the instance K35 which has rank r = 5 .

Leduc poker In our instances of the three-player Leduc poker the deck consists of s suits with r cards each. Our instances are parametric in the maximum number of bets b , which in limit hold'em is not necessarely tied to the number of players. The maximum number of raise per betting round can be either 1, 2 or 3. At the beginning of the game, players each contribute one chip to the pot. The game proceeds with two rounds of betting. In the first round, each player is dealt a private card, and in the second round, a shared board card is revealed. The minimum raise is set at 2 chips in the first round and 4 chips in the second round. We denote by L 3 brs an instance with three players with b bets per round, r ranks, and s suits. We employ the following five instances: L3132 , L3133 , L3151 , L3223 , L3523 .

F.2 Battleship game ( B ) and Sheriff game ( S )

Battleship The game is a general-sum version of the classic game Battleship, where two players take turns placing ships of varying sizes and values on two separate grids of size h × w , and then take turns firing at their opponent. Ships which have been hit at all their tiles are considered destroyed. The game ends when one player loses all their ships, or after each player has fired r shots. Each player's payoff is determined by the sum of the value of the opponent's destroyed ships minus γ ≥ 1 times the number of their own lost ships. We denote by B phwr an instance with p players on a grid of size h × w , one unit-size ship for each player, and r rounds. We consider the following four instances: B2222 , B2322 , B2323 , B2324 .

Sheriff This game is a simplified version of the Sheriff of Nottingham board game, which models the interaction between a Smuggler -who is trying to smuggle illegal items in their cargo-and the Sheriff -who's goal is stopping the Smuggler. First, the Smuggler has to decide the number n ∈ { 0 , . . . , N } of illegal items to load on the cargo. Then, the Sheriff decides whether to inspect the cargo. If they choose to inspect, and find illegal goods, the Smuggler has to pay p · n to the Sheriff. Otherwise, the Sheriff has to compensate the Smuggler with a reward of s . If the Sheriff decides not to inspect the cargo, the Sheriff's utility is 0, and the Smuggler's utility is v · n . After the Smuggler has loaded the cargo, and before the Sheriff decides whether to inspect, the Smuggler can try to bribe the Sheriff to avoid the inspection. In particular, they engage in r rounds of bargaining and, for each round i , the Smuggler proposes a bribe b i ∈ { 0 , . . . , B } , and the Sheriff accepts or declines it. Only the proposal and response from the final round r are executed. If the Sheriff accepts a bribe b r then they get b r , while the Smuggler's utility is vn -b r . Further details on the game can be found in Farina et al. [30]. An instance S pNBr has p players, N illegal items, a maximum bribe of B , and r rounds of bargaining. The other parameters are v = 5 , p = 1 , s = 1 and they are fixed across all instances. We employ the following five instances: S2122 , S2123 , S2133 , S2254 , S2264 .

F.3 The double-dummy bridge endgame ( TP )

The double-dummy bridge endgame is a benchmark introduced by Zhang et al. [101] which simulates a bridge endgame scenario. The game uses a fixed deck of playing cards that includes three ranks (2, 3, 4) of each of four suits (spades, hearts, diamonds, clubs). Spades are designated as the trump suit. There are four players involved: two defenders sitting across from each other, the dummy, and the declarer. The dummy's actions will be controlled by the declarer, so there are only

three players actively participating. However, for clarity, we will refer to all four players throughout this section.

The entire deck of cards is randomly dealt to the four players. We study the version of the game that has perfect information, meaning that all players' cards are revealed to everyone, creating a game in which all information is public ( i.e. , a double-dummy game ). The game is played in rounds called tricks . The player to the left of the declarer starts the first trick by playing a card. The suit of this card is known as the lead suit . Going in clockwise order, the other three players play a card from their hand. Players must play a card of the lead suit if they have one, otherwise, they can play any card. If a spade is played, the player with the highest spade wins the trick. Otherwise, the highest card of the lead suit wins the trick. The winner of each trick then leads the next one. At the end of the game, each player earns as many points as the number of tricks they won. In this adversarial team game, the two defenders are teammates and play against the declarer, who controls the dummy.

The specific instance that we use ( i.e. , TP3 ) has 3 ranks and perfect information. The dummy's hand is fixed as 2 ♠ 2 ♥ 3 ♥ .

F.4 Ridesharing game ( RS )

This benchmark was first introduced by Zhang et al. [101], and it models the interaction between two drivers competing to serve requests on a road network. The network is defined as an undirected graph G U = ( V U , E U ) , where each vertex v ∈ V U corresponds to a ride request to be served. Each request has a reward in R ≥ 0 , and each edge in the network has some cost. The first driver who arrives on node v ∈ V U serves the corresponding ride, and receives the corresponding reward. Once a node has been served, it stays clean until the end of the game. The game terminates when all nodes have been cleared, or when a timeout is met ( i.e. , there's a fixed time horizon T ). If the two drivers arrive simultaneously on the same vertex they both get reward 0. The final utility of each driver is computed as the sum of the rewards obtained from the beginning until the end of the game. The initial position of the two drivers is randomly selected at the beginning of the game. Finally, the two drivers can observe each other's position only when they are simultaneously on the same node, or they are in adjacent nodes.

Ridesharing games are particularly well-suited to study the computation of optimal equilibria because they are not triangle-free [28].

Setup We denote by RS p i T a ridesharing instance with p drivers, network configuration i , and horizon T . Parameter i ∈ { 1 , 2 } specifies the graph configuration. We consider the two network configurations of Zhang et al. [101], their structure is reported in Figure 2. All edges are given unitary cost. We consider a total of four instances: RS212 , RS222 , RS213 , RS223 .

0

{

1

}

1

{

.5

}

3

{

.5

}

2

{

1.5

}

4

{

4.5

}

5

{

2

}

6

{

1.5

}

0

{

1

}

1

{

.5

}

2

{

.5

}

3

{

1.5

}

4

{

1

}

5

{

2.5

}

6

{

1.5

}

7

{

5

}

G Additional experimental results

G.1 Investigation of lower equilibrium approximation

In Table 2 we show results, using the same format as Table 1 shown in the body, for the case in which the approximation ε is set to be 0 . 1% of the payoff range of the game, as opposed to the 1% threshold of the body.

We observe that none of the results change qualitatively when this increased precision is considered.

G.2 Game values and size of Mediator's strategy space

Tables 3 and 4 reports optimal equilibrium values for all games and all equilibrium concepts for which the LP algorithm was able to compute an exact value (We restrict to the cases solvable by LP because the Lagrangian relaxations only compute an ε -equilibrium, but the mediator objective in an ε -equilibrium could be arbitrarily far away from the mediator objective in an exact equilibrium). We hope that these will be good references for future researchers interested in this topic. Table 5 reports the size of the strategy space of the mediator player in the two-player zero-sum game that captures the computation of optimal equilibria, in terms of the number of decision points and edges. For correlated notions, this number may be exponential in the original game size; for communication and certification notions, it will always be polynomial.

G.3 Detailed breakdown by equilibrium and objective function (two-player games)

For each two-player game, we try three different objective functions: maximizing the utility of Player 1, maximizing the utility of Player 2, and maximizing social welfare. For each objective, we stop the optimization at the approximation level defined as 1% of the payoff range of the game.

We use online optimistic gradient descent to update the Lagrange multipliers of each player (see Appendix E.1). For each objective, we report the following information:

The runtime of the linear-programming-based algorithm ('LP') of Zhang and Sandholm [100].

The runtime of our algorithm where each agent (player or mediator) uses the Discounted CFR ('DCFR') algorithm set up with the hyperparameters recommended in the work by Brown and Sandholm [12]. In the table, we report the best runtime across all choices of the stepsize hyperparameter η ∈ { 0 . 01 , 0 . 1 , 1 . 0 , 10 . 0 } used in online optimistic gradient descent to update the Lagrange multipliers. The value of η that produces the reported runtime is noted in square brackets.

The runtime of our algorithm where each agent (player or mediator) uses the Predictive CFR + ('PCFR + ') algorithm of [33]. In the table, we report the best runtime across all choices of the stepsize hyperparameter η ∈ { 0 . 01 , 0 . 1 , 1 . 0 , 10 . 0 } used in online optimistic gradient descent to update the Lagrange multipliers. The value of η that produces the reported runtime is again noted in square brackets.

G.3.1 Results for NFCCE solution concept

G.3.2 Results for EFCCE solution concept

G.3.3 Results for EFCE solution concept

G.3.4 Results for COMM solution concept

G.3.5 Results for NFCCERT solution concept

G.3.6 Results for CCERT solution concept

Game

Maximize Player 1's utility

Maximize Player 2's utility

Maximize social welfare

LP

Ours (DCFR)

Ours (PCFR

+

)

LP

Ours (DCFR)

Ours (PCFR

+

)

LP

Ours (DCFR)

Ours (PCFR

+

)

B2222

0.00s

0.00s

[1.0]

0.01s

[1.0]

0.00s

0.01s

[1.0]

0.01s

[1.0]

0.00s

0.01s

[1.0]

0.01s

[1.0]

B2322

0.00s

0.07s

[1.0]

0.22s

[1.0]

0.00s

0.15s

[1.0]

0.34s

[1.0]

0.00s

0.22s

[1.0]

0.33s

[1.0]

B2323

7.00s

3.75s

[1.0]

7.37s

[1.0]

6.00s

4.69s

[1.0]

9.79s

[1.0]

8.00s

5.39s

[1.0]

6.08s

[1.0]

timeout

54.92s

[1.0]

1m 2s

[1.0]

timeout

59.02s

[1.0]

1m 28s

[1.0]

timeout

1m 31s

[1.0]

1m 41s

[1.0]

S2122

0.00s

0.00s

[1.0]

0.01s

[0.1]

0.00s

0.00s

[0.1]

0.00s

[0.1]

0.00s

0.00s

[1.0]

0.01s

[0.1]

S2123

0.00s

0.01s

[1.0]

[1.0]

0.00s

0.02s

[1.0]

0.03s

[0.1]

0.00s

0.01s

[1.0]

[1.0]

S2133

1.00s

0.04s

[1.0]

0.07s

[1.0]

1.00s

0.08s

[0.1]

0.09s

[0.1]

1.00s

[1.0]

0.11s

[1.0]

S2254

1m 41s

8.61s

[0.1]

15.31s

[0.1]

1m 47s

25.38s

[0.1]

23.28s

[0.1]

2m 3s

8.22s

[0.1]

16.08s

[0.1]

S2264

timeout

20.11s

[1.0]

23.99s

[0.1]

timeout

1m 9s

[0.01]

1m 5s

[0.1]

timeout

16.50s

[0.1]

29.02s

[0.1]

RS212

0.00s

0.00s

[10.0]

0.00s

[1.0]

0.00s

0.00s

[10.0]

0.00s

[1.0]

0.00s

0.00s

[1.0]

0.00s

[1.0]

RS222

0.00s

0.00s

[0.01]

0.00s

[10.0]

0.00s

0.00s

[1.0]

0.00s

[1.0]

0.00s

0.00s

[0.01]

0.00s

[0.01]

RS213

0.00s

0.01s

[10.0]

0.01s

[1.0]

0.00s

0.01s

[1.0]

0.01s

[1.0]

0.00s

0.01s

[0.1]

0.01s

[0.01]

RS223

0.00s

0.01s

[10.0]

0.01s

[1.0]

0.00s

0.01s

[1.0]

0.01s

[1.0]

0.00s

0.01s

[10.0]

0.01s

[0.1]

G.3.7 Results for CERT solution concept

G.4 Detailed breakdown by equilibrium and objective function (three-player games)

For each two-player game, we try three different objective functions: maximizing the utility of Player 1, maximizing the utility of Player 2, and maximizing the utility of Player 3. As in Appendix G.3, for each objective we stop the optimization at the approximation level defined as 1% of the payoff range of the game.

For each game and objective, we report the same information as Appendix G.3.

G.4.1 Results for NFCCE solution concept

G.4.2 Results for EFCCE solution concept

G.4.3 Results for EFCE solution concept

G.4.4 Results for COMM solution concept

G.4.5 Results for NFCCERT solution concept

G.4.6 Results for CCERT solution concept

G.4.7 Results for CERT solution concept

H Experimental results for binary search-based Lagrangian

In this section, we compare the performance of our 'direct' Lagrangian approach against our binary search-based Lagrangian approach for computing several equilibrium concepts at the approximation level defined as 1% of the payoff range of the game.

For each solution concept, we identify the same three objectives as Appendix G (maximizing each player's individual utility, and maximizing the social welfare in our two-player general-sum games). For each objective, each of the following tables compares three runtimes:

The time required by the linear program (column 'LP');

The time required by the 'direct' (non-binary search-based) Lagrangian approach, taking the fastest between the implementations using DCFR and PCFR + as the underlying noregret algorithms (column 'Lagrangian').

The time required by the binary search-based Lagrangian approach, taking the fastest between the implementations using DCFR and PCFR + as the underlying no-regret algorithms (column 'Bin.Search').

We observe that our two approaches behave similarly in small games. In larger games, especially with three players, the direct Lagrangian tends to be 2-4 times faster.

H.1 Detailed Breakdown by Equilibrium and Objective Function (Two-Player Games)

H.1.1 Results for NFCCE solution concept

H.1.3 Results for EFCE solution concept

H.1.4 Results for COMM solution concept

H.1.5 Results for NFCCERT solution concept

H.1.6 Results for CCERT solution concept

H.1.7 Results for CERT solution concept

H.2 Detailed Breakdown by Equilibrium and Objective Function (Three-Player Games)

H.2.1 Results for NFCCE solution concept

H.2.3 Results for EFCE solution concept

H.2.4 Results for COMM solution concept

H.2.5 Results for NFCCERT solution concept

H.2.6 Results for CCERT solution concept

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a learning-based framework to solve for optimal equilibria in n-player general-sum extensive-form games. The key idea is to leverage Lagrangian relaxation and convert such games to 2-player zero-sum extensive-form games which can then be solved using known learning methods that solve for minimax equilibria.\n\nSTRENGTHS:\n* significance\nI believe this work could be a significant contribution to the field as it proposes a computationally practical method that solves for optimal equilibria in n-player general-sum games. This is significant as it offers a language to speak about optimality of equilibria in addition to stability and could make game theoretic methods attractive in wider classes of practical applications. \n\n* originality\nI believe the proposed method is novel. \n\n* clarity\nThe paper is well written and relatively easy to follow. \n\n* quality\nThe theoretical derivation of the method appears sound; the empirical results seem extensive though it would be interesting to understanding additional details which I have commented below.\n\nWEAKNESSES:\nThe presentation of the empirical results seems to be primarily focused on the efficiency and feasibility of converting the problem of solving for optimal equilibria in n-player general-sum EFGs to solving minimax equilibria in two-player zero-sum games. For instance Table 1 presented runtime results of the learning-based method compared to solving LP directly. \n\nWould it be possible to provide additional details on optimality as well especially in general-sum games such as Sheriff where we know the value of max-welfare equilibria (e.g. https://www.cs.cmu.edu/~gfarina/2020/efcce-aaai20/coarse-correlation.aaai20.pdf)? Would the proposed method scale up to these instances of sheriff game?\n\nQUESTIONS:\nI find the presentation relatively easy to follow. Some minor clarifying questions below: \n\nL42: \"... including information design, and solution concepts such as ....\". This statement appears several times but it is a bit confusing to follow. I can understand the statement after reading the paper in full but consider rephrasing it or splitting it up to make it more readable? \n\nL63 nit: clarify what $T$ stands for. \n\nL145: I find the discussion on the revelation principle interesting but it's not clear how general it would apply to games. Could you comment on this and if the set of benchmark games studied in this work would satisfy this property? \n\nL191: \"All players, ... are constrained to [act/play, missing] according to $d_i$...\" is mediator considered to be included in \"All players\" in this statement? If so, what does playing according to $d_i$ mean for the mediator? Is the mediator limited to deterministic policies in this two-player game? \n\nL192: \"If nature picked ..., the utility is ...\", --> \"the utility [to the mediator] is ...\"?\n\nLIMITATIONS:\nThe authors have discussed practical limitations of the method in snippets of the main text.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on addressing the challenge of computing optimal equilibria in extensive-form games. The authors introduce the revelation principle, which transforms the problem into a linear programming (LP) task. \n\nThey propose using Lagrange relaxations to solve the LP, treating the resulting saddle-point problem as a Nash equilibrium in a zero-sum two-player game. The authors explicitly construct such zero-sum game. To efficiently find the solution, the authors employ regret minimization over conic hulls. Additionally, they highlight the flexibility of their approach by showing that other algorithms for zero-sum two-player games can be utilized, offering different convergence rate guarantees or achieving last-iteration convergence. \n\nThe paper concludes with thorough experimental evaluations, demonstrating the effectiveness and scalability of their proposed methods across various simple games.\n\nSTRENGTHS:\n- The paper addresses the significant problem of computing optimal equilibria in game theory. By reducing this problem to zero-sum games, the authors provide a framework for finding optimal equilibria. The key contribution lies in the fact that existing algorithms for zero-sum extensive form games can be directly applied to this problem. This reduction greatly enhances the understanding and applicability of the proposed approach.\n\n- The experimental evaluation conducted by the authors is robust and contributes to the strength of their work. They provide evidence of the scalability of their methods through preliminary experiments. This not only showcases the effectiveness of the proposed techniques but also demonstrates their potential for real-world applications. The inclusion of experimental results further reinforces the practical relevance of the research and adds value to the overall paper.\n\nWEAKNESSES:\nMy only concern is about the revelation principle. The reduction in the paper relies on the revelation principle, which is a fundamental concept. However, it is not clear regarding the specific conditions under which a fixed pure strategy d_i exists for different players and within which games and equilibria this concept applies.  Additionally, how to effectively find such d_i strategies is also a problem.\n\nQUESTIONS:\n- See the above weakness.\n\n- What does 'optimal direct equilibria' mean in line 162. Moerover, if the revelation principle fails, there's no fixed pure strategy d_i. In this case, how to get a similar problem of (G).\n\n- In the experiment of auction design, to demonstrate scalability, is it possible to conduct an experiment if each bidder's valuation is sampled from uniform distribution in [0,1].\n\nLIMITATIONS:\nThe authors have partially addressed the limitations of their work, though there is space for improvement (see the section Weaknesses and Questions).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the computation of optimal equilibria in multi-player extensive-form games via no-regret learning algorithms. The key idea is to take the constrained LP formulation of the optimal equilibrium problem proposed by Zhang and Sandholm and consider the saddle point formulation, which can then be solved using no-regret learning algorithms. In order to alleviate the large $\\lambda^*$, a binary-search based algorithm is proposed. Finally, the algorithms are compared with a LP-based method on several tabular games as well as for an auction-design problem.\n\nSTRENGTHS:\n- The paper is well-written. The results presented clearly and the writing is easy to follow.\n\n- The experimental results seem compelling as it has an order of magnitude advantage in wall-clock time over the LP-based method. This advantage is consistent across a set of diverse instances.\n\nWEAKNESSES:\n- I have some concerns about the LP formulation (G). Consider a $N$ player normal form game with $A$ actions per player for instance; it seems that for the correlated equilibrium in such a game, the $\\mu$ in (G) would need to have dimension $A^N$. However, no swap-regret algorithms can find an approximate correlated equilibrium with $\\tilde{O}(NA^2/\\epsilon^2)$ samples. It seems that the LP formulation (G) can lead to suboptimal dependence on the number of players.\n\n- Perhaps related the previous point, while in Line 79 the authors claim that \"Our algorithm significantly outperforms existing LP-based methods\", in the experiments it is only compared against the LP based solution to (G) proposed by Zhagn and Sandholm. Is (G) the only known LP formulation of the optimal equilibrium problem? If not, wouldn't it make sense to compare against those as well?\n\nQUESTIONS:\nHave the authors tried gradient based methods (such as extra-gradient or gradient descent-ascent) on the saddle-point problem (L1)?\n\nLIMITATIONS:\nThis paper does not have foreseeable negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper presents a novel approach to compute optimal equilibria in multi-player extensive-form games through the use of Lagrangian relaxation as a two-player zero-sum extensive-form game. Building upon the mediator augmentation game framework, the proposed computing approach significantly contributes to the reduction of zero-sum games, thereby holding substantial implications for mechanism design and information design. Furthermore, it plays a pivotal role in fostering a well-balanced hierarchy of concepts.

STRENGTHS:
-- The investigation of computing minimax equilibria in extensive-form games is an interesting and well-motivated problem.

-- Most correlation equilibrium notions pose challenges in finding optimal equilibria, whereas the framework presented in this paper offers a learning-based algorithm to compute them.

WEAKNESSES:
-- The paper lies heavily on notation that lacks intuitive explanations. To enhance reader understanding, a schematic representation of the mediator-augmented game framework should be included in the main body. 

-- The approach proposed in this paper resembles a transformation of the original game into a mediator-led Stackerberg game. It would be beneficial to discuss and compare existing research on equilibrium solutions for zero-sum games in Stackelberg games to provide a contextual background for the paper's contribution. Lack controlled experiments involving additional algorithms for equilibrium solutions in Stackelberg games.

QUESTIONS:
Why does the space of messages can be restricted by using the revelation principle? I think more details on this are necessary in the paper.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08