Title: Is Learning in Games Good for the Learners?
Venue: NeurIPS 2023
Accepted: yes
Keywords: learning in games

correlated equilibria

Stackelberg equilibria

swap regret

dynamic regret
Abstract: We consider a number of questions related to tradeoffs between reward and regret in repeated gameplay between two agents. To facilitate this,  we introduce a notion of generalized equilibrium which allows for asymmetric regret constraints, and yields polytopes of feasible values for each agent and pair of regret constraints, where we show that any such equilibrium is reachable by a pair of algorithms which maintain their regret guarantees against arbitrary opponents. As a central example, we highlight the case one agent is no-swap and the other's regret is unconstrained. We show that this captures an extension of Stackelberg equilibria with a matching optimal value, and that there exists a wide class of games where a player can significantly increase their utility by deviating from a no-swap-regret algorithm against a no-swap learner (in fact, almost any game without pure Nash equilibria is of this form). Additionally, we make use of generalized equilibria to consider tradeoffs in terms of the opponent's algorithm choice. We give a tight characterization for the maximal reward obtainable against some no-regret learner, yet we also show a class of games in which this is bounded away from the value obtainable against the class of common "mean-based" no-regret algorithms. Finally, we consider the question of learning reward-optimal strategies via repeated play with a no-regret agent when the game is initially unknown. Again we show tradeoffs depending on the opponent's learning algorithm: the Stackelberg strategy is learnable in exponential time with any no-regret agent (and in polynomial time with any no-adaptive-regret agent) for any game where it is learnable via queries, and there are games where it is learnable in polynomial time against any no-swap-regret agent but requires exponential time against a mean-based no-regret agent.

Is Learning in Games Good for the Learners?

Abstract

We consider a number of questions related to tradeoffs between reward and regret in repeated gameplay between two agents. To facilitate this, we introduce a notion of generalized equilibrium which allows for asymmetric regret constraints, and yields polytopes of feasible values for each agent and pair of regret constraints, where we show that any such equilibrium is reachable by a pair of algorithms which maintain their regret guarantees against arbitrary opponents. As a central example, we highlight the case one agent is no-swap and the other's regret is unconstrained. We show that this captures an extension of Stackelberg equilibria with a matching optimal value, and that there exists a wide class of games where a player can significantly increase their utility by deviating from a no-swap-regret algorithm against a no-swap learner (in fact, almost any game without pure Nash equilibria is of this form). Additionally, we make use of generalized equilibria to consider tradeoffs in terms of the opponent's algorithm choice. We give a tight characterization for the maximal reward obtainable against some no-regret learner, yet we also show a class of games in which this is bounded away from the value obtainable against the class of common 'mean-based' no-regret algorithms. Finally, we consider the question of learning reward-optimal strategies via repeated play with a no-regret agent when the game is initially unknown. Again we show tradeoffs depending on the opponent's learning algorithm: the Stackelberg strategy is learnable in exponential time with any no-regret agent (and in polynomial time with any noadaptive -regret agent) for any game where it is learnable via queries, and there are games where it is learnable in polynomial time against any no-swapregret agent but requires exponential time against a mean-based no-regret agent.

1 Introduction

How should two rational agents play a repeated, possibly unknown, game against one another? One natural answer - barring any knowledge of the game, or the capacity to compute potentially computationally intractible equilibria - is that they should employ some sort of learning algorithm to learn how to play over time. Indeed, there is a vast literature which studies what happens when all the players in a repeated game run (some specific type of) learning algorithms to select their actions. For example, when all players in a game simultaneously run no-(swap)-regret learning algorithms, it is known that the average strategy profile of the learners converges to a (coarse) correlated equilibrium (23; 11; 16). More recent works have studied how to design algorithms that converge to these equilibria at faster rates (7; 1; 8), performance guarantees of such equilibria compared to the optimal possible welfare (4; 17), and the specific dynamics of such learning algorithms (9; 20; 22).

In contrast, relatively little attention has been devoted to whether it is actually in the interest of these agents to run these specific learning algorithms. For example, in a setting where all agents are running no-swap-regret learning algorithms, when can an agent significantly benefit by deviating and running a different type of algorithm? And if they can, what algorithm should the agent deviate to?

1.1 Our results

We explore the following questions (and others) in the case where two agents repeatedly play a normal-form, general-sum game for T rounds.

When does reward trade off with regret? While maximizing reward is often viewed as the objective of regret minimization against arbitrary adversaries, tensions may emerge when playing against another learning agent, as one's choice of actions has a causal effect on future loss functions as determined by the opponent's algorithm. In such cases, as we discuss further below, it turns out that demanding a stronger regret guarantee (e.g. asking for no-swap-regret instead of no-external-regret) may ultimately result in lower reward for an agent. To analyze these tradeoffs, in Section 2 we introduce a notion of generalized (Φ A , Φ B ) -equilibrium , where Φ A and Φ B are the sets of 'deviation strategies' over which players A and B minimize regret (e.g. fixed actions or swap functions), presented as an asymmetric extension of the linear Φ -equilibria considered by (13). Each pair of strategy sets (Φ A , Φ B ) generates a polytope of (Φ A , Φ B ) -equilibria, where any point then yields a reward value for each player. We show that all such points are feasible: for any game and any (Φ A , Φ B ) -equilibrium φ , there is a pair of algorithms which converge to φ while maintaining their respective Φ A -regret and Φ B -regret guarantees against arbitrary opponents. Further, deviating to a strategy with fewer constraints Φ A can often result in strictly improved reward for player A . As a concrete application, we consider the question of deviating from simultaneous no-swap-regret play.

When is no-swap-regret play a stable equilibrium? What should you do if you know that your opponent in a repeated game is running a no-swap-regret algorithm to select their actions? In (10), the authors show that one utility-optimizing response (up to additive o ( T ) factors) is to play a static (mixed) strategy (your Stackelberg strategy ) and obtain the Stackelberg value of the game. However, determining your Stackelberg strategy requires some knowledge of the game, and acquiring this knowledge from repeated play may be difficult (we address the latter issue in Section 5). In comparison, it is relatively straightforward to also run a no-swap-regret learning algorithm. This begs the question: are there games where you obtain significantly less utility (i.e., at least Ω( T ) less utility) by running a no-swap-regret learning algorithm instead of playing your Stackelberg strategy?

We show that the answer is yes , such games exist and are relatively common. In fact, we provide an efficient algorithmic characterization of the games G for which both players playing a no-swap-regret learning algorithm is an ( o ( T ) -)approximate Nash equilibrium of the entire repeated 'meta-game'. The exact characterization is presented in Section 3 and is somewhat subtle, e.g., there are slightly different characterizations depending on whether we insist all pairs of no-swap-regret algorithms lead to approximate equilibria or only one specific pair, corresponding to best-case and worst-case values in the generalized equilibrium polytope. One consequence of both characterizations, however, is that for almost all games (in a measure-theoretic sense, considering arbitrarily small perturbations), in order for it to be an approximate equilibrium for both players to play a low-swap regret strategy, the game G must possess a pure Nash equilibrium. That is, in any game without a pure Nash equilibrium, it is possible for at least one of the parties to do significantly better by switching from no-swap-regret learning to playing their Stackelberg strategy.

Finally, we additionally show there are some games where playing a no-(external)-regret algorithm against another no-swap-regret learner weakly dominates playing a no-swap-regret algorithm, regardless of the specific choice of algorithms. This counters the intuition that stronger regret guarantees protect a player from worse outcomes.

Optimizing reward against no-regret learners. What if our opponent is not running a no-swapregret algorithm, but simply a no-(external)-regret algorithm? In this case, it is still possible to obtain at least the Stackelberg value of the game by playing our Stackelberg strategy (no-regret algorithms are also guaranteed to eventually learn and play the best response to this strategy). However, unlike in the no-swap-regret setting, there exist specific games and no-regret algorithms where it is possible to obtain significantly ( Ω( T ) ) more than the Stackelberg value by playing a specific dynamic strategy. This phenomenon was first observed in (10), where a specific game is given for which it is possible to obtain Ω( T ) more than Stackelberg when playing against any no-regret algorithm in the family of mean-based learning algorithms (including algorithms such as multiplicative weights and EXP3). However, many questions remain unanswered, such as understanding in which games it is possible to outperform playing one's Stackelberg strategy, and by how much.

In Section 4 we present some answers to these questions for the case of generic no-regret algorithms. Specifically, we first show that if player B is running (any) no-regret algorithm, the utility of player A (regardless of what strategy they employ) is upper bounded by Val A ( ∅ , E ) · T + o ( T ) , where Val A ( ∅ , E ) is what we call the the unconstrained-external (equilibrium) value of the game for player A , which is given by the solution to a linear program. We then show that this upper bound is asymptotically tight: there exists a no-regret algorithm L such that if player B is playing according to L , then the player A can obtain utility at least Val A ( ∅ , E ) · T -o ( T ) by playing an appropriate strategy in response.

Note that this characterization does not completely resolve the question of (10) - it requires the construction of a fairly specific no-regret algorithm L , and it is still open what is possible against specific (classes of) no-regret algorithms (e.g., multiplicative weights or mean-based algorithms). In fact, we show a property of games which, when satisfied, implies that it is impossible to obtain the unconstrained-external value against a mean-based learner.

Learning the Stackelberg strategy through repeated play. Finally, we address the question of how hard it is to actually identify the Stackelberg strategy in a game against a learning opponent. Given full knowledge of the game G , finding an agent's Stackelberg strategy is simply a computational problem which turns out to be efficiently solvable by solving several small LPs (see (6)). However, if the game (in particular, the opponent's reward matrix) is unknown, an agent must learn the Stackelberg strategy over time. Existing work on learning Stackelberg strategies (e.g., (19; 24)) generally assumes access to a best-response oracle for the game (i.e., for a specific mixed strategy, how will an opponent best-respond?). In contrast, if our opponent is playing a specific no-regret learning algorithm, they may not immediately best respond to the strategies we play! This raises the following two questions. First, when is it possible to learn the Stackelberg equilibrium of a game while playing against a learning opponent? Second, is it easier to learn this equilibrium when playing against certain classes of learning algorithms?

In Section 5 we begin by showing that it is indeed possible to convert any best-response query algorithm for finding Stackelberg equilibria via best-response queries to an adaptive strategy that learns Stackelberg equilibria via repeated play against a generic no-regret learner, albeit potentially at the cost of an exponential blow-up in the number of rounds, e.g. for a query algorithm which makes Q best-response queries, simulating it against a no-regret learner may require T = exp( Q ) rounds of play. For the special case of opponents with noadaptive -regret algorithms (such as online gradient descent), we show that only T = poly ( Q ) rounds are required in the worst case. However, in general we show that exponential runtime can be necessary. In particular, we give an example of a game with M actions where it is possible to learn the Stackelberg equilibrium in poly ( M ) rounds when playing against any no-swap-regret learning algorithm, but where it requires at least exp( M ) rounds to learn this equilibrium when playing a mean-based no-regret algorithms.

1.2 Related work

The broader literature on no-regret learning in repeated games is substantial, covering many equilibrium convergence results varying assumptions. A recent line of work (5; 10; 21) considers problems related to optimizing one's reward when competing against a no-regret learner in a game. We extend these questions to consider the relationship and regret for an optimizer, as well as to settings where properties of the game are initially unknown, and give a series of separation results in terms of various notions of equilibrium. Also relevant is the literature on analysis of no-regret trajectory dynamics, and in particular (20) which shows a game in which no-regret dynamics outperform the reward of the Nash equilibrium. Additionally, there is also prior work considering regret minimization problems involving either best-responding or otherwise strategic agents (see e.g. (3; 18)), as well as work considering alternate regret notions or behavior models for repeated Stackelberg games (e.g. (12; 15)).

1.3 Notation and preliminaries

Throughout, we consider two-player bimatrix games G = ( A,B ) , where player A ('the optimizer') has action set A = { a 1 , . . . , a M } and player B ('the learner') has action set B = { b 1 , . . . , b N } . When the optimizer plays action a i and the learner plays action b j , the players receive rewards u A ( a i , b j ) and u B ( a i , b j ) , respectively. We assume that the magnitude of each utility is bounded by a constant. The sets of mixed strategies for each player are denoted by ∆( A ) and ∆( B ) , respectively;

when the optimizer plays a mixed strategy α ∈ ∆( A ) and the learner plays β ∈ ∆( B ) , the expected reward for the optimizer is given by u A ( α, β ) = ∑ M i =1 ∑ N j =1 α i β j u A ( a i , b j ) , with u B ( α, β ) defined analogously. An action b ∈ B is a best response to a strategy α ∈ ∆( A ) if b ∈ argmax b ′ ∈B u B ( α, b ′ ) . We let BR ( α ) be the set of all such actions for player B , and likewise BR ( β ) for player A .

2 Generalized equilibria and noΦ -regret learning

Here we introduce the notions of Φ -regret and generalized equilibria, which we use to analyze the regret and reward of players in repeated bimatrix games under varying assumptions regarding the choice of regret benchmarks, the algorithms used, and the structure of the game.

Originally introduced in (14) and extended to general convex settings in (13), we consider the formulation of linear Φ -regret as it relates to bimatrix games. Given a sequence of action pairs ( a i 1 b j 1 ) , . . . , ( a i T b j T ) for T > 0 and some set of functions Φ , where each f ∈ Φ maps actions A to action profiles in ∆( A ) , we say that the Φ -regret for the optimizer (and analogously for the learner) is

Definition 1 (NoΦ -regret learning) . We say a learning algorithm L is a noΦ -regret algorithm if, for some constant c < 1 , we have that Reg Φ ( L , T ) = O ( T c ) , where Reg Φ ( L , T ) is the Φ -regret corresponding to the action sequence played by L .

̸

Some notable sets of regret comparator functions Φ are the constant maps E (corresponding to external regret ), where all input actions are mapped to the same output action, and the 'swap functions' I (corresponding to internal regret 1 ), which contain all single swap maps f ij : [ M ] → [ M ] where f ( i ) = j and f ( i ′ ) = i ′ for i ′ = i . Imposing these constraints on players in a game results in a (coarse) correlated equilibrium , which are instances of our notion of generalized equilibrium .

Definition 2 (Generalized (Φ A , Φ B ) -equilibria) . A (Φ A , Φ B ) -equilibrium φ ∈ ∆( A × B ) in a two-player game is a joint distribution over action profiles ( a, b ) such that player A cannot increase their expected reward by deviating with some strategy in Φ A and player B cannot benefit by deviating with some strategy in Φ B .

In contrast to the Φ -equilibria considered by (14; 13), here we allow constraints to be asymmetric between players. While many equilibrium notions for two-player games impose symmetric regret constraints on each player (e.g. Nash, correlated, and coarse correlated equilibria), this need not always be the case. In Section 3, we highlight Stackelberg equilibria as a motivating example for considering more general notions of asymmetric equilibria from the perspective of Φ -regret, to determine when one should deviate from simultaneous no-swap play, and in Section 4 we characterize the maximum reward attainable against no-regret learners in terms of asymmetric equilibria.

We say that the value of a game G for player A of a certain equilibrium class (Φ A , Φ B ) , denoted Val A (Φ A , Φ B ) is the maximum reward obtainable by player A at some (Φ A , Φ B ) -equilibrium (with Val B (Φ A , Φ B ) defined symmetrically for player B ). Likewise, we say that the min-value of a game for a player and an equilibrium class, denoted by e.g. MinVal A (Φ A , Φ B ) for player A , is the minimum reward for a player over all (Φ A , Φ B ) -equilibria in a game. These capture the range of feasible average rewards under repeated play via (Φ A , Φ B ) -regret dynamics.

Proposition 1. For a repeated game over T rounds where player A uses a noΦ A -regret algorithm and player B uses a noΦ B -regret algorithm, the average rewards obtained by each player are upper bounded by Val A (Φ A , Φ B ) + o (1) and Val B (Φ A , Φ B ) + o (1) , respectively, and lower bounded by MinVal A (Φ A , Φ B ) -o (1) and MinVal B (Φ A , Φ B ) -o (1) .

We consider an ε -approximate (Φ A , Φ B ) -equilibrium to be a joint profile distribution where each constraint is satisfied up to additive error ε , connecting Definitions 1 and 2 as follows.

Proposition 2 (Convergence of noΦ -regret dynamics to generalized equilibrium) . Suppose after T rounds of a game where player A plays a noΦ A -regret algorithm and player B plays a noΦ B -regret algorithm, player A has average Φ A -regret ≤ ε and player B has average Φ B -regret ≤ ε . Let

φ t := p t A × p t B denote the joint distribution over both players' actions at time t and φ := 1 T ∑ T t =1 φ t denote the time-averaged history over joint player action distributions. Then, φ is an ε -approximate (Φ A , Φ B ) -equilibrium where

for every possible deviation f A ∈ Φ A , f B ∈ Φ B . Likewise, if players A and B repeatedly play strategies corresponding to an (Φ A , Φ B ) -equilibrium, then player A is noΦ A -regret and player B is noΦ B -regret.

A general construction for noΦ -regret algorithms is given in (13), which immediately implies feasibility of dynamics which converge to some instance of any class of (Φ A , Φ B ) -equilibria in a game, possibly requiring agents to use differing algorithms if constraints are asymmetric. We show that a much stronger claim is true: such a pair of algorithms exists for any (Φ A , Φ B ) -equilibrium Ψ in a game. These algorithms also satisfy a 'best-of-both-worlds' property, in that they converge to Ψ when played together, yet simultaneously maintain their corresponding regret guarantees against arbitrary adversaries.

Theorem 1. Consider any game G . Suppose there exists a noΦ A -regret learning algorithm L A and a noΦ B -regret learning algorithm L B . For any particular (Φ A , Φ B ) -equilibrium Ψ in a game G , there exists a pair of learning algorithms ( L ∗ A (Ψ) , L ∗ B (Ψ)) such that:

The empirical sequence of play when Player A uses L ∗ A (Ψ) and Player B uses L ∗ B (Ψ) converges to Ψ .

L ∗ A (Ψ) and L ∗ B (Ψ) are noΦ A -regret and noΦ B -regret, respectively, against arbitrary adversaries.

Our approach is for the algorithms to initially implement a schedule of strategies which converges to Ψ . Yet, these algorithms also detect when their opponent disobeys the schedule by tracking their Φ -regret with respect to Ψ , and after o ( T ) violations can deviate indefinitely to playing a standalone noΦ -algorithm for all remaining rounds. Several of our results throughout make use of Theorem 1. Here we state a notable immediate implication for equilibrium selection.

Corollary 1.1. For any equilibrium scoring function Γ : ∆( A×B ) → R with a unique optimum computable in finite time, there exists a pair of learning algorithms ( L ∗ A , L ∗ B ) such that:

The empirical distribution when player A uses L ∗ A and player B uses L ∗ B converges to argmax Ψ Γ(Ψ) .

L ∗ A and L ∗ B are noΦ A -regret and noΦ B -regret, respectively, against arbitrary adversaries.

Proof. First optimize Γ over Ψ in finite time to find the unique optimum; then apply Theorem 1 to the resulting desired equilibrium.

Corollary 1.1 allows for optimizing for objectives such as total welfare or min-max utility for both players, and imposing conditions on generalized equilibria beyond Φ -regret constraints (e.g. product constraints for Nash equilibria) by assigning arbitrarily low scores to invalid strategy profiles.

In subsequent sections, we will primarily focus the function classes E and I corresponding to external and internal regret as mentioned above, as the well empty set ∅ corresponding to unconstrained regret, and we additionally will consider the case when the game G is initially unknown. Before continuing, we note that each player's values for any (Φ A , Φ B ) -equilibrium class can be expressed via a linear program, whose size is polynomial in the game dimensions for these function classes of interest.

Proposition 3. For any game G and constraints (Φ A , Φ B ) , both Val A (Φ A , Φ B ) and Val B (Φ A , Φ B ) are computable via linear programs with MN variables and poly ( M,N, | Φ A | , | Φ B | ) constraints. When Φ A and Φ B belong to {∅ , E , I} , the number of constraints is poly ( M,N ) .

In general, these values for a player may differ under distinct notions of generalized equilibria; we give several examples of such concrete value separations in Appendix A (in Theorem 8). Our results in Section 3 illustrate a particularly stark separation of this form, in which it can often be dominant to deviate to a strategy where Φ -regret constraints are violated.

3 Stability of no-swap-regret play

Here we address the following question: when is it the case that for two players in a game, it is an approximate (Nash) equilibrium for both players to play no-swap-regret strategies? More specifically, imagine a 'metagame' where at the beginning of this repeated game, both players simultaneously announce and commit to a specific adaptive (and possibly randomized) algorithm they intend to run to select actions to play in the repeated game G for the next T rounds. In this metagame, for which games G is it an o ( T ) -approximate Nash equilibrium for both players to play a no-swap-regret learning algorithm?

Of course, the answer to this question might depend on which specific no-swap-regret learning algorithm the agents declare. We therefore attempt to understand the following two questions:

Necessity: For which games G is it the case that there exists some pair of no-swap-regret algorithms which form a o ( T ) -approximate Nash equilibrium? (Equivalently, when is it never the case that playing no-swap-regret algorithms forms an approximate Nash equilibrium?)

Sufficiency: For which games G is it the case that all pairs of no-swap regret algorithms form o ( T ) -approximate Nash equilibria?

A central element of our analysis will be to consider the Stackelberg equilibria of a game.

Definition 3 (Stackelberg Equilibria) . The Stackelberg equilibrium of a game G for player A is the pair of strategies ( α, b ) given by argmax α ∈ ∆( A ) , b ∈ BR ( α ) u A ( α, b ) , and the resulting expected utility for player A is the Stackelberg value of the game, denoted Stack A . Stack B is defined symmetrically.

We can relate Stackelberg equilibria to our notions of generalized equilibria.

Proposition 4. For any game G , we have that Stack A = Val A ( ∅ , I ) .

Here, any joint distribution over action profiles where player B has zero swap regret constitutes a ( ∅ , I ) -equilibrium for a game, and the optimal value for such an equilibrium for player A coincides with the Stackelberg value. Further, each equilibrium set can be optimized over via a linear program.

Note that each value definition allows for tiebreaking in favor of player A . In general, simply playing the Stackelberg strategy α may not suffice to obtain Stack A if the best response for Player B is not unique. However, there are a number of mild conditions which are each sufficient to ensure the existence of an approximate Stackelberg strategy α ′ which yields a unique best response for player B and obtains Stack A -ε for any ε > 0 . Here we consider a minimal such condition (essentially, no action is weakly dominated without also being strictly dominated).

Assumption 1. In a game G , for each b , either BR ( α ) = { b } for some α , or b / ∈ BR ( α ) for all α . Likewise, for each a , either BR ( β ) = { a } for some β , or a / ∈ BR ( β ) for all β .

We provide an efficient algorithmic procedure to answer both questions of necessity and sufficiency for a specific game G satisfying Assumption 1. To do this, recall that when two players both employ no-swap regret strategies, they asymptotically (time-average) converge to some correlated equilibrium (here, corresponding to an ( I , I ) -equilibrium). On the other hand, by defecting from playing a no-swap regret strategy (while the other player continues playing their no-swap regret strategy), a player can guarantee their Stackelberg value for the game. Moreover, as shown by (10), this is the optimal (up to o ( T ) additive factors) best response to an opponent running a no-swap regret strategy. It thus suffices to understand how the utility a player might receive under a correlated equilibrium compares to the utility they receive under their Stackelberg strategy. For a fixed game G , let Stack A = Val A ( ∅ , I ) be the Stackelberg value for the first player, and Stack B = Val B ( I , ∅ ) be the Stackelberg value for the second player. We have the following theorem.

Theorem 2. Fix a game G satisfying Assumption 1. The following two statements hold:

There exists some pair of no-swap-regret algorithms that form an o ( T ) -approximate Nash equilibrium in the metagame iff there exists a correlated equilibrium φ in G such that u A ( φ ) = Stack A and u B ( φ ) = Stack B .

Any pair of no-swap-regret algorithms form an o ( T ) -approximate Nash equilibrium in the metagame iff for all correlated equilibria φ in G , u A ( φ ) = Stack A and u B ( φ ) = Stack B .

Moreover, given a game G , it is possible to efficiently (in polynomial time in the size of G ) check whether each of the above cases holds.

We obtain both claims by leveraging the construction in Theorem 1: best-case and worst-case correlated equilibria are feasible by some pair of no-swap-regret algorithms, and both players must simultaneously achieve close to their Stackelberg value for deviating to not be preferable. The characterization in Theorem 2 is algorithmically useful, but sheds little direct light on in which games or how often we would expect playing no-swap-regret to be an approximate equilibrium. It turns out that for many games, playing no-swap-regret is not an equilibrium; below we will show that for almost all games, if G does not have a pure Nash equilibrium, at least one player has an incentive to deviate to their Stackelberg strategy.

Definition 4. A property P of a game holds for almost all games if, given any game G , property P holds with probability 1 for the game G ′ formed by starting with G and perturbing each of the entries u A ( a i , b j ) and u B ( a i , b j ) by independent uniform random variables in the range [ -ε, ε ] (for any choice of ε ). In other words, the property holds for almost all choices of the 2 MN utility values that define a game (with respect to the standard measure on this space).

Theorem 3. For almost all games G , if G does not have a pure Nash equilibrium, then there does not exist a pair of no-swap-regret algorithms which form a o ( T ) -approximate Nash equilibrium in the metagame for G .

Sketch. We can show that if a correlated equilibrium has the same utility for a player as their Stackelberg value (a consequence of Theorem 2), then the correlated equilibrium must be a convex combination of valid Stackelberg equilibria. In almost all games, both players have unique Stackelberg equilibria (and Assumption 1 holds), which implies that this correlated equilibrium must actually be the Stackelberg strategy for both players simultaneously. This implies that it is a pure Nash equilibrium (since one action in a generic Stackelberg equilibrium is always pure).

Note that although Theorem 3 holds for almost all games, there are some important classes of games (most notably, zero-sum games) in the measure zero subset omitted by this theorem statement that both a) do not have pure Nash equilibria and b) have the property that playing no-swap-regret algorithms is an approximate equilibrium in the metagame (in particular, for zero-sum games, the Stackelberg value collapses to the value of the unique Nash equilibrium). Still, Theorem 3 shows that there are very wide classes of games for which playing no-swap-regret algorithms is not stable from the perspective of the agents.

Finally Theorem 3 requires that we deviate to our Stackelberg strategy, which may be hard to compute. One can ask whether there are games where efficient deviations - e.g., to algorithms with weaker regret guarantees - lead to strictly more utility for the deviating player. In Appendix B we show this is true in the following sense: there are games G where Val A ( E , I ) > MinVal A ( E , I ) = Val B ( I , I ) . That is, in such a game player A can possibly strictly increase their utility by switching to a lowexternal-regret strategy, and such a switch will never decrease their utility.

4 Optimal rewards against no-regret learners

Here, we characterize the feasibility of optimizing one's reward against no-(external)-regret learners in terms of generalized equilibria. In contrast to the case of no-swap-regret learners, as shown by (10) there are games in which one can obtain Ω( T ) more than the Stackelberg value over T rounds against certain no-regret algorithms by playing an appropriate adaptive strategy. A major remaining open question from this line of work is determining the best feasible reward and corresponding optimal strategy against no-regret agents in arbitrary games. We resolve this question when considering the maximum over all possible no-regret algorithms: for any game, we can compute an upper bound on the feasible reward against any no-regret algorithm, and we show that there exists a specific no-regret algorithm against which we can obtain this reward via an efficiently implementable strategy.

Theorem 4. For any game G , there exists a no-regret algorithm L and a strategy for player A such that the total reward of player A converges to Val A ( ∅ , E ) · T ± o ( T ) when player B uses L .

Here we again make use of the construction from Theorem 1. Note that by Proposition 1, this is optimal over all no-external-regret algorithms, as any adaptive strategy constitutes a no-∅ -regret

algorithm. By Proposition 3 we can identify the optimal ( ∅ , E ) -equilibrium in poly ( M,N ) time, which is sufficient to implement the algorithm L as well as our own strategy efficiently.

However, we additionally show that this bound is often unattainable against many standard noexternal-regret algorithms. A property of many such algorithms (including Multiplicative Weights, Follow the Perturbed Leader, and Exp-3) is that they are mean-based , as formulated by (5).

Definition 5 (Mean-based learning) . Let σ i,t be the cumulative reward resulting from playing action i for the first t rounds. An algorithm L is γ -mean-based if, whenever σ i,t ≤ σ j,t -γT , the probability that the algorithm selects action i in round t +1 is at most γ , for some γ = o (1) .

These algorithms resemble 'smoothed' variants of Follow the Leader; they only play actions with probability higher than o (1) if their cumulative reward thus far is not too far from optimal, and hence never play dominated strategies. However, in general, ( ∅ , E ) -equilibria may contain dominated strategies, as is also the case for coarse correlated equilibria. This allows us to show the following.

Theorem 5. Against any mean-based no-regret algorithm for player B , there are games where a T -round reward of ( Val A ( ∅ , I ) + ε ) · T cannot be reached by any adaptive strategy for player A , for any ϵ > 0 . However, for this same game, Val A ( ∅ , E ) > Val A ( ∅ , I ) .

In the appendix, we introduce a notion of 'dominated-swapping external regret' which we use to characterize a class of games for which this holds, and we give a concrete example of such a game.

5 Learning Stackelberg equilibria in unknown games

Our results thus far have highlighted the primacy of the Stackelberg reward as an objective for repeated play against a learner: it is optimal against a no-swap learner and can sometimes be optimal against a mean-based learner, and it is almost always attainable against any learner. However, until now our strategies have assumed knowledge of the entire game, which may be unrealistic in many settings for which learning in games is relevant, particularly in terms of our opponent's rewards.

Here, we consider the challenge of learning the Stackelberg strategy via repeated play against a no-regret learner when only our own rewards are known, which is unaddressed in the literature to our knowledge; much of the prior work on learning Stackelberg equilibria assumes a query model, where one can observe the best response BR ( α ) played by an opponent for any queried mixed strategy α . While here we cannot immediately observe the best response of an opponent, as their actions are selected by a learning algorithm which may be slow to adapt to changes in our behavior, we give a reduction from query algorithms of this form to strategies for choosing our actions which enable us to simulate queries to BR ( α ) against a learner, and we analyze the efficiency of this approach (in terms of rounds required for learning) under differing assumptions on the learner's algorithm.

For comparison of behavior across time horizons of varying lengths, it will be convenient for us to consider the notion of an anytime regret bound, which can be obtained from any base no-regret algorithm via doubling methods, as well as often via learning rate decay.

Definition 6 (Anytime regret algorithms) . An algorithm is an anytime noΦ -regret algorithm if satisfies Reg Φ ( t ) = O ( t c ) over the first t rounds, for some c < 1 and any t ≤ T .

We also recall the notion of adaptive regret; many no-external-regret algorithms such as Online Gradient Descent satisfy no-adaptive-regret bounds (see e.g. (25)).

Definition 7 (Adaptive regret algorithms) . An algorithm L for player B is a no-adaptiveΦ -regret algorithm if sup r,s ∈ [ T ] Reg Φ ( L , [ r, s ]) ≤ O ( T c ) , for some c < 1 , where Reg Φ ( L , [ r, s ]) = max f ∈ Φ ∑ s t = r u B ( a i t , f ( b j t )) -u B ( a i t , b j t ) .

A key distinction between adaptive-regret algorithms like OGD and mean-based algorithms like FTPL is in in their 'forgetfulness', and hence their ability to quickly adapt when rewards change. This has stark implications for the efficiency of learning Stackelberg equilibria, which we show can take exponentially longer against mean-based algorithms. As shown by (25), adaptive regret is closely connected with dynamic regret; we note that our results for adaptive-regret learners can also be extended to dynamic-regret learners.

5.1 Simulating query algorithms

Our approach will be to compute an ε -approximate Stackelberg strategy by simulating best response queries against a learner, after which point we can obtain an average reward approaching Stack A -ε in each subsequent round, calibrating ε in terms of T as desired. The query complexity for such algorithms can depend on the geometry of the best response regions of the game, and unfortunately, as shown by (24), there are 'hard' game instances which require exponentially many queries. This issue arises when the best response regions may be quite small but non-empty, as even finding a point in each region is information-theoretically difficult. We restrict our attention to games in which this does not occur, and fortunately efficient query do algorithms exist for this case.

Assumption 2. For a game G and any action b i , we have that

i.e. the volume of each BR region is either 0 or inverse polynomially large.

Proposition 5 ((19; 24)) . For a game G satisfying Assumption 2, there is an algorithm which finds an ε -approximate Stackelberg strategy for player A with Q = poly ( M,N, 1 /ε ) queries to BR ( α ) .

We note that while such algorithms can indeed obtain tighter approximation guarantees in terms of ε (e.g. O (log(1 /ε )) ), the query complexity is still inverse polynomially related to the best response region volumes; we consider only ε -approximate equilibria due to challenges which are inherent to the no-regret learning setting, as the precision with which we can simulate a query is constrained by our time horizon. The key to our approach is to play according to a mixed strategy α until it saturates the relevant window of the learner's history, which induces them to play a best response. Against no-adaptive-regret learners, a best response will be induced quickly, as their regret is bounded even over small windows. However, for arbitrary no-regret learners, we have no promises other than the cumulative regret bound, which may require saturating the entire history for each query.

Theorem 6. Suppose E ⊆ Φ . For a game satisfying Assumption 2, there is an algorithm which finds an ε -approximate Stackelberg strategy in poly (1 /ε ) Q rounds against any anytime-noΦ -regret learner, and in poly ( Q/ε ) rounds against any no-adaptiveΦ -regret calibrated for T = Θ( poly ( Q/ε )) , where Q = poly ( M,N, 1 /ε ) .

5.2 Efficiency separations for mean-based and no-swap algorithms

We show here that the exponential dependence for mean-based algorithms is necessary: there exist games where learning the Stackelberg strategy requires exponentially many rounds against a particular mean-based algorithm. However, for the games we construct, we show that it is still possible to efficiently learn the Stackelberg strategy against a noswap -regret learner.

Theorem 7. There is a distribution over games D such that for a sampled game G :

For any no-swap-regret learner used by the opponent, there is a strategy for the leader which yields an average reward of Stack A -ε in T = poly ( M/ε ) rounds.

There is a mean-based no-regret algorithm such that, when used by the opponent, there is no strategy for the leader which yields an average reward of Stack A -ε over T rounds unless T = exp(Ω( M )) .

Our construction includes a set of actions for player B which are best responses to pure actions from player A , and one such pure strategy pair will necessarily constitute the Stackelberg equilibrium; identifying each best response suffices for player A to identify the Stackelberg strategy. The game also includes a number of safety actions for player B , which yield no reward for player A with any strategy, yet allow player B to 'hedge' between multiple actions of player A . This poses a barrier to optimizing against a mean-based learner: the history must be heavily concentrated on a single action to observe the best response, and as such the history length must grow by a constant factor for each observation. However, against a no-swap-regret learner, it suffices for the optimizer to only play each action for a polynomially long window in order to identify the learner's best response; we track the accumulation of a 'swap-regret buffer' for any other action and show that it cannot be too large, limiting the number of rounds it can be played when it is not a current best response.

References

Ioannis Anagnostides, Constantinos Daskalakis, Gabriele Farina, Maxwell Fishelson, Noah Golowich, and Tuomas Sandholm. Near-optimal no-regret learning for correlated equilibria in multi-player general-sum games. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 736-749, 2022.

Sanjeev Arora, Elad Hazan, and Satyen Kale. The multiplicative weights update method: a meta-algorithm and applications. Theory of Computing , 8(6):121-164, 2012.

Maria-Florina Balcan, Avrim Blum, Nika Haghtalab, and Ariel D. Procaccia. Commitment without regrets: Online learning in stackelberg security games. In Proceedings of the Sixteenth ACM Conference on Economics and Computation , EC '15, page 61-78, New York, NY, USA, 2015. Association for Computing Machinery.

Avrim Blum, MohammadTaghi Hajiaghayi, Katrina Ligett, and Aaron Roth. Regret minimization and the price of total anarchy. In Proceedings of the fortieth annual ACM symposium on Theory of computing , pages 373-382, 2008.

Mark Braverman, Jieming Mao, Jon Schneider, and Matt Weinberg. Selling to a no-regret buyer. In Proceedings of the 2018 ACM Conference on Economics and Computation , pages 523-538, 2018.

Vincent Conitzer and Tuomas Sandholm. Computing the optimal strategy to commit to. In Proceedings of the 7th ACM Conference on Electronic Commerce , EC '06, page 82-90, New York, NY, USA, 2006. Association for Computing Machinery.

Constantinos Daskalakis, Alan Deckelbaum, and Anthony Kim. Near-optimal no-regret algorithms for zero-sum games. In Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete Algorithms , pages 235-254. SIAM, 2011.

Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. Near-optimal no-regret learning in general games. Advances in Neural Information Processing Systems , 34:2760427616, 2021.

Constantinos Daskalakis, Rafael M Frongillo, Christos H Papadimitriou, George Pierrakos, and Gregory Valiant. On learning algorithms for nash equilibria. In SAGT , pages 114-125. Springer, 2010.

Yuan Deng, Jon Schneider, and Balasubramanian Sivan. Strategizing against no-regret learners. Advances in Neural Information Processing Systems , 32, 2019.

Dean P Foster and Rakesh V Vohra. Asymptotic calibration. Biometrika , 85(2):379-390, 1998.

Denizalp Goktas, Jiayi Zhao, and Amy Greenwald. Robust no-regret learning in min-max stackelberg games, 2022.

Geoffrey J. Gordon, Amy Greenwald, and Casey Marks. No-regret learning in convex games. In Proceedings of the 25th International Conference on Machine Learning , ICML '08, page 360-367, New York, NY, USA, 2008. Association for Computing Machinery.

Amy Greenwald and Amir Jafari. A general class of no-regret learning algorithms and gametheoretic equilibria. In Bernhard Schölkopf and Manfred K. Warmuth, editors, Learning Theory and Kernel Machines , pages 2-12, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg.

Nika Haghtalab, Thodoris Lykouris, Sloan Nietert, and Alex Wei. Learning in stackelberg games with non-myopic agents, 2022.

Sergiu Hart and Andreu Mas-Colell. A simple adaptive procedure leading to correlated equilibrium. Econometrica , 68(5):1127-1150, 2000.

Jason Hartline, Vasilis Syrgkanis, and Eva Tardos. No-regret learning in bayesian games. Advances in Neural Information Processing Systems , 28, 2015.

Zhiyi Huang, Jinyan Liu, and Xiangning Wang. Learning optimal reserve price against nonmyopic bidders. CoRR , abs/1804.11060, 2018.

Joshua Letchford, Vincent Conitzer, and Kamesh Munagala. Learning and approximating the optimal strategy to commit to. In Algorithmic Game Theory , 2009.

Katrina Ligett and Georgios Piliouras. Beating the best nash without regret. SIGecom Exchanges , 10(1):23-26, 2011.

Yishay Mansour, Mehryar Mohri, Jon Schneider, and Balasubramanian Sivan. Strategizing against learners in bayesian games. In Conference on Learning Theory , pages 5221-5252. PMLR, 2022.

Panayotis Mertikopoulos, Christos Papadimitriou, and Georgios Piliouras. Cycles in adversarial regularized learning. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 2703-2717. SIAM, 2018.

Hervé Moulin and J P Vial. Strategically zero-sum games: the class of games whose completely mixed equilibria cannot be improved upon. International Journal of Game Theory , 7:201-221, 1978.

Binghui Peng, Weiran Shen, Pingzhong Tang, and Song Zuo. Learning optimal strategies to commit to. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence , AAAI'19/IAAI'19/EAAI'19. AAAI Press, 2019.

Lijun Zhang, Tianbao Yang, Rong Jin, and Zhi-Hua Zhou. Dynamic regret of strongly adaptive methods. In International Conference on Machine Learning , 2017.

A Properties and separations for generalized equilibria

A.1 Proof of Proposition 1

Proof. The set of (Φ A , Φ B ) -equilibria includes all strategy profile distributions in which both constraints are satisfied. If a player receives substantially more or less than the corresponding value, this would imply a violation of the regret constraints for at least one of the players' learning algorithms.

A.2 Proof of Proposition 2

Proof. The statement follows by observing that

which in turn are equivalent to the time-averaged utility of the play of players A and B , the timeaveraged utility for player A under a deviation f A , and the time-averaged utility for player B under a deviation f B . Applying the definition of average Φ -regret and applying the given bounds on the Φ -regret yields the conclusion of the first direction. The reverse direction follows by reversing the steps.

A.3 Proof of Proposition 4

Proof. Observe that under any strategy ( α, b ) where b ∈ BR ( α ) , player B cannot have any swapregret, and so any Stackelberg equilibrium is also a ( ∅ , I ) -equilibrium. Further, the marginal distributions over the optimal ( ∅ , I ) -equilibrium for player A over each b i cannot have distinct expected value for player A , as otherwise this would contradict optimality, and so an optimal ( ∅ , I ) -equilibrium is either a single Stackelberg equilibrium or a mixture of Stackelberg equilibria with equal value.

A.4 Proof of Proposition 3

Proof. By definition, the set of (Φ A , Φ B ) -equilibria φ is a sub-polytope of ∆( A×B ) defined via the following linear constraints:

For each f A ∈ Φ A , we have that

For each f B ∈ Φ B , we have that

The value Val A (Φ A , Φ B ) corresponds to the element φ of this polytope that maximizes ∑ i ∈ [ M ] ∑ j ∈ [ N ] φ ij u A ( a i , b j ) . Optimizing this linear function over the above polytope can be done in time poly ( M,N, | Φ A | , | Φ B | ) via any linear program solver. Computing Val B (Φ A , Φ B ) can be likewise done efficiently.

For player A , the regret comparator function sets ∅ , E , and I contain 0 , M , and M 2 elements respectively. In all three of these cases | Φ A | = poly ( M ) ; likewise, in all three of these cases | Φ B | = poly ( N ) (and thus we can efficiently compute these values when Φ A , Φ B ∈ {∅ , E , I} ).

A.5 Reward separations

We show that with respect to optimal values, these equilibrium classes are often distinct, and there exist games where values do not collapse. The separations we show here consider the equilibrium cases either where both players have identical regret constraints, or where player A is unconstrained. We note that while inspecting other cases, we identified similar examples for several other generalized equilibrium pairs, and we expect that strict separations exist between any distinct pair of generalized equilibria for the three regret notions we consider, in any direction not immediately precluded by the regret constraints. We are mostly interested in cases where B is constrained, and A may be constrained or unconstrained.

Theorem 8. For each of the following, there exists a 4 × 4 game G with rewards in { 0 , 1 , 2 } where:

Val A ( ∅ , E ) > Val A ( E , E ) > Val A ( ∅ , I ) > Val A ( I , I )

Proof. We prove both results by exhibiting a game with the desired chain of inequalities, which we found by searching random examples of 4 × 4 games with values constrained in { 0 , 1 , 2 } and computing the various values of the games with a linear programming library. The numerical values are easy to check with computation. The game G 1 := ( M A 1 , M B 1 ) satisfies the conditions for the first chain of inequalities, and the game G 2 := ( M A 2 , M B 2 ) satisfies the conditions for the second chain of inequalities. First we instantiate the game G 1 :

The corresponding values for game G 1 are simple to check:

Val A ( ∅ , E ) = 8 / 5 .

Val A ( ∅ , I ) = 4 / 3 .

Val A ( E , E ) = 1 .

Val A ( I , I ) = 0 .

Then we instantiate the game G 2 :

The corresponding values for game G 2 are simple to check:

Val A ( ∅ , E ) = 13 / 7 .

Val A ( E , E ) = 12 / 7 .

Val A ( ∅ , I ) = 5 / 3 .

Val A ( I , I ) = 4 / 3 .

B Deviation to weaker regret classes

In Section 3, we show that if two players are playing no-swap-regret strategies against one another, it is often in the interest of each player to switch to playing their Stackelberg strategy (in particular, this is true whenever the game does not have a pure Nash equilibrium). However, as we later argue, learning ones Stackelberg strategy in such a game can be difficult. It is therefore natural to ask whether there are beneficial deviations to computationally efficient strategies. In particular, is it ever in a player's interest to weaken their regret benchmark, and e.g. switch from playing a no-swap-regret strategy to a no-external-regret strategy?

We give an example showing this is true in a fairly strong sense: we exhibit a game G where if player A switches from playing a no-swap-regret algorithm to any no-external-regret algorithm, their asymptotic utility never decreases and sometimes strictly increases - i.e., there is no downside to switching to an external regret algorithm (and potentially a high upside). We have the following theorem.

Theorem 9. There exists a game G where MinVal A ( E , I ) ≥ Val A ( I , I ) and Val A ( E , I ) ≥ Val A ( I , I ) .

Proof. Consider the game G specified by the two payoff matrices

The corresponding values for this game are simple to compute:

Val A ( I , I ) = MinVal A ( I , I ) = 0 .

MinVal A ( E , I ) = 0 .

Val A ( E , I ) = 1 .

C Proof of Theorem 1

Proof. Let φ be the joint distribution over action pairs corresponding to Ψ . Let T denote the total number of steps we run the algorithm for; we will use t ≤ T as a changing step size. Suppose both player A and player B know φ 2 . We will define L ∗ A (Ψ) and L ∗ B (Ψ) in two phases: in the first phase, A and B trust their opponent and play according to deterministic sequences corresponding to approximations of φ . If either player violates the other's trust o ( T ) times, then the player defects to playing L A or L B respectively forever after.

First we elaborate upon the trusting phase. Both players consider windows of length Length ( t ) which is monotonically increasing in t and also which grows sub-linearly in t . For concreteness, we pick a sub-linear monotonic increasing growth rate of O ( √ t ) and describe how to implement the schedule of window lengths. We can keep track of a real-valued variable Z t with Z 1 = M · N , and after each window completes, update it by Z t next = Z t + 1 2 √ t where t is the step at the end of the window. To get an integral window length, we define Length ( t ) := ⌊ Z t ⌋ . Thus in this case, the Length ( t ) grows as O ( √ t ) , satisfying both conditions. Both players then compute a weighting instantiated with pairs of pure strategies by assigning c i := ⌊ Length ( t ) · φ i ⌋ example pairs (each of weight 1 / Length ( t ) ) to pure strategy pair i ∈ [ M · N ] . This weighted distribution approximates φ given Length ( t ) samples. Note that the rounding approximation is feasible given only Length ( t ) samples since ∑ M · N i =1 c i ≤ Length ( t ) . These pure strategy pair samples are then lexicographically

ordered. Then, both players act according to the pure strategies in order, thereby (over the window) achieving an ( M · N ) / Length ( t ) ℓ 1 approximation to φ :

This process repeats for every window.

The distrustful phase occurs if one of the players does not follow the agreed-upon instructions T distrust times, where T distrust is taken to be o ( T ) . After this many violations, Player A defaults to playing L A and likewise Player B defaults to playing L B ever after.

We now show that this algorithm satisfies both conditions in the theorem statement. First, if both players use L ∗ A (Ψ) and L ∗ B (Ψ) , the play converges to φ , the joint distribution of play corresponding to Ψ . This point is immediate to observe since ( M · N ) / Length ( t ) → 0 as t →∞ as Length ( t ) is monotone increasing in t .

Now we prove that both players are noΦ -regret with respect to any adversary. First we show noΦ -regret for both players in the case where Player A plays L ∗ A (Ψ) and Player B plays L ∗ B (Ψ) . Let ˆ φ t be the approximation to φ implemented over the window corresponding to final step t , and suppose that ∥ φ -ˆ φ t ∥ 1 < ε t . Recalling the proof of Theorem 1, for Player A (and analogously for Player B ) we can bound

where here we interpret φ, ˆ φ t , u A , u B ∈ R M × N as vectors over the space of all action pairs. Thus for this particular window, the overall gap from the expected reward for φ is ε t · C · √ M · N .

Then we can similarly upper bound E ( a,b ) ∼ ˆ φ t [ u A ( f A ( a ) , b )] ≤ E ( a,b ) ∼ φ [ u A ( f A ( a ) , b )] + ε t · C · M √ N for any choice of f A ∈ Φ A :

Then recall that ε t ≤ M · N Length ( t ) . Thus, overall, the average regret using due to the window is bounded by

where C 2 is another constant depending on C, M, N and where we use the shorthand Reg Φ ( · , t ) to denote the Φ -regret over the window ending in step t . Now call ˆ φ the strategy where the joint distribution ˆ φ t as previously defined gets played in each window t . Now we can bound the total Φ -regret for ˆ φ by the sum of the Φ -regrets for each window (maximizing f A ∈ Φ A over the steps in each window makes it more competitive than optimizing only one f A over the whole length T sequence). Thus for total Φ -regret, we have:

where

The last step follows since NumWindows ( T ) ≤ o ( T ) , because Length ( T ) ≤ o ( T ) .

Since we already know that the strategy φ is noΦ -regret and Length ( T ) is o ( T ) , we have proven that playing ˆ φ is noΦ -regret in the case where Player A plays L ∗ A (Ψ) and Player B plays L ∗ B (Ψ) .

The second case where the opposing player does not cooperate is easier: after at most o ( T ) steps, the player switches to an algorithm L A or L B respectively which is noΦ -regret and incurrs only o ( T ) additional regret. Thus the theorem statement holds.

D Proof of Theorem 2

Proof. We begin with the first claim. To prove the forward direction, if there exists such a φ , then choose a pair of low-swap-regret algorithms ( L A , L B ) such that the time-averaged trajectory over T rounds is guaranteed to asymptotically converge to φ (this is possible by either the results of (11), or our Theorem 1). That is, if the two players play strategy φ t at round t ∈ [ T ] , then ˆ φ = 1 T ∑ t φ t satisfies || ˆ φ -φ || ∞ = o (1) . It follows that ∑ t u A ( φ t ) ≥ T · u A ( φ ) -o ( T ) = T · Stack A -o ( T ) and therefore player A has at most an o ( T ) incentive to deviate (by (10), they can obtain at most Stack A T + o ( T ) against L B ). Symmetric logic holds for player B .

To prove the reverse direction, assume L A and L B are no-swap-regret algorithms such that ( L A , L B ) is an o ( T ) -approximate Nash equilibrium in the metagame. Since they are no-swap-regret, the time-averaged play of these two algorithms for T rounds must converge to an o (1) -approximate correlated equilibrium ˆ φ T ; moreover, since ( L A , L B ) is an o ( T ) -approximate Nash equilibrium, ˆ φ T must have the property that u A (ˆ φ T ) ≥ Stack A -o (1) and u B (ˆ φ T ) ≥ Stack B -o (1) . Taking the limit as T →∞ and selecting a convergent subsequence of the ˆ φ T , this shows there must exist a correlated equilibrium φ with the desired properties.

Likewise, similar logic proves the second claim with the following modifications. In the forward direction, we can now choose any pair of low-swap-regret algorithms ( L A , L B ) , and any correlated equilibrium φ they asymptotically converge to is guaranteed to have the property that u A ( φ ) = Stack A and u B ( φ ) = Stack B . In the reverse direction, since any correlated equilibrium is implementable by some pair of low-regret algorithms (again, by Theorem 1), the same logic shows that all correlated equilibria φ must satisfy u A ( φ ) = Stack A and u B ( φ ) = Stack B .

Finally, to see that these two conditions are efficiently checkable, note that: i. the two values Stack A and Stack B are efficiently computable given the game G , and ii. the set of correlated equilibria φ form a convex polytope defined by a small (poly ( N,M ) ) number of linear constraints (see Proposition 3). In particular, since u A ( φ ) and u B ( φ ) are simply linear functions of φ for a given game G , we can efficiently check whether there exists any point in this polytope where u A ( φ ) = Stack A and u B ( φ ) = Stack B .

E Proof of Theorem 3

Proof. We will show that (for almost all games G ) if there is a correlated equilibrium φ such that u A ( φ ) = Stack A and u B ( φ ) = Stack B , then there exists a simultaneous unique Stackelberg equilibrium for both players in G , which must be a pure Nash equilibrium. Combined with Theorem 2, this implies the theorem statement.

We will rely on the following fact: in almost all games G , both players have a unique Stackelberg strategy. To see this, consider the following method for computing A 's Stackelberg strategy. For each pure strategy b j for player B , consider the convex set A j ⊆ ∆( A ) containing the mixed strategies for player A which induce b j as a best response (i.e., A j = { α ∈ ∆( A ) | b j ∈ BR ( α ) } ). Then, for each j ∈ [ N ] , compute the strategy α j ∈ A j which maximizes u A ( α j , b j ) . The Stackelberg value Stack A is then given by max j u A ( α j , b j ) . In order for this to stem from a unique Stackelberg equilibrium, it is enough that: 1. the maximum utility is not attained by more than one j , and 2. for each j , the optimizer α j ∈ A j is unique.

These two properties are guaranteed to hold in almost all games. To see this, first note that the convex sets A j are determined entirely by the utilities u B , so we will treat these as fixed. Now, given any convex set A j , the extremal point in a randomly perturbed direction will be unique with probability 1 - but since α j is simply the extremal point of A j in the direction specified by u A ( · , b j ) (which is a randomly perturbed direction), so α j is unique in almost all games. Finally, if we perturb the magnitude of each of the utilities u A ( · , b j ) (keeping the direction the same), the maximizer max j u A ( α j , b j ) will also be unique almost surely.

Let ( α A , b A ) be the Stackelberg equilibrium for player A and let ( a B , β B ) be the Stackelberg equilibrium for player B . Now, consider the aforementioned correlated equilibrium φ ∈ ∆( A × B ) . We will begin by decomposing it into its marginals based on its first coordinate; that is, we will write φ = ∑ M i =1 λ i ( a i , β i ) for some mixed strategies β i ∈ ∆( B ) and weights λ i (with ∑ i λ i = 1 ). By the definition of correlated equilibria, note that each a i belongs to BR ( β i ) . But this means that u B ( a i , β i ) ≤ Stack B , with equality holding iff ( a i , β i ) = ( a B , β B ) (due to uniqueness of Stackelberg). Therefore, in order for u B ( φ ) = Stack B , we must have that φ = ( a B , β B ) . By symmetry, we must also have that φ = ( α A , b A ) . If both these are true, then φ is a pure strategy correlated equilibrium of the game, and is hence a pure strategy Nash equilibrium (and moreover, is also the Stackelberg equilibrium for both A and B ).

F Proof of Theorem 4

Proof. By Theorem 1, there is a pair of ∅ -regret and E -regret algorithms L ∗ A and L ∗ B which converge to a ( ∅ , E ) -equilibrium for which player A obtains Val A ( ∅ , E ) . By Proposition 1, this is optimal over all no-external-regret algorithms, as any adaptive strategy constitutes a no-∅ -regret algorithm. By Proposition 3 we can identify the optimal ( ∅ , E ) -equilibrium in poly ( M,N ) time, which is sufficient to implement the algorithms L ∗ A and L ∗ B efficiently for any desired T .

G Dominated-swapping external regret bounds for mean-based algorithms

For the following proof (of Theorem 10), we introduce the following notion of dominated-swapping external regret , a tighter upper bound on the behavior of mean-based algorithms than the standard no-external-regret guarantee.

Definition 8 (Dominated-swapping external regret) . For a game G , let D ( G ) be the set of dominated strategies for player B , i.e. b i ∈ D ( G ) if b i / ∈ BR ( α ) for all α ∈ ∆( M ) . For j, k ∈ [ N ] define g jk ( b i ) as:

i.e. g jk ( b i ) swaps b i to b k if b i is dominated and plays b j otherwise. Let E D ( G ) = { g jk : j, k ∈ [ N ] } be the set of dominated-swapping external regret comparators.

This definition leads to the following tighter upper bound on what is achievable against a mean-based no-regret algorithm.

Theorem 10. For any game G and any mean-based no-regret algorithm used by player B , there is no strategy for which the average reward of player A converges to Val A ( ∅ , E D ( G ) ) + ε , for any ε > 0 .

Proof. First, we observe that mean-based algorithms will never play a dominated strategy b i ∈ D ( G ) in more than o ( T ) rounds. As b i is dominated, there is some δ > 0 such that for every α ∈ ∆( M ) , there is some b j where u B ( α, b j ) ≥ u B ( α, b i ) + δ . Let α t denote the empirical distribution of player A 's actions up to time t . After some window of O ( γT ) = o ( T ) rounds we will have the cumulative rewards σ i,t and σ j,t satisfy σ i,t < σ j,t -δt < σ j,t -γT under any α t for some b j in each subsequent round, and so b i will never be played in more than o ( T ) rounds.

We can also see that any such noE -regret algorithm is a noE D ( G ) -regret algorithm. Suppose such an algorithm had E D ( G ) -regret ϵT , for ϵ > 0 ; then, there is some g jk for which U B ( α T , g jk ( β T )) ≥ U B ( α T , β T ) + ϵ . By the E -regret guarantee this cannot occur if j = k , as any such function g jj is

for each b i , which holds if:

̸

equivalent to the fixed deviation rule for b j . However, if this occurs for j = k , such an algorithm must have played dominated strategies in a total Ω( ϵT ) . This contradicts our assumption that no dominated strategy b i is played in more than o ( T ) rounds, and so any mean-based noE -regret algorithm is also a noE D ( G ) -regret algorithm, against which player A cannot obtain average reward which converges to any amount higher than Val A ( ∅ , E D ( G ) ) + o (1) .

H Proof of Theorem 5

Proof. Let MBRew A denote the maximal reward obtainable by player A when player B uses a mean-based algorithm. Observe that b 3 is dominated for player B , and thus cannot be included in any ( ∅ , I ) -equilibrium (by Theorem 10). Further, it will never be played by a mean-based learner for more than o ( T ) rounds, as for any distribution over a 1 and a 2 the best response is either b 1 or b 2 . As such, both Val A ( ∅ , I ) and MBRew A are at most 1 + o (1) ; a reward of 1 -o (1) is obtainable by committing to either a 1 or a 2 for each round. However, we can see that the optimal ( ∅ , E ) -equilibrium p for player A includes positive mass on ( a 1 , b 3 ) , and yields an average reward of Val A ( ∅ , E ) = 2 for player A . Let p 1 be the probability on ( a 1 , b 1 ) , let p 2 be the probability on ( a 2 , b 2 ) , let p 3 be the probability on ( a 1 , b 3 ) , and let p 0 be the remaining probability. The reward for player A is given by:

and p defines a ( ∅ , E ) -equilibrium if

Only the first constraint is non-trivial, and so the optimal ( ∅ , E ) -equilibrium for player A occurs by maximizing p 1 + p 2 +3 p 3 subject to p 2 ≥ p 3 , which yields a probability of 0.5 for both p 2 and p 3 (and 0 for p 1 and p 0 ), as well as an average reward of 2. As such, player A cannot obtain a reward approaching Val A ( ∅ , E ) , as their per-round reward is at most 1 + o (1) .

I Proof of Theorem 6

Proof. We recall that the SU algorithm from (19) finds initial points α ∗ ( b i ) in each best response region via random sampling, which takes takes 1 / poly ( ε -1 ) queries in expectation. Then, upon calibrating for O (log(1 /ε )) bits of precision SU makes poly ( M,N, log(1 /ε )) queries, each of which can be taken to be a point on some grid of spacing 1 / poly ( ε -1 ) within the simplex by the precision condition. The computed approximate Stackelberg strategy is then the optimal such point on the grid.

We first describe our strategy for simulating each query against an arbitrary anytime-no-regret learner; as E ⊆ Φ , we can restrict to considering only no-external-regret learners, as these regret constraints will always be satisfied. To implement a query q , greedily play the action whose historical frequency of play is the furthest below its target frequency in q . After O ( poly (1 /ε )) rounds, the historical distribution will be within 1 / poly ( ε -1 ) of q , and continuing the greedy selection strategy indefinitely will ensure that the history remains in a 1 / poly ( ε -1 ) -ball around q . Let t q be the time at which this occurs. After maintaining the greedy strategy for q for an additional ω ( t c q ) rounds, the anytime regret bound ensures that most frequently played item must indeed be the best response response to some point in the ball around q , provided that this ball is contained entirely inside some best response region R j . For the sampling step, a taking sufficiently small grid (but still 1 / poly ( ε -1 ) ) ensures that random sampling still suffices to find a point a point in each best response region even if our

queries may be adversarially perturbed to neighboring points on the grid, as each region is convex and has volume at least 1 / poly ( ε -1 ) . To address the issue for the line search steps, it suffices to take an additional step along each search conducted by SU before termination, where we then take each hyperplane boundary estimate to be one step inward along the grid from where our search terminates, maintaining a buffer between each hyperplane estimate in which all our points of uncertainty must lie. This adds at most a constant factor to our query complexity, and impacts our approximation by 1 / poly ( ε -1 ) , which then yields us a runtime of poly (1 /ε ) Q rounds.

For the case of a no-adaptive-regret learner, suppose such an algorithm is calibrated for T = O ( Q C 1 (1 /ε ) C 2 ) ; then, over any window of length W its regret is at most O ( ( Q C 1 (1 /ε ) C 2 ) c W -1 ) . Taking W = ω (( Q C 1 (1 /ε ) C 2 ) c ) yields a per-round regret of at most o (1) over the window, and so an algorithm must play a best response in W -o ( W ) of the rounds. For sufficiently large C 1 and C 2 , each W is large enough to yield the same precision we required for the anytime case, where now we greedily play the action whose frequency is furthest below its target since our previous query terminated , which allows us to again simulate the O ( Q ) queries in poly ( Q/ε ) rounds (accounting for the robustness checks) while yielding Θ( WQ ) = o ( T ) .

J Proof of Theorem 7

Proof. Our game consists of M actions A for the optimizer, and N = 2 M + ( M 2 ) actions for the learner, which are divided into M primary actions B , M secondary actions S , and ( M 2 ) safety actions Y .

If we restrict the learner to only playing primary actions, the game somewhat resembles a coordination game, where each pure strategy pair ( a j , b j ) is a Nash equilibrium. However, the set B is comprised of both undominated actions B U and dominated actions B D , which are unknown to the optimizer, and where each b j ∈ B d is weakly dominated by the secondary action s j . The optimizer receives reward 0 whenever the learner plays a secondary action, and so the challenge for the optimizer is to identify the pair ( a j , b j ) which maximizes u A ( a j , b j ) , for b j ∈ B D , which will be the Stackelberg equilibrium. Further, the safety actions y ij essentially allow the learner to hedge between two actions; this does not pose substantial difficulty for the optimizer when the learner is no-swap-regret, yet creates an insurmountable barrier for learning the Stackelberg equilibrium in sub-exponential time against a mean-based learner.

An instance of a game G ∈ G is specified by the partition of B into B U and B D . There is an action s j ∈ S for each j , and for each pair ( i, j ) with i < j there is an action y ij ∈ Y . The rewards for a game G are as follows. For any strategy pair, the optimizer's utility is given by:

u A ( a j , b j ) = j/M for b j ∈ B ;

u A ( a i , b j ) = 0 for b j ∈ B and with i = j ;

̸

u A ( a i , s j ) = 0 for any s j ∈ S ;

u A ( a i , y jk ) = 0 for any y jk ∈ Y ;

and the learner's utility is given by:

For b j ∈ B U

̸

For b ∈ B

For s j ∈ S

̸

̸

-u B ( a i , s j ) = 0 for i = j ; · For y ij ∈ Y : -u B ( a i , y ij ) = u B ( a j , y ij ) = 2 / 3 ; -u B ( a k , y ij ) = 0 for i, j = k .

We assume that B U is non-empty, and so there is some optimal pure Nash equilibrium ( a ∗ i , b ∗ i ) which yields a reward of i/M ; it is simple to check that this is also the Stackelberg equilibrium.

Optimizing against no-swap learners. First, we give a method for matching the Stackelberg value against an arbitrary no-swap-regret learner, which corresponds to the pair ( a j , b j ) for the largest value j such that b j ∈ B U . Consider a no-swap-regret learner which obtains a regret bound of τ = O ( T c ) over T rounds. Let SR t ( b, b ′ ) for any learner actions b and b ′ denote the t -round cumulative swap regret between b and b ′ , i.e. the total change in reward which would have occurred if b ′ was played instead for each of the first t rounds in which b was played. To model the behavior of an arbitrary no-swap-regret learner, we disallow the learner from taking any action which would increase SR t ( b, b ′ ) above τ , given the loss function for the current round, and otherwise allow the action to be chosen adversarially. While our model is deterministic for simplicity, it is straightforward to extend to the analysis to algorithms whose regret bounds hold in only expectation, e.g. by considering a distribution over values of τ in accordance with Markov's inequality (as no algorithm can have negative expected regret against arbitrary adversaries) and considering our expected regret to the Stackelberg value.

Our strategy for the optimizer is:

For each i ∈ [ M ] , play a i until either b i or s i is observed at least t ∗ > τ times;

Return a ∗ i for the largest i such that b i is observed t ∗ times.

We show that this takes at most O ( T c · M 3 ) rounds. Once a ∗ i is identified, we can commit to playing it indefinitely, at which point the learner must play b ∗ i in all but at most O ( T c · poly ( M )) rounds, and so with T = O ( poly ( M/ε )) rounds we can increase the total fraction of rounds in which ( a ∗ i , b ∗ i ) is played to 1 -ε , which yields the desired average reward bound.

The key to analyzing the runtime of our strategy is to consider the 'buffer' in regret between any pair of actions before the threshold of τ is reached, which enables us to the bound the number of rounds in which instantaneously suboptimal actions are played. Note that prior the start of window i (where a i is played), both b i and s i obtain reward 0 in each round, and as such cannot decrease their expected regret relative to any other action, as all rewards in the game are non-negative. Further, for any previous window j , both b i and s i incur regret of 1 with respect to either b j or s j , as well as between the suboptimal and optimal action in window i , and thus cannot be observed more than τ times in the window. As such, observing b i at least t ∗ times in window i indicates that b i ∈ B U (and likewise observing b i at least t ∗ times indicates that b i ∈ B D ).

̸

Any action b = BR ( a i ) will incur positive swap regret with respect to BR ( a i ) , and cannot be played in window i once SR t ( b, BR ( a i )) ≥ τ . Each action begins with SR 1 ( b, BR ( a i )) = 0 at time t = 1 ; for each of the learner's actions, we consider the rate at which its buffer decays, as well as instances in which swap regret can decrease:

Previously optimal b ∈ B ∪S \ BR ( a i ) : actions in B∪S can only accumulate negative swap regret with respect to BR ( a i ) during rounds in which they were previously optimal; any previous optimum b = BR ( a j ) for j < i was played at most t ∗ times during window j , and so we have that SR t ( b, BR ( a i )) ≥ -t ∗ .

All b ∈ B ∪ S \ BR ( a i ) : ignoring any previously accumulated regret buffer, each of these 2 M -1 actions can be played at most τ rounds during window i before exhausting their initial buffer. Accounting for possible previous optima with SR t ( b, BR ( a i )) < 0 , the number of rounds during window i in which some b ∈ B ∪ S \ BR ( a i ) is played is at most Mt ∗ +(2 M -1) τ .

Safety actions y jk ∈ Y : Suppose neither a j or a k have been played yet by the optimizer, including in the current window. As was the case for other actions which have never yielded positive instantaneous reward, y jk can be played at most τ times before SR t ( y jk , BR ( a i )) ≥ τ . If j = i , i.e. this is the first window in which y jk obtains positive instantaneous reward, the per-round regret is 1 / 3 , and so at it can be played for most 3 τ rounds. Further, y jk a obtains a regret of -2 / 3 with respect to BR ( a k ) . If k = i and the window for a j has already been completed, y jk can be played for at most 9 τ rounds, as initially we have that SR t ( y jk , BR ( a i )) ≥ -2 τ , which again increases by 1 / 3 per round. We then have that the total amount of rounds with safety actions played during window i is at most (12 M + M 2 ) τ , as there are fewer than M 2 total safety actions, and fewer than M in each of the latter cases.

This yields a per-window runtime across all actions of at most Mt ∗ +( M 2 +10 M -1) τ , which is O ( T c · M 3 ) across all windows, and so we obtain the desired result for optimizing against arbitrary no-swap-regret learners.

Optimizing against mean-based learners. Here, we show that there are mean-based no-regret algorithms for which exponentially many rounds are required for an optimizer to approximate the Stackelberg value against a learner. When considering horizons which are superpolynomial in the parameters of the game, it is most natural to consider algorithms with regret bounds which are non-trivial for smaller horizons, as well as an anytime variant of the mean-based property. We define an extension of the classical Multiplicative Weight Updates algorithm ( MWU ; see (2) for a survey), called Rounded Mean-Based Doubling, which inherits both properties in the anytime setting.

Lemma 6. When running RMBD for T rounds, the following hold at any round t ≤ T :

RMBD has cumulative regret ˜ O ( n √ t ) ;

If action j has the highest cumulative reward and σ i,t ≤ σ j,t -˜ O ( √ t ) , then action i is played with probability 0 at round t .

Proof. Let C √ t bound the regret of MWU over t rounds (where C = O ( √ log n ) ), and let D = √ 2 C + ˜ O ( n ) . We can bound the regret of RMBD over T i rounds by D √ T i via induction (which holds trivially at T 1 ). Suppose it holds for some T i . Let R ( T i ) be the true reward obtained by RMBD over T i rounds, which is at least σ j ∗ ,T i -D √ T i , where σ j ∗ ,T i is the cumulative reward of the best action over T i rounds. Consider our simulation of MWU over T i rounds using the average reward function. As the reward function is identical each round, and the cumulative reward for each action j is equivalent under averaging, the measured reward ˆ R ( T i ) from the simulated run is at most σ j ∗ ,T i after T i rounds. Upon continuing to run this instance of MWU for an additional T i rounds, the regret bound ensures that the total measured reward ˆ R ( T i +1 ) is at least σ j ∗ , 2 T i -C √ 2 T i . Rounding probabilities contributes at most an additional 2 nγT i to the regret; it suffices to implement rounding by reallocating probability mass from any p i,t < 2 γ onto other actions arbitrarily, to avoid renormalization. The total reward of RMBD over 2 T i = T i +1 is given by its cumulative reward at T i , as well as the additional reward obtained by the MWU instance over the next T i rounds, and so we

have that

which yields the bound for every T i . We can extend this to any t ∈ [ T i , T i +1 ] with at most a factor 2 increase to cumulative regret.

To bound the selection frequency of actions with suboptimal cumulative reward, we recall the meanbased analysis of MWU given in Theorem D.1 from (5), which shows that the selection frequency p k,t for action k at time t is at most γ = 2 log( √ T log n ) √ T log n if σ k,t ≤ σ j,t -γT for the action j with highest cumulative reward. As such, any action whose cumulative reward σ k,t ≤ σ j,t -˜ O ( √ t ) will be played with probability 0.

Suppose a learner plays the action with highest cumulative reward at each round for t burn = ˜ Ω( M 2 ) rounds, then plays RMBD thereafter for a total of T rounds. Note that this maintains the both properties of RMBD for all t . We show that at least T = exp(Ω( M )) rounds are required to identify the Stackelberg strategy. The optimizer must check the learner's pure best response to each a j for identification with certainty, and it is straightforward to construct a distribution in which any strategy which does not observe BR ( a j ) for all j will have linear regret to Stack A in expectation (e.g. where B U contains one action chosen uniformly at random). The difficulty in exploration of the best responses comes from the safety actions, as a j must have been played more frequently than any other action in order to not be dominated by some safety action. Let ρ j,t denote the number of rounds in which the optimizer has played a j out of the first t . Observe that by construction of the game and the properties of RMBD , an primary or secondary action b j or s j in BR ( a j ) will only be played with positive probability when:

for all k , which necessitates that ρ j,t ≥ 2 t M -˜ O ( √ t ) . Taking t burn sufficiently large, we have that ρ j,t ≥ 3 2 ρ k,t for any t ≥ t burn and all k . For any subsequent observation BR ( a k ) at t ′ , we must have that ρ k,t ′ ≥ 3 2 ρ j,t , and so the number of rounds required to play an action before observing its best response grow at a rate of at least (3 / 2) M , which completes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies several trade-offs that existed in learning in games. The first is the tradeoff between regret and rewards, for which a generalized notion of equilibria is introduced. It is then investigated whether running a no-swap regret learning algorithm is efficient. It is shown that this depends on the form of the games, in some games running a no-swap regret learning algorithm can be more efficient than employing the Stackelberg strategy. The same question is investigated for learning in game against a no-external regret learning algorithm.\n\nSTRENGTHS:\nOverall, the paper is really well-written and well-organized. The problem investigated and the results presented are interesting. The generalized notion of equilibria seems to be useful for other analyses of learning in games. It is also important to study the performance of different types of algorithms in games, (such as mean-based no-regret learning algorithms), these results can lead to further insights into algorithm designs for learning in games.\n\nWEAKNESSES:\nOverall the paper is pretty solid, a rather minor weakness is that the presentation of the paper can still be improved. \nThere are quite a number of results presented in the paper. As a result, section 1.1 is a really lengthy section. It seems like the authors are trying to summarize the question investigated, related works, and the obtained results in this section. This can lead to some confusion as some of the notions are yet to be introduced in the paper and this is still at the very beginning of the paper. I would suggest making the section more concise and putting some of the discussion in the later parts of the paper.\n\nQUESTIONS:\n1. Could the authors elaborate on why exponential weights, FTPL etc algorithms are mean-based? Also, Theorem 4 seems to be stated with respect to average reward, while Theorem 5 seems to be saying that a mean-based algorithm cannot attain the total rewards (which seems to be not surprising?), I wonder how these two Theorems should be interpreted together. \n\n2. It is mentioned that Proposition 5 can be improved, though the query complexity is still inversely proportional to the best response region volume. But from Theorem 6, it seems that through stimulating the best response queries, the complexity is independent of the best response region volume.\n\n3. It seems to be that mean-based algorithms often have no external-regret. I wonder if they can be also no swap regret? If so, how should one interpretate Theorem 7?\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission studies questions surrounding playing against a no-regret learner in a repeated game setting. These questions are motivated by previous observations that while it is known that when all players play no-regret strategies the empirical frequency of play approaches an equilibria, a player can sometimes do better by deviating to a different strategy/algorithm (that is not no-regret). \n\nIn particular, the authors focus on four questions: (1) When does reward trade off with regret? (2) Under what game settings is playing a no-swap-regret algorithm a stable equilibrium? (3) How to play against a no-regret learner? (4) How can one learn the Stackelberg strategy through repeated play against a no-regret learner?\n\nTowards answering question (1), the authors consider a generalized notation of equilibrium, (Phi-A, Phi-B)-equilibrium, where player A is restricted to playing no-Phi-A-regret strategies and player B is restricted to playing no-Phi-B-regret strategies. Each choice of (Phi-A, Phi-B) induces a polytope of (Phi-A, Phi-B)-equilibria. They show that for any equilibria in this polytope, there exists a pair of no-regret algorithms in (Phi-A, Phi-B) which converge to it. \n\nTo assist in answering question (2), the authors consider a \"metagame\", in which at the beginning of a repeated game, both players simultaneously announce and commit to an algorithm to use during the game. The authors main result towards answering (2) are a set of sufficient and necessary conditions for (a) some pair of no-swap-regret algorithms to form a Nash equilibrium in the metagame and (b) all pairs of no-swap regret algorithms to form a Nash equilibrium in the metagame. In an effort to characterize which types of games playing no-swap-regret algorithms is \"optimal\", the authors show that if a game G does not contain a pure Nash equilibria, then there does not exist a pair of no-swap-regret algorithms which form a Nash equilibrium in the metagame, when player utility functions in G are randomly perturbed. \n\nTowards answering (3), the authors show that there exists a no-(external)-regret algorithm for player B and a strategy for player A such that the average reward for player A converges to their best possible feasible reward. However, the authors show that against any mean-based no-regret learner (a popular subset of no-external-regret algorithms), there does not exist a strategy for player A which can get \"close to\" the best possible feasible reward. \n\nFinally, the authors answer (4) by showing how to learn the Stackelberg strategy by simulating best-response queries against the no-regret learner. While they show that in general this may require exponentially-many queries, polynomially-many queries are sufficient if the no-regret learner is playing a no-adaptive-regret algorithm.\n\nSTRENGTHS:\nWhile the authors are not the first to consider the general problem of playing a repeated game against a no-regret learner, this paper both introduces and addresses a (very) wide range of important and well-motivated questions surrounding the topic. The results contained in this submission provide valuable insights into this highly nuanced problem. While no one result stands out in particular, the sheer breadth of the results obtained by the authors in this submission is very impressive and the submission as a whole presents the clearest picture to-date of \"the right thing to do\" when playing against a no-regret learner.\n\nWEAKNESSES:\nWith that being said, the breadth of the results obtained by the authors makes it unclear what the main takeaway of the submission should be. At times, the submission reads like a laundry list of results about playing against a no-regret learner. Additionally, a longer discussion on related works in the main body (particularly (10) and (22)) would help someone who is not as familiar with the area better understand the main contributions of the authors.\n\nQUESTIONS:\nIn Section 3, why is Nash equilibrium the \"right\" solution concept for the metagame?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

comments: SUMMARY:
The paper explores tradeoffs between reward and regret in repeated gameplay between two agents. It introduces a concept of generalized equilibrium that allows for different regret constraints, resulting in feasible values for each agent. The paper shows that such equilibria can be reached by algorithms maintaining regret guarantees against any opponent. The paper also examines tradeoffs in terms of the opponent's algorithm choice and characterizes the maximal reward achievable against a no-regret learner. It demonstrates that different classes of no-regret algorithms can lead to varying rewards.

STRENGTHS:
- Theoretical analysis is solid and convincing.
- The problem studied is interesting. Although running no-regret dynamics leads to CCE, no work attempts to doubt "running no-regret" this thing itself.

WEAKNESSES:
I think some realistic running examples can be supplemented for better illustration.

QUESTIONS:
See in Weaknesses

LIMITATIONS:
See in Weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper addresses several interesting questions regarding the tradeoff between reward and regret in repeated gameplay between two agents. Three problems are sequentially investigated. 1. The paper provides a characterization of the setting when running a no-swap-regret learning algorithm is preferred over playing the Stackelberg strategy; it further showed that such a setting has measure zero and almost does not happen. 2. This paper shows that if the opponent is running any no-regret algorithm, the utility of the player is upper bounded by the unconstrained-external value of the game; such an upper bound is achievable for a particular no-regret algorithm of the opponent. 3. The paper shows that it is possible to convert any best-response query algorithm for finding Stackelberg equilibria via best-response queries to an adaptive strategy that learns Stackelberg equilibria via repeated play against a generic no-regret learner, albeit potentially at the cost of an exponential blow-up in the number of rounds.

STRENGTHS:
The several questions addressed in this paper are very interesting. In algorithmic game theory, many existing work focus on algorithms for finding an equilibrium via repeated gameplay. However, in repeated gameplay, the agent’s interest is often maximizing reward and has the motivation to deviate from the regret minimization algorithm. This paper studies when deviating from the regret minimization algorithm is beneficial and when it is not.

WEAKNESSES:
There are still many unsolved open questions. For example, for specific no-regret algorithm classes, it is not clear how much one can exploit.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers equilibria between agents that have arbitrary regret benchmarks (corresponding to different equilibrium concepts), and the relation between those equilibria and the interactions of no-regret learning agents.  It is known in the literature that an agent who knows that the other is playing a no-external-regret algorithm can guarantee themself the Stackelberg leader value (i.e., $Val_A(\\emptyset,\\mathcal{E})$ in the terminology of this paper).  This paper extends that results by showing that in fact it can be better to play a no-external-regret learning strategy against a no-swap-regret learner (i.e., a learning algorithm with a weaker regret guarantee can get higher utility, all else being equal).  A further result is that any generalized equilibrium is reachable by a pair of regret-minimizing learning strategies.\n\nThe paper also considers the complexity of learning Stackelberg strategies; it turns out to be easier to learn a Stackelberg leader strategy against a learning algorithm with a stronger regret guarantee, because it reacts more quickly to best-respond to changes in the other player's behavior.\n\nSTRENGTHS:\nThis is an extremely strong paper.  The question of when minimizing regret benchmarks also lead to good performance (the thing that we actually care about, in general) is really important, and this paper provides rigorous, compelling, and general answers.  The generalized equilibrium framework and the connections between learning and equilibrium are very clear and likely to have a significant impact on the learning in games literature.  I strongly expect to refer to this paper in the future.\n\nThe complexity results at the end are a nice touch as well.\n\nWEAKNESSES:\nI have no major complaints about the paper.  Here are some minor comments/issues:\n\n- I was initially very surprised by Theorem 1; a little more hand-holding about why this doesn't contradict Barman & Ligett [2015] might have helped.  (Basically, the result doesn't require the algorithm pair to _find_ an optimal equilibrium, instead it will converge without regret to an exogenously _specified_ equilibrium).\n- p.6: \"Further, each equilibrium set can be optimized over via a linear program.\": I'm not sure what this means.\n- p.8: \"Let $\\sigma_{i,t}$ be the cumulative reward resulting from playing action $i$ for the first $t$ rounds\": It would be clearer to avoid the notational collision with strategies by using a different letter\n- p.8: A few more details in the proof sketch for Theorem 5 would be helpful; it took me a while to convince myself even that the statement made sense.\n\n\n[Barman & Ligett 2015]: \"Finding Any Nontrivial Coarse Correlated Equilibrium Is Hard\"\n\nQUESTIONS:\nnone\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}