SlowGuess/NeurIPS_2023_OCR huggingface dataset:
w.brown@columbia.edu
Jon Schneider
jschnei@google.com
Kiran Vodrahalli
kirannv@google.com
We consider a number of questions related to tradeoffs between reward and regret in repeated gameplay between two agents. To facilitate this, we introduce a notion of generalized equilibrium which allows for asymmetric regret constraints, and yields polytopes of feasible values for each agent and pair of regret constraints, where we show that any such equilibrium is reachable by a pair of algorithms which maintain their regret guarantees against arbitrary opponents. As a central example, we highlight the case one agent is no-swap and the other's regret is unconstrained. We show that this captures an extension of Stackelberg equilibria with a matching optimal value, and that there exists a wide class of games where a player can significantly increase their utility by deviating from a no-swap-regret algorithm against a no-swap learner (in fact, almost any game without pure Nash equilibria is of this form). Additionally, we make use of generalized equilibria to consider tradeoffs in terms of the opponent's algorithm choice. We give a tight characterization for the maximal reward obtainable against some no-regret learner, yet we also show a class of games in which this is bounded away from the value obtainable against the class of common "mean-based" no-regret algorithms. Finally, we consider the question of learning reward-optimal strategies via repeated play with a no-regret agent when the game is initially unknown. Again we show tradeoffs depending on the opponent's learning algorithm: the Stackelberg strategy is learnable in exponential time with any no-regret agent (and in polynomial time with any no-adaptive-regret agent) for any game where it is learnable via queries, and there are games where it is learnable in polynomial time against any no-swap-regret agent but requires exponential time against a mean-based no-regret agent.
How should two rational agents play a repeated, possibly unknown, game against one another? One natural answer – barring any knowledge of the game, or the capacity to compute potentially computationally intractable equilibria – is that they should employ some sort of learning algorithm to learn how to play over time. Indeed, there is a vast literature which studies what happens when all the players in a repeated game run (some specific type of) learning algorithms to select their actions. For example, when all players in a game simultaneously run no-(swap)-regret learning algorithms, it is known that the average strategy profile of the learners converges to a (coarse) correlated equilibrium (23; 11; 16). More recent works have studied how to design algorithms that converge to these equilibria at faster rates (7; 1; 8), performance guarantees of such equilibria compared to the optimal possible welfare (4; 17), and the specific dynamics of such learning algorithms (9; 20; 22).
In contrast, relatively little attention has been devoted to whether it is actually in the interest of these agents to run these specific learning algorithms. For example, in a setting where all agents are running no-swap-regret learning algorithms, when can an agent significantly benefit by deviating and running a different type of algorithm? And if they can, what algorithm should the agent deviate to?
We explore the following questions (and others) in the case where two agents repeatedly play a normal-form, general-sum game for  $T$  rounds.
When does reward trade off with regret? While maximizing reward is often viewed as the objective of regret minimization against arbitrary adversaries, tensions may emerge when playing against another learning agent, as one's choice of actions has a causal effect on future loss functions as determined by the opponent's algorithm. In such cases, as we discuss further below, it turns out that demanding a stronger regret guarantee (e.g. asking for no-swap-regret instead of no-external-regret) may ultimately result in lower reward for an agent. To analyze these tradeoffs, in Section 2 we introduce a notion of generalized  $(\Phi_A,\Phi_B)$ -equilibrium, where  $\Phi_{A}$  and  $\Phi_B$  are the sets of "deviation strategies" over which players  $A$  and  $B$  minimize regret (e.g. fixed actions or swap functions), presented as an asymmetric extension of the linear  $\Phi$ -equilibria considered by (13). Each pair of strategy sets  $(\Phi_A,\Phi_B)$  generates a polytope of  $(\Phi_A,\Phi_B)$ -equilibria, where any point then yields a reward value for each player. We show that all such points are feasible: for any game and any  $(\Phi_A,\Phi_B)$ -equilibrium  $\varphi$ , there is a pair of algorithms which converge to  $\varphi$  while maintaining their respective  $\Phi_A$ -regret and  $\Phi_B$ -regret guarantees against arbitrary opponents. Further, deviating to a strategy with fewer constraints  $\Phi_A$  can often result in strictly improved reward for player  $A$ . As a concrete application, we consider the question of deviating from simultaneous no-swap-regret play.
When is no-swap-regret play a stable equilibrium? What should you do if you know that your opponent in a repeated game is running a no-swap-regret algorithm to select their actions? In (10), the authors show that one utility-optimizing response (up to additive  $o(T)$  factors) is to play a static (mixed) strategy (your Stackelberg strategy) and obtain the Stackelberg value of the game. However, determining your Stackelberg strategy requires some knowledge of the game, and acquiring this knowledge from repeated play may be difficult (we address the latter issue in Section 5). In comparison, it is relatively straightforward to also run a no-swap-regret learning algorithm. This begs the question: are there games where you obtain significantly less utility (i.e., at least  $\Omega(T)$  less utility) by running a no-swap-regret learning algorithm instead of playing your Stackelberg strategy?
We show that the answer is yes, such games exist and are relatively common. In fact, we provide an efficient algorithmic characterization of the games  $G$  for which both players playing a no-swap-regret learning algorithm is an  $(o(T))$ -approximate Nash equilibrium of the entire repeated "meta-game". The exact characterization is presented in Section 3 and is somewhat subtle, e.g., there are slightly different characterizations depending on whether we insist all pairs of no-swap-regret algorithms lead to approximate equilibria or only one specific pair, corresponding to best-case and worst-case values in the generalized equilibrium polytope. One consequence of both characterizations, however, is that for almost all games (in a measure-theoretic sense, considering arbitrarily small perturbations), in order for it to be an approximate equilibrium for both players to play a low-swap regret strategy, the game  $G$  must possess a pure Nash equilibrium. That is, in any game without a pure Nash equilibrium, it is possible for at least one of the parties to do significantly better by switching from no-swap-regret learning to playing their Stackelberg strategy.
Finally, we additionally show there are some games where playing a no-(external)-regret algorithm against another no-swap-regret learner weakly dominates playing a no-swap-regret algorithm, regardless of the specific choice of algorithms. This counters the intuition that stronger regret guarantees protect a player from worse outcomes.
Optimizing reward against no-regret learners. What if our opponent is not running a no-swap-regret algorithm, but simply a no-(external)-regret algorithm? In this case, it is still possible to obtain at least the Stackelberg value of the game by playing our Stackelberg strategy (no-regret algorithms are also guaranteed to eventually learn and play the best response to this strategy). However, unlike in the no-swap-regret setting, there exist specific games and no-regret algorithms where it is possible to obtain significantly  $(\Omega(T))$  more than the Stackelberg value by playing a specific dynamic strategy. This phenomenon was first observed in (10), where a specific game is given for which it is possible to obtain  $\Omega(T)$  more than Stackelberg when playing against any no-regret algorithm in the family of mean-based learning algorithms (including algorithms such as multiplicative weights and EXP3). However, many questions remain unanswered, such as understanding in which games it is possible to outperform playing one's Stackelberg strategy, and by how much.
In Section 4 we present some answers to these questions for the case of generic no-regret algorithms. Specifically, we first show that if player  $B$  is running (any) no-regret algorithm, the utility of player  $A$  (regardless of what strategy they employ) is upper bounded by  $\mathsf{Val}_A(\emptyset, \mathcal{E}) \cdot T + o(T)$ , where  $\mathsf{Val}_A(\emptyset, \mathcal{E})$  is what we call the unconstrained-external (equilibrium) value of the game for player  $A$ , which is given by the solution to a linear program. We then show that this upper bound is asymptotically tight: there exists a no-regret algorithm  $\mathcal{L}$  such that if player  $B$  is playing according to  $\mathcal{L}$ , then the player  $A$  can obtain utility at least  $\mathsf{Val}_A(\emptyset, \mathcal{E}) \cdot T - o(T)$  by playing an appropriate strategy in response.
Note that this characterization does not completely resolve the question of (10) - it requires the construction of a fairly specific no-regret algorithm  $\mathcal{L}$ , and it is still open what is possible against specific (classes of) no-regret algorithms (e.g., multiplicative weights or mean-based algorithms). In fact, we show a property of games which, when satisfied, implies that it is impossible to obtain the unconstrained-external value against a mean-based learner.
Learning the Stackelberg strategy through repeated play. Finally, we address the question of how hard it is to actually identify the Stackelberg strategy in a game against a learning opponent. Given full knowledge of the game  $G$ , finding an agent's Stackelberg strategy is simply a computational problem which turns out to be efficiently solvable by solving several small LPs (see (6)). However, if the game (in particular, the opponent's reward matrix) is unknown, an agent must learn the Stackelberg strategy over time. Existing work on learning Stackelberg strategies (e.g., (19; 24)) generally assumes access to a best-response oracle for the game (i.e., for a specific mixed strategy, how will an opponent best-respect?). In contrast, if our opponent is playing a specific no-regret learning algorithm, they may not immediately best respond to the strategies we play! This raises the following two questions. First, when is it possible to learn the Stackelberg equilibrium of a game while playing against a learning opponent? Second, is it easier to learn this equilibrium when playing against certain classes of learning algorithms?
In Section 5 we begin by showing that it is indeed possible to convert any best-response query algorithm for finding Stackelberg equilibria via best-response queries to an adaptive strategy that learns Stackelberg equilibria via repeated play against a generic no-regret learner, albeit potentially at the cost of an exponential blow-up in the number of rounds, e.g. for a query algorithm which makes  $Q$  best-response queries, simulating it against a no-regret learner may require  $T = \exp(Q)$  rounds of play. For the special case of opponents with no-adaptive-regret algorithms (such as online gradient descent), we show that only  $T = \mathrm{poly}(Q)$  rounds are required in the worst case. However, in general we show that exponential runtime can be necessary. In particular, we give an example of a game with  $M$  actions where it is possible to learn the Stackelberg equilibrium in  $\mathrm{poly}(M)$  rounds when playing against any no-swap-regret learning algorithm, but where it requires at least  $\exp(M)$  rounds to learn this equilibrium when playing a mean-based no-regret algorithms.
The broader literature on no-regret learning in repeated games is substantial, covering many equilibrium convergence results varying assumptions. A recent line of work (5; 10; 21) considers problems related to optimizing one's reward when competing against a no-regret learner in a game. We extend these questions to consider the relationship and regret for an optimizer, as well as to settings where properties of the game are initially unknown, and give a series of separation results in terms of various notions of equilibrium. Also relevant is the literature on analysis of no-regret trajectory dynamics, and in particular (20) which shows a game in which no-regret dynamics outperform the reward of the Nash equilibrium. Additionally, there is also prior work considering regret minimization problems involving either best-responding or otherwise strategic agents (see e.g. (3; 18)), as well as work considering alternate regret notions or behavior models for repeated Stackelberg games (e.g. (12; 15)).
1.3 Notation and preliminaries
Throughout, we consider two-player bimatrix games  $G = (A, B)$ , where player  $A$  ("the optimizer") has action set  $\mathcal{A} = \{a_1, \ldots, a_M\}$  and player  $B$  ("the learner") has action set  $\mathcal{B} = \{b_1, \ldots, b_N\}$ . When the optimizer plays action  $a_i$  and the learner plays action  $b_j$ , the players receive rewards  $u_A(a_i, b_j)$  and  $u_B(a_i, b_j)$ , respectively. We assume that the magnitude of each utility is bounded by a constant. The sets of mixed strategies for each player are denoted by  $\Delta(\mathcal{A})$  and  $\Delta(B)$ , respectively;
when the optimizer plays a mixed strategy  $\alpha \in \Delta (\mathcal{A})$  and the learner plays  $\beta \in \Delta (\mathcal{B})$ , the expected reward for the optimizer is given by  $u_{A}(\alpha ,\beta) = \sum_{i = 1}^{M}\sum_{j = 1}^{N}\alpha_{i}\beta_{j}u_{A}(a_{i},b_{j})$ , with  $u_{B}(\alpha ,\beta)$  defined analogously. An action  $b\in \mathcal{B}$  is a best response to a strategy  $\alpha \in \Delta (\mathcal{A})$  if  $b\in \operatorname{argmax}_{b^{\prime}\in \mathcal{B}}u_{B}(\alpha ,b^{\prime})$ . We let  $\mathsf{BR}(\alpha)$  be the set of all such actions for player  $B$ , and likewise  $\mathsf{BR}(\beta)$  for player  $A$ .
2 Generalized equilibria and no-  $\Phi$  -regret learning
Here we introduce the notions of  $\Phi$ -regret and generalized equilibria, which we use to analyze the regret and reward of players in repeated bimatrix games under varying assumptions regarding the choice of regret benchmarks, the algorithms used, and the structure of the game.
Originally introduced in (14) and extended to general convex settings in (13), we consider the formulation of linear  $\Phi$ -regret as it relates to bimatrix games. Given a sequence of action pairs  $(a_{i_1}b_{j_1}),\ldots ,(a_{i_T}b_{j_T})$  for  $T > 0$  and some set of functions  $\Phi$ , where each  $f\in \Phi$  maps actions  $\mathcal{A}$  to action profiles in  $\Delta (\mathcal{A})$ , we say that the  $\Phi$ -regret for the optimizer (and analogously for the learner) is
$$
\mathsf {R e g} _ {\Phi} (T) = \max _ {f \in \Phi} \sum_ {t = 1} ^ {T} u _ {A} (f (a _ {i _ {t}}), b _ {j _ {t}}) - u _ {A} (a _ {i _ {t}}, b _ {j _ {t}}).
$$
Definition 1 (No-  $\Phi$  -regret learning). We say a learning algorithm  $\mathcal{L}$  is a no-  $\Phi$  -regret algorithm if, for some constant  $c < 1$ , we have that  $\operatorname{Reg}_{\Phi}(\mathcal{L},T) = O(T^{c})$ , where  $\operatorname{Reg}_{\Phi}(\mathcal{L},T)$  is the  $\Phi$  -regret corresponding to the action sequence played by  $\mathcal{L}$ .
Some notable sets of regret comparator functions  $\Phi$  are the constant maps  $\mathcal{E}$  (corresponding to external regret), where all input actions are mapped to the same output action, and the "swap functions"  $\mathcal{I}$  (corresponding to internal regret $^1$ ), which contain all single swap maps  $f_{ij} : [M] \to [M]$  where  $f(i) = j$  and  $f(i') = i'$  for  $i' \neq i$ . Imposing these constraints on players in a game results in a (coarse) correlated equilibrium, which are instances of our notion of generalized equilibrium.
Definition 2 (Generalized  $(\Phi_A, \Phi_B)$ -equilibria). A  $(\Phi_A, \Phi_B)$ -equilibrium  $\varphi \in \Delta(\mathcal{A} \times \mathcal{B})$  in a two-player game is a joint distribution over action profiles  $(a, b)$  such that player  $A$  cannot increase their expected reward by deviating with some strategy in  $\Phi_A$  and player  $B$  cannot benefit by deviating with some strategy in  $\Phi_B$ .
In contrast to the  $\Phi$ -equilibria considered by (14; 13), here we allow constraints to be asymmetric between players. While many equilibrium notions for two-player games impose symmetric regret constraints on each player (e.g. Nash, correlated, and coarse correlated equilibria), this need not always be the case. In Section 3, we highlight Stackelberg equilibria as a motivating example for considering more general notions of asymmetric equilibria from the perspective of  $\Phi$ -regret, to determine when one should deviate from simultaneous no-swap play, and in Section 4 we characterize the maximum reward attainable against no-regret learners in terms of asymmetric equilibria.
We say that the value of a game  $G$  for player  $A$  of a certain equilibrium class  $(\Phi_A, \Phi_B)$ , denoted  $\mathsf{Val}_A(\Phi_A, \Phi_B)$  is the maximum reward obtainable by player  $A$  at some  $(\Phi_A, \Phi_B)$ -equilibrium (with  $\mathsf{Val}_B(\Phi_A, \Phi_B)$  defined symmetrically for player  $B$ ). Likewise, we say that the min-value of a game for a player and an equilibrium class, denoted by e.g.  $\mathsf{MinVal}_A(\Phi_A, \Phi_B)$  for player  $A$ , is the minimum reward for a player over all  $(\Phi_A, \Phi_B)$ -equilibria in a game. These capture the range of feasible average rewards under repeated play via  $(\Phi_A, \Phi_B)$ -regret dynamics.
Proposition 1. For a repeated game over  $T$  rounds where player  $A$  uses a no- $\Phi_A$ -regret algorithm and player  $B$  uses a no- $\Phi_B$ -regret algorithm, the average rewards obtained by each player are upper bounded by  $\mathsf{Val}_A(\Phi_A, \Phi_B) + o(1)$  and  $\mathsf{Val}_B(\Phi_A, \Phi_B) + o(1)$ , respectively, and lower bounded by  $\mathsf{MinVal}_A(\Phi_A, \Phi_B) - o(1)$  and  $\mathsf{MinVal}_B(\Phi_A, \Phi_B) - o(1)$ .
We consider an  $\varepsilon$ -approximate  $(\Phi_A, \Phi_B)$ -equilibrium to be a joint profile distribution where each constraint is satisfied up to additive error  $\varepsilon$ , connecting Definitions 1 and 2 as follows.
Proposition 2 (Convergence of no- $\Phi$ -regret dynamics to generalized equilibrium). Suppose after  $T$  rounds of a game where player  $A$  plays a no- $\Phi_A$ -regret algorithm and player  $B$  plays a no- $\Phi_B$ -regret algorithm, player  $A$  has average  $\Phi_A$ -regret  $\leq \varepsilon$  and player  $B$  has average  $\Phi_B$ -regret  $\leq \varepsilon$ . Let
<sup>1</sup>We refer to internal and swap regret interchangeably, as our focus is primarily on rates with respect to  $T$ .
$\varphi^t \coloneqq p_A^t \times p_B^t$  denote the joint distribution over both players' actions at time  $t$  and  $\varphi \coloneqq \frac{1}{T} \sum_{t=1}^{T} \varphi^t$  denote the time-averaged history over joint player action distributions. Then,  $\varphi$  is an  $\varepsilon$ -approximate  $(\Phi_A, \Phi_B)$ -equilibrium where
$$
\underset {(a, b) \sim \varphi} {\mathbb {E}} [ u _ {A} (a, b) ] \geq \underset {(a, b) \sim \varphi} {\mathbb {E}} [ u _ {A} (f _ {A} (a), b) ] - \varepsilon , a n d
$$
$$
\underset {(a, b) \sim \varphi} {\mathbb {E}} [ u _ {B} (a, b) ] \geq \underset {(a, b) \sim \varphi} {\mathbb {E}} [ u _ {B} (a, f _ {B} (b)) ] - \varepsilon
$$
for every possible deviation  $f_{A} \in \Phi_{A}, f_{B} \in \Phi_{B}$ . Likewise, if players  $A$  and  $B$  repeatedly play strategies corresponding to an  $(\Phi_{A}, \Phi_{B})$ -equilibrium, then player  $A$  is no-  $\Phi_{A}$ -regret and player  $B$  is no-  $\Phi_{B}$ -regret.
A general construction for no- $\Phi$ -regret algorithms is given in (13), which immediately implies feasibility of dynamics which converge to some instance of any class of  $(\Phi_A, \Phi_B)$ -equilibria in a game, possibly requiring agents to use differing algorithms if constraints are asymmetric. We show that a much stronger claim is true: such a pair of algorithms exists for any  $(\Phi_A, \Phi_B)$ -equilibrium  $\Psi$  in a game. These algorithms also satisfy a "best-of-both-worlds" property, in that they converge to  $\Psi$  when played together, yet simultaneously maintain their corresponding regret guarantees against arbitrary adversaries.
Theorem 1. Consider any game  $G$ . Suppose there exists a no- $\Phi_A$ -regret learning algorithm  $\mathcal{L}_A$  and a no- $\Phi_B$ -regret learning algorithm  $\mathcal{L}_B$ . For any particular  $(\Phi_A, \Phi_B)$ -equilibrium  $\Psi$  in a game  $G$ , there exists a pair of learning algorithms  $(\mathcal{L}_A^*(\Psi), \mathcal{L}_B^*(\Psi))$  such that:
- The empirical sequence of play when Player  $A$  uses  ${\mathcal{L}}_{A}^{ * }\left( \Psi \right)$  and Player  $B$  uses  ${\mathcal{L}}_{B}^{ * }\left( \Psi \right)$  converges to  $\Psi$  .
-  $\mathcal{L}_A^* (\Psi)$  and  $\mathcal{L}_B^* (\Psi)$  are no- $\Phi_A$ -regret and no- $\Phi_B$ -regret, respectively, against arbitrary adversaries.
Our approach is for the algorithms to initially implement a schedule of strategies which converges to  $\Psi$ . Yet, these algorithms also detect when their opponent disobeys the schedule by tracking their  $\Phi$ -regret with respect to  $\Psi$ , and after  $o(T)$  violations can deviate indefinitely to playing a standalone no- $\Phi$ -algorithm for all remaining rounds. Several of our results throughout make use of Theorem 1. Here we state a notable immediate implication for equilibrium selection.
Corollary 1.1. For any equilibrium scoring function  $\Gamma : \Delta(\mathcal{A} \times \mathcal{B}) \to \mathbb{R}$  with a unique optimum computable in finite time, there exists a pair of learning algorithms  $(\mathcal{L}_A^*, \mathcal{L}_B^*)$  such that:
- The empirical distribution when player  $A$  uses  $\mathcal{L}_A^*$  and player  $B$  uses  $\mathcal{L}_B^*$  converges to  $\operatorname{argmax}_{\Psi} \Gamma(\Psi)$ .
-  $\mathcal{L}_A^*$  and  $\mathcal{L}_B^*$  are no- $\Phi_A$ -regret and no- $\Phi_B$ -regret, respectively, against arbitrary adversaries.
Proof. First optimize  $\Gamma$  over  $\Psi$  in finite time to find the unique optimum; then apply Theorem 1 to the resulting desired equilibrium.
Corollary 1.1 allows for optimizing for objectives such as total welfare or min-max utility for both players, and imposing conditions on generalized equilibria beyond  $\Phi$ -regret constraints (e.g. product constraints for Nash equilibria) by assigning arbitrarily low scores to invalid strategy profiles.
In subsequent sections, we will primarily focus the function classes  $\mathcal{E}$  and  $\mathcal{I}$  corresponding to external and internal regret as mentioned above, as the well empty set  $\varnothing$  corresponding to unconstrained regret, and we additionally will consider the case when the game  $G$  is initially unknown. Before continuing, we note that each player's values for any  $(\Phi_A,\Phi_B)$ -equilibrium class can be expressed via a linear program, whose size is polynomial in the game dimensions for these function classes of interest.
Proposition 3. For any game  $G$  and constraints  $(\Phi_A, \Phi_B)$ , both  $\mathsf{Val}_A(\Phi_A, \Phi_B)$  and  $\mathsf{Val}_B(\Phi_A, \Phi_B)$  are computable via linear programs with  $MN$  variables and  $\mathrm{poly}(M, N, |\Phi_A|, |\Phi_B|)$  constraints. When  $\Phi_A$  and  $\Phi_B$  belong to  $\{\emptyset, \mathcal{E}, \mathcal{I}\}$ , the number of constraints is  $\mathrm{poly}(M, N)$ .