SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In general, these values for a player may differ under distinct notions of generalized equilibria; we give several examples of such concrete value separations in Appendix A (in Theorem 8). Our results in Section 3 illustrate a particularly stark separation of this form, in which it can often be dominant to deviate to a strategy where  $\Phi$ -regret constraints are violated.
3 Stability of no-swap-regret play
Here we address the following question: when is it the case that for two players in a game, it is an approximate (Nash) equilibrium for both players to play no-swap-regret strategies? More specifically, imagine a "metagame" where at the beginning of this repeated game, both players simultaneously announce and commit to a specific adaptive (and possibly randomized) algorithm they intend to run to select actions to play in the repeated game  $G$  for the next  $T$  rounds. In this metagame, for which games  $G$  is it an  $o(T)$ -approximate Nash equilibrium for both players to play a no-swap-regret learning algorithm?
Of course, the answer to this question might depend on which specific no-swap-regret learning algorithm the agents declare. We therefore attempt to understand the following two questions:
- Necessity: For which games  $G$  is it the case that there exists some pair of no-swap-regret algorithms which form a  $o(T)$ -approximate Nash equilibrium? (Equivalently, when is it never the case that playing no-swap-regret algorithms forms an approximate Nash equilibrium?)
- Sufficiency: For which games  $G$  is it the case that all pairs of no-swap regret algorithms form  $o(T)$ -approximate Nash equilibria?
A central element of our analysis will be to consider the Stackelberg equilibria of a game.
Definition 3 (Stackelberg Equilibria). The Stackelberg equilibrium of a game  $G$  for player  $A$  is the pair of strategies  $(\alpha, b)$  given by  $\operatorname{argmax}_{\alpha \in \Delta(A), b \in \mathsf{BR}(\alpha)} u_A(\alpha, b)$ , and the resulting expected utility for player  $A$  is the Stackelberg value of the game, denoted  $\mathsf{Stack}_A$ .  $\mathsf{Stack}_B$  is defined symmetrically.
We can relate Stackelberg equilibria to our notions of generalized equilibria.
Proposition 4. For any game  $G$ , we have that  $\mathsf{Stack}_A = \mathsf{Val}_A(\emptyset, \mathcal{I})$ .
Here, any joint distribution over action profiles where player  $B$  has zero swap regret constitutes a  $(\emptyset, \mathcal{I})$ -equilibrium for a game, and the optimal value for such an equilibrium for player  $A$  coincides with the Stackelberg value. Further, each equilibrium set can be optimized over via a linear program.
Note that each value definition allows for tiebreaking in favor of player  $A$ . In general, simply playing the Stackelberg strategy  $\alpha$  may not suffice to obtain  $\mathsf{Stack}_A$  if the best response for Player  $B$  is not unique. However, there are a number of mild conditions which are each sufficient to ensure the existence of an approximate Stackelberg strategy  $\alpha'$  which yields a unique best response for player  $B$  and obtains  $\mathsf{Stack}_A - \varepsilon$  for any  $\varepsilon > 0$ . Here we consider a minimal such condition (essentially, no action is weakly dominated without also being strictly dominated).
Assumption 1. In a game  $G$ , for each  $b$ , either  $\mathsf{BR}(\alpha) = \{b\}$  for some  $\alpha$ , or  $b \notin \mathsf{BR}(\alpha)$  for all  $\alpha$ . Likewise, for each  $a$ , either  $\mathsf{BR}(\beta) = \{a\}$  for some  $\beta$ , or  $a \notin \mathsf{BR}(\beta)$  for all  $\beta$ .
We provide an efficient algorithmic procedure to answer both questions of necessity and sufficiency for a specific game  $G$  satisfying Assumption 1. To do this, recall that when two players both employ no-swap regret strategies, they asymptotically (time-average) converge to some correlated equilibrium (here, corresponding to an  $(\mathcal{I},\mathcal{I})$ -equilibrium). On the other hand, by defecting from playing a no-swap regret strategy (while the other player continues playing their no-swap regret strategy), a player can guarantee their Stackelberg value for the game. Moreover, as shown by (10), this is the optimal (up to  $o(T)$  additive factors) best response to an opponent running a no-swap regret strategy. It thus suffices to understand how the utility a player might receive under a correlated equilibrium compares to the utility they receive under their Stackelberg strategy. For a fixed game  $G$ , let  $\text{Stack}_A = \text{Val}_A(\emptyset, \mathcal{I})$  be the Stackelberg value for the first player, and  $\text{Stack}_B = \text{Val}_B(\mathcal{I}, \emptyset)$  be the Stackelberg value for the second player. We have the following theorem.
Theorem 2. Fix a game  $G$  satisfying Assumption 1. The following two statements hold:
1. There exists some pair of no-swap-regret algorithms that form an  $o(T)$ -approximate Nash equilibrium in the metagame iff there exists a correlated equilibrium  $\varphi$  in  $G$  such that  $u_{A}(\varphi) = \mathsf{Stack}_{A}$  and  $u_{B}(\varphi) = \mathsf{Stack}_{B}$ .
2. Any pair of no-swap-regret algorithms form an  $o(T)$ -approximate Nash equilibrium in the metagame iff for all correlated equilibria  $\varphi$  in  $G$ ,  $u_A(\varphi) = \mathsf{Stack}_A$  and  $u_B(\varphi) = \mathsf{Stack}_B$ .
Moreover, given a game  $G$ , it is possible to efficiently (in polynomial time in the size of  $G$ ) check whether each of the above cases holds.
We obtain both claims by leveraging the construction in Theorem 1: best-case and worst-case correlated equilibria are feasible by some pair of no-swap-regret algorithms, and both players must simultaneously achieve close to their Stackelberg value for deviating to not be preferable. The characterization in Theorem 2 is algorithmically useful, but sheds little direct light on in which games or how often we would expect playing no-swap-regret to be an approximate equilibrium. It turns out that for many games, playing no-swap-regret is not an equilibrium; below we will show that for almost all games, if  $G$  does not have a pure Nash equilibrium, at least one player has an incentive to deviate to their Stackelberg strategy.
Definition 4. A property  $P$  of a game holds for almost all games if, given any game  $G$ , property  $P$  holds with probability 1 for the game  $G'$  formed by starting with  $G$  and perturbing each of the entries  $u_A(a_i, b_j)$  and  $u_B(a_i, b_j)$  by independent uniform random variables in the range  $[-\varepsilon, \varepsilon]$  (for any choice of  $\varepsilon$ ). In other words, the property holds for almost all choices of the 2MN utility values that define a game (with respect to the standard measure on this space).
Theorem 3. For almost all games  $G$ , if  $G$  does not have a pure Nash equilibrium, then there does not exist a pair of no-swap-regret algorithms which form a  $o(T)$ -approximate Nash equilibrium in the metagame for  $G$ .
Sketch. We can show that if a correlated equilibrium has the same utility for a player as their Stackelberg value (a consequence of Theorem 2), then the correlated equilibrium must be a convex combination of valid Stackelberg equilibria. In almost all games, both players have unique Stackelberg equilibria (and Assumption 1 holds), which implies that this correlated equilibrium must actually be the Stackelberg strategy for both players simultaneously. This implies that it is a pure Nash equilibrium (since one action in a generic Stackelberg equilibrium is always pure).
Note that although Theorem 3 holds for almost all games, there are some important classes of games (most notably, zero-sum games) in the measure zero subset omitted by this theorem statement that both a) do not have pure Nash equilibria and b) have the property that playing no-swap-regret algorithms is an approximate equilibrium in the metagame (in particular, for zero-sum games, the Stackelberg value collapses to the value of the unique Nash equilibrium). Still, Theorem 3 shows that there are very wide classes of games for which playing no-swap-regret algorithms is not stable from the perspective of the agents.
Finally Theorem 3 requires that we deviate to our Stackelberg strategy, which may be hard to compute. One can ask whether there are games where efficient deviations – e.g., to algorithms with weaker regret guarantees – lead to strictly more utility for the deviating player. In Appendix B we show this is true in the following sense: there are games  $G$  where  $\mathsf{Val}_A(\mathcal{E},\mathcal{I}) > \mathsf{MinVal}_A(\mathcal{E},\mathcal{I}) = \mathsf{Val}_B(\mathcal{I},\mathcal{I})$ . That is, in such a game player  $A$  can possibly strictly increase their utility by switching to a low-external-regret strategy, and such a switch will never decrease their utility.
4 Optimal rewards against no-regret learners
Here, we characterize the feasibility of optimizing one's reward against no-(external)-regret learners in terms of generalized equilibria. In contrast to the case of no-swap-regret learners, as shown by (10) there are games in which one can obtain  $\Omega(T)$  more than the Stackelberg value over  $T$  rounds against certain no-regret algorithms by playing an appropriate adaptive strategy. A major remaining open question from this line of work is determining the best feasible reward and corresponding optimal strategy against no-regret agents in arbitrary games. We resolve this question when considering the maximum over all possible no-regret algorithms: for any game, we can compute an upper bound on the feasible reward against any no-regret algorithm, and we show that there exists a specific no-regret algorithm against which we can obtain this reward via an efficiently implementable strategy.
Theorem 4. For any game  $G$ , there exists a no-regret algorithm  $\mathcal{L}$  and a strategy for player  $A$  such that the total reward of player  $A$  converges to  $\mathsf{Val}_A(\emptyset, \mathcal{E}) \cdot T \pm o(T)$  when player  $B$  uses  $\mathcal{L}$ .
Here we again make use of the construction from Theorem 1. Note that by Proposition 1, this is optimal over all no-external-regret algorithms, as any adaptive strategy constitutes a no- $\emptyset$ -regret
algorithm. By Proposition 3 we can identify the optimal  $(\emptyset, \mathcal{E})$ -equilibrium in  $\mathrm{poly}(M, N)$  time, which is sufficient to implement the algorithm  $\mathcal{L}$  as well as our own strategy efficiently.
However, we additionally show that this bound is often unattainable against many standard noexternal-regret algorithms. A property of many such algorithms (including Multiplicative Weights, Follow the Perturbed Leader, and Exp-3) is that they are mean-based, as formulated by (5).
Definition 5 (Mean-based learning). Let  $\sigma_{i,t}$  be the cumulative reward resulting from playing action  $i$  for the first  $t$  rounds. An algorithm  $\mathcal{L}$  is  $\gamma$ -mean-based if, whenever  $\sigma_{i,t} \leq \sigma_{j,t} - \gamma T$ , the probability that the algorithm selects action  $i$  in round  $t + 1$  is at most  $\gamma$ , for some  $\gamma = o(1)$ .
These algorithms resemble "smoothed" variants of Follow the Leader; they only play actions with probability higher than  $o(1)$  if their cumulative reward thus far is not too far from optimal, and hence never play dominated strategies. However, in general,  $(\emptyset, \mathcal{E})$ -equilibria may contain dominated strategies, as is also the case for coarse correlated equilibria. This allows us to show the following.
Theorem 5. Against any mean-based no-regret algorithm for player  $B$ , there are games where a  $T$ -round reward of  $(\mathsf{Val}_A(\emptyset, \mathcal{I}) + \varepsilon) \cdot T$  cannot be reached by any adaptive strategy for player  $A$ , for any  $\epsilon > 0$ . However, for this same game,  $\mathsf{Val}_A(\emptyset, \mathcal{E}) > \mathsf{Val}_A(\emptyset, \mathcal{I})$ .
In the appendix, we introduce a notion of "dominated-swapping external regret" which we use to characterize a class of games for which this holds, and we give a concrete example of such a game.
5 Learning Stackelberg equilibria in unknown games
Our results thus far have highlighted the primacy of the Stackelberg reward as an objective for repeated play against a learner: it is optimal against a no-swap learner and can sometimes be optimal against a mean-based learner, and it is almost always attainable against any learner. However, until now our strategies have assumed knowledge of the entire game, which may be unrealistic in many settings for which learning in games is relevant, particularly in terms of our opponent's rewards.
Here, we consider the challenge of learning the Stackelberg strategy via repeated play against a no-regret learner when only our own rewards are known, which is unaddressed in the literature to our knowledge; much of the prior work on learning Stackelberg equilibria assumes a query model, where one can observe the best response  $\mathsf{BR}(\alpha)$  played by an opponent for any queried mixed strategy  $\alpha$ . While here we cannot immediately observe the best response of an opponent, as their actions are selected by a learning algorithm which may be slow to adapt to changes in our behavior, we give a reduction from query algorithms of this form to strategies for choosing our actions which enable us to simulate queries to  $\mathsf{BR}(\alpha)$  against a learner, and we analyze the efficiency of this approach (in terms of rounds required for learning) under differing assumptions on the learner's algorithm.
For comparison of behavior across time horizons of varying lengths, it will be convenient for us to consider the notion of an anytime regret bound, which can be obtained from any base no-regret algorithm via doubling methods, as well as often via learning rate decay.
Definition 6 (Anytime regret algorithms). An algorithm is an anytime no- $\Phi$ -regret algorithm if it satisfies  $\mathsf{Reg}_{\Phi}(t) = O(t^{c})$  over the first  $t$  rounds, for some  $c < 1$  and any  $t \leq T$ .
We also recall the notion of adaptive regret; many no-external-regret algorithms such as Online Gradient Descent satisfy no-adaptive-regret bounds (see e.g. (25)).
Definition 7 (Adaptive regret algorithms). An algorithm  $\mathcal{L}$  for player  $B$  is a no-adaptive-  $\Phi$ -regret algorithm if  $\sup_{r,s\in [T]}\mathsf{Reg}_{\Phi}(\mathcal{L},[r,s])\leq O(T^{c})$ , for some  $c < 1$ , where  $\mathsf{Reg}_{\Phi}(\mathcal{L},[r,s]) = \max_{f\in \Phi}\sum_{t = r}^{s}u_{B}(a_{it},f(b_{jt})) - u_{B}(a_{it},b_{jt})$ .
A key distinction between adaptive-regret algorithms like OGD and mean-based algorithms like FTPL is in their "forgetfulness", and hence their ability to quickly adapt when rewards change. This has stark implications for the efficiency of learning Stackelberg equilibria, which we show can take exponentially longer against mean-based algorithms. As shown by (25), adaptive regret is closely connected with dynamic regret; we note that our results for adaptive-regret learners can also be extended to dynamic-regret learners.
5.1 Simulating query algorithms
Our approach will be to compute an  $\varepsilon$ -approximate Stackelberg strategy by simulating best response queries against a learner, after which point we can obtain an average reward approaching  $\mathsf{Stack}_A - \varepsilon$  in each subsequent round, calibrating  $\varepsilon$  in terms of  $T$  as desired. The query complexity for such algorithms can depend on the geometry of the best response regions of the game, and unfortunately, as shown by (24), there are "hard" game instances which require exponentially many queries. This issue arises when the best response regions may be quite small but non-empty, as even finding a point in each region is information-theoretically difficult. We restrict our attention to games in which this does not occur, and fortunately efficient query do algorithms exist for this case.
Assumption 2. For a game  $G$  and any action  $b_{i}$ , we have that
$$
\operatorname * {P r} _ {\alpha \sim \operatorname {U n i f} (\Delta (M))} \left[ b _ {i} \in \mathsf {B R} (\alpha) \right] \in \{0 \} \cup \left[ 1 / \operatorname {p o l y} (\varepsilon^ {- 1}), 1 \right],
$$
i.e. the volume of each BR region is either 0 or inverse polynomially large.
Proposition 5 ((19; 24)). For a game  $G$  satisfying Assumption 2, there is an algorithm which finds an  $\varepsilon$ -approximate Stackelberg strategy for player  $A$  with  $Q = \mathrm{poly}(M,N,1 / \varepsilon)$  queries to  $\mathsf{BR}(\alpha)$ .
We note that while such algorithms can indeed obtain tighter approximation guarantees in terms of  $\varepsilon$  (e.g.  $O(\log(1/\varepsilon))$ ), the query complexity is still inverse polynomially related to the best response region volumes; we consider only  $\varepsilon$ -approximate equilibria due to challenges which are inherent to the no-regret learning setting, as the precision with which we can simulate a query is constrained by our time horizon. The key to our approach is to play according to a mixed strategy  $\alpha$  until it saturates the relevant window of the learner's history, which induces them to play a best response. Against no-adaptive-regret learners, a best response will be induced quickly, as their regret is bounded even over small windows. However, for arbitrary no-regret learners, we have no promises other than the cumulative regret bound, which may require saturating the entire history for each query.
Theorem 6. Suppose  $\mathcal{E} \subseteq \Phi$ . For a game satisfying Assumption 2, there is an algorithm which finds an  $\varepsilon$ -approximate Stackelberg strategy in  $\mathrm{poly}(1 / \varepsilon)^Q$  rounds against any anytime-no- $\Phi$ -regret learner, and in  $\mathrm{poly}(Q / \varepsilon)$  rounds against any no-adaptive- $\Phi$ -regret calibrated for  $T = \Theta(\mathrm{poly}(Q / \varepsilon))$ , where  $Q = \mathrm{poly}(M, N, 1 / \varepsilon)$ .
5.2 Efficiency separations for mean-based and no-swap algorithms
We show here that the exponential dependence for mean-based algorithms is necessary: there exist games where learning the Stackelberg strategy requires exponentially many rounds against a particular mean-based algorithm. However, for the games we construct, we show that it is still possible to efficiently learn the Stackelberg strategy against a no-swap-regret learner.
Theorem 7. There is a distribution over games  $\mathcal{D}$  such that for a sampled game  $G$ :
- For any no-swap-regret learner used by the opponent, there is a strategy for the leader which yields an average reward of  $\mathsf{Stack}_A - \varepsilon$  in  $T = \mathrm{poly}(M / \varepsilon)$  rounds.
- There is a mean-based no-regret algorithm such that, when used by the opponent, there is no strategy for the leader which yields an average reward of  $\mathsf{Stack}_A - \varepsilon$  over  $T$  rounds unless  $T = \exp (\Omega (M))$ .
Our construction includes a set of actions for player  $B$  which are best responses to pure actions from player  $A$ , and one such pure strategy pair will necessarily constitute the Stackelberg equilibrium; identifying each best response suffices for player  $A$  to identify the Stackelberg strategy. The game also includes a number of safety actions for player  $B$ , which yield no reward for player  $A$  with any strategy, yet allow player  $B$  to "hedge" between multiple actions of player  $A$ . This poses a barrier to optimizing against a mean-based learner: the history must be heavily concentrated on a single action to observe the best response, and as such the history length must grow by a constant factor for each observation. However, against a no-swap-regret learner, it suffices for the optimizer to only play each action for a polynomially long window in order to identify the learner's best response; we track the accumulation of a "swap-regret buffer" for any other action and show that it cannot be too large, limiting the number of rounds it can be played when it is not a current best response.
[18] Zhiyi Huang, Jinyan Liu, and Xiangning Wang. Learning optimal reserve price against non-myopic bidders. CoRR, abs/1804.11060, 2018.
[19] Joshua Letchford, Vincent Conitzer, and Kamesh Munagala. Learning and approximating the optimal strategy to commit to. In Algorithmic Game Theory, 2009.
[20] Katrina Ligett and Georgios Piliouras. Beating the best nash without regret. SIGecom Exchanges, 10(1):23-26, 2011.
[21] Yishay Mansour, Mehryar Mohri, Jon Schneider, and Balasubramanian Sivan. Strategizing against learners in bayesian games. In Conference on Learning Theory, pages 5221-5252. PMLR, 2022.
[22] Panayotis Mertikopoulos, Christos Papadimitriou, and Georgios Piliouras. Cycles in adversarial regularized learning. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 2703-2717. SIAM, 2018.
[23] Hervé Moulin and J P Vial. Strategically zero-sum games: the class of games whose completely mixed equilibria cannot be improved upon. International Journal of Game Theory, 7:201-221, 1978.
[24] Binghui Peng, Weiran Shen, Pingzhong Tang, and Song Zuo. Learning optimal strategies to commit to. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI'19/IAAI'19/EAAI'19. AAAI Press, 2019.
[25] Lijun Zhang, Tianbao Yang, Rong Jin, and Zhi-Hua Zhou. Dynamic regret of strongly adaptive methods. In International Conference on Machine Learning, 2017.
A Properties and separations for generalized equilibria
Proof. The set of  $(\Phi_A, \Phi_B)$ -equilibria includes all strategy profile distributions in which both constraints are satisfied. If a player receives substantially more or less than the corresponding value, this would imply a violation of the regret constraints for at least one of the players' learning algorithms.
Proof. The statement follows by observing that
$$
\underset {(a, b) \sim \varphi} {\mathbb {E}} \left[ u _ {\{A, B \}} (a, b) \right] = \frac {1}{T} \sum_ {t = 1} ^ {T} \underset {(a, b) \sim \varphi^ {t}} {\mathbb {E}} \left[ u _ {\{A, B \}} (a, b) \right]
$$
$$
\underset {(a, b) \sim \varphi} {\mathbb {E}} [ u _ {A} (f _ {A} (a), b) ] = \frac {1}{T} \sum_ {t = 1} ^ {T} \underset {(a, b) \sim \varphi^ {t}} {\mathbb {E}} [ u _ {A} (f _ {A} (a), b) ]
$$
$$
\underset {(a, b) \sim \varphi} {\mathbb {E}} [ u _ {B} (a, f _ {B} (b)) ] = \frac {1}{T} \sum_ {t = 1} ^ {T} \underset {(a, b) \sim \varphi^ {t}} {\mathbb {E}} [ u _ {B} (a, f _ {B} (b)) ]
$$
which in turn are equivalent to the time-averaged utility of the play of players  $A$  and  $B$ , the time-averaged utility for player  $A$  under a deviation  $f_{A}$ , and the time-averaged utility for player  $B$  under a deviation  $f_{B}$ . Applying the definition of average  $\Phi$ -regret and applying the given bounds on the  $\Phi$ -regret yields the conclusion of the first direction. The reverse direction follows by reversing the steps.
A.3 Proof of Proposition 4
Proof. Observe that under any strategy  $(\alpha, b)$  where  $b \in \mathsf{BR}(\alpha)$ , player  $B$  cannot have any swap-regret, and so any Stackelberg equilibrium is also a  $(\emptyset, \mathcal{I})$ -equilibrium. Further, the marginal distributions over the optimal  $(\emptyset, \mathcal{I})$ -equilibrium for player  $A$  over each  $b_i$  cannot have distinct expected value for player  $A$ , as otherwise this would contradict optimality, and so an optimal  $(\emptyset, \mathcal{I})$ -equilibrium is either a single Stackelberg equilibrium or a mixture of Stackelberg equilibria with equal value.
Proof. By definition, the set of  $(\Phi_A,\Phi_B)$ -equilibria  $\varphi$  is a sub-polytope of  $\Delta (\mathcal{A}\times \mathcal{B})$  defined via the following linear constraints:
- For each  $f_A \in \Phi_A$ , we have that
$$
\sum_ {i \in [ M ]} \sum_ {j \in [ N ]} \varphi_ {i j} u _ {A} (a _ {i}, b _ {j}) \geq \sum_ {i \in [ M ]} \sum_ {j \in [ N ]} \varphi_ {i j} u _ {A} (a _ {f (i)}, b _ {j}).
$$
- For each  $f_{B} \in \Phi_{B}$ , we have that