SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {i \in [ M ]} \sum_ {j \in [ N ]} \varphi_ {i j} u _ {B} (a _ {i}, b _ {j}) \geq \sum_ {i \in [ M ]} \sum_ {j \in [ N ]} \varphi_ {i j} u _ {B} (a _ {f (i)}, b _ {j}).
$$
The value  $\mathsf{Val}_A(\Phi_A, \Phi_B)$  corresponds to the element  $\varphi$  of this polytope that maximizes  $\sum_{i \in [M]} \sum_{j \in [N]} \varphi_{ij} u_A(a_i, b_j)$ . Optimizing this linear function over the above polytope can be done in time poly $(M, N, |\Phi_A|, |\Phi_B|)$  via any linear program solver. Computing  $\mathsf{Val}_B(\Phi_A, \Phi_B)$  can be likewise done efficiently.
For player  $A$ , the regret comparator function sets  $\emptyset, \mathcal{E}$ , and  $\mathcal{I}$  contain  $0, M$ , and  $M^2$  elements respectively. In all three of these cases  $|\Phi_A| = \mathrm{poly}(M)$ ; likewise, in all three of these cases  $|\Phi_B| = \mathrm{poly}(N)$  (and thus we can efficiently compute these values when  $\Phi_A, \Phi_B \in \{\emptyset, \mathcal{E}, \mathcal{I}\}$ ).
A.5 Reward separations
We show that with respect to optimal values, these equilibrium classes are often distinct, and there exist games where values do not collapse. The separations we show here consider the equilibrium cases either where both players have identical regret constraints, or where player  $A$  is unconstrained. We note that while inspecting other cases, we identified similar examples for several other generalized equilibrium pairs, and we expect that strict separations exist between any distinct pair of generalized equilibria for the three regret notions we consider, in any direction not immediately precluded by the regret constraints. We are mostly interested in cases where  $B$  is constrained, and  $A$  may be constrained or unconstrained.
Theorem 8. For each of the following, there exists a  $4 \times 4$  game  $G$  with rewards in  $\{0,1,2\}$  where:
1.  $\mathsf{Val}_A(\emptyset, \mathcal{E}) > \mathsf{Val}_A(\emptyset, \mathcal{I}) > \mathsf{Val}_A(\mathcal{E}, \mathcal{E}) > \mathsf{Val}_A(\mathcal{I}, \mathcal{I})$
2.  $\mathsf{Val}_A(\emptyset, \mathcal{E}) > \mathsf{Val}_A(\mathcal{E}, \mathcal{E}) > \mathsf{Val}_A(\emptyset, \mathcal{I}) > \mathsf{Val}_A(\mathcal{I}, \mathcal{I})$
Proof. We prove both results by exhibiting a game with the desired chain of inequalities, which we found by searching random examples of  $4 \times 4$  games with values constrained in  $\{0,1,2\}$  and computing the various values of the games with a linear programming library. The numerical values are easy to check with computation. The game  $G_{1} := (M_{A_{1}}, M_{B_{1}})$  satisfies the conditions for the first chain of inequalities, and the game  $G_{2} := (M_{A_{2}}, M_{B_{2}})$  satisfies the conditions for the second chain of inequalities. First we instantiate the game  $G_{1}$ :
$$
M _ {A _ {1}} := \left[ \begin{array}{c c c c} 1 & 0 & 0 & 0 \\ 1 & 0 & 0 & 1 \\ 2 & 2 & 0 & 2 \\ 0 & 2 & 0 & 0 \end{array} \right] \qquad M _ {B _ {1}} := \left[ \begin{array}{c c c c} 0 & 2 & 0 & 0 \\ 1 & 1 & 1 & 0 \\ 1 & 0 & 2 & 0 \\ 1 & 0 & 0 & 1 \end{array} \right]
$$
The corresponding values for game  $G_{1}$  are simple to check:
1.  $\mathsf{Val}_A(\emptyset, \mathcal{E}) = 8/5$ .
2.  $\mathsf{Val}_A(\emptyset, \mathcal{I}) = 4/3$ .
3.  $\mathsf{Val}_A(\mathcal{E},\mathcal{E}) = 1$
4.  $\mathsf{Val}_A(\mathcal{I},\mathcal{I}) = 0$
Then we instantiate the game  $G_{2}$ :
$$
M _ {A _ {2}} := \left[ \begin{array}{c c c c} 2 & 0 & 1 & 0 \\ 2 & 1 & 1 & 0 \\ 0 & 2 & 1 & 2 \\ 2 & 0 & 2 & 1 \end{array} \right] \qquad M _ {B _ {2}} := \left[ \begin{array}{c c c c} 1 & 0 & 1 & 2 \\ 0 & 1 & 2 & 0 \\ 1 & 0 & 2 & 0 \\ 0 & 2 & 1 & 1 \end{array} \right]
$$
The corresponding values for game  $G_{2}$  are simple to check:
1.  $\mathsf{Val}_A(\emptyset, \mathcal{E}) = 13/7$ .
2.  $\mathsf{Val}_A(\mathcal{E},\mathcal{E}) = 12 / 7$
3.  $\mathsf{Val}_A(\emptyset, \mathcal{I}) = 5/3$ .
4.  $\mathsf{Val}_A(\mathcal{I},\mathcal{I}) = 4 / 3$
B Deviation to weaker regret classes
In Section 3, we show that if two players are playing no-swap-regret strategies against one another, it is often in the interest of each player to switch to playing their Stackelberg strategy (in particular, this is true whenever the game does not have a pure Nash equilibrium). However, as we later argue, learning ones Stackelberg strategy in such a game can be difficult. It is therefore natural to ask whether there are beneficial deviations to computationally efficient strategies. In particular, is it ever in a player's interest to weaken their regret benchmark, and e.g. switch from playing a no-swap-regret strategy to a no-external-regret strategy?
We give an example showing this is true in a fairly strong sense: we exhibit a game  $G$  where if player  $A$  switches from playing a no-swap-regret algorithm to any no-external-regret algorithm, their asymptotic utility never decreases and sometimes strictly increases – i.e., there is no downside to switching to an external regret algorithm (and potentially a high upside). We have the following theorem.
Theorem 9. There exists a game  $G$  where  $\mathsf{MinVal}_A(\mathcal{E},\mathcal{I}) \geq \mathsf{Val}_A(\mathcal{I},\mathcal{I})$  and  $\mathsf{Val}_A(\mathcal{E},\mathcal{I}) \geq \mathsf{Val}_A(\mathcal{I},\mathcal{I})$ .
Proof. Consider the game  $G$  specified by the two payoff matrices
$$
M _ {A} := \left[ \begin{array}{c c c} 0 & 0 & 2 \\ 0 & 0 & 1 \\ 0 & 1 & 1 \end{array} \right] \qquad M _ {B} := \left[ \begin{array}{c c c} 2 & 1 & 1 \\ 0 & 2 & 1 \\ 0 & 0 & 2 \end{array} \right].
$$
The corresponding values for this game are simple to compute:
1.  $\mathsf{Val}_A(\mathcal{I},\mathcal{I}) = \mathsf{MinVal}_A(\mathcal{I},\mathcal{I}) = 0.$
2.  $\mathsf{MinVal}_A(\mathcal{E},\mathcal{I}) = 0$
3.  $\mathsf{Val}_A(\mathcal{E},\mathcal{I}) = 1$
Proof. Let  $\varphi$  be the joint distribution over action pairs corresponding to  $\Psi$ . Let  $T$  denote the total number of steps we run the algorithm for; we will use  $t \leq T$  as a changing step size. Suppose both player  $A$  and player  $B$  know  $\varphi^2$ . We will define  $\mathcal{L}_A^*(\Psi)$  and  $\mathcal{L}_B^*(\Psi)$  in two phases: in the first phase,  $A$  and  $B$  trust their opponent and play according to deterministic sequences corresponding to approximations of  $\varphi$ . If either player violates the other's trust  $o(T)$  times, then the player defects to playing  $\mathcal{L}_A$  or  $\mathcal{L}_B$  respectively forever after.
First we elaborate upon the trusting phase. Both players consider windows of length  $\mathrm{Length}(t)$  which is monotonically increasing in  $t$  and also which grows sub-linearly in  $t$ . For concreteness, we pick a sub-linear monotonic increasing growth rate of  $\mathcal{O}(\sqrt{t})$  and describe how to implement the schedule of window lengths. We can keep track of a real-valued variable  $Z_{t}$  with  $Z_{1} = M \cdot N$ , and after each window completes, update it by  $Z_{t_{\mathrm{next}}} = Z_{t} + \frac{1}{2\sqrt{t}}$  where  $t$  is the step at the end of the window. To get an integral window length, we define  $\mathrm{Length}(t) \coloneqq \lfloor Z_t \rfloor$ . Thus in this case, the  $\mathrm{Length}(t)$  grows as  $O(\sqrt{t})$ , satisfying both conditions. Both players then compute a weighting instantiated with pairs of pure strategies by assigning  $c_{i} \coloneqq \lfloor \mathrm{Length}(t) \cdot \varphi_{i} \rfloor$  example pairs (each of weight  $1 / \mathrm{Length}(t)$ ) to pure strategy pair  $i \in [M \cdot N]$ . This weighted distribution approximates  $\varphi$  given  $\mathrm{Length}(t)$  samples. Note that the rounding approximation is feasible given only  $\mathrm{Length}(t)$  samples since  $\sum_{i=1}^{M \cdot N} c_{i} \leq \mathrm{Length}(t)$ . These pure strategy pair samples are then lexicographically
${}^{2}\varphi$  can be communicated from Player  $A$  to Player  $B$  during a burn-in phase of length  $> M \cdot  N$  ,the dimension of the discrete joint distribution over pure player strategy pairs.
ordered. Then, both players act according to the pure strategies in order, thereby (over the window) achieving an  $(M\cdot N) / \mathrm{Length}(t)\ell_1$  approximation to  $\varphi$ :
$$
\sum_ {i = 1} ^ {M \cdot N} \left| \varphi_ {i} - \frac {c _ {i}}{\operatorname {L e n g t h} (t)} \right| = \sum_ {i = 1} ^ {M \cdot N} \left| \varphi_ {i} - \frac {\lfloor \operatorname {L e n g t h} (t) \cdot \varphi_ {i} \rfloor}{\operatorname {L e n g t h} (t)} \right| \leq \frac {M \cdot N}{\operatorname {L e n g t h} (t)}.
$$
This process repeats for every window.
The distrustful phase occurs if one of the players does not follow the agreed-upon instructions  $T_{\mathrm{distrust}}$  times, where  $T_{\mathrm{distrust}}$  is taken to be  $o(T)$ . After this many violations, Player  $A$  defaults to playing  $L_{A}$  and likewise Player  $B$  defaults to playing  $L_{B}$  ever after.
We now show that this algorithm satisfies both conditions in the theorem statement. First, if both players use  $\mathcal{L}_A^* (\Psi)$  and  $\mathcal{L}_B^* (\Psi)$ , the play converges to  $\varphi$ , the joint distribution of play corresponding to  $\Psi$ . This point is immediate to observe since  $(M\cdot N) / \mathrm{Length}(t)\to 0$  as  $t\to \infty$  as  $\mathrm{Length}(t)$  is monotone increasing in  $t$ .
Now we prove that both players are no- $\Phi$ -regret with respect to any adversary. First we show no- $\Phi$ -regret for both players in the case where Player  $A$  plays  $\mathcal{L}_A^*(\Psi)$  and Player  $\bar{B}$  plays  $\mathcal{L}_B^*(\Psi)$ . Let  $\hat{\varphi}_t$  be the approximation to  $\varphi$  implemented over the window corresponding to final step  $t$ , and suppose that  $\|\varphi - \hat{\varphi}_t\|_1 < \varepsilon_t$ . Recalling the proof of Theorem 1, for Player  $A$  (and analogously for Player  $B$ ) we can bound
$$
\begin{array}{l} \left| \underset {(a, b) \sim \varphi} {\mathbb {E}} [ u _ {A} (a, b) ] - \underset {(a, b) \sim \hat {\varphi} _ {t}} {\mathbb {E}} [ u _ {A} (a, b) ] \right| = \left| (\varphi - \hat {\varphi} _ {t}) ^ {\top} u _ {A} \right| \\ \leq \left\| \varphi - \hat {\varphi} _ {t} \right\| _ {1} \cdot \left\| u _ {A} \right\| _ {2} \leq \varepsilon_ {t} \cdot C \cdot \sqrt {M \cdot N}, \\ \end{array}
$$
where here we interpret  $\varphi, \hat{\varphi}_t, u_A, u_B \in \mathbb{R}^{M \times N}$  as vectors over the space of all action pairs. Thus for this particular window, the overall gap from the expected reward for  $\varphi$  is  $\varepsilon_t \cdot C \cdot \sqrt{M \cdot N}$ .
Then we can similarly upper bound  $\mathbb{E}_{(a,b)\sim \hat{\varphi}_t}[u_A(f_A(a),b)]\leq \mathbb{E}_{(a,b)\sim \varphi}[u_A(f_A(a),b)] + \varepsilon_t\cdot C\cdot M\sqrt{N}$  for any choice of  $f_{A}\in \Phi_{A}$ :
$$
\begin{array}{l} (*) = \left| \underset {(a, b) \sim \varphi} {\mathbb {E}} \left[ u _ {A} (f _ {A} (a), b) \right] - \underset {(a, b) \sim \hat {\varphi} _ {t}} {\mathbb {E}} \left[ u _ {A} (f _ {A} (a), b) \right] \right| \\ = \left| \sum_ {k = 1} ^ {M} \sum_ {j = 1} ^ {N} \left(\hat {\varphi} _ {t} (k, j) - \varphi (k, j)\right) \cdot \sum_ {i = 1} ^ {M} f _ {A} \left(a _ {k}\right) _ {i} \cdot u (\cdot , b _ {j}) \right| \\ \leq \| \varphi - \hat {\varphi} _ {t} \| _ {1} \cdot \| [ f _ {A} (a _ {1}) ^ {\top} u _ {A} (\cdot , b _ {1}), \dots , f _ {A} (a _ {M}) ^ {\top} u _ {A} (\cdot , b _ {N}) ] \| _ {2} \\ \leq \varepsilon_ {t} \cdot \sqrt {M \cdot N} \cdot \max  _ {k, j} \| f _ {A} (a _ {k}) \| _ {2} \cdot \| u _ {A} (\cdot , b _ {j}) \| _ {2} \\ \leq \varepsilon_ {t} \cdot \sqrt {M \cdot N} \cdot 1 \cdot \sqrt {M \cdot C ^ {2}} \\ = \varepsilon_ {t} \cdot M \cdot \sqrt {N} \cdot C. \\ \end{array}
$$
Then recall that  $\varepsilon_t \leq \frac{M \cdot N}{\mathrm{Length}(t)}$ . Thus, overall, the average regret using due to the window is bounded by
$$
\frac {1}{\operatorname {L e n g t h} (t)} \operatorname {R e g} _ {\Phi} (\hat {\varphi} _ {t}, t) \leq \frac {1}{\operatorname {L e n g t h} (t)} \operatorname {R e g} _ {\Phi} (\varphi , t) + C _ {2} \cdot \frac {1}{\operatorname {L e n g t h} (t)},
$$
where  $C_2$  is another constant depending on  $C, M, N$  and where we use the shorthand  $\mathsf{Reg}_{\Phi}(\cdot, t)$  to denote the  $\Phi$ -regret over the window ending in step  $t$ . Now call  $\hat{\varphi}$  the strategy where the joint distribution  $\hat{\varphi}_t$  as previously defined gets played in each window  $t$ . Now we can bound the total  $\Phi$ -regret for  $\hat{\varphi}$  by the sum of the  $\Phi$ -regrets for each window (maximizing  $f_A \in \Phi_A$  over the steps in each window makes it more competitive than optimizing only one  $f_A$  over the whole length  $T$  sequence). Thus for total  $\Phi$ -regret, we have:
$$
\operatorname {R e g} _ {\Phi} (\hat {\varphi}, T) \leq \operatorname {R e g} _ {\Phi} (\varphi , T) + \text {N u m W i n d o w s} (T) \cdot C _ {2} \leq \operatorname {R e g} _ {\Phi} (\varphi , T) + o (T),
$$
$$
\text{NumWindows}(T):= \min_{\substack{\sum_{t = 1}^{k}\text{Length} (t)\geq T}}k.
$$
The last step follows since  $\mathsf{NumWindows}(T) \leq o(T)$ , because  $\mathsf{Length}(T) \leq o(T)$ .
Since we already know that the strategy  $\varphi$  is no- $\Phi$ -regret and  $\mathsf{Length}(T)$  is  $o(T)$ , we have proven that playing  $\hat{\varphi}$  is no- $\Phi$ -regret in the case where Player  $A$  plays  $\mathcal{L}_A^*(\Psi)$  and Player  $B$  plays  $\mathcal{L}_B^*(\Psi)$ .
The second case where the opposing player does not cooperate is easier: after at most  $o(T)$  steps, the player switches to an algorithm  $\mathcal{L}_A$  or  $\mathcal{L}_B$  respectively which is no-  $\Phi$ -regret and incurs only  $o(T)$  additional regret. Thus the theorem statement holds.
Proof. We begin with the first claim. To prove the forward direction, if there exists such a  $\varphi$ , then choose a pair of low-swap-regret algorithms  $(\mathcal{L}_A,\mathcal{L}_B)$  such that the time-averaged trajectory over  $T$  rounds is guaranteed to asymptotically converge to  $\varphi$  (this is possible by either the results of (11), or our Theorem 1). That is, if the two players play strategy  $\varphi_t$  at round  $t\in [T]$ , then  $\hat{\varphi} = \frac{1}{T}\sum_{t}\varphi_{t}$  satisfies  $||\hat{\varphi} -\varphi ||_{\infty} = o(1)$ . It follows that  $\sum_{t}u_{A}(\varphi_{t})\geq T\cdot u_{A}(\varphi) - o(T) = T\cdot \mathsf{Stack}_{A} - o(T)$  and therefore player  $A$  has at most an  $o(T)$  incentive to deviate (by (10), they can obtain at most  $\mathsf{Stack}_A T + o(T)$  against  $\mathcal{L}_B$ ). Symmetric logic holds for player  $B$ .
To prove the reverse direction, assume  $\mathcal{L}_A$  and  $\mathcal{L}_B$  are no-swap-regret algorithms such that  $(\mathcal{L}_A,\mathcal{L}_B)$  is an  $o(T)$ -approximate Nash equilibrium in the metagame. Since they are no-swap-regret, the time-averaged play of these two algorithms for  $T$  rounds must converge to an  $o(1)$ -approximate correlated equilibrium  $\hat{\varphi}_T$ ; moreover, since  $(\mathcal{L}_A,\mathcal{L}_B)$  is an  $o(T)$ -approximate Nash equilibrium,  $\hat{\varphi}_T$  must have the property that  $u_A(\hat{\varphi}_T) \geq \mathsf{Stack}_A - o(1)$  and  $u_B(\hat{\varphi}_T) \geq \mathsf{Stack}_B - o(1)$ . Taking the limit as  $T \to \infty$  and selecting a convergent subsequence of the  $\hat{\varphi}_T$ , this shows there must exist a correlated equilibrium  $\varphi$  with the desired properties.
Likewise, similar logic proves the second claim with the following modifications. In the forward direction, we can now choose any pair of low-swap-regret algorithms  $(\mathcal{L}_A,\mathcal{L}_B)$ , and any correlated equilibrium  $\varphi$  they asymptotically converge to is guaranteed to have the property that  $u_{A}(\varphi) = \mathsf{Stack}_{A}$  and  $u_{B}(\varphi) = \mathsf{Stack}_{B}$ . In the reverse direction, since any correlated equilibrium is implementable by some pair of low-regret algorithms (again, by Theorem 1), the same logic shows that all correlated equilibria  $\varphi$  must satisfy  $u_{A}(\varphi) = \mathsf{Stack}_{A}$  and  $u_{B}(\varphi) = \mathsf{Stack}_{B}$ .
Finally, to see that these two conditions are efficiently checkable, note that: i. the two values  $\mathsf{Stack}_A$  and  $\mathsf{Stack}_B$  are efficiently computable given the game  $G$ , and ii. the set of correlated equilibria  $\varphi$  form a convex polytope defined by a small  $(\mathrm{poly}(N,M))$  number of linear constraints (see Proposition 3). In particular, since  $u_{A}(\varphi)$  and  $u_{B}(\varphi)$  are simply linear functions of  $\varphi$  for a given game  $G$ , we can efficiently check whether there exists any point in this polytope where  $u_{A}(\varphi) = \mathsf{Stack}_{A}$  and  $u_{B}(\varphi) = \mathsf{Stack}_{B}$ .
Proof. We will show that (for almost all games  $G$ ) if there is a correlated equilibrium  $\varphi$  such that  $u_{A}(\varphi) = \mathsf{Stack}_{A}$  and  $u_{B}(\varphi) = \mathsf{Stack}_{B}$ , then there exists a simultaneous unique Stackelberg equilibrium for both players in  $G$ , which must be a pure Nash equilibrium. Combined with Theorem 2, this implies the theorem statement.
We will rely on the following fact: in almost all games  $G$ , both players have a unique Stackelberg strategy. To see this, consider the following method for computing  $A$ 's Stackelberg strategy. For each pure strategy  $b_{j}$  for player  $B$ , consider the convex set  $A_{j} \subseteq \Delta(\mathcal{A})$  containing the mixed strategies for player  $A$  which induce  $b_{j}$  as a best response (i.e.,  $A_{j} = \{\alpha \in \Delta(\mathcal{A}) \mid b_{j} \in \mathsf{BR}(\alpha)\}$ ). Then, for each  $j \in [N]$ , compute the strategy  $\alpha_{j} \in A_{j}$  which maximizes  $u_{A}(\alpha_{j}, b_{j})$ . The Stackelberg value  $\mathsf{Stack}_{A}$  is then given by  $\max_{j} u_{A}(\alpha_{j}, b_{j})$ . In order for this to stem from a unique Stackelberg equilibrium, it is enough that: 1. the maximum utility is not attained by more than one  $j$ , and 2. for each  $j$ , the optimizer  $\alpha_{j} \in A_{j}$  is unique.
These two properties are guaranteed to hold in almost all games. To see this, first note that the convex sets  $A_{j}$  are determined entirely by the utilities  $u_{B}$ , so we will treat these as fixed. Now, given any convex set  $A_{j}$ , the extremal point in a randomly perturbed direction will be unique with probability 1 – but since  $\alpha_{j}$  is simply the extremal point of  $A_{j}$  in the direction specified by  $u_{A}(\cdot, b_{j})$  (which is a randomly perturbed direction), so  $\alpha_{j}$  is unique in almost all games. Finally, if we perturb the magnitude of each of the utilities  $u_{A}(\cdot, b_{j})$  (keeping the direction the same), the maximizer  $\max_{j} u_{A}(\alpha_{j}, b_{j})$  will also be unique almost surely.
Let  $(\alpha_A, b_A)$  be the Stackelberg equilibrium for player  $A$  and let  $(a_B, \beta_B)$  be the Stackelberg equilibrium for player  $B$ . Now, consider the aforementioned correlated equilibrium  $\varphi \in \Delta(A \times B)$ . We will begin by decomposing it into its marginals based on its first coordinate; that is, we will write  $\varphi = \sum_{i=1}^{M} \lambda_i(a_i, \beta_i)$  for some mixed strategies  $\beta_i \in \Delta(\mathcal{B})$  and weights  $\lambda_i$  (with  $\sum_i \lambda_i = 1$ ). By the definition of correlated equilibria, note that each  $a_i$  belongs to  $\mathsf{BR}(\beta_i)$ . But this means that  $u_B(a_i, \beta_i) \leq \mathsf{Stack}_B$ , with equality holding iff  $(a_i, \beta_i) = (a_B, \beta_B)$  (due to uniqueness of Stackelberg). Therefore, in order for  $u_B(\varphi) = \mathsf{Stack}_B$ , we must have that  $\varphi = (a_B, \beta_B)$ . By symmetry, we must also have that  $\varphi = (\alpha_A, b_A)$ . If both these are true, then  $\varphi$  is a pure strategy correlated equilibrium of the game, and is hence a pure strategy Nash equilibrium (and moreover, is also the Stackelberg equilibrium for both  $A$  and  $B$ ).
F Proof of Theorem 4
Proof. By Theorem 1, there is a pair of  $\emptyset$ -regret and  $\mathcal{E}$ -regret algorithms  $\mathcal{L}_A^*$  and  $\mathcal{L}_B^*$  which converge to a  $(\emptyset, \mathcal{E})$ -equilibrium for which player  $A$  obtains  $\mathsf{Val}_A(\emptyset, \mathcal{E})$ . By Proposition 1, this is optimal over all no-external-regret algorithms, as any adaptive strategy constitutes a no- $\emptyset$ -regret algorithm. By Proposition 3 we can identify the optimal  $(\emptyset, \mathcal{E})$ -equilibrium in  $\mathrm{poly}(M, N)$  time, which is sufficient to implement the algorithms  $\mathcal{L}_A^*$  and  $\mathcal{L}_B^*$  efficiently for any desired  $T$ .
G Dominated-swapping external regret bounds for mean-based algorithms
For the following proof (of Theorem 10), we introduce the following notion of dominated-swapping external regret, a tighter upper bound on the behavior of mean-based algorithms than the standard no-external-regret guarantee.
Definition 8 (Dominated-swapping external regret). For a game  $G$ , let  $D(G)$  be the set of dominated strategies for player  $B$ , i.e.  $b_{i} \in D(G)$  if  $b_{i} \notin \mathsf{BR}(\alpha)$  for all  $\alpha \in \Delta(M)$ . For  $j, k \in [N]$  define  $g_{jk}(b_{i})$  as:
$$
g _ {j k} (b _ {i}) = \left\{ \begin{array}{l l} b _ {j} & b _ {i} \notin D (G) \\ b _ {k} & b _ {i} \in D (G) \end{array} \right.
$$
i.e.  $g_{jk}(b_i)$  swaps  $b_i$  to  $b_k$  if  $b_i$  is dominated and plays  $b_j$  otherwise. Let  $\mathcal{E}_{D(G)} = \{g_{jk} : j, k \in [N]\}$  be the set of dominated-swapping external regret comparators.
This definition leads to the following tighter upper bound on what is achievable against a mean-based no-regret algorithm.
Theorem 10. For any game  $G$  and any mean-based no-regret algorithm used by player  $B$ , there is no strategy for which the average reward of player  $A$  converges to  $\mathsf{Val}_A(\emptyset, \mathcal{E}_{D(G)}) + \varepsilon$ , for any  $\varepsilon > 0$ .
Proof. First, we observe that mean-based algorithms will never play a dominated strategy  $b_{i} \in D(G)$  in more than  $o(T)$  rounds. As  $b_{i}$  is dominated, there is some  $\delta > 0$  such that for every  $\alpha \in \Delta(M)$ , there is some  $b_{j}$  where  $u_{B}(\alpha, b_{j}) \geq u_{B}(\alpha, b_{i}) + \delta$ . Let  $\alpha_{t}$  denote the empirical distribution of player  $A$ 's actions up to time  $t$ . After some window of  $O(\gamma T) = o(T)$  rounds we will have the cumulative rewards  $\sigma_{i,t}$  and  $\sigma_{j,t}$  satisfy  $\sigma_{i,t} < \sigma_{j,t} - \delta t < \sigma_{j,t} - \gamma T$  under any  $\alpha_{t}$  for some  $b_{j}$  in each subsequent round, and so  $b_{i}$  will never be played in more than  $o(T)$  rounds.
We can also see that any such no- $\mathcal{E}$ -regret algorithm is a no- $\mathcal{E}_{D(G)}$ -regret algorithm. Suppose such an algorithm had  $\mathcal{E}_{D(G)}$ -regret  $\epsilon T$ , for  $\epsilon > 0$ ; then, there is some  $g_{jk}$  for which  $U_B(\alpha_T, g_{jk}(\beta_T)) \geq U_B(\alpha_T, \beta_T) + \epsilon$ . By the  $\mathcal{E}$ -regret guarantee this cannot occur if  $j = k$ , as any such function  $g_{jj}$  is
equivalent to the fixed deviation rule for  $b_{j}$ . However, if this occurs for  $j \neq k$ , such an algorithm must have played dominated strategies in a total  $\Omega(\epsilon T)$ . This contradicts our assumption that no dominated strategy  $b_{i}$  is played in more than  $o(T)$  rounds, and so any mean-based no- $\mathcal{E}$ -regret algorithm is also a no- $\mathcal{E}_{D(G)}$ -regret algorithm, against which player  $A$  cannot obtain average reward which converges to any amount higher than  $\mathsf{Val}_A(\emptyset, \mathcal{E}_{D(G)}) + o(1)$ .
H Proof of Theorem 5
b1	b2	b3
a1	1,1	0,0	3,0
a2	0,0	1,1	0,0
Figure 1: Game where  $\mathsf{Val}_A(\emptyset, \mathcal{E}) > \mathsf{Val}_A(\emptyset, \mathcal{I}) = \mathsf{MBrew}_A$