SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Let  $\mathsf{MBRew}_A$  denote the maximal reward obtainable by player  $A$  when player  $B$  uses a mean-based algorithm. Observe that  $b_{3}$  is dominated for player  $B$ , and thus cannot be included in any  $(\emptyset, \mathcal{I})$ -equilibrium (by Theorem 10). Further, it will never be played by a mean-based learner for more than  $o(T)$  rounds, as for any distribution over  $a_{1}$  and  $a_{2}$  the best response is either  $b_{1}$  or  $b_{2}$ . As such, both  $\mathsf{Val}_A(\emptyset, \mathcal{I})$  and  $\mathsf{MBRew}_A$  are at most  $1 + o(1)$ ; a reward of  $1 - o(1)$  is obtainable by committing to either  $a_{1}$  or  $a_{2}$  for each round. However, we can see that the optimal  $(\emptyset, \mathcal{E})$ -equilibrium  $p$  for player  $A$  includes positive mass on  $(a_{1}, b_{3})$ , and yields an average reward of  $\mathsf{Val}_A(\emptyset, \mathcal{E}) = 2$  for player  $A$ . Let  $p_{1}$  be the probability on  $(a_{1}, b_{1})$ , let  $p_{2}$  be the probability on  $(a_{2}, b_{2})$ , let  $p_{3}$  be the probability on  $(a_{1}, b_{3})$ , and let  $p_{0}$  be the remaining probability. The reward for player  $A$  is given by:
$$
\operatorname {R e w} _ {A} (p) = p _ {1} + p _ {2} + 3 p _ {3}
$$
and  $p$  defines a  $(\emptyset, \mathcal{E})$ -equilibrium if
$$
\operatorname {R e w} _ {B} (p) \geq \operatorname {R e w} _ {B} \left(p \rightarrow b _ {i}\right)
$$
for each  $b_{i}$ , which holds if:
$$
p _ {1} + p _ {2} \geq p _ {1} + p _ {3};
$$
$$
p _ {1} + p _ {2} \geq p _ {2};
$$
$$
p _ {1} + p _ {2} \geq 0.
$$
Only the first constraint is non-trivial, and so the optimal  $(\emptyset, \mathcal{E})$ -equilibrium for player  $A$  occurs by maximizing  $p_1 + p_2 + 3p_3$  subject to  $p_2 \geq p_3$ , which yields a probability of 0.5 for both  $p_2$  and  $p_3$  (and 0 for  $p_1$  and  $p_0$ ), as well as an average reward of 2. As such, player  $A$  cannot obtain a reward approaching  $\mathsf{Val}_A(\emptyset, \mathcal{E})$ , as their per-round reward is at most  $1 + o(1)$ .
I Proof of Theorem 6
Proof. We recall that the SU algorithm from (19) finds initial points  $\alpha^{*}(b_{i})$  in each best response region via random sampling, which takes  $1 / \mathrm{poly}(\varepsilon^{-1})$  queries in expectation. Then, upon calibrating for  $O(\log(1/\varepsilon))$  bits of precision SU makes poly $(M, N, \log(1/\varepsilon))$  queries, each of which can be taken to be a point on some grid of spacing  $1 / \mathrm{poly}(\varepsilon^{-1})$  within the simplex by the precision condition. The computed approximate Stackelberg strategy is then the optimal such point on the grid.
We first describe our strategy for simulating each query against an arbitrary anytime-no-regret learner, as  $\mathcal{E} \subseteq \Phi$ , we can restrict to considering only no-external-regret learners, as these regret constraints will always be satisfied. To implement a query  $q$ , greedily play the action whose historical frequency of play is the furthest below its target frequency in  $q$ . After  $O(\mathrm{poly}(1 / \varepsilon))$  rounds, the historical distribution will be within  $1 / \mathrm{poly}(\varepsilon^{-1})$  of  $q$ , and continuing the greedy selection strategy indefinitely will ensure that the history remains in a  $1 / \mathrm{poly}(\varepsilon^{-1})$ -ball around  $q$ . Let  $t_q$  be the time at which this occurs. After maintaining the greedy strategy for  $q$  for an additional  $\omega(t_q^c)$  rounds, the anytime regret bound ensures that most frequently played item must indeed be the best response response to some point in the ball around  $q$ , provided that this ball is contained entirely inside some best response region  $R_j$ . For the sampling step, a taking sufficiently small grid (but still  $1 / \mathrm{poly}(\varepsilon^{-1})$ ) ensures that random sampling still suffices to find a point a point in each best response region even if our
queries may be adversarially perturbed to neighboring points on the grid, as each region is convex and has volume at least  $1 / \mathrm{poly}(\varepsilon^{-1})$ . To address the issue for the line search steps, it suffices to take an additional step along each search conducted by SU before termination, where we then take each hyperplane boundary estimate to be one step inward along the grid from where our search terminates, maintaining a buffer between each hyperplane estimate in which all our points of uncertainty must lie. This adds at most a constant factor to our query complexity, and impacts our approximation by  $1 / \mathrm{poly}(\varepsilon^{-1})$ , which then yields us a runtime of  $\mathrm{poly}(1 / \varepsilon)^Q$  rounds.
For the case of a no-adaptive-regret learner, suppose such an algorithm is calibrated for  $T = O(Q^{C_1}(1 / \varepsilon)^{C_2})$ ; then, over any window of length  $W$  its regret is at most  $O\left((Q^{C_1}(1 / \varepsilon)^{C_2})^c W^{-1}\right)$ . Taking  $W = \omega ((Q^{C_1}(1 / \varepsilon)^{C_2})^c)$  yields a per-round regret of at most  $o(1)$  over the window, and so an algorithm must play a best response in  $W - o(W)$  of the rounds. For sufficiently large  $C_1$  and  $C_2$ , each  $W$  is large enough to yield the same precision we required for the anytime case, where now we greedily play the action whose frequency is furthest below its target since our previous query terminated, which allows us to again simulate the  $O(Q)$  queries in  $\mathrm{poly}(Q / \varepsilon)$  rounds (accounting for the robustness checks) while yielding  $\Theta (WQ) = o(T)$ .
J Proof of Theorem 7
Proof. Our game consists of  $M$  actions  $\mathcal{A}$  for the optimizer, and  $N = 2M + \binom{M}{2}$  actions for the learner, which are divided into  $M$  primary actions  $\mathcal{B}$ ,  $M$  secondary actions  $\mathcal{S}$ , and  $\binom{M}{2}$  safety actions  $\mathcal{Y}$ .
If we restrict the learner to only playing primary actions, the game somewhat resembles a coordination game, where each pure strategy pair  $(a_{j},b_{j})$  is a Nash equilibrium. However, the set  $\mathcal{B}$  is comprised of both undominated actions  $\mathcal{B}_U$  and dominated actions  $\mathcal{B}_D$ , which are unknown to the optimizer, and where each  $b_{j} \in \mathcal{B}_{d}$  is weakly dominated by the secondary action  $s_j$ . The optimizer receives reward 0 whenever the learner plays a secondary action, and so the challenge for the optimizer is to identify the pair  $(a_j,b_j)$  which maximizes  $u_A(a_j,b_j)$ , for  $b_{j} \in \mathcal{B}_{D}$ , which will be the Stackelberg equilibrium. Further, the safety actions  $y_{ij}$  essentially allow the learner to hedge between two actions; this does not pose substantial difficulty for the optimizer when the learner is no-swap-regret, yet creates an insurmountable barrier for learning the Stackelberg equilibrium in sub-exponential time against a mean-based learner.
An instance of a game  $G \in \mathcal{G}$  is specified by the partition of  $\mathcal{B}$  into  $\mathcal{B}_U$  and  $\mathcal{B}_D$ . There is an action  $s_j \in S$  for each  $j$ , and for each pair  $(i,j)$  with  $i < j$  there is an action  $y_{ij} \in \mathcal{V}$ . The rewards for a game  $G$  are as follows. For any strategy pair, the optimizer's utility is given by:
-  $u_A(a_j, b_j) = j / M$  for  $b_j \in \mathcal{B}$ ;
-  $u_A(a_i, b_j) = 0$  for  $b_j \in \mathcal{B}$  and with  $i \neq j$ ;
-  $u_A(a_i, s_j) = 0$  for any  $s_j \in S$ ;
-  $u_A(a_i, y_{jk}) = 0$  for any  $y_{jk} \in \mathcal{V}$ ;
and the learner's utility is given by:
- For  $b_{j} \in \mathcal{B}_{U}$ :
$$
\begin{array}{l} - u _ {B} \left(a _ {j}, b _ {j}\right) = 1; \\ - u _ {B} \left(a _ {i}, b _ {j}\right) = 0 \text {f o r} i \neq j; \\ \end{array}
$$
- For  $b_{j} \in \mathcal{B}_{D}$ :
- For  $s_j \in S$ :
$$
\begin{array}{l} - u _ {B} \left(a _ {i}, b _ {j}\right) = 0 \text {f o r a n y} i; \\ - u _ {B} \left(a _ {j}, s _ {j}\right) = 1 \text {i f} b _ {j} \in \mathcal {B} _ {D}; \\ - u _ {B} \left(a _ {j}, s _ {j}\right) = 0 \text {i f} b _ {j} \in \mathcal {B} _ {U}; \\ \end{array}
$$
$$
- u _ {B} \left(a _ {i}, s _ {j}\right) = 0 \text {f o r} i \neq j;
$$
- For  $y_{ij} \in \mathcal{V}$ :
$$
\begin{array}{l} - u _ {B} \left(a _ {i}, y _ {i j}\right) = u _ {B} \left(a _ {j}, y _ {i j}\right) = 2 / 3; \\ - u _ {B} \left(a _ {k}, y _ {i j}\right) = 0 \text {f o r} i, j \neq k. \\ \end{array}
$$
We assume that  $\mathcal{B}_U$  is non-empty, and so there is some optimal pure Nash equilibrium  $(a_i^*, b_i^*)$  which yields a reward of  $i / M$ ; it is simple to check that this is also the Stackelberg equilibrium.
Optimizing against no-swap learners. First, we give a method for matching the Stackelberg value against an arbitrary no-swap-regret learner, which corresponds to the pair  $(a_{j},b_{j})$  for the largest value  $j$  such that  $b_{j}\in \mathcal{B}_{U}$ . Consider a no-swap-regret learner which obtains a regret bound of  $\tau = O(T^{c})$  over  $T$  rounds. Let  $\mathsf{SR}_t(b,b')$  for any learner actions  $b$  and  $b^{\prime}$  denote the  $t$ -round cumulative swap regret between  $b$  and  $b^{\prime}$ , i.e. the total change in reward which would have occurred if  $b^{\prime}$  was played instead for each of the first  $t$  rounds in which  $b$  was played. To model the behavior of an arbitrary no-swap-regret learner, we disallow the learner from taking any action which would increase  $\mathsf{SR}_t(b,b')$  above  $\tau$ , given the loss function for the current round, and otherwise allow the action to be chosen adversarially. While our model is deterministic for simplicity, it is straightforward to extend to the analysis to algorithms whose regret bounds hold in only expectation, e.g. by considering a distribution over values of  $\tau$  in accordance with Markov's inequality (as no algorithm can have negative expected regret against arbitrary adversaries) and considering our expected regret to the Stackelberg value.
Our strategy for the optimizer is:
- For each  $i \in [M]$ , play  $a_i$  until either  $b_i$  or  $s_i$  is observed at least  $t^* > \tau$  times;
- Return  $a_{i}^{*}$  for the largest  $i$  such that  $b_{i}$  is observed  $t^{*}$  times.
We show that this takes at most  $O(T^{c} \cdot M^{3})$  rounds. Once  $a_{i}^{*}$  is identified, we can commit to playing it indefinitely, at which point the learner must play  $b_{i}^{*}$  in all but at most  $O(T^{c} \cdot \mathrm{poly}(M))$  rounds, and so with  $T = O(\mathrm{poly}(M / \varepsilon))$  rounds we can increase the total fraction of rounds in which  $(a_{i}^{*}, b_{i}^{*})$  is played to  $1 - \varepsilon$ , which yields the desired average reward bound.
The key to analyzing the runtime of our strategy is to consider the "buffer" in regret between any pair of actions before the threshold of  $\tau$  is reached, which enables us to the bound the number of rounds in which instantaneously suboptimal actions are played. Note that prior the start of window  $i$  (where  $a_{i}$  is played), both  $b_{i}$  and  $s_i$  obtain reward 0 in each round, and as such cannot decrease their expected regret relative to any other action, as all rewards in the game are non-negative. Further, for any previous window  $j$ , both  $b_{i}$  and  $s_i$  incur regret of 1 with respect to either  $b_{j}$  or  $s_j$ , as well as between the suboptimal and optimal action in window  $i$ , and thus cannot be observed more than  $\tau$  times in the window. As such, observing  $b_{i}$  at least  $t^*$  times in window  $i$  indicates that  $b_{i} \in \mathcal{B}_{U}$  (and likewise observing  $b_{i}$  at least  $t^*$  times indicates that  $b_{i} \in \mathcal{B}_{D}$ ).
Any action  $b \neq \mathsf{BR}(a_i)$  will incur positive swap regret with respect to  $\mathsf{BR}(a_i)$ , and cannot be played in window  $i$  once  $\mathsf{SR}_t(b, \mathsf{BR}(a_i)) \geq \tau$ . Each action begins with  $\mathsf{SR}_1(b, \mathsf{BR}(a_i)) = 0$  at time  $t = 1$ ; for each of the learner's actions, we consider the rate at which its buffer decays, as well as instances in which swap regret can decrease:
- Previously optimal  $b \in \mathcal{B} \cup S \setminus \mathsf{BR}(a_i)$ : actions in  $\mathcal{B} \cup S$  can only accumulate negative swap regret with respect to  $\mathsf{BR}(a_i)$  during rounds in which they were previously optimal; any previous optimum  $b = \mathsf{BR}(a_j)$  for  $j < i$  was played at most  $t^*$  times during window  $j$ , and so we have that  $\mathsf{SR}_t(b, \mathsf{BR}(a_i)) \geq -t^*$ .
- All  $b \in \mathcal{B} \cup \mathcal{S} \setminus \mathsf{BR}(a_i)$ : ignoring any previously accumulated regret buffer, each of these  $2M - 1$  actions can be played at most  $\tau$  rounds during window  $i$  before exhausting their initial buffer. Accounting for possible previous optima with  $\mathsf{SR}_t(b, \mathsf{BR}(a_i)) < 0$ , the number of rounds during window  $i$  in which some  $b \in \mathcal{B} \cup \mathcal{S} \setminus \mathsf{BR}(a_i)$  is played is at most  $M t^* + (2M - 1)\tau$ .
- Safety actions  $y_{jk} \in \mathcal{V}$ : Suppose neither  $a_j$  or  $a_k$  have been played yet by the optimizer, including in the current window. As was the case for other actions which have never yielded positive instantaneous reward,  $y_{jk}$  can be played at most  $\tau$  times before  $\mathsf{SR}_t(y_{jk},\mathsf{BR}(a_i)) \geq \tau$ . If  $j = i$ , i.e., this is the first window in which  $y_{jk}$  obtains positive instantaneous reward, the per-round regret is 1/3, and so at it can be played for most  $3\tau$  rounds. Further,  $y_{jk}$  a obtains a regret of  $-2/3$  with respect to  $\mathsf{BR}(a_k)$ . If  $k = i$  and the window for  $a_j$  has already been completed,  $y_{jk}$  can be played for at most  $9\tau$  rounds, as initially we have that  $\mathsf{SR}_t(y_{jk},\mathsf{BR}(a_i)) \geq -2\tau$ , which again increases by 1/3 per round. We then have that the total amount of rounds with safety actions played during window  $i$  is at most  $(12M + M^2)\tau$ , as there are fewer than  $M^2$  total safety actions, and fewer than  $M$  in each of the latter cases.
This yields a per-window runtime across all actions of at most  $M t^{*} + (M^{2} + 10M - 1)\tau$ , which is  $O(T^{c}\cdot M^{3})$  across all windows, and so we obtain the desired result for optimizing against arbitrary no-swap-regret learners.
Optimizing against mean-based learners. Here, we show that there are mean-based no-regret algorithms for which exponentially many rounds are required for an optimizer to approximate the Stackelberg value against a learner. When considering horizons which are superpolynomial in the parameters of the game, it is most natural to consider algorithms with regret bounds which are non-trivial for smaller horizons, as well as an anytime variant of the mean-based property. We define an extension of the classical Multiplicative Weight Updates algorithm (MWU; see (2) for a survey), called Rounded Mean-Based Doubling, which inherits both properties in the anytime setting.
Algorithm 1 Rounded Mean-Based Doubling (RMBD)
Initialize and run MWU for  $T_{1}:= 2$  rounds and  $n$  actions.  
Let  $T_{2}:= 2T_{1}$  and  $i:= 2$ .  
while  $T_{i}\leq T$  do  
Initialize MWU for  $T_{i}$  rounds and  $n$  actions.  
Simulate running MWU for  $T_{i - 1}$  rounds, using the average of the first  $T_{i - 1}$  rewards each round.  
For  $T_{i - 1}$  rounds, run MWU with action probabilities rounded to multiples of  $4\gamma = \tilde{O}(T_i^{-1 / 2})$ .  
Let  $T_{i + 1} = 2T_{i}$  and  $i:= i + 1$ .  
end while
Lemma 6. When running RMBD for  $T$  rounds, the following hold at any round  $t \leq T$ :
- RMBD has cumulative regret  $\tilde{O}(n\sqrt{t})$ ;
- If action  $j$  has the highest cumulative reward and  $\sigma_{i,t} \leq \sigma_{j,t} - \tilde{O}(\sqrt{t})$ , then action  $i$  is played with probability  $O$  at round  $t$ .
Proof. Let  $C\sqrt{t}$  bound the regret of MWU over  $t$  rounds (where  $C = O(\sqrt{\log n})$ ), and let  $D = \sqrt{2}C + \tilde{O}(n)$ . We can bound the regret of RMBD over  $T_i$  rounds by  $D\sqrt{T_i}$  via induction (which holds trivially at  $T_1$ ). Suppose it holds for some  $T_i$ . Let  $R(T_i)$  be the true reward obtained by RMBD over  $T_i$  rounds, which is at least  $\sigma_{j^*,T_i} - D\sqrt{T_i}$ , where  $\sigma_{j^*,T_i}$  is the cumulative reward of the best action over  $T_i$  rounds. Consider our simulation of MWU over  $T_i$  rounds using the average reward function. As the reward function is identical each round, and the cumulative reward for each action  $j$  is equivalent under averaging, the measured reward  $\hat{R}(T_i)$  from the simulated run is at most  $\sigma_{j^*,T_i}$  after  $T_i$  rounds. Upon continuing to run this instance of MWU for an additional  $T_i$  rounds, the regret bound ensures that the total measured reward  $\hat{R}(T_{i+1})$  is at least  $\sigma_{j^*,2T_i} - C\sqrt{2T_i}$ . Rounding probabilities contribute at most an additional  $2n\gamma T_i$  to the regret; it suffices to implement rounding by reallocating probability mass from any  $p_{i,t} < 2\gamma$  onto other actions arbitrarily, to avoid renormalization. The total reward of RMBD over  $2T_i = T_{i+1}$  is given by its cumulative reward at  $T_i$ , as well as the additional reward obtained by the MWU instance over the next  $T_i$  rounds, and so we
$$
\begin{array}{l} R \left(T _ {i + 1}\right) = R \left(T _ {i}\right) + \hat {R} \left(T _ {i + 1}\right) - \hat {R} \left(T _ {i}\right) \\ \geq \sigma_ {j ^ {*}, T _ {i + 1}} - D \sqrt {T _ {i}} - C \sqrt {2 T _ {i}} - 2 n \gamma T _ {i} \\ \geq \sigma_ {j ^ {*}, T _ {i + 1}} - D \sqrt {T _ {i + 1}}, \\ \end{array}
$$
which yields the bound for every  $T_{i}$ . We can extend this to any  $t \in [T_i, T_{i+1}]$  with at most a factor 2 increase to cumulative regret.
To bound the selection frequency of actions with suboptimal cumulative reward, we recall the mean-based analysis of MWU given in Theorem D.1 from (5), which shows that the selection frequency  $p_{k,t}$  for action  $k$  at time  $t$  is at most  $\gamma = \frac{2\log(\sqrt{T}\log n)}{\sqrt{T}\log n}$  if  $\sigma_{k,t} \leq \sigma_{j,t} - \gamma T$  for the action  $j$  with highest cumulative reward. As such, any action whose cumulative reward  $\sigma_{k,t} \leq \sigma_{j,t} - \tilde{O}(\sqrt{t})$  will be played with probability 0.
Suppose a learner plays the action with highest cumulative reward at each round for  $t_{\mathrm{burn}} = \hat{\Omega}(M^2)$  rounds, then plays RMBD thereafter for a total of  $T$  rounds. Note that this maintains the both properties of RMBD for all  $t$ . We show that at least  $T = \exp(\Omega(M))$  rounds are required to identify the Stackelberg strategy. The optimizer must check the learner's pure best response to each  $a_j$  for identification with certainty, and it is straightforward to construct a distribution in which any strategy which does not observe  $\mathsf{BR}(a_j)$  for all  $j$  will have linear regret to  $\mathsf{Stack}_A$  in expectation (e.g. where  $\mathcal{B}_U$  contains one action chosen uniformly at random). The difficulty in exploration of the best responses comes from the safety actions, as  $a_j$  must have been played more frequently than any other action in order to not be dominated by some safety action. Let  $\rho_{j,t}$  denote the number of rounds in which the optimizer has played  $a_j$  out of the first  $t$ . Observe that by construction of the game and the properties of RMBD, an ordinary or secondary action  $b_j$  or  $s_j$  in  $\mathsf{BR}(a_j)$  will only be played with positive probability when:
$$
\begin{array}{l} \rho_ {j, t} \geq \frac {2}{3} (\rho_ {j, t} + \rho_ {k, t}) - \tilde {O} (\sqrt {t}) \\ = 2 \rho_ {k, t} - \tilde {O} (\sqrt {t}) \\ \end{array}
$$
for all  $k$ , which necessitates that  $\rho_{j,t} \geq \frac{2t}{M} - \tilde{O}(\sqrt{t})$ . Taking  $t_{\mathrm{burn}}$  sufficiently large, we have that  $\rho_{j,t} \geq \frac{3}{2}\rho_{k,t}$  for any  $t \geq t_{\mathrm{burn}}$  and all  $k$ . For any subsequent observation  $\mathsf{BR}(a_k)$  at  $t'$ , we must have that  $\rho_{k,t'} \geq \frac{3}{2}\rho_{j,t}$ , and so the number of rounds required to play an action before observing its best response grow at a rate of at least  $(3/2)^M$ , which completes the proof.