SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof sketch. We sketch the proof for the deterministic tie-breaking, as the randomized one needs just an application of Azuma-Hoeffding. We first rewrite  $\sum_{t\in [T]}U_P(\mathbf{h}_t,y_t)$  partitioned in the principal's utility for each round-specific forecast that induces action  $i$  as the best-response from the agent, for all actions  $i\in \mathcal{A}_A$ :  $\sum_{t\in [T]}U_P(\mathbf{h}_t,y_t) = \sum_{i\in \mathcal{A}_A}\sum_{t\in [T]}w_i(\mathbf{p}_t)\cdot U_P(\mathbf{h}_t,i)$ . This equivalence holds because each  $\mathbf{p}_t$  maps to a single best response  $i\in \mathcal{A}_A$  (deterministic tie-breaking). Because of the linearity of the principal's reward in the principal's strategy:  $\sum_{i\in \mathcal{A}_A}\sum_{t\in [T]}w_i(\mathbf{p}_t)\cdot U_P(\mathbf{h}_t,i) = \sum_{i\in \mathcal{A}_A}n_T(i)\cdot U_P(\bar{\mathbf{h}}_T,i)$ , where  $n_T(i) = n_{[0,T]}(i)$ . Adding and subtracting  $U_P(\bar{\mathbf{p}}_T,i)$  from the above, we now need to bound the quantity:  $\sum_{i\in \mathcal{A}_A}n_T(i)(U_P(\bar{\mathbf{p}}_T(i),i) + \langle U_P(\cdot ,i),\bar{\mathbf{h}}_T(i) - \bar{\mathbf{p}}_T(i)\rangle)$ . The first term is upper bounded by  $V^{\star}T$ ; note that  $i\in \mathrm{BR}(\bar{\mathbf{p}}_T)$  (and  $V^{\star} = \max_{\mathbf{h}}\max_{y\in \mathrm{BR}(\mathbf{h})}U_P(\mathbf{h},y))$  since  $\mathbf{p}_t(i)\in P_i$  and hence that should also be true for the average of  $\mathbf{p}_t(i)$  over  $t$  rounds. The second summand is bounded by the calibration error of Definition 2.3.
2A similar upper-bound on the principal's utility was proved for no-swap-regret agents [20]. While we prove the theorem directly for calibration, an alternative proof in App. B shows that calibration implies no-swap regret.
Figure 2: Pictorial representation for notation  $B_{2}(S,\varepsilon)$  (left) and  $B_{2}(S, - \varepsilon)$  (right).
On the other hand, it may seem that because the agent's behavior is less specified when she uses calibrated forecasts (as opposed to full knowledge), the principal may only be able to extract much less utility compared to  $V^{\star}$ . Again, we show that this is not the case and that there exist algorithms for the principal such that the sequence of strategies  $\{\mathbf{h}_t\}_{t \in [T]}$  is asymptotically approaching  $V^{\star}$ .
Theorem 3.2. There exists an algorithm for the principal in CSGs that achieves average utility:  $\lim_{T\to \infty}\frac{1}{T}\sum_{t\in [T]}U_P(\mathbf{h}_t,y_t)\geq V^\star$
Algorithm 1 is an explore-then-commit algorithm; it first estimates an appropriate strategy for the principal  $\tilde{\mathbf{h}}$  (EXPLORE), and then repeatedly plays it until the end (COMMIT). In the remainder of the section, we sketch the proof for Theorem 3.2 and point to exact lemma statements in the Appendix. Let  $T_{1}, T_{2}$  denote the set of rounds that belong in the EXPLORE and COMMIT phase respectively.
To elaborate on the objectives of the EXPLORE phase, let us first consider a setting with zero calibration error, where the agent's forecasting algorithm is perfectly and adaptively calibrated, leading to  $y_{t} = \mathrm{BR}(\mathbf{h}_{t})$  at every round. The task for the EXPLORE phase simplifies to identifying a near-optimal strategy  $\tilde{\mathbf{h}}$  through best response oracles that satisfies  $U_{P}(\tilde{\mathbf{h}},\mathrm{BR}(\tilde{\mathbf{h}}))\geq V^{\star} - \varepsilon_{1}$  for a predetermined  $\varepsilon_{1}$ . We formalize this property in (P1). Given that the agent is perfectly calibrated, in the COMMIT phase, the agent always plays  $\tilde{y} = \mathrm{BR}(\tilde{\mathbf{h}})$ , leading to an upper bound of  $\varepsilon_1|T_2|$  on the Stackelberg regret. Hence, Algorithm 1's regret is bounded by  $V^{\star}|T_{1}| + \varepsilon_{1}|T_{2}|$ .
(P1)  $U_P(\tilde{\mathbf{h}}, \tilde{y}) \geq V^\star - \varepsilon_1$  for  $\tilde{y} \in \mathsf{BR}(\tilde{\mathbf{h}})$ , i.e.,  $(\tilde{\mathbf{h}}, \tilde{y})$  is an approximate Stackelberg equilibrium.
Moving away from the idealized setting, we must account for possible discrepancies between  $y_{t}$  and  $\mathrm{BR}(\mathbf{h}_t)$  due to calibration error. This introduces: (i) An increased sample complexity  $|T_1|$  in the EXPLORE phase, given the necessity to learn a near-optimal strategy from noisy responses; (ii) Potential deviations from the action  $\tilde{y} = \mathrm{BR}(\tilde{\mathbf{h}})$  in the COMMIT phase due to miscalibrations in belief. To address the first challenge, we employ Algorithm 2, which constructs an approximate best response oracle by repeatedly interacting with a calibrated agent. For the second challenge, we require our learned policy  $\tilde{\mathbf{h}}$  to be robust against inaccurate forecasts. This is reflected in condition (P2), which necessitates the ball of radius  $\varepsilon_2$  around  $\tilde{\mathbf{h}}$  to be fully contained in the polytope  $P_{\tilde{y}}$ . The critical insight from (P2) is: for any forecast  $\mathbf{p}_t$  that results in a best response  $y_{t} = \mathrm{BR}(\mathbf{p}_{t})\neq \tilde{y}$ , there must be a minimum distance of  $\varepsilon_{2}$  separating  $\mathbf{p}_t$  from  $\tilde{\mathbf{h}}$ . We will now proceed to formalize (P2), but before delving into that, it is important to introduce some additional notations.
Let  $B_{2}(x,\varepsilon)$  denote the ball of radius  $\varepsilon$  around  $x$ , i.e.,  $B_{2}(x,\varepsilon) \triangleq \{x^{\prime} : \| x - x^{\prime}\|_{2} \leq \varepsilon\}$ . For a convex set  $S \in \mathbb{R}^{n}$ , we use  $B_{2}(S,\varepsilon)$  to denote the union of all balls of radius  $\varepsilon$  around the set, i.e.,  $B_{2}(S,\varepsilon) \triangleq \bigcup_{x \in S} B_{2}(x,\varepsilon)$ . For a convex set  $S \in \mathbb{R}^{n}$ , we use  $B_{2}(S, -\varepsilon)$  to denote the set of all points in  $S$  that are "safely" inside  $S$  (i.e., all the points in a ball of radius  $\varepsilon$  around them still belong in  $S$ ):  $B_{2}(S, -\varepsilon) \triangleq \{x \in S : B_{2}(x,\varepsilon) \subseteq S\}$ . We call this last set, the  $\varepsilon$ -conservative of  $S$ . See Figure 2 for a pictorial illustration of the notations.
Algorithm 1: Explore-Then-Commit
Input: Target precision  $\varepsilon$ , time horizon  $T$ .
EXPLORE: Find  $\varepsilon$ -optimal strategy  $\tilde{\mathbf{h}} \in \mathcal{H}_P$  using Algorithm 2.
COMMIT: Repeatedly play  $\tilde{\mathbf{h}}$  for the rest of the rounds.
Given the above notations and for tuned  $\varepsilon_{2}$ , the pair  $(\tilde{\mathbf{h}},\tilde{y})$  returned by Algorithm  $2^{3}$  satisfies property (P1) and the additional property (P2):
(P2)  $\tilde{\mathbf{h}}\in B_2(P_{\tilde{y}}, - \varepsilon_2)$ , i.e.,  $\tilde{\mathbf{h}}$  lies robustly within the best-response polytope for  $\tilde{y}$ .
Given (P2), the regret of the COMMIT phase can be decomposed to when  $y_{t} = \tilde{y}$ , and when  $y_{t} \neq \tilde{y}$ :
$$
\sum_ {t \in T _ {2}} \left(V ^ {\star} - U _ {P} (\tilde {\mathbf {h}}, y _ {t})\right) = \sum_ {t \in T _ {2}: y _ {t} = \tilde {y}} \left(V ^ {\star} - U _ {P} (\tilde {\mathbf {h}}, y _ {t})\right) + \sum_ {t \in T _ {2}: y _ {t} \neq \tilde {y}} \left(V ^ {\star} - U _ {P} (\tilde {\mathbf {h}}, y _ {t})\right) \tag {3}
$$
When  $y_{t} = \tilde{y}$ , (P1) guarantees that  $V^{\star} - U_{P}(\tilde{\mathbf{h}},\tilde{y})\leq \varepsilon_{1}$ , so the first term is at most  $\varepsilon_1\cdot |T_2|$ .
When  $y_{t} \neq \tilde{y}$ , let  $A = \mathcal{A}_A \setminus \{\tilde{y}\}$ . For  $i \in A$ , the definition of binning function  $w_{i}$  guarantees that the probability of playing action  $i$  on forecast  $\mathbf{p}_t$  is exactly  $w_{i}(\mathbf{p}_{t})$ . Based on this observation, the second term of Equation (3) can be further bounded as
$$
\sum_ {t \in T _ {2}} \sum_ {i \in A} w _ {i} (\mathbf {p} _ {t}) (V ^ {\star} - U _ {P} (\tilde {\mathbf {h}}, i)) \leq \sum_ {i \in A} \sum_ {t \in T _ {2}} w _ {i} (\mathbf {p} _ {t}) V ^ {\star} = \sum_ {i \in A} n _ {T _ {2}} (i) V ^ {\star}. \tag {4}
$$
Using Definition 2.2 of the calibration error and properties of the  $\ell_2$  and  $\ell_{\infty}$  norms, we can further express  $n_{T_2}(i)$  as follows
$$
n _ {T _ {2}} (i) = \frac {\operatorname {C a l E r r} _ {i} \left(\mathbf {h} _ {T _ {2}} , \mathbf {p} _ {T _ {2}}\right) \cdot | T _ {2} |}{\| \bar {\mathbf {p}} _ {T _ {2}} (i) - \tilde {\mathbf {h}} \| _ {\infty}} \leq \frac {\sqrt {m} \cdot \operatorname {C a l E r r} _ {i} \left(\mathbf {h} _ {T _ {2}} , \mathbf {p} _ {T _ {2}}\right) \cdot | T _ {2} |}{\| \bar {\mathbf {p}} _ {T _ {2}} (i) - \tilde {\mathbf {h}} \| _ {2}} \overset {(\mathbf {P 2})} {\leq} \frac {\sqrt {m} \cdot r _ {\delta} (| T _ {2} |) \cdot | T _ {2} |}{\varepsilon_ {2}},
$$
where for the second inequality is: since  $\tilde{\mathbf{h}}$  lies in the  $\varepsilon_{2}$ -conservative of  $P_{\tilde{y}}$ , and  $\bar{\mathbf{p}}_{T_2}(i)$  belongs to a different and non-intersecting polytope  $P_{i}$ , we know that  $\| \bar{\mathbf{p}}_{T_2}(i) - \tilde{\mathbf{h}} \|_2 \geq \varepsilon_2$ . See Figure 3 for a geometric interpretation. Finally, bounding the sample complexity for the EXPLOR phase (Lemma C.4) and taking asymptotics gives the result.
The proof sketch above hinges on being able to identify a strategy for the principal  $\tilde{\mathbf{h}}$  with properties (P1), (P2). We outline below how this is done through Algorithm 2.
At a high level, Algorithm 2 proceeds as follows. It builds an initialization set  $\mathcal{I}$  which consists of pairs  $(\mathbf{h},y)$  where  $\mathbf{h}$  approximately belongs to the  $\varepsilon_{2}$ -conservative of  $y$ 's best response polytope:  $B_{2}(P_{y}, - \varepsilon_{2})$ . Using these points in the initialization set  $\mathcal{I}$ , the algorithm then solves the convex optimization problem of  $\max_{\mathbf{h}}U_P(\mathbf{h},y_i)$  using membership queries to  $B_{2}(P_{y}, - \varepsilon_{2})$  and the points of  $\mathcal{I}$  as initial points. The final solution is obtained as the maximum of all these convex programs.
There are 2 things left to be specified for Algorithm 2; how to build the membership oracle to  $B_{2}(P_{y}, - \varepsilon_{2})$  given that  $U_{A}(\cdot)$  is unknown to the principal, and how to build the initialization set  $\mathcal{I}$  to contain at least one point that is well
Figure 3: Relationship between the robust policy  $\tilde{\mathbf{h}}$  and the average predictions  $\bar{\mathbf{p}}_{T_2}(i)$ : Given that  $\tilde{\mathbf{h}}$  is in the conservative region  $B_2(P_{\tilde{y}}, -\varepsilon_2)$ , any average prediction  $\bar{\mathbf{p}}_{T_2}(i)$  that triggers action  $i \neq \tilde{y}$  during the COMMIT phase must fall outside of  $P_{\tilde{y}}$  and thus have a distance of at least  $\varepsilon_2$  from  $\tilde{\mathbf{h}}$ .
centered in  $P_{y^{\star}}$ . To build set  $\mathcal{I}$ , Algorithm 2 samples strategies  $\mathbf{h}_i$  uniformly at random for enough
<sup>3</sup>With post-processing if necessary, see Appendix C.5 for more details.
Algorithm 2: Principal's Learning Algorithm for the Optimal Commitment
Originally, initialization set  $\mathcal{I}\gets \emptyset$ $/*$  build initialization set with pairs  $(\mathbf{h},y)$  , where  $\mathbf{h}\in B_2(P_y, - \varepsilon_2)$ $^ { / } \ast$    
for  $i\in [O(\log (T) / \mathrm{Volume}(\eta)]$  do Sample a strategy  $\mathbf{h}_t\in \mathcal{H}_P$  uniformly at random. Query APPROXMEM (Algorithm 4) for action  $y_{i}\in \mathcal{A}_{A}$ $\mathcal{I}\leftarrow \mathcal{I}\cup (y_i,\mathbf{h}_i)$ $/*$  optimize  $U_{P}(\mathbf{h},y)$  using membership queries to  $B_{2}(P_{y}, - \varepsilon_{2})$  and set I  $^ { / } \ast$    
for  $(y_{i},\mathbf{h}_{i})\in \mathcal{I}$  do Solve Equation (5) using LSV [42] (initial point:  $\mathbf{h}_i$  , oracle:  $\varepsilon_{1}$  -approximate membership oracle to  $B_{2}(P_{y_{i}}, - \varepsilon_{2})$  simulated by APPROXMEM (Algorithm 4) max  $U_{P}(\mathbf{h},y_{i})$  subject to  $\mathbf{h}\in B_2(P_{y_i}, - \varepsilon_2)$  (5) h  $\tilde{\mathbf{h}}_{y_i}\gets \varepsilon '$  -optimal solution of Equation (5)   
 $\tilde{y}\gets \arg \max_{y_i\in \mathcal{I}}U_P(\tilde{\mathbf{h}}_{y_i},y_i)$    
RETURN  $\tilde{\mathbf{h}}_{\tilde{y}}$
number of rounds to ensure at least one sample in the  $\eta /2$ -ball around the center of  $P_{y^{\star}}$  with high probability. To build the approximate membership oracle for the  $\varepsilon_2$ -conservative best-response polytope for an action  $y$ , we use APPROXMEM (Algorithm 4 in Appendix C.2). Specifically, on input  $\mathbf{h} \in \mathcal{H}_P$ , APPROXMEM either asserts  $\mathbf{h} \in B_2(P_y, -\varepsilon_2 + \varepsilon_1)$  or  $\mathbf{h} \notin B_2(P_y, -\varepsilon_2 - \varepsilon_1)$  with probability at least  $1 - \varepsilon_3$ . To do this, it samples  $\Phi$  points in proximity to  $\mathbf{h}$  and plays each one repeatedly for  $l$  rounds, while registering the best-response action observed for each one of these. If the most frequent best-response for all  $\mathbf{h}_{\phi}$  is  $y$ , then we can conclude with good probability that  $\mathbf{h}$  was inside  $B_2(P_y, -\varepsilon_2)$ . See Lemma C.2 for more details.
4 Forecasting Algorithm for Adaptive Calibration
In this section, we examine whether there exist natural forecasting procedures that satisfy our Definition 2.3 about adaptively calibrated forecasts. We answer this question positively.
Theorem 4.1. For all  $\varepsilon >0$  and all binnings  $\Pi = \{w_{i}:\mathbb{R}^{m}\to [0,1],i\in [k]\}$ , there exists a parameter-free forecasting procedure that is  $(\varepsilon ,\Pi)$ -adaptively calibrated with rate  $r_{\delta}(t) = O(\sqrt{\log(kmt)} /t)$ . Moreover, when  $\Pi$  is a continuous binning (i.e., each  $w_{i}$  is continuous), there exists a forecasting procedure that is  $(0,\Pi)$ -adaptively calibrated with the same rate.
To prove the theorem, we use two main tools; the first one is a well-known algorithm of Luo and Schapire [44] (ADANORMALHEDGE) applied for online learning in the sleeping experts problem (see Appendix D for details). Roughly speaking, the sleeping experts is a standard online learning problem with  $T$  rounds and  $N$  experts, where at each round  $t$  there is only a subset of the experts being "awake" to be considered by the learner and report their predictions. Let  $I_{t,i}$  be the binary variable indicating whether expert  $i$  was awake at round  $t$  ( $I_{t,i} = 1$ ) or asleep ( $I_{t,i} = 0$ ). The interaction protocol between the learner and the adversary at each round  $t$  is: (i) The learner observes which experts are awake, i.e.,  $\{I_{t,i}\}_{i \in [N]}$ . (ii) The learner selects a probability distribution  $\pi_t \in \Delta([N])$  supported on the set of active experts  $A_t \triangleq \{i : I_{t,i} = 1\}$ . (iii) The adversary selects a loss vector  $\{\ell_{t,i}\}_{i \in [N]}$ . (iv) The learner incurs expected loss  $\hat{\ell}_t = \mathbb{E}_{i \sim \pi_t}[\ell_{t,i}]$ . ADANORMALHEDGE is a parameter-free online learning algorithm that when applied on the sleeping experts problem (and with appropriate initialization) obtains regret  $\mathrm{Reg}_t(i) = O(\sqrt{T_i \log(NT_i)})$ , where  $T_i = \sum_{\tau \in [t]} I_{\tau,i}$ .
The second tool that we use is No-Regret vs. Best-Response dynamics (NRBR) [34]. NRBR are a form of no-regret dynamics between two players, where one of the players must also best-respond on average. Essentially, at each round  $t \in [T]$ , the forecasting algorithm with the calibration rate of Theorem 4.1 outputs a randomized forecast  $\mathbf{p}_t \in \mathcal{F}_P$ , by simulating an interaction between two players described below. For the first player, we construct a sleeping experts problem instance, where the set of experts is  $\mathcal{G} = \{g_{(s,i,j,\sigma)} : s \in [T], i \in \mathcal{A}_A, j \in \mathcal{A}_P, \sigma \in \{\pm 1\}\}$ . For each  $g_{(s,i,j,(\sigma))} \in \mathcal{G}$
and  $t \in [T]$ , we define the loss, sleeping/awake indicator, and instantaneous regret respectively as:
$$
\ell_ {t, g _ {(s, i, j, \sigma)}} \triangleq L _ {g _ {(s, i, j, \sigma)}} \left(\mathbf {h} _ {t}, \mathbf {p} _ {t}\right) = w _ {i} \left(\mathbf {p} _ {t}\right) \cdot \boldsymbol {\sigma} \cdot \left(h _ {t, j} - p _ {t, j}\right); \tag {6}
$$
$$
I _ {t, g _ {(s, i, j, \sigma)}} \triangleq \mathbf {1} \{t \geq s \}; \tag {7}
$$
$$
r _ {t, g} \triangleq I _ {t, g} \cdot \left(\ell_ {t, g} - \hat {\ell} _ {t}\right).
$$
where by  $h_{t,j}, p_{t,j}$  we denote the  $j$ -th coordinate of  $\mathbf{h}_t$  and  $\mathbf{p}_t$  respectively. We defined the losses for our newly constructed sleeping experts' instance as above to make sure that there is a direct correspondence with the calibration error. Similar ideas for calibration (albeit not for the notion of adaptivity we consider) have been used in [41, 34]. We describe next the player interaction in NRBR.
Player 1. Runs ADANORMALHEDGE on expert set  $\mathcal{G}$  with a pre-specified prior  $\pi_0$  over  $\mathcal{G}$  and feedback specified in Equations (6), (7). At each round  $t$ , Player 1 computes distribution  $\pi_t \in \Delta(A_t(\mathcal{G}))$ , where  $A_t(\mathcal{G})$  denotes the set of active experts  $g_{(s,i,j,\sigma)} \in \mathcal{G}$  with  $I_{t,g_{(s,i,j,\sigma)}} = 1$ .
Player 2. Best responds to  $\pi_t$  by selecting  $Q_t \in \Delta(\mathcal{F}_P)$  that satisfies:
$$
\max  _ {\mathbf {h} _ {t} \in \mathcal {H} _ {P}} \underset {\mathbf {p} _ {t} \sim Q _ {t}} {\mathbb {E}} [ \ell_ {t, g} ] = \max  _ {\mathbf {h} _ {t} \in \mathcal {H} _ {P}} \underset {\mathbf {p} _ {t} \sim Q _ {t}} {\mathbb {E}} [ L _ {g} (\mathbf {h} _ {t}, \mathbf {p} _ {t}) ] \leq \varepsilon . \tag {8}
$$
After simulating the game above, the algorithm outputs forecast  $\mathbf{p}_t\sim Q_t$ . The existence of such a distribution  $Q_{t}$  is justified by the min-max theorem ([34, Fact 4.1] or [28, Theorem 5]). In the Appendix, we also give an explicit formula for  $Q_{t}$  in the special case of  $m = 2$ . When  $\Pi_0$  is continuous, player 2 can select a deterministic  $\mathbf{p}_t$  that achieves Equation (8) with  $\varepsilon = 0$ . This stronger property is justified by the outgoing fixed-point theorem [28, Theorem 4]. Note that this algorithm inherits its parameter-free property directly from ADANORMALHEDGE. We are now ready to provide a proof sketch for Theorem 4.1.
Proof sketch of Theorem 4.1. Fix an instance of the NRBR game outlined above. We begin by Definition 2.2 of calibration error translated in the sleeping experts instance that we defined above:
$$
\mathrm {C a l E r r} _ {i} (\mathbf {h} _ {s: t}, \mathbf {p} _ {s: t}) \cdot (t - s) = \max _ {j \in [ m ]} \max _ {\sigma \in \{\pm 1 \}} \sum_ {\tau \in [ s, t ]} I _ {\tau , g _ {(s, i, j, \sigma)}} \ell_ {\tau , g _ {(s, i, j, \sigma)}}
$$
We add and subtract in the above  $\sum_{\tau} \hat{\ell}_{\tau}$  to make the regret of the ADANORMALHEDGE on the sleeping experts instance appear and so the aforementioned becomes:
$$
\mathrm {C a l E r r} _ {i} (\mathbf {h} _ {s: t}, \mathbf {p} _ {s: t}) \cdot (t - s) = \underbrace {\operatorname {R e g} _ {t} (g _ {(s , i , j , \sigma)})} _ {O (\sqrt {(t - s) \cdot \log (k m t)})} + \max _ {j \in \mathcal {A} _ {P}} \max _ {\sigma \in \{\pm 1 \}} \underbrace {\sum_ {\tau \in [ s , t ]} I _ {\tau , g _ {(s , i , j , \sigma)} \hat {\ell} _ {\tau , g _ {(s , i , j , \sigma)}}}} _ {\mathcal {T}}
$$
where for the regret, we have substituted the regret obtained by ADANORMALHEDGE. Note that  $\hat{\ell}_{\tau} = \mathbb{E}_{g\sim \pi_{\tau}}[\ell_{\tau ,g}]$ . This, together with Equation (6) helps us translate the sleeping experts' loss to a loss that depends on  $\mathbf{h}_{\tau},\mathbf{p}_{\tau}$ . Adding and subtracting the term  $\sum_{\tau \in [s,t]}\mathbb{E}_{\substack{\mathbf{p}\sim Q_{\tau}}}^{g\sim \pi_{\tau}}[L_g(\mathbf{h}_{\tau},\mathbf{p})]$ :
$$
\mathcal{T}\leq \sum_{\tau \in [s,t]}\max_{\mathbf{h}_{\tau}\in \Delta (\mathcal{A}_{P})}\underset {g\sim \pi_{\tau}}{\mathbb{E}}_{\substack{\mathbf{p}\sim Q_{\tau}}}[L_{g}(\mathbf{h}_{\tau},\mathbf{p})] + \sum_{\tau \in [s,t]}\underset {g\sim P_{\tau}}{\mathbb{E}}\left[L_{g}(\mathbf{h}_{\tau},\mathbf{p}_{\tau}) - \underset {\mathbf{p}\in Q_{\tau}}{\mathbb{E}}[L_{g}(\mathbf{h}_{\tau},\mathbf{p})]\right]
$$
The first term above is upper bounded by  $\varepsilon \cdot (t - s)$  (Equation (8)) (or 0 in the continuous setting). The second can be bound with martingale concentration inequalities.
5 Continuous Games
In this section, we generalize our results for the case of continuous Stackelberg games. The supplementary material can be found in Appendix E.
Continuous Stackelberg Games. We use again  $\mathcal{A}_P$  and  $\mathcal{A}_A$  to denote the principal and the agent action spaces, respectively. Both  $\mathcal{A}_A$ ,  $\mathcal{A}_P$  are convex, compact sets where  $\mathcal{A}_P \subset \mathbb{R}^m$  and  $\mathcal{A}_A \subset \mathbb{R}^k$ . The utilities of the principal and the agent are given by continuous functions  $U_P: \mathcal{A}_P \times \mathcal{A}_A \to \mathbb{R}_+$  and  $U_A: \mathcal{A}_P \times \mathcal{A}_A \to \mathbb{R}_+$ . In this setting, we assume that both the principal and the agent can only play deterministic strategies, i.e.,  $\mathcal{H}_P = \mathcal{A}_P$ . For  $x \in \mathcal{A}_P$ , let  $\mathrm{BR}(x)$  be the best-response function that is implicitly defined as  $\nabla_2 U_A(x, \mathrm{BR}(x)) = 0$ . Our continuous games satisfy Assumption 5.1: (i)-(iii) are standard assumptions used in previous works (e.g., [24]), but (iv) cannot be derived from (i) and (ii) without further assumptions on the correlation between  $x, y$ . Nevertheless, (iv) (and the conditions under which it holds) has been justified in settings such as strategic classification [21, 51].
Algorithm 3: Lazy Gradient Descent without a Gradient (LAZYGDwOG)
Initialize  $\mathbf{h}_0 = 0$    
for epoch  $\phi \geq 0$  do Sample  $S_{t}$  uniformly at random from the unit sphere  $\mathbb{S}^{m - 1}$  Play  $\mathbf{h}_{\phi} = \mathbf{x}_{\phi} + \delta_{\phi}S_{\phi}$  for  $M$  rounds. /\* avg feedback gets close to BR(ht)\*/ Observe agent's responses  $y_{\phi ,1},\dots ,y_{\phi ,M}$  Update action  $\mathbf{x}_{\phi +1}\gets \mathrm{Proj}_{B_2(A_P, - \delta_\phi)}(\mathbf{x}_\phi +\gamma_\phi \frac{m}{\delta_\phi} S_tU_P(\mathbf{h}_\phi ,\frac{1}{M}\sum_{i\in [M]}y_{\phi ,i}))$
Assumption 5.1. Utility functions  $U_{P}, U_{A}$ , and the domain  $\mathcal{A}_P$  satisfy the following:
(i) For all  $x \in \mathcal{A}_P, y \in \mathcal{A}_A$ ,  $U_P(x,y)$  is  $L_1$ -Lipschitz and concave in  $x$ ,  $L_2$ -Lipschitz in  $y$ , and bounded by  $W_P$  in  $\ell_2$  norm.
(ii) The best-response function  $\mathsf{BR}:\mathcal{A}_P\to \mathcal{A}_A$  is  $L_{\mathrm{BR}}$  -Lipschitz.
(iii) Regularity of the feasible set  $\mathcal{A}_P = \mathcal{H}_P = \mathcal{F}_P$ :
- The diameter is bounded:  $\mathrm{diam}(\mathcal{F}_P) = \sup_{\mathbf{h},\mathbf{h}'\in \mathcal{F}_P}\| \mathbf{h} - \mathbf{h}'\| _2\leq D_P$
$B(0,r)\subseteq \mathcal{A}_P\subseteq B(0,R).$
(iv) The function  $U_P(\mathbf{h}, \mathsf{BR}(\mathbf{h}))$  is concave with respect to  $\mathbf{h}$ , and has Lipschitz constant  $L_U$ .
The main result of this section is to show that even in continuous CSGs, we can approximate asymptotically  $V^{\star}$  for the principal's utility, and that no better utility is actually achieved.
Theorem 5.2. For continuous CSGs satisfying Assumption 5.1, for all  $\varepsilon_0 > 0$ , there exists a finite binning  $\Pi_0$  such that if the agent is  $(0,\Pi_0)^4$  - adaptively calibrated and the principal runs an appropriately parametrized instance of LAZYGDwOG (Algorithm 3) then:  $\lim_{\Phi \to \infty}\frac{1}{\Phi M}\sum_{\phi \in [\Phi ]}\sum_{i\in [M]}U_P(\mathbf{h}_\phi ,y_\phi ,i)\geq V^\star -\varepsilon_0$ . Moreover, for any sequence of the principal's actions  $\mathbf{h}_{[1:T]}$ , it holds that:  $\lim_{T\to \infty}\frac{1}{T}\sum_{t\in [T]}U_P(\mathbf{h}_t,y_t)\leq V^\star +\varepsilon_0$ .