SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since  $\tilde{\mathbf{h}}$  is an  $\varepsilon$ -optimal strategy, by Assumption C.11, we know that the true optimal  $\mathbf{h}_{\tilde{y}}^{\star}$  satisfies  $\| \mathbf{h}_{\tilde{y}}^{\star} - \tilde{\mathbf{h}} \|_{\infty} \leq c2^{L}\varepsilon$ . According to Lemma C.10, every coordinate of  $\mathbf{h}_{\tilde{y}}^{\star}$  is a rational number with denominator at most  $2^{2mL}$ . Combining the two guarantees, we know that  $S$  must contain the true optimal  $\mathbf{h}_{\tilde{y}}^{\star}$ , which satisfies  $\mathbf{h}_{\tilde{y}}^{\star} \in P_{\tilde{y}}$ . Suppose this point is  $\mathbf{h}^j$ . According to the above claim, we have  $\mathbf{h}_{\mathrm{query}}^j \in B_2(P_{\tilde{y}}, -\lambda)$ . Since we simulated the APPROXMEM oracle with  $\varepsilon_1 = \varepsilon_2 = \frac{\lambda}{2}$ , it must return TRUE on strategies that belong to  $B_2(P_{\tilde{y}}^{-\varepsilon_2}, -\varepsilon_1) = B_2(P_y, -\lambda)$ , therefore, it must return TRUE on  $\mathbf{h}_{\mathrm{query}}^j$ . This argument shows that POSTPROCESS always return valid strategies.
However, it could be the case that APPROXMEM returns TRUE on strategies  $h^j$  that are not  $\mathbf{h}_{\tilde{y}}^{\star}$ . We argue that the guarantee of POSTPROCESS is unaffected by showing that if APPROXMEM returns TRUE on any  $\mathbf{h}_{\mathrm{query}}^j$ , then we must have  $\mathbf{h}_{\mathrm{query}}^j \in B_2(P_{\tilde{y}}^{-\varepsilon_2}, + \varepsilon_1) \subseteq P_{\tilde{y}}$  from the guarantee of APPROXMEM (see Lemma C.2). Applying the above claim again, we have that  $\mathbf{h} \in B_2(P_{\tilde{y}}, - \lambda)$  for the returned strategy  $\mathbf{h}$ , which proves (i).
As for (ii), note that
$$
\begin{array}{l} \| \mathbf {h} - \tilde {\mathbf {h}} \| _ {2} \leq \| \tilde {\mathbf {h}} - \mathbf {h} ^ {j} \| _ {2} + \| \mathbf {h} ^ {j} - \mathbf {h} _ {\text {q u e r y}} ^ {j} \| _ {2} + \| \mathbf {h} _ {\text {q u e r y}} ^ {j} - \mathbf {h} \| _ {2} \\ \leq \sqrt {m} \| \tilde {\mathbf {h}} - \mathbf {h} ^ {j} \| _ {\infty} + \frac {2 \lambda}{\eta} \left(\| \mathbf {h} ^ {j} - \mathbf {h} _ {0} \| _ {2} + \| \mathbf {h} _ {\text {q u e r y}} ^ {j} - \mathbf {h} _ {0} \| _ {2}\right) \\ \leq c \sqrt {m} 2 ^ {L} \varepsilon + \frac {4 \lambda \sqrt {m}}{\eta}. \\ \end{array}
$$
Finally, we analyze the failure probability and sample complexity of POSTPROCESS. Since the agent's adaptive calibration error is uniformly bounded by  $r_{\delta}(\cdot)$  with probability  $1 - \delta$ , and the
probability that there exists an incorrect answer from APPROXMEM is bounded by  $T \cdot T^{-2} = T^{-1}$  conditioned on agent's calibration error being bounded, the failure probability of POSTPROCESS is at most  $\delta + T^{-1}$ .
For the sample complexity, since  $|S| \leq 2^m$  and each  $\mathbf{h}^j \in S$  requires simulating APPROXMEM $_{\tilde{y}}$  with  $O\left(1.25^m r_\delta^{-1}\left(\frac{\lambda}{k\sqrt{m}}\right)\log T\right)$  samples (see Lemma C.2 and proposition C.3), the total number of samples required is  $O\left(2^{2m}r_{\delta}^{-1}\left(\frac{\lambda}{k\sqrt{m}}\right)\log T\right)$ .
Theorem C.13. Suppose the utility functions satisfy Assumptions C.9 and C.11. If the principal uses Algorithm 2 with parameters  $\varepsilon_1 = \sqrt{m}\varepsilon_2 = \left(\frac{\eta}{m2^{4mL}}\right)^{\gamma_0}, \varepsilon_3 = \delta + T^{-2}$  for the APPROXMEM oracle and  $\varepsilon' = 2^{-4mL - 1}$  for LSV, then feed the output of LSV into POSTPROCESS( $\lambda$ ), then with probability at least  $1 - \delta - T^{-1} - \varepsilon'$  the final solution satisfies the following two properties:
(P1)  $U_P(\tilde{\mathbf{h}}, y^\star) \geq V^\star - \frac{5m\lambda}{\eta}$ , i.e.,  $\tilde{\mathbf{h}}$  is an approximate Stackelberg equilibrium.
(P2)  $\tilde{\mathbf{h}}\in B_2(P_{y^\star}, - \lambda)$ , i.e.,  $\tilde{\mathbf{h}}$  lies robustly within the best-response polytope for  $y^{\star}$ .
The total number of samples needed is expressed in Equation (14).
Proof. The analysis for the failure probability is the same with Theorem C.8.
Now we consider the optimization problem for the optimal polytope  $P_{y^{\star}}$ , with initial point  $\mathbf{h}_0$  that is  $\frac{\eta}{2}$ -centered in  $P_{y^{\star}}$ . Let  $\hat{\mathbf{h}}$  be the solution output by LSV, and let  $\tilde{\mathbf{h}}$  be the solution output by running POSTPROCESS on  $\hat{\mathbf{h}}$ . According to Lemma C.7,  $\hat{\mathbf{h}}$  satisfies  $\hat{\mathbf{h}} \in B_2(P_{y^{\star}}^{-\varepsilon_2}, + \varepsilon') \subseteq B_2(P_{y^{\star}},\varepsilon')$  and the suboptimality of  $\hat{\mathbf{h}}$  is at most  $O(\varepsilon' + \varepsilon_2) \leq 2^{-4mL}$ . Since  $\varepsilon' \leq 2^{-4mL}$ , by Lemma C.12,  $\tilde{\mathbf{h}} \in B_2(P_{y^{\star}}, - \lambda)$ , which proves (P2).
To prove (P1), note that since  $\| \hat{\mathbf{h}} -\tilde{\mathbf{h}}\| _2\leq \sqrt{m}\left(c2^L\varepsilon ' + \frac{4\lambda}{\eta}\right)$  by Lemma C.12, we have
$$
\begin{array}{l} V ^ {\star} - U _ {P} (\tilde {\mathbf {h}}, y ^ {\star}) \leq \left(V ^ {\star} - U _ {P} (\hat {\mathbf {h}}, y ^ {\star})\right) + \left(U _ {P} (\hat {\mathbf {h}}, y ^ {\star}) - U _ {P} (\tilde {\mathbf {h}}, y ^ {\star})\right) \\ \leq 2 ^ {- 4 m L} + m \left(c 2 ^ {L} \varepsilon^ {\prime} + \frac {4 \lambda}{\eta}\right) \leq 2 ^ {- 3 m L} + \frac {4 m \lambda}{\eta} \leq \frac {5 m \lambda}{\eta}. \\ \end{array}
$$
Finally, we compute the total number of samples. According to Lemma C.2 and proposition C.3, each call to the APPROXMEM oracle requires  $O\left(\sqrt{mr}_{\delta}^{-1}\left(\frac{\varepsilon_2}{k\sqrt{m}}\right)\log T\right) = O\left(\sqrt{mr}_{\delta}^{-1}\left(\frac{1}{km}\left(\frac{\eta}{m2^{4mL}}\right)^{\gamma_0}\right)\log T\right)$  samples, and Lemma C.7 suggests that LSV makes  $O\left(m^2\log^{O(1)}\left(\frac{m}{\varepsilon'\eta}\right)\right) = O\left(m^3L\log^{O(1)}\left(\frac{m}{\eta}\right)\right)$  oracle calls, where we run  $O(V^{-1}\log T)$  LSV instances for each initial point in the initialization set. In addition, POSTPROCESS takes  $O\left(2^{2m}r_{\delta}^{-1}\left(\frac{\lambda}{k\sqrt{m}}\right)\log T\right)$  samples (Lemma C.12), and initialization takes  $O\left(V^{-1}1.25^m r_{\delta}^{-1}\left(\frac{\eta}{k\sqrt{m}}\right)\log^2 T\right)$  samples (Lemma C.6). Adding them all up, the total number of samples is
$$
\tilde {O} \left(V ^ {- 1} 1. 2 5 ^ {m} r _ {\delta} ^ {- 1} \left(\frac {\eta}{k \sqrt {m}}\right) + V ^ {- 1} m ^ {3. 5} L r _ {\delta} ^ {- 1} \left(\frac {1}{k m} \left(\frac {\eta}{m 2 ^ {4 m L}}\right) ^ {\gamma_ {0}}\right) + 2 ^ {2 m} r _ {\delta} ^ {- 1} \left(\frac {\lambda}{k \sqrt {m}}\right)\right). \tag {14}
$$
where  $\tilde{O}$  hildes logarithm factors in  $T,m,\eta^{-1}$
C.6 Proof of Theorem 3.2
In this section, we present the main theorem (Theorem 3.2) in Section 3. We first formally restate the theorem with exact convergence rate.
Theorem C.14 (Formal version of Theorem 3.2). There exists an algorithm for the principal in CSGs that achieves average utility with probability  $1 - o(1)$ :
$$
\lim  _ {T \rightarrow \infty} \frac {1}{T} \sum_ {t \in [ T ]} U _ {P} \left(\mathbf {h} _ {t}, y _ {t}\right) \geq V ^ {\star}. \tag {15}
$$
Specifically, for calibrated agents with calibration error rate  $r_{\delta}(t) = O(t^{-1 / \beta})$
(I) If the principal runs the EXPLORE-THEN-COMMIT algorithm (Algorithm 1) where the explore phase follows the parameter settings in Theorem C.8 with  $\varepsilon' = O\left(T^{-\frac{1}{\gamma_1}}C_1^{\frac{1}{\gamma_1}}\eta^{-\frac{\gamma_1 - 1}{\gamma_1}}\right)$ , then the limit in (15) approaches  $V^{\star}$  with rate
$$
\tilde {O} \left(T ^ {- \frac {1}{\gamma_ {1}}} C _ {1} ^ {\frac {1}{\gamma_ {1}}} \eta^ {- \frac {\gamma_ {1} - 1}{\gamma_ {1}}}\right),
$$
$$
C _ {1} \triangleq V ^ {- 1} m ^ {\frac {5 + m \gamma_ {0} + \beta (2 \gamma_ {0} + 1)}{2}} k ^ {\beta}, \quad \gamma_ {1} \triangleq \gamma_ {0} \left(\frac {m}{2} + \beta\right) + 1.
$$
(II) If the principal runs the EXPLORE-THEN-COMMIT algorithm (Algorithm 1) where the explore phase follows the parameter settings in Theorem C.13 with  $\lambda = O\left(\left(\frac{\eta C_2}{m}\right)^{\frac{1}{\gamma_2}}T^{-\frac{1}{\gamma_2}}\right)$ , then the limit in (15) approaches  $V^{\star}$  with rate
$$
\tilde {O} \left(T ^ {- \frac {1}{\gamma_ {2}}} \left(\frac {m}{\eta}\right) ^ {1 - \frac {1}{\gamma_ {2}}} C _ {2} ^ {\frac {1}{\gamma_ {2}}}\right),
$$
$$
C _ {2} \triangleq V ^ {- 1} 2 ^ {5 m L \beta \gamma_ {0}} k ^ {\beta}, \quad \gamma_ {2} \triangleq \beta \gamma_ {0} + 1.
$$
Proof. According to Lemmas C.4 and C.5, we have that with probability  $1 - o(1)$ , Algorithm 2 (with potential post-processing) returns a policy  $\tilde{\mathbf{h}}$  within  $N$  samples, where  $\tilde{\mathbf{h}}$  satisfies
(P1)  $U_{P}(\tilde{\mathbf{h}},y^{\star})\geq V^{\star} - \varepsilon_{\mathrm{opt}}$ , i.e.,  $\tilde{\mathbf{h}}$  is an approximate Stackelberg equilibrium.
(P2)  $\tilde{\mathbf{h}}\in B_2(P_{y^{\star}}, - \varepsilon_{\mathrm{robust}})$ , i.e.,  $\tilde{\mathbf{h}}$  lies robustly within the best-response polytope for  $y^\star$ .
With properties (P1) and (P2), the arguments in Section 3 give us
$$
V ^ {\star} T - \sum_ {t \in [ T ]} U _ {P} \left(\mathbf {h} _ {t}, y _ {t}\right) \lesssim N + \varepsilon_ {\text {o p t}} T + \frac {k \sqrt {m} T r _ {\delta} (T)}{\varepsilon_ {\text {r o b u s t}}}.
$$
Now we plug in specific instantiations of Algorithm 1.
(I) Suppose the principal follows the parameter settings in Theorem C.8. In this case, we have  $\varepsilon_{\mathrm{opt}} = 3\varepsilon^{\prime}$ ,  $\varepsilon_{\mathrm{robust}} = \varepsilon^{\prime}$ , and
$$
\begin{array}{l} N = \tilde {O} \left(V ^ {- 1} m ^ {\frac {5 + m \gamma_ {0}}{2}} \left(\eta \varepsilon^ {\prime}\right) ^ {- \frac {m \gamma_ {0}}{2}} r _ {\delta} ^ {- 1} \left(\frac {\left(\varepsilon^ {\prime} \eta\right) ^ {\gamma_ {0}}}{k m ^ {\gamma_ {0}} \sqrt {m}}\right)\right) \\ = \tilde {O} \left(V ^ {- 1} m ^ {\frac {5 + m \gamma_ {0} + \beta (2 \gamma_ {0} + 1)}{2}} k ^ {\beta} \left(\eta \varepsilon^ {\prime}\right) ^ {- \gamma_ {0} \left(\frac {m}{2} + \beta\right)}\right). \\ \end{array}
$$
Finally, optimizing over  $\varepsilon^{\prime}$  gives the claimed bound.
(II) Suppose the principal follows the parameter settings in Theorem C.13. In this case, we have  $\varepsilon_{\mathrm{opt}} = \frac{5m\lambda}{\eta}, \varepsilon_{\mathrm{robust}} = \lambda,$  and
$$
\begin{array}{l} N = \tilde {O} \left(V ^ {- 1} 1. 2 5 ^ {m} r _ {\delta} ^ {- 1} \left(\frac {\eta}{k \sqrt {m}}\right) + V ^ {- 1} m ^ {3. 5} L r _ {\delta} ^ {- 1} \left(\frac {1}{k m} \left(\frac {\eta}{m 2 ^ {4 m L}}\right) ^ {\gamma_ {0}}\right) + 2 ^ {2 m} r _ {\delta} ^ {- 1} \left(\frac {\lambda}{k \sqrt {m}}\right)\right) \\ = \tilde {O} \left(V ^ {- 1} 1. 2 5 ^ {m} (k \sqrt {m}) ^ {\beta} \eta^ {- \beta} + V ^ {- 1} 2 ^ {5 m L \beta \gamma_ {0}} k ^ {\beta} \lambda^ {- \beta \gamma_ {0}}\right). \\ \end{array}
$$
Finally, optimizing over  $\lambda$  gives the claimed bound.
Remark C.15 (Representation length). If we assume the optimal solution  $\mathbf{h}^{\star}$  lies in the grid of size poly( $\varepsilon^{-1}$ ), then in case (II) of Theorem C.14, we have  $2^{2mL} = \text{poly}(\varepsilon^{-1})$ , which leads to a rate of  $\tilde{O}\left(poly(\varepsilon, k, m, \eta^{-1}, V^{-1}) \cdot T^{-\frac{1}{\gamma_2}}\right)$ , where  $\gamma_2$  is a constant that only depends on the agent's calibration error rate  $\beta$ . This is the result of replacing Lemma C.10 with the finite grid assumption.
Remark C.16 (Adaptive regret versus calibration). Our primary focus lies on calibration due to its characterization of agents' beliefs and the fact that it provides both upper and lower bounds to the principal's utility. This is particularly useful for the learning direction, as denoted by the lower bounds in Theorems 3.2 and C.14. However, a different form of adaptive guarantee would suffice here: one concerning (external) regret. Nevertheless, we do not focus on regret as a characterization as it doesn't offer the same upper bound guarantees — in fact, the principal could potentially extract more utility than  $V^{\star}$ . Additionally, regret-based assumptions tend to overly emphasize the agent's optimization techniques rather than maintaining a consistent belief about the action being executed.
D Supplementary Material for Section 4
D.1 Background on Sleeping Experts and ADANORMALHEDGE
We start the exposition of this part by introducing the sleeping experts problem [10, 31]. For each expert  $i \in [N]$  and round  $t \in [T]$ , let  $\ell_{t,i} \in [0,1]$  be the loss of expert  $i$ , and let  $I_{t,i}$  be an indicator that takes value  $I_{t,i} = 1$  if expert  $i$  is active at round  $t$  and  $I_{t,i} = 0$  if asleep. The interaction protocol at each round  $t$  goes as follows: The indicators  $(I_{t,i})_{i \in [N]}$  are revealed to the learner. The learner selects a probability distribution  $\pi_t \in \Delta([N])$  that is supported only on the set of active experts  $A_t \triangleq \{i : I_{t,i} = 1\}$ . The adversary selects a loss vector  $(\ell_{t,i})_{i \in [N]}$ . The learner then suffers expected loss  $\hat{\ell}_t = \mathbb{E}_{i \sim \pi_t}[\ell_{t,i}]$ . The regret with respect to each expert  $i$  only accounts for the rounds when  $i$  is awake, which, together with the fact that  $\pi_t$  is only supported on active experts, implies that
$$
\operatorname {R e g} _ {T} (i) = \sum_ {t \in [ T ]} I _ {t, i} \left(\hat {\ell} _ {t} - \ell_ {t, i}\right) \quad \Rightarrow \quad \operatorname {R e g} _ {T} = \max  _ {i} \operatorname {R e g} _ {T} (i) \tag {16}
$$
One of the algorithms that can be used to provide sublinear regret for the sleeping experts problem is ADANORMALHEDGE [44]. ADANORMALHEDGE is a powerful, parameter-free algorithm which provides regret bounds in terms of the cumulative magnitude of the instantaneous regrets, defined as:  $r_{t,i} = \hat{\ell}_t - \ell_{t,i}$  for all experts  $i \in [N]$ . As its name suggests, ADANORMALHEDGE uses the well-known algorithm HEDGE as a backbone; HEDGE maintains a probability distribution over experts at each round  $t$  and draws an expert from said distribution. After the expert's loss is revealed, the probability distribution for the next round  $t + 1$  is updated using a multiplicative weights argument. For bandit feedback (i.e., when only the chosen expert's loss is revealed to the learner), the multiplicative weights update rule uses an inverse propensity scoring estimator for each expert's loss in place of their real loss. The new element that ADANORMALHEDGE brings to the table is a way of defining the weights at each round  $t$ ; specifically, the weights are updated proportionally to the sum of instantaneous regret for each expert until round  $t$ . This allows the learner to obtain finer control over the total regret without needing extra parameters to tune the algorithm at each round. The exact regret guarantee that ADANORMALHEDGE obtains is stated formally below.
Lemma D.1 (ADANORMALHEDGE [44]). Let  $r_{t,i} = I_{t,i}\left(\hat{\ell}_t - \ell_{t,i}\right)$  be the instantaneous regret of any active expert  $i\in A_t$  at round  $t$ , and  $c_{t,i} = |r_{t,i}|$ . Then, ADANORMALHEDGE with prior  $q\in \Delta ([N])$  selects experts according to the following distribution
$$
\pi_ {t, i} \propto q _ {i} I _ {t, i} w \left(R _ {t - 1, i}, C _ {t - 1, i}\right), \text {w h e r e}
$$
$$
R _ {t - 1, i} = \sum_ {\tau \in [ t - 1 ]} r _ {\tau , i}, C _ {t - 1, i} = \sum_ {\tau \in [ t - 1 ]} c _ {\tau , i},
$$
$$
w (R, C) = \frac {1}{2} \left(\Phi (R + 1, C + 1) - \Phi (R - 1, C + 1)\right),
$$
$$
\Phi (R, C) = \exp \left(\frac {\operatorname* {m a x} \{0 , R \} ^ {2}}{3 C}\right)
$$
The regret of ADANORMALHEDGE against any distribution over experts  $u \in \Delta([N])$  is bounded by
$$
\operatorname {R e g} _ {T} (u) \leq O \left(\sqrt {\langle u , C _ {T} \rangle \cdot \left(D _ {K L} (u \| q) + \log \log T + \log \log N)\right)}\right).
$$
where by  $D_{KL}(u||q)$  we denote the KL-divergence between distributions  $u$  and  $q$ .
ADANORMALHEDGE can be used to obtain adaptive regret bounds by creating a sleeping expert  $(i, s)$  for each  $i \in [N]$ ,  $s \in [T]$  that has the same loss as expert  $i$  but is only awake after  $s$ .
Corollary D.2. Running ADANORMALHEDGE for the sleeping expert setting with prior  $q_{(i,s)} \propto \frac{1}{s^2}$  gives regret
$$
\operatorname {R e g} _ {t} ((i, s)) \leq O \left(\sqrt {(t - s) (\log (N s) + \log \log T)}\right),
$$
where  $T_{i} = \sum_{t=1}^{T} I_{t,i}$  is the total number of rounds in which  $i$  is active.
D.2 Formula for Computing  $Q_{t}$  when  $m = 2$
To obtain the explicit formula for  $Q_{t}$ , we first discretize the space of forecasts  $\mathcal{F}_P = [0,1]$  (since we focus on the case where  $m = 2$ ) to form set  $\mathcal{F}_P^\varepsilon = \{0,\varepsilon ,2\varepsilon ,\ldots ,1 - \varepsilon ,1\}$ . Then, we have that for each  $\hat{\mathbf{p}}\in \mathcal{F}_P^\varepsilon$ :
$$
\begin{array}{l} \underset {g \sim \pi_ {t}} {\mathbb {E}} \left[ \ell_ {t, g} \right] = \underset {g \sim \pi_ {t}} {\mathbb {E}} \left[ L _ {g} \left(\mathbf {h} _ {t}, \mathbf {p}\right) \right] = \sum_ {g \in A _ {t} (\mathcal {G})} \pi_ {t, g} w _ {i} (\mathbf {p}) \sigma \left(\mathbf {h} _ {t} - \mathbf {p}\right) \\ = \left(\mathbf {h} _ {t} - \mathbf {p}\right) \underbrace {\sum_ {i \in \mathcal {A} _ {A}} w _ {i} (\mathbf {p}) \sum_ {s \leq t} \left(\pi_ {t , g (s , i , + 1)} - \pi_ {t , g (s , i , - 1)}\right)} _ {Z _ {\mathbf {p}}} \tag {17} \\ \end{array}
$$
where we have omitted index  $j$  from the sleeping expert  $g$  since because  $m = 2$ , we can focus on a single action  $j$ . We assume WLOG that no forecast  $\mathbf{p} \in \mathcal{F}_P^\varepsilon$  falls exactly on the boundary of best response polytopes, so there is no tie-breaking needed. From Equation (17), we have that:
$$
\max  _ {\mathbf {h} _ {t} \in \mathcal {H} _ {P}} \underset {g \sim \pi_ {t}} {\mathbb {E}} \left[ \ell_ {t, g} \right] = \max  \left\{Z _ {\mathbf {p}}, 0 \right\} - \mathbf {p} Z _ {\mathbf {p}} \tag {18}
$$
where the equation also uses the fact that for  $m = 2$ ,  $\max_{\mathbf{h}_t \in \mathcal{H}_P} \mathbf{h}_t = 1$ .
In the final step, we map  $\mathbf{p}$  to the discretized grid of  $\mathcal{F}_P^\varepsilon$ . Let  $j\varepsilon, (j + 1)\varepsilon$  be two adjacent discretized points and  $q \in [0,1]$  such that:  $qZ_{j\varepsilon} + (1 - q)Z_{(j + 1)\varepsilon}0$ . Then, setting  $q_{t,j\varepsilon} = q$  and  $q_{t,(j + 1)\varepsilon} = 1 - q$  and using Equation (18) gives that
$$
\max_{\mathbf{h}_{t}\in \mathcal{H}_{P}}\underset { \begin{array}{c}g\sim \pi_{t}\\ \mathbf{p}\sim Q_{t} \end{array} }{\mathbb{E}}\left[\ell_{t,g}\right]\leq \varepsilon .
$$
D.3 Proof of Theorem 4.1
Proof of Theorem 4.1. We first specify how to build a set of sleeping experts settings from our problem definition. For that, consider the following set of experts:
$$
\mathcal {G} = \left\{g _ {(s, i, j, \sigma)}: s \in [ T ], i \in \mathcal {A} _ {A}, j \in \mathcal {A} _ {P}, \sigma \in \{\pm 1 \} \right\},
$$
i.e., we create a different expert for each round, each principal-agent action pair, and each  $\sigma$  (the use of which will be made clear in the next paragraph). For each expert  $g_{(s,i,j,(\sigma))} \in \mathcal{G}$  and  $t \in [T]$ , we define the loss, sleeping/awake indicator, and instantaneous regret respectively as:
$$
\ell_ {t, g _ {(s, i, j, \sigma)}} \triangleq L _ {g _ {(i, i, j, \sigma)}} (\mathbf {h} _ {t}, \mathbf {p} _ {t}) = w _ {i} (\mathbf {p} _ {t}) \cdot \boldsymbol {\sigma} \cdot (h _ {t, j} - p _ {t, j});
$$
$$
I _ {t, g _ {(s, i, j, \sigma)}} \triangleq \mathbf {1} \{t \geq s \};
$$
where by  $h_{t,j}, p_{t,j}$  we denote the  $j$ -th coordinate of the  $\mathbf{h}$  and  $\mathbf{p}$  vectors respectively.
Running ADANORMALHEDGE on the instance with  $\mathcal{G}$  that we specified above, with prior  $q_{g_{(s,i,j,\sigma)}}\propto \frac{1}{s^2}$  [44, Section 5.1] guarantees that  $\forall g_{(s,i,j,\sigma)}\in \mathcal{G}$ ,
$$
\operatorname {R e g} _ {t} \left(g _ {(s, i, j, \sigma)}\right) = \sum_ {\tau \in [ t ]} r _ {t, g _ {(s, i, \sigma)}} \leq \widetilde {O} \left(\sqrt {(t - s) \log (k m T)}\right). \tag {19}
$$
where  $\tilde{O}(\cdot)$  hides lower order poly-logarithmic terms. Therefore, by simulating the NRBR dynamics, we obtain that with probability at least  $\geq 1 - \delta$ ,  $\forall i \in [k]$  and  $1 \leq s < t \leq T$ ,
$$
\begin{array}{l} \operatorname {C a l E r r} _ {i} \left(\mathbf {h} _ {s: t}, \mathbf {p} _ {s: t}\right) = \frac {1}{t - s} \max  _ {j \in \mathcal {A} _ {P}} \max  _ {\sigma \in \{\pm 1 \}} \sum_ {\tau = s} ^ {t} I _ {\tau , g _ {(s, i, j, \sigma)}} \cdot \ell_ {\tau , g _ {(s, i, j, \sigma)}} (Definition2.3) \\ = \frac {1}{t - s} \max _ {j \in \mathcal {A} _ {P}} \max _ {\sigma \in \{\pm 1 \}} \underbrace {\sum_ {\tau = s} ^ {t} I _ {\tau , g _ {(s , i , j , \sigma)}} \cdot \left(\ell_ {\tau , g _ {(s , i , j , \sigma)}} - \underset {g \sim \pi_ {\tau}} {\mathbb {E}} \left[ \ell_ {\tau , g} \right]\right)} _ {\mathrm {R e g} _ {t} \left(g _ {(s, i, j, \sigma)}\right)} \\ + \frac {1}{t - s} \max _ {j \in \mathcal {A} _ {P}} \max _ {\sigma \in \{\pm 1 \}} \sum_ {\tau = s} ^ {t} I _ {\tau , g _ {(s, i, j, \sigma)}} \cdot \underset {g \sim \pi_ {\tau}} {\mathbb {E}} [ \ell_ {\tau , g} ] \\ = \frac {1}{t - s} \max  _ {j \in \mathcal {A} _ {P}} \max  _ {\sigma \in \{\pm 1 \}} \operatorname {R e g} _ {t} \left(g _ {(s, i, j, \sigma)}\right) + \frac {1}{t - s} \max  _ {j \in \mathcal {A} _ {P}} \max  _ {\sigma \in \{\pm 1 \}} \underbrace {\sum_ {\tau = s} ^ {t} \mathbb {E} _ {g \sim \pi_ {\tau}} \left[ L _ {g _ {(s , i , j , \sigma)}} \left(\mathbf {h} _ {\tau} , \mathbf {p} _ {\tau}\right) \right]} _ {\tau} (20) \\ \end{array}
$$
where for the first derivation we add and subtract  $\sum_{\tau} \hat{\ell}_{\tau}$  and use that because of the NRBR dynamics:  $\hat{\ell}_{\tau} = \mathbb{E}_{g \sim P_{\tau}}[\ell_{\tau, g}]$ , and for the last derivation we have used the definition of  $\ell_{\tau, g} = L_g(\mathbf{h}_{\tau}, \mathbf{p}_{\tau})$  (Equation (6)). We next upper bound the term  $\mathcal{T}$  as follows:
$$
\begin{array}{l} \mathcal{T} = \sum_{\tau = s}^{t}\underset { \begin{array}{c}g\sim \pi_{\tau}\\ \mathbf{p}\sim Q_{\tau} \end{array} }{\mathbb{E}}[L_{g}(\mathbf{h}_{\tau},\mathbf{p})] + \sum_{\tau = s}^{t}\underset { \begin{array}{c}g\sim \pi_{\tau}\\ \mathbf{p}\sim Q_{\tau} \end{array} }{\mathbb{E}}[L_{g}(\mathbf{h}_{\tau},\mathbf{p}_{\tau})] - \sum_{\tau = s}^{t}\underset { \begin{array}{c}g\sim \pi_{\tau}\\ \mathbf{p}\sim Q_{\tau} \end{array} }{\mathbb{E}}[L_{g}(\mathbf{h}_{\tau},\mathbf{p})] \\ \leq \sum_ {\tau = s} ^ {t} \max  _ {\mathbf {h} _ {\tau} \in \Delta (A _ {P})} \underset {\mathbf {p} \sim Q _ {\tau}} {\mathbb {E}} \left[ L _ {g} (\mathbf {h} _ {\tau}, \mathbf {p}) \right] + \sum_ {\tau = s} ^ {t} \underset {g \sim \pi_ {\tau}} {\mathbb {E}} \left[ L _ {g} (\mathbf {h} _ {\tau}, \mathbf {p} _ {\tau}) - \underset {\mathbf {p} \in Q _ {\tau}} {\mathbb {E}} \left[ L _ {g} (\mathbf {h} _ {\tau}, \mathbf {p}) \right] \right] \\ \leq \varepsilon \cdot (t - s) + \sqrt {(t - s) \log ((t - s) / \delta)} \\ \end{array}
$$
where the first inequality is by the property of  $\mathbf{h}_{\tau}$  being the best strategy for the principal, and the last one uses the fact that  $\max_{\mathbf{h}_{\tau} \in \Delta(A_P)} \mathbb{E}_{\substack{\mathbf{p} \sim Q_{\tau}}}^{g \sim \pi_{\tau}}[L_g(\mathbf{h}_{\tau}, \mathbf{p})] \leq \varepsilon$  from the NRBR Equation (8) and a martingale concentration bound on the second term.
Plugging the upper bound for  $Q$  back to Equation (20) and using the regret bound for AdaNormal-Hedge (Equation (19)) we get:
$$
\operatorname {C a l E r r} _ {i} \left(\mathbf {h} _ {s: t}, \mathbf {p} _ {s: t}\right) \leq \widetilde {O} \left(\frac {\log (k m T)}{t - s}\right) + O \left(\frac {\log ((t - s) / \delta)}{t - s}\right) + \varepsilon \leq r _ {t} (\delta) + \epsilon .
$$
E Supplementary Material for Section 5