SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We outline next how LAZYGDwOG works. LAZYGDwOG is a variant of the gradient descent without a gradient algorithm (GDwOG) of Flaxman et al. [26]. The main new component of the algorithm is that it separates the time horizon into epochs and for each epoch it runs an update of the GDwOG algorithm. During all the rounds that comprise an epoch ( $M$  in total), LAZYGDwOG presents the same (appropriately smoothed-out) strategy to the agent and observes the  $M$  different responses from the agent. The intuition behind repeating the same strategy for  $M$  rounds is that the principal wants to give the opportunity to the agent to recalibrate for a better forecast, i.e.,  $\lim_{M\to \infty}\frac{1}{M} |\{i\in [M]:\| \mathbf{p}_i - \mathbf{h}\| \geq \varepsilon_0\}| = 0$ . The remainder of the proof for Theorem 5.2 focuses on showing that when the calibrated forecasts converge to  $\mathbf{h}_t$ , then the principal's utility converges to the utility they would have gotten if the agent was perfectly best responding to  $\mathbf{h}_t$ .
6 Discussion and future directions
In this paper we introduced and studied learning in CSGs, where the agents best respond to the principal's actions based on calibrated forecasts that they have about them. Our work opens up several exciting avenues for future research. First, although our main results prove asymptotic convergence, it is an open question whether our exact convergence rates can be improved both for general CSGs and for specific cases of Stackelberg games (e.g., strategic classification in more general models compared to [21], pricing [1]). Second, it is an interesting question whether our definition of adaptive calibration (Def. 2.3) can actually hold for the sum over all binning functions, instead of just the maximum. Finally, to provide our asymptotic convergence results we assumed that the principal has access to the agent's calibration rate  $r_{\delta}(\cdot)$ ; some information regarding how  $\mathbf{p}_t$  's relate to  $\mathbf{h}_t$  's is necessary to leverage the fact that agents are calibrated. But we think that in some specific settings (e.g., strategic classification) there may actually exist extra information regarding the forecasts (compared to just knowing  $r_{\delta}(\cdot)$ ) that can be leveraged to design learning algorithms for the principal with faster convergence rates. We discuss these directions in more detail in Appendix F.
<sup>4</sup>We can define  $(0, \Pi)$ -adaptive calibration in continuous CSGs due to the continuous case in Theorem 4.1.
[37] Kwang-Sung Jun, Francesco Orabona, Stephen Wright, and Rebecca Willett. Improved strongly adaptive online learning using coin betting. In Artificial Intelligence and Statistics (AISTATS), pages 943-951. PMLR, 2017.
[38] Sham M Kakade and Dean P Foster. Deterministic calibration and Nash equilibrium. Journal of Computer and System Sciences, 74(1):115-130, 2008.
[39] Christopher Kiekintveld, Towhidul Islam, and Vladik Kreinovich. Security games with interval uncertainty. In Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, pages 231-238, 2013.
[40] Christian Kroer, Gabriele Farina, and Tuomas Sandholm. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 32, 2018.
[41] Daniel Lee, Georgy Noarov, Mallesh Pai, and Aaron Roth. Online minimax multiobjective optimization: Multicalbeating and other applications. In Advances in Neural Information Processing Systems, volume 35, pages 29051-29063, 2022.
[42] Yin Tat Lee, Aaron Sidford, and Santosh S Vempala. Efficient convex optimization with membership oracles. In Conference on Learning Theory (COLT), pages 1292-1294. PMLR, 2018.
[43] Joshua Letchford, Vincent Conitzer, and Kamesh Munagala. Learning and approximating the optimal strategy to commit to. In Algorithmic Game Theory, pages 250-262. Springer, 2009.
[44] Haipeng Luo and Robert E Schapire. Achieving all with no parameters: Adanormalhedge. In Conference on Learning Theory (COLT), pages 1286-1304. PMLR, 2015.
[45] Yishay Mansour, Mehryar Mohri, Jon Schneider, and Balasubramanian Sivan. Strategizing against learners in Bayesian games. In Conference on Learning Theory (COLT), pages 5221-5252. PMLR, 2022.
[46] Vidya Muthukumar and Anant Sahai. Robust commitments and partial reputation. In Proceedings of the 20th ACM Conference on Economics and Computation (EC), pages 637-638, 2019.
[47] Binghui Peng, Weiran Shen, Pingzhong Tang, and Song Zuo. Learning optimal strategies to commit to. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 33, pages 2149-2156, 2019.
[48] James Pita, Manish Jain, Milind Tambe, Fernando Ordónez, and Sarit Kraus. Robust solutions to Stackelberg games: Addressing bounded rationality and limited observations in human cognition. Artificial Intelligence, 174(15):1142-1171, 2010.
[49] Milind Tambe. Security and game theory: algorithms, deployed systems, lessons learned. Cambridge university press, 2011.
[50] Haifeng Xu, Long Tran-Thanh, and Nicholas R Jennings. Playing repeated security games with no prior knowledge. In Proceedings of the 2016 International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), pages 104-112, 2016.
[51] Tijana Zrnic, Eric Mazumdar, Shankar Sastry, and Michael Jordan. Who leads and who follows in strategic classification? In Advances in Neural Information Processing Systems, volume 34, pages 15257-15269, 2021.
[52] Song Zuo and Pingzhong Tang. Optimal machine strategies to commit to in two-person repeated games. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 29, 2015.
A Calibrated Forecasts Standard Definition
We give below the standard definition for asymptotic calibration of Foster and Vohra [30] for a sequence of binary outcomes, i.e.,  $\mathbf{h}_t\in A = \{0,1\} ,\forall t\in [T]$ . The forecasts  $\mathbf{p}_t$  take values in  $C = [0,1]$ . Let  $X$  denote the adaptive adversary generating the events' sequence (which is of infinite size), where the  $T$  first events are  $\mathbf{h}_1,\dots ,\mathbf{h}_T$ .
Definition A.1. A forecasting procedure  $\sigma$  is asymptotically calibrated if and only if for any adaptive adversary  $X$  that generates the sequence  $\mathbf{h}_1, \dots, \mathbf{h}_T \in A$  and the forecasting algorithm  $\sigma$  that generates (possibly random) forecasts  $\mathbf{p}_1, \dots, \mathbf{p}_T \in C$  on the same sequence, we have that the calibration score  $C_T(X, \sigma)$  goes to 0 as  $T \to \infty$ :
$$
C _ {T} (X, \sigma) \triangleq \sum_ {\mathbf {p} \in \mathcal {F} _ {P}} \frac {n _ {T} (\mathbf {p} ; \mathbf {h} , \sigma)}{T} | \rho_ {T} (\mathbf {p}; \mathbf {h}, \sigma) - \mathbf {p} | \tag {9}
$$
where  $n_T(\mathbf{p}; \mathbf{h}, \sigma) \triangleq \sum_{t \in [T]} \mathbf{1} \{ \mathbf{p}_t = \mathbf{p} \}$  is the number of times that  $\sigma$  predicts  $\mathbf{p}$  and  $\rho_T(\mathbf{p}; \mathbf{h}, \sigma) \triangleq \frac{\sum_{t \in [T]} \mathbf{h}_t \mathbf{1} \{ \mathbf{p}_t = \mathbf{p} \}}{n_T(\mathbf{p}; \mathbf{h}, \sigma)}$  be the fraction (empirical probability) of these times that the actual event was 1.
Note that in Eq. (9), while  $\mathcal{F}_P$  contains an infinite number of distinct  $\mathbf{p}$ 's (hence an infinite number of summands), for every finite  $T$ , there is only a finite number of  $\mathbf{p}$  where  $n_T(\mathbf{p}; \mathbf{h}, \sigma)$  is nonzero. Therefore,  $C_T$  is well-defined and finite.
Equivalently, the above definition states that for the infinite binning [28]  $\Pi = \{w_{x}(\mathbf{p}): x \in C\}$  where  $w_{x}(\mathbf{p}) = \mathbf{1}\{\mathbf{p} = x\}$ , the calibration score can be equivalently expressed as
$$
C _ {T} (X, \sigma) \triangleq \sum_ {w _ {x} \in \Pi} \frac {n _ {T} (x)}{T} \left\| \overline {{\mathbf {h}}} _ {T} (x) - \overline {{\mathbf {p}}} _ {T} (x) \right\|,
$$
where  $n_T(x) \triangleq \sum_{t=1}^{T} w_x(\mathbf{p}_t)$  is the number of times that forecast  $\mathbf{p}_t$  falls into bin  $x$ ,  $\bar{\mathbf{p}}_T(x) \triangleq \sum_{t=1}^{T} \frac{w_x(\mathbf{p}_t)}{n_T(x)} \cdot \mathbf{p}_t$  is the average forecast that activates bin  $x$ , which is equal to  $\sum_{t=1}^{T} \frac{w_x(\mathbf{p}_t)}{n_T(x)} \cdot x = x$  because  $w_x(\mathbf{p}_t)$  is nonzero if and only if  $\mathbf{p}_t = x$ , and  $\bar{\mathbf{h}}_T(x) \triangleq \sum_{t=1}^{T} \frac{w_x(\mathbf{p}_t)}{n_T(x)} \cdot \mathbf{h}_t$  is the average outcome corresponding to bin  $x$ . It follows that the score  $C_T$  is a sum of the calibration errors during interval  $[1:T]$  for all bins (with CalErr defined in Definition 2.3).
$$
C _ {T} (X, \sigma) = \sum_ {w _ {x} \in \Pi} \operatorname {C a l E r r} _ {x} \left(\mathbf {h} _ {1: T}, \mathbf {p} _ {1: T}\right).
$$
B Calibrated Forecasts Lead to No Swap Regret
In this section, we show the connection between no-swap-regret agents and adaptively calibrated ones. As a reminder, no-swap-regret agents (translated to our setting and notation for the ease of exposition) are defined as follows.
Definition B.1 (Agent's swap regret [10]). For a sequence of principal's strategies  $\mathbf{h}_1, \dots, \mathbf{h}_T \in \mathcal{H}_P$  and agent's actions  $y_1, \dots, y_T \in \mathcal{A}_A$ , the swap regret is defined as
$$
\mathrm {S w a p R e g} (\mathbf {h} _ {1: T}, y _ {1: T}) = \max _ {\pi : \mathcal {A} _ {A} \to \mathcal {A} _ {A}} \sum_ {t \in [ T ]} U _ {A} (\mathbf {h} _ {t}, \pi (y _ {t})) - \sum_ {t \in [ T ]} U _ {A} (\mathbf {h} _ {t}, y _ {t}).
$$
We say that an agent is a no-swap-regret agent, if for the sequence of actions  $\{y_t\}_{t \in [T]}$  that they are playing it holds that  $\operatorname{SwapReg}(\mathbf{h}_{1:T}, y_{1:T}) = o(T)$ .
We next show that calibrated forecasts lead to no swap regret actions for the agent.
Lemma B.2 (Calibrated forecasts lead to no swap regret). If the agent is  $(\varepsilon, \Pi)$ -adaptively calibrated, then the agent's swap regret on the sequence  $\mathbf{h}_{1:T}$  is bounded by the calibration error as follows:
- If the agent breaks ties deterministically, then with probability  $\geq 1 - \delta$ ,
$$
\operatorname {S w a p R e g} \left(\mathbf {h} _ {1: T}, y _ {1: T}\right) \leq 2 U _ {\max } m k T \left(r _ {\delta} (T) + \varepsilon\right) \in o (T).
$$
- If the agent breaks ties randomly, then with probability  $\geq 1 - 2\delta$
$$
\operatorname {S w a p R e g} \left(\mathbf {h} _ {1: T}, y _ {1: T}\right) \leq U _ {\max } \left(O \left(\sqrt {T k \log \left(\frac {k}{\delta}\right)}\right) + 2 m k T \left(r _ {\delta} (T) + \varepsilon\right)\right) \in o (T).
$$
where  $U_{\mathrm{max}} = \max_{\mathbf{h} \in \mathcal{A}_P} \max_{y \in \mathcal{A}_A} U_A(\mathbf{h}, y)$  is the maximum utility the agent can obtain (without constraining the agent to play best responses).
Proof. We first present the proof for the case that the agents break ties deterministically. To simplify notation, we use  $n_T(i) \coloneqq n_{[0:T]}(i)$ ,  $\bar{\mathbf{p}}_T \coloneqq \bar{\mathbf{p}}_{[0:T]}(i)$ , and  $\mathbf{h}_T(i) \coloneqq \mathbf{h}_{[0:T]}(i)$ .
Fix a  $\pi : \mathcal{A}_A \to \mathcal{A}_A$ . Then, with probability at least  $1 - \delta$ , we have that:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} U _ {A} \left(\mathbf {h} _ {t}, \pi \left(y _ {t}\right)\right) - \sum_ {t = 1} ^ {T} U _ {A} \left(\mathbf {h} _ {t}, y _ {t}\right) \\ = \sum_ {i \in \mathcal {A} _ {A}} \sum_ {t = 1} ^ {T} \mathbf {1} \left\{y _ {t} = i \right\} \left(U _ {A} \left(\mathbf {h} _ {t}, \pi (i)\right) - U _ {A} \left(\mathbf {h} _ {t}, i\right)\right) \quad (\text {r e w r i t i n g} y _ {t} \text {a s t h e e x a c t a c t i o n}) \\ \stackrel {(a)} {=} \sum_ {i \in \mathcal {A} _ {A}} \sum_ {t = 1} ^ {T} w _ {i} (\mathbf {p} _ {t}) \left(\left\langle \mathbf {h} _ {t}, U _ {A} (\cdot , \pi (i)) \right\rangle - \left\langle \mathbf {h} _ {t}, U _ {A} (\cdot , i) \right\rangle\right) (10) \\ = \sum_ {i \in \mathcal {A} _ {A}} n _ {T} (i) \left(\left\langle \bar {\mathbf {h}} _ {T} (i), U _ {A} (\cdot , \pi (i)) \right\rangle - \left\langle \bar {\mathbf {h}} _ {T} (i), U _ {A} (\cdot , i) \right\rangle\right) \\ = \sum_ {i \in \mathcal {A} _ {A}} n _ {T} (i) \left(\left\langle \bar {\mathbf {h}} _ {T} (i) - \bar {\mathbf {p}} _ {T} (i), U _ {A} (\cdot , \pi (i)) \right\rangle + \left\langle \bar {\mathbf {p}} _ {T} (i), U _ {A} (\cdot , \pi (i)) - U _ {A} (\cdot , i) \right\rangle + \left\langle \bar {\mathbf {p}} _ {T} (i) - \bar {\mathbf {h}} _ {T} (i), U _ {A} (\cdot , i) \right\rangle\right) \\ \stackrel {(b)} {\leq} \sum_ {i \in \mathcal {A} _ {A}} n _ {T} (i) \left\| \bar {\mathbf {p}} _ {T} (i) - \bar {\mathbf {h}} _ {T} (i) \right\| _ {\infty} \left(\| U _ {A} (\cdot , \pi (i)) \| _ {1} + \| U _ {A} (\cdot , i) \| _ {1}\right) \\ = 2 U _ {\max } m \cdot \sum_ {i \in \mathcal {A} _ {A}} T \cdot \operatorname {C a l E r r} _ {i} \left(\mathbf {h} _ {1: T}, \mathbf {p} _ {1: T}\right) (Def.2.3) \\ \leq 2 U _ {\max } m k T \left(r _ {\delta} (T) + \varepsilon\right). \\ \end{array}
$$
In the above equations, step (a) is due to the fact that agents best respond with a deterministic tie-breaking rule:  $y_{t} = i$  if and only if  $i \in \mathsf{BR}(\mathbf{p}_t)$  and  $i \succ j, \forall j \neq i$ , which is equivalent to  $w_{i}(\mathbf{p}_{t}) = 1$ . We have also used  $U_{A}(\cdot ,i)$  to denote the  $m$ -dimensional vector where the  $j$ th entry is the utility  $U_{A}(j,i)$ . Step (b) is because the second term
$$
\langle \bar {\mathbf {p}} _ {T} (i), U _ {A} (\cdot , \pi (i)) - U _ {A} (\cdot , i) \rangle = U _ {A} (\bar {\mathbf {p}} _ {T} (i), \pi (i)) - U _ {A} (\bar {\mathbf {p}} _ {T} (i), i)
$$
is non-positive since each  $\mathbf{p}_t$  with  $w_{i}(\mathbf{p}_{t}) = 1$  belongs to the best response polytope  $P_{i}$ , so does their average:  $\bar{\mathbf{p}}_t(i)\in P_i\iff i\in \mathrm{BR}(\bar{\mathbf{p}}_t(i))$
Since the above inequality holds for any  $\pi$ , it also holds after taking the maximum over all  $\pi : \mathcal{A}_A \to \mathcal{A}_A$ . Therefore, we have the same bound for the agent's swap regret.
Next, we move to the case when the agent breaks ties randomly. For a fixed  $\pi$ , we have that at every time step  $t$ ,
$$
\mathbb {E} _ {t - 1} \left[ U _ {A} \left(\mathbf {h} _ {t}, \pi (y _ {t})\right) \right] = \frac {\sum_ {i \in \mathrm {B R} (\mathbf {p} _ {t})} U _ {A} \left(\mathbf {h} _ {t} , \pi (i)\right)}{| \mathrm {B R} (\mathbf {p} _ {t}) |} = \sum_ {i \in \mathcal {A} _ {A}} w _ {i} (\mathbf {p} _ {t}) U _ {A} \left(\mathbf {h} _ {t}, \pi (i)\right).
$$
Therefore, by Azuma-Hoeffding's inequality, w.p.  $\geq 1 - \delta^{\prime}$ , we have
$$
\sum_ {t = 1} ^ {T} U _ {A} \left(\mathbf {h} _ {t}, \pi \left(y _ {t}\right)\right) \leq \sum_ {t = 1} ^ {T} w _ {i} \left(\mathbf {p} _ {t}\right) \sum_ {i \in \mathcal {A} _ {A}} U _ {A} \left(\mathbf {h} _ {t}, \pi (i)\right) + O \left(\sqrt {T \log \left(\frac {1}{\delta^ {\prime}}\right)}\right). \tag {11}
$$
Since all actions in  $\mathbb{R}(\mathbf{p}_t)$  have the same utility for the agents, we also have
$$
U _ {A} (\mathbf {h} _ {t}, y _ {t}) = \frac {\sum_ {i \in \mathrm {B R} (\mathbf {p} _ {t})} U _ {A} (\mathbf {h} _ {t} , i)}{| \mathrm {B R} (\mathbf {p} _ {t}) |} = \sum_ {i \in \mathcal {A} _ {A}} w _ {i} (\mathbf {p} _ {t}) U _ {A} (\mathbf {h} _ {t}, i).
$$
Therefore, using Equations (11) and (12), we have that with probability at least  $1 - \delta'$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} U _ {A} \left(\mathbf {h} _ {t}, \pi \left(y _ {t}\right)\right) - \sum_ {t = 1} ^ {T} U _ {A} \left(\mathbf {h} _ {t}, y _ {t}\right) \leq \sum_ {i \in \mathcal {A} _ {A}} \sum_ {t = 1} ^ {T} w _ {i} \left(\mathbf {p} _ {t}\right) \left(U _ {A} \left(\mathbf {h} _ {t}, \pi (i)\right) - U _ {A} \left(\mathbf {h} _ {t}, i\right)\right) \\ + O \left(\sqrt {T \log \left(\frac {1}{\delta^ {\prime}}\right)}\right). \tag {12} \\ \end{array}
$$
We can use the same arguments as above (from Equation (10) onwards) to bound the first term on the right hand side by  $2U_{\mathrm{max}}mkT(r_{\delta}(T) + \varepsilon)$  with probability  $1 - \delta$ . Finally, setting  $\delta' = \delta / M$  where  $M = k^k$  is the number of possible swap functions, and applying the union bound, we conclude that with probability  $\geq 1 - 2\delta$ , the swap regret is bounded by
$$
\operatorname {S w a p R e g} \left(\mathbf {h} _ {1: T}, y _ {1: T}\right) \leq U _ {\max } \left(O \left(\sqrt {T \left(k \log k + \log \left(\frac {1}{\delta}\right)\right)}\right) + 2 m k T \left(r _ {\delta} (T) + \varepsilon\right)\right).
$$
C Supplementary Material for Section 3
C.1 Proof of Theorem 3.1
We first formally restate the theorem with exact convergence rates (not just asymptotically).
Theorem C.1 (Formal version of Theorem 3.1). Assume that the agent is  $(\varepsilon, \Pi)$ -adaptively calibrated with rate  $r_{\delta}(\cdot)$  and let  $U_{\mathrm{max}} = \max_{\mathbf{h} \in \mathcal{H}_P} \max_{y \in \mathcal{A}_A} U_P(\mathbf{h}, y)$ . Then, for any sequence  $\{\mathbf{h}_t\}_{t \in [T]}$  for the principal's strategies in a repeated CSG, with probability at least  $1 - 2\delta$ , the principal's utility is upper bounded as:
$$
\sum_ {t \in [ T ]} U _ {P} \left(\mathbf {h} _ {t}, y _ {t}\right) \leq V ^ {\star} T + \alpha \left(U _ {\max }, m, k, T, r _ {\delta}, \delta , \varepsilon\right)
$$
where  $\alpha (U_{\mathrm{max}},m,k,T,r_{\delta},\delta ,\varepsilon) = U_{\mathrm{max}}\cdot m\cdot k\cdot T\cdot (r_{\delta}(T) + \varepsilon)$  when the agent uses deterministic tie-breaking and  $\alpha (U_{\mathrm{max}},m,k,T,r_{\delta},\delta ,\varepsilon) = U_{\mathrm{max}}\cdot m\cdot k\cdot T\cdot (r_{\delta}(T) + \varepsilon +\sqrt{T\log(1 / \delta)})$  when the agent uses randomized tie-breaking.
Proof. To simplify notation, we use  $n_T(i) \coloneqq n_{[0,T]}(i)$ . When the agent follows deterministic tie-breaking, we have:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} U _ {P} \left(\mathbf {h} _ {t}, y _ {t}\right) = \sum_ {i \in \mathcal {A} _ {A}} \sum_ {t = 1} ^ {T} w _ {i} \left(\mathbf {p} _ {t}\right) U _ {P} \left(\mathbf {h} _ {t}, i\right) \\ = \sum_ {i \in \mathcal {A} _ {A}} n _ {T} (i) U _ {P} (\bar {\mathbf {h}} _ {T} (i), i) \quad \text {(l i n e a r i t y o f} U _ {P} \text {i n t h e p r i n c i p a l s t r a t e g y)} \\ = \sum_ {i \in \mathcal {A} _ {A}} n _ {T} (i) \left(U _ {P} (\bar {\mathbf {p}} _ {T} (i), i) + \left\langle U _ {P} (\cdot , i), \bar {\mathbf {h}} _ {T} (i) - \bar {\mathbf {p}} _ {T} (i) \right\rangle\right) \\ (\pm \sum_ {i} U _ {P} (\bar {\mathbf {p}} _ {T} (i), i)) \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {\mathrm {(a)}} {\leq} \sum_ {i \in \mathcal {A} _ {A}} n _ {T} (i) V ^ {\star} + \sum_ {i \in \mathcal {A} _ {A}} n _ {T} (i) \left\| \bar {\mathbf {p}} _ {T} (i) - \bar {\mathbf {h}} _ {T} (i) \right\| _ {\infty} \cdot \| U _ {P} (\cdot , i) \| _ {1} \\ \stackrel {(b)} {\leq} V ^ {\star} T + U _ {\max } m \sum_ {i \in \mathcal {A} _ {A}} T \cdot \operatorname {C a l E r r} _ {i} \left(\mathbf {h} _ {1: T}, \mathbf {p} _ {1: T}\right) \\ \leq V ^ {\star} T + U _ {\max } \cdot k \cdot m \cdot T \cdot \left(r _ {\delta} (T) + \epsilon\right) \\ \end{array}
$$
where (a) is because  $U_{P}(\bar{\mathbf{p}}_{T}(i), i) \leq V^{\star}$  (since  $i \in \mathbb{R}(\bar{\mathbf{p}}_{T})$ ), the Hölder's inequality, and the fact that  $\| \cdot \|_2 \leq \| \cdot \|_1$ , and (b) is because of the definition of  $U_{\max}$  and Definition 2.3. The proof for the randomized tie-breaking setting has an extra term from Azuma-Hoeffding's inequality, similar to the proof of Theorem B.2.
C.2 Approximate membership oracle to conservative polytopes
In this section, we formally present the algorithm (Algorithm 4 for constructing an approximate membership oracle to the conservative best response polytope for each of the agent's action. The sample complexity of the oracle will be presented in Lemma C.2.
Algorithm 4: Approximate membership oracle for the conservative best response polytope (APPROXMEM)
Input: query  $\mathbf{h}\in \mathcal{H}_P$  original polytope  $P_{y}$ $(y\in \mathcal{A}_A)$  , approximation factor  $\varepsilon_{1}$  , conservatism factor  $\varepsilon_{2}$  , failure probability  $\varepsilon_{3}$    
Parameters: Number of epochs  $\Phi$  radius  $R$  calibration error  $\varepsilon_{\mathrm{cal}}$    
Let  $l$  be s.t.  $r_\delta (l) = \frac{\varepsilon_{\mathrm{cal}}}{k\sqrt{m}}$    
for epoch  $\phi \in [\Phi ]$  do Sample a point  $\mathbf{h}_{\phi}$  such that  $\| \mathbf{h}_{\phi} - \mathbf{h}\| _2 = R$  /\* Sample point  $\mathbf{h}_{\phi}$  close to h \*/ if  $\mathbf{h}_{\phi}\notin \mathcal{H}_{P}$  then RETURN FALSE /\* No longer inside the feasible set \*/ else Play strategy  $\mathbf{h}_{\phi}$  for  $l$  rounds.  $y_{\phi}\gets$  most frequent best-response action from agent during the  $l$  rounds. if  $y_{\phi}\neq y$  then RETURN FALSE /\* h is too close to  $P_{y_{\phi}}$  /\* For membership, output  $y_{\Phi}$  if all  $\{y_{\phi}\}_{\phi \in [\Phi ]}$  agree, and N/A otherwise \*/ RETURN TRUE
In Lemma C.2, we show that the parameters  $\Phi, \varepsilon_{\mathrm{cal}}, R$  can be tuned to achieve a wide range of parameters  $(\varepsilon_1, \varepsilon_2, \varepsilon_3)$ . In Proposition C.3, we provide explicit instantiations of the parameters for three special cases of  $(\varepsilon_1, \varepsilon_2, \varepsilon_3)$  that will be useful in later sections.
Lemma C.2. If the agent is  $(\varepsilon, \Pi)$ -adaptively calibrated with rate  $r_{\delta}(\cdot)$  and infinitesimal  $\varepsilon$ , and the parameters  $\Phi, R, \varepsilon_{cal}$  satisfy:
$$
\varepsilon_ {1} + \varepsilon_ {2} - R \geq \varepsilon_ {\text {c a l}} \tag {Condition 1}
$$
$$
\Phi \geq 1 0 \sqrt {m} \left(1 - \left(\frac {\varepsilon_ {c a l} + \varepsilon_ {2}}{R}\right) ^ {2}\right) ^ {\frac {m - 1}{2}} \log \left(\frac {1}{\varepsilon_ {3} - \delta}\right) \tag {Condition2}
$$
Then Algorithm 4 (APPROXMEM) returns an  $\varepsilon_{1}$ -approximate membership oracle to  $P_{y}^{-\varepsilon_{2}} = B_{2}(P_{y}, - \varepsilon_{2})$  with probability  $1 - \varepsilon_{3}$ , using no more than  $N_{\varepsilon_1,\varepsilon_2,\varepsilon_3} = O\left(\Phi r_{\delta}^{-1}(\frac{\varepsilon_{cal}}{k\sqrt{m}})\right)$  rounds of interactions with the agent.
Specifically, with probability  $1 - \varepsilon_{3}$ , APPROXMEM either returns TRUE which asserts that  $\mathbf{h} \in B_{2}(P_{y}^{-\varepsilon_{2}}, + \varepsilon_{1})$ , or returns FALSE which asserts that  $\mathbf{h} \notin B_{2}(P_{y}^{-\varepsilon_{2}}, -\varepsilon_{1})$ .
Proposition C.3 (Parameter settings). Assume  $\varepsilon_3 = \delta + T^{-2}$ . In the following three cases:  $\frac{\varepsilon_1}{\varepsilon_2} = \Theta(\sqrt{m}), 1, o(1)$ , the proposed setting of parameters  $\varepsilon_{cal}, R, \Phi$  satisfy (Condition 1) and (Condition 2) simultaneously.
Case I.  $\frac{\varepsilon_1}{\varepsilon_2} = \Theta (\sqrt{m})$
$$
\varepsilon_ {c a l} = \varepsilon_ {2}, \quad R = \frac {\varepsilon_ {1}}{2}, \quad \Phi = 1 0 \sqrt {m} \left(1 - \frac {4 \varepsilon_ {2}}{\varepsilon_ {1}}\right) ^ {\frac {m - 1}{2}} \log (T) = O (\sqrt {m} \log (T)).
$$
Case II.  $\frac{\varepsilon_1}{\varepsilon_2} = 1$
$$
\varepsilon_ {c a l} = 0. 1 \varepsilon_ {2}, \quad R = 1. 9 \varepsilon_ {2}, \quad \Phi = 1. 2 5 ^ {m} \log (T).
$$
Case III.  $\frac{\varepsilon_1}{\varepsilon_2} = o(1)$
$$
\varepsilon_ {c a l} = \frac {\varepsilon_ {1}}{6}, \quad R = \left(\varepsilon_ {2} + \frac {\varepsilon_ {1}}{6}\right) \left(1 + \frac {\varepsilon_ {1}}{2 \varepsilon_ {2}}\right), \quad \Phi = 1 0 \sqrt {m} \left(\frac {\varepsilon_ {2}}{\varepsilon_ {1}}\right) ^ {\frac {m}{2}} \log (T).
$$