SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nika Haghtalab<sup>1</sup>, Chara Podimata<sup>2</sup>, and Kunhe Yang<sup>1</sup>
<sup>1</sup>University of California, Berkeley, {nika,kunheyang}@berkeley.edu  
<sup>2</sup>MIT & Archimedes, podimata@mit.edu
In this paper, we introduce a generalization of the standard Stackelberg Games (SGs) framework: Calibrated Stackelberg Games (CSGs). In CSGs, a principal repeatedly interacts with an agent who (contrary to standard SGs) does not have direct access to the principal's action but instead best-responds to calibrated forecasts about it. CSG is a powerful modeling tool that goes beyond assuming that agents use ad hoc and highly specified algorithms for interacting in strategic settings and thus more robustly addresses real-life applications that SGs were originally intended to capture. Along with CSGs, we also introduce a stronger notion of calibration, termed adaptive calibration, that provides fine-grained any-time calibration guarantees against adversarial sequences. We give a general approach for obtaining adaptive calibration algorithms and specialize them for finite CSGs. In our main technical result, we show that in CSGs, the principal can achieve utility that converges to the optimum Stackelberg value of the game both in finite and continuous settings, and that no higher utility is achievable. Two prominent and immediate applications of our results are the settings of learning in Stackelberg Security Games and strategic classification, both against calibrated agents.
Stackelberg games (SGs) are a canonical model for strategic principal-agent interactions, considering a principal (or "leader") that commits to a strategy  $\mathbf{h}$  and an agent (or "follower") who observes this strategy and best respond by taking action  $\mathrm{BR}(\mathbf{h})$ . These games are inspired by real-world applications such as economic policy design (where a tax policymaker establishes rules for triggering audits before taxes are filed), defense (where a principal allocates security resources to high-risk targets before vulnerabilities are exploited) and many more, see e.g., [7, 22, 49, 35, 21, 15, 17]. By anticipating the agent's best-response, a principal who knows the agent's payoff function can calculate the optimal Stackelberg strategy guaranteeing her utility  $V^{\star}$ . In recent years, repeated SGs have gained popularity in addressing settings where the agent's payoff function is unknown to the principal. In this setting, the principal, who can only observe the agents' actions, aims to deploy a sequence of strategies  $\mathbf{h}_1,\ldots ,\mathbf{h}_T$  over  $T$  rounds whose average payoff is at least as good as  $V^{\star}$ , i.e., the value of her optimal strategy had she known the agent's payoffs in advance.
Despite the original intent, repeated SGs are often studied under strict assumptions on the agent's knowledge and algorithmic behavior. Examples include requiring the agent to best respond per round using  $y_{t} = \mathsf{BR}(\mathbf{h}_{t})$  [7, 21], necessitating the agent to precisely know the principal's strategy at all times (e.g., the attacker must anticipate the exact probabilistic allocation of the defender's security resources), or employing one of many online optimization algorithms whose every detail (down to the learning step size) can significantly impact the principal's utility [51].
In this paper, instead of working with such restrictive and often unrealistic assumptions on the agent's knowledge and behavior, we build on foundational decision theoretic concepts, such as forecasts and calibration [19, 29, 30]. In practice, while agents may not observe the principal's true strategies  $\mathbf{h}_t$ , they can form calibrated forecasts — a notion of consistency in beliefs about  $\mathbf{h}_t$  — to which they then best respond. Indeed, such a decision-theoretic perspective on game dynamics led to seminal results on converging to correlated and Nash equilibria in simultaneous multi-player games [29, 38]. Our work brings the perspective of calibrated forecasts to principal-agent games. We introduce Calibrated Stackelberg Games (CSG)—a class that is more general than standard SGs—and ask:
Q1. What characterizes principal's optimal utility in CSGs?
Q2. Are there natural forecasting algorithms for the agent that satisfy calibration?
Our Contributions. We answer both questions completely. For Q1, we show that the principal's optimal utility converges exactly to  $V^{\star}$ . For Q2, we give a general approach for obtaining a fine-grained any-time notion of calibration of independent interest and further specializing it to games.
Before we delve into the details of our contributions, we highlight two key aspects of our results. First, calibration is a common property of forecasting procedures shared by many algorithms, not any one particular algorithm defining the agent's behavior. Despite not constraining the agent to any particular algorithm, our answer to Q1 shows that the principal can meaningfully converge to  $V^{\star}$ , which is the value she could have achieved in a single-shot game had she known the agent's utility. Second, our definition and results immediately apply to two important Stackelberg settings; Stackelberg Security Games [49, 7, 33] and strategic classification [21]. As such, we obtain the first results for learning against calibrated agents in these settings too.
Our work contributes two concepts of independent interest (Section 2): first, CSGs that directly generalize the standard model of repeated SGs, and second, a notion of calibration, termed adaptive calibration. This notion which draws inspiration from adaptive regret bounds in online learning, provides fine-grained calibration guarantees for adversarial sequences.
Beyond the introduction of these models, we address an important property of CSGs in our answer to Q1. We show that the principal's optimal utility in CSGs converges to  $V^{\star}$ , nothing more or less, in games with finite (Section 3) or continuous (Section 5) actions spaces. Note that  $V^{\star}$  is a benchmark that gives both players more power: the principal knows the agent's utility and the agent observes the principal's strategy. We find it somewhat surprising then that the optimal achievable principal utility in CSGs, in which both players work with significantly less knowledge, converges to  $V^{\star}$  exactly.
As for our newly introduced notion of adaptive calibration (Section 4), we provide an answer to Q2 by giving a general approach for creating adaptively calibrated forecasting algorithms. This shows that adaptive calibration is not just an abstract notion, rather, it is a natural property with deep roots in the theory of online learning. Indeed, to obtain these results we draw inspirations from recent advances in the multicalibration literature [34] regarding simulating no-regret and best-response dynamics and the sleeping experts problem setting which has been a staple of the online learning literature [8, 31]. Furthermore, we specialize our approach for strategic settings by showing how standard calibration concepts (such as the "binning function") can be adapted to account for the agent's best-responses.
Repeated Stackelberg games. Learning optimal Stackelberg strategies has been studied in the offline [17] and the online setting, where only instantaneous best-responses are observable (i.e., no access to a best-response oracle). Key applications include Stackelberg Security Games (e.g., [12, 7, 47, 50]) and strategic classification (e.g., [21, 15, 3, 4]). Another line of work treats repeated games as an extensive form game and studies optimal strategies for infinite [52] or finite [16] horizons. Other works consider learning in the presence of non-myopic agents that best respond by maximizing discounted utilities [5, 33, 2]. The main distinction to our work is that in our setting, the agents have only calibrated forecasts regarding the principal's strategies (rather than full knowledge of them).
Stackelberg games beyond best responses. Recent works have studied variants of repeated Stackelberg games with different agent strategic behaviors beyond best responding. One example is no-regret learning, in which previous works have extensively investigated the relationship between the principal's cumulative utility and the single-shot Stackelberg value against agents that use mean-based learning algorithms [13], gradient descent [24, 25], no-external regret algorithms [13, 20, 51],
no-internal/swap regret algorithms [20, 45], and no-counterfactual internal regret algorithms [14]. Another research direction assumes agents approximately best respond due to uncertainty in the principal's strategy [11, 6, 46] or their own [43, 39, 40] and study robust Stackelberg equilibria [48, 32]. Most of the works here assume that the principal knows the agent's utility function with the exception of [51]. Core differences to our framework are that (1) we work in an online learning setting where the principal has to learn the agent's utility function from their responses; (2) we do not assume a specific agent algorithm but focus on properties of agent beliefs that are shared by many algorithms.
Calibration and application in games. The study of calibration, introduced by Dawid [19], dates back to seminal work by Foster and Vohra [30] and Hart [36] that showed the existence of asymptotic online calibration against any adversarial sequence. Applying calibration to game dynamics, Foster and Vohra [29] introduced the concept of calibrated learning, which refers to a player best responding to calibrated forecasts of others' actions. They demonstrated that the game dynamics of all players performing calibrated learning converge to the set of correlated equilibria. This is complemented by the results of [38, 27, 28] that show smooth and continuous variants of calibrated learning dynamics converge to Nash equilibrium. Our work differs from the above by studying game dynamics that converge to a Stackelberg equilibrium, where only the follower (agent) performs calibrated learning.
Adaptivity and sleeping experts. The notion of adaptive calibration introduced in Section 2 is related to the study of adaptivity of regret bounds in online learning [44, 18, 37]. Our design of adaptively calibrated forecasting algorithms builds on the multi-objective learning perspective of online (multi-)calibration [41, 34] and the powerful tool of sleeping experts [8, 31, 44] which has proven useful in various applications such as fairness [9].
2 Model & preliminaries
We begin this section with some basic definitions about forecasts, calibration, and games, and then introduce the class of games that we study; Calibrated Stackelberg Games (CSGs).
Adaptively Calibrated Forecasts. We use  $A$  to denote the space of outcomes and  $C \supseteq A$  to denote the space of forecasts. A (stochastic) forecasting procedure  $\sigma$  is an online procedure that takes any adversarial sequence of outcomes  $\mathbf{h}_t \in A$  for  $t \in [T]$ , and on round  $t$  outputs (possibly at random) forecast  $\mathbf{p}_t \in C$  solely based on outcomes and forecasts  $\mathbf{h}_{\tau}, \mathbf{p}_{\tau}$ , for  $\tau \in [t - 1]$ . To define calibrated forecasts, let us first introduce the notion of binning functions.
Definition 2.1 (Binning [28]). We call a set  $\Pi = \{w_i\}_{i\in [n]}a$  binning function, if each  $w_{i}:C\to [0,1]$  maps forecasts to real values in [0, 1], and for all  $\mathbf{p}\in C$  we have  $\sum_{i\in [n]}w_i(\mathbf{p}) = 1$
With the above binning functions, we define the adaptive calibration error with respect to  $\Pi$  as follows. At a high level, conditioned on any bin, the calibration error measures the difference between the expected forecasts that fall in that bin and the corresponding expected outcome.
Definition 2.2 (II-Adaptive Calibration Error). For any time interval  $[s, t]$ , let  $\mathbf{p}_{s:t}$  be the sequence of forecasts and  $\mathbf{h}_{s:t}$  be the sequence of outcomes. For a given binning  $\Pi = \{w_i\}_{i \in [n]}$  with size  $n$  and  $\forall i \in [n]$ , define the II-adaptive calibration error as
$$
\operatorname {C a l E r r} _ {i} \left(\mathbf {h} _ {s: t}, \mathbf {p} _ {s: t}\right) \triangleq \frac {n _ {[ s , t ]} (i)}{t - s} \cdot \left\| \bar {\mathbf {p}} _ {[ s, t ]} (i) - \bar {\mathbf {h}} _ {[ s, t ]} (i) \right\| _ {\infty}, \tag {1}
$$
where during interval  $[s,t]$ ,  $n_{[s,t]}(i) \triangleq \sum_{\tau=s}^{t} w_i(\mathbf{p}_{\tau})$  is the effective number of times that the forecast belongs to bin  $i$  (i.e., bin  $i$  is activated),  $\bar{\mathbf{p}}_{[s,t]}(i) \triangleq \sum_{\tau=s}^{t} \frac{w_i(\mathbf{p}_{\tau})}{n_{[s,t]}(i)} \cdot \mathbf{p}_{\tau}$  is the expected forecast that activates bin  $i$ ,  $\bar{\mathbf{h}}_{[s,t]}(i) \triangleq \sum_{\tau=s}^{t} \frac{w_i(\mathbf{p}_{\tau})}{n_{[s,t]}(i)} \cdot \mathbf{h}_{\tau}$  is the expected outcomes corresponding to bin  $i$ .
We say that a forecasting procedure is adaptively calibrated if it achieves vanishing calibration error on any adversarial sequence of outcomes and any sub-interval of time steps.
Definition 2.3  $((\varepsilon, \Pi)$ -Adaptively Calibrated Forecasts). A forecasting procedure  $\sigma$  is  $\varepsilon$ -adaptively calibrated to binning  $\Pi = \{w_i\}_{i \in [n]}$  with rate  $r_{\delta}(\cdot) \in o(1)$ , if for all adversarial sequences of actions  $\mathbf{h}_1, \dots, \mathbf{h}_T$ , where  $\mathbf{h}_t \in A$ ,  $\sigma$  outputs forecasts  $\mathbf{p}_t \in C$  for  $t \in [T]$  such that with probability at least  $1 - \delta$ , we have that  $\forall s, t$  such that  $1 \leq s < t \leq T$ , and  $\forall i \in [n]$ :
$$
\operatorname {C a l E r r} _ {i} \left(\mathbf {h} _ {s: t}, \mathbf {p} _ {s: t}\right) \leq r _ {\delta} (t - s) + \varepsilon .
$$
We remark that without adaptivity (i.e., for  $s = 1$  and  $t = T$ ), Definition 2.2 is weaker than the standard definition of calibration (e.g., [30], listed for completeness in Appendix A) in two ways: (1) standard calibration takes each prediction  $\mathbf{p} \in C$  as an independent bin, thus having infinitely many binning functions:  $w_{\mathbf{p}}(\cdot) = \delta_{\mathbf{p}}(\cdot)$ . Instead, we only require calibration with respect to the predefined binning  $\Pi$  which only contains a finite number of binning functions; (2) standard calibration cares about the summation over calibration error across bins, but we only consider the maximum error.
Stackelberg Games. A Stackelberg game is defined as the tuple  $(\mathcal{A}_P, \mathcal{A}_A, U_P, U_A)$ , where  $\mathcal{A}_P$  and  $\mathcal{A}_A$  are the principal and the agent action spaces respectively, and  $U_P: \mathcal{A}_P \times \mathcal{A}_A \to \mathbb{R}_+$  and  $U_A: \mathcal{A}_P \times \mathcal{A}_A \to \mathbb{R}_+$  are the principal and the agent utility functions respectively. For ease of exposition, we work with finite Stackelberg games (i.e.,  $|\mathcal{A}_P| = m$  and  $|\mathcal{A}_A| = k$ ) and generalize our results to continuous games in Section 5. When the principal plays action  $x \in \mathcal{A}_P$  and the agent plays action  $y \in \mathcal{A}_A$ , then the principal and the agent receive utilities  $U_P(x, y)$  and  $U_A(x, y)$  respectively. We also define the principal's strategy space as the simplex over actions:  $\mathcal{H}_P = \Delta(\mathcal{A}_P)$ . For a strategy  $\mathbf{h} \in \mathcal{H}_P$ , we oftentimes abuse notation slightly and write  $U_P(\mathbf{h}, y) := \mathbb{E}_{x \sim \mathbf{h}}[U_P(x, y)]$ .
Repeated Stackelberg games capture the repeated interaction between a principal and an agent over  $T$  rounds. What distinguishes Stackelberg games from other types of games is the inter-temporal relationship between the principal's action/strategy and the agent's response; specifically, the principal first commits to a strategy  $\mathbf{h}_t \in \mathcal{H}_P$  and the agent subsequently best-responds to it with  $y_t \in \mathcal{A}_A$ . Let  $\mathbf{p}_t \in \mathcal{F}_P = \mathcal{H}_P$  be the agent's belief regarding the principal's strategy at round  $t$ . In standard Stackelberg games:  $\mathbf{p}_t = \mathbf{h}_t$ , i.e., the agent has full knowledge of the principal's strategy. In this paper, we consider games where the agent does not in general know  $\mathbf{h}_t$  when playing, but they only best-respond according to their belief  $\mathbf{p}_t$ . The agent's best-response to belief  $\mathbf{p}_t$  according to her underlying utility function  $U_A$  is action  $y_t \in \mathcal{A}_A$  such that
$$
y _ {t} \in \mathbb {B R} (\mathbf {p} _ {t}) \quad \text {w h e r e} \quad \mathbb {B R} (\mathbf {p} _ {t}) = \underset {y \in \mathcal {A} _ {A}} {\operatorname {a r g m a x}} \underset {x \sim \mathbf {p} _ {t}} {\mathbb {E}} [ U _ {A} (x, y) ]. \tag {2}
$$
We often overload notation and write  $U_A(\mathbf{p}, y) \coloneqq \mathbb{E}_{x \sim \mathbf{p}}[U_A(x, y)]$ . Note that from Equation (2), the best-responses to  $\mathbf{p}_t$  form a set. If this set is not a singleton, we use either a deterministic or a randomized tie-breaking rule. For the deterministic tie-breaking rule, the agent breaks ties according to a predefined preference rule  $\succ$  over the set of actions  $\mathcal{A}_A$ . For the randomized tie-breaking rule, the agent chooses  $y_t$  by sampling from the set  $\mathrm{BR}(\mathbf{p}_t)$  uniformly at random, i.e.,  $y_t \sim \mathrm{Unif}(\mathrm{BR}(\mathbf{p}_t))$ .
The Stackelberg value of the game is the principal's optimal utility when the agent best responds:
$$
V^{\star} = \max_{\mathbf{h}^{\star}\in \mathcal{H}_{P}}\max_{y^{\star}\in \mathsf{BR}(\mathbf{h}^{\star})}U_{P}(\mathbf{h}^{\star},y^{\star}).
$$
In the above definition  $\mathbf{h}^{\star}$  is referred to as the principal's optimal strategy.
For an agent's action  $y \in \mathcal{A}_A$ , we define the corresponding best-response polytope  $P_y$  as the set of all of the agent's beliefs that induce  $y$  as the agent's best-response, i.e.,  $P_y = \{\mathbf{p} \in \mathcal{F}_P : y \in \mathrm{BR}(\mathbf{p})\}$ . We make the following standard assumption, which intuitively means that there are sufficiently many strategies that induce  $y^{\star}$  as the agent's best-response.
Assumption 2.4 (Regularity). The principal's optimal strategy  $\mathbf{h}^{\star} \in \Delta(\mathcal{A}_P)$  and the agent's optimal action  $y^{\star} \in \mathrm{BR}(\mathbf{h}^{\star})$  satisfy a regularity condition:  $P_{y^{\star}}$  contains an  $\ell_2$  ball of radius  $\eta > 0$ .
Interaction Protocol for Calibrated Stackelberg Games (CSGs)
(1) The principal plays strategy  $\mathbf{h}_t\in \mathcal{H}_P$
(2) The agent without observing  $\mathbf{h}_t$  forms a calibrated prediction  $\mathbf{p}_t$  (Def. 2.5) about  $\mathbf{h}_t$ .
(3) The agent best responds to  $\mathbf{p}_t$  by playing  $y_{t}\in \mathrm{BR}(\mathbf{p}_{t})$  ( $+$  tie-breaking).
(4) The principal observes  $y_{t}$  and experiences utility  $U_{P}(\mathbf{h}_{t},y_{t})$
(5) The agent observes  $\mathbf{h}_t$ , or an action sampled from  $\mathbf{h}_t$ .
Figure 1: Principal-Agent Interaction Protocol in a Round of a CSG
Calibrated Stackelberg Games. In CSGs (see Figure 1 for the principal-agent interaction protocol $^{1}$ ), the agent forms  $(\varepsilon, \Pi)$ -adaptively calibrated forecasts as their beliefs  $\mathbf{p}_t$  regarding  $\mathbf{h}_t$ .
<sup>1</sup>If the agent observes action  $x_{t}\sim \mathbf{h}_{t}$  instead of the mixed strategy  $\mathbf{h}_t$  , then they can still calibrate to the sequence of  $\mathbf{h}_t$  with an additional (vanishing) error term that comes from concentration inequalities.
We first define binning functions that are especially appropriate for forecasts in games. In CSGs, we define  $\Pi$  based on whether  $i$  is a best-response to the input calibrated forecast, i.e.,  $\forall \mathbf{p} \in \mathcal{F}_P$ :
$$
w _ {i} (\mathbf {p}) = \mathbf {1} \{i \in \mathrm {B R} (\mathbf {p}), i \succ j, \forall j \neq i \} \quad \text {(f o r t h e d e t e r m i n i s t i c t i e - b r e a k i n g)}
$$
$$
w _ {i} (\mathbf {p}) = \frac {\mathbf {1} \{i \in \mathrm {B R} (\mathbf {p}) \}}{| \mathrm {B R} (\mathbf {p}) |} \quad \text {(f o r t h e r a n d o m i z e d t i e - b r e a k i n g)}
$$
Note that both binning functions meet the conditions of Definition 2.1. Applying Definition 2.3 for calibrated agent forecasts in CSGs we have the following:
Definition 2.5 ( $\varepsilon$ -Adaptively Calibrated Agent for CSGs). The agent is called  $\varepsilon$ -adaptively calibrated with rate  $r_{\delta}(\cdot) \in o(1)$ , if for any sequence of principal strategies  $\mathbf{h}_1, \dots, \mathbf{h}_T \in \mathcal{H}_P$  the agent takes a sequence of actions  $y_1, \ldots, y_T$  that satisfy the following requirements: 1) there is a sequence of forecasts  $\mathbf{p}_t \in \mathcal{F}_P$  for  $t \in [T]$ , such that  $y_t \in \mathbb{R}(\mathbf{p}_t)$ , and 2) forecasts  $\mathbf{p}_1, \ldots, \mathbf{p}_T$  are  $\varepsilon$ -calibrated for binning  $\Pi$  with rate  $r_{\delta}(\cdot)$  with respect to the principal's strategies  $\mathbf{h}_1, \dots, \mathbf{h}_T$ .
We next review the fundamental constructs from Equation (1) and their intuitive meaning in this setting.  $n_{[s,t]}(i)\triangleq \sum_{\tau \in [s,t]}w_i(\mathbf{p}_\tau)$  is now the expected number of times that the forecast has induced action  $i$  from the agent as their best response during interval  $[s,t]$ ,  $\bar{\mathbf{p}}_{[s,t]}(i)\triangleq \sum_{\tau \in [s,t]}w_i(\mathbf{p}_\tau)\cdot \mathbf{p}_\tau /n_{[s,t]}(i)$  is the expected forecast that induces action  $i$  from the agent as their best response during interval  $[s,t]$ , and  $\bar{\mathbf{h}}_{[s,t]}(i)\triangleq \sum_{\tau \in [s,t]}w_i(\mathbf{p}_\tau)\cdot \mathbf{h}_\tau /n_{[s,t]}(i)$  is the expected principal strategy that induces action  $i$  from the agent as their best response during interval  $[s,t]$ . The requirement for an agent to be calibrated is quite mild, as the forecasts are binned only according to the best-response they induce.
3 Principal's learning algorithms
In this section (see Appendix C for full proofs and convergence rates), we study the relationship between the principal's Stackelberg value  $V^{\star}$  and the best utility the principal can obtain from learning to play a sequence of strategies  $\{\mathbf{h}_t\}_{t\in [T]}$  against calibrated agents, i.e.,  $\frac{1}{T}\sum_{t\in [T]}U_P(\mathbf{h}_t,y_t)$ . The relationship between  $V^{\star}$  and  $\frac{1}{T}\sum_{t\in [T]}U_P(\mathbf{h}_t,y_t)$  is not a priori clear. In the case of calibrated forecasts, the agents do not know the exact  $\mathbf{h}_t$  when they choose their response. Instead, they base their decisions on the history of the principal's strategies so far. A principal then may be able to create historical patterns that lead the agents to worse actions, thus obtaining better utility himself. Indeed, several works have shown how historical patterns can afford the principal much better utility than  $V^{\star}$  when the agents are no-regret [13, 20]. Surprisingly, we show that this is not the case when the agents are calibrated;  $\sum_{t\in [T]}\bar{U}_P(\mathbf{h}_t,y_t)$  is upper bounded by  $TV^{\star}$  and a term that is sublinear in  $T$  and depends on the calibration parameters.
Theorem 3.1. Assume that the agent is  $(\varepsilon, \Pi)$ -as calibrated with rate  $r_{\delta}(\cdot)$  and negligible  $\varepsilon$ . Then, for any sequence  $\{\mathbf{h}_t\}_{t \in [T]}$  for the principal's strategies in a CSG, with probability at least  $1 - 2\delta$ , the principal's utility is upper bounded as:  $\lim_{T \to \infty} \frac{1}{T} \sum_{t \in [T]} U_P(\mathbf{h}_t, y_t) \leq V^\star$ .