Title: Calibrated Stackelberg Games: Learning Optimal Commitments Against Calibrated Agents
Venue: NeurIPS 2023
Accepted: yes
Keywords: calibration

Stackelberg games

learning in repeated games

strategic agents

best response

strategic classification

Stackelberg Security Games
Abstract: In this paper, we introduce a generalization of the standard Stackelberg Games (SGs) framework: _Calibrated Stackelberg Games_. In CSGs, a principal repeatedly interacts with an agent who (contrary to standard SGs) does not have direct access to the principal's action but instead best responds to _calibrated forecasts_ about it. CSG is a powerful modeling tool that goes beyond assuming that agents use ad hoc and highly specified algorithms for interacting in strategic settings to infer the principal's actions  and thus more robustly addresses real-life applications that SGs were originally intended to capture. Along with CSGs, we also introduce a stronger notion of calibration, termed _adaptive calibration_, that provides fine-grained any-time calibration guarantees against adversarial sequences. We give a general approach for obtaining adaptive calibration algorithms and specialize them for finite CSGs. In our main technical result, we show that in CSGs, the principal can achieve utility that converges to the optimum Stackelberg value of the game both in _finite_ and _continuous_ settings and that no higher utility is achievable. Two prominent and immediate applications of our results are the settings of learning in Stackelberg Security Games and strategic classification, both against _calibrated_ agents.

Calibrated Stackelberg Games: Learning Optimal Commitments Against Calibrated Agents

Abstract

In this paper, we introduce a generalization of the standard Stackelberg Games (SGs) framework: Calibrated Stackelberg Games (CSGs) . In CSGs, a principal repeatedly interacts with an agent who (contrary to standard SGs) does not have direct access to the principal's action but instead best-responds to calibrated forecasts about it. CSG is a powerful modeling tool that goes beyond assuming that agents use ad hoc and highly specified algorithms for interacting in strategic settings and thus more robustly addresses real-life applications that SGs were originally intended to capture. Along with CSGs, we also introduce a stronger notion of calibration, termed adaptive calibration , that provides fine-grained any-time calibration guarantees against adversarial sequences. We give a general approach for obtaining adaptive calibration algorithms and specialize them for finite CSGs. In our main technical result, we show that in CSGs, the principal can achieve utility that converges to the optimum Stackelberg value of the game both in finite and continuous settings, and that no higher utility is achievable. Two prominent and immediate applications of our results are the settings of learning in Stackelberg Security Games and strategic classification, both against calibrated agents.

1 Introduction

Stackelberg games (SGs) are a canonical model for strategic principal-agent interactions, considering a principal (or 'leader') that commits to a strategy h and an agent (or 'follower') who observes this strategy and best respond by taking action BR ( h ) . These games are inspired by real-world applications such as economic policy design (where a tax policymaker establishes rules for triggering audits before taxes are filed), defense (where a principal allocates security resources to high-risk targets before vulnerabilities are exploited) and many more, see e.g., [7, 22, 49, 35, 21, 15, 17]. By anticipating the agent's best-response, a principal who knows the agent's payoff function can calculate the optimal Stackelberg strategy guaranteeing her utility V ⋆ . In recent years, repeated SGs have gained popularity in addressing settings where the agent's payoff function is unknown to the principal. In this setting, the principal, who can only observe the agents' actions, aims to deploy a sequence of strategies h 1 , . . . , h T over T rounds whose average payoff is at least as good as V ⋆ , i.e., the value of her optimal strategy had she known the agent's payoffs in advance.

Despite the original intent, repeated SGs are often studied under strict assumptions on the agent's knowledge and algorithmic behavior. Examples include requiring the agent to best respond per round using y t = BR ( h t ) [7, 21], necessitating the agent to precisely know the principal's strategy at all times (e.g., the attacker must anticipate the exact probabilistic allocation of the defender's security resources), or employing one of many online optimization algorithms whose every detail (down to the learning step size) can significantly impact the principal's utility [51].

In this paper, instead of working with such restrictive and often unrealistic assumptions on the agent's knowledge and behavior, we build on foundational decision theoretic concepts, such as forecasts and calibration [19, 29, 30]. In practice, while agents may not observe the principal's true strategies h t , they can form calibrated forecasts -anotion of consistency in beliefs about h t -to which they then best respond. Indeed, such a decision-theoretic perspective on game dynamics led to seminal results on converging to correlated and Nash equilibria in simultaneous multi-player games [29, 38]. Our work brings the perspective of calibrated forecasts to principal-agent games. We introduce Calibrated Stackelberg Games (CSG) -a class that is more general than standard SGs- and ask:

Q1. What characterizes principal's optimal utility in CSGs?

Q2. Are there natural forecasting algorithms for the agent that satisfy calibration?

Our Contributions. We answer both questions completely. For Q1 , we show that the principal's optimal utility converges exactly to V ⋆ . For Q2 , we give a general approach for obtaining a finegrained any-time notion of calibration of independent interest and further specializing it to games.

Before we delve into the details of our contributions, we highlight two key aspects of our results. First, calibration is a common property of forecasting procedures shared by many algorithms, not any one particular algorithm defining the agent's behavior. Despite not constraining the agent to any particular algorithm, our answer to Q1 shows that the principal can meaningfully converge to V ⋆ , which is the value she could have achieved in a single-shot game had she known the agent's utility. Second, our definition and results immediately apply to two important Stackelberg settings; Stackelberg Security Games [49, 7, 33] and strategic classification [21]. As such, we obtain the first results for learning against calibrated agents in these settings too.

Our work contributes two concepts of independent interest (Section 2): first, CSGs that directly generalize the standard model of repeated SGs, and second, a notion of calibration, termed adaptive calibration . This notion which draws inspiration from adaptive regret bounds in online learning, provides fine-grained calibration guarantees for adversarial sequences.

Beyond the introduction of these models, we address an important property of CSGs in our answer to Q1 . We show that the principal's optimal utility in CSGs converges to V ⋆ , nothing more or less, in games with finite (Section 3) or continuous (Section 5) actions spaces. Note that V ⋆ is a benchmark that gives both players more power: the principal knows the agent's utility and the agent observes the principal's strategy. We find it somewhat surprising then that the optimal achievable principal utility in CSGs, in which both players work with significantly less knowledge, converges to V ⋆ exactly.

As for our newly introduced notion of adaptive calibration (Section 4), we provide an answer to Q2 by giving a general approach for creating adaptively calibrated forecasting algorithms. This shows that adaptive calibration is not just an abstract notion, rather, it is a natural property with deep roots in the theory of online learning. Indeed, to obtain these results we draw inspirations from recent advances in the multicalibration literature [34] regarding simulating no-regret and best-response dynamics and the sleeping experts problem setting which has been a staple of the online learning literature [8, 31]. Furthermore, we specialize our approach for strategic settings by showing how standard calibration concepts (such as the 'binning function') can be adapted to account for the agent's best-responses.

1.1 Related work

Repeated Stackelberg games. Learning optimal Stackelberg strategies has been studied in the offline [17] and the online setting, where only instantaneous best-responses are observable (i.e., no access to a best-response oracle). Key applications include Stackelberg Security Games (e.g., [12, 7, 47, 50]) and strategic classification (e.g., [21, 15, 3, 4]). Another line of work treats repeated games as an extensive form game and studies optimal strategies for infinite [52] or finite [16] horizons. Other works consider learning in the presence of non-myopic agents that best respond by maximizing discounted utilities [5, 33, 2]. The main distinction to our work is that in our setting, the agents have only calibrated forecasts regarding the principal's strategies (rather than full knowledge of them).

Stackelberg games beyond best responses. Recent works have studied variants of repeated Stackelberg games with different agent strategic behaviors beyond best responding. One example is no-regret learning, in which previous works have extensively investigated the relationship between the principal's cumulative utility and the single-shot Stackelberg value against agents that use meanbased learning algorithms [13], gradient descent [24, 25], no-external regret algorithms [13, 20, 51],

no-internal/swap regret algorithms [20, 45], and no-counterfactual internal regret algorithms [14]. Another research direction assumes agents approximately best respond due to uncertainty in the principal's strategy [11, 6, 46] or their own [43, 39, 40] and study robust Stackelberg equilibria [48, 32]. Most of the works here assume that the principal knows the agent's utility function with the exception of [51]. Core differences to our framework are that (1) we work in an online learning setting where the principal has to learn the agent's utility function from their responses; (2) we do not assume a specific agent algorithm but focus on properties of agent beliefs that are shared by many algorithms.

Calibration and application in games. The study of calibration, introduced by Dawid [19], dates back to seminal work by Foster and Vohra [30] and Hart [36] that showed the existence of asymptotic online calibration against any adversarial sequence. Applying calibration to game dynamics, Foster and Vohra [29] introduced the concept of calibrated learning , which refers to a player best responding to calibrated forecasts of others' actions. They demonstrated that the game dynamics of all players performing calibrated learning converge to the set of correlated equilibria. This is complemented by the results of [38, 27, 28] that show smooth and continuous variants of calibrated learning dynamics converge to Nash equilibrium. Our work differs from the above by studying game dynamics that converge to a Stackelberg equilibrium, where only the follower (agent) performs calibrated learning.

Adaptivity and sleeping experts. The notion of adaptive calibration introduced in Section 2 is related to the study of adaptivity of regret bounds in online learning [44, 18, 37]. Our design of adaptively calibrated forecasting algorithms builds on the multi-objective learning perspective of online (multi-)calibration [41, 34] and the powerful tool of sleeping experts [8, 31, 44] which has proven useful in various applications such as fairness [9].

2 Model & preliminaries

We begin this section with some basic definitions about forecasts, calibration, and games, and then introduce the class of games that we study; Calibrated Stackelberg Games (CSGs).

Adaptively Calibrated Forecasts. We use A to denote the space of outcomes and C ⊇ A to denote the space of forecasts. A (stochastic) forecasting procedure σ is an online procedure that takes any adversarial sequence of outcomes h t ∈ A for t ∈ [ T ] , and on round t outputs (possibly at random) forecast p t ∈ C solely based on outcomes and forecasts h τ , p τ , for τ ∈ [ t -1] . To define calibrated forecasts, let us first introduce the notion of binning functions .

Definition 2.1 (Binning [28]) . We call a set Π = { w i } i ∈ [ n ] a binning function , if each w i : C → [0 , 1] maps forecasts to real values in [0 , 1] , and for all p ∈ C we have ∑ i ∈ [ n ] w i ( p ) = 1 .

With the above binning functions, we define the adaptive calibration error with respect to Π as follows. At a high level, conditioned on any bin, the calibration error measures the difference between the expected forecasts that fall in that bin and the corresponding expected outcome.

Definition 2.2 ( Π -Adaptive Calibration Error) . For any time interval [ s, t ] , let p s : t be the sequence of forecasts and h s : t be the sequence of outcomes. For a given binning Π = { w i } i ∈ [ n ] with size n , and ∀ i ∈ [ n ] , define the Π -adaptive calibration error as

where during interval [ s, t ] , n [ s,t ] ( i ) ≜ ∑ t τ = s w i ( p τ ) is the effective number of times that the forecast belongs to bin i (i.e., bin i is activated), ¯ p [ s,t ] ( i ) ≜ ∑ t τ = s w i ( p τ ) n [ s,t ] ( i ) · p τ is the expected forecast that activates bin i , ¯ h [ s,t ] ( i ) ≜ ∑ t τ = s w i ( p τ ) n [ s,t ] ( i ) · h τ is the expected outcomes corresponding to bin i .

We say that a forecasting procedure is adaptively calibrated if it achieves vanishing calibration error on any adversarial sequence of outcomes and any sub-interval of time steps.

Definition 2.3 ( ( ε, Π) -Adaptively Calibrated Forecasts) . A forecasting procedure σ is ε -adaptively calibrated to binning Π = { w i } i ∈ [ n ] with rate r δ ( · ) ∈ o (1) , if for all adversarial sequences of actions h 1 , · · · , h T , where h t ∈ A , σ outputs forecasts p t ∈ C for t ∈ [ T ] such that with probability at least 1 -δ, we have that ∀ s, t such that 1 ≤ s < t ≤ T , and ∀ i ∈ [ n ] :

We remark that without adaptivity (i.e., for s = 1 and t = T ), Definition 2.2 is weaker than the standard definition of calibration (e.g., [30], listed for completeness in Appendix A) in two ways: (1) standard calibration takes each prediction p ∈ C as an independent bin, thus having infinitely many binning functions: w p ( · ) = δ p ( · ) . Instead, we only require calibration with respect to the predefined binning Π which only contains a finite number of binning functions; (2) standard calibration cares about the summation over calibration error across bins, but we only consider the maximum error.

Stackelberg Games. A Stackelberg game is defined as the tuple ( A P , A A , U P , U A ) , where A P and A A are the principal and the agent action spaces respectively, and U P : A P ×A A → R + and U A : A P × A A → R + are the principal and the agent utility functions respectively. For ease of exposition, we work with finite Stackelberg games (i.e., |A P | = m and |A A | = k ) and generalize our results to continuous games in Section 5. When the principal plays action x ∈ A P and the agent plays action y ∈ A A , then the principal and the agent receive utilities U P ( x, y ) and U A ( x, y ) respectively. We also define the principal's strategy space as the simplex over actions: H P = ∆( A P ) . For a strategy h ∈ H P , we oftentimes abuse notation slightly and write U P ( h , y ) := E x ∼ h [ U P ( x, y )] .

Repeated Stackelberg games capture the repeated interaction between a principal and an agent over T rounds. What distinguishes Stackelberg games from other types of games is the inter-temporal relationship between the principal's action/strategy and the agent's response; specifically, the principal first commits to a strategy h t ∈ H P and the agent subsequently best-responds to it with y t ∈ A A . Let p t ∈ F P = H P be the agent's belief regarding the principal's strategy at round t . In standard Stackelberg games: p t = h t , i.e., the agent has full knowledge of the principal's strategy. In this paper, we consider games where the agent does not in general know h t when playing, but they only best-respond according to their belief p t . The agent's best-response to belief p t according to her underlying utility function U A is action y t ∈ A A such that

We often overload notation and write U A ( p , y ) := E x ∼ p [ U A ( x, y )] . Note that from Equation (2), the best-responses to p t form a set . If this set is not a singleton, we use either a deterministic or a randomized tie-breaking rule. For the deterministic tie-breaking rule, the agent breaks ties according to a predefined preference rule ≻ over the set of actions A A . For the randomized tie-breaking rule, the agent chooses y t by sampling from the set BR ( p t ) uniformly at random, i.e., y t ∼ Unif ( BR ( p t )) .

The Stackelberg value of the game is the principal's optimal utility when the agent best responds:

In the above definition h ⋆ is referred to as the principal's optimal strategy.

For an agent's action y ∈ A A , we define the corresponding best-response polytope P y as the set of all of the agent's beliefs that induce y as the agent's best-response, i.e., P y = { p ∈ F P : y ∈ BR ( p ) } . We make the following standard assumption, which intuitively means that there are sufficiently many strategies that induce y ⋆ as the agent's best-response.

Assumption 2.4 (Regularity) . The principal's optimal strategy h ⋆ ∈ ∆( A P ) and the agent's optimal action y ⋆ ∈ BR ( h ⋆ ) satisfy a regularity condition : P y ⋆ contains an ℓ 2 ball of radius η > 0 .

Interaction Protocol for Calibrated Stackelberg Games (CSGs)

(1) The principal plays strategy h t ∈ H P .

(2) The agent without observing h t forms a calibrated prediction p t (Def. 2.5) about h t .

(3) The agent best responds to p t by playing y t ∈ BR ( p t ) ( + tie-breaking).

(4) The principal observes y t and experiences utility U P ( h t , y t ) .

(5) The agent observes h t , or an action sampled from h t . 1

Calibrated Stackelberg Games. In CSGs (see Figure 1 for the principal-agent interaction protocol 1 ), the agent forms ( ε, Π) -adaptively calibrated forecasts as their beliefs p t regarding h t .

We first define binning functions that are especially appropriate for forecasts in games. In CSGs, we define Π based on whether i is a best-response to the input calibrated forecast, i.e., ∀ p ∈ F P :

̸

Note that both binning functions meet the conditions of Definition 2.1. Applying Definition 2.3 for calibrated agent forecasts in CSGs we have the following:

Definition 2.5 ( ε -Adaptively Calibrated Agent for CSGs) . The agent is called ε -adaptively calibrated with rate r δ ( · ) ∈ o (1) , if for any sequence of principal strategies h 1 , · · · , h T ∈ H P the agent takes a sequence of actions y 1 , . . . , y T that satisfy the following requirements: 1) there is a sequence of forecasts p t ∈ F P for t ∈ [ T ] , such that y t ∈ BR ( p t ) , and 2) forecasts p 1 , . . . , p T are ε -calibrated for binning Π with rate r δ ( · ) with respect to the principal's strategies h 1 , · · · , h T .

Wenext review the fundamental constructs from Equation (1) and their intuitive meaning in this setting. n [ s,t ] ( i ) ≜ ∑ τ ∈ [ s,t ] w i ( p τ ) is now the expected number of times that the forecast has induced action i from the agent as their best response during interval [ s, t ] , ¯ p [ s,t ] ( i ) ≜ ∑ τ ∈ [ s,t ] w i ( p τ ) · p τ /n [ s,t ] ( i ) is the expected forecast that induces action i from the agent as their best response during interval [ s, t ] , and ¯ h [ s,t ] ( i ) ≜ ∑ τ ∈ [ s,t ] w i ( p τ ) · h τ /n [ s,t ] ( i ) is the expected principal strategy that induces action i from the agent as their best response during interval [ s, t ] . The requirement for an agent to be calibrated is quite mild, as the forecasts are binned only according to the best-response they induce.

3 Principal's learning algorithms

In this section (see Appendix C for full proofs and convergence rates), we study the relationship between the principal's Stackelberg value V ⋆ and the best utility the principal can obtain from learning to play a sequence of strategies { h t } t ∈ [ T ] against calibrated agents, i.e., 1 T ∑ t ∈ [ T ] U P ( h t , y t ) . The relationship between V ⋆ and 1 T ∑ t ∈ [ T ] U P ( h t , y t ) is not a priori clear. In the case of calibrated forecasts, the agents do not know the exact h t when they choose their response. Instead, they base their decisions on the history of the principal's strategies so far. A principal then may be able to create historical patterns that lead the agents to worse actions, thus obtaining better utility himself. Indeed, several works have shown how historical patterns can afford the principal much better utility than V ⋆ when the agents are no-regret [13, 20]. Surprisingly, we show that this is not the case when the agents are calibrated; ∑ t ∈ [ T ] U P ( h t , y t ) is upper bounded by TV ⋆ and a term that is sublinear in T and depends on the calibration parameters. 2

Theorem 3.1. Assume that the agent is ( ε, Π) -as calibrated with rate r δ ( · ) and negligible ε . Then, for any sequence { h t } t ∈ [ T ] for the principal's strategies in a CSG, with probability at least 1 -2 δ , the principal's utility is upper bounded as: lim T →∞ 1 T ∑ t ∈ [ T ] U P ( h t , y t ) ≤ V ⋆ .

Proof sketch. We sketch the proof for the deterministic tie-breaking, as the randomized one needs just an application of Azuma-Hoeffding. We first rewrite ∑ t ∈ [ T ] U P ( h t , y t ) partitioned in the principal's utility for each round-specific forecast that induces action i as the best-response from the agent, for all actions i ∈ A A : ∑ t ∈ [ T ] U P ( h t , y t ) = ∑ i ∈A A ∑ t ∈ [ T ] w i ( p t ) · U P ( h t , i ) . This equivalence holds because each p t maps to a single best response i ∈ A A (deterministic tie-breaking). Because of the linearity of the principal's reward in the principal's strategy: ∑ i ∈A A ∑ t ∈ [ T ] w i ( p t ) · U P ( h t , i ) = ∑ i ∈A A n T ( i ) · U P ( ¯ h T , i ) , where n T ( i ) = n [0 ,T ] ( i ) . Adding and subtracting U P (¯ p T , i ) from the above, we now need to bound the quantity: ∑ i ∈A A n T ( i )( U P (¯ p T ( i ) , i ) + ⟨ U P ( · , i ) , ¯ h T ( i ) -¯ p T ( i ) ⟩ ) . The first term is upper bounded by V ⋆ T ; note that i ∈ BR (¯ p T ) (and V ⋆ = max h max y ∈ BR ( h ) U P ( h , y ) ) since p t ( i ) ∈ P i and hence that should also be true for the average of p t ( i ) over t rounds. The second summand is bounded by the calibration error of Definition 2.3.

ε

B

2

(

x

,

ε

)

x

ε

S

B

2

(

x

,

ε

)

x

ε

S

B

2

(

S

,

ε

) =

⋃

x

∈

S

B

2

(

x

,

ε

)

B

2

(

S

, -

ε

) = {

x

∈

S

:

B

2

(

x

,

ε

) ⊆

S

}

On the other hand, it may seem that because the agent's behavior is less specified when she uses calibrated forecasts (as opposed to full knowledge), the principal may only be able to extract much less utility compared to V ⋆ . Again, we show that this is not the case and that there exist algorithms for the principal such that the sequence of strategies { h t } t ∈ [ T ] is asymptotically approaching V ⋆ .

Theorem 3.2. There exists an algorithm for the principal in CSGs that achieves average utility: lim T →∞ 1 T ∑ t ∈ [ T ] U P ( h t , y t ) ≥ V ⋆ .

Algorithm 1 is an explore-then-commit algorithm; it first estimates an appropriate strategy for the principal ˜ h (EXPLORE), and then repeatedly plays it until the end (COMMIT). In the remainder of the section, we sketch the proof for Theorem 3.2 and point to exact lemma statements in the Appendix. Let T 1 , T 2 denote the set of rounds that belong in the EXPLORE and COMMIT phase respectively.

To elaborate on the objectives of the EXPLORE phase, let us first consider a setting with zero calibration error, where the agent's forecasting algorithm is perfectly and adaptively calibrated, leading to y t = BR ( h t ) at every round. The task for the EXPLORE phase simplifies to identifying a near-optimal strategy ˜ h through best response oracles that satisfies U P ( ˜ h , BR ( ˜ h )) ≥ V ⋆ -ε 1 for a predetermined ε 1 . We formalize this property in (P1) . Given that the agent is perfectly calibrated, in the COMMIT phase, the agent always plays ˜ y = BR ( ˜ h ) , leading to an upper bound of ε 1 | T 2 | on the Stackelberg regret. Hence, Algorithm 1's regret is bounded by V ⋆ | T 1 | + ε 1 | T 2 | .

(P1) U P ( ˜ h , ˜ y ) ≥ V ⋆ -ε 1 for ˜ y ∈ BR ( ˜ h ) , i.e., ( ˜ h , ˜ y ) is an approximate Stackelberg equilibrium.

̸

Moving away from the idealized setting, we must account for possible discrepancies between y t and BR ( h t ) due to calibration error. This introduces: (i) An increased sample complexity | T 1 | in the EXPLORE phase, given the necessity to learn a near-optimal strategy from noisy responses; (ii) Potential deviations from the action ˜ y = BR ( ˜ h ) in the COMMIT phase due to miscalibrations in belief. To address the first challenge, we employ Algorithm 2, which constructs an approximate best response oracle by repeatedly interacting with a calibrated agent. For the second challenge, we require our learned policy ˜ h to be robust against inaccurate forecasts. This is reflected in condition (P2) , which necessitates the ball of radius ε 2 around ˜ h to be fully contained in the polytope P ˜ y . The critical insight from (P2) is: for any forecast p t that results in a best response y t = BR ( p t ) = ˜ y , there must be a minimum distance of ε 2 separating p t from ˜ h . We will now proceed to formalize (P2) , but before delving into that, it is important to introduce some additional notations.

Let B 2 ( x, ε ) denote the ball of radius ε around x , i.e., B 2 ( x, ε ) ≜ { x ′ : ∥ x -x ′ ∥ 2 ≤ ε } . For a convex set S ∈ R n , we use B 2 ( S, ε ) to denote the union of all balls of radius ε around the set, i.e., B 2 ( S, ε ) ≜ ⋃ x ∈ S B 2 ( x, ε ) . For a convex set S ∈ R n , we use B 2 ( S, -ε ) to denote the set of all points in S that are 'safely' inside S (i.e., all the points in a ball of radius ε around them still belong in S ): B 2 ( S, -ε ) ≜ { x ∈ S : B 2 ( x, ε ) ⊆ S } . We call this last set, the ε -conservative of S . See Figure 2 for a pictorial illustration of the notations.

Algorithm 1: Explore-Then-Commit

Input:

Target precision ε , time horizon T .

EXPLORE: Find

ε -optimal strategy ˜ h ∈ H P using Algorithm 2.

COMMIT: Repeatedly play ˜ h for the rest of the rounds.

Given the above notations and for tuned ε 2 , the pair ( ˜ h , ˜ y ) returned by Algorithm 2 3 satisfies property (P1) and the additional property (P2) :

(P2) ˜ h ∈ B 2 ( P ˜ y , -ε 2 ) , i.e., ˜ h lies robustly within the best-response polytope for ˜ y .

̸

Given (P2) , the regret of the COMMIT phase can be decomposed to when y t = ˜ y , and when y t = ˜ y :

̸

When y t = ˜ y , (P1) guarantees that V ⋆ -U P ( ˜ h , ˜ y ) ≤ ε 1 , so the first term is at most ε 1 · | T 2 | .

̸

When y t = ˜ y , let A = A A \ { ˜ y } . For i ∈ A , the definition of binning function w i guarantees that the probability of playing action i on forecast p t is exactly w i ( p t ) . Based on this observation, the second term of Equation (3) can be further bounded as

Using Definition 2.2 of the calibration error and properties of the ℓ 2 and ℓ ∞ norms, we can further express n T 2 ( i ) as follows

where for the second inequality is: since ˜ h lies in the ε 2 -conservative of P ˜ y , and ¯ p T 2 ( i ) belongs to a different and non-intersecting polytope P i , we know that ∥ ¯ p T 2 ( i ) -˜ h ∥ 2 ≥ ε 2 . See Figure 3 for a geometric interpretation. Finally, bounding the sample complexity for the EXPLORE phase (Lemma C.4) and taking asymptotics gives the result.

The proof sketch above hinges on being able to identify a strategy for the principal ˜ h with properties (P1) , (P2) . We outline below how this is done through Algorithm 2.

At a high level, Algorithm 2 proceeds as follows. It builds an initialization set I which consists of pairs ( h , y ) where h approximately belongs to the ε 2 -conservative of y 's best response polytope: B 2 ( P y , -ε 2 ) . Using these points in the initialization set I , the algorithm then solves the convex optimization problem of max h U P ( h , y i ) using membership queries to B 2 ( P y , -ε 2 ) and the points of I as initial points. The final solution is obtained as the maximum of all these convex programs.

There are 2 things left to be specified for Algorithm 2; how to build the membership oracle to B 2 ( P y , -ε 2 ) given that U A ( · ) is unknown to the principal, and how to build the initialization set I to contain at least one point that is well

˜

h

P

˜

y

B

2

(

P

˜

y

, -

ε

2

)

h

⋆

Pi

Pj

¯

p

T

2

(

i

)

¯

p

T

2

(

j

)

ε

2

ε

2

̸

centered in P y ⋆ . To build set I , Algorithm 2 samples strategies h i uniformly at random for enough

number of rounds to ensure at least one sample in the η / 2 -ball around the center of P y ⋆ with high probability. To build the approximate membership oracle for the ε 2 -conservative best-response polytope for an action y , we use APPROXMEM (Algorithm 4 in Appendix C.2). Specifically, on input h ∈ H P , APPROXMEM either asserts h ∈ B 2 ( P y , -ε 2 + ε 1 ) or h / ∈ B 2 ( P y , -ε 2 -ε 1 ) with probability at least 1 -ε 3 . To do this, it samples Φ points in proximity to h and plays each one repeatedly for l rounds, while registering the best-response action observed for each one of these. If the most frequent best-response for all h ϕ is y , then we can conclude with good probability that h was inside B 2 ( P y , -ε 2 ) . See Lemma C.2 for more details.

4 Forecasting Algorithm for Adaptive Calibration

In this section, we examine whether there exist natural forecasting procedures that satisfy our Definition 2.3 about adaptively calibrated forecasts. We answer this question positively.

Theorem 4.1. For all ε > 0 and all binnings Π = { w i : R m → [0 , 1] , i ∈ [ k ] } , there exists a parameter-free forecasting procedure that is ( ε, Π) -adaptively calibrated with rate r δ ( t ) = O ( √ log( kmt ) /t ) . Moreover, when Π is a continuous binning (i.e., each w i is continuous), there exists a forecasting procedure that is (0 , Π) -adaptively calibrated with the same rate.

To prove the theorem, we use two main tools; the first one is a well-known algorithm of Luo and Schapire [44] (ADANORMALHEDGE) applied for online learning in the sleeping experts problem (see Appendix D for details). Roughly speaking, the sleeping experts is a standard online learning problem with T rounds and N experts, where at each round t there is only a subset of the experts being 'awake' to be considered by the learner and report their predictions. Let I t,i be the binary variable indicating whether expert i was awake at round t ( I t,i = 1 ) or asleep ( I t,i = 0 ). The interaction protocol between the learner and the adversary at each round t is: (i) The learner observes which experts are awake, i.e., { I t,i } i ∈ [ N ] . (ii) The learner selects a probability distribution π t ∈ ∆([ N ]) supported on the set of active experts A t ≜ { i : I t,i = 1 } . (iii) The adversary selects a loss vector { ℓ t,i } i ∈ [ N ] . (iv) The learner incurs expected loss ˆ ℓ t = E i ∼ π t [ ℓ t,i ] . ADANORMALHEDGE is a parameter-free online learning algorithm that when applied on the sleeping experts problem (and with appropriate initialization) obtains regret Reg t ( i ) = O ( √ T i log( NT i )) , where T i = ∑ τ ∈ [ t ] I τ,i .

The second tool that we use is No-Regret vs. Best-Response dynamics (NRBR) [34]. NRBR are a form of no-regret dynamics between two players, where one of the players must also best-respond on average. Essentially, at each round t ∈ [ T ] , the forecasting algorithm with the calibration rate of Theorem 4.1 outputs a randomized forecast p t ∈ F P , by simulating an interaction between two players described below. For the first player, we construct a sleeping experts problem instance, where the set of experts is G = { g ( s,i,j,σ ) : s ∈ [ T ] , i ∈ A A , j ∈ A P , σ ∈ {± 1 }} . For each g ( s,i,j, ( σ ) ∈ G

and t ∈ [ T ] , we define the loss, sleeping/awake indicator, and instantaneous regret respectively as:

where by h t,j , p t,j we denote the j -th coordinate of h t and p t respectively. We defined the losses for our newly constructed sleeping experts' instance as above to make sure that there is a direct correspondence with the calibration error. Similar ideas for calibration (albeit not for the notion of adaptivity we consider) have been used in [41, 34]. We describe next the player interaction in NRBR.

Player 1. Runs ADANORMALHEDGE on expert set G with a pre-specified prior π 0 over G and feedback specified in Equations (6), (7). At each round t , Player 1 computes distribution π t ∈ ∆( A t ( G )) , where A t ( G ) denotes the set of active experts g ( s,i,j,σ ) ∈ G with I t,g ( s,i,j,σ ) = 1 .

Player 2. Best responds to π t by selecting Q t ∈ ∆( F P ) that satisfies:

After simulating the game above, the algorithm outputs forecast p t ∼ Q t . The existence of such a distribution Q t is justified by the min-max theorem ([34, Fact 4.1] or [28, Theorem 5]). In the Appendix, we also give an explicit formula for Q t in the special case of m = 2 . When Π 0 is continuous, player 2 can select a deterministic p t that achieves Equation (8) with ε = 0 . This stronger property is justified by the outgoing fixed-point theorem [28, Theorem 4]. Note that this algorithm inherits its parameter-free property directly from ADANORMALHEDGE. We are now ready to provide a proof sketch for Theorem 4.1.

Proof sketch of Theorem 4.1. Fix an instance of the NRBR game outlined above. We begin by Definition 2.2 of calibration error translated in the sleeping experts instance that we defined above:

We add and subtract in the above ∑ τ ˆ ℓ τ to make the regret of the ADANORMALHEDGE on the sleeping experts instance appear and so the aforementioned becomes:

where for the regret, we have substituted the regret obtained by ADANORMALHEDGE. Note that ˆ ℓ τ = E g ∼ π τ [ ℓ τ,g ] . This, together with Equation (6) helps us translate the sleeping experts' loss to a loss that depends on h τ , p τ . Adding and subtracting the term ∑ τ ∈ [ s,t ] E g ∼ πτ p ∼ Qτ [ L g ( h τ , p )] :

The first term above is upper bounded by ε · ( t -s ) (Equation (8)) (or 0 in the continuous setting). The second can be bound with martingale concentration inequalities.

5 Continuous Games

In this section, we generalize our results for the case of continuous Stackelberg games. The supplementary material can be found in Appendix E.

Continuous Stackelberg Games. We use again A P and A A to denote the principal and the agent action spaces, respectively. Both A A , A P are convex, compact sets where A P ⊂ R m and A A ⊂ R k . The utilities of the principal and the agent are given by continuous functions U P : A P ×A A → R + and U A : A P ×A A → R + . In this setting, we assume that both the principal and the agent can only play deterministic strategies, i.e., H P = A P . For x ∈ A P , let BR ( x ) be the best-response function that is implicitly defined as ∇ 2 U A ( x, BR ( x )) = 0 . Our continuous games satisfy Assumption 5.1: (i)-(iii) are standard assumptions used in previous works (e.g., [24]), but (iv) cannot be derived from (i) and (ii) without further assumptions on the correlation between x, y . Nevertheless, (iv) (and the conditions under which it holds) has been justified in settings such as strategic classification [21, 51].

Assumption 5.1. Utility functions U P , U A , and the domain A P satisfy the following:

(i) For all x ∈ A P , y ∈ A A , U P ( x, y ) is L 1 -Lipschitz and concave in x , L 2 -Lipschitz in y , and bounded by W P in ℓ 2 norm.

(ii) The best-response function BR : A P →A A is L BR -Lipschitz.

(iii) Regularity of the feasible set A P = H P = F P :

The diameter is bounded: diam ( F P ) = sup h , h ′ ∈F P ∥ h -h ′ ∥ 2 ≤ D P .

B (0 , r ) ⊆ A P ⊆ B (0 , R ) .

(iv) The function U P ( h , BR ( h )) is concave with respect to h , and has Lipschitz constant L U .

The main result of this section is to show that even in continuous CSGs, we can approximate asymptotically V ⋆ for the principal's utility, and that no better utility is actually achieved.

Theorem 5.2. For continuous CSGs satisfying Assumption 5.1, for all ε 0 > 0 , there exists a finite binning Π 0 such that if the agent is (0 , Π 0 ) 4 -adaptively calibrated and the principal runs an appropriately parametrized instance of LAZYGDWOG (Algorithm 3) then: lim Φ →∞ M →∞ 1 Φ M ∑ ϕ ∈ [Φ] ∑ i ∈ [ M ] U P ( h ϕ , y ϕ,i ) ≥ V ⋆ -ε 0 . Moreover, for any sequence of the principal's actions h [1: T ] , it holds that: lim T →∞ 1 T ∑ t ∈ [ T ] U P ( h t , y t ) ≤ V ⋆ + ε 0 .

We outline next how LAZYGDWOG works. LAZYGDWOG is a variant of the gradient descent without a gradient algorithm (GDWOG) of Flaxman et al. [26]. The main new component of the algorithm is that it separates the time horizon into epochs and for each epoch it runs an update of the GDWOG algorithm. During all the rounds that comprise an epoch ( M in total), LAZYGDWOG presents the same (appropriately smoothed-out) strategy to the agent and observes the M different responses from the agent. The intuition behind repeating the same strategy for M rounds is that the principal wants to give the opportunity to the agent to recalibrate for a better forecast, i.e., lim M →∞ 1 M |{ i ∈ [ M ] : ∥ p i -h ∥ ≥ ε 0 }| = 0 . The remainder of the proof for Theorem 5.2 focuses on showing that when the calibrated forecasts converge to h t , then the principal's utility converges to the utility they would have gotten if the agent was perfectly best responding to h t .

6 Discussion and future directions

In this paper we introduced and studied learning in CSGs, where the agents best respond to the principal's actions based on calibrated forecasts that they have about them. Our work opens up several exciting avenues for future research. First, although our main results prove asymptotic convergence, it is an open question whether our exact convergence rates can be improved both for general CSGs and for specific cases of Stackelberg games (e.g., strategic classification in more general models compared to [21], pricing [1]). Second, it is an interesting question whether our definition of adaptive calibration (Def. 2.3) can actually hold for the sum over all binning functions, instead of just the maximum. Finally, to provide our asymptotic convergence results we assumed that the principal has access to the agent's calibration rate r δ ( · ) ; some information regarding how p t 's relate to h t 's is necessary to leverage the fact that agents are calibrated. But we think that in some specific settings (e.g., strategic classification) there may actually exist extra information regarding the forecasts (compared to just knowing r δ ( · ) ) that can be leveraged to design learning algorithms for the principal with faster convergence rates. We discuss these directions in more detail in Appendix F.

References

The value of knowing a demand curve: Bounds on regret for online posted-price auctions. In 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings. , pages 594-605. IEEE, 2003.

Jacob D Abernethy, Rachel Cummings, Bhuvesh Kumar, Sam Taggart, and Jamie H Morgenstern. Learning auctions with robust incentive guarantees. Advances in Neural Information Processing Systems , 32, 2019.

Saba Ahmadi, Hedyeh Beyhaghi, Avrim Blum, and Keziah Naggita. The strategic perceptron. In Proceedings of the 22nd ACM Conference on Economics and Computation (EC) , pages 6-25, 2021.

Saba Ahmadi, Avrim Blum, and Kunhe Yang. Fundamental bounds on online strategic classification. In Proceedings of the 24th ACM Conference on Economics and Computation (EC) , pages 22-58, 2023.

Kareem Amin, Afshin Rostamizadeh, and Umar Syed. Learning prices for repeated auctions with strategic buyers. In Advances in Neural Information Processing Systems , volume 26, 2013.

Bo An, David Kempe, Christopher Kiekintveld, Eric Shieh, Satinder Singh, Milind Tambe, and Yevgeniy Vorobeychik. Security games with limited surveillance. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , volume 26, pages 1241-1248, 2012.

Maria-Florina Balcan, Avrim Blum, Nika Haghtalab, and Ariel D Procaccia. Commitment without regrets: Online learning in Stackelberg security games. In Proceedings of the 16th ACM Conference on Economics and Computation (EC) , pages 61-78, 2015.

Avrim Blum. Empirical support for winnow and weighted-majority algorithms: Results on a calendar scheduling domain. Machine Learning , 26:5-23, 1997.

Avrim Blum and Thodoris Lykouris. Advancing subgroup fairness via sleeping experts. In Innovations in Theoretical Computer Science Conference (ITCS) , volume 151, pages 55:1-55:24, 2020.

Avrim Blum and Yishay Mansour. From external to internal regret. Journal of Machine Learning Research , 8(6), 2007.

Avrim Blum, Nika Haghtalab, and Ariel Procaccia. Lazy defenders are almost optimal against diligent attackers. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , volume 28, 2014.

Avrim Blum, Nika Haghtalab, and Ariel Procaccia. Learning optimal commitment to overcome insecurity. In Advances in Neural Information Processing Systems , volume 27, pages 1826-1834, 2014.

Mark Braverman, Jieming Mao, Jon Schneider, and Matt Weinberg. Selling to a no-regret buyer. In Proceedings of the 19th ACM Conference on Economics and Computation (EC) , pages 523-538, 2018.

Modibo K Camara, Jason D Hartline, and Aleck Johnsen. Mechanisms for a no-regret agent: Beyond the common prior. In Proceedings of the 61st Annual Symposium on Foundations of Computer Science (FOCS) , pages 259-270. IEEE, 2020.

Yiling Chen, Yang Liu, and Chara Podimata. Learning strategy-aware linear classifiers. In Advances in Neural Information Processing Systems , volume 33, pages 15265-15276, 2020.

Natalie Collina, Eshwar Ram Arunachaleswaran, and Michael Kearns. Efficient stackelberg strategies for finitely repeated games. In Proceedings of the 2023 International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS) , AAMAS '23, page 643-651, 2023.

Vincent Conitzer and Tuomas Sandholm. Computing the optimal strategy to commit to. In Proceedings of the 7th ACM Conference on Electronic Commerce (EC) , pages 82-90, 2006.

Amit Daniely, Alon Gonen, and Shai Shalev-Shwartz. Strongly adaptive online learning. In International Conference on Machine Learning (ICML) , pages 1405-1411. PMLR, 2015.

A Philip Dawid. The well-calibrated bayesian. Journal of the American Statistical Association , 77(379):605-610, 1982.

Yuan Deng, Jon Schneider, and Balasubramanian Sivan. Strategizing against no-regret learners. In Advances in Neural Information Processing Systems , volume 32, pages 1579-1587, 2019.

Jinshuo Dong, Aaron Roth, Zachary Schutzman, Bo Waggoner, and Zhiwei Steven Wu. Strategic classification from revealed preferences. In Proceedings of the 19th ACM Conference on Economics and Computation (EC) , pages 55-70, 2018.

Fei Fang, Thanh Nguyen, Benjamin Ford, Nicole Sintov, and Milind Tambe. Introduction to green security games. In International Joint Conference on Artificial Intelligence (IJCAI) , 2015.

Uriel Feige and Gideon Schechtman. On the optimality of the random hyperplane rounding technique for max cut. Random Structures & Algorithms , 20(3):403-440, 2002.

Tanner Fiez, Benjamin Chasnov, and Lillian J Ratliff. Convergence of learning dynamics in stackelberg games. arXiv preprint arXiv:1906.01217 , 2019.

Tanner Fiez, Benjamin Chasnov, and Lillian Ratliff. Implicit learning dynamics in Stackelberg games: Equilibria characterization, convergence analysis, and empirical study. In International Conference on Machine Learning (ICML) , pages 3133-3144. PMLR, 2020.

Abraham D. Flaxman, Adam Tauman Kalai, and H. B. McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. In Proceedings of the 15th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , 2004.

Dean P Foster and Sergiu Hart. Smooth calibration, leaky forecasts, finite recall, and Nash dynamics. Games and Economic Behavior , 109:271-293, 2018.

Dean P Foster and Sergiu Hart. Forecast hedging and calibration. Journal of Political Economy , 129(12):3447-3490, 2021.

Dean P Foster and Rakesh V Vohra. Calibrated learning and correlated equilibrium. Games and Economic Behavior , 21(1-2):40, 1997.

Dean P Foster and Rakesh V Vohra. Asymptotic calibration. Biometrika , 85(2):379-390, 1998.

Yoav Freund, Robert E Schapire, Yoram Singer, and Manfred K Warmuth. Using and combining predictors that specialize. In Proceedings of the 39th Annual ACM Symposium on Theory of Computing (STOC) , pages 334-343, 1997.

Jiarui Gan, Minbiao Han, Jibang Wu, and Haifeng Xu. Robust stackelberg equilibria. In Proceedings of the 24th ACM Conference on Economics and Computation (EC) , page 735, 2023.

Nika Haghtalab, Thodoris Lykouris, Sloan Nietert, and Alexander Wei. Learning in Stackelberg games with non-myopic agents. In Proceedings of the 23rd ACM Conference on Economics and Computation (EC) , pages 917-918, 2022.

Nika Haghtalab, Michael I Jordan, and Eric Zhao. A unifying perspective on multi-calibration: Unleashing game dynamics for multi-objective learning. arXiv preprint arXiv:2302.10863 , 2023.

Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, and Mary Wootters. Strategic classification. In Innovations in Theoretical Computer Science Conference (ITCS) , pages 111-122, 2016.

Sergiu Hart. Calibrated forecasts: The minimax proof. arXiv preprint arXiv:2209.05863 , 2022.

Kwang-Sung Jun, Francesco Orabona, Stephen Wright, and Rebecca Willett. Improved strongly adaptive online learning using coin betting. In Artificial Intelligence and Statistics (AISTATS) , pages 943-951. PMLR, 2017.

Sham M Kakade and Dean P Foster. Deterministic calibration and Nash equilibrium. Journal of Computer and System Sciences , 74(1):115-130, 2008.

Christopher Kiekintveld, Towhidul Islam, and Vladik Kreinovich. Security games with interval uncertainty. In Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems , pages 231-238, 2013.

Christian Kroer, Gabriele Farina, and Tuomas Sandholm. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , volume 32, 2018.

Daniel Lee, Georgy Noarov, Mallesh Pai, and Aaron Roth. Online minimax multiobjective optimization: Multicalibeating and other applications. In Advances in Neural Information Processing Systems , volume 35, pages 29051-29063, 2022.

Yin Tat Lee, Aaron Sidford, and Santosh S Vempala. Efficient convex optimization with membership oracles. In Conference on Learning Theory (COLT) , pages 1292-1294. PMLR, 2018.

Joshua Letchford, Vincent Conitzer, and Kamesh Munagala. Learning and approximating the optimal strategy to commit to. In Algorithmic Game Theory , pages 250-262. Springer, 2009.

Haipeng Luo and Robert E Schapire. Achieving all with no parameters: Adanormalhedge. In Conference on Learning Theory (COLT) , pages 1286-1304. PMLR, 2015.

Yishay Mansour, Mehryar Mohri, Jon Schneider, and Balasubramanian Sivan. Strategizing against learners in Bayesian games. In Conference on Learning Theory (COLT) , pages 52215252. PMLR, 2022.

Vidya Muthukumar and Anant Sahai. Robust commitments and partial reputation. In Proceedings of the 20th ACM Conference on Economics and Computation (EC) , pages 637-638, 2019.

Binghui Peng, Weiran Shen, Pingzhong Tang, and Song Zuo. Learning optimal strategies to commit to. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , volume 33, pages 2149-2156, 2019.

James Pita, Manish Jain, Milind Tambe, Fernando Ordónez, and Sarit Kraus. Robust solutions to Stackelberg games: Addressing bounded rationality and limited observations in human cognition. Artificial Intelligence , 174(15):1142-1171, 2010.

Milind Tambe. Security and game theory: algorithms, deployed systems, lessons learned . Cambridge university press, 2011.

Haifeng Xu, Long Tran-Thanh, and Nicholas R Jennings. Playing repeated security games with no prior knowledge. In Proceedings of the 2016 International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS) , pages 104-112, 2016.

Tijana Zrnic, Eric Mazumdar, Shankar Sastry, and Michael Jordan. Who leads and who follows in strategic classification? In Advances in Neural Information Processing Systems , volume 34, pages 15257-15269, 2021.

Song Zuo and Pingzhong Tang. Optimal machine strategies to commit to in two-person repeated games. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , volume 29, 2015.

A Calibrated Forecasts Standard Definition

We give below the standard definition for asymptotic calibration of Foster and Vohra [30] for a sequence of binary outcomes, i.e., h t ∈ A = { 0 , 1 } , ∀ t ∈ [ T ] . The forecasts p t take values in C = [0 , 1] . Let X denote the adaptive adversary generating the events' sequence (which is of infinite size), where the T first events are h 1 , . . . , h T .

Definition A.1. A forecasting procedure σ is asymptotically calibrated if and only if for any adaptive adversary X that generates the sequence h 1 , · · · , h T ∈ A and the forecasting algorithm σ that generates (possibly random) forecasts p 1 , · · · , p T ∈ C on the same sequence, we have that the calibration score C T ( X,σ ) goes to 0 as T →∞ :

where n T ( p ; h , σ ) ≜ ∑ t ∈ [ T ] 1 { p t = p } is the number of times that σ predicts p and ρ T ( p ; h , σ ) ≜ ∑ t ∈ [ T ] h t 1 { p t = p } n T ( p ; h ,σ ) be the fraction (empirical probability) of these times that the actual event was 1 .

Note that in Eq. (9), while F P contains an infinite number of distinct p 's (hence an infinite number of summands), for every finite T , there is only a finite number of p where n T ( p ; h , σ ) is nonzero. Therefore, C T is well-defined and finite.

Equivalently, the above definition states that for the infinite binning [28] Π = { w x ( p ) : x ∈ C } where w x ( p ) = 1 { p = x } , the calibration score can be equivalently expressed as

where n T ( x ) ≜ ∑ T t =1 w x ( p t ) is the number of times that forecast p t falls into bin x , ¯ p T ( x ) ≜ ∑ T t =1 w x ( p t ) n T ( x ) · p t is the average forecast that activates bin x , which is equal to ∑ T t =1 w x ( p t ) n T ( x ) · x = x because w x ( p t ) is nonzero if and only if p t = x , and ¯ h T ( x ) ≜ ∑ T t =1 w x ( p t ) n T ( x ) · h t is the average outcome corresponding to bin x . It follows that the score C T is a sum of the calibration errors during interval [1 : T ] for all bins (with CalErr defined in Definition 2.3).

B Calibrated Forecasts Lead to No Swap Regret

In this section, we show the connection between no-swap-regret agents and adaptively calibrated ones. As a reminder, no-swap-regret agents (translated to our setting and notation for the ease of exposition) are defined as follows.

Definition B.1 (Agent's swap regret [10]) . For a sequence of principal's strategies h 1 , · · · , h T ∈ H P and agent's actions y 1 , · · · , y T ∈ A A , the swap regret is defined as

We say that an agent is a no-swap-regret agent, if for the sequence of actions { y t } t ∈ [ T ] that they are playing it holds that SwapReg( h 1: T , y 1: T ) = o ( T ) .

We next show that calibrated forecasts lead to no swap regret actions for the agent.

Lemma B.2 (Calibrated forecasts lead to no swap regret) . If the agent is ( ε, Π) -adaptively calibrated, then the agent's swap regret on the sequence h 1: T is bounded by the calibration error as follows:

If the agent breaks ties deterministically, then with probability ≥ 1 -δ,

If the agent breaks ties randomly, then with probability ≥ 1 -2 δ ,

where U max = max h ∈A P max y ∈A A U A ( h , y ) is the maximum utility the agent can obtain (without constraining the agent to play best responses).

Proof. We first present the proof for the case that the agents break ties deterministically. To simplify notation, we use n T ( i ) := n [0: T ] ( i ) , ¯ p T := ¯ p [0: T ] ( i ) , and ¯ h T ( i ) := ¯ h [0: T ] ( i ) .

Fix a π : A A →A A . Then, with probability at least 1 -δ , we have that:

̸

In the above equations, step (a) is due to the fact that agents best respond with a deterministic tiebreaking rule: y t = i if and only if i ∈ BR ( p t ) and i ≻ j, ∀ j = i , which is equivalent to w i ( p t ) = 1 . We have also used U A ( · , i ) to denote the m -dimensional vector where the j th entry is the utility U A ( j, i ) . Step (b) is because the second term

is non-positive since each p t with w i ( p t ) = 1 belongs to the best response polytope P i , so does their average: ¯ p t ( i ) ∈ P i ⇐⇒ i ∈ BR (¯ p t ( i )) .

Since the above inequality holds for any π , it also holds after taking the maximum over all π : A A → A A . Therefore, we have the same bound for the agent's swap regret.

Next, we move to the case when the agent breaks ties randomly. For a fixed π , we have that at every time step t ,

Therefore, by Azuma-Hoeffding's inequality, w.p. ≥ 1 -δ ′ , we have

Since all actions in BR ( p t ) have the same utility for the agents, we also have

Therefore, using Equations (11) and (12), we have that with probability at least 1 -δ ′ ,

We can use the same arguments as above (from Equation (10) onwards) to bound the first term on the right hand side by 2 U max mkT ( r δ ( T ) + ε ) with probability 1 -δ . Finally, setting δ ′ = δ/M where M = k k is the number of possible swap functions, and applying the union bound, we conclude that with probability ≥ 1 -2 δ , the swap regret is bounded by

C Supplementary Material for Section 3

C.1 Proof of Theorem 3.1

We first formally restate the theorem with exact convergence rates (not just asymptotically).

Theorem C.1 (Formal version of Theorem 3.1) . Assume that the agent is ( ε, Π) -adaptively calibrated with rate r δ ( · ) and let U max = max h ∈H P max y ∈A A U P ( h , y ) . Then, for any sequence { h t } t ∈ [ T ] for the principal's strategies in a repeated CSG, with probability at least 1 -2 δ , the principal's utility is upper bounded as:

where α ( U max , m, k, T, r δ , δ, ε ) = U max · m · k · T · ( r δ ( T ) + ε ) when the agent uses deterministic tie-breaking and α ( U max , m, k, T, r δ , δ, ε ) = U max · m · k · T · ( r δ ( T ) + ε + √ T log(1 /δ )) when the agent uses randomized tie-breaking.

Proof. To simplify notation, we use n T ( i ) := n [0 ,T ] ( i ) . When the agent follows deterministic tie-breaking, we have:

where (a) is because U P (¯ p T ( i ) , i ) ≤ V ⋆ (since i ∈ BR (¯ p T ) ), the Hö(lder's inequality, and the fact that ∥ · ∥ 2 ≤ ∥ · ∥ 1 , and (b) is because of the definition of U max and Definition 2.3. The proof for the randomized tie-breaking setting has an extra term from Azuma-Hoeffding's inequality, similar to the proof of Theorem B.2.

C.2 Approximate membership oracle to conservative polytopes

In this section, we formally present the algorithm (Algorithm 4 for constructing an approximate membership oracle to the conservative best response polytope for each of the agent's action. The sample complexity of the oracle will be presented in Lemma C.2.

Algorithm 4: Approximate membership oracle for the conservative best response polytope (APPROXMEM)

Input: query h ∈ H P , original polytope P y ( y ∈ A A ) , approximation factor ε 1 , conservatism factor ε 2 , failure probability ε 3

̸

In Lemma C.2, we show that the parameters Φ , ε cal , R can be tuned to achieve a wide range of parameters ( ε 1 , ε 2 , ε 3 ) . In Proposition C.3, we provide explicit instantiations of the parameters for three special cases of ( ε 1 , ε 2 , ε 3 ) that will be useful in later sections.

Lemma C.2. If the agent is ( ε, Π) -adaptively calibrated with rate r δ ( · ) and infinitesimal ε , and the parameters Φ , R, ε cal satisfy:

Then Algorithm 4 ( APPROXMEM ) returns an ε 1 -approximate membership oracle to P -ε 2 y = B 2 ( P y , -ε 2 ) with probability 1 -ε 3 , using no more than N ε 1 ,ε 2 ,ε 3 = O ( Φ r -1 δ ( ε cal k √ m ) ) rounds of interactions with the agent.

Specifically, with probability 1 -ε 3 , APPROXMEM either returns TRUE which asserts that h ∈ B 2 ( P -ε 2 y , + ε 1 ) , or returns FALSE which asserts that h / ∈ B 2 ( P -ε 2 y , -ε 1 ) .

Proposition C.3 (Parameter settings) . Assume ε 3 = δ + T -2 . In the following three cases: ε 1 ε 2 = Θ( √ m ) , 1 , o (1) , the proposed setting of parameters ε cal , R, Φ satisfy (Condition 1) and (Condition 2) simultaneously.

Case II. ε 1 ε 2 = 1 .

Case III. ε 1 ε 2 = o (1) .

Proof of Lemma C.2. Before we delve into the specifics of the proof, we introduce some notation. In order to make sure that h ϕ is such that ∥ h -h ϕ ∥ = R , we do the following: h ϕ ← h + R S ϕ , where S ϕ is sampled uniformly at random from the equator S ∩ H , where S = { s ∈ R m : ∥ s ∥ 2 2 = 1 } is the unit sphere and H = { s ∈ R m : ⟨ s , 1 ⟩ = 0 } is an equatorial hyperplane ( 1 ≜ (1 , · · · , 1) ∈ R m ). Note that this is because we want h ϕ to remain a valid probability distribution, i.e., that ⟨ h ϕ , 1 ⟩ = 1 and h ϕ ≥ 0 coordinate-wise; indeed, since we already have ⟨ h , 1 ⟩ = 1 , we need to make sure that (1) ⟨ S ϕ , 1 ⟩ = 0 , which is guaranteed by S ϕ ∈ H ; (2) h ϕ ≥ 0 , which is guaranteed by returning FALSE whenever h ϕ / ∈ H P .

For the rest of the proof, we condition on the following success event:

Since the agent is ( ε, Π) -adaptively calibrated with rate r δ , we have Pr( E ) ≥ 1 -δ .

Recall that ε cal = k √ mr δ ( l ) denote the error that comes from calibration error. We first show that conditioned on E , ∀ ϕ ∈ [Φ] , h ϕ ∈ B 2 ( P y ϕ , ε cal ) . Let l y ϕ be the number of times that agent plays y ϕ during the l repeats, then we have l y ϕ ≥ l/k because y ϕ is the most frequently played action. Then, the calibration error bound in Definition 2.3 guarantees that

where the first inequality in equation (13) is because of the norm property ∥ x ∥ 2 ≤ d ∥ x ∥ ∞ for a vector x ∈ R d . Since ¯ p ( y ϕ ) ∈ P y ϕ because the agent always best responds to forecasts, we obtain h ϕ ∈ B 2 ( P y ϕ , ε cal ) .

We then prove the following two claims:

(C1) If h ∈ B 2 ( P -ε 2 y , -ε 1 ) , then APPROXMEM returns TRUE.

(C2) If h / ∈ B 2 ( P -ε 2 y , + ε 1 ) , then APPROXMEM returns FALSE with probability ≥ 1 -( ε 3 -δ ) .

Indeed, if the following two claims hold, then we have established that APPROXMEM asserts one of two cases correctly with probability ≥ 1 -( ε 3 -δ ) conditioned on E . Together with the fact that Pr( E ) ≥ 1 -δ , proves the lemma.

̸

Proof of (C1). Suppose h ∈ B 2 ( P -ε 2 y , -ε 1 ) . Then we know that B 2 ( h , ε 1 ) ⊆ P -ε 2 y = B 2 ( P y , -ε 2 ) , which further implies B 2 ( h , ε 1 + ε 2 ) ⊆ P y . Therefore, the distance between h and any other polytope P y ′ ( y ′ = y ) must be lower bounded by ε 1 + ε 2 . By triangle inequality, ∀ h ′ ∈ P y ′ where y ′ = y , we have

̸

where the last step follows from (Condition 1).

Since this holds for all h ′ ∈ P y ′ , it implies h ϕ ̸∈ B 2 ( P y ′ , ε cal ) whenever y ′ = y . Together with the fact that h ϕ ∈ B 2 ( P y ϕ , ε cal ) , we must have y ϕ = y for all epochs ϕ ∈ [Φ] . Therefore, APPROXMEM always returns TRUE.

̸

Proof of (C2). Suppose h / ∈ B 2 ( P -ε 2 y , + ε 1 ) . We first analyze the probability of returning FALSE for a fixed epoch ϕ ∈ [Φ] . Let ∂S be the boundary of a convex set S , then by triangle inequality, we have

For the first term, we have d ( h , ∂P y ) ≤ ε 2 since

For the second term, we have

√

because P y is a convex set. Combining the two bounds, we know that d ( h , ∂B 2 ( P y , ε cal )) ≤ ε cal + ε 2 .

Since h ϕ is uniformly sampled from the sphere of radius R around h , by convexity of B 2 ( P y , ε cal ) and the rotation invariance property of a unit sphere, we have

where v is a unit vector pointing in the direction of (one of) the projection from h to ∂B 2 ( P y , ε cal ) , and e 1 = (1 , 0 , · · · , 0) ∈ R m . According to [23, Lemma 9], we can further lower bound the probability by

Finally, the probability that no epoch returns FALSE (failure of APPROXMEM) is at most

Since δ ′ ≪ 1 , the above inequality holds as long as

which follows from (Condition 2). Therefore, h / ∈ B 2 ( P -ε 2 y , + ε 1 ) , then APPROXMEM returns FALSE with probability ≥ 1 -( ε 3 -δ ) .

C.3 Complexity of the explore algorithm

Lemma C.4 (Sample complexity of Algorithm 2) . Using Algorithm 2, the principal can find with probability at least 1 -( δ + ε opt + T -1 ) , a strategy ˜ h ∈ H P that satisfies the two properties in Lemma C.5. We refer the reader to Theorems C.8 and C.13 for the sample complexity under two different instantiations of Algorithm 2.

Lemma C.5 (Optimality and robustness of output policy) . Under the conditions of Lemma C.4, the output ˜ h of Algorithm 2 satisfies the following two properties:

(P1) U P ( ˜ h , y ⋆ ) ≥ V ⋆ -ε opt , i.e., ˜ h is an approximate Stackelberg equilibrium.

(P2) ˜ h ∈ B 2 ( P y ⋆ , -ε robust ) , i.e., ˜ h lies robustly within the best-response polytope for y ⋆ .

We refer the reader to Theorems C.8 and C.13 for the exact values of ε robust , ε opt under two different instantiations of Algorithm 2.

Before we formally state the proof of Lemmas C.4 and C.5, we first show the sample complexity guarantee of the exploration phase of Algorithm 2 (Lemma C.6), then state a useful lemma from prior work on convex optimization from membership queries [42] (Lemma C.7).

Lemma C.6 (Initialization set of Algorithm 2) . Let V be the volume of an ℓ 2 ball of radius η 2 in R m . Suppose the principal samples h uniformly from H P for O ( V -1 log T ) times and calls APPROXMEM ( ε 1 = ε 2 = η/ 4 , ε 3 = δ + T -2 / 2) for the membership of each of them, then with probability at least 1 -δ -T -1 , the initialization set I contains ( h 0 , y 0 ) where y 0 = y ⋆ is the optimal target, and h 0 is η 2 -centered in P y ⋆ , i.e., h 0 ∈ B 2 ( P y ⋆ , -η 2 ) .

The total number of samples required for the initialization phase is O ( V -1 1 . 25 m r -1 δ ( η k √ m ) log 2 T ) .

Proof of Lemma C.6. By regularity assumption, there exists ˙ h ∈ P y ⋆ , s.t. B 2 ( ˙ h , η ) ∈ P y ⋆ . Therefore, ∀ h ′ ∈ B 2 ( ˙ h , η 2 ) , we have ˙ h ∈ B 2 ( P y ⋆ , -η 2 ) . Moreover, since h ′ lies robustly inside P y ⋆ , on the success event of APPROXMEM ( ε 1 = ε 2 = η 4 ) , APPROXMEM ( h ′ ) must return membership y ⋆ , otherwise we must have h ′ / ∈ B 2 ( P -η 4 y ⋆ , -η 4 ) = B 2 ( P y ⋆ , -η 2 ) , a contradiction. Since the set of all such h ′ takes up nontrivial volume V in H P , we know that O ( V -1 log T ) uniform samples are guaranteed to hit one with probability 1 -1 (2 T ) .

Nowweupper bound the failure probability of the initialization phase. Since the agent is miscalibrated with probability δ , random sampling fails to discover centered h ′ with probability 1 (2 T ) , and the

probability that one of the answers from APPROXMEM is wrong with probability 1 (2 T ) , the total failure probability is 1 -δ -T -1 as desired.

As for the sample complexity, note that according to Lemma C.2 and proposition C.3, each call to APPROXMEM takes O ( 1 . 25 m r -1 δ ( η k √ m ) log T ) samples, and the initialization phase calls APPROXMEM for O ( V -1 log T ) times, the total sample complexity is O ( V -1 1 . 25 m r -1 δ ( η k √ m ) log 2 T ) .

Lemma C.7 (LSV performance guarantee, Theorems 14 and 15 in [42]) . Let K be a convex set specified by a membership oracle, a point x 0 ∈ R n , and η > 0 such that B 2 ( x 0 , η / 2 ) ⊆ K ⊆ B 2 ( x 0 , 1) . There exists a universal constant γ 0 > 1 such that for any convex function f given by an evaluation oracle and any ε ′ > 0 , there is a randomized algorithm that computes a point z ∈ B 2 ( K,ε ′ ) such that with probability ≥ 1 -ε ′ ,

with constant probability using O ( n 2 log O (1) ( n / ε ′ η ) ) calls to the ε 1 -approximate membership oracle and evaluation oracle, where ε 1 = ( ε ′ η / n ) γ 0 .

Note that in Lemma C.7, the required accuracy ( ε 1 ) of the approximate membership oracle is orders of magnitudes smaller than the suboptimality ( ε ) of the output solution, i.e., ε 1 ≪ ε . Let ˜ h denote the ε -optimal solution returned by LSV. Since we use APPROXMEM which is an ε 1 -approximate membership oracle to K = P -ε 2 y , ˜ h will lie in B 2 ( K,ε ) = B 2 ( P -ε 2 y , ε ) = B 2 ( P y , -ε 2 + ε ) . In order to find a near-optimal solution that lies robustly within P y , we can either simulate APPROXMEM with more epochs to guarantee ε 1 ≪ ε ≈ ε 2 (see Appendix C.4 for more details), or perform a post-process to ˜ h that pushes it further inside the polytope (see Appendix C.5 for more details).

C.4 More epochs to guarantee robustness

In this section, we show how to use more epochs in the simulation of each APPROXMEM oracle to obtain higher accuracy feedbacks. This approach directly guarantees that the output of LSV lies robustly within the polytope.

Theorem C.8. If the principal uses Algorithm 2 with parameters APPROXMEM ( ε 1 = ( ε ′ η n ) γ 0 , ε 2 = 2 ε ′ , ε 3 = δ + T -2 ) and LSV ( ε ′ ) , then with probability at least 1 -δ -T -1 -ε ′ , the final solution ˜ h satisfies the following two properties:

(P1) U P ( ˜ h , y ⋆ ) ≥ V ⋆ -3 ε ′ , i.e., ˜ h is an approximate Stackelberg equilibrium.

(P2) ˜ h ∈ B 2 ( P y ⋆ , -ε ′ ) , i.e., ˜ h lies robustly within the best-response polytope for y ⋆ .

Proof. We first analyze the failure probability of the algorithm. Since the agent's adaptive calibration error is uniformly bounded by r δ ( · ) with probability 1 -δ , the probability that there exists an incorrect answer from APPROXMEM is bounded by T · T -2 = T -1 conditioned on agent's calibration error being bounded, and the probability that LSV returns a bad solution is bounded by ε ′ , the total failure probability of the described algorithm is at most δ + T -1 + ε ′ .

Now we consider the optimization problem for the optimal polytope P y ⋆ , with initial point h 0 that is η 2 -centered in P y ⋆ . There exists such a pair ( h 0 , y ⋆ ) ∈ I according to Lemma C.6. Let ˜ h be the solution output by LSV. According to Lemma C.7, ˜ h ∈ B 2 ( P -ε 2 y ⋆ , + ε ′ ) ⊆ B 2 ( P y ⋆ , -ε ′ ) , which proves (P2) . For (P1) , note that Lemma C.7 also guarantees

Finally, we compute the total number of samples. By Lemma C.6, the initialization epoch takes O ( V -1 1 . 25 m r -1 δ ( η k √ m ) log 2 T ) samples. According to Lemma C.2 and proposi- tion C.3, each call to the APPROXMEM oracle requires O ( √ m ( ε 2 ε 1 ) m 2 r -1 δ ( ε 1 k √ m ) log( T ) ) = O ( √ m ( m ηε ′ ) mγ 0 2 r -1 δ ( ( ε ′ η ) γ 0 km γ 0 √ m ) log( T ) ) samples, and Lemma C.7 suggests that there LSV makes O ( m 2 log O (1) ( m ε ′ η ) ) oracle calls, where we run O ( V -1 log T ) LSV instances for each initial point in the initialization set. Putting together, the total number of samples is

where O hides logarithm terms in T, m, 1 /η, 1 /ε ′ .

C.5 Representation length and post-processing

In this section, we show a post-processing of the output of LSV [42] under the representation length assumption.

Assumption C.9 (Utility Representation Length) . Suppose the utility functions U P and U A are rational with denominator at most a , and normalized to be [0 , 1] . Therefore, the game's utility representation length is L = 2 mn log a .

The next lemma shows that under Assumption C.9, the optimal solution h ⋆ also has a finite representation length.

Lemma C.10. For any y ∈ A A , let h ⋆ y ∈ R m be the principal's optimal strategy in P y that achieves max h ⋆ ∈ P y U P ( h ⋆ , y ) . Suppose the utility functions U P , U A satisfy Assumption C.9. Then for all i ∈ A P , the i -th coordinate of h ⋆ y is a rational number with denominator at most 2 2 mL .

Proof. The proof of this lemma is similar to [12, Lemma 10], which follows from well-known results in linear programming. We first note that h ⋆ is the solution of the linear programming:

̸

where U P ( h , y ) is linear in h and the best response polytope P y can be represented as a system { h : A h T ⪰ b } , where the set of constraints are the ones that define probability simplex H P , together with the constraints of the form U P ( h , y ) -U P ( h , y ′ ) ≥ 0 , ∀ y ′ = y . Suppose A is normalized so that each entry is an integer. By Assumption C.9, each coefficient is at most 2 L . Since the solution to the above LP is the intersection of m independent constraints of A (denoted with D ), we have h ⋆ y,i = det( D i ) det( D ) by Cramer's rule, where D i is D with the i -th column replaced by b . According to Hardamard's inequality,

We also make the following assumption on the optimal strategy h ⋆ : any near-optimal strategy ˜ h must lie in the neighborhood of h ⋆ .

Assumption C.11 (Near-optimal strategies) . There exists a constant c s.t. for all ε ≤ 2 -4 mL and all strategy ˜ h such that U P ( ˜ h .y ⋆ ) ≤ V ⋆ -ε , we have ∥ ˜ h -h ⋆ ∥ ∞ ≤ c 2 L ε .

Now we are ready to define the post-processing algorithm.

Lemma C.12 (Post-processing) . Suppose that Assumptions C.9 and C.11 holds. Assume ε ≤ 2 -4 mL , ˜ h is an ε -approximate optimal strategy in P ˜ y , and h 0 is η / 2 -centered in P ˜ y , i.e., B 2 ( h 0 , η / 2 ) ⊆ P ˜ y , then using no more than O ( 2 2 m r -1 δ ( λ k √ m ) log T ) rounds of interactions with the agent, POSTPROCESS (Algorithm 5) outputs a strategy h that satisfies with probability ≥ 1 -δ -T -1 :

Algorithm 5: Post-Processing (POSTPROCESS)

Input: non-robust strategy ( ˜ h , ˜ y ) s.t. ˜ h ∈ B 2 ( P ˜ y , ε ) , robustness parameter λ , intial point h 0 Output: robust strategy h

S ← set of strategies h ′ ∈ H P s.t. each coordinate h ′ i is a rational number with denominator at most 2 2 mL and | ˜ h i -h i | ≤ c 2 L ε

for h j ∈ S do

Query APPROXMEM ˜ y ( h j query ) with ε 1 = ε 2 = λ 2 , ε 3 = δ + T -2

if APPROXMEM returns TRUE then

(i) h lies robustly inside P ˜ y : h ∈ B 2 ( P ˜ y , -λ ) ;

Proof. To prove the lemma, we make the following claim: For all h 1 ∈ H P , if h 1 ∈ P ˜ y and h 2 = ( 1 -2 λ η ) h 1 + 2 λ η h 0 , then we have h 2 ∈ B 2 ( P ˜ y , -λ ) . In fact, for all unit vectors s ∈ R m , we have

Since h 1 ∈ P ˜ y and h 0 + η 2 s ∈ B 2 ( h 0 , η / 2 ) ⊆ P ˜ y from the assumption that h 0 is η / 2 -centered, their convex combination must also lie in P ˜ y because P ˜ y is convex. Thus, we have h 2 + λ s ∈ P ˜ y . Since the above holds for all unit vectors s , we conclude that B 2 ( h 2 , λ ) ⊆ P ˜ y . We have thus proved the claim.

With the above claim, we first show that when all oracle calls return correctly, S contains at least one point h j such that APPROXMEM returns TRUE on h j query, and the post-processed version of this strategy satisfies (i) and (ii).

Since ˜ h is an ε -optimal strategy, by Assumption C.11, we know that the true optimal h ⋆ ˜ y satisfies ∥ h ⋆ ˜ y -˜ h ∥ ∞ ≤ c 2 L ε . According to Lemma C.10, every coordinate of h ⋆ ˜ y is a rational number with denominator at most 2 2 mL . Combining the two guarantees, we know that S must contains the true optimal h ⋆ ˜ y , which satisfies h ⋆ ˜ y ∈ P ˜ y . Suppose this point is h j . According to the above claim, we have h j query ∈ B 2 ( P ˜ y , -λ ) . Since we simulated the APPROXMEM oracle with ε 1 = ε 2 = λ 2 , it must return TRUE on strategies that belong to B 2 ( P -ε 2 ˜ y , -ε 1 ) = B 2 ( P y , -λ ) , therefore, it must return TRUE on h j query. This argument shows that POSTPROCESS always return valid strategies.

However, it could be the case that APPROXMEM returns TRUE on strategies h j that are not h ⋆ ˜ y . We argue that the guarantee of POSTPROCESS is unaffected by showing that if APPROXMEM returns TRUE on any h j query, then we must have h j query ∈ B 2 ( P -ε 2 ˜ y , + ε 1 ) ⊆ P ˜ y from the guarantee of APPROXMEM (see Lemma C.2). Applying the above claim again, we have that h ∈ B 2 ( P ˜ y , -λ ) for the returned strategy h , which proves (i).

As for (ii), note that

Finally, we analyze the failure probability and sample complexity of POSTPROCESS. Since the agent's adaptive calibration error is uniformly bounded by r δ ( · ) with probability 1 -δ , and the

probability that there exists an incorrect answer from APPROXMEM is bounded by T · T -2 = T -1 conditioned on agent's calibration error being bounded, the failure probability of POSTPROCESS is at most δ + T -1 .

For the sample complexity, since | S | ≤ 2 m and each h j ∈ S requires simulating APPROXMEM ˜ y with O ( 1 . 25 m r -1 δ ( λ k √ m ) log T ) samples (see Lemma C.2 and proposition C.3), the total number of samples required is O ( 2 2 m r -1 δ ( λ k √ m ) log T ) .

Theorem C.13. Suppose the utility functions satisfy Assumptions C.9 and C.11. If the principal uses Algorithm 2 with parameters ε 1 = √ mε 2 = ( η m 2 4 mL ) γ 0 , ε 3 = δ + T -2 for the APPROXMEM oracle and ε ′ = 2 -4 mL -1 for LSV , then feed the output of LSV into POSTPROCESS ( λ ) , then with probability at least 1 -δ -T -1 -ε ′ the final solution satisfies the following two properties:

(P1) U P ( ˜ h , y ⋆ ) ≥ V ⋆ -5 mλ η , i.e., ˜ h is an approximate Stackelberg equilibrium.

(P2) ˜ h ∈ B 2 ( P y ⋆ , -λ ) , i.e., ˜ h lies robustly within the best-response polytope for y ⋆ .

The total number of samples needed is expressed in Equation (14) .

Proof. The analysis for the failure probability is the same with Theorem C.8 .

Now we consider the optimization problem for the optimal polytope P y ⋆ , with initial point h 0 that is η 2 -centered in P y ⋆ . Let ˆ h be the solution output by LSV, and let ˜ h be the solution output by running POSTPROCESS on ˆ h . According to Lemma C.7, ˆ h satisfies ˆ h ∈ B 2 ( P -ε 2 y ⋆ , + ε ′ ) ⊆ B 2 ( P y ⋆ , ε ′ ) , and the suboptimality of ˆ h is at most O ( ε ′ + ε 2 ) ≤ 2 -4 mL . Since ε ′ ≤ 2 -4 mL , by Lemma C.12, ˜ h ∈ B 2 ( P y ⋆ , -λ ) , which proves (P2) .

To prove (P1) , note that since ∥ ˆ h -˜ h ∥ 2 ≤ √ m ( c 2 L ε ′ + 4 λ η ) by Lemma C.12, we have

Finally, we compute the total number of samples. According to Lemma C.2 and proposition C.3, each call to the APPROXMEM oracle requires O ( √ mr -1 δ ( ε 2 k √ m ) log T ) = O ( √ mr -1 δ ( 1 km ( η m 2 4 mL ) γ 0 ) log T ) samples, and Lemma C.7 suggests that LSV makes O ( m 2 log O (1) ( m ε ′ η ) ) = O ( m 3 L log O (1) ( m η ) ) oracle calls, where we run O ( V -1 log T ) LSV instances for each initial point in the initialization set. In addition, POSTPROCESS takes O ( 2 2 m r -1 δ ( λ k √ m ) log T ) samples (Lemma C.12), and initialization takes O ( V -1 1 . 25 m r -1 δ ( η k √ m ) log 2 T ) samples (Lemma C.6). Adding them all up, the total number of samples is

where ˜ O hildes logarithm factors in T, m, η -1 .

C.6 Proof of Theorem 3.2

In this section, we present the main theorem (Theorem 3.2) in Section 3. We first formally restate the theorem with exact convergence rate.

Theorem C.14 (Formal version of Theorem 3.2) . There exists an algorithm for the principal in CSGs that achieves average utility with probability 1 -o (1) :

Specifically, for calibrated agents with calibration error rate r δ ( t ) = O ( t -1 /β ) ,

(I) If the principal runs the EXPLORE-THEN-COMMIT algorithm (Algorithm 1) where the explore phase follows the parameter settings in Theorem C.8 with ε ′ = O ( T -1 γ 1 C 1 γ 1 1 η -γ 1 -1 γ 1 ) , then the limit in (15) approaches V ⋆ with rate

where

(II) If the principal runs the EXPLORE-THEN-COMMIT algorithm (Algorithm 1) where the explore phase follows the parameter settings in Theorem C.13 with λ = O ( ( ηC 2 m ) 1 γ 2 T -1 γ 2 ) , then the limit in (15) approaches V ⋆ with rate

Proof. According to Lemmas C.4 and C.5, we have that with probability 1 -o (1) , Algorithm 2 (with potential post-processing) returns a policy ˜ h within N samples, where ˜ h satisfies

(P1) U P ( ˜ h , y ⋆ ) ≥ V ⋆ -ε opt , i.e., ˜ h is an approximate Stackelberg equilibrium.

(P2) ˜ h ∈ B 2 ( P y ⋆ , -ε robust ) , i.e., ˜ h lies robustly within the best-response polytope for y ⋆ .

With properties (P1) and (P2) , the arguments in Section 3 give us

Now we plug in specific instantiations of Algorithm 1.

(I) Suppose the principal follows the parameter settings in Theorem C.8. In this case, we have ε opt = 3 ε ′ , ε robust = ε ′ , and

Finally, optimizing over ε ′ gives the claimed bound.

(II) Suppose the principal follows the parameter settings in Theorem C.13. In this case, we have ε opt = 5 mλ η , ε robust = λ , and

Finally, optimizing over λ gives the claimed bound.

where

Remark C.15 (Representation length) . If we assume the optimal solution h ⋆ lies in the grid of size poly ( ε -1 ) , then in case (II) of Theorem C.14, we have 2 2 mL = poly ( ε -1 ) , which leads to a rate of ˜ O ( poly ( ε, k, m, η -1 , V -1 ) · T -1 γ 2 ) , where γ 2 is a constant that only depends on the agent's calibration error rate β . This is the result of replacing Lemma C.10 with the finite grid assumption.

Remark C.16 (Adaptive regret versus calibration) . Our primary focus lies on calibration due to its characterization of agents' beliefs and the fact that it provides both upper and lower bounds to the principal's utility. This is particularly useful for the learning direction, as denoted by the lower bounds in Theorems 3.2 and C.14. However, a different form of adaptive guarantee would suffice here: one concerning (external) regret. Nevertheless, we do not focus on regret as a characterization as it doesn't offer the same upper bound guarantees - in fact, the principal could potentially extract more utility than V ⋆ . Additionally, regret-based assumptions tend to overly emphasize the agent's optimization techniques rather than maintaining a consistent belief about the action being executed.

D Supplementary Material for Section 4

D.1 Background on Sleeping Experts and ADANORMALHEDGE

We start the exposition of this part by introducing the sleeping experts problem [10, 31]. For each expert i ∈ [ N ] and round t ∈ [ T ] , let ℓ t,i ∈ [0 , 1] be the loss of expert i , and let I t,i be an indicator that takes value I t,i = 1 if expert i is active at round t and I t,i = 0 if asleep. The interaction protocol at each round t goes as follows: The indicators ( I t,i ) i ∈ [ N ] are revealed to the learner. The learner selects a probability distribution π t ∈ ∆([ N ]) that is supported only on the set of active experts A t ≜ { i : I t,i = 1 } . The adversary selects a loss vector ( ℓ t,i ) i ∈ [ N ] . The learner then suffers expected loss ˆ ℓ t = E i ∼ π t [ ℓ t,i ] . The regret with respect to each expert i only accounts for the rounds when i is awake, which, together with the fact that π t is only supported on active experts, implies that

One of the algorithms that can be used to provide sublinear regret for the sleeping experts problem is ADANORMALHEDGE [44]. ADANORMALHEDGE is a powerful, parameter-free algorithm which provides regret bounds in terms of the cumulative magnitude of the instantaneous regrets, defined as: r t,i = ˆ ℓ t -ℓ t,i for all experts i ∈ [ N ] . As its name suggests, ADANORMALHEDGE uses the wellknown algorithm HEDGE as a backbone; HEDGE maintains a probability distribution over experts at each round t and draws an expert from said distribution. After the expert's loss is revealed, the probability distribution for the next round t +1 is updated using a multiplicative weights argument. For bandit feedback (i.e., when only the chosen expert's loss is revealed to the learner), the multiplicative weights update rule uses an inverse propensity scoring estimator for each expert's loss in place of their real loss. The new element that ADANORMALHEDGE brings to the table is a way of defining the weights at each round t ; specifically, the weights are updated proportionally to the sum of instantaneous regret for each expert until round t . This allows the learner to obtain finer control over the total regret without needing extra parameters to tune the algorithm at each round. The exact regret guarantee that ADANORMALHEDGE obtains is stated formally below.

Lemma D.1 (ADANORMALHEDGE [44]) . Let r t,i = I t,i ( ˆ ℓ t -ℓ t,i ) be the instantaneous regret of any active expert i ∈ A t at round t , and c t,i = | r t,i | . Then, ADANORMALHEDGE with prior q ∈ ∆([ N ]) selects experts according to the following distribution

The regret of ADANORMALHEDGE against any distribution over experts u ∈ ∆([ N ]) is bounded by

where by D KL ( u || q ) we denote the KL-divergence between distributions u and q .

ADANORMALHEDGE can be used to obtain adaptive regret bounds by creating a sleeping expert ( i, s ) for each i ∈ [ N ] , s ∈ [ T ] that has the same loss as expert i but is only awake after s .

Corollary D.2. Running ADANORMALHEDGE for the sleeping expert setting with prior q ( i,s ) ∝ 1 s 2 gives regret

where T i = ∑ T t =1 I t,i is the total number of rounds in which i is active.

D.2 Formula for Computing Q t when m = 2

To obtain the explicit formula for Q t , we first discretize the space of forecasts F P = [0 , 1] (since we focus on the case where m = 2 ) to form set F ε P = { 0 , ε, 2 ε, . . . , 1 -ε, 1 } . Then, we have that for each ˆ p ∈ F ε P :

where we have omitted index j from the sleeping expert g since because m = 2 , we can focus on a single action j . We assume WLOG that no forecast p ∈ F ε P falls exactly on the boundary of best response polytopes, so there is no tie-breaking needed. From Equation (17), we have that:

where the equation also uses the fact that for m = 2 , max h t ∈H P h t = 1 .

In the final step, we map p to the discretized grid of F ε P . Let jε, ( j +1) ε be two adjacent discretized points and q ∈ [0 , 1] such that: qZ jε +(1 -q ) Z ( j +1) ε 0 . Then, setting q t,jε = q and q t, ( j +1) ε = 1 -q and using Equation (18) gives that

D.3 Proof of Theorem 4.1

Proof of Theorem 4.1. We first specify how to build a set of sleeping experts settings from our problem definition. For that, consider the following set of experts:

i.e., we create a different expert for each round, each principal-agent action pair, and each σ (the use of which will be made clear in the next paragraph). For each expert g ( s,i,j, ( σ ) ∈ G and t ∈ [ T ] , we define the loss, sleeping/awake indicator, and instantaneous regret respectively as:

where by h t,j , p t,j we denote the j -th coordinate of the h and p vectors respectively.

Running ADANORMALHEDGE on the instance with G that we specified above, with prior q g ( s,i,j,σ ) ∝ 1 s 2 [44, Section 5.1] guarantees that ∀ g ( s,i,j,σ ) ∈ G ,

where ˜ O ( · ) hides lower order poly-logarithmic terms. Therefore, by simulating the NRBR dynamics, we obtain that with probability at least ≥ 1 -δ , ∀ i ∈ [ k ] and 1 ≤ s < t ≤ T ,

where for the first derivation we add and subtract ∑ τ ˆ ℓ τ and use that because of the NRBR dynamics: ˆ ℓ τ = E g ∼ P τ [ ℓ τ,g ] , and for the last derivation we have used the definition of ℓ τ,g = L g ( h τ , p τ ) (Equation (6)). We next upper bound the term T as follows:

where the first inequality is by the property of h τ being the best strategy for the principal, and the last one uses the fact that max h τ ∈ ∆( A P ) E g ∼ πτ p ∼ Qτ [ L g ( h τ , p )] ≤ ε from the NRBR Equation (8) and a martingale concentration bound on the second term.

Plugging the upper bound for Q back to Equation (20) and using the regret bound for AdaNormalHedge (Equation (19)) we get:

E Supplementary Material for Section 5

E.1 Proof of Theorem 5.2

Theorem 5.2 (Restated) . For continuous CSGs satisfying Assumption 5.1, for all ε 0 > 0 , there exists a finite binning Π 0 such that if the agent is (0 , Π 0 ) - adaptively calibrated and the principal runs an appropriately parametrized instance of LAZYGDWOG (Algorithm 3) then:

Moreover, for any sequence of the principal's actions h [1: T ] , it holds that:

E.1.1 Proof of Lower Bound

Proof. Before delving into the proof of the lower bound, we first introduce some notations. Let C ( h ) ≜ U P ( h , BR ( h )) . Let V ⋆ δ ≜ max h ∈ B 2 ( A P , -δ ) C ( h ) be the optimal utility restricted in the smaller strategy set B 2 ( A P , -δ ) . We use y ϕ ≜ 1 M ∑ s ∈ [ M ] y ϕ,s to denote the average feedback that LAZYGDWOG uses to update the strategies.

We first consider any fixed ε > 0 . Combining the guarantees of Lemmas E.4 and E.5, we conclude that there exists a finite binning Π 0 and M ε < ∞ , such that if the agent is (0 , Π 0 ) -adaptively calibrated, then ∀ M ≥ M ε , the following two inequalities are satisfied at the same time:

Set the parameters according to γ ϕ = γ 0 m -1 2 ϕ -3 4 and δ ϕ ≡ δ = δ 0 m 1 2 Φ -1 4 in Algorithm 3, then similar arguments to [51, Theorem 3.1] guarantee that

where (a) is from Equation (21).

Now we upper bound the difference between V ⋆ and V ⋆ δ = max h ∈ B 2 ( A P , -δ ) C ( h ) , then we have

where the second inequality follows from Assumption 5.1 that C ( h ) is L U -Lipschitz.

The next step is to upper bound the difference between the actual average utility and 1 Φ ∑ ϕ ∈ [Φ] E [ C ( h ϕ )] . From Equation (22), we have

Finally, putting the above inequalities together, we obtain

Taking the limit of Φ →∞ , the above inequalities imply

Since the above arguments hold for all ε > 0 , taking ε = ε 0 L BR D P +1 proves the theorem.

E.1.2 Proof of Upper Bound

For a fixed ε > 0 , let D ε = { x 1 , · · · , x I } be an ε -grid of F P under ℓ 2 distance, and let Π 0 be the continuous binning specified by Equation (26). We have:

In the above inequalities that lead to (23), step (a) is because U P is L 2 -Lipschitz in the second argument and BR ( · ) is L BR -Lipschitz, and the fact that w i ( p t ) > 0 only when ∥ p t -x i ∥ 2 < 2 ε . In step (b), we used Jensen's inequality because U P in concave in the first argument. Step (c) follows from the definition of n T ( i ) and h T ( i ) in Definition 2.3. The last inequality (d) uses the fact that U P is L 1 -Lipschitz in the first argument to decompose U P ( p T ( i ) , BR ( x i )) into calibration error (i.e., term (B)) and U P ( p T ( i ) , BR ( x i )) where the strategy that the agent best responds to is close to the principal's strategy (i.e., term (A)).

We can further bound ( A ) and ( B ) in Equation (23) respectively as follows:

and

Therefore, putting the above bounds together, we obtain that with probability ≥ 1 -δ,

Since the above derivation holds for any ε > 0 , it suffices to take ε such that 2( L 1 + L 2 L BR ) ε = ε 0 . Finally, since | D ε | < ∞ and r δ ( T ) = o (1) , taking the limit of T →∞ proves the upper bound:

E.2 Key lemma: asymptotically correct forecast

In this section, we state and prove the key lemma for establishing Theorem 5.2. Intuitively, this lemma states that for any strategy h ∈ A P , as long as the principal repeatedly plays h for enough rounds, the fraction of times where the agent's forecast is close to h will converge to 1.

Lemma E.1. For any ε 0 > 0 , there exists a finite binning Π 0 , such that if the principal repeatedly plays any h ∈ A P for M rounds and the agent's forecasts p 1: M are (0 , Π 0 ) - adaptively calibrated, then:

In particular, if the calibration error (defined in Definition 2.3) has rate r ( · ) ∈ o (1) with respect to Π 0 , then

Proof of Lemma E.1. We first describe the construction of Π 0 . For ε = 1 4 ε 0 , let D ε = { x 1 , · · · , x I } be an ε -grid of F P under ℓ 2 distance, and Λ( p ; x, R ) ≜ ( R -∥ p -x ∥ 2 ) + be the tent function with center x and radius R . Consider the following binning

Clearly, | Π 0 | = I < ∞ because the diameter of F P is bounded as stated in Theorem 5.1. We can also verify that Π 0 satisfies ∑ i ∈ [ I ] w i ( p ) = 1 for all p ∈ F P because w i ( p ) is defined as the normalized tent function.

Now we prove that Π 0 satisfies the desired property. Since the agent is adaptively calibrated to Π 0 , we have that ∀ i ∈ [ I ] ,

Since | D ( δ ) ε | ≤ | D ε | = I < ∞ , taking the sum of calibration error over bins in D ( δ ) ε , we obtain

We can further lower bound (28) and get:

In the above inequalities, (a) and (b) are both due to triangle inequalities, and (c) is because ∥ h -x i ∥ 2 ≥ δ from the definition of D ( δ ) ε in (27) and ∥ x i -p s ∥ 2 < 2 ε whenever w i ( p s ) > 0 ⇐⇒ Λ( p s ; x i , 2 ε ) > 0 . Together with (28), the above set of inequalities imply

On the other hand, since D ε is an ε -grid of F P , if ∥ p s -h ∥ 2 ≥ ε 0 , there must exist x i ∈ D ε such that ∥ x i -p s ∥ 2 ≤ ε , which implies

As for the weight that w i assigns to p s , we also have

Therefore, we have

Finally, combining inequalities (29) and (30), we conclude that

which proves (25). The proof is complete by taking the limit of M → 0 , which guarantees r ( M ) → 0 and immediately implies the convergence result in (24).

Note that the rate in Equation (25) does not depend on strategy h . Therefore, in the context of running LAZYGDWOG (Algorithm 3), we can turn Lemma E.1 into the following uniform convergence result across epochs:

Proposition E.2. For any ε 0 > 0 , there exists a finite binning Π 0 , such that ∀ Φ > 0 , if the principal runs LAZYGDWOG for Φ epochs where each epoch has length M , and the agent's forecasts ( p ϕ,s ) ϕ ∈ [Φ] ,s ∈ [ M ] are adaptively calibrated with respect to Π 0 , then we have the following uniform convergence guarantee:

Remark E.3. Note that the rate in (25) has a polynomial dependency on | Π 0 | , which, due to the construction in the proof of Lemma E.1, ends up being exponential in m because it is the size of a ε 0 4 grid of the domain A P . To improve on this exponential dependency, one possible approach is to design an adaptive calibration algorithm for the agent that achieves the stronger notion of ℓ 1 calibration, which is more common in recent literature. For example, Hart [36], Foster and Vohra [29, 30] are defined using ℓ 1 calibration error rather than ℓ ∞ . Another approach is to avoid using naive conversion from ℓ ∞ to ℓ 1 calibration error in (28) , which leads to a polynomial dependency on the number of bins. These two approaches are equivalent ways of formulating the problem, and they both lead to interesting open directions.

E.3 More auxiliary lemmas: approximate best response and closeness in utility

In this section, we use the results in Appendix E.2 to show that the average feedback 1 M ∑ s ∈ [ M ] y ϕ,s in epoch ϕ ∈ [Φ] is close to the best response BR ( h ϕ ) (Lemma E.4), and that the principal's average utility in this epoch is close to U P ( h ϕ , BR ( h ϕ )) (Lemma E.5).

Lemma E.4. For any ε 1 > 0 , there exists a finite binning Π 0 and M 0 < ∞ such that when agent's forecasts p 1: t are adaptively calibrated with respect to Π 0 , then we have that ∀ M ≥ M 0 ,

Proof. Let ε 0 = ε 1 2 L BR and Π 0 be the binning that satisfies Proposition E.2 for parameter ε 0 . Therefore, we know from Equation (31) in Proposition E.2 that for ε 2 = ε 1 2 · D P · L BR there exists M 0 such that ∀ M ≥ M 0 ,

Using Lipschitzness of the best response mapping BR ( · ) , we have that ∀ ϕ ∈ [Φ] ,

Lemma E.5. For any ε 1 > 0 , there exists a finite binning Π 0 and M 0 < ∞ such that when agent's forecasts p 1: t are adaptively calibrated with respect to Π 0 , then we have that ∀ M ≥ M 0 ,

Proof. The proof of this lemma is very similar to that of Lemma E.4, with a different choice of constants ε 0 and ε 2 . Note that since U P is L 2 -Lipschitz in the second argument, we have

The rest of the proof follows from Lemma E.4 by choosing ε 0 = ε 1 2 L 2 L BR and ε 2 = ε 1 2 · D P · L BR L 2 .

F Future Directions

Adaptive Calibration Although the results in this paper are all stated in terms of the ℓ ∞ -calibration error (e.g., maximum instead of sum over bins), a lot of the existing calibration literature focuses on ℓ 1 -calibration error [29, 30]. It is an interesting problem whether we can get ℓ 1 -adaptive calibration error bounds without a polynomial dependency in the number of binning functions, where obtaining such bounds lead to polynomial improvements on the dependency of m (the number of agent's actions). In the case of continuous calibration, it is an open problem to obtain uniform (adaptive) calibration error bounds for parametric or nonparametric continuous binning function classes. Resolving this open problem could lead to a better rate for the learning direction of Theorem 5.2, as the current result uses naive ℓ ∞ -toℓ 1 conversion of calibration error that leads to linear dependency on the number of binning functions, which turns out to be exponential in the dimension of the principal's action space. See Remark E.3 for more details.

Convex optimization from membership oracles In the case of finite Stackelberg games, we use the results of Lee et al. [42] to solve the constrained convex optimization problem in each polytope from approximate membership queries, where the key step is a polynomial reduction from approximate separation oracles to approximate membership oracles. However, the precision of the constructed separation oracle is worse than the precision of the membership oracle, which naturally leads to a

worse precision of the final optimization solution. To be more specific, given an ε -membership oracle to a convex set K , Lee et al. [42] can only guarantee the returned solution to be ε 1 /γ 0 -optimal and contained in B 2 ( K,ε 1 /γ 0 ) , where γ 0 > 6 is a fixed constant. An understudied open direction in the optimization community is whether a variant of Lee et al. [42] can return a solution in B 2 ( K,ε + δ ) for δ that is any tunable parameter, with runtime depending polynomially on 1 /δ .

Open Problem F.1. Given an ε -approximate membership oracle of a convex set K ⊆ R m and an evaluation oracle of a convex function f , is there an optimization algorithm that, under sufficient regularity conditions, for any tunable δ , finds a near-optimal solution ˆ x such that f (ˆ x ) ≤ min x ∈ K f ( x ) + ε 1 /γ 0 and ˆ x ∈ B 2 ( K,ε + δ ) within poly ( m, 1 /δ, log(1 /ε )) calls to both oracles?

We note that resolving Open Problem F.1 will immediately lead to exponentially improved rates for our Theorem C.14 for any finite Stackelberg games. To see this, recall that the algorithms that achieve Theorem C.14 require repeated calls to the constructed APPROXMEM oracle. This oracle provides an ε 1 -approximate response, indicating whether or not the query point belongs to the ε 2 -conservative version of each best response polytope. The number of samples used in simulating each oracle call is exponential in ε 2 ε 1 . The algorithm by Lee et al. [42] amplifies the inaccuracy of APPROXMEM at a polynomial rate of γ 0 . Consequently, imposing a constraint of ε 1 ≲ ε γ 0 2 . However, resolving Open Problem F.1 removes this constraint and introduces a much milder one, i.e., ε 1 ≲ ε 2 -δ , where δ is a tunable parameter. This adjustment leads to exponential improvement in the final sample complexity.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript introduces the concept of calibrated forecasts in repeated Stackelberg games (SG) and proposes two concepts: the calibrated Stackelberg games (CSG) that generalizes the standard SGs and the adaptive calibrated forecast. The technical contribution is as follows: First, a principal's learning algorithm against adaptively calibrated agents is proposed where the average utility of the principal converges to the Stackelberg value, which is the best possible average utility under this setting. Second, a forecasting algorithm that meets the concept of calibrated forecast is proposed, which means that the agents can actually perform calibrated forecasts for the principal's action. Third, even for continuous Stackelberg games where the action sets of principal and agent are continuous, there will be principal's learning algorithm against certainly adaptively calibrated agents where the average utility of the principal converges to the Stackelberg value, the best possible one.\n\nSTRENGTHS:\nThe notion of calibrated forecast, on which the paper is entirely based, seems much reasonable; compared to the original definition, it is generalized by introducing a binning function, enabling us to represent some rules for choosing the best response of agents (deterministic/randomized tie-breaking rule in the manuscript). Although its assumption seems much stronger and unrealistic, this work succeeded in proposing forecasting algorithms that meets the conditions of adaptive calibration by combining online algorithms with the study on novel game dynamics; its technical contribution seems far non-trivial. For the principal, this work succeed in proposing learning algorithms that achieves the best possible average utility asymptotically, meaning that the proposed learning algorithm is asymptotically the best one under this setting.\n\nWEAKNESSES:\nThe connection between the proposed concepts (calibrated Stackelberg games and adaptive calibrated forecast) and the applications the manuscript claims (Stackelberg security games and strategic classification) is unclear from the manuscript although it is claimed that the results \"immediately apply\". Thus, it looks like that this work addresses an artificial setting in Stackelberg games. To avoid this, the authors should carefully review the connection between the proposed concepts and the applications at least in the Appendix.\n\nMinor comment:\nIn Theorem 5.2, unlike Theorem 3.1, the binning \\Pi_0 is fixed as Eq. (25), but it is described only in the Appendix. As far as I read the main part, I'm afraid that the binning is irresponsible and not related to the agent's policy. For the sake of completeness, please consider describing the actual formula for the binning and its meaning in the main article.\n\nQUESTIONS:\nP.3, l.128: \"Definition 2.2 is weaker than the standard definition of calibration...\" What does \"weaker\" mean? As far as I understand, introducing a binning function \\Pi is a generalization compared to the standard definition but it does not strengthen or weaken the assumption.\n\nLIMITATIONS:\nThe authors describe some limitations (and thus future directions) of this work in the conclusion section and in Appendix F.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors consider a problem of Calibrated Stackelberg Games (CSG), which is a generalization of the Stackelberg Games. These framework differ from the standard online learning problems as in the SG framework instead of only having a single learner entity, there is a principal and an agent. The key difficulty introduced by the CSG framework is that the agent needs to respond to the action of the principal without being able to observe it. Instead, the agent have to forecast what they expect the principal's action to be and aim to best respond to that believed action from the principal. \nAs this learning objective is more challenging than classic learning problems, the authors consider the question of whether the principal can achieve optimal utility (V* being the utility of the principal's action with the highest best response) in this calibrated game. \nAn important part of the paper is dedicated to the construction of adaptively calibrated forecasts and of the CSG protocol. Then, the authors present and analyse an algorithm that can achieve optimal utility with high probability. \nThe principal's algorithm is a simple explore then commit strategy. This exploration starts with  $log T/\\eta$ uniform uniform strategies samples, for which the agent returns an associated response. Then the principal tries to find approximately optimal strategies for each of the agent's response, and ends up picking the one of these that yields the highest utility. \nThey then analyze the performance of the algorithm and show that it assumptotically reaches the optimal utility for discrete and continuous games.\n\nSTRENGTHS:\nThis paper studies a generalization of the Stackelberg games, which are challenging in the online learning framework, and show that it is possible to reach optimal utility for the principal even if the agent doesn't have access to the strategy picked by the principal ahead of time. \nThese results are novel, and the analysis of the algorithm builds upon standard online learning algorithms. The authors took particular care in connecting the CSG framework with other online learning problems such as sleeping experts.\n\nThis work provides some good preliminary baselines for the SCG problem, and should provide a strong foundation for future works to build upon it.\n\nWEAKNESSES:\nThe main weakness of the paper is that the results provided, meaning that the algorithm presented can find the optimal utility, only holds asymptotically, making it difficult to appear useful in practice, when we only have a limited time-horizon. \nIt would be useful to discuss extensions of this work that could achieve stronger guarantees in finite time horizons.\n\nQUESTIONS:\nDo you think that it is possible to extend your work beyond the asymptotic guarantees that you provide?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper defines and studies a new Stackelberg games setup. Rather than making some standard assumptions --- e.g. that the principal and/or the agent exhibit specific types of play (e.g. agent playing no regret), or assuming access to the agent’s best response oracle, etc. --- this paper only assumes that the agent will be playing by best-responding to (appropriately) calibrated forecasts of the principal play. In this Calibrated Stackelberg Games setup, the authors show that: (1) The principal, by only knowing that the agent will be playing in a calibrated way, can achieve exactly (no more and no less than) the Stackelberg value of the game over a repeated interaction, by following an explore-then-commit style algorithm; (2) The agent has an efficient algorithm for producing said calibrated forecasts (in fact, strengthened by the notion of adaptivity --- meaning that calibration should hold over all subintervals of the time axis). These results are further complemented by e.g. considering both finite and continuous action spaces, as well as an adaptive calibration algorithm.\n\nSTRENGTHS:\nAs intended by the authors, they are able to demonstrate that Calibrated Stackelberg games indeed show potential towards relaxing/moving away from various restrictive assumptions (on what the principal and the agent observe and how they play) in the literature. An important moral takeaway is that the same old Stackelberg value --- which one might have expected might in fact require the principal and the agent to (more or less) explicitly observe each other’s play --- can be achieved over time by only asking the agent to use calibrated forecasts of the principal’s play (which is a fairly mild requirement), and the principal to know and use the fact that the agent exhibits calibration. \n\nIn fact, this conclusion is carefully shown to hold (existentially and algorithmically) both for finite and continuous action spaces --- which, while to be morally expected from standard minimax reasoning, still takes work to rigorously establish and shows thoroughness on the authors’ part.\n \nIn terms of the techniques employed, the exploration process for achieving this value on the principal’s side is, in fact, not very straightforward to design both for finite and continuous action spaces; so, there is a healthy dose of sophistication involved. The adaptive calibration algorithm, on the other hand, quite straightforwardly follows from existing techniques at the intersection of no-regret dynamics and online multiobjective settings.\n\nWEAKNESSES:\nI did not spot any technical weaknesses, and the contribution of this paper to both the Stackelberg games and the calibration literatures is solid. So overall, the paper does not have any significant weaknesses. However, \n\nI would, however, like to point out that the principal needing, in certain parts of this new framework, to know the calibration rate of the agent is not to be taken lightly given that an important motivation of this paper is to relax, as much as possible, the prevalent specificity, in existing literature, of the requirements on what the principal and the agent should know. I would appreciate it if the authors could provide further elaboration of this, beyond the brief mention in the conclusion.  \n\nSecondly, while having *adaptive* calibration guarantees is nice, compared to regular marginal ones, I’m not clear on how or whether this adaptivity interacts with the proposed theory of CSG or is more or less orthogonal? In other words --- okay, using the standard sleeping experts technique for establishing adaptive online guarantees, it is possible to make the agent calibrated on every [s, t] rather than just on [1, T]; but does that really matter for e.g. being able to achieve the value V* in the process of play, or any other desirable Stackelberg properties? Since the main point of the paper is to propose a theory of calibrated Stackelberg games, it is important to be clear on whether this is an essential element of such a theory or was just added to the paper for good measure. If this is in fact an essential element, I’d like to ask the authors to clarify this.\n\nI also did find the presentation suboptimal --- the paper reads quite densely; especially, in my experience, when it comes to the proof sketch after Theorem 3.2 in Section 3. I invite the authors to revamp that part of the presentation for the rebuttal phase. Some specific things that I’d appreciate would be (1) alleviating the notational/explanatory tedium related to condition (P1) --- I am still not clear on how strong or weak it is, and where exactly things must break if it wasn’t required; (2) adding a high-level description (preferably involving more prose) of how the agent’s calibration figures into what the algorithm for the principal does.\n\nQUESTIONS:\nFor the substantive questions, please see my questions above on: (1) the principal needing to know the calibration rate; (2) the requirement of adaptivity of calibration; (3) some elements of Section 3 such as property (P1) etc. \n\nHere, I’ll quickly list a sampling of a few typos and notational issues:\nLine 300: bound*ed*\nLine 285: Where is the notation L_g defined?\nLine 276: Brackets around sigma in the subscript for g\nLine 222: Probably meant to say that (h hat, y) is an equilibrium rather than just h hat\nLine 201: Where is h bar_T defined?\nLine 174: The fundamental constructs from Definition 2.2 are reviewed, not from Eq 1\nLine 102: converged *to a* Stackelberg equilibrium\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}