Title: Regret Minimization in Stackelberg Games with Side Information
Venue: NeurIPS 2024
Accepted: yes
Keywords: Stackelberg game

context

online learning
Abstract: Algorithms for playing in Stackelberg games have been deployed in real-world domains including airport security, anti-poaching efforts, and cyber-crime prevention. However, these algorithms often fail to take into consideration the additional information available to each player (e.g. traffic patterns, weather conditions, network congestion), a salient feature of reality which may significantly affect both players' optimal strategies. We formalize such settings as Stackelberg games with side information, in which both players observe an external context before playing. The leader commits to a (context-dependent) strategy, and the follower best-responds to both the leader's strategy and the context. We focus on the online setting in which a sequence of followers arrive over time, and the context may change from round-to-round. In sharp contrast to the non-contextual version, we show that it is impossible for the leader to achieve good performance (measured by regret) in the full adversarial setting.  Motivated by our impossibility result, we show that no-regret learning is possible in two natural relaxations: the setting in which the sequence of followers is chosen stochastically and the sequence of contexts is adversarial, and the setting in which the sequence of contexts is stochastic and the sequence of followers is chosen by an adversary.

Regret Minimization in Stackelberg Games with Side Information

Abstract

Algorithms for playing in Stackelberg games have been deployed in real-world domains including airport security, anti-poaching efforts, and cyber-crime prevention. However, these algorithms often fail to take into consideration the additional information available to each player (e.g. traffic patterns, weather conditions, network congestion), which may significantly affect both players' optimal strategies. We formalize such settings as Stackelberg games with side information , in which both players observe an external context before playing. The leader commits to a (contextdependent) strategy, and the follower best-responds to both the leader's strategy and the context. We focus on the online setting in which a sequence of followers arrive over time, and the context may change from round-to-round. In sharp contrast to the non-contextual version, we show that it is impossible for the leader to achieve noregret in the full adversarial setting. Motivated by this result, we show that no-regret learning is possible in two natural relaxations: the setting in which the sequence of followers is chosen stochastically and the sequence of contexts is adversarial, and the setting in which contexts are stochastic and follower types are adversarial.

1 Introduction

A Stackelberg game [30, 31] is a strategic interaction between two utility-maximizing players in which one player (the leader ) is able to commit to a (possibly mixed) strategy before the other player (the follower ) takes an action. While Stackelberg's original formulation was used to model economic competition between firms, Stackelberg games have been used to study a wide range of topics in computing ranging from incentives in algorithmic decision-making [15] to radio spectrum utilization [32]. Perhaps the most successful application of Stackelberg games to solve real-world problems is in the field of security, where the analysis of Stackelberg security games has led to new methods in domains such as passenger screening at airports [8], wildlife protection efforts in conservation areas [10], the deployment of Federal Air Marshals on board commercial flights [18], and patrol boat schedules for the United States Coast Guard [1]. 1

However in many real-world settings which are typically modeled as Stackelberg games, the payoffs of the players often depend on additional contextual information which is not captured by the Stackelberg game framework. For example, in airport security the severity of an attack (as well as the 'benefit' of a successful attack to the attacker) depends on factors such as the arrival and departure city of a flight, the whether there are VIP passengers on board, and the amount of valuable cargo on the aircraft. Additionally, there may be information in the time leading up to the attack attempt which may help the security service determine the type of attack which is coming [17]. For instance, in wildlife protection settings factors such as the weather and time of year may make certain species of

Maria-Florina Balcan

wildlife easier or harder to defend from poaching, and information such as the location of tire tracks may provide context about which animals are being targeted. As a result, the optimal strategy of both the leader and the follower may change significantly depending on the side information available.

Overview of our results. In order to capture the additional information that the leader and follower may have at their disposal, we formalize such settings as Stackelberg games with side information . Specifically, we consider a setting in which a leader interacts with a sequence of followers in an online fashion. At each time-step, the leader gets to see payoff-relevant information about the current round in the form of a context . After observing the context, the leader commits to a mixed strategy over a finite set of actions, and the follower best-responds to both (1) the leader's strategy and (2) the context in order to maximize their utility. We allow the follower in each round to be one of K types . Each follower type corresponds to a different mapping from contexts, leader strategies, and follower actions to utilities. While the leader may observe the context before committing to their mixed strategy, they do not get to observe the follower's type until after the round is over. Under this setting, the goal of the leader is to minimize their regret with respect to the best policy (i.e. the best mapping from contexts to mixed strategies) in hindsight, with respect to the realized sequence of followers and contexts.

Weshow that in the fully adversarial setting (i.e. the setting in which both the sequence of contexts and follower types is chosen by an adversary), no-regret learning is not possible, even when the policy class is highly structured. We show this via a reduction from the problem of online linear thresholding, for which it is known that no no-regret learning algorithm exists. Motivated by this impossibility result, we study two natural relaxations: (1) a setting in which the sequence of contexts is chosen by an adversary and the sequence of follower types is chosen stochastically, and (2) a setting in which the sequence of contexts is chosen stochastically and the sequence of follower types is chosen by an adversary.

In the stochastic follower setting we show that the greedy algorithm (Algorithm 1), which estimates the leader's expected utility for the given context and plays the mixed strategy which maximizes their estimate, achieves no-regret as long as the total variation distance between their estimate and the true distribution is decreasing with time. We then show how to instantiate the leader's estimation procedure so that the regret of Algorithm 1 is O (min { K,A f } √ T log( T )) , where T is the time horizon, K is the number of follower types, and A f is the number of follower actions. In the stochastic context setting, we show the leader can obtain O ( √ KT log( T )+ K ) regret by playing Hedge over a finite set of policies (Algorithm 2). An important intermediate result in both settings is that it is (nearly) without loss of generality to consider leader policies which map to a finite set of mixed strategies E z , given context z . 2

Next, we extend our algorithms for both types of adversary to the setting in which the leader does not get to observe the follower's type after each round, but instead only gets to observe their action. We refer to this type of feedback as bandit feedback . Both of our extensions to bandit feedback make use of the notion of a barycentric spanner [4], a special basis under which bounded loss estimators may be obtained for all leader mixed strategies. In the bandit stochastic follower setting, we use the fact that in addition to being bounded, a loss estimator constructed using a barycentric spanner has low variance, in order to show that a natural extension of our greedy algorithm obtains ˜ O ( T 2 / 3 ) regret.We also make use of barycentric spanners in the (bandit) stochastic context setting, albeit in a different way. Here, our algorithm proceeds by splitting the time horizon into blocks, and using a barycentric spanner to estimate the leader's utility from playing according to a set of special policies in each block. We then play Hedge over these policies to obtain ˜ O ( T 2 / 3 ) leader regret. 3 See Table 1 for a summary of our results.

Related work. Letchford et al. [21] consider the problem of learning the leader's optimal mixed strategy in the repeated Stackelberg game setting against a perfectly rational follower with an unknown payoff matrix. Peng et al. [23] study the same setting as Letchford et al. [21]. They provide improved rates and prove nearly-matching lower bounds. Learning algorithms to recover the leader's optimal mixed strategy have also been studied in Stackelberg security games [5, 7, 26, 6].

Our work builds off of several results established for online learning in (non-contextual) Stackelberg games in Balcan et al. [5]. In particular, our results in Section 4.2 and Appendix C.2 may be viewed as a generalization of their results to the setting in which the payoffs of both players depend on an external context. Roughly speaking, Balcan et al. [5] show that it without loss to play Hedge over a finite set

of mixed strategies in order to obtain no-regret against an adversarially-chosen sequence of follower types. In order to handle the additional side information available in our setting, we instead play Hedge over a finite set of policies , each of which map to a finite set of (context-dependent) mixed strategies. However, the discretization argument is more nuanced in our setting. In particular, it is not without loss of generality to consider a finite set of policies. As a result, we need to bound the additional regret incurred by the leader due to the discretization. More recent work on learning in Stackelberg games provides improved regret rates in the full feedback [9] and bandit feedback [6] settings, and considers the effects of non-myopic followers [13] and followers who respond to calibrated predictions [14].

Lauffer et al. [20] study a Stackelberg game setting in which there is an underlying (probabilistic) state space which affects the leader's rewards, and there is a single (unknown) follower type. In contrast, we study a setting in which the sequence of follower types and/or contexts may be chosen adversarially. Sessa et al. [25] study a repeated game setting in which the players receive additional information (i.e. a context) at each round, much like in our setting. However, their focus is on repeated normal-form games, which require different tools and techniques to analyze compared to the repeated Stackelberg game setting we consider. Other work has also considered repeated normal-form games which change over time in different ways. In particular, Zhang et al. [33], Anagnostides et al. [3] study learning dynamics in time-varying game settings, and Harris et al. [16] study a meta-learning setting in which the game being played changes after a fixed number of rounds.

Finally, our problem may be viewed is a special case of the contextual bandit setting with adversariallychosen utilities [28, 29, 24], where the learner gets to observe 'extra information' in the form of the follower's type (Section 4) or the follower's action (Section 5). However, there is much to gain from taking advantage of the additional information and structure that is present in our setting. Besides having generally worse regret rates, another reason not to use off-the-shelf adversarial contextual bandit algorithms in our setting is that they typically require either (1) the learner to know the set of contexts they will face beforehand (the transductive setting; Syrgkanis et al. [28, 29], Rakhlin and Sridharan [24]) or (2) for there to exist a small set of contexts such that any two policies behave differently on at least one context in the set (the small separator setting; Syrgkanis et al. [28]). We require no such assumptions.

2 Setting and background

Notation. We use [ N ] := { 1 , . . . , N } to denote the set of natural numbers up to and including N ∈ N and cl ( P ) to denote the closure of the set P . x [ a ] denotes the a -th component of vector x , and ∆( A ) denotes the probability simplex over the set A . TV( p , q ) = 1 2 ∫ | p ( x ) -q ( x ) | dx is the total variation distance between distributions p and q , and E t [ x ] = E [ x |F t ] is shorthand for the expected value of the random variable x , conditioned on the filtration up to (but not including) time t . All proofs may be found in the Appendix. Finally, while we present our results for general Stackelberg games with side information, our results are readily applicable to the special case of Stackelberg security games with side information.

Our setting. Weconsider a game between a leader and a sequence of followers. In each round t ∈ [ T ] , Nature reveals a context z t ∈ Z ⊆ R d to both players. 4 The leader moves first by playing some mixed strategy x t ∈ X ⊆ R A over a set of (finite) leader actions A , i.e., x t ∈ ∆( A ) . The size of A

is A := |A| . Having observed the leader's mixed strategy, the follower best-responds to both x t and z t by playing some action a f ∈ A f , where A f is the (finite) set of follower actions and A f := |A f | .

Definition 2.1 (Follower Best-Response) . Follower f 's best-response to context z and mixed strategy x is b f ( z , x ) ∈ arg max a f ∈A f ∑ a l ∈A x [ a l ] · u f ( z , a l , a f ) , where u f : Z × A × A f → [0 , 1] is follower f 's utility function. In the case of ties, we assume that there is a fixed and known ordering over actions which determines how the follower best-responds, i.e. if a > a ′ for a, a ′ ∈ A f then the follower will break ties between a and a ′ in favor of a . 5

We allow for the follower in round t (denoted by f t ) to be one of K ≥ 1 follower types { α (1) , . . . , α ( K ) } (where K ≤ T ). Follower type α ( i ) is characterized by utility function u α ( i ) : Z × A × A f → [0 , 1] , i.e. given a context z , leader action a l , and follower action a f , a follower of type α ( i ) would receive utility u α ( i ) ( z , a l , a f ) . We assume that the set of all possible follower types and their utility functions are known to the leader, but that the follower's type at round t is not revealed to the leader until after the round is over. We denote the leader's utility function by u : Z×A×A f → [0 , 1] and assume it is known to the leader. We often use the shorthand u ( z , x , a f ) = ∑ a l ∈A x [ a l ] · u ( z , a l , a f ) to denote the leader's expected utility of playing mixed strategy x under context z against follower action a f . Follower f t 's expected utility u f t ( z , x , a f ) is defined analogously.

A leader policy π : Z → X is a (possibly random) mapping from contexts to mixed strategies. If the leader using policy π t in round t and observes context z t , their strategy x t is given by x t ∼ π t ( z t ) .

Definition 2.2 (Optimal Policy) . Given a sequence of followers f 1 , . . . , f T and contexts z 1 , . . . , z T , the strategy given by the leader's optimal-in-hindsight policy for context z is π ∗ ( z ) ∈ arg max x ∈X ∑ T t =1 u ( z , x , b f t ( z , x )) · { z t = z } .

✶ We measure the leader's performance against the optimal policy via the notion of contextual Stackelberg regret ( regret for short).

Definition 2.3 (Contextual Stackelberg Regret) . Given a sequence of followers f 1 , . . . , f T and contexts z 1 , . . . , z T , the leader's contextual Stackelberg regret is R ( T ) := ∑ T t =1 u ( z t , π ∗ ( z t ) , b f t ( z t , π ∗ ( z t ))) -u ( z t , x t , b f t ( z t , x t )) , where x 1 , . . . , x T is the sequence of mixed strategies played by the leader.

If an algorithm achieves regret R ( T ) = o ( T ) , we say that it is a no-regret algorithm. We consider three ways in which Nature can select the sequence of contexts/followers:

If the sequence of contexts (resp. follower types) are drawn i.i.d. from some fixed distribution, we say that the sequence of contexts (resp. follower types) are chosen stochastically .

If Nature chooses the sequence of contexts (resp. follower types) before the first round in order to harm the leader (possibly using knowledge of the leader's algorithm), we say that the sequence of contexts (resp. follower types) are chosen by a non-adaptive adversary .

If Nature chooses context z t (resp. follower f t ) before round t in order to harm the leader (possibly using knowledge of the leader's algorithm and the outcomes of the prior t -1 rounds), we say that the sequence of contexts (resp. follower types) are chosen by an adaptive adversary .

Our impossibility results in Section 3 hold when both the sequence of contexts and the sequence of follower types are chosen by either type of adversary. Our positive results in Section 4 hold when either the sequence of contexts or the sequence of follower types are chosen by either type of adversary (and the other sequence is chosen stochastically). Our extension to bandit feedback (Section 5, where the leader only gets to observe the follower's best-response instead of their type) holds whenever one sequence is chosen by a non-adaptive adversary and the other sequence is chosen stochastically.

3 On the impossibility of fully adversarial no-regret learning

We begin with a negative result: no-regret learning is not possible in the setting of Section 2 if the sequence of contexts and the sequence of followers is chosen by an adversary. While this is not necessarily surprising given that Definition 2.3 allows for the optimal policy π ∗ to be arbitrarily

RegretMinimizerforOnline

LinearThresholding

2.gt = h2(xt)

RegretMinimizer

Zt=h(wt)

Nature

GameswithSide

3.yt

Information

1.0t

complex, we show that this result holds even when the policy class to which π ∗ belongs is highly structured . We show this via a reduction to the online linear thresholding problem, for which it is known that no-regret learning is impossible.

Online linear thresholding. The online linear thresholding problem is a repeated two-player game between a learner and an adversary. Before the first round, an adversary chooses a cutoff s ∈ [0 , 1] which is unknown to the learner. In each round, the adversary chooses a point ω t ∈ [0 , 1] and reveals it to the learner. ω t is assigned label y t = 1 if ω t > s and label y t = -1 otherwise. Given ω t , the learner makes a guess g t ∈ [0 , 1] (the probability they place on y t = 1 ), and receives utility u OLT ( ω t , g t ) = g t · ✶ { y t = 1 } +(1 -g t ) · ✶ { y t = -1 } . The learner gets to observe y t after round t is over. The learner's policy π t : [0 , 1] → [0 , 1] is a mapping from points in [0 , 1] to guesses in [0 , 1] . The optimal policy π ∗ makes guess π ∗ ( ω t ) = 1 if ω t > s and π ∗ ( ω t ) = 0 otherwise. The learner's regret after T rounds is given by R OLT ( T ) = T -∑ T t =1 u OLT ( ω t , g t ) , since the optimal policy achieves utility 1 in every round. In order to prove a lower bound on contextual Stackelberg regret in our setting, we make use of the following well-known lower bound on regret in the online linear thresholding setting (see e.g. [12]).

Lemma 3.1. Any algorithm suffers regret R OLT ( T ) = Ω( T ) in the online linear thresholding problem when the sequence of points ω 1 , . . . , ω T is chosen by an adversary.

Theorem 3.2. If an adversary can choose both the sequence of contexts z 1 , . . . , z T and the sequence of followers f 1 , . . . , f T , no algorithm can achieve better than Ω( T ) contextual Stackelberg regret in expectation over the internal randomness of the algorithm, even when π ∗ is restricted to come from the set of linear thresholding functions.

The reduction from online linear thresholding proceeds by creating an instance of our setting such that the sequence of contexts z 1 , . . . , z T correspond to the sequence of points ω 1 , . . . , ω T encountered by the learner, and the sequence of follower types f 1 , . . . , f T correspond to the sequence of labels y 1 , . . . , y T . We then show that a no-regret algorithm in the online thresholding problem can be obtained by using an algorithm which minimizes contextual Stackelberg regret on the constructed game instance as a black box. However this is a contradiction, since by Lemma 3.1 the online thresholding problem is not online learnable by any algorithm. See Figure 1 for a visualization of our reduction.

Intuitively, this reduction works because the adversary can 'hide' information about the follower's type f t in the context z t . However, there exists a family of problem instances in which learning this relationship between contexts and follower types as hard as learning the threshold in the online linear thresholding problem, for which no no-regret learning algorithm exists by Lemma 3.1.

4 Limiting the power of the adversary

Motivated by the impossibility result of Section 3, we study two natural relaxations of the fully adversarial setting: one in which the sequence of followers is chosen stochastically but the contexts are chosen adversarially (Section 4.1) and one in which the sequence of contexts is chosen stochastically but followers are chosen adversarially (Section 4.2). In both settings we allow the adversary to be adaptive.

An important structural results for both Section 4.1 and Section 4.2 is that for any context z ∈ Z , the leader incurs only negligible regret by restricting themselves to policies which map to mixed

strategies in some finite (and computable) set E z . In order to state this result formally, we need to introduce the notion of a contextual best-response region , which is a generalization of the notion of a best-response region in (non-contextual) Stackelberg games (e.g. [21, 5]).

Definition 4.1 (Contextual Follower Best-Response Region) . For follower type α ( i ) , follower action a f ∈ A f , and context z ∈ Z , let X z ( α ( i ) , a f ) ⊆ X denote the set of all leader mixed strategies such that a follower of type α ( i ) best-responds to all x ∈ X z ( α ( i ) , a f ) by playing action a f under context z , i.e., X z ( α ( i ) , a f ) = { x ∈ X : b α ( i ) ( z , x ) = a f } .

Definition 4.2 (Contextual Best-Response Region) . For a given function σ : { α (1) , . . . , α ( K ) } → A f , let X z ( σ ) denote the set of all leader mixed strategies such that under context z , a follower of type α ( i ) plays action σ ( α ( i ) ) for all i ∈ [ K ] , i.e. X z ( σ ) = ∩ i ∈ [ K ] X z ( α ( i ) , σ ( α ( i ) )) .

For a fixed contextual best-response region X z ( σ ) , we refer to the corresponding σ as the bestresponse function for region X z ( σ ) , as it maps each follower type to its best-response for every leader strategy x ∈ X z ( σ ) . We sometimes use σ ( z , x ) to refer to the best-response function associated with mixed strategy x under context z , and we use Σ z to refer to the set of all best-response functions under context z . Note that | Σ z | ≤ A K f for any context z ∈ Z . This gives us an upper-bound on the number of best-response regions for a given context.

One useful property of all contextual best-response regions is that they are convex and bounded polytopes. To see this, observe that every contextual follower best-response region (and therefore every contextual best-response region) is (1) a subset of ∆ d and (2) the intersection of finitelymany half-spaces. While every X z ( σ ) is convex and bounded, it is not necessarily closed. If every contextual best-response region were closed, it would be without loss of generality for the leader to restrict themselves to the set of policies which map every context to an extreme point of some contextual best-response region. In what follows, we show that the leader does not 'lose too much' (as measured by regret) by restricting themselves to policies which map to some approximate extreme point of a contextual best-response region.

Definition 4.3 ( δ -approximate extreme points) . Fix a context z ∈ Z and consider the set of all nonempty contextual best-response regions. For δ > 0 , E z ( δ ) is the set of leader mixed strategies such that for all best-response functions σ and any x ∈ ∆( A l ) that is an extreme point of cl ( X z ( σ )) , x ∈ E z ( δ ) if x ∈ X z ( σ ) . Otherwise there is some x ′ ∈ E z ( δ ) such that x ′ ∈ X z ( σ ) and ∥ x ′ -x ∥ 1 ≤ δ .

Note that Definition 4.3 is constructive. We set δ = O ( 1 T ) so that the additional regret from only considering policies which map to points in ∪ z ∈Z E z ( δ ) is negligible. As a result, we use the shorthand E z := E z ( δ ) throughout the sequel. The following lemma is a generalization of Lemma 4.3 in Balcan et al. [5] to our setting, and its proof uses similar techniques from convex analysis.

Lemma 4.4. For any sequence of followers f 1 , . . . f T and any leader policy π , there exists a policy π ( E ) : Z → ∪ z ∈Z E z that, when given context z , plays a mixed strategy in E z and guarantees that ∑ T t =1 u ( z t , π ( z t ) , b f t ( z t , π ( z t ))) -u ( z t , π ( E ) ( z t ) , b f t ( z t , π ( E ) ( z t ))) ≤ 1 . Moreover, the same result holds in expectation over any distribution over follower types F .

Since we do not restrict the context space to be finite, the leader cannot pre-compute E z for every z ∈ Z before the game begins. Instead, they can compute E z t in round t before they commit to their mixed strategy. While E z t is computatable, it may be exponentially large in A f and K . However this is to be expected as Li et al. [22] show that in its general form, solving the non-contextual version of the online Stackelberg game problem is NP-Hard.

4.1 Stochastic follower types and adversarial contexts

In this setting we allow the sequence of contexts to be chosen by an adversary, but we restrict the sequence of followers to be sampled i.i.d. from some (unknown) distribution over follower types F . When picking context z t , we allow the adversary to have knowledge of F and f 1 , . . . , f t -1 , but not f t . Under this relaxation, our measure of algorithm performance is expected contextual Stackelberg regret, where the expectation is taken over the randomness in the distribution over follower types.

Definition 4.5 (Expected Contextual Stackelberg Regret) . Given a distribution over followers F and a sequence of contexts z 1 , . . . , z T , the leader's expected contextual Stackeleberg regret is E [ R ( T )] := E f 1 ,...,f T ∼F [ ∑ T t =1 u ( z t , π ∗ ( z t ) , b f t ( z t , π ∗ ( z t ))) -u ( z t , x t , b f t ( z t , x t ))] , where π ∗ is the optimal policy given knowledge of z 1 , . . . , z T and F .

Algorithm 1: Learning with stochastic follower types: full feedback

Under this setting, the utility for policy π may be written as E f 1 ,...,f T ∼F [ ∑ T t =1 u ( z t , π ( z t ) , b f t ( z t , π ( z t ))) ] = ∑ T t =1 E f 1 ,...,f t -1 ∼F [ E t [ u ( z t , π ( z t ) , b f t ( z t , π ( z t )))]] . Our algorithm (Algorithm 1) proceeds by estimating the inner expectation E t [ u ( z t , π ( z t ) , b f t ( z t , π ( z t )))] as

and acting greedily with respect to our estimate. Here ̂ p t ( b f t ( z t , π ( z t )) = a f ) is the (estimated) probability that the follower's best-response is a f , given context z t and leader mixed strategy π ( z t ) . As we will see, different instantiations of ̂ p t will lead to different regret rates for Algorithm 1. However, before instantiating Algorithm 1 with a specific estimation procedure, we provide a general result which bounds the regret of Algorithm 1 in terms of the total variation distance between the sequence { ̂ p t } t ∈ [ T ] and the true distribution p .

Theorem 4.6. Let p ( z , x ) := [ p ( b f t ( z , x ) = a f )] a f ∈A f and ̂ p t ( z , x ) := [ ̂ p t ( b f t ( z , x ) = a f )] a f ∈A f . The expected contextual Stackelberg regret (Definition 4.5) of Algorithm 1 satisfies

Theorem 4.6 shows that the regret of Algorithm 1 is proportional to how well it estimates p ( z , x ) over time (as measured by total variation distance), on (1) the sequence of contexts chosen by the adversary and (2) the sequence of mixed strategies played by Algorithm 1 and the (near-)optimal policy π ( E ) . While we instantiate Algorithm 1 in the setting where there are finitely-many follower types and follower actions, Theorem 4.6 opens the door to provide meaningful regret guarantees in settings in which there are infinitely-many follower types and/or follower actions. 6 We now instantiate the estimation procedure in Algorithm 1 in two different ways to get end-to-end regret guarantees. First, the leader can get regret O ( K √ T log T ) by estimating the distribution of follower types directly.

The leader can obtain a complementary regret bound of O ( A f √ T log T ) if they instead estimate the probability that the follower best-responds with action a f ∈ A f , given a particular context z and leader mixed strategy x . 7 In what follows, we use ✶ ( σ ( z , x ) = a f ) ∈ { 0 , 1 } K to refer to the indicator vector whose i -th component is ✶ { σ ( z , x ) ( α ( i ) ) = a f } , i.e. the indicator that a follower of type α ( i ) best-responds to context z and mixed strategy x by playing action a f .

4.2 Stochastic contexts and adversarial follower types

We now consider the setting in which the sequence of contexts are drawn i.i.d. from some unknown distribution P and the follower f t is chosen by an adversary with knowledge of P and z 1 , . . . , z t -1 ,

but not z t . As was the case in Section 4.1, we consider a relaxed notion of regret which compares the performance of the leader to the best policy in expectation, although now the expectation is taken with respect to the distribution over contexts P .

Definition 4.9 (Expected Contextual Stackelberg Regret, II) . Given a distribution over contexts P and a sequence of followers f 1 , . . . , f T , the leader's expected contextual Stackeleberg regret is

where π ∗ is the optimal policy given knowledge of f 1 , . . . , f T and P .

Our key insight is that when the sequence of contexts is generated stochastically, to obtain no-regret it suffices to (1) play a standard, off-the-shelf online learning algorithm (e.g. Hedge) over a finite (albeit exponentially-large) set of policies in order to find one which is approximately optimal and then (2) bound the resulting discretization error.

Lemma 4.10. When the sequence of contexts is determined stochastically, the expected utility of any fixed policy π may be written as

Using Lemma 4.10, we now show that it suffices to play Hedge over a finite set of policies Π in order for the leader to obtain no-regret (Algorithm 2). The key step in our analysis is to show that the discretization error is small for our chosen policy class Π . 8 For a given weight vector ω ∈ R K , let π ( ω ) ( z ) := arg max z ∈E z ∑ K i =1 u ( z , x , b α ( i ) ( z , x )) · ω [ i ] . For a given set of weight vectors Ω , we set Π to be the induced policy class, i.e. Π := { π ( ω ) } ω ∈ Ω .

Theorem 4.11. If Ω = { ω : ω ∈ ∆ K , T · ω [ i ] ∈ N , ∀ i ∈ [ K ] } and η = √ log Π T , then Algorithm 2 obtains expected contextual Stackelberg regret (Definition 4.9) E [ R ( T )] = O ( √ KT log T + K ) .

We conclude by briefly comparing our results with those of the non-contextual Stackelberg game setting of Balcan et al. [5]. In particular, the setting of this subsection may be viewed as a generalization of the setting of Balcan et al. [5] in which the leader and follower utilities at time t also depend on a stochastically-generated context z t . When |Z| = 1 , we recover their setting exactly. Under their non-contextual setting, there is only one set of approximate extreme points E z , and so we write the E = E z . Here it suffices to consider the set of constant 'policies' which always map to one of the (approximate) extreme points in E . Plugging this choice of Π into Algorithm 2, we recover their algorithm (and therefore also their regret rates) exactly.

However, it is also worth noting that more care must be taken to obtain regret guarantees against an adaptive adversary in our setting compared to the non-contextual setting of Balcan et al. [5]. 9 In particular, we need to bound the discretization error due to considering a finite set of policies, but it is without loss of generality to consider a finite set of mixed strategies in the non-contextual setting.

Contexts: non-stochastic, Followers: stochastic

12

baseline

Algorithm 1

10

Algorithm 2

Cumulative Reward

8

6

4

Z

0

0

10

20

30

40

50

Time

(a) Non-stochastic contexts, stochas-

' tic follower types.

20

Cumulative Reward

15

10

20

10

30

40

Contexts: stochastic, Followers: non-stochastic baseline

Algorithm 1

Algorithm 2

50

Time

(b) Stochastic contexts, stochastic follower types.

20

15

10

5

0

20

0

Contexts: stochastic,Followers: stochastic baseline

Algorithm 1

Algorithm 2

10

30

40

50

Time

(c) Stochastic contexts, stochastic follower types.

4.3 Simulations

We empirically evaluate the performance of Algorithm 1 and Algorithm 2 on synthetically-generated data. We consider a setup in which K = 5 , A = A f = 3 , and the context dimension d = 3 . Utility functions are linear in both the context and player actions, and are sampled u.a.r. from [ -1 , 1] 3 × 3 × 3 .

We compare the cumulative reward of our algorithms to each other and the algorithm of Balcan et al. [5] (which does not leverage side information) as a baseline. We simulate non-stochastic context arrivals in Figure 2a by displaying the same context for T/ 4 time-steps in a row. Follower types are chosen u.a.r. from each of the five follower types. In Figure 2b, contexts are generated stochastically by sampling each component u.a.r. from [ -1 , 1] . Followers are chosen non-stochastically by deterministically cycling over the five types. In Figure 2c, both contexts and follower types are chosen stochastically. Specifically, contexts are generated as in Figure 2b and follower types are generated as in Figure 2a.

We find that Algorithm 1 and Algorithm 2 perform similarly across instances, and both significantly out-perform the baseline of Balcan et al. [5]. It would be interesting to find instances for which Algorithm 1 (resp. Algorithm 2) performs poorly whenever followers (resp. contexts) are chosen non-stochastically.

5 Extension to bandit feedback

We have so far assumed that the leader gets to observe the follower's type after each round. However this assumption may not always hold in real-world Stackelberg game settings. For example, in cyber security domains it may be hard to deduce the organization responsible for a failed cyber attack. In wildlife protection, a very successful poacher may never be seen by the park rangers. Instead, the leader may only be able to observe the action the follower takes at each round. Following previous work on learning in non-contextual Stackelberg games, we refer to this type of feedback as bandit feedback. What can we say about the leader's ability to learn under bandit feedback when there is side information?

While our impossibility result of Section 3 immediately applies to this more challenging setting, our algorithms from Section 4 do not. This is because we can no longer compute quantities such as ✶ { f t = α ( i ) } or b f t ( z t , x ) for an arbitrary mixed strategy x from just follower f t 's action alone. We still assume that the follower is one of K different types, although f t is now never revealed to the leader.

We allow ourselves two relaxations when designing learning algorithms which operate under bandit feedback. First, while the leader's utility function may still depend on the context z t , we assume that the follower's utility is a function of the leader's mixed strategy x t alone, i.e. u f ( z , x , a f ) = u f ( x , a f ) for all z ∈ Z . This allows us to drop the dependence on z t from both the follower's best response and the set of approximate extreme points, i.e. b f ( z , x ) becomes b f ( x ) and E z becomes E . Furthermore, our definitions of contextual follower best-response region (Definition 4.1) and contextual best-response region (Definition 4.2) collapse to their non-contextual counterparts. Depending on the application domain, the assumption that only the leader's utility depends on the side non-

Cumulative Reward

information may be reasonable. For instance, while an institution would prefer that a server with less traffic is hacked compared to one with more, a hacker might only care about the information hosted on the server (which may not be related to network traffic patterns). Second, we design algorithms with regret guarantees which only hold against a non-adaptive adversary. 10 Despite these relaxations, the problem of learning under bandit feedback still remains challenging because of the exponentially large size of E . While a natural first step is to estimate p ( b f t ( x ) = a f ) (i.e. the probability that follower at round t best-responds with action a f when the leader plays mixed strategy x ) for all x ∈ E and a f ∈ A f , doing so naively would take exponentially-many rounds, due to the size of E .

Building off of results in the non-contextual setting of Balcan et al. [5], we leverage the fact that the leader's utility for different mixed strategies is not independent. Instead, they are linearly related through the frequency of follower types which take a particular action, given a particular leader mixed strategy. Therefore, it suffices to estimate this linear function (which can be done using as few as K samples) to get an unbiased estimate of p ( b f t ( x ) = a f ) for any x ∈ E and a f ∈ A f . Borrowing from the literature on linear bandits, we use a barycentric spanner [4] to estimate {{ p ( b f t ( x ) = a f ) } a f ∈A f } x ∈E in both partial adversarial settings we consider. A barycentric spanner for compact vector space W is a special basis such that any vector in W may be expressed as a linear combination of elements in the basis, with each linear coefficient being in the range [ -1 , 1] .

In Appendix C.1, we use the property that estimators constructed using barycentric spanners have low variance to show that an explore-then-exploit algorithm achieves O ( K 2 / 3 A 2 / 3 f T 2 / 3 log 1 / 3 T ) expected contextual Stackelberg regret in the setting with stochastic follower types and adversarial contexts. Specifically, our algorithm (Algorithm 3) plays a special set of K mixed strategies N times each, then uses barycentric spanners to estimate { p t ( ✶ ( σ ( z , x ) = a f ) ) } a f ∈A f for all x ∈ X and z ∈ Z , after which Algorithm 3 plays greedily like in Section 4.1.

In Appendix C.2, we use the property that estimators constructed using barycentric spanners are bounded to design a reduction to our algorithm in Section 4.2 which achieves O ( KA 1 / 3 f T 2 / 3 log 1 / 3 T ) expected contextual Stackelberg regret whenever the sequence of contexts is chosen stochastically and the sequence of follower types is chosen by an adversary. Finally, while it may be possible to obtain O ( √ T ) regret without using barycentric spanners, this would come at the cost of a linear dependence on |E| (and therefore an exponential dependence on K and A f ) in regret.

6 Conclusion

We initiate the study of Stackelberg games with side information, which despite the presence of side information in many Stackelberg game settings, has not received attention from the community. We focus on the online setting in which the leader faces a sequence of contexts and follower types. We show that when both sequences are chosen adversarially, no-regret learning is not possible even for highly structured policy classes. When either sequence is chosen stochastically, we obtain algorithms with ˜ O ( √ T ) regret. We also explore an extension to bandit feedback, in which we obtain ˜ O ( T 2 / 3 ) regret in both settings. There are several exciting avenues for future research; we highlight two below.

1. Intermediate forms of adversary. The two relaxations of the fully adversarial setting that we consider, while natural, rule out the leader learning about the follower's type from the context. Although we prove that learning is impossible in the fully adversarial setting, our lower bound does not rule out, e.g. settings where the mapping from contexts to follower types has finite Littlestone dimension. It would be interesting to further explore this direction to pin down when no-regret learning is possible.

2. ˜ O ( T 1 / 2 ) regret under bandit feedback. Bernasconi et al. [6] obtain O ( T 1 / 2 ) regret when learning in non-contextual Stackelberg games under bandit feedback against an adversarially-chosen sequence of follower types via a reduction to adversarial linear bandits. However, applying similar steps to Bernasconi et al. in our setting results in a reduction to a generalization of the (adversarial) contextual bandit problem for which we are not aware of any regret minimizing algorithm. Nevertheless, we view exploring whether ˜ O ( T 1 / 2 ) contextual Stackelberg regret is possible under bandit feedback as a natural and exciting future direction.

References

Bo An, Fernando Ordóñez, Milind Tambe, Eric Shieh, Rong Yang, Craig Baldwin, Joseph DiRenzo III, Kathryn Moretti, Ben Maule, and Garrett Meyer. A deployed quantal responsebased patrol planning system for the us coast guard. Interfaces , 43(5):400-420, 2013.

Bo An, Milind Tambe, and Arunesh Sinha. Stackelberg security games (ssg) basics and application overview. Improving Homeland Security Decisions , page 485, 2017.

Ioannis Anagnostides, Ioannis Panageas, Gabriele Farina, and Tuomas Sandholm. On the convergence of no-regret learning dynamics in time-varying games. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023. URL http://papers.nips.cc/paper_files/paper/2023/hash/ 34f1c2e7ab91b6fa481ad0286a08ad02-Abstract-Conference.html .

Baruch Awerbuch and Robert Kleinberg. Online linear optimization and adaptive routing. J. Comput. Syst. Sci. , 74(1):97-114, 2008. doi: 10.1016/J.JCSS.2007.04.016. URL https: //doi.org/10.1016/j.jcss.2007.04.016 .

Maria-Florina Balcan, Avrim Blum, Nika Haghtalab, and Ariel D. Procaccia. Commitment without regrets: Online learning in stackelberg security games. In Tim Roughgarden, Michal Feldman, and Michael Schwarz, editors, Proceedings of the Sixteenth ACM Conference on Economics and Computation, EC '15, Portland, OR, USA, June 15-19, 2015 , pages 61-78. ACM, 2015. doi: 10.1145/2764468.2764478. URL https://doi.org/10.1145/2764468. 2764478 .

Martino Bernasconi, Matteo Castiglioni, Andrea Celli, Alberto Marchesi, Francesco Trovò, and Nicola Gatti. Optimal rates and efficient algorithms for online bayesian persuasion. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 2329 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 2164-2183. PMLR, 2023. URL https://proceedings.mlr.press/v202/ bernasconi23a.html .

Avrim Blum, Nika Haghtalab, and Ariel D. Procaccia. Learning optimal commitment to overcome insecurity. pages 1826-1834, 2014. URL https://proceedings.neurips.cc/ paper/2014/hash/cc1aa436277138f61cda703991069eaf-Abstract.html .

Matthew Brown, Arunesh Sinha, Aaron Schlenker, and Milind Tambe. One size does not fit all: A game-theoretic approach for dynamically and effectively screening for threats. In Dale Schuurmans and Michael P. Wellman, editors, Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA , pages 425-431. AAAI Press, 2016. doi: 10.1609/AAAI.V30I1.10023. URL https://doi.org/10.1609/aaai. v30i1.10023 .

Constantinos Daskalakis and Vasilis Syrgkanis. Learning in auctions: Regret is hard, envy is easy. Games Econ. Behav. , 134:308-343, 2022. doi: 10.1016/J.GEB.2022.03.001. URL https://doi.org/10.1016/j.geb.2022.03.001 .

Fei Fang, Peter Stone, and Milind Tambe. When security games go green: Designing defender strategies to prevent poaching and illegal fishing. In Qiang Yang and Michael J. Wooldridge, editors, Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015 , pages 2589-2595. AAAI Press, 2015. URL http://ijcai.org/Abstract/15/367 .

Anupam Gupta. 14: Online learning: Experts and bandits. 15850: Advanced Algorithms course notes , 2023.

Nika Haghtalab. Lecture 12: Introduction to online learning 2. CS6781: Theoretical Foundations of Machine Learning course notes , 2020.

Nika Haghtalab, Thodoris Lykouris, Sloan Nietert, and Alexander Wei. Learning in stackelberg games with non-myopic agents. In David M. Pennock, Ilya Segal, and Sven Seuken, editors, EC '22: The 23rd ACM Conference on Economics and Computation, Boulder, CO, USA, July 11 - 15, 2022 , pages 917-918. ACM, 2022. doi: 10.1145/3490486.3538308. URL https://doi.org/10.1145/3490486.3538308 .

Nika Haghtalab, Chara Podimata, and Kunhe Yang. Calibrated stackelberg games: Learning optimal commitments against calibrated agents. 2023. URL http://papers.nips.cc/paper_files/paper/2023/hash/ c23ccf9eedf87e4380e92b75b24955bb-Abstract-Conference.html .

Moritz Hardt, Nimrod Megiddo, Christos H. Papadimitriou, and Mary Wootters. Strategic classification. In Madhu Sudan, editor, Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science, Cambridge, MA, USA, January 14-16, 2016 , pages 111-122. ACM, 2016. doi: 10.1145/2840728.2840730. URL https://doi.org/10.1145/2840728. 2840730 .

Keegan Harris, Ioannis Anagnostides, Gabriele Farina, Mikhail Khodak, Steven Wu, and Tuomas Sandholm. Meta-learning in games. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=uHaWaNhCvZD .

Indiana Intelligence Fusion Center Iifc. 8 signs of terrorism, Jul 2022. URL https://www.in. gov/iifc/8-signs-of-terrorism/ .

Manish Jain, Jason Tsai, James Pita, Christopher Kiekintveld, Shyamsunder Rathi, Milind Tambe, and Fernando Ordóñez. Software assistants for randomized patrol planning for the LAX airport police and the federal air marshal service. Interfaces , 40(4):267-290, 2010. doi: 10.1287/INTE.1100.0505. URL https://doi.org/10.1287/inte.1100.0505 .

Debarun Kar, Thanh H Nguyen, Fei Fang, Matthew Brown, Arunesh Sinha, Milind Tambe, and Albert Xin Jiang. Trends and applications in stackelberg security games. Handbook of dynamic game theory , pages 1-47, 2017.

Niklas T. Lauffer, Mahsa Ghasemi, Abolfazl Hashemi, Yagiz Savas, and Ufuk Topcu. No-regret learning in dynamic stackelberg games. IEEE Trans. Autom. Control. , 69(3):1418-1431, 2024. doi: 10.1109/TAC.2023.3330797. URL https://doi.org/10.1109/TAC.2023.3330797 .

Joshua Letchford, Vincent Conitzer, and Kamesh Munagala. Learning and approximating the optimal strategy to commit to. In Marios Mavronicolas and Vicky G. Papadopoulou, editors, Algorithmic Game Theory, Second International Symposium, SAGT 2009, Paphos, Cyprus, October 18-20, 2009. Proceedings , volume 5814 of Lecture Notes in Computer Science , pages 250-262. Springer, 2009. doi: 10.1007/978-3-642-04645-2\_23. URL https://doi.org/10. 1007/978-3-642-04645-2_23 .

Yuqian Li, Vincent Conitzer, and Dmytro Korzhyk. Catcher-evader games. pages 329-337, 2016. URL http://www.ijcai.org/Abstract/16/054 .

Binghui Peng, Weiran Shen, Pingzhong Tang, and Song Zuo. Learning optimal strategies to commit to. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019 , pages 2149-2156. AAAI Press, 2019. doi: 10.1609/AAAI.V33I01.33012149. URL https://doi.org/10.1609/aaai.v33i01. 33012149 .

Alexander Rakhlin and Karthik Sridharan. BISTRO: an efficient relaxation-based method for contextual bandits. In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 , volume 48 of JMLR Workshop and Conference Proceedings , pages 19771985. JMLR.org, 2016. URL http://proceedings.mlr.press/v48/rakhlin16.html .

Pier Giuseppe Sessa, Ilija Bogunovic, Andreas Krause, and Maryam Kamgarpour. Contextual games: Multi-agent learning with side information. 2020. URL https://proceedings.

neurips.cc/paper/2020/hash/f9afa97535cf7c8789a1c50a2cd83787-Abstract. html .

Arunesh Sinha, Debarun Kar, and Milind Tambe. Learning adversary behavior in security games: APACmodel perspective. pages 214-222, 2016. URL http://dl.acm.org/citation.cfm? id=2936958 .

Arunesh Sinha, Fei Fang, Bo An, Christopher Kiekintveld, and Milind Tambe. Stackelberg security games: Looking beyond a decade of success. In Jérôme Lang, editor, Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden , pages 5494-5501. ijcai.org, 2018. doi: 10.24963/IJCAI.2018/775. URL https://doi.org/10.24963/ijcai.2018/775 .

Vasilis Syrgkanis, Akshay Krishnamurthy, and Robert E. Schapire. Efficient algorithms for adversarial contextual learning. In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 , volume 48 of JMLR Workshop and Conference Proceedings , pages 2159-2168. JMLR.org, 2016. URL http://proceedings.mlr.press/v48/ syrgkanis16.html .

Vasilis Syrgkanis, Haipeng Luo, Akshay Krishnamurthy, and Robert E. Schapire. Improved regret bounds for oracle-based adversarial contextual bandits. pages 3135-3143, 2016. URL https://proceedings.neurips.cc/paper/2016/hash/ dfa92d8f817e5b08fcaafb50d03763cf-Abstract.html .

Heinrich Von Stackelberg. Market structure and equilibrium . Springer Science & Business Media, 1934.

Bernhard von Stengel and Shmuel Zamir. Leadership games with convex strategy sets. Games Econ. Behav. , 69(2):446-457, 2010. doi: 10.1016/J.GEB.2009.11.008. URL https://doi. org/10.1016/j.geb.2009.11.008 .

Jin Zhang and Qian Zhang. Stackelberg game for utility-based cooperative cognitiveradio networks. In Edward W. Knightly, Carla-Fabiana Chiasserini, and Xiaojun Lin, editors, Proceedings of the 10th ACM Interational Symposium on Mobile Ad Hoc Networking and Computing, MobiHoc 2009, New Orleans, LA, USA, May 18-21, 2009 , pages 23-32. ACM, 2009. doi: 10.1145/1530748.1530753. URL https://doi.org/10.1145/1530748.1530753 .

Mengxiao Zhang, Peng Zhao, Haipeng Luo, and Zhi-Hua Zhou. No-regret learning in timevarying zero-sum games. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 26772-26808. PMLR, 2022. URL https://proceedings.mlr. press/v162/zhang22an.html .

A Appendix for Section 3: On the impossibility of fully adversarial no-regret learning

Theorem 3.2. If an adversary can choose both the sequence of contexts z 1 , . . . , z T and the sequence of followers f 1 , . . . , f T , no algorithm can achieve better than Ω( T ) contextual Stackelberg regret in expectation over the internal randomness of the algorithm, even when π ∗ is restricted to come from the set of linear thresholding functions.

Proof. We proceed via proof by contradiction. Assume that there exists an algorithm ALG which achieves o ( T ) contextual Stackelberg regret against an adversarially-chosen sequence of contexts and follower types. Note that at every time-step, ALG takes as input a context z t and produces a mixed strategy x t .

We now describe the family of contextual Stackelberg game instances we reduce to. Consider the setting in which there are two follower types ( α (1) and α (2) ) and two leader/follower actions ( A = A f = { a 1 , a 2 } ). Suppose that the context space is of the form Z = [0 , 1] , and that regardless of the realized context or leader mixed strategy, the best-response of follower type α (1) is to play action a 1 ( b α (1) ( z , x ) = a 1 , ∀ z ∈ Z , x ∈ X ) and the best-response of follower type α (2) is to play action a 2 ( b α (2) ( z , x ) = a 2 , ∀ z ∈ Z , x ∈ X ). Since the follower's best-response does not depend on the leader's mixed strategy or the context, we use the shorthand b f t := b f t ( z t , x t ) . Finally, suppose that the leader's utility function is given by u ( z , a l , a f ) = ✶ { a l = a f } . Note that this is a special case of our general setting (described in Section 2). The reduction from online linear thresholding proceeds as follows. In each round t ∈ [ T ] ,

Given a point ω t ∈ [0 , 1] , we give the context z t = ω t as input to ALG .

In return, we receive mixed strategy x t ∈ ∆( { a 1 , a 2 } ) from ALG . We set g t = x t [1] . 11

Play according to g t , and receive label y t and utility u OLT ( ω t , g t ) from Nature. Give follower type

and utility u ( z t , x t , b f t ) = x t [1] · ✶ { b f t = a 1 } + x t [2] · ✶ { b f t = a 2 } as input to ALG .

Observe that under this reduction,

since if z > s , f t = α (1) and otherwise f t = α (0) . By playing according to π ∗ , we can ensure that u ( z t , π ∗ ( z t ) , b f t ) = 1 for all t ∈ [ T ] . π ∗ must then be optimal, because 1 is the largest possible per-round utility that the leader can receive.

Since ALG achieves no-contextual-Stackelberg-regret, we know by Definition 2.3 that

To conclude, it suffices to show that R OLT ( T ) = o ( T ) using Equation (2) and Equation (3). Applying Equation (2), we see that

By construction, ✶ { b f t = a 1 } = ✶ { y t = 1 } , ✶ { b f t = a 2 } = ✶ { y t = -1 } , x t [1] = g t , and x t [2] = 1 -g t . Substituting this into Equation (4), we see that

By Equation (3) and Equation (5), we can conclude that R OLT ( T ) = o ( T ) . However, this is a contradiction since no no-regret learning algorithm exists for the online linear thresholding problem by Lemma 3.1. Therefore it must not be possible to achieve no-contextual-Stackelberg-regret whenever the sequence of contexts and follower types is chosen by an adversary.

B Appendix for Section 4: Limiting the power of the adversary

Lemma 4.4. For any sequence of followers f 1 , . . . f T and any leader policy π , there exists a policy π ( E ) : Z → ∪ z ∈Z E z that, when given context z , plays a mixed strategy in E z and guarantees that ∑ T t =1 u ( z t , π ( z t ) , b f t ( z t , π ( z t ))) -u ( z t , π ( E ) ( z t ) , b f t ( z t , π ( E ) ( z t ))) ≤ 1 . Moreover, the same result holds in expectation over any distribution over follower types F .

Proof. Observe that for any z ,

for every possible setting of a (1) , . . . , a ( K ) , and then taking the maximum of all feasible solutions. Since Equation (6) is an optimization over contextual best-response region X z ( a (1) , . . . , a ( K ) ) and all contextual best-response regions are convex polytopes, π ∗ ( z ) will be an extreme point of some contextual best-response region, although it may not be attained. Overloading notation, let X z ( π ∗ ( z )) denote the contextual best-response region corresponding to π ∗ ( z ) , i.e., π ∗ ( z ) ∈ X z ( π ∗ ( z )) . Since for a fixed context z ∈ Z the leader's utility is a linear function of x over the convex polytope X z ( π ∗ ( z )) , there exists a point x ( z ) ∈ cl ( X z ( π ∗ ( z ))) such that

Let x ′ ( z ) denote the corresponding point in E z such that ∥ x ′ ( z ) -x ( z ) ∥ 1 ≤ δ . (Such a point will always exist by Definition 4.3.) Since u ∈ [0 , 1] and is linear in x for a fixed context and follower best-response,

Summing over all unique z encountered by the algorithm over T time-steps, we obtain the desired result for the policy π ( E ) which plays mixed strategy π ( E ) ( z ) = x ′ ( z ) when given context z . Finally, observe that the same line of reasoning holds whenever we are interested in the optimal policy in expectation with respect to some distribution F over followers , as is the case in, e.g. Section 4.1 (with ∑ T t =1 ✶ { z t = z , f t = α ( i ) } replaced with P ( f = α ( i ) ) ).

B.1 Section 4.1: Stochastic follower types and adversarial contexts

Theorem 4.6. Let p ( z , x ) := [ p ( b f t ( z , x ) = a f )] a f ∈A f and ̂ p t ( z , x ) := [ ̂ p t ( b f t ( z , x ) = a f )] a f ∈A f . The expected contextual Stackelberg regret (Definition 4.5) of Algorithm 1 satisfies

Proof of Theorem 4.6. For any z ∈ Z and t ∈ [ T ] ,

Since utilities are bounded in [0 , 1] and the expectations E t and ̂ E t are taken with respect to p and ̂ p t respectively, we can upper-bound ̂ E t [ u ( z , π t ( z ) , b f ( z , π t ( z )))] -E t [ u ( z , π t ( z ) , b f ( z , π t ( z )))] by

Putting everything together, we get that

We now use this fact to bound the expected regret. By Lemma 4.4,

By repeating the same steps as above, we can upper-bound

Corollary 4.7. If ̂ p t = { ̂ p t ( f t = α ( i ) ) } i ∈ [ K ] , ̂ p t +1 ( f = α ( i ) ) = 1 t ∑ t τ =1 ✶ { f τ = α ( i ) } , and ̂ p 1 ( f = α ( i ) ) = 1 K for i ∈ [ K ] , then the regret of Algorithm 1 satisfies E [ R ( T )] = O ( K √ T log( T )) .

Proof. For t ≥ 2 ,

simultaneously for all t ∈ [ T ] and i ∈ [ K ] , with probability at least 1 -1 T 2 . Dropping the dependence of p , ̂ p t on z and x , we can conclude that

(since K ≤ T ), and so

Corollary 4.8. If ̂ p t ( z , x ) = { ̂ p t ( ✶ ( σ ( z , x ) = a f ) ) } a f ∈A f , ̂ p t +1 ( ✶ ( σ ( z , x ) = a ) ) = 1 t ∑ t τ =1 ✶ { b f τ ( z , x ) = a } , and ̂ p 1 ( ✶ ( σ ( z , x ) = a ) ) = 1 A f for a f ∈ A f , then the regret of Algorithm 1 satisfies E [ R ( T )] = O ( A f √ T log( T )) .

Proof. For t ≥ 2 ,

for any z ∈ Z , x ∈ X . By Hoeffding's inequality,

simultaneously for all t ∈ [ T ] and i ∈ [ K ] , with probability at least 1 -1 T 2 . Using this fact, we can conclude that

(since K ≤ T ) and

B.2 Section 4.2: Stochastic contexts and adversarial follower types

The following regret guarantee for Hedge is a well-known result. (See, e.g. Gupta [11].)

Lemma B.1. Hedge enjoys expected regret rate O ( T log n ) when there are n actions, the learning rate is chosen to be η = √ log n T , and the sequence of utilities for each arm are chosen by an adversary.

√

Lemma 4.10. When the sequence of contexts is determined stochastically, the expected utility of any fixed policy π may be written as

Proof. For any fixed policy π ,

where the second line uses the fact that f t cannot depend on z t , and the result follows from the fact that z 1 , . . . , z T are i.i.d.

Theorem 4.11. If Ω = { ω : ω ∈ ∆ K , T · ω [ i ] ∈ N , ∀ i ∈ [ K ] } and η = √ log Π T , then Algorithm 2 obtains expected contextual Stackelberg regret (Definition 4.9) E [ R ( T )] = O ( √ KT log T + K ) .

Let π ( ω ∗ ) denote the optimal-in-hindsight policy in Π .

To conclude, it suffices to bound the discretization error, as

which follows from applying the standard regret guarantee of Hedge (Lemma B.1 in the Appendix). Applying Lemma 4.10,

where the inequality follows from adding and subtracting ∑ K i =1 E z [ u ( z , π ( E ) ( z ) , b α ( i ) ( z , π ( E ) ( z )))] · T · ω ∗ [ i ] and ∑ K i =1 E z [ u ( z , π ( ω ∗ ) ( z ) , b α ( i ) ( z , π ( ω ∗ ) ( z )))] · T · ω ∗ [ i ] . Finally, we can upper-bound the discretization error by

by using the fact that the sender's utility is bounded in [0 , 1] . Piecing everything together and observing that | Π | ≤ T K gives us the desired regret guarantee.

C Appendix for Section 5: Extension to bandit feedback

C.1 Stochastic follower types and adversarial contexts

Recall from Section 4.1 that ✶ ( σ ( x ) = a f ) ∈ { 0 , 1 } K is the indicator vector whose i -th component is ✶ { σ ( x ) ( α ( i ) ) = a f } , i.e. the indicator that a follower of type α ( i ) best-responds to mixed strategy x by playing action a f . For any fixed policy π , we can write the sender's expected utility in round t as where p ( b f ( π ( z )) = a f ) := E f ∼F [ ✶ { b f ( z ) = a f } ] , the first line follows from the assumption of a non-adaptive adversary, the second line follows from the fact that f 1 , . . . , f T are drawn i.i.d., and

✶ Given this reformulation, a natural approach is to estimate p ( ✶ ( σ ( π ( z t )) = a f ) ) as ̂ p ( ✶ ( σ ( π ( z t )) = a f ) ) and act greedily with respect to our estimate, like we did in Section 4.1. To do so, we define the set W := { ✶ ( σ = a f ) | ∀ a f ∈ A f , σ ∈ Σ } and estimate p ( b ) for every element b in the barycentric spanner B := { b (1) , . . . , b ( K ) } of W . 12

where P ( f = α ( i ) ) is the probability that follower f is of type α ( i ) . Note that p ( ✶ ( σ ( π ( z t )) = a f ) ) (and therefore E f t [ u ( z t , π ( z t ) , b f t ( π ( z t )))] ) is linear in ( σ ( π ( z t )) = a f ) .

We estimate p ( b ) as follows: For every b ∈ B , there must be a mixed strategy x ( b ) and follower action a ( b ) such that b = ✶ ( σ x ( b ) = a ( b ) ) . Therefore, if the leader plays mixed strategy x ( b ) N times, we set ̂ p ( b ) = 1 N ∑ t ∈ [ N ] ✶ { b f t ( x ( b ) ) = a ( b ) } . Given estimates { ̂ p ( b ) } b ∈B , we can estimate p ( ✶ ( σ ( x ) = a f ) ) for any x ∈ E and a f ∈ A f as where λ i ( ✶ ( σ ( x ) = a f ) ) ∈ [ -1 , 1] for i ∈ [ K ] are the coefficients from the barycentric spanner. 13 Note that this is an unbiased estimator, due to the fact that p ( ( σ ( x ) = a f ) ) is a linear function.

✶ Algorithm 3 plays each mixed strategy x ( b ) for b ∈ B N > 0 times in order to obtain an estimate of each p ( b ) . 14 It then uses these estimates to construct estimates for all ✶ ( σ ( x ) = a f ) (and therefore also E f [ u ( z , x , b f ( x ))] for all x ∈ E and z ∈ Z ). Finally, in the remaining rounds Algorithm 3 acts greedily with respect to its estimate, much like in Algorithm 1.

Theorem C.1. If N = O ( A 2 / 3 f T 2 / 3 log 1 / 3 ( T ) K 1 / 3 ) , then the expected contextual Stackelberg regret of Algorithm 3 (Definition 4.5) satisfies

Proof Sketch. The key step in our analysis is to show that for any best-response function σ ∈ Σ and follower action a f ∈ A f , Var( ̂ p ( ✶ ( σ = a f ) )) ≤ K N (Lemma C.3). Using this fact, we can bound the cumulative total variation distance between ̂ p ( ✶ ( σ ( x t ) = a f ) and p ( ✶ ( σ ( x t ) = a f )) for any sequence of mixed strategies and follower actions in the 'exploit' phase (Lemma C.5). The rest of the analysis follows similarly to the proof of Corollary 4.7.

C.2 Stochastic contexts and adversarial follower types

We now turn our attention to learning under bandit feedback when the sequence of contexts is chosen stochastically and the choice of follower type is adversarial. While we still use barycentric spanners to estimate {{ ̂ p ( ✶ ( σ ( x ) = a f ) ) } a f ∈A f } x ∈E , we can no longer do all of our exploration 'up front' like in Appendix C.1 because the follower types are now adversarially chosen. Instead, we follow the technique of Balcan et al. [5] and split the time horizon into Z consecutive, evenly-sized blocks. In block B τ , we pick a random time-step to estimate p τ ( ✶ ( σ ( x ( b ) ) = a ( b ) ) ) , i.e. the probability that a follower in block B τ best-responds to mixed strategy x ( b ) by playing action a ( b ) , for every element in our barycentric spanner B . If whenever the leader plays x ( b ) the follower best-responds with action a ( b ) , we set ̂ p τ ( ✶ ( σ ( x ( b ) ) = a ( b ) ) ) = 1 . Otherwise we set ̂ p τ ( ✶ ( σ ( x ( b ) ) = a ( b ) ) ) = 0 . Since the time-step in which we play x ( b ) is chosen uniformly from all time-steps in B τ , ̂ p τ ( ✶ ( σ ( x ( b ) ) = a ( b ) ) ) is an unbiased estimate of p τ ( ✶ ( σ ( x ( b ) ) = a ( b ) ) ) . While ̂ p τ ( ✶ ( σ ( x ( b ) ) = a ( b ) ) ) no longer has low variance since it must be recomputed separately for every block B τ , it is still bounded. Therefore, we can use our estimates { ̂ p τ ( ✶ ( σ ( x ( b ) ) = a ( b ) ) ) } b ∈B , along with the corresponding linear coefficients from the barycentric spanner, to get a bounded (and unbiased) estimate for every p ( ( σ ( x ) = a f ) ) .

✶ Once we have estimates for {{ p τ ( ✶ ( σ ( x ) = a f ) ) } a f ∈A f } x ∈E , we proceed via a reduction to Algorithm 2. In particular, in every block B τ we use our estimates {{ ̂ p τ ( ✶ ( σ ( x ) = a f ) ) } a f ∈A f } x ∈E to construct an (unbiased and bounded) estimate of the average utility for all policies in our finite policy class Π during block B τ . At the end of each block, we feed this estimate into the Hedge update step, which updates the weights of all policies for the next block. Finally, when we are not exploring (i.e. estimating p τ ( ✶ ( σ ( x ( b ) ) = a ( b ) ) ) for some b ∈ B ), we sample the leader's policy according to the distribution over policies given by Hedge from the previous block. This process is summarized in Algorithm 4.

Theorem C.2. If N = O ( T 2 / 3 A 1 / 3 f log 1 / 3 T ) , then Algorithm 4 obtains expected contextual Stackelberg regret (Definition 4.9)

Proof Sketch. The analysis proceeds similarly to the analysis of Theorem 6.1 in Balcan et al. [5]. We highlight the key differences here. The first key difference is that while Balcan et al. [5] play Hedge over a finite set of leader strategies, we play Hedge over a finite set of leader policies , each of which map to one of finitely-many leader strategies for a given context. Second, unlike in Balcan et al. [5] it is not sufficient to only estimate {{ p τ ( ✶ ( σ ( x ) = a f ) ) } a f ∈A f } x ∈E to obtain an unbiased estimate of the utility of each policy in Π in each time block-we must also specify a context (or set of contexts) to use in our estimator. We show that it suffices to select a context uniformly at random from the contexts { z t } t ∈ B τ encountered in the block.

C.3 Proofs for Appendix C.1: Stochastic follower types and adversarial contexts

Lemma C.3. For any σ f ∈ Σ and a f ∈ A f , Var( ̂ p ( ✶ ( σ = a f ) )) ≤ K N .

Algorithm 3: Learning with stochastic follower types: bandit feedback

Play mixed strategy x ( b ( i ) ) , observe best-response a f ( i -1) · N + τ end

end

Proof.

̸

̸

Observe that since (1) the follower in each round is drawn independently from F and (2) the rounds used to compute ̂ p ( b ( i ) ) do not overlap with the rounds used to compute ̂ p ( b ( j ) ) for j = i , ̂ p ( b ( i ) ) and ̂ p ( b ( j ) ) are independent random variables for j = i . Therefore

̸

̸

̸

We now turn our focus to E [ ̂ p 2 ( b ( j ) )] . Observe that

Plugging this into our expression for Var( ̂ p ( ✶ ( σ = a f ) )) , we see that

̸

where the last line follows from the fact that λ j ( ✶ ( σ = a f ) ) ∈ [ -1 , 1] and p ( b ( j ) ) ∈ [0 , 1] .

Proof. By the definition of ̂ π ,

the desired result may be obtained by rearranging terms.

Lemma C.5. For any sequence of mixed strategies x NK +1 , . . . , x T ,

with probability at least 1 -1 T .

Proof. By Lemma C.3 and a Hoeffding bound, we have that

with probability at least 1 -δ , for any particular ( σ, a f ) pair. Taking a union bound over the randomness in estimating p ( b (1) ) , . . . , p ( b ( K ) ) , we see that

with probability at least 1 -δ , simultaneously for all ( σ, a f ) . The desired result follows by summing over T and A f , and setting δ = 1 T .

Theorem C.1. If N = O ( A 2 / 3 f T 2 / 3 log 1 / 3 ( T ) K 1 / 3 ) , then the expected contextual Stackelberg regret of Algorithm 3 (Definition 4.5) satisfies

Proof.

where the last line follows from Lemma C.5. The desired result follows by the setting of N .

C.4 Proofs for Appendix C.2: Stochastic contexts and adversarial follower types

Definition C.6. Let u τ ( π ) := ∑ a f ∈A f u ( z τ , π ( z τ ) , a f ) · p τ ( ✶ ( σ ( π ( z τ )) = a f ) ) and ̂ u τ ( π ) := ∑ a f ∈A f u ( z τ , π ( z τ ) , a f ) · ∑ K j =1 λ j ( ✶ ( σ ( π ( z τ )) = a f ) ) · ̂ p τ ( b ( j ) ) , where z τ ∼ Unif { z t : t ∈ B τ } , B = { b (1) , . . . , b ( K ) } is the Barycentric spanner for W , and ̂ p ( b ) = 1 if b f t ( b ) ( x ( b ) ) = a ( b ) f and ̂ p ( b ) = 0 otherwise.

Lemma C.7. For any fixed policy π , E { z τ } t ∈ Bτ E [ ̂ u τ ( π )] = E z τ ∼P [ u τ ( π )] = E z τ ∼P [ ∑ a f ∈A f u ( z τ , π ( z τ ) , a f ) · p τ ( ✶ ( σ ( π ( z τ )) = a f ) )] , where the second expectation is taken over the randomness in selecting the explore time-steps and in drawing z τ ∼ Unif { z t : t ∈ B τ } . Moreover, ̂ u τ ( π ) ∈ [ -KA f , KA f ] .

Algorithm 4: Learning with stochastic contexts: bandit feedback

Choose random perturbation over B and explore time-steps in B τ uniformly at random Choose a time-step in B τ uniformly at random whose context will be used as z τ

for

t

∈

If

t

B

τ

do is an explore time-step, play the corresponding mixed strategy

a

f

(

b

t

t

=

a

Otherwise (

t

, set

p

τ

(

b

t

) = 1

. Otherwise, set is an exploit round), sample

π

t

p

∼

end

For each policy π ( ω ) ∈ Π , compute

Proof.

The following lemma is analogous to Equation (1) in Balcan et al. [5].

Lemma C.8.

where R N,κ is an upper-bound on the regret of (full-information) Hedge which takes as input a sequence of N losses/utilities which are bounded in [ -κ, κ ] and are parameterized by z 1 , . . . , z N .

)

τ

(

b

p

t

,

t

) = 0

a

l,t

.

∼

̂

̂

π

t

(

z

t

)

.

x

(

b

t

)

in

B

. If

Proof.

where the first line uses Lemma C.7 and the fact that z 1 , . . . , z T ∼ P are i.i.d., and R N,κ is the regret of Hedge after N time-steps when losses are bounded in [ -κ, κ ] . Rearranging terms and using the fact that the expected regret of Hedge after N time-steps is at most √ Nκ log | Π | gets us the desired result.

Theorem C.2. If N = O ( T 2 / 3 A 1 / 3 f log 1 / 3 T ) , then Algorithm 4 obtains expected contextual Stackelberg regret (Definition 4.9)

Proof.

where the second line follows from Lemma 4.4, the third from splitting the time horizon into blocks, the fourth from loss due to exploration, the fifth due to reformulating the reward as a function of different follower actions, the sixth due to linearity of expectation, and the seventh line follows from the fact that (1) π τ is independent of z t for all t ∈ B τ and (2) z 1 , . . . , z T are independent.

where the first line comes from multiplying and dividing by | B τ | , the second line comes from the definition of p τ , the third from the definition of u τ , the fourth follows from linearity of expectation and the fact that z 1 , . . . , z T are i.i.d., the fifth follows from applying Lemma C.8, and the sixth line follows from the definition of B and the fact that | Π | ≤ N K . Setting N gets us the final result.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: As our work is theoretical, there are no positive or negative societal impacts. However, we hope that our work will one day lead to the design and implementation of better algorithms for solving Stackelberg security games in practice, which would have a positive societal impact in domains such as airport security and the protection of wildlife against poaching.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper examines online learning within Stackelberg games, incorporating additional contextual information. Specifically, at each round $t$, both the follower and the leader observe a shared context $z_t$, which impacts their respective utilities. The leader, who is also the learner in this online learning framework, then selects a mixed strategy $x_t$​ from its set of possible actions. Following this, the follower observes the leader's actions and chooses a best-response action based on both the leader's action and the given context. The learner's objective is to choose a sequence of probability distributions that maximizes overall utility. The authors evaluate the performance of no-regret algorithms using the concept of *\"policy regret,\"* aiming to achieve sublinear regret relative to the optimal *\"fixed contextual policy\"* that assigns a probability distribution to each context.\n\nThe authors present several positive and negative results:\n\n1. When the context and the type of players are adversarially selected, there is no $o(T)$-regret algorithm.\n\n2. When the types of players arrive according to a fixed probability distribution but the contexts are adversarially selected, the authors provide an $O(T^{1/2})$-regret algorithm.\n\n3. When the contexts arrive according to a fixed probability distribution but the types of followers arrive according to a fixed probability distribution, the authors provide an $O(T^{1/2})$-regret algorithm.\n\n4. Finally, the authors extend their results to the bandit case by providing O(T2/3)O(T^{2/3})O(T2/3)-regret algorithms.\n\nSTRENGTHS:\nI find the problem of online learning in Stackelberg games, where both the follower and the leader observe contextual side information, to be well-motivated and of significant interest to the game theory and learning community. The authors have thoroughly examined several aspects of the problem. Despite the negative results in the case of adversarially selected contexts and types, they provide positive results in the stochastic case. Furthermore, the results appear solid and present considerable technical interest.\n\nWEAKNESSES:\nThe regret bounds for the bandit case are not tight. However I believe the paper provides interesting first results for an interesting problem.\n\nQUESTIONS:\nWhat are the main challenges of extending your results in infinite action games with convex structure?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper studies an online Stackelberg game, where the leader plays with a different follower type in a different context in each time step. The paper takes an online learning approach to solving this problem. It first that given that the context space is infinite, it is not possible to achieve sublinear regret via a reduction to an online linear threasholding problem. Therefore, the authors consider relaxed cases, where either the context or the follower type is chosen stochastically in each round. In both cases, the authors present algorithms that guanrantee sublinear regret, both for the full feedback and bandit feedback settings.

STRENGTHS:
(+) The model is well motivated.

(+) The paper is clear and well presented. All analyses look sound and rigorous.

(+) The results presented are very complete, covering all cases of the model and both positive and corresponding negative results.

WEAKNESSES:
(-) The main impossibility result appears to reply on the fact that the context space is infinite and the model is non-linear w.r.t. the context. 

(-) The analysis of the relaxed cases (stochastic follower type, or stochastic context) looks fairly standard and the results are expected. So overall the paper is more like an application of existing techniques to a problem motivated by a new context.

QUESTIONS:
Would the impossiblity result change if the context space is finite, or if the players' utility functions are linear w.r.t. the context.

LIMITATIONS:
The authors didn't seem to have addressed this explicitly, but the work is theoretical anyway, so this is minor.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a study of Stackelberg games with contextual side information, impacting their strategies in a game theoretic setting. The authors introduce a framework for analyzing online Stackelberg games, where a leader faces a sequence of followers, and both or either sequences—contexts and follower types—can be adversarially chosen. The paper contributes by showing the limitations of traditional non-contextual strategies and offering new algorithms that can handle stochastic elements in either the context or the follower sequences.\n\nSTRENGTHS:\n- The paper addresses a novel aspect of Stackelberg games by incorporating side information, which is realistically present in many practical applications but often ignored in theoretical models.\n- The paper is technically sound with rigorous proofs and a clear exposition of both theoretical and practical implications of the findings.\n- The paper is well-written and organized. Concepts are introduced systematically, and the flow from problem statement to results is logical and easy to follow.\n\nWEAKNESSES:\n- The paper could improve by providing numerical experiments or case studies that demonstrate the efficacy of the proposed algorithms.\n- The practical implications are clear for certain fields, but the paper could further elaborate on how these findings might influence other areas of research or industry applications.\n\nQUESTIONS:\n- The paper would definitely benefit from the addition of numerical experiments or case studies.\n- Could the authors provide more clarity on how the adversarial model for context selection was validated? Are there empirical data or specific scenarios where this model reflects real-world conditions?\n- How would the author compare the results with existing methods for handling contextual information in game theory, such as contextual bandits or online learning with expert advice? Can the authors comment on how their approach compares to existing methods in terms of computational efficiency and practical deployability in real systems?\n- How does the proposed algorithm perform if the model of side information is mis-specified? For instance, if the actual distribution of contexts or follower types deviates significantly from the stochastic model assumed, what is the impact on the regret bounds?\n- Several theoretical assumptions are crucial and well presented in the paper. How sensitive are the main results to these assumptions? If some of these assumptions might not hold, how would this affect the applicability of the results?\n- The paper could benefit from a deeper discussion on the limitations regarding the scalability of the algorithms when the number of contexts or follower types is large. What are the computational implications?\n\nLIMITATIONS:\nThe authors have not explicitly addressed limitations or potential negative societal impacts of their work. A clearer identification of potential limitations, such as dependency on the accurate modeling of side information and follower behavior, would strengthen the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper studied the regret minimization in Stackelberg games with side information which consider the additional information available to each player. The paper found that achieving no-regret learning is impossible in fully adversarial settings. However, it demonstrated that no-regret learning is achievable in scenarios where either the sequence of contexts or followers is chosen stochastically.

STRENGTHS:
The idea of using side-information to learn to play Stackelberg games is interesting.
Compared to previous work on learning in Stackelberg games, this paper takes into consideration the additional information available to both the leader and followers at each round, which is more complicated and realistic.
The paper provide an impossibility result and also identifies a setting where no-regret learning is possible. An algorithm is also provided to achieve no-regret.

WEAKNESSES:
This paper lacks experimental results to verify the theoretical analysis.

QUESTIONS:
I do not have any questions for the authors.

LIMITATIONS:
The limitations have been addressed properly.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-15