Sample-Efficient Learning of Stackelberg Equilibria in General-Sum Games

1 Introduction . Real-world problems such as economic design and policy making can often be modeled as multiagent games that involves two levels of thinking : The policy maker—as a player in this game— needs to reason about the other player ’ s optimal behaviors given her decision , in order to inform her own optimal decision making . Consider for example the optimal taxation problem in the AI Economist [ 53 ] , a game modeling a real-world social-economic system involving a leader ( e.g . the government ) and a group of interacting followers ( e.g . citizens ) . The leader sets a tax rate which determines an economics-like game for the followers ; the followers then play in this game with the objective to maximize their own reward ( such as individual productivity ) . However , the goal of the leader is to maximize her own reward ( such as overall equality ) which is in general different from the followers ’ rewards , making these games general-sum [ 38 ] . Such two-level thinking appears broadly in other applications as well such as in automated mechanism design [ 11 , 12 ] , optimal auctions [ 10 , 14 ] , security games [ 43 ] , reward shaping [ 25 ] , and so on . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Another key feature in such games is that the players are asymmetric , and they act in turns : the leader first plays , then the follower sees the leader ’ s action and then adapts to it . This makes symmetric solution concepts such as Nash equilibrium [ 30 ] not always appropriate . A more natural solution concept for these games is the Stackelberg equilibrium : the leader ’ s optimal strategy , assuming the followers play their best response to the leader [ 42 , 13 ] . The Stackelberg equlibrium is often the desired solution concept in many of the aforementioned applications . Furthermore , it is of compelling interest to understand the learning of Stackelberg equilibria from samples , as it is often the case that we can only learn about the game through interactively deploying policies and observing the ( noisy ) feedback from the game [ 53 ] . This may be the case even when the game rules are perfectly known but not yet represented in a desired form , as argued in the line of work on empirical game theory [ 50 , 48 , 20 ] . Despite the motivations , theoretical studies of learning Stackelberg equilibria in general-sum games remain open , in particular when we can only learn from noisy samples of the rewards . A line of work provides guarantees for finding Stackelberg equilibria in general-sum games , but restricts attention to either the full observation setting ( so that the exact game is observable ) or with an exact best-response oracle [ 13 , 27 , 47 , 34 ] . These results lay out a foundation for analyzing the Stackelberg equilibrium , but do not generalize to the bandit feedback setting in which the game can only be learned from random samples of the rewards . Another line of work considers the sample complexity of learning the Nash equilibrium in Markov games [ 35 , 4 , 5 , 28 , 51 , 52 ] , which also do not imply algorithms for finding the Stackelberg equilibrium in these games as the Nash is in general different from the Stackelberg equilibrium in general-sum games . In this work , we study the sample complexity of learning Stackelberg equilibrium in general-sum games . We focus on general-sum games with two players ( one leader and one follower ) , in which we wish to learn an approximate Stackelberg equilbrium for the leader from random samples . Our contributions can be summarized as follows . • As a warm-up , we consider bandit games in which the two players play an action in turns and observe their own rewards . We identify a fundamental gap between the exact Stackelberg value and its estimated version from finite samples , which can not be closed information-theoretically regardless of the algorithm ( Section 3.1 ) . We then propose a rigorous definition gapε for this gap , and show that it is possible to sample-efficiently learn the ( gapε + ε ) -approximate Stackelberg equilibrium with Õ ( AB/ε2 ) samples , where A , B are the number of actions for the two players . We further show a matching lower bound Ω ( AB/ε2 ) ( Section 3.2 ) . We also establish similar results for learning Stackelberg in simultaneous matrix games ( Appendix B ) . • We consider bandit-RL games in which the leader ’ s action determines an episodic Markov Decision Process ( MDP ) for the follower . We show that a ( gapε + ε ) approximate Stackelberg equilibrium for the leader can be found in Õ ( H5S2AB/ε2 ) episodes of play , where H , S are the horizon length and number of states for the follower ’ s MDP , and A , B are the number of actions for the two players ( Section 4 ) . Our algorithm utilizes recently developed reward-free reinforcement learning techniques to enable fast exploration for the follower within the MDPs . • Finally , we consider linear bandit games in which the action spaces for the two players can be arbitrarily large , but the reward is a linear function of a d-dimensional feature representation of the actions . We design an algorithm that achieves Õ ( d2/ε2 ) sample complexity upper bound for linear bandit games ( Section 5 ) . This only depends polynomially on the feature dimension instead of the size of the action spaces , and has at most an Õ ( d ) gap from the lower bound . 1.1 Related work . Since the seminal paper of [ 46 ] , notions of equilibria in games and their algorithmic computation have received wide attention [ see , e.g. , 9 , 41 ] . For the scope of this paper , this section focuses on reviewing results that related to learning Stackelberg equilibria . Learning Stackelberg equilibria in zero-sum games The first category of results study twoplayer zero-sum games , where the rewards of the two players sum to zero . Most results along this line focus on the bilinear or convex-concave setting [ see , e.g. , 21 , 32 , 31 , 37 , 16 ] , where the Stackelberg equilibrium coincide with the Nash equilibrium due to Von Neumann ’ s minimax theorem [ 46 ] . Results for learning Stackelberg equilibria beyond convex-concave setting are much more recent , with Rafique et al . [ 36 ] , Nouiehed et al . [ 33 ] considering the nonconvex-concave setting , and Fiez et al . [ 15 ] , Jin et al . [ 19 ] , Marchesi et al . [ 29 ] considering the nonconvex-nonconcave setting . Marchesi et al . [ 29 ] provide sample complexity results for learning Stackelberg with infinite strategy spaces , using discretization techniques that may scale exponentially in the dimension without further assumptions on the problem structure . We remark that a crucial property of zero-sum games is that any two strategies giving similar rewards for the follower will also give similar rewards for the leader . This is no longer true in general-sum games , and prevents most statistical results for learning Stackelberg equilibria in the zero-sum setting from generalizing to the general-sum setting . Learning Stackelberg equilibria in general-sum games The computational complexity of finding Stackelberg equilibria in games with simultaneous play ( “ computing optimal strategy to commit to ” ) is studied in [ 13 , 26 , 47 , 22 , 1 , 7 ] . These results assume full observation of the payoff function , and show that several versions of matrix games and extensive-form ( multi-step ) games admit polynomial time algorithms . Vasal [ 45 ] designs computationally efficient algorithms for computing Stackelberg in certain “ conditionally independent controlled ” Markov games . Another line of work considers learning Stackelberg with a “ best response oracle ” [ 27 , 6 , 34 ] , that returns the follower ’ s exact best response strategy when a leader ’ s strategy is queried . This oracle and the noisy reward oracle we assume are in general incomparable ( can not simulate each other regardless of the number of queries ) , and thus our sample complexity results do not imply each other . The recent work of Sessa et al . [ 39 ] proposes the StackelUCB algorithm to sample-efficiently learn a Stackelberg game where the opponent ’ s response function has a linear structure in a certain kernel space , and the observation noise is added in the action space instead of the reward ( thus a different noisy feedback model from ours ) . Lastly , Fiez et al . [ 15 ] study the local convergence of first-order algorithms for finding Stackelberg equilibria in general-sum games . Their result also assumes exact feedback and do not allow sampling errors . The AI Economist [ 53 ] studies the optimal taxation problem by learning the Stackelberg equilibrium via a two-level reinforcement learning approach . Learning equilibria in Markov games A recent line of results [ 4 , 5 , 51 , 52 ] consider learning Markov games [ 40 ] —a generalization of Markov decision process to the multi-agent setting . We remark that all three settings studied in this paper can be cast as special cases of general-sum Markov games , which is studied by [ 28 ] . In particular , Liu et al . [ 28 ] provides sample complexity guarantees for finding Nash equilibria , correlated equilibria , or coarse correlated equilibria of the general-sum Markov games . These Nash-finding algorithms are related to our setting , but do not imply results for learning Stackelberg ( see Section 3.2 and Appendix C.5 for detailed discussions ) . 2 Preliminaries . Bandit games A general-sum two-player bandit game can be described by a tuple M = ( A , B , r1 , r2 ) , which defines the following game played by two players , a leader and a follower : • The leader plays an action a ∈ A , with |A| = A . • The follower sees the action played by the leader , and plays an action b ∈ B , with |B| = B . • The follower observes a ( potentially random ) reward r2 ( a , b ) ∈ [ 0 , 1 ] . The leader also observes her own reward r1 ( a , b ) ∈ [ 0 , 1 ] . Note that this is a special case of a general-sum turn-based Markov game with two steps [ 40 , 4 ] . This game is also a turn-based variant of the simultaneous matrix game considered in [ 13 , 27 ] ( for which we also provide results in Appendix B ) . Best response , Stackelberg equilibrium Let µi ( a , b ) : = E [ ri ( a , b ) ] ( i = 1 , 2 ) denote the mean rewards . For each leader action a , the best response set BR0 ( a ) is the set of follower actions that maximize µ2 ( a , · ) : BR0 ( a ) : = { b : µ2 ( a , b ) = max b′∈B µ2 ( a , b ′ ) } . ( 1 ) Given the best-response set BR0 ( a ) , we define the function φ0 : A → [ 0 , 1 ] as the leader ’ s value function when the follower plays the worst-case best response ( henceforth the “ exact φ-function ” ) : φ0 ( a ) : = min b∈BR0 ( a ) µ1 ( a , b ) , ( 2 ) This is the value function for the leader action a , assuming the follower plays the best response to a and breaks ties in the best response set against the leader ’ s favor . This is known as pessimistic tie breaking and provides a worst-case guarantee for the leader [ 13 ] . We remark that here restricting b to pure strategies ( deterministic actions ) is without loss of generality , as there is at least one pure strategy that maximizes µ2 ( a , · ) and ( among the maximizers ) minimize µ1 ( a , · ) , among all mixed strategies . The Stackelberg Equilibrium ( henceforth also “ Stackelberg ” ) for the leader is the “ best response to the best response ” , i.e . any action a ? that maximizes φ0 [ 42 ] : a ? ∈ arg max a∈A φ0 ( a ) . ( 3 ) We are interested in finding approximate solutions to the Stackelberg equilibrium , that is , an action â that approximately maximizes φ0 ( a ) . Note that as the leader ’ s action is seen by the follower , it suffices to only consider pure strategies for the leader too ( this is equivalent to the “ optimal committment to pure strategies ” problem of [ 13 ] ) . We also remark that , while we consider pessimistic tie breaking ( definition ( 2 ) ) in this paper , similar results hold in the optimistic setting as well in which the follower breaks ties in favor of the leader . We defer the statements and proofs of these results to Appendix A . Real-world example ( AI Economist ) : Consider the following ( simplified ) optimal taxation problem in the AI Economist [ 53 ] as an example of a bandit game . The government ( leader ) determines the tax rate a ∈ A for the follower . The citizen ( follower ) then chooses the amount of labor b ∈ B she wishes to perform . The rewards for the two players are different in general : For example , the citizen ’ s reward r2 ( a , b ) can be her post-tax income per labor , and the government ’ s reward r1 ( a , b ) can be a weighted average of its tax income and some measure of the citizen ’ s welfare ( e.g . not too much labor ) . We remark that the actual AI Economist is more similar to a bandit-RL game where the follower plays sequentially in an MDP determined by the leader , which we study in Section 4 . Sample-efficient learning with bandit feedback In this paper we consider the bandit feedback setting , that is , the algorithm can not directly observe the mean rewards µ1 ( · , · ) and µ2 ( · , · ) , and can only query ( a , b ) and obtain random samples ( r1 ( a , b ) , r2 ( a , b ) ) . Our goal is to determine the number of samples in order to find an approximate maximizer of φ0 ( a ) . Note that the bandit feedback setting assumes observation noise in the rewards . As we will see in Section 3 , this noise turns out to bring in a fundamental challenge for learning Stackelberg equilibria that is not present in ( and thus not directly solved by ) existing work on learning Stackelberg , which assumes either exact observation of the mean rewards [ 13 , 26 ] , or the best-response oracle that can query a and obtain the exact best response set BR0 ( a ) [ 27 , 34 ] . Markov decision processes We also present the basics of a Markov Decision Processes ( MDPs ) , which will be useful for the bandit-RL games in Section 4 . We consider episodic MDPs defined by a tuple ( H , S , B , d1 , P , r ) , where H is the horizon length , S is the state space , B is the action space1 , P = { Ph ( ·|s , b ) : h ∈ [ H ] , s ∈ S , b ∈ B } is the transition probabilities , and r = { rh : S × B → [ 0 , 1 ] , h ∈ [ H ] } are the ( potentially random ) reward functions . A policy π = { πbh ( ·|s ) ∈ ∆B : h ∈ [ H ] , s ∈ S } for the player is a set of probability distributions over actions given the state . In this paper we consider the exploration setting as the protocol of interacting with MDPs , similar as in [ 3 , 17 ] . The learning agent is able to play episodes repeatedly , where in each episode at step h ∈ { 1 , . . . , H } , the agent observes state sh ∈ S , takes an action bh ∼ πh ( ·|sh ) , observes her reward rh = rh ( sh , bh ) ∈ [ 0 , 1 ] , and transits to the next state sh+1 ∼ Ph ( ·|sh , bh ) . The initial state is received from the MDP : s1 ∼ d1 ( · ) . The overall value function ( return ) of a policy π is defined as V ( π ) : = Eπ [ ∑H h=1 rh ( sh , bh ) ] . 1The notation B indicates that the MDP is played by the follower ( cf . Section 4 ) ; we reserve A as the leader ’ s action space in this paper .

Summary: The authors study the problem of learning Stackelberg equilibria from random samples in general sum games. The work will be of interest to both theoreticians of Stackelberg games and those who work on applications. The main contributions are: - In bandit games, which are two-step Markov games where the leader and follower only observe their own rewards, the authors show that there is a constant gap between the optimal and the best strategy the leader can attain with a finite number of samples. This gap is a central feature of the paper—it exists in all settings that are analyzed by the authors, and the sample complexity results give you performance that approaches the best possible up to the gap. The authors give a tight bound on the number of samples required which is linear in the size of the action sets of the leader and follower and quadratic in the approximation ratio relative to the gap. The algorithm queries each leader action a fixed number of times (which is the same across all actions) to construct an estimate of the follower's best response function that has the desired properties. - For bandit-RL games, where the leader selects an action that induces an MDP for the follower, the authors show that a quite similar algorithm can be applied. The main challenge is that the follower has to learn the optimal policy for each MDP (this is an RL setting with bandit feedback to the follower), and this requires an application of some recent work to yield appropriate estimates. The bandit-RL game setting reduces to bandit games when the horizon is of length 1. The derived sample complexity matches that of the bandit game for the horizon = 1 subcase, showing that the dependence on these parameters is tight. However, the scaling in horizon length is extreme—more than quintic. There is no known lower bound for the dependence on horizon. - In linear bandit games, which are bandit games, but where the rewards are a linear function of the features of each action. The algorithm the authors state for this setting scales quadratically in the dimension of the feature space and quadratically in the approximation ratio. There is a factor of the dimension between this and the lower bound they derive.