Sample-Efficient Learning of Stackelberg Equilibria in General-Sum Games

1 Introduction . Real-world problems such as economic design and policy making can often be modeled as multiagent games that involves two levels of thinking : The policy maker—as a player in this game— needs to reason about the other player ’ s optimal behaviors given her decision , in order to inform her own optimal decision making . Consider for example the optimal taxation problem in the AI Economist [ 53 ] , a game modeling a real-world social-economic system involving a leader ( e.g . the government ) and a group of interacting followers ( e.g . citizens ) . The leader sets a tax rate which determines an economics-like game for the followers ; the followers then play in this game with the objective to maximize their own reward ( such as individual productivity ) . However , the goal of the leader is to maximize her own reward ( such as overall equality ) which is in general different from the followers ’ rewards , making these games general-sum [ 38 ] . Such two-level thinking appears broadly in other applications as well such as in automated mechanism design [ 11 , 12 ] , optimal auctions [ 10 , 14 ] , security games [ 43 ] , reward shaping [ 25 ] , and so on . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Another key feature in such games is that the players are asymmetric , and they act in turns : the leader first plays , then the follower sees the leader ’ s action and then adapts to it . This makes symmetric solution concepts such as Nash equilibrium [ 30 ] not always appropriate . A more natural solution concept for these games is the Stackelberg equilibrium : the leader ’ s optimal strategy , assuming the followers play their best response to the leader [ 42 , 13 ] . The Stackelberg equlibrium is often the desired solution concept in many of the aforementioned applications . Furthermore , it is of compelling interest to understand the learning of Stackelberg equilibria from samples , as it is often the case that we can only learn about the game through interactively deploying policies and observing the ( noisy ) feedback from the game [ 53 ] . This may be the case even when the game rules are perfectly known but not yet represented in a desired form , as argued in the line of work on empirical game theory [ 50 , 48 , 20 ] . Despite the motivations , theoretical studies of learning Stackelberg equilibria in general-sum games remain open , in particular when we can only learn from noisy samples of the rewards . A line of work provides guarantees for finding Stackelberg equilibria in general-sum games , but restricts attention to either the full observation setting ( so that the exact game is observable ) or with an exact best-response oracle [ 13 , 27 , 47 , 34 ] . These results lay out a foundation for analyzing the Stackelberg equilibrium , but do not generalize to the bandit feedback setting in which the game can only be learned from random samples of the rewards . Another line of work considers the sample complexity of learning the Nash equilibrium in Markov games [ 35 , 4 , 5 , 28 , 51 , 52 ] , which also do not imply algorithms for finding the Stackelberg equilibrium in these games as the Nash is in general different from the Stackelberg equilibrium in general-sum games . In this work , we study the sample complexity of learning Stackelberg equilibrium in general-sum games . We focus on general-sum games with two players ( one leader and one follower ) , in which we wish to learn an approximate Stackelberg equilbrium for the leader from random samples . Our contributions can be summarized as follows . • As a warm-up , we consider bandit games in which the two players play an action in turns and observe their own rewards . We identify a fundamental gap between the exact Stackelberg value and its estimated version from finite samples , which can not be closed information-theoretically regardless of the algorithm ( Section 3.1 ) . We then propose a rigorous definition gapε for this gap , and show that it is possible to sample-efficiently learn the ( gapε + ε ) -approximate Stackelberg equilibrium with Õ ( AB/ε2 ) samples , where A , B are the number of actions for the two players . We further show a matching lower bound Ω ( AB/ε2 ) ( Section 3.2 ) . We also establish similar results for learning Stackelberg in simultaneous matrix games ( Appendix B ) . • We consider bandit-RL games in which the leader ’ s action determines an episodic Markov Decision Process ( MDP ) for the follower . We show that a ( gapε + ε ) approximate Stackelberg equilibrium for the leader can be found in Õ ( H5S2AB/ε2 ) episodes of play , where H , S are the horizon length and number of states for the follower ’ s MDP , and A , B are the number of actions for the two players ( Section 4 ) . Our algorithm utilizes recently developed reward-free reinforcement learning techniques to enable fast exploration for the follower within the MDPs . • Finally , we consider linear bandit games in which the action spaces for the two players can be arbitrarily large , but the reward is a linear function of a d-dimensional feature representation of the actions . We design an algorithm that achieves Õ ( d2/ε2 ) sample complexity upper bound for linear bandit games ( Section 5 ) . This only depends polynomially on the feature dimension instead of the size of the action spaces , and has at most an Õ ( d ) gap from the lower bound . 1.1 Related work . Since the seminal paper of [ 46 ] , notions of equilibria in games and their algorithmic computation have received wide attention [ see , e.g. , 9 , 41 ] . For the scope of this paper , this section focuses on reviewing results that related to learning Stackelberg equilibria . Learning Stackelberg equilibria in zero-sum games The first category of results study twoplayer zero-sum games , where the rewards of the two players sum to zero . Most results along this line focus on the bilinear or convex-concave setting [ see , e.g. , 21 , 32 , 31 , 37 , 16 ] , where the Stackelberg equilibrium coincide with the Nash equilibrium due to Von Neumann ’ s minimax theorem [ 46 ] . Results for learning Stackelberg equilibria beyond convex-concave setting are much more recent , with Rafique et al . [ 36 ] , Nouiehed et al . [ 33 ] considering the nonconvex-concave setting , and Fiez et al . [ 15 ] , Jin et al . [ 19 ] , Marchesi et al . [ 29 ] considering the nonconvex-nonconcave setting . Marchesi et al . [ 29 ] provide sample complexity results for learning Stackelberg with infinite strategy spaces , using discretization techniques that may scale exponentially in the dimension without further assumptions on the problem structure . We remark that a crucial property of zero-sum games is that any two strategies giving similar rewards for the follower will also give similar rewards for the leader . This is no longer true in general-sum games , and prevents most statistical results for learning Stackelberg equilibria in the zero-sum setting from generalizing to the general-sum setting . Learning Stackelberg equilibria in general-sum games The computational complexity of finding Stackelberg equilibria in games with simultaneous play ( “ computing optimal strategy to commit to ” ) is studied in [ 13 , 26 , 47 , 22 , 1 , 7 ] . These results assume full observation of the payoff function , and show that several versions of matrix games and extensive-form ( multi-step ) games admit polynomial time algorithms . Vasal [ 45 ] designs computationally efficient algorithms for computing Stackelberg in certain “ conditionally independent controlled ” Markov games . Another line of work considers learning Stackelberg with a “ best response oracle ” [ 27 , 6 , 34 ] , that returns the follower ’ s exact best response strategy when a leader ’ s strategy is queried . This oracle and the noisy reward oracle we assume are in general incomparable ( can not simulate each other regardless of the number of queries ) , and thus our sample complexity results do not imply each other . The recent work of Sessa et al . [ 39 ] proposes the StackelUCB algorithm to sample-efficiently learn a Stackelberg game where the opponent ’ s response function has a linear structure in a certain kernel space , and the observation noise is added in the action space instead of the reward ( thus a different noisy feedback model from ours ) . Lastly , Fiez et al . [ 15 ] study the local convergence of first-order algorithms for finding Stackelberg equilibria in general-sum games . Their result also assumes exact feedback and do not allow sampling errors . The AI Economist [ 53 ] studies the optimal taxation problem by learning the Stackelberg equilibrium via a two-level reinforcement learning approach . Learning equilibria in Markov games A recent line of results [ 4 , 5 , 51 , 52 ] consider learning Markov games [ 40 ] —a generalization of Markov decision process to the multi-agent setting . We remark that all three settings studied in this paper can be cast as special cases of general-sum Markov games , which is studied by [ 28 ] . In particular , Liu et al . [ 28 ] provides sample complexity guarantees for finding Nash equilibria , correlated equilibria , or coarse correlated equilibria of the general-sum Markov games . These Nash-finding algorithms are related to our setting , but do not imply results for learning Stackelberg ( see Section 3.2 and Appendix C.5 for detailed discussions ) . 2 Preliminaries . Bandit games A general-sum two-player bandit game can be described by a tuple M = ( A , B , r1 , r2 ) , which defines the following game played by two players , a leader and a follower : • The leader plays an action a ∈ A , with |A| = A . • The follower sees the action played by the leader , and plays an action b ∈ B , with |B| = B . • The follower observes a ( potentially random ) reward r2 ( a , b ) ∈ [ 0 , 1 ] . The leader also observes her own reward r1 ( a , b ) ∈ [ 0 , 1 ] . Note that this is a special case of a general-sum turn-based Markov game with two steps [ 40 , 4 ] . This game is also a turn-based variant of the simultaneous matrix game considered in [ 13 , 27 ] ( for which we also provide results in Appendix B ) . Best response , Stackelberg equilibrium Let µi ( a , b ) : = E [ ri ( a , b ) ] ( i = 1 , 2 ) denote the mean rewards . For each leader action a , the best response set BR0 ( a ) is the set of follower actions that maximize µ2 ( a , · ) : BR0 ( a ) : = { b : µ2 ( a , b ) = max b′∈B µ2 ( a , b ′ ) } . ( 1 ) Given the best-response set BR0 ( a ) , we define the function φ0 : A → [ 0 , 1 ] as the leader ’ s value function when the follower plays the worst-case best response ( henceforth the “ exact φ-function ” ) : φ0 ( a ) : = min b∈BR0 ( a ) µ1 ( a , b ) , ( 2 ) This is the value function for the leader action a , assuming the follower plays the best response to a and breaks ties in the best response set against the leader ’ s favor . This is known as pessimistic tie breaking and provides a worst-case guarantee for the leader [ 13 ] . We remark that here restricting b to pure strategies ( deterministic actions ) is without loss of generality , as there is at least one pure strategy that maximizes µ2 ( a , · ) and ( among the maximizers ) minimize µ1 ( a , · ) , among all mixed strategies . The Stackelberg Equilibrium ( henceforth also “ Stackelberg ” ) for the leader is the “ best response to the best response ” , i.e . any action a ? that maximizes φ0 [ 42 ] : a ? ∈ arg max a∈A φ0 ( a ) . ( 3 ) We are interested in finding approximate solutions to the Stackelberg equilibrium , that is , an action â that approximately maximizes φ0 ( a ) . Note that as the leader ’ s action is seen by the follower , it suffices to only consider pure strategies for the leader too ( this is equivalent to the “ optimal committment to pure strategies ” problem of [ 13 ] ) . We also remark that , while we consider pessimistic tie breaking ( definition ( 2 ) ) in this paper , similar results hold in the optimistic setting as well in which the follower breaks ties in favor of the leader . We defer the statements and proofs of these results to Appendix A . Real-world example ( AI Economist ) : Consider the following ( simplified ) optimal taxation problem in the AI Economist [ 53 ] as an example of a bandit game . The government ( leader ) determines the tax rate a ∈ A for the follower . The citizen ( follower ) then chooses the amount of labor b ∈ B she wishes to perform . The rewards for the two players are different in general : For example , the citizen ’ s reward r2 ( a , b ) can be her post-tax income per labor , and the government ’ s reward r1 ( a , b ) can be a weighted average of its tax income and some measure of the citizen ’ s welfare ( e.g . not too much labor ) . We remark that the actual AI Economist is more similar to a bandit-RL game where the follower plays sequentially in an MDP determined by the leader , which we study in Section 4 . Sample-efficient learning with bandit feedback In this paper we consider the bandit feedback setting , that is , the algorithm can not directly observe the mean rewards µ1 ( · , · ) and µ2 ( · , · ) , and can only query ( a , b ) and obtain random samples ( r1 ( a , b ) , r2 ( a , b ) ) . Our goal is to determine the number of samples in order to find an approximate maximizer of φ0 ( a ) . Note that the bandit feedback setting assumes observation noise in the rewards . As we will see in Section 3 , this noise turns out to bring in a fundamental challenge for learning Stackelberg equilibria that is not present in ( and thus not directly solved by ) existing work on learning Stackelberg , which assumes either exact observation of the mean rewards [ 13 , 26 ] , or the best-response oracle that can query a and obtain the exact best response set BR0 ( a ) [ 27 , 34 ] . Markov decision processes We also present the basics of a Markov Decision Processes ( MDPs ) , which will be useful for the bandit-RL games in Section 4 . We consider episodic MDPs defined by a tuple ( H , S , B , d1 , P , r ) , where H is the horizon length , S is the state space , B is the action space1 , P = { Ph ( ·|s , b ) : h ∈ [ H ] , s ∈ S , b ∈ B } is the transition probabilities , and r = { rh : S × B → [ 0 , 1 ] , h ∈ [ H ] } are the ( potentially random ) reward functions . A policy π = { πbh ( ·|s ) ∈ ∆B : h ∈ [ H ] , s ∈ S } for the player is a set of probability distributions over actions given the state . In this paper we consider the exploration setting as the protocol of interacting with MDPs , similar as in [ 3 , 17 ] . The learning agent is able to play episodes repeatedly , where in each episode at step h ∈ { 1 , . . . , H } , the agent observes state sh ∈ S , takes an action bh ∼ πh ( ·|sh ) , observes her reward rh = rh ( sh , bh ) ∈ [ 0 , 1 ] , and transits to the next state sh+1 ∼ Ph ( ·|sh , bh ) . The initial state is received from the MDP : s1 ∼ d1 ( · ) . The overall value function ( return ) of a policy π is defined as V ( π ) : = Eπ [ ∑H h=1 rh ( sh , bh ) ] . 1The notation B indicates that the MDP is played by the follower ( cf . Section 4 ) ; we reserve A as the leader ’ s action space in this paper .

Summary: The paper considers the problem of online learning the Stackelberg Equilibrium in general-sum games. The authors identify a fundamental gap between the exact and the estimated version of the Stackelberg value. Then they consider three general-sum games setting: bandit games, bandit-RL games, and linear bandit games, designing algorithms for these settings and providing sample complexity results.