Can Reinforcement Learning Efficiently Find Stackelberg-Nash Equilibria in General-Sum Markov Games?

1 INTRODUCTION . Reinforcement learning ( RL ) has achieved striking empirical successes in solving complicated realworld sequential decision-making problems ( Mnih et al. , 2015 ; Duan et al. , 2016 ; Silver et al. , 2016 ; 2017 ; 2018 ; Agostinelli et al. , 2019 ; Akkaya et al. , 2019 ) . Motivated by these successes , multi-agent extensions of RL algorithms recently have gained great popularity in decision-making problems involving multiple interacting agents ( Busoniu et al. , 2008 ; Hernandez-Leal et al. , 2018 ; 2019 ; OroojlooyJadid & Hajinezhad , 2019 ; Zhang et al. , 2019 ) . Multi-agent RL is often modeled as a Markov game ( Littman , 1994 ) where , at each time step , each player ( agent ) takes an action simultaneously at each state of the environment , observe her own immediate reward , and the environment evolves into a next state . Here both the reward of each player and the state transition depends on the actions of all players . From the perspective of each player , her goal is to find a policy that maximizes her expected total reward in the presence of other agents . In Markov games , depending on the structure of the reward functions , the relationship among the players can be either collaborative , where each player has the same reward function , or competitive , where the sum of the reward function is equal to zero , or mixed , which corresponds to a general-sum game . While most of existing theoretical results focus on the collaborative or two-player competitive settings , the mixed setting is oftentimes more pertinent to real-world multi-agent applications . Moreover , in addition to having diverse reward functions , the players might also have asymmetric roles in the Markov game — the players might be divided into leaders and followers , where the leaders ’ joint policy determines a general-sum game for the followers . Games with such a leaderfollower structure is popular in applications such as mechanism design ( Conitzer & Sandholm , 2002 ; Roughgarden , 2004 ; Garg & Narahari , 2005 ; Kang & Wu , 2014 ) , security games ( Tambe , 2011 ; Korzhyk et al. , 2011 ; Balcan et al. , 2015 ) , incentive design ( Zheng et al. , 1984 ; Ratliff et al. , 2014 ; Chen et al. , 2016 ; Ratliff & Fiez , 2020 ) , and model-based RL ( Rajeswaran et al. , 2020 ) . Consider a simplified economic system that consists of a government and a group of companies , where the companies purchase or sell goods , and the government collects taxes from transactions . Such a problem can be viewed as a multi-player general-sum game , where the government serves as the leader and the companies are followers ( Zheng et al. , 2020 ) . In particular , when the government sets a tax rate , the companies form a general-sum game themselves , whose reward functions depend on the tax rate . Each company aims to maximize their own revenue , and thus ideally they achieve a Nash equilibrium ( NE ) of the induced game . Whereas the goal of the government might be achieving the social welfare , which might be measured via certain fairness metrics computed by the revenues of the companies . In multi-player Markov games with such a leader-follower structure , the desired solution concept is the Stackelberg-Nash equilibrium ( SNE ) ( Başar & Olsder , 1998 ) . In the setting where there is a single leader , SNE corresponds to a pair of leader ’ s policy π∗ and followers ’ joint policy ν∗ that satisfies the following two properties : ( i ) when the leader adopts π∗ , ν∗ is the best-response policy of the followers , i.e. , ν∗ is a Nash equilibrium of the followers ’ subgame induced by π∗ ; and ( ii ) π∗ is the optimal policy of the leader assuming the followers always adopt the best response . We are interested in finding an SNE in a multi-player Markov game when the reward functions and Markov transition kernel are unknown . In particular , we focus on the setting with a single leader and the state transitions only depend on the leader ’ s actions . That is , the followers ’ actions only affect the rewards received by the leader and followers . For such a game , we are interested in the following question : Can we develop reinforcement learning methods that provably find Stackelberg-Nash equilibria in leader-controlled general-sum games with sample efficiency ? To this end , we consider both online and offline RL settings , where in the former , we learn the SNE in a trial-an-error fashion by interacting with the environment and generating data , and in the latter , we learn the SNE from a given dataset that is collected a priori . For the online setting , as the transition model is unknown , to achieve sample efficiency , the equilibrium-finding algorithm also needs to take the exploration-exploitation tradeoff into consideration . Although the similar challenge has been studied in zero-sum Markov game , it seems unclear how to incorporate popular exploration mechanisms such as optimism in the face of uncertainty ( Sutton & Barto , 2018 ) into SNE finding . Meanwhile , under the offline setting , as the RL agent has no control of data collection , it is ideal to design an RL algorithm with theoretical guarantees for an arbitrary dataset that might not be sufficiently explorative . Our contributions Our contributions are three-fold . First , for the episodic leader-controlled general-sum game , under the online and offline settings respectively , we propose optimistic and pessimistic variants of the least-squares value iteration ( LSVI ) algorithm . In particular , in a version of LSVI , we estimate the optimal action-value function of the leader via least-squares regression and construct an estimate of the SNE by solving the SNE of the multi-matrix game for each state , whose payoff matrices are given by the leader ’ s estimated action-value function and the followers ’ reward functions . Moreover , we add a UCB exploration bonus to the least-squares solution to achieve optimism in the online setting . Whereas in the offline setting , pessimism is achieved by subtracting a penalty function constructed using the offline data , which is equal to the negative bonus function . Moreover , these algorithms are readily able to incorporate function approximators and we showcase the version with linear function approximation . Second , under the online setting , we prove that our optimistic LSVI algorithm achieves a sublinear Õ ( H2 √ d3K ) regret , where K is the number of episodes , H is the horizon , d is the dimension of the feature mapping , and Õ ( · ) omits logarithmic terms . Finally , under the offline setting , we establish an upper bound on the suboptimality of the proposed algorithm for an arbitrary dataset with K trajectories . Our upper bound yields a sublinear Õ ( H2 √ d3/K ) rate as long as the dataset has sufficient coverage over the trajectory induced by the desired SNE . Related work In the sequel , we discuss the related works on learning Stackelberg games . We defer more related works on RL for solving NE in Markov games and single-agent RL to §A . Learning Stackelberg games As for solving Stackelberg-Nash equilibrium , most of the existing results focus on the normal form game , which is equivalent to our Markov game with H = 1 . Letchford et al . ( 2009 ) ; Blum et al . ( 2014 ) ; Peng et al . ( 2019 ) study learning Stackelberg equilibrium with a best response oracle . In addition , Fiez et al . ( 2019 ) study the local convergence of firstorder methods for finding Stackelberg equilibria in general-sum games with differentiable reward functions , and Ghadimi & Wang ( 2018 ) ; Chen et al . ( 2021a ) ; Hong et al . ( 2020 ) analyze the global convergence of first-order methods for achieving global optimality of bilevel optimization . A more related work is Bai et al . ( 2021 ) , which studies the matrix Stackelberg game with bandit feedback . This work also studies an RL extension where the leader has a finite action set and the follower is faced with an MDP specified by the leader ’ s action . In comparison , we assume the leader knows the reward functions and the main challenge lies in the unknown and leader-controlled transitions . Thus , our setting is different from that in Bai et al . ( 2021 ) . Furthermore , a more relevant work is ( Bucarey et al. , 2019b ) , which establishes the Bellman equation and value iteration algorithm for solving SNE in leader-controlled Markov games . In comparison , we establish modifications of least-squares value iteration that are tailored to online and offline settings . Notation See §B for details . 2 PRELIMINARIES . In this section , we introduce the formulation of the general-sum simultaneous-move Markov games , Stackelberg-Nash equilibrium , and the linear structure we use in this paper . 2.1 GENERAL-SUM SIMULTANEOUS-MOVE MARKOV GAMES . In this setting , two levels of hierarchy in decision making are considered : one leader l and N followers { fi } i∈ [ N ] . Specifically , we define an episodic version of general-sum simultaneousmoves Markov game by the tuple ( S , Al , Af = { Afi } i∈ [ N ] , H , rl , rf = { rfi } i∈ [ N ] , P ) , where S is the state space , Al and Af are the sets of actions of the leader and the followers respectively , H is the number of steps in each episode , rl = { rl , h : S × Al × Af → [ −1 , 1 ] } Hh=1 and rfi = { rfi , h : S × Al × Af → [ −1 , 1 ] } Hh=1 are reward functions of the leader and the followers respectively , and P = { Ph : S × Al × Af × S → [ 0 , 1 ] } Hh=1 is a collection of transition kernels . Here Al ×Af = Al ×Af1 × · · · × AfN . Throughout this paper , we also let ? be some element in { l , f1 , · · · , fN } . Moreover , for any ( h , x , a ) ∈ [ H ] × S ×Al and b = { bi ∈ Afi } i∈ [ N ] , we use the shorthands r ? , h ( x , a , b ) = r ? , h ( x , a , b1 , · · · , bN ) and Ph ( · |x , a , b ) = Ph ( · |x , a , b1 , · · · , bN ) . Policy and Value Function . A stochastic policy π = { πh : S → ∆ ( Al ) } Hh=1 of the leader is a set of probability distributions over actions given the state . Meanwhile , a stochastic joint policy of the followers is defined by ν = { νfi } i∈ [ N ] , where νfi = { νfi , h : S → ∆ ( Afi ) } Hh=1 . We use the notation πh ( a |x ) and νfi , h ( bi |x ) to denote the probability of taking action a ∈ Al or bi ∈ Afi for state x at step h under policy π , νfi respectively . Throughout this paper , for any ν = { νfi } i∈ [ N ] and b = { bi } i∈ [ N ] , we use the shorthand νh ( b |x ) = νf1 , h ( b1 |x ) × · · · × νfN , h ( bN |x ) . Given policies ( π , ν = { νfi } i∈ [ N ] ) , the action-value ( Q ) and state-value ( V ) functions for the leader and followers are defined by Qπ , ν ? , h ( x , a , b ) = Eπ , ν , h , x , a , b [ H∑ t=h r ? , h ( xt , at , bt ) ] , V π , ν ? , h ( x ) = Ea∼πh ( · | x ) , b∼νh ( · | x ) Q π , ν ? , h ( x , a , b ) , ( 2.1 ) where the expectation Eπ , ν , h , x , a , b is taken over state-action pairs induced by the policies ( π , ν = { νfi } i∈ [ N ] ) and the transition probability , when initializing the process with the triplet ( s , a , b = { bi } i∈ [ N ] ) at step h. For notational simplicity , when h , x , a , b are clear from the context , we omit h , x , a , b from Eπ , ν , h , x , a , b . By the definition in ( 2.1 ) , we have the Bellman equation V π , ν ? , h = 〈Q π , ν ? , h , πh × νh〉Al×Af , Q π , ν ? , h = r ? , h + PhV π , ν ? , h+1 , ∀ ? ∈ { l , f1 , · · · , fN } , ( 2.2 ) where πh × νh represents πh × νf1 , h × · · · × νfN , h. Here Ph is the operator which is defined by ( Phf ) ( x , a , b ) = E [ f ( x′ ) |x′ ∼ Ph ( x′ |x , a , b ) ] ( 2.3 ) for any function f : S → R and ( x , a , b ) ∈ S ×Al ×Af .

Summary: The paper studies the problem of multi-agent Markov games with one leader and multiple followers. Moreover, the games considered are controlled by the leader, meaning that the transition function only depends on the action of the leader. The goal is to find the Stackelberg-Nash equilibrium of the game by means of reinforcement learning-based algorithms, both for the offline and online setting. The algorithms proposed have guarantees in terms of regret which is sub-linear in the number of episodes for the online setting and converges to zero in the offline setting in the size of the dataset.