Gradient play in stochastic games: stationary points, convergence, and sample complexity

1 INTRODUCTION . Multi-agent systems find applications in a wide range of societal systems , e.g . electric grid , traffic networks , smart building and smart cities etc . Given the complexity of these systems , multi-agent reinforcement learning ( MARL ) has gained increasing attention in recent years ( Daneshfar & Bevrani , 2010 ; Shalev-Shwartz et al. , 2016 ; Vidhate & Kulkarni , 2017 ; Xu et al. , 2020 ) . Among MARL algorithms , policy gradient-type methods are highly popular because of their flexibility and capability to incorporate structured state and action spaces . However , while many recent works ( Zhang et al. , 2018 ; Chen et al. , 2018 ; Wai et al. , 2018 ; Li et al. , 2019 ; Qu et al. , 2020 ) have studied the sample complexity of multi-agent policy gradient algorithms , due to a lack of understanding of the optimization landscape in these multi-agent learning problems , most works can only show convergence to a first-order stationary point . Deeper understanding of the quality of these stationary points is missing even in the simple identical-reward multi-agent RL setting . In this paper , we examine this problem from a game-theoretic perspective . We model the multi-agent system as a stochastic game ( SG ) where agents can have different reward functions , and study the dynamical behavior of first-order ( gradient-based ) learning methods . The study of SGs dates back to as early as the 1950s by Shapley ( 1953 ) with a series of followup works on developing NE-seeking algorithms , especially in the RL setting ( e.g . ( Littman , 1994 ; Bowling & Veloso , 2000 ; Shoham et al. , 2003 ; Buşoniu et al. , 2010 ; Lanctot et al. , 2017 ; Zhang et al. , 2019a ) and citations within ) . While well-known classical algorithms for solving SGs are mostly value-based , such as Nash-Q learning ( Hu & Wellman , 2003 ) , Hyper-Q learning ( Tesauro , 2003 ) , and WoLF-PHC ( Bowling & Veloso , 2001 ) , gradient-based algorithms have also started to gain popularity in recent years due to their advantages as mentioned earlier ( e.g . ( Abdallah & Lesser , 2008 ; Zhang & Lesser , 2010 ; Foerster et al. , 2017 ) ) . In this work , we aim to gain a deeper understanding of the structure and quality of first-order stationary points for these gradient-based methods , with a particular focus on answering the following questions : 1 ) How do the first-order stationary points relate to the NEs of the underlying game ? , 2 ) Do gradient-based algorithms guarantee convergence to a NE ? , 3 ) What is the stability of the individual NEs ? , and 4 ) How should agents learn using local samples from the environment ? . These questions have already been widely discussed in other settings , e.g. , one-shot ( stateless ) finiteaction games ( Shapley , 1964 ; Crawford , 1985 ; Jordan , 1993 ; Krishna & Sjöström , 1998 ; Shamma & Arslan , 2005 ; Kohlberg & Mertens , 1986 ; Van Damme , 1991 ) , one-shot continuous games ( Mazumdar et al. , 2020 ) , zero-sum linear quadratic ( LQ ) games Zhang et al . ( 2019b ) , etc . There are both negative and positive results depending on the settings . For one-shot continuous games , ( Mazumdar et al. , 2020 ) proved a negative result suggesting that gradient flow has stationary points ( even local maxima ) that are not necessarily NEs . Conversely , Zhang et al . ( 2019b ) designed projected nested-gradient methods that provably converge to NEs in zero-sum LQ games . However , much less is known in the tabular setting of SGs with finite state-action spaces . Contributions . In our paper , we consider the gradient play algorithm for the infinite time-discounted reward SGs where an agent ’ s individual policy is directly parameterized by the probability of choosing an action from the agent ’ s own action space at a given state . We focus on the tabular setting where state and action spaces are finite . Through generalizing the gradient domination property in ( Agarwal et al. , 2020 ) to the multi-agent setting , we first establish the equivalence of first-order stationary policies and Nash equilibria . We then show that strict NEs are locally asymptotically stable under gradient play and provide a local convergence rate analysis . Additionally , we study the global convergence for a special class of SGs called Markov potential games ( MPGs ) ( González-Sánchez & Hernández-Lerma , 2013 ; Macua et al. , 2018 ; Leonardos et al. , 2021 ) , which includes identical reward multi-agent RL ( Tan , 1993 ; Claus & Boutilier , 1998 ; Panait & Luke , 2005 ) as an important special case . In this setting , we first show that exact gradient play can find an -Nash equilibrium within O ( |S| ∑ i |Ai| 2 ) steps . Then , we design a sample-based gradient play algorithm and show that it can find an -Nash equilibrium with high probability in a fullydecentralized manner using Õ ( n 6 poly ( 1 1−γ , |S| , maxi |Ai| ) ) samples , where |S| , |Ai| denote the size of the state space and action space of agent i respectively . The convergence rate shows that the number of iterations to reach an -NE scales linearly with the number of agents . In the sample-based learning , agents only need to observe the state , their own actions , and their own rewards . The key enabler of the learning is the existence of an underlying averaged MDP for each agent when other agents ’ policies are fixed . Our learning method can be viewed as a model-based policy evaluation method with respect to agents ’ averaged MDPs . This averaged MDP concept could be applied to design many other MARL algorithms , especially policy-evaluation-based methods . We also study the local geometry around some special classes of equilibrium points , showing that strict NEs are local maxima of the total potential function and that fully mixed NEs are saddle points . Lastly , all the algorithms studied in this paper have been numerical tested and results are provided in Appendix A . Comparison to other MARL algorithms : For MPGs with continuous state and action spaces , there are studies about learning either the open-loop ( González-Sánchez & Hernández-Lerma , 2013 ; Zazo et al. , 2016 ) or the closed-loop ( Macua et al. , 2018 ) NEs for MPGs . These works generally assume full model information and solve the problem via optimal control . There are two recent arXiv preprints ( Mguni , 2020 ; Leonardos et al. , 2021 ; Mguni et al. , 2021 ) studying MPGs that are similar to our MPG setting . In particular , Leonardos et al . ( 2021 ) also studies gradient play for MPG . Both of our papers share similar results on MPGs but the sample-based methods are designed from different perspectives.1 In addition , our papers studies general SGs besides MPG . Moreover , our concept of “ averaged ” MDPs could also serve as a useful tool for the design and analysis of other MARL algorithms . Beyond these MPG works , decentralized Q-learning introduced in Arslan & Yüksel ( 2016 ) might be the closest to the setting considered in this paper . They consider the identical interest case and only show asymptotic convergence to the set of NEs . There are other recent works that also study learning for general-sum or zero-sum stochastic games . However the settings they consider are different from our setting , for example , Daskalakis et al . ( 2021 ) considers convergence to NE for two player zero-sum games , while Song et al . ( 2021 ) considers convergence to correlated equilibrium for finite time horizon general-sum games . On the other hand , Zhang et al . ( 2018 ) ; Li et al . ( 2019 ) ; Qu et al . ( 2019 ) consider slightly different MARL settings , where agents collaboratively maximize the summation of agents ’ reward with either full or partial state observation . They also require communication between neighboring agents for a better global coordination . 1Leonardos et al . ( 2021 ) considers Monte Carlo , model-free gradient estimation . The sample complexity is derived under the condition that the estimation is unbiased , which is difficult to hold in general . Interestingly , both sample complexities are O ( 1 6 ) . It is an interesting question to study whether such dependence is fundamental or not . We also remark that Leonardos et al . ( 2021 ) and this work are done in parallel . 2 PROBLEM SETTING AND PRELIMINARIES . We consider a stochastic game ( SG ) M = ( N , S , A = A1 × · · · × An , P , r = ( r1 , . . . , rn ) , γ , ρ ) with n agents ( Shapley , 1953 ) which is specified by an agent set N = { 1 , 2 , . . . , n } , a finite state space S , a finite action space Ai for each agent i ∈ N , a transition model P where P ( s′|s , a ) = P ( s′|s , a1 , . . . , an ) is the probability of transitioning into state s′ upon taking action a : = ( a1 , . . . , an ) in state s where ai ∈ Ai is action of agent i , agent i ’ s reward function ri : S ×A → [ 0 , 1 ] , a discount factor γ ∈ [ 0 , 1 ) , and an initial state distribution ρ over S. A stochastic policy π : S → ∆ ( A ) ( where ∆ ( A ) is the probability simplex over A ) specifies a strategy in which agents choose their actions jointly based on the current state in a stochastic fashion , i.e . Pr ( at|st ) = π ( at|st ) . A distributed stochastic policy is a special subclass of stochastic policies , with π = π1× . . .×πn , where πi : S → ∆ ( Ai ) . For distributed stochastic policies , each agent takes its action based on the current state s independently of other agents ’ choices of actions , i.e . : Pr ( at|st ) = π ( at|st ) = n∏ i=1 πi ( ai , t|st ) , at = ( a1 , t , . . . , an , t ) . For notational simplicity , we define : πI ( aI |s ) : = ∏ i∈I πi ( ai|s ) , where I ⊆ N is an index set . Further , we use the notation −i to denote the index set N\ { i } . We consider direct distributed policy parameterization , where agent i ’ s policy is parameterized by θi : πi , θi ( ai|s ) = θi , ( s , ai ) , i = 1 , 2 , . . . , n. ( 1 ) For notational simplicity , we abbreviate πi , θi ( ai|s ) as πθi ( ai|s ) , and θi , ( s , ai ) as θs , ai . Here θi ∈ ∆ ( Ai ) |S| , i.e . θi is subject to the constraints θs , ai ≥ 0 and ∑ ai∈Ai θs , ai = 1 for all s ∈ S. The global joint policy is given by : πθ ( a|s ) = ∏n i=1 πθi ( ai|s ) = ∏n i=1 θs , ai . We use Xi : = ∆ ( Ai ) |S| , X : = X1 × · · · × Xn to denote the feasible region of θi and θ . Agent i ’ s value function V θi : S → R , i ∈ N is defined as the discounted sum of future rewards starting at state s via executing πθ , i.e . V θi ( s ) : = E [ ∞∑ t=0 γtri ( st , at ) ∣∣ πθ , s0 = s ] , where the expectation is with respect to the random trajectory τ = ( st , at , ri , t ) ∞t=0 where at ∼ πθ ( ·|st ) , st+1 = P ( ·|st , at ) . We denote agent i ’ s total reward starting from initial state s0 ∼ ρ as : Ji ( θ ) = Ji ( θ1 , . . . , θn ) : = Es0∼ρV θi ( s0 ) . In the game setting , Nash equilibrium is often used to characterize the performance of agents ’ policies . Definition 1 . ( Nash equilibrium ) A policy θ∗ = ( θ∗1 , . . . , θ∗n ) is called a Nash equilibrium ( NE ) if Ji ( θ ∗ i , θ ∗ −i ) ≥ Ji ( θ′i , θ∗−i ) , ∀θ′i ∈ Xi , i ∈ N The equilibrium is called a strict NE if the inequality holds strictly for all θ′i ∈ Xi and i ∈ N . The equilibrium is called a pure NE if θ∗ corresponds to a deterministic policy . The equilibrium is called a mixed NE if it is not pure . Further , the equilibrium is called a fully mixed NE if every entry of θ∗ is strictly positive , i.e . : θ∗s , ai > 0 , ∀ ai ∈ Ai , ∀ s ∈ S , i ∈ N We define the discounted state visitation distribution dθ of a policy πθ given an initial state distribution ρ as : dθ ( s ) : = Es0∼ρ ( 1− γ ) ∞∑ t=0 γtPrθ ( st = s|s0 ) , ( 2 ) where Prθ ( st = s|s0 ) is the state visitation probability that st = s when executing πθ starting at state s0 . Throughout the paper , we make the following assumption on the SGs we study . Assumption 1 . The stochastic gameM satisfies : dθ ( s ) > 0 , ∀s ∈ S , ∀θ ∈ X . Assumption 1 requires that every state is visited with positive probability , which is a standard assumption for convergence proofs in the RL literature ( e.g . ( Agarwal et al. , 2020 ; Mei et al. , 2020 ) ) . Similar to centralized RL , we define agent i ’ sQ-functionQθi : S×A → R and its advantage function Aθi : S ×A → R as : Qθi ( s , a ) : = E [ ∞∑ t=0 γtri ( st , at ) ∣∣ πθ , s0 = s , a0 = a ] , Aθi ( s , a ) : = Qθi ( s , a ) − V θi ( s ) . ‘ Averaged ’ Markov decision process ( MDP ) : We further define agent i ’ s ‘ averaged ’ Q-function Qθi : S ×Ai → R and ‘ averaged ’ advantage-function Aθi : S ×Ai → R as : Qθi ( s , ai ) : = ∑ a−i πθ−i ( a−i|s ) Qθi ( s , ai , a−i ) , Aθi ( s , ai ) : = ∑ a−i πθ−i ( a−i|s ) Aθi ( s , ai , a−i ) . ( 3 ) Similarly , we define agent i ’ s ‘ averaged ’ transition probability distribution P θi : S × S ×Ai → R , and ‘ averaged ’ reward rθi : S ×Ai → R as : P θi ( s ′|s , ai ) : = ∑ a−i πθ−i ( a−i|s ) P ( s′|s , ai , a−i ) , rθi ( s , ai ) : = ∑ a−i πθ−i ( a−i|s ) ri ( s , ai , a−i ) From its definition , the averaged Q-function satisfies the following Bellman equation : Lemma 1 . Qθi satisfies : Qθi ( s , ai ) = rθi ( s , ai ) + γ ∑ s′ , a′i πθi ( a ′ i|s′ ) P θi ( s ′|s , ai ) Qθi ( s ′ , a′i ) ( 4 ) Lemma 1 suggests that the averaged Q-function Qθi is indeed the Q-function for the MDP defined on action space Ai , with rθi , P θi as its stage reward and transition probability respectively . We define this MDP as the ‘ averaged ’ MDP of agent i , i.e. , Mθi = ( S , Ai , P θi , rθi , γ , ρ ) . The notion of an ‘ averaged ’ MDP will serve as an important intuition when designing the sample-based algorithm . Note that the ‘ averaged ’ MDP is only well-defined when the policies of the other agents θ−i are kept fixed . When this is indeed the case , agent i can be treated as an independent learner with respect to its own ‘ averaged ’ MDP . Thus , various classical policy evaluation RL algorithms can then be applied .

Summary: The paper studies the global convergence of policy gradient for multiagent Markov potential games. The authors define a notion of potential markov games that seems natural and show convergence to Nash policies when the agents use gradient ascent independently on their policies (using direct parametrization). Observe that the authors assume that the agents have full knowledge of their utilities and their derivatives (their algorithm is deterministic). They also provide convergence for the stochastic variant of policy gradient (given samples). They moreover show that deterministic Nash policies always exist and show that fully mixed policies are saddle points. Saddle points might be of higher order, so first order methods might fail to avoid the fully mixed policies because you need to check third (or higher) order derivatives (so techniques from Ge et al do not help to show avoidance as claimed, the authors need to remove that comment since Ge et al argue about second order stationarity).