Gradient play in stochastic games: stationary points, convergence, and sample complexity

1 INTRODUCTION . Multi-agent systems find applications in a wide range of societal systems , e.g . electric grid , traffic networks , smart building and smart cities etc . Given the complexity of these systems , multi-agent reinforcement learning ( MARL ) has gained increasing attention in recent years ( Daneshfar & Bevrani , 2010 ; Shalev-Shwartz et al. , 2016 ; Vidhate & Kulkarni , 2017 ; Xu et al. , 2020 ) . Among MARL algorithms , policy gradient-type methods are highly popular because of their flexibility and capability to incorporate structured state and action spaces . However , while many recent works ( Zhang et al. , 2018 ; Chen et al. , 2018 ; Wai et al. , 2018 ; Li et al. , 2019 ; Qu et al. , 2020 ) have studied the sample complexity of multi-agent policy gradient algorithms , due to a lack of understanding of the optimization landscape in these multi-agent learning problems , most works can only show convergence to a first-order stationary point . Deeper understanding of the quality of these stationary points is missing even in the simple identical-reward multi-agent RL setting . In this paper , we examine this problem from a game-theoretic perspective . We model the multi-agent system as a stochastic game ( SG ) where agents can have different reward functions , and study the dynamical behavior of first-order ( gradient-based ) learning methods . The study of SGs dates back to as early as the 1950s by Shapley ( 1953 ) with a series of followup works on developing NE-seeking algorithms , especially in the RL setting ( e.g . ( Littman , 1994 ; Bowling & Veloso , 2000 ; Shoham et al. , 2003 ; Buşoniu et al. , 2010 ; Lanctot et al. , 2017 ; Zhang et al. , 2019a ) and citations within ) . While well-known classical algorithms for solving SGs are mostly value-based , such as Nash-Q learning ( Hu & Wellman , 2003 ) , Hyper-Q learning ( Tesauro , 2003 ) , and WoLF-PHC ( Bowling & Veloso , 2001 ) , gradient-based algorithms have also started to gain popularity in recent years due to their advantages as mentioned earlier ( e.g . ( Abdallah & Lesser , 2008 ; Zhang & Lesser , 2010 ; Foerster et al. , 2017 ) ) . In this work , we aim to gain a deeper understanding of the structure and quality of first-order stationary points for these gradient-based methods , with a particular focus on answering the following questions : 1 ) How do the first-order stationary points relate to the NEs of the underlying game ? , 2 ) Do gradient-based algorithms guarantee convergence to a NE ? , 3 ) What is the stability of the individual NEs ? , and 4 ) How should agents learn using local samples from the environment ? . These questions have already been widely discussed in other settings , e.g. , one-shot ( stateless ) finiteaction games ( Shapley , 1964 ; Crawford , 1985 ; Jordan , 1993 ; Krishna & Sjöström , 1998 ; Shamma & Arslan , 2005 ; Kohlberg & Mertens , 1986 ; Van Damme , 1991 ) , one-shot continuous games ( Mazumdar et al. , 2020 ) , zero-sum linear quadratic ( LQ ) games Zhang et al . ( 2019b ) , etc . There are both negative and positive results depending on the settings . For one-shot continuous games , ( Mazumdar et al. , 2020 ) proved a negative result suggesting that gradient flow has stationary points ( even local maxima ) that are not necessarily NEs . Conversely , Zhang et al . ( 2019b ) designed projected nested-gradient methods that provably converge to NEs in zero-sum LQ games . However , much less is known in the tabular setting of SGs with finite state-action spaces . Contributions . In our paper , we consider the gradient play algorithm for the infinite time-discounted reward SGs where an agent ’ s individual policy is directly parameterized by the probability of choosing an action from the agent ’ s own action space at a given state . We focus on the tabular setting where state and action spaces are finite . Through generalizing the gradient domination property in ( Agarwal et al. , 2020 ) to the multi-agent setting , we first establish the equivalence of first-order stationary policies and Nash equilibria . We then show that strict NEs are locally asymptotically stable under gradient play and provide a local convergence rate analysis . Additionally , we study the global convergence for a special class of SGs called Markov potential games ( MPGs ) ( González-Sánchez & Hernández-Lerma , 2013 ; Macua et al. , 2018 ; Leonardos et al. , 2021 ) , which includes identical reward multi-agent RL ( Tan , 1993 ; Claus & Boutilier , 1998 ; Panait & Luke , 2005 ) as an important special case . In this setting , we first show that exact gradient play can find an -Nash equilibrium within O ( |S| ∑ i |Ai| 2 ) steps . Then , we design a sample-based gradient play algorithm and show that it can find an -Nash equilibrium with high probability in a fullydecentralized manner using Õ ( n 6 poly ( 1 1−γ , |S| , maxi |Ai| ) ) samples , where |S| , |Ai| denote the size of the state space and action space of agent i respectively . The convergence rate shows that the number of iterations to reach an -NE scales linearly with the number of agents . In the sample-based learning , agents only need to observe the state , their own actions , and their own rewards . The key enabler of the learning is the existence of an underlying averaged MDP for each agent when other agents ’ policies are fixed . Our learning method can be viewed as a model-based policy evaluation method with respect to agents ’ averaged MDPs . This averaged MDP concept could be applied to design many other MARL algorithms , especially policy-evaluation-based methods . We also study the local geometry around some special classes of equilibrium points , showing that strict NEs are local maxima of the total potential function and that fully mixed NEs are saddle points . Lastly , all the algorithms studied in this paper have been numerical tested and results are provided in Appendix A . Comparison to other MARL algorithms : For MPGs with continuous state and action spaces , there are studies about learning either the open-loop ( González-Sánchez & Hernández-Lerma , 2013 ; Zazo et al. , 2016 ) or the closed-loop ( Macua et al. , 2018 ) NEs for MPGs . These works generally assume full model information and solve the problem via optimal control . There are two recent arXiv preprints ( Mguni , 2020 ; Leonardos et al. , 2021 ; Mguni et al. , 2021 ) studying MPGs that are similar to our MPG setting . In particular , Leonardos et al . ( 2021 ) also studies gradient play for MPG . Both of our papers share similar results on MPGs but the sample-based methods are designed from different perspectives.1 In addition , our papers studies general SGs besides MPG . Moreover , our concept of “ averaged ” MDPs could also serve as a useful tool for the design and analysis of other MARL algorithms . Beyond these MPG works , decentralized Q-learning introduced in Arslan & Yüksel ( 2016 ) might be the closest to the setting considered in this paper . They consider the identical interest case and only show asymptotic convergence to the set of NEs . There are other recent works that also study learning for general-sum or zero-sum stochastic games . However the settings they consider are different from our setting , for example , Daskalakis et al . ( 2021 ) considers convergence to NE for two player zero-sum games , while Song et al . ( 2021 ) considers convergence to correlated equilibrium for finite time horizon general-sum games . On the other hand , Zhang et al . ( 2018 ) ; Li et al . ( 2019 ) ; Qu et al . ( 2019 ) consider slightly different MARL settings , where agents collaboratively maximize the summation of agents ’ reward with either full or partial state observation . They also require communication between neighboring agents for a better global coordination . 1Leonardos et al . ( 2021 ) considers Monte Carlo , model-free gradient estimation . The sample complexity is derived under the condition that the estimation is unbiased , which is difficult to hold in general . Interestingly , both sample complexities are O ( 1 6 ) . It is an interesting question to study whether such dependence is fundamental or not . We also remark that Leonardos et al . ( 2021 ) and this work are done in parallel . 2 PROBLEM SETTING AND PRELIMINARIES . We consider a stochastic game ( SG ) M = ( N , S , A = A1 × · · · × An , P , r = ( r1 , . . . , rn ) , γ , ρ ) with n agents ( Shapley , 1953 ) which is specified by an agent set N = { 1 , 2 , . . . , n } , a finite state space S , a finite action space Ai for each agent i ∈ N , a transition model P where P ( s′|s , a ) = P ( s′|s , a1 , . . . , an ) is the probability of transitioning into state s′ upon taking action a : = ( a1 , . . . , an ) in state s where ai ∈ Ai is action of agent i , agent i ’ s reward function ri : S ×A → [ 0 , 1 ] , a discount factor γ ∈ [ 0 , 1 ) , and an initial state distribution ρ over S. A stochastic policy π : S → ∆ ( A ) ( where ∆ ( A ) is the probability simplex over A ) specifies a strategy in which agents choose their actions jointly based on the current state in a stochastic fashion , i.e . Pr ( at|st ) = π ( at|st ) . A distributed stochastic policy is a special subclass of stochastic policies , with π = π1× . . .×πn , where πi : S → ∆ ( Ai ) . For distributed stochastic policies , each agent takes its action based on the current state s independently of other agents ’ choices of actions , i.e . : Pr ( at|st ) = π ( at|st ) = n∏ i=1 πi ( ai , t|st ) , at = ( a1 , t , . . . , an , t ) . For notational simplicity , we define : πI ( aI |s ) : = ∏ i∈I πi ( ai|s ) , where I ⊆ N is an index set . Further , we use the notation −i to denote the index set N\ { i } . We consider direct distributed policy parameterization , where agent i ’ s policy is parameterized by θi : πi , θi ( ai|s ) = θi , ( s , ai ) , i = 1 , 2 , . . . , n. ( 1 ) For notational simplicity , we abbreviate πi , θi ( ai|s ) as πθi ( ai|s ) , and θi , ( s , ai ) as θs , ai . Here θi ∈ ∆ ( Ai ) |S| , i.e . θi is subject to the constraints θs , ai ≥ 0 and ∑ ai∈Ai θs , ai = 1 for all s ∈ S. The global joint policy is given by : πθ ( a|s ) = ∏n i=1 πθi ( ai|s ) = ∏n i=1 θs , ai . We use Xi : = ∆ ( Ai ) |S| , X : = X1 × · · · × Xn to denote the feasible region of θi and θ . Agent i ’ s value function V θi : S → R , i ∈ N is defined as the discounted sum of future rewards starting at state s via executing πθ , i.e . V θi ( s ) : = E [ ∞∑ t=0 γtri ( st , at ) ∣∣ πθ , s0 = s ] , where the expectation is with respect to the random trajectory τ = ( st , at , ri , t ) ∞t=0 where at ∼ πθ ( ·|st ) , st+1 = P ( ·|st , at ) . We denote agent i ’ s total reward starting from initial state s0 ∼ ρ as : Ji ( θ ) = Ji ( θ1 , . . . , θn ) : = Es0∼ρV θi ( s0 ) . In the game setting , Nash equilibrium is often used to characterize the performance of agents ’ policies . Definition 1 . ( Nash equilibrium ) A policy θ∗ = ( θ∗1 , . . . , θ∗n ) is called a Nash equilibrium ( NE ) if Ji ( θ ∗ i , θ ∗ −i ) ≥ Ji ( θ′i , θ∗−i ) , ∀θ′i ∈ Xi , i ∈ N The equilibrium is called a strict NE if the inequality holds strictly for all θ′i ∈ Xi and i ∈ N . The equilibrium is called a pure NE if θ∗ corresponds to a deterministic policy . The equilibrium is called a mixed NE if it is not pure . Further , the equilibrium is called a fully mixed NE if every entry of θ∗ is strictly positive , i.e . : θ∗s , ai > 0 , ∀ ai ∈ Ai , ∀ s ∈ S , i ∈ N We define the discounted state visitation distribution dθ of a policy πθ given an initial state distribution ρ as : dθ ( s ) : = Es0∼ρ ( 1− γ ) ∞∑ t=0 γtPrθ ( st = s|s0 ) , ( 2 ) where Prθ ( st = s|s0 ) is the state visitation probability that st = s when executing πθ starting at state s0 . Throughout the paper , we make the following assumption on the SGs we study . Assumption 1 . The stochastic gameM satisfies : dθ ( s ) > 0 , ∀s ∈ S , ∀θ ∈ X . Assumption 1 requires that every state is visited with positive probability , which is a standard assumption for convergence proofs in the RL literature ( e.g . ( Agarwal et al. , 2020 ; Mei et al. , 2020 ) ) . Similar to centralized RL , we define agent i ’ sQ-functionQθi : S×A → R and its advantage function Aθi : S ×A → R as : Qθi ( s , a ) : = E [ ∞∑ t=0 γtri ( st , at ) ∣∣ πθ , s0 = s , a0 = a ] , Aθi ( s , a ) : = Qθi ( s , a ) − V θi ( s ) . ‘ Averaged ’ Markov decision process ( MDP ) : We further define agent i ’ s ‘ averaged ’ Q-function Qθi : S ×Ai → R and ‘ averaged ’ advantage-function Aθi : S ×Ai → R as : Qθi ( s , ai ) : = ∑ a−i πθ−i ( a−i|s ) Qθi ( s , ai , a−i ) , Aθi ( s , ai ) : = ∑ a−i πθ−i ( a−i|s ) Aθi ( s , ai , a−i ) . ( 3 ) Similarly , we define agent i ’ s ‘ averaged ’ transition probability distribution P θi : S × S ×Ai → R , and ‘ averaged ’ reward rθi : S ×Ai → R as : P θi ( s ′|s , ai ) : = ∑ a−i πθ−i ( a−i|s ) P ( s′|s , ai , a−i ) , rθi ( s , ai ) : = ∑ a−i πθ−i ( a−i|s ) ri ( s , ai , a−i ) From its definition , the averaged Q-function satisfies the following Bellman equation : Lemma 1 . Qθi satisfies : Qθi ( s , ai ) = rθi ( s , ai ) + γ ∑ s′ , a′i πθi ( a ′ i|s′ ) P θi ( s ′|s , ai ) Qθi ( s ′ , a′i ) ( 4 ) Lemma 1 suggests that the averaged Q-function Qθi is indeed the Q-function for the MDP defined on action space Ai , with rθi , P θi as its stage reward and transition probability respectively . We define this MDP as the ‘ averaged ’ MDP of agent i , i.e. , Mθi = ( S , Ai , P θi , rθi , γ , ρ ) . The notion of an ‘ averaged ’ MDP will serve as an important intuition when designing the sample-based algorithm . Note that the ‘ averaged ’ MDP is only well-defined when the policies of the other agents θ−i are kept fixed . When this is indeed the case , agent i can be treated as an independent learner with respect to its own ‘ averaged ’ MDP . Thus , various classical policy evaluation RL algorithms can then be applied .

Summary: This paper studies the optimization landscape of first-order methods in stochastic games. In particular, it studies the connections between first-order stationary points and Nash equilibrium and its convergent property and learning stability. For general-sum SGs, local convergence results is provided. For Markov potential games, global convergence results are provided.