Global Convergence of Multi-Agent Policy Gradient in Markov Potential Games

1 INTRODUCTION . Multi-agent reinforcement learning ( MARL ) has been the fundamental driver of numerous recent advances in Artificial Intelligence ( AI ) and Machine Learning ( ML ) ranging from super-human performance in competitive game-playing ( Silver et al. , 2016 ; 2018 ; Brown & Sandholm , 2019 ; Jaderberg et al. , 2019 ) and multi-tasking ( Mnih et al. , 2015 ; OpenAI , 2018 ; Vinyals et al. , 2019 ) to robotics , autonomous-driving and cyber-physical systems ( Busoniu et al. , 2008 ; Zhang et al. , 2019 ) . However , despite the popularity of MARL algorithms to analyze these systems in practice , the theory that underpins their empirical success lags behind ( Dafoe et al. , 2020 ) . Many state-of-the-art theoretical results concern single-agent RL systems , typically modelled as single-agent Markov Decision Processes ( MDPs ) ( Bertsekas , 2000 ; Panait & Luke , 2005 ; Sutton & Barto , 2018 ) . The main challenge when transitioning from single to multi-agent RL settings is the computation of Nash policies . For n ≥ 2 agents , a Nash policy is defined to be a profile of policies ( π∗1 , ... , π∗n ) so that by fixing the policies of all agents but i , π∗i is optimal for the resulting single-agent MDP and this is true for all 1 ≤ i ≤ n 1 ( Definition 1 ) . In multi-agent settings , Nash policies may not be unique in principle and , unlike singe-agent MDPs , agents ’ rewards may differ dramatically between them . A common approach to compute Nash policies in MDPs is the use of policy gradient methods . The significant progress in the analysis of such methods ( see Agarwal et al . ( 2020 ) and references therein ) has mainly concerned the single-agent case or the case of pure common interests ( identical agents ) ( Wang & Sandholm , 2002 ; Panait & Luke , 2005 ) : the convergence properties of policy gradient in general MARL settings remain poorly understood . Recent steps towards a theory for multi-agent settings involve Daskalakis et al . ( 2020 ) who show convergence of independent policy gradient to the optimal policy for two-agent zero-sum stochastic games , Wei et al . ( 2021 ) who improve the result of Daskalakis et al . ( 2020 ) using optimistic policy gradient and Zhao et al . ( 2021 ) who study extensions of Natural Policy Gradient using function approximation . It is worth noting that the positive results 1Analogue of Nash equilibrium notion . of Daskalakis et al . ( 2020 ) ; Wei et al . ( 2021 ) and Zhao et al . ( 2021 ) depend on the fact that two-agent stochastic zero-sum games satisfy the min-max equals max-min property ( Shapley , 1953 ) . If we move away from the extremes of single-agent or purely competitive settings ( two-agents , zero-sum ) , a lot of these regularities , and in particular the value-uniqueness property , cease to hold . However , building a theory to analyze problems of cooperation between two or more agents constitutes a primary open challenge for the fields of AI and ML ( Dafoe et al. , 2020 ; Dafoe et al. , 2021 ) . Based on the above , our work is motivated by the following natural question : Can we get ( provable ) convergence guarantees for multi-agent RL settings in which agents have aligned incentives , i.e. , in which coordination is desirable ? Model and Informal Statement of Results . To make progress in this direction , we study a class of n-agent MDPs that naturally generalize normal form potential games ( Monderer & Shapley , 1996 ) , the archetypal model of interactions between multiple agents with aligned , yet not necessarily identical interests , called Markov Potential Games ( MPGs ) . In words , a multi-agent MDP is an MPG as long as there exists a ( state-dependent ) real-valued potential function Φ so that if an agent i changes their policy ( and the rest of the agents keep their policy unchanged ) , the difference in agent i ’ s value/utility , V i , is captured by the difference in the value of Φ ( Definition 2 ) . Our first task is to understand the structural properties of MPGs and their Nash policies . Rather surprisingly , many insights from normal-form potential games do not carry over as MPGs involve settings with purely competitive ( zero-sum ) interactions at some states . Moreover , Markov games in which every state-interaction is a potential game are not necessarily MPGs . These findings suggest that MPGs form a class of MDPs with rich structure which challenges our intuition on the nature of cooperation in state-based interactions . On the other hand , MPGs trivially include MDPs of pure common interests ( MDPs in which agents have identical rewards ) and showcase intuitively expected properties such as the existence of deterministic Nash policies . Our structural results are as follows . Theorem 1.1 ( Structural Properties of MPGs ) . The following facts are true for MPGs with n-agents . ( a ) There always exists a deterministic Nash policy profile ( see Theorem 3.1 ) . ( b ) We can construct MDPs for which each state is a ( normal-form ) potential game but which are not MPGs . This can be true regardless of whether the whole MDP is competitive or cooperative in nature ( see Examples 1 and 2 , respectively ) . On the opposite side , we can construct MDPs that are MPGs , but which include states that are purely competitive ( i.e. , zero-sum games ) , see Figure 3 . ( c ) We provide sufficient conditions so that an MDP is an MPG . These include cases where each state is a ( normal-form ) potential game and the transition probabilities are not affected by agents actions or the reward functions satisfy certain regularity conditions between different states ( see conditions C1 and C2 in Proposition 3.2 ) . We then turn to our motivating question above and , in our main contribution , we answer it in the affirmative . We show that if every agent i independently runs ( with simultaneous updates ) projected gradient ascent ( PGA ) on their policy ( using their value V i ) , then , after O ( 1/ 2 ) iterations , the system will reach an -approximate Nash policy . Here , independence means that ( PGA ) requires only local information to determine the updates , i.e. , each agent ’ s own rewards , actions , and view of the environment . Such protocols are naturally motivated in distributed settings where all information about type of interaction and other agents ’ actions is encoded in the agent ’ s environment . For the finite samples analogue , we show that the system will reach an -approximate Nash policy after O ( 1/ 6 ) iterations . Our main convergence results are summarized in the following ( informal ) Theorem . Theorem 1.2 ( Convergence of Policy Gradient ( Informal ) ) . Consider an MPG with n agents and let > 0 . ( a ) Exact Gradients : If each agent i runs independent policy gradient using direct parameterization on their policy and the updates are simultaneous , then the learning dynamics reach an -Nash policy after O ( 1/ 2 ) iterations . ( b ) Finite Samples : If each agent i runs stochastic policy gradient using greedy parameterization ( see equation 3 ) on their policy and the updates are simultaneous , then the learning dynamics reach an -Nash policy after O ( 1/ 6 ) iterations . The formal statements for cases ( a ) and ( b ) are provided in Theorems 4.2 and 4.4 , respectively . The technical details are presented in Section 4 . The main step in the proof of Theorem 1.2 establishes that Projected Gradient Ascent ( PGA ) on the potential function generates the same dynamics as each agent running independent PGA on their value function . This follows from a straightforward derivation of an agent-wise version of the single-agent gradient domination property which can be used to show that every ( approximate ) stationary point ( Definition 4 ) of the potential function is an ( approximate ) Nash policy ( Lemma 4.1 ) . If agents do not have access to exact gradients , the key is to get an unbiased sample for the gradient of the value functions and prove that it has bounded variance ( in terms of the parameters of the MPG ) . This is established by requiring agents to perform stochastic PGA with α-greedy exploration ( equation 3 ) . The main idea is that this parameterization stays away from the boundary of the simplex throughout its trajectory ( Daskalakis et al. , 2020 ) . Other Related Works . Our paper contributes to the growing literature on cooperative AI and ML ( Carroll et al. , 2019 ; Dafoe et al. , 2020 ) . The results on convergence of MARL algorithms are scarce and largely restricted to purely competitive ( Daskalakis et al. , 2020 ; Wei et al. , 2021 ; Zhao et al. , 2021 ) or purely cooperative ( Wang & Sandholm , 2002 ; Bard et al. , 2020 ) settings . As Daskalakis et al . ( 2020 ) argue , the current frontier concerns the extension to settings that are not zero-sum , involve more than two agents and/or are cooperative in nature , albeit likely for weaker solution concepts . Our current paper proceeds precisely in this direction , and in fact , it does so without reverting to a weaker solution concept . Concerning the setup , our paper contributes to the rapidly growing literature on MPGs and variations thereof which is showcased by the works of Marden ( 2012 ) ; Valcarcel Macua et al . ( 2018 ) ; Mguni ( 2021 ) ; Mguni et al . ( 2021 ) and the partially concurrent work of Zhang et al . ( 2021 ) . Marden ( 2012 ) study Markov games that are potential at every state and which satisfy a strong additional state-transitivity property . Under the same assumption , Mguni ( 2021 ) derive an analogous result to our Theorem 3.1 on the existence of deterministic Nash policies . Valcarcel Macua et al . ( 2018 ) provide an analytical way to find a closed-loop Nash policy in a class of Markov games that is more closely related to our current MPG setting ( cf . Proposition 3.2 ) . Zhang et al . ( 2021 ) study the same class of MPGs , and present additional practical applications . By introducing the notion of averaged MDPs , they derive an alternative , model-based policy evaluation method , which , interestingly , establishes the same sample-complexity ( of O ( −6 ) ) as our model-free estimation . As pointed out by Zhang et al . ( 2021 ) , it will be instructive to further explore this connection . Concerning the technical parts , our methods are related to Daskalakis et al . ( 2020 ) ; Agarwal et al . ( 2020 ) ; Kakade & Langford ( 2002 ) and to Davis & Drusvyatskiy ( 2018 ) ; Bubeck ( 2015 ) ; Nemirovski et al . ( 2009 ) . 2 PRELIMINARIES . Markov Decision Processes ( MDPs ) . We consider a setting with n agents who select actions in a shared Markov Decision Process ( MDP ) . Formally , an MDP is a tuple G = ( S , N , { Ai , Ri } i∈N , P , γ , ρ ) , where S is a finite state space of size S = |S| , N = { 1 , 2 , . . . , n } is the set of agents , and Ai is a finite action space of size Ai = |Ai| for each agent i ∈ N with generic element ai ∈ Ai . We write A = ∏ i∈N Ai and A−i = ∏ j 6=iAj to denote the joint action spaces of all agents and of all agents other than i with generic elements a = ( ai ) i∈N and a−i = ( aj ) j 6=i , respectively . Ri : S × A → [ −1 , 1 ] is the reward function of agent i ∈ N , i.e. , Ri ( s , ai , a−i ) is the instantaneous reward of agent i when they take action ai and all other agents take actions a−i at state s ∈ S. P is the transition probability function , for which P ( s′ | s , a ) is the probability of transitioning from s to s′ when a ∈ A is chosen by the agents . Finally , γ ( same for all agents ) is a discount factor for future rewards and ρ ∈ ∆ ( S ) is a distribution for the initial state at time t = 0.2 Whenever time is relevant , we will index the above terms with t. In particular , at each time step t ≥ 0 , all agents i ∈ N observe the state st ∈ S , select actions at = ( ai , t , a−i , t ) , receive rewards ri , t : = Ri ( st , at ) , and transition to the next state st+1 ∼ P ( · | st , at ) . We will write τ = ( st , at , rt ) t≥0 to denote the trajectories of the system , where rt : = ( ri , t ) , i ∈ N . Policies and Value Functions . For each agent i ∈ N , a deterministic , stationary policy πi : S → Ai specifies the action of agent i at each state s ∈ S , i.e. , πi ( s ) = ai ∈ Ai for each s ∈ S. A stochastic , stationary policy πi : S → ∆ ( Ai ) specifies a probability distribution over the actions of agent i for each state s ∈ S. Accordingly , πi ∈ Πi : = ∆ ( Ai ) S . In this case , we will write ai ∼ πi ( · | s ) to denote the randomized action of agent i at state s ∈ S. As above , we will write π = ( πi ) i∈N ∈ Π : = ×i∈N∆ ( Ai ) S and π−i = ( πj ) i 6=j∈N ∈ Π−i : = ×i6=j∈N∆ ( Aj ) S to denote the joint policies of all agents and of all agents other than i , respectively . A joint policy π induces a distribution Prπ over trajectories τ = ( st , at , rt ) t≥0 , where s0 is drawn from the initial state distribution ρ and ai , t is drawn from πi ( · | st ) for all i ∈ N . 2We will write ∆ ( X ) to denote the set of probability distributions over any set X . The value function , V is : Π → R , gives the expected reward of agent i ∈ N when s0 = s and the agents draw their actions , at = ( ai , t , a−i , t ) , at time t ≥ 0 from policy π = ( πi , π−i ) V is ( π ) : = Eπ [ ∑∞ t=0 γtri , t | s0 = s ] . ( 1 ) We also write V iρ ( π ) = Es∼ρ [ V is ( π ) ] if the initial state is random and follows distribution ρ . The solution concept that we will be focusing on are the Nash Policies which are formally defined next . Definition 1 ( -Nash Policy ) . A joint policy π∗ = ( π∗i ) i∈N is an -Nash policy if there exists an ≥ 0 so that for each agent i ∈ N , V is ( π∗i , π∗−i ) ≥ V is ( πi , π∗−i ) − , for all πi ∈ ∆ ( Ai ) S , and all s ∈ S. If = 0 , then π∗ is a called a Nash policy . In this case , π∗i maximizes each agent i ’ s value function for each starting state s ∈ S given the policies , π∗−i = ( π∗j ) j 6=i , of all other agents j 6= i ∈ N . The definition of a Nash policy remains the same if s ∼ ρ ( random starting state ) .

Summary: In this paper, the authors present a new definition of Markov Potential Games (MPG). They discuss the relationship between MPG, normal form potential game, and Markov games where every state-game is a potential game.  They study MPGs by presenting sufﬁcient conditions for MPGs and the existence of deterministic Nash policies.  They show the conference of independent-playing policy gradient to Nash policies, in both exact gradient case and finite sample case. Experiment results validate the congergence.