Global Convergence of Multi-Agent Policy Gradient in Markov Potential Games

1 INTRODUCTION . Multi-agent reinforcement learning ( MARL ) has been the fundamental driver of numerous recent advances in Artificial Intelligence ( AI ) and Machine Learning ( ML ) ranging from super-human performance in competitive game-playing ( Silver et al. , 2016 ; 2018 ; Brown & Sandholm , 2019 ; Jaderberg et al. , 2019 ) and multi-tasking ( Mnih et al. , 2015 ; OpenAI , 2018 ; Vinyals et al. , 2019 ) to robotics , autonomous-driving and cyber-physical systems ( Busoniu et al. , 2008 ; Zhang et al. , 2019 ) . However , despite the popularity of MARL algorithms to analyze these systems in practice , the theory that underpins their empirical success lags behind ( Dafoe et al. , 2020 ) . Many state-of-the-art theoretical results concern single-agent RL systems , typically modelled as single-agent Markov Decision Processes ( MDPs ) ( Bertsekas , 2000 ; Panait & Luke , 2005 ; Sutton & Barto , 2018 ) . The main challenge when transitioning from single to multi-agent RL settings is the computation of Nash policies . For n ≥ 2 agents , a Nash policy is defined to be a profile of policies ( π∗1 , ... , π∗n ) so that by fixing the policies of all agents but i , π∗i is optimal for the resulting single-agent MDP and this is true for all 1 ≤ i ≤ n 1 ( Definition 1 ) . In multi-agent settings , Nash policies may not be unique in principle and , unlike singe-agent MDPs , agents ’ rewards may differ dramatically between them . A common approach to compute Nash policies in MDPs is the use of policy gradient methods . The significant progress in the analysis of such methods ( see Agarwal et al . ( 2020 ) and references therein ) has mainly concerned the single-agent case or the case of pure common interests ( identical agents ) ( Wang & Sandholm , 2002 ; Panait & Luke , 2005 ) : the convergence properties of policy gradient in general MARL settings remain poorly understood . Recent steps towards a theory for multi-agent settings involve Daskalakis et al . ( 2020 ) who show convergence of independent policy gradient to the optimal policy for two-agent zero-sum stochastic games , Wei et al . ( 2021 ) who improve the result of Daskalakis et al . ( 2020 ) using optimistic policy gradient and Zhao et al . ( 2021 ) who study extensions of Natural Policy Gradient using function approximation . It is worth noting that the positive results 1Analogue of Nash equilibrium notion . of Daskalakis et al . ( 2020 ) ; Wei et al . ( 2021 ) and Zhao et al . ( 2021 ) depend on the fact that two-agent stochastic zero-sum games satisfy the min-max equals max-min property ( Shapley , 1953 ) . If we move away from the extremes of single-agent or purely competitive settings ( two-agents , zero-sum ) , a lot of these regularities , and in particular the value-uniqueness property , cease to hold . However , building a theory to analyze problems of cooperation between two or more agents constitutes a primary open challenge for the fields of AI and ML ( Dafoe et al. , 2020 ; Dafoe et al. , 2021 ) . Based on the above , our work is motivated by the following natural question : Can we get ( provable ) convergence guarantees for multi-agent RL settings in which agents have aligned incentives , i.e. , in which coordination is desirable ? Model and Informal Statement of Results . To make progress in this direction , we study a class of n-agent MDPs that naturally generalize normal form potential games ( Monderer & Shapley , 1996 ) , the archetypal model of interactions between multiple agents with aligned , yet not necessarily identical interests , called Markov Potential Games ( MPGs ) . In words , a multi-agent MDP is an MPG as long as there exists a ( state-dependent ) real-valued potential function Φ so that if an agent i changes their policy ( and the rest of the agents keep their policy unchanged ) , the difference in agent i ’ s value/utility , V i , is captured by the difference in the value of Φ ( Definition 2 ) . Our first task is to understand the structural properties of MPGs and their Nash policies . Rather surprisingly , many insights from normal-form potential games do not carry over as MPGs involve settings with purely competitive ( zero-sum ) interactions at some states . Moreover , Markov games in which every state-interaction is a potential game are not necessarily MPGs . These findings suggest that MPGs form a class of MDPs with rich structure which challenges our intuition on the nature of cooperation in state-based interactions . On the other hand , MPGs trivially include MDPs of pure common interests ( MDPs in which agents have identical rewards ) and showcase intuitively expected properties such as the existence of deterministic Nash policies . Our structural results are as follows . Theorem 1.1 ( Structural Properties of MPGs ) . The following facts are true for MPGs with n-agents . ( a ) There always exists a deterministic Nash policy profile ( see Theorem 3.1 ) . ( b ) We can construct MDPs for which each state is a ( normal-form ) potential game but which are not MPGs . This can be true regardless of whether the whole MDP is competitive or cooperative in nature ( see Examples 1 and 2 , respectively ) . On the opposite side , we can construct MDPs that are MPGs , but which include states that are purely competitive ( i.e. , zero-sum games ) , see Figure 3 . ( c ) We provide sufficient conditions so that an MDP is an MPG . These include cases where each state is a ( normal-form ) potential game and the transition probabilities are not affected by agents actions or the reward functions satisfy certain regularity conditions between different states ( see conditions C1 and C2 in Proposition 3.2 ) . We then turn to our motivating question above and , in our main contribution , we answer it in the affirmative . We show that if every agent i independently runs ( with simultaneous updates ) projected gradient ascent ( PGA ) on their policy ( using their value V i ) , then , after O ( 1/ 2 ) iterations , the system will reach an -approximate Nash policy . Here , independence means that ( PGA ) requires only local information to determine the updates , i.e. , each agent ’ s own rewards , actions , and view of the environment . Such protocols are naturally motivated in distributed settings where all information about type of interaction and other agents ’ actions is encoded in the agent ’ s environment . For the finite samples analogue , we show that the system will reach an -approximate Nash policy after O ( 1/ 6 ) iterations . Our main convergence results are summarized in the following ( informal ) Theorem . Theorem 1.2 ( Convergence of Policy Gradient ( Informal ) ) . Consider an MPG with n agents and let > 0 . ( a ) Exact Gradients : If each agent i runs independent policy gradient using direct parameterization on their policy and the updates are simultaneous , then the learning dynamics reach an -Nash policy after O ( 1/ 2 ) iterations . ( b ) Finite Samples : If each agent i runs stochastic policy gradient using greedy parameterization ( see equation 3 ) on their policy and the updates are simultaneous , then the learning dynamics reach an -Nash policy after O ( 1/ 6 ) iterations . The formal statements for cases ( a ) and ( b ) are provided in Theorems 4.2 and 4.4 , respectively . The technical details are presented in Section 4 . The main step in the proof of Theorem 1.2 establishes that Projected Gradient Ascent ( PGA ) on the potential function generates the same dynamics as each agent running independent PGA on their value function . This follows from a straightforward derivation of an agent-wise version of the single-agent gradient domination property which can be used to show that every ( approximate ) stationary point ( Definition 4 ) of the potential function is an ( approximate ) Nash policy ( Lemma 4.1 ) . If agents do not have access to exact gradients , the key is to get an unbiased sample for the gradient of the value functions and prove that it has bounded variance ( in terms of the parameters of the MPG ) . This is established by requiring agents to perform stochastic PGA with α-greedy exploration ( equation 3 ) . The main idea is that this parameterization stays away from the boundary of the simplex throughout its trajectory ( Daskalakis et al. , 2020 ) . Other Related Works . Our paper contributes to the growing literature on cooperative AI and ML ( Carroll et al. , 2019 ; Dafoe et al. , 2020 ) . The results on convergence of MARL algorithms are scarce and largely restricted to purely competitive ( Daskalakis et al. , 2020 ; Wei et al. , 2021 ; Zhao et al. , 2021 ) or purely cooperative ( Wang & Sandholm , 2002 ; Bard et al. , 2020 ) settings . As Daskalakis et al . ( 2020 ) argue , the current frontier concerns the extension to settings that are not zero-sum , involve more than two agents and/or are cooperative in nature , albeit likely for weaker solution concepts . Our current paper proceeds precisely in this direction , and in fact , it does so without reverting to a weaker solution concept . Concerning the setup , our paper contributes to the rapidly growing literature on MPGs and variations thereof which is showcased by the works of Marden ( 2012 ) ; Valcarcel Macua et al . ( 2018 ) ; Mguni ( 2021 ) ; Mguni et al . ( 2021 ) and the partially concurrent work of Zhang et al . ( 2021 ) . Marden ( 2012 ) study Markov games that are potential at every state and which satisfy a strong additional state-transitivity property . Under the same assumption , Mguni ( 2021 ) derive an analogous result to our Theorem 3.1 on the existence of deterministic Nash policies . Valcarcel Macua et al . ( 2018 ) provide an analytical way to find a closed-loop Nash policy in a class of Markov games that is more closely related to our current MPG setting ( cf . Proposition 3.2 ) . Zhang et al . ( 2021 ) study the same class of MPGs , and present additional practical applications . By introducing the notion of averaged MDPs , they derive an alternative , model-based policy evaluation method , which , interestingly , establishes the same sample-complexity ( of O ( −6 ) ) as our model-free estimation . As pointed out by Zhang et al . ( 2021 ) , it will be instructive to further explore this connection . Concerning the technical parts , our methods are related to Daskalakis et al . ( 2020 ) ; Agarwal et al . ( 2020 ) ; Kakade & Langford ( 2002 ) and to Davis & Drusvyatskiy ( 2018 ) ; Bubeck ( 2015 ) ; Nemirovski et al . ( 2009 ) . 2 PRELIMINARIES . Markov Decision Processes ( MDPs ) . We consider a setting with n agents who select actions in a shared Markov Decision Process ( MDP ) . Formally , an MDP is a tuple G = ( S , N , { Ai , Ri } i∈N , P , γ , ρ ) , where S is a finite state space of size S = |S| , N = { 1 , 2 , . . . , n } is the set of agents , and Ai is a finite action space of size Ai = |Ai| for each agent i ∈ N with generic element ai ∈ Ai . We write A = ∏ i∈N Ai and A−i = ∏ j 6=iAj to denote the joint action spaces of all agents and of all agents other than i with generic elements a = ( ai ) i∈N and a−i = ( aj ) j 6=i , respectively . Ri : S × A → [ −1 , 1 ] is the reward function of agent i ∈ N , i.e. , Ri ( s , ai , a−i ) is the instantaneous reward of agent i when they take action ai and all other agents take actions a−i at state s ∈ S. P is the transition probability function , for which P ( s′ | s , a ) is the probability of transitioning from s to s′ when a ∈ A is chosen by the agents . Finally , γ ( same for all agents ) is a discount factor for future rewards and ρ ∈ ∆ ( S ) is a distribution for the initial state at time t = 0.2 Whenever time is relevant , we will index the above terms with t. In particular , at each time step t ≥ 0 , all agents i ∈ N observe the state st ∈ S , select actions at = ( ai , t , a−i , t ) , receive rewards ri , t : = Ri ( st , at ) , and transition to the next state st+1 ∼ P ( · | st , at ) . We will write τ = ( st , at , rt ) t≥0 to denote the trajectories of the system , where rt : = ( ri , t ) , i ∈ N . Policies and Value Functions . For each agent i ∈ N , a deterministic , stationary policy πi : S → Ai specifies the action of agent i at each state s ∈ S , i.e. , πi ( s ) = ai ∈ Ai for each s ∈ S. A stochastic , stationary policy πi : S → ∆ ( Ai ) specifies a probability distribution over the actions of agent i for each state s ∈ S. Accordingly , πi ∈ Πi : = ∆ ( Ai ) S . In this case , we will write ai ∼ πi ( · | s ) to denote the randomized action of agent i at state s ∈ S. As above , we will write π = ( πi ) i∈N ∈ Π : = ×i∈N∆ ( Ai ) S and π−i = ( πj ) i 6=j∈N ∈ Π−i : = ×i6=j∈N∆ ( Aj ) S to denote the joint policies of all agents and of all agents other than i , respectively . A joint policy π induces a distribution Prπ over trajectories τ = ( st , at , rt ) t≥0 , where s0 is drawn from the initial state distribution ρ and ai , t is drawn from πi ( · | st ) for all i ∈ N . 2We will write ∆ ( X ) to denote the set of probability distributions over any set X . The value function , V is : Π → R , gives the expected reward of agent i ∈ N when s0 = s and the agents draw their actions , at = ( ai , t , a−i , t ) , at time t ≥ 0 from policy π = ( πi , π−i ) V is ( π ) : = Eπ [ ∑∞ t=0 γtri , t | s0 = s ] . ( 1 ) We also write V iρ ( π ) = Es∼ρ [ V is ( π ) ] if the initial state is random and follows distribution ρ . The solution concept that we will be focusing on are the Nash Policies which are formally defined next . Definition 1 ( -Nash Policy ) . A joint policy π∗ = ( π∗i ) i∈N is an -Nash policy if there exists an ≥ 0 so that for each agent i ∈ N , V is ( π∗i , π∗−i ) ≥ V is ( πi , π∗−i ) − , for all πi ∈ ∆ ( Ai ) S , and all s ∈ S. If = 0 , then π∗ is a called a Nash policy . In this case , π∗i maximizes each agent i ’ s value function for each starting state s ∈ S given the policies , π∗−i = ( π∗j ) j 6=i , of all other agents j 6= i ∈ N . The definition of a Nash policy remains the same if s ∼ ρ ( random starting state ) .

Summary: The paper introduces a novel class of Markov Games that generalize classical potential games in the normal-form representation. They show that such games always admit deterministic Nash policies, and that they enjoy some interesting properties that they make them a more suitable generalization of potential games than other attempts in the literature. The main result of the paper is that, in Markov Potential Games, policy gradient algorithms converge to Nash. The authors also provide an experimental analysis on congestion games.