Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning

1 INTRODUCTION . Policy gradient ( PG ) methods have played a major role in recent developments of reinforcement learning ( RL ) algorithms ( Silver et al. , 2014 ; Schulman et al. , 2015a ; Haarnoja et al. , 2018 ) . Among the many PG variants , trust region learning ( Kakade & Langford , 2002 ) , with two typical embodiments of Trust Region Policy Optimisation ( TRPO ) ( Schulman et al. , 2015a ) and Proximal Policy Optimisation ( PPO ) ( Schulman et al. , 2017 ) algorithms , offer supreme empirical performance in both discrete and continuous RL problems ( Duan et al. , 2016 ; Mahmood et al. , 2018 ) . The effectiveness of trust region methods largely stems from their theoretically-justified policy iteration procedure . By optimising the policy within a trustable neighbourhood of the current policy , thus avoiding making aggressive updates towards risky directions , trust region learning enjoys the guarantee of monotonic performance improvement at every iteration . In multi-agent reinforcement learning ( MARL ) settings ( Yang & Wang , 2020 ) , naively applying policy gradient methods by considering other agents as a part of the environment can lose its effectiveness . This is intuitively clear : once a learning agent updates its policy , so do its opponents ; this however changes the loss landscape of the learning agent , thus harming the improvement effect from the PG update . As a result , applying independent PG updates in MARL offers poor convergence property ( Claus & Boutilier , 1998 ) . To address this , a learning paradigm named centralised training with decentralised execution ( CTDE ) ( Lowe et al. , 2017b ; Foerster et al. , 2018 ; Zhou et al. , 2021 ) was developed . In CTDE , each agent is equipped with a joint value function which , during ∗First two authors contribute equally . Code is available at https : //github.com/cyanrain7/ Trust-Region-Policy-Optimisation-in-Multi-Agent-Reinforcement-Learning . training , has access to the global state and opponents ’ actions . With the help of the centralised value function that accounts for the non-stationarity caused by others , each agent adapts its policy parameters accordingly . As such , the CTDE paradigm allows a straightforward extension of single-agent PG theorems ( Sutton et al. , 2000 ; Silver et al. , 2014 ) to multi-agent scenarios ( Lowe et al. , 2017b ; Kuba et al. , 2021 ; Mguni et al. , 2021 ) . Consequently , fruitful multi-agent policy gradient algorithms have been developed ( Foerster et al. , 2018 ; Peng et al. , 2017 ; Zhang et al. , 2020 ; Wen et al. , 2018 ; 2020 ; Yang et al. , 2018 ) . Unfortunately , existing CTDE methods offer no solution of how to perform trust region learning in MARL . Lack of such an extension impedes agents from learning monotonically improving policies in a stable manner . Recent attempts such as IPPO ( de Witt et al. , 2020a ) and MAPPO ( Yu et al. , 2021 ) have been proposed to fill such a gap ; however , these methods are designed for agents that are homogeneous ( i.e. , sharing the same action space and policy parameters ) , which largely limits their applicability and potentially harm the performance . As we show in Proposition 1 later , parameter sharing could suffer from an exponentially-worse suboptimal outcome . On the other hand , although IPPO/MAPPO can be practically applied in a non-parameter sharing way , it still lacks the essential theoretical property of trust region learning , which is the monotonic improvement guarantee . In this paper , we propose the first theoretically-justified trust region learning framework in MARL . The key to our findings are the multi-agent advantage decomposition lemma and the sequential policy update scheme . With the advantage decomposition lemma , we introduce a multi-agent policy iteration procedure that enjoys the monotonic improvement guarantee . To implement such a procedure , we propose two practical algorithms : Heterogeneous-Agent Trust Region Policy Optimisation ( HATRPO ) and Heterogeneous-Agent Proximal Policy Optimisation ( HAPPO ) . HATRPO/HAPPO adopts the sequential update scheme , which saves the cost of maintaining a centralised critic for each agent in CTDE . Importantly , HATRPO/HAPPO does not require homogeneity of agents , nor any other restrictive assumptions on the decomposibility of the joint Q-function ( Rashid et al. , 2018 ) . We evaluate HATRPO and HAPPO on benchmarks of StarCraftII and Multi-Agent MuJoCo against strong baselines such as MADDPG ( Lowe et al. , 2017a ) , IPPO ( de Witt et al. , 2020b ) and MAPPO ( Yu et al. , 2021 ) ; results clearly demonstrate its state-of-the-art performance across all tested tasks . 2 PRELIMINARIES . In this section , we first introduce problem formulation and notations for MARL , and then briefly review trust region learning in RL and discuss the difficulty of extending it to MARL . We end by surveying existing MARL work that relates to trust region methods and show their limitations . 2.1 COOPERATIVE MARL PROBLEM FORMULATION AND NOTATIONS . We consider a Markov game ( Littman , 1994 ) , which is defined by a tuple 〈N , S , A , P , r , γ〉 . Here , N = { 1 , . . . , n } denotes the set of agents , S is the finite state space , A = ∏n i=1Ai is the product of finite action spaces of all agents , known as the joint action space , P : S ×A× S → [ 0 , 1 ] is the transition probability function , r : S ×A→ R is the reward function , and γ ∈ [ 0 , 1 ) is the discount factor . The agents interact with the environment according to the following protocol : at time step t ∈ N , the agents are at state st ∈ S ; every agent i takes an action ait ∈ Ai , drawn from its policy πi ( ·|st ) , which together with other agents ’ actions gives a joint action at = ( a1t , . . . , a n t ) ∈ A , drawn from the joint policy π ( ·|st ) = ∏n i=1 π i ( ·i|st ) ; the agents receive a joint reward rt = r ( st , at ) ∈ R , and move to a state st+1 with probability P ( st+1|st , at ) . The joint policy π , the transition probabililty function P , and the initial state distribution ρ0 , induce a marginal state distribution at time t , denoted by ρtπ . We define an ( improper ) marginal state distribution ρπ , ∑∞ t=0 γ tρtπ . The state value function and the state-action value function are defined : Vπ ( s ) , Ea0 : ∞∼π , s1 : ∞∼P [ ∑∞ t=0 γ trt ∣∣ s0 = s ] and Qπ ( s , a ) , Es1 : ∞∼P , a1 : ∞∼π [ ∑∞ t=0 γ trt ∣∣ s0 = s , a0 = a ] . The advantage function is written as Aπ ( s , a ) , Qπ ( s , a ) − Vπ ( s ) . In this paper , we consider a fully-cooperative setting where all agents share the same reward function , aiming to maximise the expected total reward : J ( π ) , Es0 : ∞∼ρ0 : ∞π , a0 : ∞∼π [ ∞∑ t=0 γtrt ] . Throughout this paper , we pay close attention to the contribution to performance from different subsets of agents . Before proceeding to our methods , we introduce following novel definitions . Definition 1 . Let i1 : m denote an ordered subset { i1 , . . . , im } of N , and let −i1 : m refer to its complement . We write ik when we refer to the kth agent in the ordered subset . Correspondingly , the multi-agent state-action value function is defined as Qi1 : mπ ( s , ai1 : m ) , Ea−i1 : m∼π−i1 : m [ Qπ ( s , ai1 : m , a−i1 : m ) ] , and for disjoint sets j1 : k and i1 : m , the multi-agent advantage function is Ai1 : mπ ( s , aj1 : k , ai1 : m ) , Qj1 : k , i1 : mπ ( s , aj1 : k , ai1 : m ) −Qj1 : kπ ( s , aj1 : k ) . ( 1 ) Hereafter , the joint policies π = ( π1 , . . . , πn ) and π̄ = ( π̄1 , . . . , π̄n ) shall be thought of as the “ current ” , and the “ new ” joint policy that agents update towards , respectively . 2.2 TRUST REGION ALGORITHMS IN REINFORCEMENT LEARNING . Trust region methods such as TRPO ( Schulman et al. , 2015a ) were proposed in single-agent RL with an aim of achieving a monotonic improvement of J ( π ) at each iteration . Formally , it can be described by the following theorem . Theorem 1 . ( Schulman et al. , 2015a , Theorem 1 ) Let π be the current policy and π̄ be the next candidate policy . We define Lπ ( π̄ ) = J ( π ) + Es∼ρπ , a∼π̄ [ Aπ ( s , a ) ] , DmaxKL ( π , π̄ ) = maxs DKL ( π ( ·|s ) , π̄ ( ·|s ) ) . Then the inequality of J ( π̄ ) ≥ Lπ ( π̄ ) − CDmaxKL ( π , π̄ ) ( 2 ) holds , where C = 4γmaxs , a |Aπ ( s , a ) | ( 1−γ ) 2 . The above theorem states that as the distance between the current policy π and a candidate policy π̄ decreases , the surrogate Lπ ( π̄ ) , which involves only the current policy ’ s state distribution , becomes an increasingly accurate estimate of the actual performance metric J ( π̄ ) . Based on this theorem , an iterative trust region algorithm is derived ; at iteration k + 1 , the agent updates its policy by πk+1 = arg max π ( Lπk ( π ) − CDmaxKL ( πk , π ) ) . Such an update guarantees a monotonic improvement of the policy , i.e. , J ( πk+1 ) ≥ J ( πk ) . To implement this procedure in practical settings with parameterised policies πθ , Schulman et al . ( 2015a ) approximated the KL-penalty with a KL-constraint , which gave rise to the TRPO update of θk+1 = arg max θ Lπθk ( πθ ) , subject to Es∼ρπθk [ DKL ( πθk , πθ ) ] ≤ δ . ( 3 ) At each iteration k + 1 , TRPO constructs a KL-ball Bδ ( πθk ) around the policy πθk , and optimises πθk+1 ∈ Bδ ( πθk ) to maximise Lπθk ( πθ ) . By Theorem 1 , we know that the surrogate objective Lπθk ( πθ ) is close to the true reward J ( πθ ) within Bδ ( πθk ) ; therefore , πθk leads to improvement . Furthermore , to save the cost on Es∼ρπθk [ DKL ( πθk , πθ ) ] when computing Equation ( 3 ) , Schulman et al . ( 2017 ) proposed an approximation solution to TRPO that uses only first order derivatives , known as PPO . PPO optimises the policy parameter θk+1 by maximising the PPO-clip objective of LPPOπθk ( πθ ) = Es∼ρπθk , a∼πθk [ min ( πθ ( a|s ) πθk ( a|s ) Aπθk ( s , a ) , clip ( πθ ( a|s ) πθk ( a|s ) , 1± ) Aπθk ( s , a ) ) ] . ( 4 ) The clip operator replaces the ratio πθ ( a|s ) πθk ( a|s ) with 1+ or 1− , depending on whether or not the ratio is beyond the threshold interval . This effectively enables PPO to control the size of policy updates .

Summary: This paper proposes two algorithms, HATRPO and HAPPO, which extend TRPO and PPO to cooperative multi-agent settings, respectively. Unlike existing MARL algorithms such as IPPO and MAPPO, HATRPO/HAPPO can guarantee monotonic policy improvements,  taking advantage of the proposed multi-agent advantage decomposition lemma and sequential policy update scheme. In addition, HATRPO/HAPPO can be applied to heterogeneous agents and does not require the joint value function being decomposable. Experimental results on Mulit-Agent MuJoCo demonstrate the competitive performance of the proposed methods.