Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning

1 INTRODUCTION . Policy gradient ( PG ) methods have played a major role in recent developments of reinforcement learning ( RL ) algorithms ( Silver et al. , 2014 ; Schulman et al. , 2015a ; Haarnoja et al. , 2018 ) . Among the many PG variants , trust region learning ( Kakade & Langford , 2002 ) , with two typical embodiments of Trust Region Policy Optimisation ( TRPO ) ( Schulman et al. , 2015a ) and Proximal Policy Optimisation ( PPO ) ( Schulman et al. , 2017 ) algorithms , offer supreme empirical performance in both discrete and continuous RL problems ( Duan et al. , 2016 ; Mahmood et al. , 2018 ) . The effectiveness of trust region methods largely stems from their theoretically-justified policy iteration procedure . By optimising the policy within a trustable neighbourhood of the current policy , thus avoiding making aggressive updates towards risky directions , trust region learning enjoys the guarantee of monotonic performance improvement at every iteration . In multi-agent reinforcement learning ( MARL ) settings ( Yang & Wang , 2020 ) , naively applying policy gradient methods by considering other agents as a part of the environment can lose its effectiveness . This is intuitively clear : once a learning agent updates its policy , so do its opponents ; this however changes the loss landscape of the learning agent , thus harming the improvement effect from the PG update . As a result , applying independent PG updates in MARL offers poor convergence property ( Claus & Boutilier , 1998 ) . To address this , a learning paradigm named centralised training with decentralised execution ( CTDE ) ( Lowe et al. , 2017b ; Foerster et al. , 2018 ; Zhou et al. , 2021 ) was developed . In CTDE , each agent is equipped with a joint value function which , during ∗First two authors contribute equally . Code is available at https : //github.com/cyanrain7/ Trust-Region-Policy-Optimisation-in-Multi-Agent-Reinforcement-Learning . training , has access to the global state and opponents ’ actions . With the help of the centralised value function that accounts for the non-stationarity caused by others , each agent adapts its policy parameters accordingly . As such , the CTDE paradigm allows a straightforward extension of single-agent PG theorems ( Sutton et al. , 2000 ; Silver et al. , 2014 ) to multi-agent scenarios ( Lowe et al. , 2017b ; Kuba et al. , 2021 ; Mguni et al. , 2021 ) . Consequently , fruitful multi-agent policy gradient algorithms have been developed ( Foerster et al. , 2018 ; Peng et al. , 2017 ; Zhang et al. , 2020 ; Wen et al. , 2018 ; 2020 ; Yang et al. , 2018 ) . Unfortunately , existing CTDE methods offer no solution of how to perform trust region learning in MARL . Lack of such an extension impedes agents from learning monotonically improving policies in a stable manner . Recent attempts such as IPPO ( de Witt et al. , 2020a ) and MAPPO ( Yu et al. , 2021 ) have been proposed to fill such a gap ; however , these methods are designed for agents that are homogeneous ( i.e. , sharing the same action space and policy parameters ) , which largely limits their applicability and potentially harm the performance . As we show in Proposition 1 later , parameter sharing could suffer from an exponentially-worse suboptimal outcome . On the other hand , although IPPO/MAPPO can be practically applied in a non-parameter sharing way , it still lacks the essential theoretical property of trust region learning , which is the monotonic improvement guarantee . In this paper , we propose the first theoretically-justified trust region learning framework in MARL . The key to our findings are the multi-agent advantage decomposition lemma and the sequential policy update scheme . With the advantage decomposition lemma , we introduce a multi-agent policy iteration procedure that enjoys the monotonic improvement guarantee . To implement such a procedure , we propose two practical algorithms : Heterogeneous-Agent Trust Region Policy Optimisation ( HATRPO ) and Heterogeneous-Agent Proximal Policy Optimisation ( HAPPO ) . HATRPO/HAPPO adopts the sequential update scheme , which saves the cost of maintaining a centralised critic for each agent in CTDE . Importantly , HATRPO/HAPPO does not require homogeneity of agents , nor any other restrictive assumptions on the decomposibility of the joint Q-function ( Rashid et al. , 2018 ) . We evaluate HATRPO and HAPPO on benchmarks of StarCraftII and Multi-Agent MuJoCo against strong baselines such as MADDPG ( Lowe et al. , 2017a ) , IPPO ( de Witt et al. , 2020b ) and MAPPO ( Yu et al. , 2021 ) ; results clearly demonstrate its state-of-the-art performance across all tested tasks . 2 PRELIMINARIES . In this section , we first introduce problem formulation and notations for MARL , and then briefly review trust region learning in RL and discuss the difficulty of extending it to MARL . We end by surveying existing MARL work that relates to trust region methods and show their limitations . 2.1 COOPERATIVE MARL PROBLEM FORMULATION AND NOTATIONS . We consider a Markov game ( Littman , 1994 ) , which is defined by a tuple 〈N , S , A , P , r , γ〉 . Here , N = { 1 , . . . , n } denotes the set of agents , S is the finite state space , A = ∏n i=1Ai is the product of finite action spaces of all agents , known as the joint action space , P : S ×A× S → [ 0 , 1 ] is the transition probability function , r : S ×A→ R is the reward function , and γ ∈ [ 0 , 1 ) is the discount factor . The agents interact with the environment according to the following protocol : at time step t ∈ N , the agents are at state st ∈ S ; every agent i takes an action ait ∈ Ai , drawn from its policy πi ( ·|st ) , which together with other agents ’ actions gives a joint action at = ( a1t , . . . , a n t ) ∈ A , drawn from the joint policy π ( ·|st ) = ∏n i=1 π i ( ·i|st ) ; the agents receive a joint reward rt = r ( st , at ) ∈ R , and move to a state st+1 with probability P ( st+1|st , at ) . The joint policy π , the transition probabililty function P , and the initial state distribution ρ0 , induce a marginal state distribution at time t , denoted by ρtπ . We define an ( improper ) marginal state distribution ρπ , ∑∞ t=0 γ tρtπ . The state value function and the state-action value function are defined : Vπ ( s ) , Ea0 : ∞∼π , s1 : ∞∼P [ ∑∞ t=0 γ trt ∣∣ s0 = s ] and Qπ ( s , a ) , Es1 : ∞∼P , a1 : ∞∼π [ ∑∞ t=0 γ trt ∣∣ s0 = s , a0 = a ] . The advantage function is written as Aπ ( s , a ) , Qπ ( s , a ) − Vπ ( s ) . In this paper , we consider a fully-cooperative setting where all agents share the same reward function , aiming to maximise the expected total reward : J ( π ) , Es0 : ∞∼ρ0 : ∞π , a0 : ∞∼π [ ∞∑ t=0 γtrt ] . Throughout this paper , we pay close attention to the contribution to performance from different subsets of agents . Before proceeding to our methods , we introduce following novel definitions . Definition 1 . Let i1 : m denote an ordered subset { i1 , . . . , im } of N , and let −i1 : m refer to its complement . We write ik when we refer to the kth agent in the ordered subset . Correspondingly , the multi-agent state-action value function is defined as Qi1 : mπ ( s , ai1 : m ) , Ea−i1 : m∼π−i1 : m [ Qπ ( s , ai1 : m , a−i1 : m ) ] , and for disjoint sets j1 : k and i1 : m , the multi-agent advantage function is Ai1 : mπ ( s , aj1 : k , ai1 : m ) , Qj1 : k , i1 : mπ ( s , aj1 : k , ai1 : m ) −Qj1 : kπ ( s , aj1 : k ) . ( 1 ) Hereafter , the joint policies π = ( π1 , . . . , πn ) and π̄ = ( π̄1 , . . . , π̄n ) shall be thought of as the “ current ” , and the “ new ” joint policy that agents update towards , respectively . 2.2 TRUST REGION ALGORITHMS IN REINFORCEMENT LEARNING . Trust region methods such as TRPO ( Schulman et al. , 2015a ) were proposed in single-agent RL with an aim of achieving a monotonic improvement of J ( π ) at each iteration . Formally , it can be described by the following theorem . Theorem 1 . ( Schulman et al. , 2015a , Theorem 1 ) Let π be the current policy and π̄ be the next candidate policy . We define Lπ ( π̄ ) = J ( π ) + Es∼ρπ , a∼π̄ [ Aπ ( s , a ) ] , DmaxKL ( π , π̄ ) = maxs DKL ( π ( ·|s ) , π̄ ( ·|s ) ) . Then the inequality of J ( π̄ ) ≥ Lπ ( π̄ ) − CDmaxKL ( π , π̄ ) ( 2 ) holds , where C = 4γmaxs , a |Aπ ( s , a ) | ( 1−γ ) 2 . The above theorem states that as the distance between the current policy π and a candidate policy π̄ decreases , the surrogate Lπ ( π̄ ) , which involves only the current policy ’ s state distribution , becomes an increasingly accurate estimate of the actual performance metric J ( π̄ ) . Based on this theorem , an iterative trust region algorithm is derived ; at iteration k + 1 , the agent updates its policy by πk+1 = arg max π ( Lπk ( π ) − CDmaxKL ( πk , π ) ) . Such an update guarantees a monotonic improvement of the policy , i.e. , J ( πk+1 ) ≥ J ( πk ) . To implement this procedure in practical settings with parameterised policies πθ , Schulman et al . ( 2015a ) approximated the KL-penalty with a KL-constraint , which gave rise to the TRPO update of θk+1 = arg max θ Lπθk ( πθ ) , subject to Es∼ρπθk [ DKL ( πθk , πθ ) ] ≤ δ . ( 3 ) At each iteration k + 1 , TRPO constructs a KL-ball Bδ ( πθk ) around the policy πθk , and optimises πθk+1 ∈ Bδ ( πθk ) to maximise Lπθk ( πθ ) . By Theorem 1 , we know that the surrogate objective Lπθk ( πθ ) is close to the true reward J ( πθ ) within Bδ ( πθk ) ; therefore , πθk leads to improvement . Furthermore , to save the cost on Es∼ρπθk [ DKL ( πθk , πθ ) ] when computing Equation ( 3 ) , Schulman et al . ( 2017 ) proposed an approximation solution to TRPO that uses only first order derivatives , known as PPO . PPO optimises the policy parameter θk+1 by maximising the PPO-clip objective of LPPOπθk ( πθ ) = Es∼ρπθk , a∼πθk [ min ( πθ ( a|s ) πθk ( a|s ) Aπθk ( s , a ) , clip ( πθ ( a|s ) πθk ( a|s ) , 1± ) Aπθk ( s , a ) ) ] . ( 4 ) The clip operator replaces the ratio πθ ( a|s ) πθk ( a|s ) with 1+ or 1− , depending on whether or not the ratio is beyond the threshold interval . This effectively enables PPO to control the size of policy updates .

Summary: This paper introduces a trust region policy optimization method for cooperative multiagent reinforcement learning with a improvement guarantee. This is accomplished by imposing an arbitrary hierarchy on the players so that each player optimizes its policy w.r.t. the players who updated before it in the hierarchy. To my knowledge, this is the first extension of PPO and TRPO to the MARL setting with an improvement guarantee. Experiments demonstrate the efficacy of the approach.