Settling the Variance of Multi-Agent Policy Gradients

1 Introduction . Policy gradient ( PG ) methods refer to the category of reinforcement learning ( RL ) algorithms where the parameters of a stochastic policy are optimised with respect to the expected reward through gradient ascent . Since the earliest embodiment of REINFORCE [ 40 ] , PG methods , empowered by deep neural networks [ 33 ] , are among the most effective model-free RL algorithms on various kinds of tasks [ 2 , 6 , 10 ] . However , the performance of PG methods is greatly affected by the variance of the PG estimator [ 35 , 37 ] . Since the RL agent behaves in an unknown stochastic environment , which is often considered as a black box , the randomness of expected reward can easily become very large with increasing sizes of the state and action spaces ; this renders PG estimators with high variance , which concequently leads to low sample efficiency and unsuccessful trainings [ 9 , 35 ] . To address the large variance issue and improve the PG estimation , different variance reduction methods were developed [ 9 , 23 , 35 , 48 ] . One of the most successfully applied and extensively studied methods is the control variate subtraction [ 8 , 9 , 37 ] , also known as the baseline trick . A baseline is a scalar random variable , which can be subtracted from the state-action value samples in the PG estimates so as to decrease the variance , meanwhile introducing no bias to its expectation . Baselines can be implemented through a constant value [ 8 , 12 ] or a value that is dependent on the state [ 8 , 9 , 11 , 33 ] such as the state value function , which results in the zero-centered advantage function and the advantage actor-critic algorithm [ 18 ] . State-action dependent baselines can also be applied [ 9 , 41 ] , although they are reported to have no advantages over state-dependent baselines [ 35 ] . ⇤Equal contribution . †Corresponding author < yaodong.yang @ pku.edu.cn > . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . When it comes to multi-agent reinforcement learning ( MARL ) [ 43 ] , PG methods can naturally be applied . One naive approach is to make each agent disregard its opponents and model them as part of the environment . In such a setting , single-agent PG methods can be applied in a fully decentralised way ( hereafter referred as decentralised training ( DT ) ) . Although DT has numerous drawbacks , for example the non-stationarity issue and poor convergence guarantees [ 3 , 16 ] , it demonstrates excellent empirical performance in certain tasks [ 19 , 26 ] . A more rigorous treatment is to extend the PG theorem to the multi-agent policy gradient ( MAPG ) theorem , which induces a learning paradigm known as centralised training with decentralised execution ( CTDE ) [ 7 , 42 , 45 , 46 ] . In CTDE , each agent during training maintains a centralised critic which takes joint information as input , for example COMA [ 7 ] takes joint state-action pair ; meanwhile , it learns a decentralised policy that only depends on the local state , for execution . Learning the centralised critic helps address the non-stationarity issue encountered in DT [ 7 , 16 ] ; this makes CTDE an effective framework for implementing MAPG and successful applications have been achieved in many real-world tasks [ 13 , 17 , 22 , 38 , 39 , 50 ] . Unfortunately , compared to single-agent PG methods , MAPG methods suffer more from the large variance issue . This is because in multi-agent settings , the randomness comes not only from each agent ’ s own interactions with the environment but also other agents ’ explorations . In other words , an agent would not be able to tell if an improved outcome is due to its own behaviour change or other agents ’ actions . Such a credit assignment problem [ 7 , 36 ] is believed to be one of the main reasons behind the large variance of CTDE methods [ 16 ] ; yet , despite the intuition being built , there is still a lack of mathematical treatment for understanding the contributing factors to the variance of MAPG estimators . As a result , addressing the large variance issue in MARL is still challenging . One relevant baseline trick in MARL is the application of a counterfactual baseline , introduced in COMA [ 7 ] ; however , COMA still suffers from the large variance issue empirically [ 19 ] . In this work , we analyse the variance of MAPG estimates mathematically . Specifically , we try to quantify the contributions of the number of agents and the effect of multi-agent explorations to the variance of MAPG estimators . One natural outcome of our analysis is the optimal baseline ( OB ) , which achieves the minimal variance for MAPG estimators . Our OB technique can be seamlessly plugged into any existing MAPG methods . We incorporate it in COMA [ 7 ] and a multi-agent version of PPO [ 29 ] , and demonstrate its effectiveness by evaluating the resulting algorithms against the state-of-the-art algorithms . Our main contributions are summarised as follows : 1 . We rigorously quantify the excess variance of the CTDE MAPG estimator to that of the DT one and prove that the order of such excess depends linearly on the number of agents , and quadratically on agents ’ exploration terms ( i.e. , the local advantages ) . 2 . We demonstrate that the counterfactual baseline of COMA reduces the noise induced by other agents , but COMA still faces the large variance due to agent ’ s own exploration . 3 . We derive that there exists an optimal baseline ( OB ) , which minimises the variance of an MAPG estimator , and introduce a surrogate version of OB that can be easily implemented in any MAPG algorithms with deep neural networks . 4 . We show by experiments that OB can effectively decrease the variance of MAPG estimates in COMA and multi-agent PPO , stabilise and accelerate training in StarCraftII and Multi-Agent MuJoCo environments . 2 Preliminaries & Background . In this section , we provide the prelimiaries for the MAPG methods in MARL . We introduce notations and problem formulations in Section 2.1 , present the MAPG theorem in Section 2.2 , and finally , review two existing MAPG methods that our OB can be applied to in Section 2.3 . 2.1 Multi-Agent Reinforcement Learning Problem . We formulate a MARL problem as a Markov game [ 14 ] , represented as a tuple hN , S , A , P , r , i . Here , N = { 1 , . . . , n } is the set of agents , S is the state space , A = Qn i=1 Ai is the product of action spaces of the n agents , known as the joint action space , P : S ⇥A⇥ S ! R is the transition probability kernel , r : S ⇥A ! R is the reward function ( with |r ( s , a ) | , 8s 2 S , a 2 A ) , and 2 [ 0 , 1 ) is the discount factor . Each agent i possesses a parameter vector ✓i , which concatenated with parameters of other agents , gives the joint parameter vector ✓ . At a time step t 2 N , the agents are at state st 2 S. An agent i takes an action ait 2 Ai drawn from its stochastic policy ⇡i✓ ( ·|st ) parametrised by ✓i ( 1 ) , simultaneously with other agents , which together gives a joint action at = ( a1t , . . . , ant ) 2 A , drawn from the joint policy ⇡✓ ( at|st ) = Qn i=1 ⇡ i ✓ ( a i t|st ) . The system moves to state st+1 2 S with probability mass/density P ( st+1|st , at ) . A trajectory is a sequence ⌧ = hst , at , rti1t=0 of states visited , actions taken , and rewards received by the agents in an interaction with the environment . The joint policy ⇡✓ , the transition kernel P , and the initial state distribution d0 , induce the marginal state distributions at time t , i.e. , dt✓ ( s ) , which is a probability mass when S is discrete , and a density function when S is continuous . The total reward at time t 2 N is defined as Rt , P1 k=0 krt+k . The state value function V✓ and the state-action value function Q✓ are given by V✓ ( s ) , Ea0:1⇠⇡✓ , s1:1⇠P h R0 s0 = s i , Q✓ ( s , a ) , Es1:1⇠P , a1:1⇠⇡✓ h R0 s0 = s , a0 = a i . The advantage function is defined as A✓ ( s , a ) , Q✓ ( s , a ) V✓ ( s ) . The goal of the agents is to maximise the expected total reward J ( ✓ ) , Es⇠d0 ⇥ V✓ ( s ) ⇤ . In this paper , we write ( i1 , . . . , ik ) to denote the set of all agents excluding i1 , . . . , ik ( we drop the bracket when k = 1 ) . We define the multi-agent state-action value function for agents i1 , . . . , ik as Qi1 , ... , ik✓ s , a ( i1 , ... , ik ) , E a ( i1 , ... , ik ) ⇠⇡ ( i1 , ... , ik ) ✓ ⇥ Q✓ s , a ( i1 , ... , ik ) , a ( i1 , ... , ik ) ⇤ ( 2 ) , which is the expected total reward once agents i1 , . . . , ik have taken their actions . Note that for k = 0 , this becomes the state value function , and for k = n , this is the usual state-action value function . As such , we can define the multi-agent advantage function as Ai1 , ... , ik✓ ⇣ s , a ( j1 , ... , jm ) , a ( i1 , ... , ik ) ⌘ , Qj1 , ... , jm , i1 , ... , ik✓ ⇣ s , a ( j1 , ... , jm , i1 , ... , ik ) ⌘ Qj1 , ... , jm✓ ⇣ s , a ( j1 , ... , jm ) ⌘ , ( 1 ) which is the advantage of agents i1 , . . . , ik playing ai1 , . . . , aik , given aj1 , . . . , ajm . When m = n 1 and k = 1 , this is often referred to as the local advantage of agent i [ 7 ] . 2.2 The Multi-Agent Policy Gradient Theorem . The Multi-Agent Policy Gradient Theorem [ 7 , 47 ] is an extension of the Policy Gradient Theorem [ 33 ] from RL to MARL , and provides the gradient of J ( ✓ ) with respect to agent i ’ s parameter , ✓i , as r✓iJ ( ✓ ) = Es0:1⇠d0:1✓ , a i0:1⇠⇡ i✓ , ai0:1⇠⇡i✓ '' 1X t=0 tQ✓ ( st , a it , a i t ) r✓i log ⇡i✓ ( ait|st ) # From this theorem we can derive two MAPG estimators , one for CTDE ( i.e. , giC ) , where learners can query for the joint state-action value function , and one for DT ( i.e. , giD ) , where every agent can only query for its own state-action value function . These estimators , respectively , are given by giC = 1X t=0 tQ̂ st , a it , a i t r✓i log ⇡i✓ ait st and giD = 1X t=0 tQ̂i st , ait r✓i log ⇡i✓ ait st , where st ⇠ dt✓ , ait ⇠ ⇡i✓ ( ·|st ) , a i t ⇠ ⇡ i✓ ( ·|st ) . Here , Q̂ is a ( joint ) critic which agents query for values of Q✓ ( s , a ) . Similarly , Q̂i is a critic providing values of Qi✓ ( s , a i ) . The roles of the critics are , in practice , played by neural networks that can be trained with TD-learning [ 7 , 34 ] . For the purpose of this paper , we assume they give exact values . In CTDE , a baseline [ 7 ] is any function b ( s , a i ) . For any such function , we can easily prove that Es⇠dt✓ , a i⇠⇡ i✓ , ai⇠⇡i✓ ⇥ b ( s , a i ) r✓i log ⇡i✓ ( ai s ) ⇤ = 0 , ( for proof see Appendix A or [ 7 ] ) which allows augmenting the CTDE estimator as follows giC ( b ) = 1X t=0 t h Q̂ ( st , a it , a i t ) b ( st , a it ) i r✓i log ⇡i✓ ( ait|st ) , ( 2 ) ( 1 ) It could have been more clear to write ⇡i✓i to highlight that ⇡ i depends only on ✓i and no other parts of ✓ . Our notation , however , allows for more convenience in the later algebraic manipulations . ( 2 ) The notation `` a '' and `` ( a i , ai ) '' , as well as `` a ⇠ ⇡✓ '' and `` a i ⇠ ⇡ i✓ , a i ⇠ ⇡i✓ '' , are equivalent . We write ai and a when we refer to the action and joint action as to values , and ai and a as to random variables . and it has exactly the same expectation as giC = g i C ( 0 ) , but can lead to different variance properties . In this paper , we study total variance , which is the sum of variances of all components giC , j ( b ) of a vector estimator giC ( b ) . We note that one could consider the variance of every component of parameters , and for each of them choose a tuned baseline [ 23 ] . This , however , in light of neural networks with overwhelming parameter sizes used in deep MARL seems not to have practical applications .

Summary: This paper analyzes the variance of gradient estimates in policy-gradient-based multi-agent RL, specifically for standard multi-agent policy gradient (including both centralized and decentralized training) and for the COMA gradient. The paper then derives an optimal baseline (OB) as well as a tractable surrogate for variance reduction. The non-optimality of the variance of standard MAPG and COMA is shown. Numerically, lower variance of the OB is shown in an analytical example, and performance (in terms of reward and win rate) improvement is shown when the OB is implemented on top of MAPPO and COMA in the discrete-action StarCraft micromanagement scenarios and continuous-action multi-agent MuJoCo.