Multi-agent Policy Optimization with Approximatively Synchronous Advantage Estimation

1 INTRODUCTION . Reinforcement learning ( RL ) algorithms have shown amazing performance on many singleagent ( SA ) environment tasks ( Mnih et al. , 2013 ) ( Jaderberg et al. , 2016 ) ( Oh et al. , 2018 ) . However , for many real-world problems , the environment is much more complex where RL agents often need to cooperate with other agents . For example , taxi scheduling ( Nguyen et al. , 2018 ) and network control ( Chu et al. , 2019 ) . In cooperative multi-agent tasks , each agent is treated as an independent decision-maker , but can be trained together to learn cooperation . The common goal is to maximize the global return in the perspective of a team of agents . To deal with such tasks , the architecture of centralized training and decentralized executions ( CTDE ) is proposed ( Oliehoek & Vlassis , 2007 ) ( Jorge et al. , 2016 ) . The basic idea of CTDE is to construct a centralized policy evaluator , which only works during training and is accessable to global information . At the same time , each agent is assigned with a local policy for decentralized execution . The role of the evaluator is to evaluate agents ’ local policies differentially from the global perspective . A challenge in construction of centralized evaluator is multi-agent credit assignment ( Chang et al. , 2004 ) : in cooperative settings , joint actions typically generate only global rewards , making it difficult for each agent to deduce its own contribution to the team ’ s success . Credit assignment requires differentiate evaluation for agents ’ local policies , but designing individual reward function for each agent is often complicated and lacks of generalization ( Grzes , 2017 ) ( Mannion et al. , 2018 ) . Current policy based MARL methods generally realize credit assignment by introducing differentiate value functions or advantage functions ( Foerster et al. , 2018 ) ( Lowe et al. , 2017 ) . However , these value functions or advantage functions are estimated asynchronously but decentralized policies are updated synchronously , as shown in figure 1 ( b ) , which results in natural estimation bias . In this paper , we propose a novel policy based MARL method called multi-agent policy optimization with approximatively synchronous advantage estimation ( ASAE ) . In our work , we first define the counter-factual scenes , in which MA advantage estimation can be converted to SA advantage estimation . For certain agent , each counter-factual scene is assigned with a SA advantage . Then the marginal advantage function is defined as the expectation of SA advantages on distribution of counter-factual scenes , and credit assignment is realized by constructing different scenes ’ distribution for different agents . Moreover , in order to achieve synchronous advantage estimation , an approximation of other agents ’ joint future policy is introduced . To ensure the approximation is reliable , a restriction is applied to the original multi-agent policy optimization ( MAPO ) problem . The approximate optimization problem is simplified and broken down into multiple sub-problems , which has a similar form to trust region policy optimization ( TRPO ) problem . And the sub-problems are finally solved by proximal policy optimization ( PPO ) method . We have two contributions in this work : ( 1 ) A novel advantage estimation method called marginal advantage estimation , which realizes credit assignment for MARL is proposed . More importantly , this method provides a channel for various SA advantage functions expanding to multi-agent system . ( 2 ) A simple yet effective method for approximatively synchronous advantage estimation is firstly proposed . 2 RELATED WORK . A common challenge in cooperative multi-agent tasks is credit assignment . RL algorithms designed for single-agent tasks , ignore credit assignment and take other agents as part of partial observable environment . Such algorithms perform poorly in complex cooperative tasks which require high coordination ( Lowe et al. , 2017 ) . To deal with the challenge , some value based MARL methods estimate a local Q value for each agent , and the shared global Q value is then constructed through these local Q values . Value decomposition network ( VDN ) constructs the global Q value by simply adding all local Q values together ( Sunehag et al. , 2018 ) . And in QMIX algorithm ( Rashid et al. , 2018 ) , the global Q value is obtained by mixing local Q values with a neural network . In mean field multi-agent methods , local Q values are defined on agent pairs . The mapping from local Q values to the global Q value is established by measuring the influence of each agent pair ’ s joint action to the global return ( Yang et al. , 2018 ) . Similarly , for policy based MARL methods , credit assignment is generally realized through differentiated evaluation with CTED structure . Some naive policy based methods estimate local Q values for individual agents with a centralized critic ( Lowe et al. , 2017 ) , resulting in large variance . Some other methods try to introduce advantage function in MARL . Counter-factual multi-agent policy gradient ( COMA ) method ( Foerster et al. , 2018 ) is inspired by the idea of difference reward ( Wolpert & Tumer , 2002 ) and provides a naive yet effective approach for differentiated advantage estimation in cooperative MARL . In COMA , a centralized critic is used to predict the joint Q value function Qπ ( s , u ) of joint action u under state s. And the advantage for agent a is defined as Aa ( s , u ) = Q ( s , u ) − ∑ u′a πa ( u′a|τa ) Q ( s , ( u−a , u′a ) ) ( 1 ) where τ and π represent trajectory and policy respectively . a and -a denote current agent and the set of other agents respectively . COMA introduces a counter-factual baseline , which assumes that other agents take fixed actions , as shown in figure 1 ( b ) . COMA performs synchronous updates with asynchronous estimation , which leads to lagging and biased advantage estimation . In contrast , asynchronous estimation & asynchronous updating is more reliable yet more complicated . An ideal approach is synchronous estimation & synchronous updating . However , it requires prediction of other agents ’ future policies . 3 BACKGROUND . We consider a most general setting of partially observable , full cooperative multi-agent tasks , which can be described as a stochastic game defined by a tuple G = < S , U , P , r , Z , O , n , γ > . The true state of environment s ∈ S is unavailable to all agents . At each time step , n agents identified by a ∈ A ( A = { 1 , 2 , · · · , n } ) receive their local observations za ∈ Z , and take actions ua ∈ U simultaneously . The joint observation Z = Zn is acquired by the observation function O ( s , a ) : S × A → Z . The next state is determined by joint action u ∈ U ( U = Un ) and the transition function P ( s′|s , u ) : S×U×S → [ 0 , 1 ] . The reward function r ( s , u ) : S×U→ R is shared by all agents , so as the discounted return Gt = ∑∞ t+i γ trt+i . γ ∈ [ 0 , 1 ) is a discount factor . In policy based MARL with CTED architecture , each agent has a local trajectory τa consists of historical observation and action { ( za0 , ua0 ) , ( ua1 , za1 ) , · · · } . And an independent policy πa ( ua|τa ) is constructed for each agent on their local trajectory . Action-state value function Qπ ( s , u ) and state value function V π ( s ) are used to evaluate joint policy . The advantage function is Aπ ( s , u ) = Qπ ( s , u ) −V π ( s ) . For clarity , symbols in bold are used to denote the joint variable of group agents . In single-agent policy optimization problems ( Schulman et al. , 2015a ) , the objective is to maximize the expected action state value functionEπθ [ Qπθ ] . Similarly , for MAPO with CTDE structure , each agent optimize its local policy individually with estimated Q values from centralized critic . Under this circumstance , the overall objective is for agent a = 1 to n : max θa E ( πθa , π−a ) [ Qa ( πθa , π−a ) ] ( 2 ) Where Q values can be substituted by advantages to reduce the variance . 4 APPROXIMATIVELY SYNCHRONOUS ADVANTAGE ESTIMATION IN MULTI-AGENT SYSTEM . In this section , we first introduce marginal advantage estimation which expands advantage functions of SARL to MARL as well to realize credit assignment . And then , we describe how to realize approximatively synchronous advantage estimation based on the marginal advantage function in MAPO problem . 4.1 MARGINAL ADVANTAGE ESTIMATION . In this subsection , we are going to solve the challenge of credit assignment through the proposed marginal advantage estimation . We first consider an counter-factual way where advantages are estimated asynchronously but policies are updated synchronously , as shown in figure 1 ( b ) . In this case , a counter-factual scene can be defined as : at certain state , for agent a , other agent always take fixed actions . In partially observable , full cooperative multi-agent settings , the counter-factual advantage of agent a ’ s action ua under state s is derived based on the joint action ’ s value ( or joint Q value ) function Q ( s , u ) Aa ( s , u ) = Aa ( s , ( ua , u−a ) ) = Q ( s , u ) − ∫ ua Q ( s , u−a , ua ) dπa ( ua|τa ) ( 3 ) From the view of agent a , the counter-factual advantage depends on other agents ’ joint action u−a , which is a random variable and u−a ∼ π−a . In order to remove the dependency , the marginal Q value function of agent a is defined as Qa ( s , ua ) = Eu−a∼π−a [ Q ( s , ( ua , u−a ) ) ] ( 4 ) Notice that in CTED structure , policy πa ( ua|τa ) and π−a ( u−a|τ−a ) are independent . By replacing joint Q value function with marginal Q value function , the marginal advantage function is derived Aa ( s , ua ) = Qa ( s , ua ) − ∫ ua Qa ( s , ua ) dπa ( ua|τa ) = ∫ u−a Q ( s , ua , u−a ) dπ−a ( u−a|τ−a ) − ∫ ua ∫ u−a Q ( s , ua , u−a ) dπ−a ( u−a|τ−a ) dπa ( ua|τa ) = ∫ u−a [ Q ( s , ua , u−a ) − ∫ ua Q ( s , ua , u−a ) dπa ( ua|τa ) ] dπ−a ( u−a|τ−a ) = ∫ u−a Aa ( s , u ) dπ−a ( u−a|τ−a ) ( 5 ) Such replacement will not change the result of advantage estimation because the substitution of joint Q value is its expectation . Form equation ( 5 ) , for different agent , the value of marginal advantage is different , which realizes credit assignment . It can be easily proved that if counter-factual advantage Aa ( s , u ) is an unbiased estimation of joint Q value Q ( s , u ) , then marginal advantage is also an unbiased estimation of marginal Q value ( Appendix I ) . In a counter-factual scene , from the view of agent a , other agents and their fix joint actions u−a can be regarded as part of the environment . Let ( s , u−a ) = sctf and counter-factual advantage function can be written as Aa ( s , u ) =Aa ( sctf , ua ) =Q ( sctf , u a ) − ∫ ua Q ( sctf , u a ) dπa ( ua|τa ) ( 6 ) In counter-factual scenes , counter-factual advantage function is identical to advantage function in SARL , which means the counter-factual advantage in equation ( 5 ) can be replaced by any form of advantage function used in SARL . For example , considering using TD residual δat = r ( st , u a t ) + γV ( st+1 ) −V ( st ) as an estimation of joint advantage Aa ( st , ut ) , the marginal advantages could be written as Aa ( st , ut ) : = Eu−a∼π−a [ ∞∑ l=0 γlδat+l ] Aa ( st , ut ) : = Eu−a∼π−a [ δ a t ] ( 7 ) The former is unbiased estimation , but has high variance . The latter is biased estimation for any V 6= V π , but has much lower variance . These two methods can be combined for compromise between bias and variance ( Schulman et al. , 2015b ) . As agents ’ policies are independent , the expectation in equation ( 5 ) can be split into a ( n− 1 ) -layer integration , which is complicated . For simplicity and efficiency , the Monte-Carlo ( MC ) sampling can be applied as a substitution . Aa ( st , ut ) = ∫ u−a Aa ( st , ut ) dπ−a ≈ 1 m m∑ u−a Aa ( st , ut ) ( 8 ) Where m is the number of other agents ’ joint action samples . The principle of one step process to calculate marginal advantage with TD residual is shown in figure 2 . Firstly , based on the last true state st , m joint action samples are sampled . These samples are then reorganized . Take agent 1 as example , action u1 , t from Sa1 is combined with other agents ’ action samples from Sa2 to Sam respectively . As a result , m reorganized new samples are acquired . Based on these new samples , one step simulations are executed and m counter-factual rewards and states are acquired , which are used to calculate the estimation of marginal advantage . At last , the next true state is selected form counter-factual states . Both methods in equation ( 7 ) use V value predictor and require interactive simulation . Agent needs to interact with environment to get extra samples . In this work , we consider using centralized critic to predict jointQ values , and the marginal advantages can be directly calculated with theseQ values , which avoids interactive simulation .

Summary: The paper deals with the problem of credit assignment and synchronous estimation in cooperative multi-agent reinforcement learning problems. The authors introduce marginal advantage functions and use them for the estimation of the counterfactual advantage function. These functions permit to decompose the Multi-Agent Policy Optimization Problem in Single Agent Policy Optimization subproblems, which are solved using TRPO.