Coordinated Proximal Policy Optimization

1 Introduction . Cooperative Multi-Agent Reinforcement Learning ( CoMARL ) shows great promise for solving various real-world tasks , such as traffic light control ( Wu et al. , 2020 ) , sensor network management ( Sharma and Chauhan , 2020 ) and autonomous vehicle coordination ( Yu et al. , 2019 ) . In such applications , a team of agents aim to maximize a joint expected utility through a single global reward . Since multiple agents coexist in a common environment and learn and adapt their behaviour concurrently , the arising non-stationary issue makes it difficult to design an efficient learning method ( Hernandez-Leal et al. , 2017 ; Papoudakis et al. , 2019 ) . Recently , a number of CoMARL ∗Corresponding authors . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . methods based on Centralized Training Decentralized Execution ( CTDE ) ( Foerster et al. , 2016 ) have been proposed , including policy-based ( Lowe et al. , 2017 ; Foerster et al. , 2018 ; Wang et al. , 2020 ; Yu et al. , 2021 ) and value-based methods ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ; Mahajan et al. , 2019 ) . While generally having more stable convergence properties ( Gupta et al. , 2017 ; Song et al. , 2019 ; Wang et al. , 2020 ) and being naturally suitable for problems with stochastic policies ( Deisenroth et al. , 2013 ; Su et al. , 2021 ) , policy-based methods still receive less attention from the community and generally possess inferior performance against value-based methods , as evidenced in the StarCraft II benchmark ( Samvelyan et al. , 2019 ) . The performance discrepancy between policy-based and value-based methods can be largely attributed to the inadequate utilization of the centralized training procedure in the CTDE paradigm . Unlike value-based methods that directly optimize the policy via centralized training of value functions using extra global information , policy-based methods only utilize centralized value functions for state/action evaluation such that the policy can be updated to increase the likelihood of generating higher values ( Sutton et al. , 2000 ) . In other words , there is an update lag between intermediate value functions and the final policy in policy-based methods , and merely coordinating over value functions is insufficient to guarantee satisfactory performance ( Grondman et al. , 2012 ; Fujimoto et al. , 2018 ) . To this end , we propose the Coordinated Proximal Policy Optimization ( CoPPO ) algorithm , a multiagent extension of PPO ( Schulman et al. , 2017 ) , to directly coordinate over the agents ’ policies by dynamically adapting the step sizes during the agents ’ policy update processes . We first prove a relationship between a lower bound of joint policy performance and the update of policies . Based on this relationship , a monotonic joint policy improvement can be achieved through optimizing an ideal objective . To improve scalability and credit assignment , and to cope with the potential high variance due to non-stationarity , a series of transformations and approximations are then conducted to derive an implementable optimization objective in the final CoPPO algorithm . While originally aiming at monotonic joint policy improvement , CoPPO ultimately realizes a direct coordination over the policies at the level of each agent ’ s policy update step size . Concretely , by taking other agents ’ policy update into consideration , CoPPO is able to achieve dynamic credit assignment that helps to indicate a proper update step size to each agent during the optimization procedure . In the empirical study , an extremely hard version of the penalty game ( Claus and Boutilier , 1998 ) is used to verify the efficacy and interpretability of CoPPO . In addition , the evaluation on the StarCraft II micromanagement benchmark further demonstrates the superior performance of CoPPO against several strong baselines . The paper is organized as follows : Section 2 provides a background introduction . Section 3 introduces the derivation process of CoPPO . Section 4 presents the experimental studies , and Section 5 reviews some related works . Finally , Section 6 concludes the paper . 2 Background . We model the fully cooperative MARL problem as a Dec-POMDP ( Oliehoek and Amato , 2016 ) which is defined by a tuple G = 〈N , S , Ω , O , A , R , P , γ〉 . N is the number of agents and S is the set of true states of the environment . Agent i ∈ { 1 , . . . , N } obtains its partial observation oi ∈ Ω according to the observation function O ( s , i ) , where s ∈ S. Each agent has an action-observation history τ i ∈ T ≡ ( Ω × A ) ∗ . At each timestep , agent i chooses an action ai ∈ A according to its policy π ( ai|τ i ) , and we use a to denote the joint action { a1 , . . . , aN } . The environment then returns the reward signal R ( s , a ) that is shared by all agents , and shifts to the next state according to the transition function P ( s′|s , a ) . The joint action-value function induced by a joint policy π is defined as : Qπ ( st , at ) = Est+1 : ∞ , at+1 : ∞ [ ∑∞ t′=0 γ t′Rt+t′ |st , at ] , where γ ∈ [ 0 , 1 ) is the discounted factor . We denote the joint action of agents other than agent i as a−i , and π−i , τ−i follow a similar convention . Joint policy π can be parameterized by θ = { θ1 , . . . , θN } , where θi is the parameter set of agent i ’ s policy . Our problem setting follows the CTDE paradigm ( Foerster et al. , 2016 ) , in which each agent executes its policy conditioned only on the partially observable information , but the policies can be trained centrally by using extra global information . Value-based MARL In CTDE value-based methods such as ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ; Mahajan et al. , 2019 ) , an agent selects its action by performing an arg max operation over the local action-value function , i.e . πi ( τ i ) = arg maxai Q i ( τ i , ai ) . Without loss in generality , the update rule for value-based methods can be formulated as follows : ∆θi ∝ Eπ [ ( R ( s , a ) + max a′ Qtot ( s , a′ ) −Qtot ( s , a ) ) ∂Qtot ∂Qi ∇θiQi ( τ i , ai ) ] , ( 1 ) where θi represents the parameter of Qi , and Qtot is the global action-value function . The centralized training procedure enables value-based methods to factorize Qtot into some local action-values . Eq . ( 1 ) is essentially Q-learning if rewriting ∂Q tot ∂Qi ∇θiQ i ( τ i , ai ) as∇θiQtot . The partial derivative is actually a credit assignment term that projects the step size of Qtot to that of Qi ( Wang et al. , 2020 ) . Policy-based MARL In CTDE policy-based methods such as ( Foerster et al. , 2018 ; Lowe et al. , 2017 ; Wang et al. , 2020 ; Yu et al. , 2021 ) , an agent selects its action from an explicit policy πi ( ai|τ i ) . The vanilla multi-agent policy gradient algorithm updates the policy using the following formula : ∆θi ∝ Eπ [ Qtotπ ( s , a ) ∇θiπi ( ai|τ i ) ] . ( 2 ) In order to reduce the variance and address the credit assignment issue , COMA ( Foerster et al. , 2018 ) replace Qtot with the counterfactual advantage : Aπ ( s , a ) = Qπ ( s , ( ai , a−i ) ) − Eâi∼πi [ Qπ ( s , ( âi , a−i ) ) ] . This implies that fixing the actions of other agents ( i.e . a−i ) , an agent will evaluate the action it has actually taken ( i.e . ai ) by comparing with the average effect of other actions it may have taken . 3 Coordinated Proximal Policy Optimization . In policy-based methods , properly limiting the policy update step size is proven to be effective in single-agent settings ( Schulman et al. , 2015a , 2017 ) . In cases when there are multiple policies , it is also crucial for each agent to take other agents ’ update into account when adjusting its own step size . Driven by this insight , we propose the CoPPO algorithm to adaptively adjust the step sizes during the update of the policies of multiple agents . 3.1 Monotonic Joint Policy Improvement . The performance of joint policy π is defined as : J ( π ) .= Ea∼π , s∼ρπ [ ∑∞ t=0 γ tRt+1 ( s , a ) ] , where ρπ is the unnormalized discounted visitation frequencies when the joint actions are chosen from π . Then the difference between the performance of two joint policies , say π and π̃ , can be expressed as the accumulation of the global advantage over timesteps ( see Appendix A.1 for proof ) : J ( π̃ ) − J ( π ) = Ea∼π̃ , s∼ρπ̃ [ Aπ ( s , a ) ] , ( 3 ) where Aπ ( s , a ) = Qπ ( s , a ) − V π ( s ) is the joint advantage function . This equation indicates that if the joint policy π is updated to π̃ , then the performance will improve when the update increases the probability of taking `` good '' joint actions so that ∑ a π̃ ( a|s ) Aπ ( s , a ) > 0 for every s. Modeling the dependency of ρπ̃ on π̃ involves the complex dynamics of the environment , so we extend the approach proposed in ( Kakade and Langford , 2002 ) to derive an approximation of J ( π̃ ) , denoted as J̃π ( π̃ ) : J̃π ( π̃ ) . = J ( π ) + Ea∼π̃ , s∼ρπ [ Aπ ( s , a ) ] . ( 4 ) Note that if the policy is differentiable , then J̃π ( π̃ ) matches J ( π̃ ) to first order ( see Appendix A.2 for proof ) . Quantitatively , we measure the difference between two joint policies using the maximum total variation divergence ( Schulman et al. , 2015a ) , which is defined by : DmaxTV [ π‖π̃ ] . = maxsDTV [ π ( ·|s ) ‖π̃ ( ·|s ) ] , where DTV [ π ( ·|s ) ‖π̃ ( ·|s ) ] = 12 ∫ A |π ( a|s ) − π̃ ( a|s ) |da ( the definition for the discrete case is simply replacing the integral with a summation , and our results remain valid in such case ) . Using the above notations , we can derive the following theorem : Theorem 1 . Let = maxs , a |Aπ ( s , a ) | , αi = √ 1 2D max TV [ π i||π̃i ] , 1 ≤ i ≤ N , and N be the total number of agents , then the error of the approximation in Eq . ( 4 ) can be explicitly bounded as follows : ∣∣∣J ( π̃ ) − J̃π ( π̃ ) ∣∣∣ ≤ 4 [ 1− γ∏Ni=1 ( 1− αi ) 1− γ − 1 ] . ( 5 ) Proof . See Appendix A.3 . As shown above , the upper bound is influenced by αi , N and . By definition , we have αi ≤ 1 and ≥ 0 , thus the upper bound will increase when αi increases for any i , implying that it becomes harder to make precise approximation when any individual of the agents dramatically updates their policies . Also , the growth in the number of agents can raise the difficulty for approximation . As for , from Eq . ( 5 ) we can roughly conclude that a larger advantage value can cause higher approximation error , and this is reasonable because J̃π ( π̃ ) approximates J ( π̃ ) by approximating the expectation over Aπ . Transforming the inequality in Eq . ( 5 ) leads to J ( π̃ ) ≥ J̃π ( π̃ ) −4 ( 1−γ ∏N i=1 ( 1−αi ) 1−γ − 1 ) . Thus the joint policy can be iteratively updated by : πnew = arg max π̃ [ J̃πold ( π̃ ) − 4 ( 1− γ ∏N i=1 ( 1− αi ) 1− γ − 1 ) ] . ( 6 ) Eq . ( 6 ) involves a complete search in the joint observation space and action space for computing and αi , making it difficult to be applied to large-scale settings . In the next subsection , several transformations and approximations are employed to this objective to achieve better scalability .

Summary: This paper proposes a new CTDE algorithm CoPPO for cooperative MARL. CoPPO is based on the extended objective of TRPO in MARL settings. CoPPO uses the clip objective of PPO to implement the trust region constraint of TRPO and proposes a double clipping trick to adjust the step sizes of different agents.