Monotonic Improvement Guarantees under Non-stationarity for Decentralized PPO

1 INTRODUCTION . In cooperative multi-agent reinforcement learning ( MARL ) , a team of agents must coordinate their behavior to maximize a single cumulative return ( Panait & Luke , 2005 ) . In such a setting , partial observability and/or communication constraints necessitate the learning of decentralized policies that condition only on the local action-observation history of each agent . In a simulated or laboratory setting , decentralized policies can often be learned in a centralized fashion , i.e. , Centralized Training with Decentralized Execution ( CTDE ) ( Oliehoek & Amato , 2016 ) , which allows agents to access each other ’ s observations and unobservable extra state information during training . Actor-critic algorithms ( Konda & Tsitsiklis , 2000 ) are a natural approach to CTDE because critics can exploit centralized training by conditioning on extra information not available to the decentralized policies ( Lowe et al. , 2017 ; Foerster et al. , 2017 ) . Unfortunately , such actor-critic methods have long been outperformed by value-based methods such as QMIX ( Rashid et al. , 2018 ) on MARL benchmark tasks such as Starcraft Multi-Agent Challenge ( SMAC ) ( Samvelyan et al. , 2019 ) . However , two recent actor-critic algorithms ( Schröder de Witt et al. , 2020 ; Yu et al. , 2021 ) have upended this ranking by outperforming previously dominant MARL methods , such as MADDPG ( Lowe et al. , 2017 ) and value-decomposed Q-learning ( Sunehag et al. , 2017 ; Rashid et al. , 2018 ) . Both algorithms are multi-agent extensions of Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) but one uses decentralized critics , i.e. , independent PPO ( IPPO ) ( Schröder de Witt et al. , 2020 ) , and the other uses centralized critics , i.e. , multi-agent PPO ( MAPPO ) ( Yu et al. , 2021 ) . One key feature of PPO-based methods is the use of ratios ( between the policy probabilities before and after updating ) in the objective . Both IPPO and MAPPO extend this feature of PPO to the multi-agent setting by computing ratios separately for each agent ’ s policy during training , which we call independent ratios . Unfortunately , until now there has been no theoretical justification for the use of such independent ratios . In this paper we show that the analysis that underpins the monotonic policy improvement guarantee for PPO ( Schulman et al. , 2015 ) does not carry over to the use of independent ratios in IPPO and MAPPO . Instead , a direct application of this analysis leads to a joint policy optimization and suggests the use of joint ratios , i.e. , computing ratios between joint policies . The difference is crucial because , based on the existing trust region analysis for PPO , only a joint ratios approach enjoys a monotonic policy improvement guarantee . Moreover , as independent ratios consider only the change in one agent ’ s policy and ignore the fact that the other agents ’ policies also change , the transition dynamics underlying these independent ratios are non-stationary ( Papoudakis et al. , 2019 ) , breaking the assumptions in the monotonic improvement analysis ( Schulman et al. , 2015 ) . While some studies attempt to extend the monotonic improvement analysis to MARL ( Wen et al. , 2021 ; Li & He , 2020 ) , they primarily consider optimizing policies with joint ratios , rather than independent ratios as in our paper , and are thus not applicable to IPPO or MAPPO . To address this gap , we provide a new monotonic improvement analysis that holds even when the transition dynamics are non-stationary . We show that , despite this non-stationarity , a monotonic improvement guarantee still arises as a result of enforcing the trust region constraint over joint policies , i.e. , a centralized trust region constraint . In other words , constraining the update of joint policies in centralized training addresses the non-stationarity of learning decentralized policies . This new analysis implies that independent ratios can also enjoy the same performance guarantee as joint ratios if the centralized trust region constraint is properly enforced by bounding independent ratios . In this way both IPPO and MAPPO can guarantee monotonic policy improvement . We also provide a theoretical foundation for proximal ratio clipping by showing that centralized trust region can be enforced in a principled way by bounding independent ratios based on the number of agents in training . Furthermore , we show that the surrogate objectives optimized in IPPO and MAPPO are essentially equivalent when their critics converge to a fixed point . Finally , we provide empirical results that support the hypothesis that the strong performance of IPPO and MAPPO is a direct result of enforcing such a trust region constraint via clipping in centralized training . Particularly , we show that good values of the hyperparameters for the clipping range are highly sensitive to the number of agents , as these hyperparameters , together with the number of agents , effectively determine the size of the centralized trust region . Moreover , we show that IPPO and MAPPO have comparable performance on SMAC maps with varied difficulty and numbers of agents . This comparable performance also implies that the way of training critics could be less crucial in practice than enforcing a trust region constraint . 2 RELATED WORK . The use of trust region optimization in MARL traces back to parameter-sharing TRPO ( PSTRPO ) ( Gupta et al. , 2017 ) , which combines parameter sharing with TRPO for cooperative multiagent continuous control but provides no theoretical support . Our analysis showing that a trust region constraint is pivotal to guarantee performance improvement in MARL applies to PS-TRPO , among other algorithms . Multi-agent trust region learning ( MATRL ) ( Wen et al. , 2021 ) uses a trust region for independent learning with a game-theoretical analysis in the policy space . MATRL considers independent learning and proposes to enforce a trust region constraint by approximating the stable fixed point via a meta-game . Despite the improvement guarantee for joint policies , solving a meta-game itself can be challenging because its complexity increases exponentially in the number of agents . We instead consider centralized learning and enforce the trust region constraint in a centralized and scalable way . Multi-Agent TRPO ( MATRPO ) directly extends TRPO to the multi-agent case ( Li & He , 2020 ) and divides the trust region by the number of agents . However , the analysis assumes a private reward for each agent , which yields different theoretical results from ours . Non-stationarity has been discussed in multi-agent mirror descent with trust region decomposition ( Li et al. , 2021 ) , which first decomposes the trust region for each decentralized policy and then approximates the KL divergence through additional training . However , this method needs to learn a fully centralized action-value function and thus becomes becomes impractical when there are many agents . 3 BACKGROUND . Dec-MDPs . We consider a fully cooperative multi-agent task in which a team of cooperative agents choose sequential actions in a stochastic environment . It can be modeled as a decentralized Markov decision process ( Dec-MDP ) , defined by a tuple { N , S , A , P , r , ρ0 , γ } , where N , { 1 , . . . , N } denotes the set ofN agents and s ∈ S , S1×S2× ... ×SN describes the joint state of the environment . The initial state s0 ∼ ρ0 is drawn from distribution ρ0 , and at each time step t , all agents k ∈ N choose simultaneously one action akt ∈ Ak , yielding a joint action at , a1t × a2t × ... × aNt ∈ A , A1 ×A2 × ... ×AN . After executing the joint action at in state st , the next state st+1 ∼ P ( st , at ) is drawn from transition kernel P and a collaborative reward rt = r ( st , at ) is returned . In a DecMDP , each agent k ∈ N has a local state skt ∈ Sk , and chooses its actions with a decentralized policy akt ∼ πk ( ·|skt ) based only on its local state . The collaborating team of agents aims to learn a joint policy , π ( at|st ) , ∏N k=1 πk ( a k t |skt ) , that maximizes their expected discounted return , η ( π ) , E ( st , at ) [ ∑∞ t=0 γ trt ] , where γ ∈ [ 0 , 1 ) is a discount factor . Policy optimization methods . For single-agent RL that is modeled as an infinite-horizon discounted Markov decision process ( MDP ) { S , A , P , r , ρ0 , γ } , the performance for a policy π ( a|s ) is defined as : η ( π ) = E ( st , at ) [ ∑∞ t=0 γ tr ( st , at ) ] . The action-value function Qπ and value function Vπ are defined as : Qπ ( st , at ) = Est+1∼p ( ·|st , at ) , at+1∼π ( ·|st+1 ) [ ∞∑ l=0 γlr ( st+l , at+l ) ] , Vπ ( st ) = Eat∼π ( ·|st ) [ Qπ ( st , at ) ] . Let the advantage function be Aπ ( s , a ) = Qπ ( s , a ) − Vπ ( s ) ; the following useful identity expresses the expected return of another policy π̃ in terms of the advantage over π ( Kakade & Langford , 2002 ) : η ( π̃ ) = η ( π ) + ∑ s ρπ̃ ( s ) ∑ a π̃ ( a|s ) Aπ ( s , a ) , where ρπ̃ ( s ) is the state distribution induced by π̃ . The complex dependency of ρπ̃ ( s ) on π̃ makes the righthand side difficult to optimize directly . Schulman et al . ( 2015 ) proposed to consider the following surrogate objective Lπ ( π̃ ) = η ( π ) + ∑ s ρπ ( s ) ∑ a π̃ ( a|s ) Aπ ( s , a ) = η ( π ) + E ( s , a ) ∼ρπ [ π̃ ( a|s ) π ( a|s ) Aπ ( s , a ) ] , where the ρπ̃ is replaced with ρπ . Define DmaxTV ( π , π̃ ) , maxsDTV ( π ( ·|s ) , π̃ ( ·|s ) ) , where DTV is the total variation ( TV ) divergence . Theorem 1 . ( Theorem 1 in Schulman et al . ( 2015 ) ) Let α = DmaxTV ( π , π̃ ) . Then the following bound holds η ( π̃ ) ≥ Lπ ( π̃ ) − 4 γ ( 1− γ ) 2 α2 , where = maxs , a|Aπ ( s , a ) | . This theorem forms the foundation of policy optimization methods , including Trust Region Policy Optimization ( TRPO ) ( Schulman et al. , 2015 ) and Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) . TRPO suggests a robust way to take large update steps by using a constraint , rather than a penalty , on the TV divergence , and considers the following practical optimization problem , TRPO : max π̃ E ( s , a ) ∼ρπ [ π̃ ( a|s ) π ( a|s ) Aπ ( s , a ) ] , s.t . DmaxTV ( π ( ·|s ) , π̃ ( ·|s ) ) ≤ α . ( 1 ) This constrained optimization is complicated as it requires using conjugate gradient algorithms with a quadratic approximation to the constraint . PPO simplifies the above optimization by clipping probability ratios λπ̃ = π̃ ( a|s ) π ( a|s ) to form a lower bound of Lπ ( π̃ ) : PPO : max π̃ E ( s , a ) ∼ρπ [ min ( λπ̃Aπ ( s , a ) , clip ( λπ̃ , 1− , 1 + ) Aπ ( s , a ) ) ] , ( 2 ) where is a hyperparameter to specify the clipping range . Independent PPO and Multi-Agent PPO . Both IPPO ( Schröder de Witt et al. , 2020 ) and MAPPO ( Yu et al. , 2021 ) optimize decentralized policies with independent ratios . In particular , the main objective IPPO and MAPPO optimize is max π̃k E ( sk , ak ) ∼ρπk [ min ( λπ̃kA k ( sk , ak ) , clip ( λπ̃k , 1− , 1 + ) Ak ( sk , ak ) ) ] ∀k ∈ { 1 , 2 , ... , N } , ( 3 ) where λπ̃k = π̃k ( a k|sk ) πk ( ak|sk ) denotes the ratio between the decentralized policy probabilities of agent k before and after updating . The difference between IPPO and MAPPO lies in how they estimate the advantage function : IPPO learns a decentralized advantage functionAk ( sk , ak ) , ∑∞ t=0 [ r ( s k t , a k t ) ] − V ( sk ) based on the local information ( sk , ak ) for each agent , while MAPPO uses a centralized critic that conditions on centralized state information s : Ak ( sk , ak ) , Es−k [ ∑∞ t=0 [ r ( s k t , a k t ) ] − V ( s ) ] , where −k refers the set of all agents except agent k. Both methods use parameter sharing between agents . Consequently , as all agents share the same actor and critic networks , one can ignore the agent specifier k in the objective and use experience from all agents to update the actor and critic networks : max π̃θ ∑ k E ( sk , ak ) ∼ρπθ [ min ( λθAφ ( s k , ak ) , clip ( λθ , 1− , 1 + ) Aφ ( sk , ak ) ) ] , ( 4 ) where λθ = π̃θ ( a k|sk ) πθ ( ak|sk ) , and θ , φ are shared parameters for policy and advantage networks . The use of independent ratios together with parameter sharing has shown strong empirical results in various MARL benchmark tasks ( Schröder de Witt et al. , 2020 ; Yu et al. , 2021 ) .

Summary: This paper provides analysis showing monotonic improvements when in cooperative MARL settings, where independent, as opposed to joint ratios (over agents) are used. From a single agent’s perspective, a non-stationary distribution is experienced since other players are simultaneously performing updates to their policies, breaking the standard monotonic improvement property in TRPO. The authors’ propose a surrogate objective for decentralized policies, and show that the improvement in expected return can be bounded in terms of the sum of these (decentralized) surrogate objectives and a term proportionate to the total variation divergence between the 2 policies---analogous to the result by Schulman et al in 2005. Similarly, the authors proceed by showing that the TV divergence between policies can be bounded by enforcing constraints on independent ratios, where the strictness of these bounds depends on the number of agents. The trust region constraint is then split (independently) over all agents, and the authors show that prior methods like IPPO and MAPPO maximize an objective consistent with the proposed theory.