Asynchronous Multi-Agent Actor-Critic with Macro-Actions

1 INTRODUCTION . In recent years , multi-agent policy gradient methods using the actor-critic framework have achieved impressive success in solving a variety of cooperative and competitive domains ( Lowe et al. , 2017 ; Foerster et al. , 2018 ; Du et al. , 2019 ; Iqbal & Sha , 2019 ; Vinyals et al. , 2019 ; Li et al. , 2019 ; Wang et al. , 2020 ; Yang et al. , 2020 ; Zhou et al. , 2020 ; Baker et al. , 2020 ; Su et al. , 2021 ; Wang et al. , 2021 ; Du et al. , 2021 ) . However , as these methods assume synchronized primitive-action execution over agents , they struggle to solve tasks that involve long-term reasoning and asynchronous behavior , such as real-world multi-robot applications ( e.g. , search and rescue ( Queralta et al. , 2020 ) , package delivery ( Choudhury et al. , 2021 ) and warehouse service ( Xiao et al. , 2020 ) ) . The Macro-Action Decentralized Partially Observable Markov Decision Process ( MacDecPOMDP ) ( Amato et al. , 2014 ; 2019 ) provides a general formalism for multi-agent asynchronous collaborative decision-making under uncertainty . Macro-actions represent temporally extended actions that have ( potentially ) different durations . This introduces asynchronous high-level decisionmaking over agents , as agents can start and terminate macro-actions at different timesteps . Such asynchronicity actually makes multi-agent reinforcement learning ( MARL ) more challenging because it is difficult to determine what information to use and when to update agents ’ policies from either the decentralized or centralized perspective . Despite several efforts made recently to enable agents to learn asynchronous hierarchical policies such as extending DQN ( Mnih et al. , 2015 ) to learn macro-action-value functions ( Xiao et al. , 2019 ) , transferring MacDec-POMDPs to event-driven processes with continuous timing ( Menda et al. , 2019 ) , and adapting a single-agent option-critic framework ( Bacon et al. , 2017 ) to multi-agent domains to learn all components ( e.g . low-level policy , high-level abstraction , high-level policy ) from scratch ( Chakravorty et al. , 2019 ) , none of them provides a principled way for optimizing macroaction-based policies via asynchronous policy gradients to solve general multi-agent problems with asynchronous decision-making . In this paper , we propose a group of macro-action-based multi-agent actor-critic methods to generalize the current primitive-action-based multi-agent actor-critic methods to multi-agent problems with macro-actions as well as allowing asynchronous policy optimization . First , we formulate a macroaction-based independent actor-critic ( Mac-IAC ) method . Although independent learning suffers from a theoretical curse of environmental non-stationarity , it allows fully online learning and may still work well in certain domains . Second , we introduce a macro-action-based centralized actorcritic ( Mac-CAC ) method , for the case where full communication is available during execution . We also formulate a centralized training for decentralized execution ( CTDE ) paradigm ( Kraemer & Banerjee , 2016 ; Oliehoek et al. , 2008 ) variant of our method . CTDE has gained popularity since such methods can learn better decentralized policies by using centralized information during training . Current primitive-action-based multi-agent actor-critic methods typically use a centralized critic to optimize each decentralized actor . However , the asynchronous joint macro-action execution from the centralized perspective could be very different with the completion time being very different from each agent ’ s decentralized perspective . To this end , we first present a Naive Independent Actor with Centralized Critic ( Naive IACC ) method that naively uses a joint macro-action-value function as the critic for each actor ’ s policy gradient estimation ; and then propose an Independent Actor with Individual Centralized Critic ( Mac-IAICC ) method addressing the above challenge . We evaluate our proposed methods on diverse macro-action-based multi-agent problems : a benchmark Box Pushing domain ( Xiao et al. , 2019 ) , a variant of the Overcooked domain ( Wu et al. , 2021 ) and a larger warehouse service domain ( Xiao et al. , 2019 ) . Experimental results show that our methods are able to learn high-quality solutions while primitive-action-based methods can not , and show the strength of Mac-IAICC for learning decentralized policies over Naive IAICC and Mac-IAC . To our knowledge , this is the first general formalization of macro-action-based multi-agent actor-critic frameworks considering the three state-of-the-art multi-agent training paradigms . 2 BACKGROUND . This section first introduces the formal definitions of the Dec-POMDP and the MacDec-POMDP , and then reviews single-agent and multi-agent actor-critic policy gradient methods with primitiveactions . We also provide an overview of value-based MARL methods with macro-actions . 2.1 DEC-POMDPS AND MACDEC-POMDPS . The decentralized partially observable Markov decision processes ( Dec-POMDP ) ( Oliehoek & Amato , 2016 ) is a general framework to model fully cooperative multi-agent tasks , where agents make decisions in a decentralized way based on only local information . Formally , a Dec-POMDP is defined by a tuple 〈I , S , A , Ω , T , O , R , H , γ〉 , where I is a set of agents ; S is the environmental state space ; A = ×i∈IAi is the joint primitive-action space over each agent ’ s primitive-action set Ai ; Ω = ×i∈IΩi is the joint primitive-observation space over each agent ’ s primitive-observation set Ωi . At every timestep , under a state s , agents synchronously execute a joint primitive-action~a = ×i∈Iai , each individually selected by an agent using a policy πi : HAi × Ai → [ 0 , 1 ] , a mapping from local primitive observation-action historyHAi to primitive-actions . The environment then transits to a new state s′ according to a state transition function T ( s , ~a , s′ ) = P ( s′ | s , ~a ) . Agents receive a global reward r ( s , ~a ) issued by a reward function R : S×A→ R , and obtain a joint primitive-observation ~o = ×i∈Ioi drawn from an observation function O ( ~o , ~a , s′ ) = P ( ~o | ~a , s′ ) in state s′ . The objective is to find a joint policy ~π = ×iπi such that the expected sum of discounted rewards from an initial state , V ~π ( s ( 0 ) ) = E [ ∑H−1 t=0 γ tr ( s ( t ) , ~a ( t ) ) | s ( 0 ) , ~π ] , gets optimized , where γ ∈ [ 0 , 1 ] is a discount factor , and H is the number of ( primitive ) timesteps until the problem terminates ( the horizon ) . The macro-action decentralized partially observable Markov decision process ( MacDecPOMDP ) ( Amato et al. , 2014 ; 2019 ) incorporates the option framework ( Sutton et al. , 1999 ) into the Dec-POMDP by defining each agent ’ s macro-action as a tuple mi = 〈Imi , πmi , βmi〉 , where the initiation set Imi ⊂ HMi defines how to initiate a macro-action based on macro-observationaction history HMi at the high-level ; πmi : H A i × Ai → [ 0 , 1 ] is the low-level policy for the execution of a macro-action ; and a stochastic termination function βmi : H A i → [ 0 , 1 ] determines how to terminate a macro-action based on primitive-observation-action history HAi at the low-level . A MacDec-POMDP is thus formally defined by a tuple 〈I , S , A , M , Ω , ζ , T , O , Z , R , H , γ〉 , where I , S , A , Ω , T , O , R , H and γ remain the same definitions as in the Dec-POMDP ; M = ×i∈IMi is the joint macro-action space over each agent ’ s macro-action space Mi ; ζ = ×i∈Iζi is the joint macroobservation space over each agent ’ s macro-observation space ζi ; and Z = { Zi } i∈I is a set of macroobservation likelihood models . During execution , each agent independently selects a macro-action mi using a high-level policy Ψi : HMi ×Mi → [ 0 , 1 ] , a mapping from macro-observation-action his- tory to macro-actions , and captures a macro-observation zi ∈ ζi according to the macro-observation probability function Zi ( zi , mi , s′ ) = P ( zi | mi , s′ ) when the macro-action terminates in a state s′ . The objective of solving MacDec-POMDPs with finite horizon is to find a joint high-level policy ~Ψ = ×i∈IΨi that maximizes the value , V ~Ψ ( s ( 0 ) ) = E [ ∑H−1 t=0 γ tr ( s ( t ) , ~a ( t ) ) | s ( 0 ) , ~π , ~Ψ ] . 2.2 SINGLE-AGENT ACTOR-CRITIC . In single-agent reinforcement learning , the policy gradient theorem ( Sutton et al. , 2000 ) formulates a principled way to optimize a parameterized policy πθ via gradient ascent on the policy ’ s performance defined as J ( θ ) = Eπθ [ ∑∞ t=0 γ tr ( s ( t ) , a ( t ) ) ] . In POMDPs , the gradient w.r.t . parameters of a observation-action history-based policy πθ ( a | h ) is expressed as : ∇θJ ( θ ) = Eπθ [ ∇θ log πθ ( a | h ) Qπθ ( h , a ) ] ( 1 ) where , h is often maintained by having a RNN in the policy network ( Hausknecht & Stone , 2015 ) . The actor-critic framework ( Konda & Tsitsiklis , 2000 ) learns an on-policy action-value function Qπθφ ( h , a ) ( critic ) via temporal-difference ( TD ) learning ( Sutton , 1988 ) to approximate the actionvalue for the policy ( actor ) updates . Variance reduction is commonly achieved by training a historyvalue function V πθw ( h ) and using it as a baseline ( Weaver & Tao , 2001 ) as well as bootstrapping to estimate the action-value . Accordingly , the actor-critic policy gradient can be written as : ∇θJ ( θ ) = Eπθ [ ∇θ log πθ ( a | h ) ( r + γV πθw ( h ′ ) − V πθw ( h ) ) ] ( 2 ) where , r is the immediate reward received by the agent at the corresponding timestep . 2.3 INDEPENDENT ACTOR-CRITIC . The single-agent actor-critic algorithm can be adapted to multi-agent problems in a simple way such that each agent independently learns its own actor and critic while treating other agents as part of the world ( Foerster et al. , 2018 ) . We consider a variance reduction version of independent actor-critic ( IAC ) with the policy gradient as follows : ∇θiJ ( θi ) = E~π~θ [ ∇θi log πθi ( ai | hi ) ( r + γV πθi wi ( h ′ i ) − V πθi wi ( hi ) ) ] ( 3 ) where , r is a shared reward over agents at every timestep . Due to other agents ’ policy updating and exploring , from any agent ’ s local perspective , the environment appears non-stationary which can lead to unstable learning of the critic without convergence guarantees ( Lowe et al. , 2017 ) . This instability often prevents IAC from learning high-quality cooperative policies . 2.4 INDEPENDENT ACTOR WITH CENTRALIZED CRITIC . To address the above difficulties existing in independent learning approaches , centralized training for decentralized execution ( CTDE ) provides agents with access to global information during offline training while allowing agents to rely on only local information during online decentralized execution . Typically , the key idea of exploiting CTDE with actor-critic is to train a joint action-value function , Q~π~θφ ( x , ~a ) , as the centralized critic and use it to compute gradients w.r.t . the parameters of each decentralized policy ( Foerster et al. , 2018 ; Lowe et al. , 2017 ) , which can be formulated as : ∇θiJ ( θi ) = E~π~θ [ ∇θi log πθi ( ai | hi ) Q ~π~θ φ ( x , ~a ) ] ( 4 ) where , x represents the available centralized information ( e.g. , joint observation , joint observationaction history , or the true state ) . Although the centralized critic in Eq . 4 can facilitate the update of decentralized policies in the direction that optimizes global collaborative performance , it also introduces extra variance over other agents ’ actions ( Lyu et al. , 2021 ; Wang et al. , 2021 ) . Therefore , we consider the version of independent actor with centralized critic ( IACC ) with a general variance reduction trick ( Foerster et al. , 2018 ; Su et al. , 2021 ) , the policy gradient of which is : ∇θiJ ( θi ) = E~π~θ [ ∇θi log πθi ( ai | hi ) ( r + γV ~π~θ w ( x ′ ) − V ~π~θw ( x ) ) ] ( 5 )

Summary: This paper considers the problem of MacDec-POMDPs where it requires agents to be capable of performing asynchronously without waiting for other agents to terminate. As there is no multi-agent policy gradient method with macro-actions for MacDec-POMDPs, this paper fill this gap and integrates the macro-action-value into multi-agent policy gradient and proposes (i) macro-action-based independent actor-critic (Mac-IAC) method, (ii) macro-action-based centralized actor-critic (Mac-CAC) method, (iii) Naive Independent Actor with Centralized Critic (Naive IACC) as well as Independent Actor with Individual Centralized Critic (Mac-IAICC) via CTDE. Experimental results show that the proposed methods outperforms vanilla baselines.