Asynchronous Multi-Agent Actor-Critic with Macro-Actions

1 INTRODUCTION . In recent years , multi-agent policy gradient methods using the actor-critic framework have achieved impressive success in solving a variety of cooperative and competitive domains ( Lowe et al. , 2017 ; Foerster et al. , 2018 ; Du et al. , 2019 ; Iqbal & Sha , 2019 ; Vinyals et al. , 2019 ; Li et al. , 2019 ; Wang et al. , 2020 ; Yang et al. , 2020 ; Zhou et al. , 2020 ; Baker et al. , 2020 ; Su et al. , 2021 ; Wang et al. , 2021 ; Du et al. , 2021 ) . However , as these methods assume synchronized primitive-action execution over agents , they struggle to solve tasks that involve long-term reasoning and asynchronous behavior , such as real-world multi-robot applications ( e.g. , search and rescue ( Queralta et al. , 2020 ) , package delivery ( Choudhury et al. , 2021 ) and warehouse service ( Xiao et al. , 2020 ) ) . The Macro-Action Decentralized Partially Observable Markov Decision Process ( MacDecPOMDP ) ( Amato et al. , 2014 ; 2019 ) provides a general formalism for multi-agent asynchronous collaborative decision-making under uncertainty . Macro-actions represent temporally extended actions that have ( potentially ) different durations . This introduces asynchronous high-level decisionmaking over agents , as agents can start and terminate macro-actions at different timesteps . Such asynchronicity actually makes multi-agent reinforcement learning ( MARL ) more challenging because it is difficult to determine what information to use and when to update agents ’ policies from either the decentralized or centralized perspective . Despite several efforts made recently to enable agents to learn asynchronous hierarchical policies such as extending DQN ( Mnih et al. , 2015 ) to learn macro-action-value functions ( Xiao et al. , 2019 ) , transferring MacDec-POMDPs to event-driven processes with continuous timing ( Menda et al. , 2019 ) , and adapting a single-agent option-critic framework ( Bacon et al. , 2017 ) to multi-agent domains to learn all components ( e.g . low-level policy , high-level abstraction , high-level policy ) from scratch ( Chakravorty et al. , 2019 ) , none of them provides a principled way for optimizing macroaction-based policies via asynchronous policy gradients to solve general multi-agent problems with asynchronous decision-making . In this paper , we propose a group of macro-action-based multi-agent actor-critic methods to generalize the current primitive-action-based multi-agent actor-critic methods to multi-agent problems with macro-actions as well as allowing asynchronous policy optimization . First , we formulate a macroaction-based independent actor-critic ( Mac-IAC ) method . Although independent learning suffers from a theoretical curse of environmental non-stationarity , it allows fully online learning and may still work well in certain domains . Second , we introduce a macro-action-based centralized actorcritic ( Mac-CAC ) method , for the case where full communication is available during execution . We also formulate a centralized training for decentralized execution ( CTDE ) paradigm ( Kraemer & Banerjee , 2016 ; Oliehoek et al. , 2008 ) variant of our method . CTDE has gained popularity since such methods can learn better decentralized policies by using centralized information during training . Current primitive-action-based multi-agent actor-critic methods typically use a centralized critic to optimize each decentralized actor . However , the asynchronous joint macro-action execution from the centralized perspective could be very different with the completion time being very different from each agent ’ s decentralized perspective . To this end , we first present a Naive Independent Actor with Centralized Critic ( Naive IACC ) method that naively uses a joint macro-action-value function as the critic for each actor ’ s policy gradient estimation ; and then propose an Independent Actor with Individual Centralized Critic ( Mac-IAICC ) method addressing the above challenge . We evaluate our proposed methods on diverse macro-action-based multi-agent problems : a benchmark Box Pushing domain ( Xiao et al. , 2019 ) , a variant of the Overcooked domain ( Wu et al. , 2021 ) and a larger warehouse service domain ( Xiao et al. , 2019 ) . Experimental results show that our methods are able to learn high-quality solutions while primitive-action-based methods can not , and show the strength of Mac-IAICC for learning decentralized policies over Naive IAICC and Mac-IAC . To our knowledge , this is the first general formalization of macro-action-based multi-agent actor-critic frameworks considering the three state-of-the-art multi-agent training paradigms . 2 BACKGROUND . This section first introduces the formal definitions of the Dec-POMDP and the MacDec-POMDP , and then reviews single-agent and multi-agent actor-critic policy gradient methods with primitiveactions . We also provide an overview of value-based MARL methods with macro-actions . 2.1 DEC-POMDPS AND MACDEC-POMDPS . The decentralized partially observable Markov decision processes ( Dec-POMDP ) ( Oliehoek & Amato , 2016 ) is a general framework to model fully cooperative multi-agent tasks , where agents make decisions in a decentralized way based on only local information . Formally , a Dec-POMDP is defined by a tuple 〈I , S , A , Ω , T , O , R , H , γ〉 , where I is a set of agents ; S is the environmental state space ; A = ×i∈IAi is the joint primitive-action space over each agent ’ s primitive-action set Ai ; Ω = ×i∈IΩi is the joint primitive-observation space over each agent ’ s primitive-observation set Ωi . At every timestep , under a state s , agents synchronously execute a joint primitive-action~a = ×i∈Iai , each individually selected by an agent using a policy πi : HAi × Ai → [ 0 , 1 ] , a mapping from local primitive observation-action historyHAi to primitive-actions . The environment then transits to a new state s′ according to a state transition function T ( s , ~a , s′ ) = P ( s′ | s , ~a ) . Agents receive a global reward r ( s , ~a ) issued by a reward function R : S×A→ R , and obtain a joint primitive-observation ~o = ×i∈Ioi drawn from an observation function O ( ~o , ~a , s′ ) = P ( ~o | ~a , s′ ) in state s′ . The objective is to find a joint policy ~π = ×iπi such that the expected sum of discounted rewards from an initial state , V ~π ( s ( 0 ) ) = E [ ∑H−1 t=0 γ tr ( s ( t ) , ~a ( t ) ) | s ( 0 ) , ~π ] , gets optimized , where γ ∈ [ 0 , 1 ] is a discount factor , and H is the number of ( primitive ) timesteps until the problem terminates ( the horizon ) . The macro-action decentralized partially observable Markov decision process ( MacDecPOMDP ) ( Amato et al. , 2014 ; 2019 ) incorporates the option framework ( Sutton et al. , 1999 ) into the Dec-POMDP by defining each agent ’ s macro-action as a tuple mi = 〈Imi , πmi , βmi〉 , where the initiation set Imi ⊂ HMi defines how to initiate a macro-action based on macro-observationaction history HMi at the high-level ; πmi : H A i × Ai → [ 0 , 1 ] is the low-level policy for the execution of a macro-action ; and a stochastic termination function βmi : H A i → [ 0 , 1 ] determines how to terminate a macro-action based on primitive-observation-action history HAi at the low-level . A MacDec-POMDP is thus formally defined by a tuple 〈I , S , A , M , Ω , ζ , T , O , Z , R , H , γ〉 , where I , S , A , Ω , T , O , R , H and γ remain the same definitions as in the Dec-POMDP ; M = ×i∈IMi is the joint macro-action space over each agent ’ s macro-action space Mi ; ζ = ×i∈Iζi is the joint macroobservation space over each agent ’ s macro-observation space ζi ; and Z = { Zi } i∈I is a set of macroobservation likelihood models . During execution , each agent independently selects a macro-action mi using a high-level policy Ψi : HMi ×Mi → [ 0 , 1 ] , a mapping from macro-observation-action his- tory to macro-actions , and captures a macro-observation zi ∈ ζi according to the macro-observation probability function Zi ( zi , mi , s′ ) = P ( zi | mi , s′ ) when the macro-action terminates in a state s′ . The objective of solving MacDec-POMDPs with finite horizon is to find a joint high-level policy ~Ψ = ×i∈IΨi that maximizes the value , V ~Ψ ( s ( 0 ) ) = E [ ∑H−1 t=0 γ tr ( s ( t ) , ~a ( t ) ) | s ( 0 ) , ~π , ~Ψ ] . 2.2 SINGLE-AGENT ACTOR-CRITIC . In single-agent reinforcement learning , the policy gradient theorem ( Sutton et al. , 2000 ) formulates a principled way to optimize a parameterized policy πθ via gradient ascent on the policy ’ s performance defined as J ( θ ) = Eπθ [ ∑∞ t=0 γ tr ( s ( t ) , a ( t ) ) ] . In POMDPs , the gradient w.r.t . parameters of a observation-action history-based policy πθ ( a | h ) is expressed as : ∇θJ ( θ ) = Eπθ [ ∇θ log πθ ( a | h ) Qπθ ( h , a ) ] ( 1 ) where , h is often maintained by having a RNN in the policy network ( Hausknecht & Stone , 2015 ) . The actor-critic framework ( Konda & Tsitsiklis , 2000 ) learns an on-policy action-value function Qπθφ ( h , a ) ( critic ) via temporal-difference ( TD ) learning ( Sutton , 1988 ) to approximate the actionvalue for the policy ( actor ) updates . Variance reduction is commonly achieved by training a historyvalue function V πθw ( h ) and using it as a baseline ( Weaver & Tao , 2001 ) as well as bootstrapping to estimate the action-value . Accordingly , the actor-critic policy gradient can be written as : ∇θJ ( θ ) = Eπθ [ ∇θ log πθ ( a | h ) ( r + γV πθw ( h ′ ) − V πθw ( h ) ) ] ( 2 ) where , r is the immediate reward received by the agent at the corresponding timestep . 2.3 INDEPENDENT ACTOR-CRITIC . The single-agent actor-critic algorithm can be adapted to multi-agent problems in a simple way such that each agent independently learns its own actor and critic while treating other agents as part of the world ( Foerster et al. , 2018 ) . We consider a variance reduction version of independent actor-critic ( IAC ) with the policy gradient as follows : ∇θiJ ( θi ) = E~π~θ [ ∇θi log πθi ( ai | hi ) ( r + γV πθi wi ( h ′ i ) − V πθi wi ( hi ) ) ] ( 3 ) where , r is a shared reward over agents at every timestep . Due to other agents ’ policy updating and exploring , from any agent ’ s local perspective , the environment appears non-stationary which can lead to unstable learning of the critic without convergence guarantees ( Lowe et al. , 2017 ) . This instability often prevents IAC from learning high-quality cooperative policies . 2.4 INDEPENDENT ACTOR WITH CENTRALIZED CRITIC . To address the above difficulties existing in independent learning approaches , centralized training for decentralized execution ( CTDE ) provides agents with access to global information during offline training while allowing agents to rely on only local information during online decentralized execution . Typically , the key idea of exploiting CTDE with actor-critic is to train a joint action-value function , Q~π~θφ ( x , ~a ) , as the centralized critic and use it to compute gradients w.r.t . the parameters of each decentralized policy ( Foerster et al. , 2018 ; Lowe et al. , 2017 ) , which can be formulated as : ∇θiJ ( θi ) = E~π~θ [ ∇θi log πθi ( ai | hi ) Q ~π~θ φ ( x , ~a ) ] ( 4 ) where , x represents the available centralized information ( e.g. , joint observation , joint observationaction history , or the true state ) . Although the centralized critic in Eq . 4 can facilitate the update of decentralized policies in the direction that optimizes global collaborative performance , it also introduces extra variance over other agents ’ actions ( Lyu et al. , 2021 ; Wang et al. , 2021 ) . Therefore , we consider the version of independent actor with centralized critic ( IACC ) with a general variance reduction trick ( Foerster et al. , 2018 ; Su et al. , 2021 ) , the policy gradient of which is : ∇θiJ ( θi ) = E~π~θ [ ∇θi log πθi ( ai | hi ) ( r + γV ~π~θ w ( x ′ ) − V ~π~θw ( x ) ) ] ( 5 )

Summary: The paper presents a method for learning policy gradient based methods on macro-actions in environments where multiple agents initiate and terminate actions at different timesteps. The paper identifies a suitable framework (MacDec-POMDPs) for these tasks and develops a series of actor-critic mechanisms to allow agents to coordinate asynchronous macros actions. Four main macro-action algorithms are proposed: * Mac-IAC - that applies a variance reduced version of actor critic directly to the multi-agent domain * Mac-CAC - which treats all agents as a single global agent * Mac-IACC - described as naive, which uses a centralised critic but derives independent policy gradients for each agent * Mac-IAICC - which again uses a centralised critic like mechanism but one where the termination times of the individual agent correct the traces used to train this central critic. Again this derives independent policy gradients for each agent.  The experimental section presents three models: box pushing, overcooked and warehouse. From my reading of the paper, these models provide predefined options to all agents (so all agents have access to all options), where some options are extended and some are single step. Rewards are defined that are broadcast to all agents, and are used to learn the coordination policy (high level) but not the low level options, in the case of the macro action algorithms. Two non-macro action methods are evaluated against Mac-IAC and Mac-CAC on Box-pushing and Overcooked. This finds that macro-action methods are better than the primitive action methods (less interesting - see notes) and that there are situations in which Mac-IAC is better but more typically Mac-CAC outperforms Mac-IAC when there is a significant need for coordination. The next series of experiments compares the four macro-actions methods and show (most pertinantly) that the Mac-IAICC method achieves better coordination than Mac-IAC, nearing the performance of Mac-CAC even in the most coordination intensive conditions. Naive Mac-IACC improves upon Mac-IAC in terms of coordination but suffers in coordination intensive tasks. This supports the authors' argument that the variance in the shared critic for Naive Mac-IACC is damaging and that the proposed individual centralised critic approach for Mac-IAICC significantly improves upon this.