Coordinated Multi-Agent Exploration Using Shared Goals

1 INTRODUCTION . Cooperative multi-agent reinforcement learning ( MARL ) is an increasingly important field . Indeed , many real-world problems are naturally modeled using MARL techniques . For instance , tasks from areas as diverse as robot fleet coordination ( Swamy et al. , 2020 ; Hüttenrauch et al. , 2019 ) and autonomous traffic control ( Bazzan , 2008 ; Sunehag et al. , 2018 ) fit MARL formulations . To address MARL problems , early work followed the independent single-agent reinforcement learning paradigm ( Tampuu et al. , 2015 ; Tan , 1993 ; Matignon et al. , 2012 ) . However , more recently , specifically tailored techniques such as monotonic value function factorization ( QMIX ) ( Rashid et al. , 2018 ) , multi-agent deep deterministic policy gradient ( MADDPG ) ( Lowe et al. , 2017 ) , and counterfactual multi-agent policy gradients ( COMA ) ( Foerster et al. , 2018 ) have been developed . Those methods excel in a multi-agent setting because they address the non-stationary issue of MARL and develop communication protocols between agents . Despite those advances and the resulting reported performance improvements , a common issue remained : all of the aforementioned methods use exploration techniques from classical algorithms . Specifically , these methods employ noise-based exploration , i.e. , the exploration policy is a noisy version of the actor policy . For instance , Lowe et al . ( 2017 ) add Ornstein-Uhlenbeck ( OU ) ( Uhlenbeck & Ornstein , 1930 ) noise or Gaussian noise to the actor policy . Foerster et al . ( 2016 ) ; Rashid et al . ( 2018 ) ; Yang et al . ( 2018 ) ; Foerster et al . ( 2017 ) use variants of ǫ-greedy exploration , where a random suboptimal action is selected with probability ǫ . It was recognized recently that use of classical exploration techniques is sub-optimal in a multi-agent reinforcement learning setting . Specifically , Mahajan et al . ( 2019 ) show that QMIX with ǫ-greedy exploration results in slow exploration and sub-optimality . Mahajan et al . ( 2019 ) improve exploration by conditioning an agent ’ s behavior on a shared latent variable controlled by a hierarchical policy . Even more recently , Wang et al . ( 2020 ) encourage coordinated exploration by considering the influence of one agent ’ s behavior on other agents ’ behaviors . While all of the aforementioned exploration techniques for multi-agent reinforcement learning significantly improve results , they suffer from a common challenge : agents struggle to identify states that are worth exploring , and don ’ t coordinate their exploration efforts toward those states . To give an example , consider a push-box task , where two agents need to jointly push a heavy box to a specific location before observing a reward . In this situation , instead of exploring the environment independently , agents need to coordinate pushing the box within the environment to find the specific location . To address this issue , we propose coordinated multi-agent exploration ( CMAE ) , where multiple agents share a common goal . We achieve this by first projecting the joint state space to multiple subspaces . We develop a normalized entropy ( Cover & Thomas. , 1991 ) -based technique to select a goal from the under-explored subspaces . Then , exploration policies are trained to reach the goals in a coordinated manner . To show that CMAE improves results , we evaluate our approach on various environments with sparse-rewards from Wang et al . ( 2020 ) , and the sparse-reward version of the Starcraft multi-agent challenge ( SMAC ) ( Samvelyan et al. , 2019 ) , which requires coordinated actions among agents over extended time steps before observing a reward . The experimental results show that our approach needs only 1 % − 5 % of environment steps to achieve similar or better average test episode returns than current state-of-the-art baselines . 2 PRELIMINARIES . In this section , we define the multi-agent Markov decision process ( MDP ) in Sec . 2.1 , and introduce the multi-agent reinforcement learning setting in Sec . 2.2 . 2.1 MULTI-AGENT MARKOV DECISION PROCESS . We model a cooperative multi-agent system as a multi-agent Markov decision process ( MDP ) . An n-agent MDP is defined by a tuple G = ( S , A , T , R , Z , O , n , γ , H ) . S is the global state space of the environment . A is the action space . At each time step t , each agent ’ s policy πi , i ∈ { 1 , . . . , n } , selects an action ati ∈ A . All selected actions form a joint action a t ∈ An . The transition function T maps the current state st and the joint action at to the next state st+1 , i.e. , T : S ×An → S . All agents receive a collective reward rt ∈ R according to the reward function R : S × An → R. The goal of all agents ’ policies is to maximize the collective expected return ∑H t=0 γ trt , where γ ∈ [ 0 , 1 ] is the discount factor , H is the horizon , and rt is the collective reward obtained at timestep t. Each agent i observes local observation oti ∈ Z according to the observation function O : S → Z . Note , observations usually reveal partial information about the global state . For instance , suppose the global state contains the location of agents , while the local observation of an agent may only contain the location of other agents within a limited distance . All agents ’ local observations form a joint observation , denoted by ot . A global state space S is the product of component spaces Vi , i.e. , S = ∏M i=1 Vi , where Vi ⊆ R ( Samvelyan et al. , 2019 ; Lowe et al. , 2017 ; Rashid et al. , 2018 ; Foerster et al. , 2018 ; Mahajan et al. , 2019 ) . We refer to Vi as a ‘ state component. ’ The set of all component spaces of a product space is referred to as the component set . For instance , the component set of S is { Vi|i ∈ { 1 , . . . , M } } . Each entity , e.g. , agents , objects , etc. , in the environment are described by a set of state components . We refer to a set of state components that is associated with an entity in the environment as an ‘ entity set. ’ For instance , in a 2-agent push-box environment , where two agents can only collaboratively push a box to a goal location , we have the global state space S = ∏6 i=1 Vi , where { V1 , V2 } , { V3 , V4 } , { V5 , V6 } represent the location of agent one , agent two , and the box , separately . Consequently , { V1 , V2 } , { V3 , V4 } , { V5 , V6 } are three entity sets . 2.2 MULTI-AGENT REINFORCEMENT LEARNING . In this paper , we follow the standard centralized training and decentralized execution ( CTDE ) paradigm ( Lowe et al. , 2017 ; Rashid et al. , 2018 ; Foerster et al. , 2018 ; Mahajan et al. , 2019 ) . That is , at training time , the learning algorithm has access to all agents ’ local observations , actions , and the global state . At execution time , i.e. , at test time , each individual agent ’ s policy only has access to its own local observation . The proposed CMAE is applicable to off-policy MARL methods ( e.g. , Rashid et al. , 2018 ; Lowe et al. , 2017 ; Sunehag et al. , 2018 ; Matignon et al. , 2012 ) . In off-policy MARL , exploration poli- Algorithm 1 : Training with Coordinated Multi-Agent Exploration ( CMAE ) Initialize exploration policies { µi } n i=1 , target policies { πi } n i=1 , counters { ck } K k=1 ; Initialize the environment and replay buffer D ; Initialize α = 1 ; for episode = 1 . . .M do Reset the environment , and observe global states st and observations ot = ( ot1 . . . o t n ) ; for t = 1 . . . T do UpdateCounter ( { ck } K k=1 , s t , ot ) ; Select actions at using a mixture of exploration and target policies αµi + ( 1− α ) πi , α decreases linearly to 0 ; Apply at to the environment ; Observe rewards rt , state st+1 , and local observations ot+1 ; Add transition tuple { st , ot , a , st+1 , ot+1 , rt } to D ; TrainTarget ( { πi } n i=1 , D ) ; end TrainExp ( { µi } n i=1 , { ck } K k=1 , D ) ; end cies µi , i ∈ { 1 , . . . , n } are responsible for collecting data from the environment . The data in the form of transition tuple ( st , ot , at , st+1 , ot+1 ) is stored in a replay memory D , i.e. , D = { ( st , ot , at , st+1 , ot+1 ) } t. The target policies are trained using transition tuples from the replay memory . 3 COORDINATED MULTI-AGENT EXPLORATION ( CMAE ) . In the following we first present an overview of CMAE before we discuss the method more formally . Overview : The goal is to train the target policies { πi } i∈ { 1 , ... , n } of n agents to maximize the environment episode return . Classical off-policy algorithms ( Lowe et al. , 2017 ; Rashid et al. , 2018 ) typically use a noisy version of the target policies πi as the exploration policies µi , i.e. , to collect data actions are selected based on exploration policies µi . In contrast , in CMAE , we propose to train the exploration policies by training with a modified reward . Specifically , target polices are trained to maximize the usual external episode return . In contrast , exploration policies are trained to collect data from subspaces that haven ’ t been well explored . We find this strategy to significantly improve training of target policies in the multi-agent reinforcement learning setting because this strategy can encourage multiple agents to jointly explore configurations of the state space . Alg . 1 summarizes this approach . At each step , a mixture of the exploration policies { µi } n i=1 and target policies { πi } n i=1 are used to select actions . The resulting experience tuple is then stored in a replay memory D. The target policies are trained directly using the data within the replay memory D at each step . Note that the exploration policies will only be updated at the end of each episode using a reshaped reward that encourages exploration polices to explore under-explored subspaces in a collaborative manner . In the following we will provide details about how we propose to train the exploration policies . 3.1 TRAINING OF EXPLORATION POLICIES . To train the exploration policies µi , i ∈ { 1 , . . . , n } we use a modified reward r̂ . This modified reward specifies the goal of the exploration . For example , in the two-agent push-box task , we specify a specific joint location of both agents and the box as a goal . Note , the agents will ignore all external rewards and only see positive reward when the goal , i.e. , the specified position is reached . The reward for the goal is set to b while the rewards for all other situations are zero . To find the goal situation we use K counters ck , k ∈ { 1 , . . . , K } . A counter ck operates on a low-dimensional subspace Sk of the state space S , i.e. , Sk ⊆ S . Occurrence of every configuration sk ∈ Sk within the low-dimensional subspace will be recorded using the current replay buffer D. Algorithm 2 : Train Exploration Policies ( TrainExp ) Input : exploration policies { µi } n i=1 , counters { ck } K k=1 , replay buffer D ; Initialize bonus b ; Compute normalized entropy η ( k ) of subspace k based on associated counter ck ; k∗ = argmink η ( k ) ; Sample a batch B = { si } M i=1 from D ; g = argmins∈B ck∗ ( projk∗ ( s ) ) ; for { st , ot , a , st+1 , ot+1 , rt } ∈ D do if st == g then rt = b ; else rt = 0 ; end Update { µi } n i=1 by { s t , ot , a , st+1 , ot+1 , rt } end Let projk be the projection from global state space to the subspace k. Formally , we obtain ck ( sk ) = ∑ s∈D ✶ [ projk ( s ) = sk ] , where ✶ [ · ] is the indicator function ( 1 if argument is true ; 0 otherwise ) and projk ( s ) denotes the restriction of state s ∈ S to subspace Sk . Note that we are successively incrementing the counts , i.e. , the counters ck are not recomputed from scratch every time we train the exploration policies . We subsequently normalize the counters ck ( sk ) into a probability distribution pk ( sk ) = ck ( sk ) / ∑ ŝk∈Sk ck ( ŝk ) which is then used to compute a normalized entropy ηk = H/Hmax = − ( ∑ s∈Sk pk ( s ) log pk ( s ) ) / log ( |Sk| ) . We select the subspace k ∗ with the smallest normalized entropy . From this subspace we choose the joint goal state g by first sampling a batch of states B from the replay buffer . From those states we select in a second step the state with the smallest count as the goal state g , i.e. , g = argmins∈B ck∗ ( s ) . Sampling of states is performed in order to avoid selecting unreachable states as a goal , i.e. , we encourage to explore states that we have seen rarely but at least once . Given the goal state g , we train the exploration policies µi using the replay buffer D modified by a revised reward r̂ = b if sk∗ = g. Note , r̂ = 0 otherwise . Consequently , the exploration policies µi focus exclusively on achieving the desired goal g. This strategy is summarized in Alg . 2 . As an alternative to the aforementioned subspace selection method , one could use probabilistic subspace selection , where the probability of a subspace being chosen is inversely proportional to its normalized entropy . The two different subspace selection approaches result in different exploration behaviors . Specifically , the probabilistic subspace selection will encourage exploration policies to explore more subspaces while the smallest normalized entropy method focuses on the most underexplored subspace .

Summary: The paper introduces an exploration bonus tailored to multi-agent learning in the CTDE (centralized training, decentralizing execution) setting. The bonus works by: 1) dividing up the observation space into subspaces (in this case, corresponding to the entities, pairs of entities, triplets of entities, etc), 2) maintaining running counters within each subspace of every possible configuration within that subspace, 3) identifying the lowest entropy subspace, 4) sampling the replay buffer to find a rarely occurring (but importantly, reachable) “goal” state within the lowest entropy subspace, and finally, 5) rewarding the exploration policy for visiting the goal state. The authors test their method in two domains: 1) a series of coordinated multi-agent exploration challenges in grid worlds, and 2) the Starcraft Multi-Agent Challenge (SMAC). They demonstrate much faster learning over a series of baselines.