Coordinated Multi-Agent Exploration Using Shared Goals

1 INTRODUCTION . Cooperative multi-agent reinforcement learning ( MARL ) is an increasingly important field . Indeed , many real-world problems are naturally modeled using MARL techniques . For instance , tasks from areas as diverse as robot fleet coordination ( Swamy et al. , 2020 ; Hüttenrauch et al. , 2019 ) and autonomous traffic control ( Bazzan , 2008 ; Sunehag et al. , 2018 ) fit MARL formulations . To address MARL problems , early work followed the independent single-agent reinforcement learning paradigm ( Tampuu et al. , 2015 ; Tan , 1993 ; Matignon et al. , 2012 ) . However , more recently , specifically tailored techniques such as monotonic value function factorization ( QMIX ) ( Rashid et al. , 2018 ) , multi-agent deep deterministic policy gradient ( MADDPG ) ( Lowe et al. , 2017 ) , and counterfactual multi-agent policy gradients ( COMA ) ( Foerster et al. , 2018 ) have been developed . Those methods excel in a multi-agent setting because they address the non-stationary issue of MARL and develop communication protocols between agents . Despite those advances and the resulting reported performance improvements , a common issue remained : all of the aforementioned methods use exploration techniques from classical algorithms . Specifically , these methods employ noise-based exploration , i.e. , the exploration policy is a noisy version of the actor policy . For instance , Lowe et al . ( 2017 ) add Ornstein-Uhlenbeck ( OU ) ( Uhlenbeck & Ornstein , 1930 ) noise or Gaussian noise to the actor policy . Foerster et al . ( 2016 ) ; Rashid et al . ( 2018 ) ; Yang et al . ( 2018 ) ; Foerster et al . ( 2017 ) use variants of ǫ-greedy exploration , where a random suboptimal action is selected with probability ǫ . It was recognized recently that use of classical exploration techniques is sub-optimal in a multi-agent reinforcement learning setting . Specifically , Mahajan et al . ( 2019 ) show that QMIX with ǫ-greedy exploration results in slow exploration and sub-optimality . Mahajan et al . ( 2019 ) improve exploration by conditioning an agent ’ s behavior on a shared latent variable controlled by a hierarchical policy . Even more recently , Wang et al . ( 2020 ) encourage coordinated exploration by considering the influence of one agent ’ s behavior on other agents ’ behaviors . While all of the aforementioned exploration techniques for multi-agent reinforcement learning significantly improve results , they suffer from a common challenge : agents struggle to identify states that are worth exploring , and don ’ t coordinate their exploration efforts toward those states . To give an example , consider a push-box task , where two agents need to jointly push a heavy box to a specific location before observing a reward . In this situation , instead of exploring the environment independently , agents need to coordinate pushing the box within the environment to find the specific location . To address this issue , we propose coordinated multi-agent exploration ( CMAE ) , where multiple agents share a common goal . We achieve this by first projecting the joint state space to multiple subspaces . We develop a normalized entropy ( Cover & Thomas. , 1991 ) -based technique to select a goal from the under-explored subspaces . Then , exploration policies are trained to reach the goals in a coordinated manner . To show that CMAE improves results , we evaluate our approach on various environments with sparse-rewards from Wang et al . ( 2020 ) , and the sparse-reward version of the Starcraft multi-agent challenge ( SMAC ) ( Samvelyan et al. , 2019 ) , which requires coordinated actions among agents over extended time steps before observing a reward . The experimental results show that our approach needs only 1 % − 5 % of environment steps to achieve similar or better average test episode returns than current state-of-the-art baselines . 2 PRELIMINARIES . In this section , we define the multi-agent Markov decision process ( MDP ) in Sec . 2.1 , and introduce the multi-agent reinforcement learning setting in Sec . 2.2 . 2.1 MULTI-AGENT MARKOV DECISION PROCESS . We model a cooperative multi-agent system as a multi-agent Markov decision process ( MDP ) . An n-agent MDP is defined by a tuple G = ( S , A , T , R , Z , O , n , γ , H ) . S is the global state space of the environment . A is the action space . At each time step t , each agent ’ s policy πi , i ∈ { 1 , . . . , n } , selects an action ati ∈ A . All selected actions form a joint action a t ∈ An . The transition function T maps the current state st and the joint action at to the next state st+1 , i.e. , T : S ×An → S . All agents receive a collective reward rt ∈ R according to the reward function R : S × An → R. The goal of all agents ’ policies is to maximize the collective expected return ∑H t=0 γ trt , where γ ∈ [ 0 , 1 ] is the discount factor , H is the horizon , and rt is the collective reward obtained at timestep t. Each agent i observes local observation oti ∈ Z according to the observation function O : S → Z . Note , observations usually reveal partial information about the global state . For instance , suppose the global state contains the location of agents , while the local observation of an agent may only contain the location of other agents within a limited distance . All agents ’ local observations form a joint observation , denoted by ot . A global state space S is the product of component spaces Vi , i.e. , S = ∏M i=1 Vi , where Vi ⊆ R ( Samvelyan et al. , 2019 ; Lowe et al. , 2017 ; Rashid et al. , 2018 ; Foerster et al. , 2018 ; Mahajan et al. , 2019 ) . We refer to Vi as a ‘ state component. ’ The set of all component spaces of a product space is referred to as the component set . For instance , the component set of S is { Vi|i ∈ { 1 , . . . , M } } . Each entity , e.g. , agents , objects , etc. , in the environment are described by a set of state components . We refer to a set of state components that is associated with an entity in the environment as an ‘ entity set. ’ For instance , in a 2-agent push-box environment , where two agents can only collaboratively push a box to a goal location , we have the global state space S = ∏6 i=1 Vi , where { V1 , V2 } , { V3 , V4 } , { V5 , V6 } represent the location of agent one , agent two , and the box , separately . Consequently , { V1 , V2 } , { V3 , V4 } , { V5 , V6 } are three entity sets . 2.2 MULTI-AGENT REINFORCEMENT LEARNING . In this paper , we follow the standard centralized training and decentralized execution ( CTDE ) paradigm ( Lowe et al. , 2017 ; Rashid et al. , 2018 ; Foerster et al. , 2018 ; Mahajan et al. , 2019 ) . That is , at training time , the learning algorithm has access to all agents ’ local observations , actions , and the global state . At execution time , i.e. , at test time , each individual agent ’ s policy only has access to its own local observation . The proposed CMAE is applicable to off-policy MARL methods ( e.g. , Rashid et al. , 2018 ; Lowe et al. , 2017 ; Sunehag et al. , 2018 ; Matignon et al. , 2012 ) . In off-policy MARL , exploration poli- Algorithm 1 : Training with Coordinated Multi-Agent Exploration ( CMAE ) Initialize exploration policies { µi } n i=1 , target policies { πi } n i=1 , counters { ck } K k=1 ; Initialize the environment and replay buffer D ; Initialize α = 1 ; for episode = 1 . . .M do Reset the environment , and observe global states st and observations ot = ( ot1 . . . o t n ) ; for t = 1 . . . T do UpdateCounter ( { ck } K k=1 , s t , ot ) ; Select actions at using a mixture of exploration and target policies αµi + ( 1− α ) πi , α decreases linearly to 0 ; Apply at to the environment ; Observe rewards rt , state st+1 , and local observations ot+1 ; Add transition tuple { st , ot , a , st+1 , ot+1 , rt } to D ; TrainTarget ( { πi } n i=1 , D ) ; end TrainExp ( { µi } n i=1 , { ck } K k=1 , D ) ; end cies µi , i ∈ { 1 , . . . , n } are responsible for collecting data from the environment . The data in the form of transition tuple ( st , ot , at , st+1 , ot+1 ) is stored in a replay memory D , i.e. , D = { ( st , ot , at , st+1 , ot+1 ) } t. The target policies are trained using transition tuples from the replay memory . 3 COORDINATED MULTI-AGENT EXPLORATION ( CMAE ) . In the following we first present an overview of CMAE before we discuss the method more formally . Overview : The goal is to train the target policies { πi } i∈ { 1 , ... , n } of n agents to maximize the environment episode return . Classical off-policy algorithms ( Lowe et al. , 2017 ; Rashid et al. , 2018 ) typically use a noisy version of the target policies πi as the exploration policies µi , i.e. , to collect data actions are selected based on exploration policies µi . In contrast , in CMAE , we propose to train the exploration policies by training with a modified reward . Specifically , target polices are trained to maximize the usual external episode return . In contrast , exploration policies are trained to collect data from subspaces that haven ’ t been well explored . We find this strategy to significantly improve training of target policies in the multi-agent reinforcement learning setting because this strategy can encourage multiple agents to jointly explore configurations of the state space . Alg . 1 summarizes this approach . At each step , a mixture of the exploration policies { µi } n i=1 and target policies { πi } n i=1 are used to select actions . The resulting experience tuple is then stored in a replay memory D. The target policies are trained directly using the data within the replay memory D at each step . Note that the exploration policies will only be updated at the end of each episode using a reshaped reward that encourages exploration polices to explore under-explored subspaces in a collaborative manner . In the following we will provide details about how we propose to train the exploration policies . 3.1 TRAINING OF EXPLORATION POLICIES . To train the exploration policies µi , i ∈ { 1 , . . . , n } we use a modified reward r̂ . This modified reward specifies the goal of the exploration . For example , in the two-agent push-box task , we specify a specific joint location of both agents and the box as a goal . Note , the agents will ignore all external rewards and only see positive reward when the goal , i.e. , the specified position is reached . The reward for the goal is set to b while the rewards for all other situations are zero . To find the goal situation we use K counters ck , k ∈ { 1 , . . . , K } . A counter ck operates on a low-dimensional subspace Sk of the state space S , i.e. , Sk ⊆ S . Occurrence of every configuration sk ∈ Sk within the low-dimensional subspace will be recorded using the current replay buffer D. Algorithm 2 : Train Exploration Policies ( TrainExp ) Input : exploration policies { µi } n i=1 , counters { ck } K k=1 , replay buffer D ; Initialize bonus b ; Compute normalized entropy η ( k ) of subspace k based on associated counter ck ; k∗ = argmink η ( k ) ; Sample a batch B = { si } M i=1 from D ; g = argmins∈B ck∗ ( projk∗ ( s ) ) ; for { st , ot , a , st+1 , ot+1 , rt } ∈ D do if st == g then rt = b ; else rt = 0 ; end Update { µi } n i=1 by { s t , ot , a , st+1 , ot+1 , rt } end Let projk be the projection from global state space to the subspace k. Formally , we obtain ck ( sk ) = ∑ s∈D ✶ [ projk ( s ) = sk ] , where ✶ [ · ] is the indicator function ( 1 if argument is true ; 0 otherwise ) and projk ( s ) denotes the restriction of state s ∈ S to subspace Sk . Note that we are successively incrementing the counts , i.e. , the counters ck are not recomputed from scratch every time we train the exploration policies . We subsequently normalize the counters ck ( sk ) into a probability distribution pk ( sk ) = ck ( sk ) / ∑ ŝk∈Sk ck ( ŝk ) which is then used to compute a normalized entropy ηk = H/Hmax = − ( ∑ s∈Sk pk ( s ) log pk ( s ) ) / log ( |Sk| ) . We select the subspace k ∗ with the smallest normalized entropy . From this subspace we choose the joint goal state g by first sampling a batch of states B from the replay buffer . From those states we select in a second step the state with the smallest count as the goal state g , i.e. , g = argmins∈B ck∗ ( s ) . Sampling of states is performed in order to avoid selecting unreachable states as a goal , i.e. , we encourage to explore states that we have seen rarely but at least once . Given the goal state g , we train the exploration policies µi using the replay buffer D modified by a revised reward r̂ = b if sk∗ = g. Note , r̂ = 0 otherwise . Consequently , the exploration policies µi focus exclusively on achieving the desired goal g. This strategy is summarized in Alg . 2 . As an alternative to the aforementioned subspace selection method , one could use probabilistic subspace selection , where the probability of a subspace being chosen is inversely proportional to its normalized entropy . The two different subspace selection approaches result in different exploration behaviors . Specifically , the probabilistic subspace selection will encourage exploration policies to explore more subspaces while the smallest normalized entropy method focuses on the most underexplored subspace .

Summary: The paper proposes to improve the exponential sample complexity of finding a coordinated multi-agent strategy by learning an exploration policy for each agent that conditions on a shared goal. The exploration policy is mixed with the normal RL policy according to a parameter alpha, which is scaled down over time. The shared goal that agents pursue is selected by using an explicit counter mechanism over objects in the environment.