Divide and Explore: Multi-Agent Separate Exploration with Shared Intrinsic Motivations

1 INTRODUCTION . Deep reinforcement learning is a very important learning framework to train intelligent agents to solve complex tasks ( Sutton & Barto , 2018 ) . Despite recent success stories in a variety of real tasks ( Silver et al. , 2016 ; Vinyals et al. , 2019 ; Lee et al. , 2020 ) , the difficulty of exploration remains a major challenge for DRL when the reward is sparse or deceptive ( Burda et al. , 2018 ) . Recent research in exploration has made significant progresses thanks to the idea of intrinsic motivation ( Schmidhuber , 2010 ) . Intrinsic motivation could be viewed as objectives for unsupervised reinforcement learning ( Levine , 2020 ) , and their goal is usually not to guide the agent to complete a specific task , but to help the agent to discover interesting states and trajectories which could help the agent to understand the environment and find possible target solutions for specific tasks . Many previous works on intrinsic motivation ( Raileanu & Rocktäschel , 2020 ; Burda et al. , 2018 ; 2019 ) can be understood as proposing some kind of metrics to measure the novelty of the states , and then using the metric as an intrinsic reward to encourage agents to reach novel states , such as count-based exploration ( Bellemare et al. , 2016 ) and ICM ( Pathak et al. , 2017 ) . Despite of the significant progresses that have been made , in theory , efficient exploration is still very challenging if not impossible , due to the infeasible size of the state space to be explored . Similar difficulties are usually seen in the state-space search problems in theoretical computer science . For example , in SMT or SAT problems ( De Moura & Bjørner , 2008 ) , one is required to search for a feasible solution to satisfy certain constraints , and these problems are usually NP complete because of the exponential size of the state space . A very common class of practical algorithms to solve these search problems is divide and conquer . The algorithm first tries to divide the state space into certain pieces and then search for results in each piece of state space . A significant benefit of this approach is it can be easily modified to take advantage of modern distributed/concurrent computing hardware , because it can naturally divide a hard problem into smaller and simpler ones , allowing different computing nodes to complete the tasks asynchronously . Divide-and-conquer type of algorithms are among the state-of-the art algorithms to solve SMT or SAT problems . In this paper , we take inspiration from divide and conquer algorithms in theoretical computer science and try to adapt them to the exploration problems in RL . Different from traditional exploration mechanisms with a single exploring agent , we train a number of concurrent exploring agents . We carefully design our intrinsic reward such that each exploring agent is encouraged to explore only one region in the state space . In other words , the state space is automatically divided into several components , where each agent is responsible for exploring one of these components . In order to achieve this , we first choose a reward function based on intrinsic motivation , e.g . countbased reward or ICM . This function is able to output a novelty score for a given transition . We initialize K agents , each agent is equipped with its own version of reward function . Then we design a multi-party intrinsic reward for each agent consisting of two parts : ( 1 ) a reward to discourage the agent to visit states which has already been explored by other agents . ( 2 ) a self exploration reward to encourage the agent to visit novel states which has not been visited by itself . Agents run in a fully asynchronous manner , no agent has to wait for other other agent ’ s progress to take exploration steps . All agents communicate with each other through shared memory , under the abstraction of a simple read-write shared object , which can be made lock-free , i.e. , the learning system can always progress regardless of some computing nodes may delay or fail . These properties make our algorithm highly scalable . Our experiments show that the proposed method is highly efficient and is able to achieve state-of-the-art results in many RL tasks such as MiniGrid and Vizdoom . 2 RELATED WORK . Intrinsic Motivation The study of intrinsic motivation has been of interest for some time ( Schmidhuber , 2010 ) . Earlier works on intrinsic motivation tended to analyse with psychology and combined with reinforcement learning in tabular setting ( Deci et al. , 1981 ; Singh et al. , 2005 ) . With the breakthroughs of deep reinforcement learning , there has been a surge of interest in applying intrinsic motivation with deep neural networks for hard exploration RL problems , in which countbased exploration and curiosity-driven exploration are two main promising research branches . For count-based methods , the agent is encouraged to explore rarely visited states , while for curiositydriven methods , the agent is encouraged to explore via learning the world model . Count ( Bellemare et al. , 2016 ) extends count-based methods to non-tabular setting by deriving pseudo-counts with a density model . Ostrovski et al . ( 2017 ) adapts Bellemare et al . ( 2016 ) with PixelCNN . RND ( Burda et al. , 2018 ) uses random network distillation to measure the familiarity of states , and RGU ( Badia et al. , 2020 ) combines RND with an episodic novelty module . ICM ( Pathak et al. , 2017 ) builds a forward dynamic model and an inverse dynamic model to measure the curiosity for exploration . RIDE ( Raileanu & Rocktäschel , 2020 ) has similar networks with ICM while using a different intrinsic reward combined with episodic state visitation counts . AMIGo ( Campero et al. , 2020 ) proposes a meta-learning framework for generating adversarially motivated intrinsic goals in curriculum to guide agent exploration . RAPID ( Zha et al. , 2021 ) ranks past trajectories with episodic scores and does imitation learning for distillation . AGAC ( Flet-Berliac et al. , 2021 ) formulates an actor-critic framework with adversary . Our work is different from above by building a multi-agent learning mechanism to accelerate exploration . Distributed Reinforcement Learning The scalability of large scale distributed learning in DRL is a main challenge for reinforcement learning . Most of previous works scale up deep reinforcement learning with distributed asynchronous SGD ( Dean et al. , 2012 ) by distributing RL components . Gorila ( Nair et al. , 2015 ) scales DQN ( Mnih et al. , 2015 ) with distributed Q networks and replay buffers . A3C ( Mnih et al. , 2016 ) uses multiple workers for training and collects all gradients for parameter synchronous . Unlike A3C , IMPALA ( Espeholt et al. , 2018 ) transfers all trajectories to multiple central learners for distributed learning . Ape-X ( Horgan et al. , 2018 ) consists of a distributed replay buffers and a synchronous learner for value-based methods . SEED RL ( Espeholt et al. , 2019 ) and OpenAI Rapid ( OpenAI , 2018 ) apply central GPU based inference for acceleration . For most of the above works , the main parallelism exploited by the distributed framework is data parallelism , e.g . they use multiple nodes for collecting data , and divide minibatches among computing nodes and aggregate on the gradients computed by each node . Our model , on the other hand , can be viewed as taking advantage a new parallelism specific to RL problems — state space parallelism . Instead of dividing minibatches for the nodes , we let the agents divide the state space into pieces , which greatly improves exploration efficiency . 3 BACKGROUND . 3.1 PROBLEM FORMULATION . In this paper , we consider an RL problem formulated as a Markov Decision Process ( MDP ) , which is defined as a tuple MDP = { S , A , P , R , γ } , where S is the state space , A is the action space of the agent , P is the transition probabilities which represents the stochastic transition function S×A→ S , R ( s , a ) is the reward function after the agent executes an action , γ is the discount factor . The goal is to maximize the expected accumulated rewards R = E ( ∑T t=0 γ tr ( st , at ) ) the agent receives when executing series of optimal actions in episodes with policy π . In our method , other than the rewards rt received from environment at timestep t , our agents also receive a intrinsic reward Rt as an guidance for efficient exploration . The intrinsic reward is a parameterized function of state and action Rt ( s , a ) = fθ ( s , a ) , where the parameters θ = θ ( M ) depends on the agent ’ s history or replay buffer M . For example , in count-based reward , θ is the number of visits at each state . Instead of a single exploring agent , our proposed method trains K independent exploring agents . The exploring agents do not communicate or affect each other directly , however they indirectly affect each other through the intrinsic rewards . Each agent uses its own history or replay memory to train its own version of fθ ( the intrinsic motivation model ) . For any agent , the total intrinsic reward used for agent training is formed by combining all trained reward functions of all the agents . 3.2 CONCURRENT SHARED OBJECTS . A commonly used primitive in concurrent or distributed computing is concurrent shared objects . It is basically the user interface of concurrent data structures . A concurrent shared objects is composed of several methods which all processes can call . The behavior of a shared object is defined by its sequential execution traces . For example , a concurrent Set object has three methods , add ( v ) , find ( v ) and remove ( v ) . The behavior of this concurrent object is defined by the sequential executions of an ordinary ( non-concurrent ) Set data type . Also , a property called ” linearizability ” is applied to guarantee consistent usage of the concurrent object . Linearizability basically says that for any concurrent execution traces of the concurrent object , one can find a sequential execution trace which yields exactly the same results and the sequential execution trace preserves the real time ordering of the concurrent execution trace . In other words , all the methods of the concurrent objects can be viewed as executed atomically . In this paper , we assume all concurrent objects are linearizable . 4 DIVIDE AND EXPLORE . We design a learning mechanism named Divide-and-Explore ( D & E ) to realize the idea mentioned above . An overview of D & E is illustrated in Fig.1 . The learning framework consists of n independent agents { A1 , A2 , . . . , An } , and each agent tries to explore a particular region in the state space . We let replay memory M i denote the set of all the past experiences of the ith agent Ai in the form ( sit , a i t , r i t , s i t+1 ) . Let M = ( M 1 , · · ·Mn ) be the collection of replay memories of all agents . Note that in practice we usually don ’ t need to store all the past experiences in the replay memory , but some sufficient statistics of the experiences is enough . Our algorithm first specifies a learnable intrinsic motivation model fθ as a reward function , and θ is the model ’ s parameters . For each agent Ai , we keep track of a distinct model f i = fθi , where θi = θ ( M i ) is a function of the replay memory of agent Ai . When agent Ai takes a transition ( sit , a i t , r i t , s i t+1 ) , for every agent A j , our training architecture computes a intrinsic reward r̂i , jt based on the transition and the agent ’ s f j , as following r̂i , jt = f j ( sit , a i t , r i t , s i t+1 ) ( 1 ) Intuitively , r̂i , jt measures the transition ’ s novelty for agent A j . If r̂i , jt is large , it means that the state si+1 is rarely visited and not explored fully by Agent Aj . The intrinsic motivation model fθ can be selected with any kind of count-based , curiosity-driven and memory-based exploration methods ( e.g . Pathak et al . ( 2017 ) ; Burda et al . ( 2018 ) ) according to the specified task .

Summary: This paper proposes a distributed algorithm for exploration in reinforcement learning, based off of the principle of "divide and conquer".  This is implemented as a multi-agent system where each agent is run on it's own node in parallel and given a custom intrinsic reward to motivate exploration and diversity.  These intrinsic rewards are built out of standard intrinsic reward methods for exploration, but are modified so each agent not only gets reward from it's own intrinsic motivation function but also gets reward by how their actions score with respect to the other agent's intrinsic motivation functions.  This motivates each agent to specialize to find the strategies the other agents' intrinsic motivation functions would see as novel.