Divide and Explore: Multi-Agent Separate Exploration with Shared Intrinsic Motivations
Authors: Xiao Jing
Zhenwei Zhu
Hongliang Li
Xin Pei
Yoshua Bengio
Tong Che
Hongyong Song
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep Reinforcement Learning
Exploration
Intrinsic Motivation
Distributed Learning

Abstract: One of the greatest challenges of reinforcement learning is efficient exploration, especially when training signals are sparse or deceptive. The main difficulty of exploration lies in the size and complexity of the state space, which makes simple approaches such as exhaustive search infeasible. Our work is based on two important observations. On one hand, modern computing platforms are extremely scalable in terms of number of computing nodes and cores, which can complete asynchronous and well load-balanced computational tasks very fast. On the other hand,  Divide-and-Conquer is a commonly used technique in computer science to solve similar problems (such as SAT) of doing efficient search in extremely large state space. In this paper, we apply the idea of divide-and-conquer in the context of intelligent exploration. The resulting exploration scheme can be combined with various specific intrinsic rewards designed for the given task. In our exploration scheme, the learning algorithm can automatically divide the state space into regions, and each agent is assigned to explore one of these regions. All the agents run asynchronously and they can be deployed onto modern distributed computing platforms. Our experiments show that the proposed method is highly efficient and is able to achieve state-of-the-art results in many RL tasks such as MiniGrid and Vizdoom.  

Under review as a conference paper at ICLR 2022
DIVIDE AND EXPLORE: MULTI-AGENT SEPARATE Ex- PLORATION WITH SHARED INTRINSIC MOTIVATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
One of the greatest challenges of reinforcement learning is efficient exploration. especially when training signals are sparse or deceptive. The main difficulty of exploration lies in the size and complexity of the state space, which makes simple approaches such as exhaustive search infeasible. Our work is based on two im- portant observations. On one hand, modern computing platforms are extremely scalable in terms of number of computing nodes and cores, which can complete asynchronous and well load-bat bal--allaeeddd computational tasks very fast. On the other hand, Divide-and and-Conquer a commonly used technique in computer science to solve similar problems (such SAT) of doing efficient search in extremely large state space. In this paper, we apply the idea of divide-and-conquer in the context of intelligent exploration. The resulting exploration scheme can be combined with various specific intrinsic rewards designed for the given task. In our exploration scheme, the learning algorithm can automatically divide the state space into re- gions, and each agent is assigned to explore one of these regions. All the agents run asynchronously and they can be deployed onto modern distributed computing platforms. Our experiments show that the proposed method is highly efficient and is able to achieve state-of-the- results in many RL tasks such as MiniGrid and Vizdoom.
INTRODUCTION
Deep reinforcement learning is a very important learning framework to train intelligent agents to solve complex tasks (Sutton Barto 2018). Despite recent success stories in variety of real tasks (Silver et al. 2016) Vinyals et al. 2019 Lee et al. 2020), the difficulty of exploration remains a ma- jor challenge for DRL when the reward is sparse or deceptive (Burda et al. 2018). Recent research in exploration has made significant progresses thanks to the idea of intrinsic motivation (Schmid- huber 2010). Intrinsic motivation could be viewed as objectives for unsupervised reinforcement learning (Levine 2020), and their goal is usually not to guide the agent to complete specific task, but to help the agent to discover interesting states and trajectories which could help the agent to un- derstand the environment and find possible target solutions for specific tasks. Many previous works on intrinsic motivation (Raileanu & Rocktäschel, 2020; Burda et al., 20188) 2019) can be understood as proposing some kind of metrics to measure the novelty of the states, and then using the metric as an intrinsic reward to encourage agents to reach novel states, such as count-based exploration (Bellemare et al. 2016) and ICM (Pathak et al. 2017).
Lee
Despite of the sigmificant progresses that have been made, in theory, efficient exploration is still very challenging impossible, due the infeasible size of the state space to be explored. Similar difficulties are usually seen in the state-space search problems in theoretical computer science. For example, SMT or SAT problems (De Moura & Björner 2008), one required search for feasible solution to satisfy certain constraints, and these problems usually NP complete because of the exponential size of the state space.
A very common class of practical algorithms to solve these search problems is divide and conquer. The algorithm first tries to divide the state space into certain pieces and then search for results in each piece of state space. A significant benefit of this approach is it can be easily modified to take advantage of modern distributed/cooccrreen computing hardware, because it can naturally divide hard problem into smaller and simpler ones, allowing different computing nodes to complete

---

Under review as a conference paper at ICLR 2022
the tasks asynchronously. Aivide-and-conuue type of algorithms are among the state-of-the art algorithms to solve SMT or SAT problems.
In this paper, take inspiration from divide and conquer algorithms in theoretical computer science and try to adapt them to the exploration problems in RL. Different from traditional exploration mechanisms with single exploring agent, we train number of concurrent exploring agents. We carefully design our intrinsic reward such that each exploring agent is encouraged to explore only one region in the state space. In other words, the state space is automatically divided into several components, where each agent is responsible for exploring one of these components.
In order to achieve this, we first choose reward function based on intrinsic motivation, e-g. count- based reward or ICM. This function is able to output a novelty score for a given transition. We initialize K agents, each agent equipped with its own version of reward function. Then we design multi-party intrinsic reward for each agent consisting of two parts: (1) a reward to discourage the agent to visit states which has already been explored by other agents. (2) a self exploration reward to encourage the agent to visit novel states which has not been visited by itself. Agents run in a fully asynchronous manner, no agent has to wait for other other agent's progress to take exploration steps. All agents communicate with each other through shared memory, under the abstraction of a simple read- wead-write shared object, which can be made lock-free, i.c., the learning system can always progress regardless of some computing nodes may delay or fail. These properties make our algorithm highly scalable. Our experiments show that the proposed method is highly efficient and is able to achieve state-of-the-ar results in many RL tasks such as MiniGrid and Vizdoom.
2 RELATED WORK
Intrinsic Motivation The study of intrinsic motivation has been of interest for some time (Schmidhuber 2010). Earlier works on intrinsic motivation tended to analyse with psychology and combined with reinforcement learning in tabular setting (Deci et al. [981] [Singh et al.) (005). With the breakthroughs of deep reinforcement learning, there has been a surge of interest in applying intrinsic motivation with deep neural networks for hard exploration RL problems, in which count- based exploration and curiosity-driven exploration are two main promising research branches. For count-based methods, the agent is encouraged to explore rarely visited states, while for curiosity- driven methods, the agent is encouraged to explore via learning the world model. Count (Bellemare) al., 2016) extends count-based methods to non- non-tabular setting by deriving pseudo- counts with a density model. Ostrovski et al. (2017) adapts Bellemare et al. (2016) with PixeICNN. RND (Burda cet al. 2018) uses random network distillation to measure the familiarity of states, and RGU (Badia et al. 2020) combines RND with an episodic novelty module. ICM(Pa et al. (017) builds for- ward dynamic model and an inverse dynamic model to measure the curiosity for exploration. RIDE [Raileanu & Rocktäschel 2020) has similar networks with ICM while using a different intrinsic reward combined with episodic state visitation counts. AMIGo (Campero et al.] 2020) proposes meta-learning framework for generating adversarially motivated intrinsic goals in currieulum to guide agent exploration. RAPID (Zha et al. (20T)) ranks past trajectories with episodic scores and does imitation learning for distillation. AGAC (Flet-Berliac al., (202T) formulates an actor-critic framework with adversary. Our work is different from above by building a multi-agent learning mechanism to accelerate exploration.
Distributed Reinforcement Learning The scalability of large scale distributed learning in DRL is a main challenge for reinforcement learning. Most of previous works scale up deep reinforcement learning with distributed asynchronous SGD (Dean et al. 2012) by distributing RL components. Gorila (Nair et al.) 2015) scales DQN (Mnih et al., 2015) with distributed Q networks and replay buffers. A3C (Miih et al.) 2016) uses multpple workers for training and collects all gradients for parameter synchronous. Unlike A3C, IMPALA (Espeholt et al. 2018) transfers all trajectories to multiple central learners for distributed learning. Ape-X (Horgan et al., 2018) consists of dis- tributed replay buffers and a synchronous learner for value-based methods. SEED RL (Espeholl) et al. (219) and OpenAl Rapid (OpenAl 2018) apply central GPU based inference for acceleration. For most of the above works, the main parallelism exploited by the distributed framework is data parallelism, e.g. they use multiple nodes for collecting data, and divide minibatches among comput- ing nodes and aggregate on the gradients computed by each node. Our model, on the other hand, can be viewed taking advantage new parallelism specific to RL. problems state space parallelism.
2

---

Under review as a conference paper at ICLR 2022
Instead of dividing minibatches for the nodes, let the agents divide the state space into pieces, which greatly improves exploration efficiency.
BACKGROUND
3.1 PROBLEM FORMULATION
In this paper, we consider an RL problem formulated as a Markov Decision Process(MDP), which is defined as a tuple MDP- = {S, A, P. R, 7}. where S is the state space, A is the action space of the agent, P is the transition probabilities which represents the stochastic transition function S A + S, R(s, a) is the reward function after the agent executes an action, Y is the discount factor. The goal is to maximize the expected accumulated rewards R = E(Σ/=0 (rr, ac)) the agent receives when executing series of optimal actions in episodes with policy π.
In method, other than the rewards r, received from environment at timestep our agents also receive a intrinsic reward Rg as an guidance for efficient exploration. The intrinsic reward is a parameterized function of state and action (s, a) fo(s, af.s,a the parameters ((M) depends on the agent's history or replay buffer M. For example, in count-based reward, 0 is the number of visits each state.
Instead of a single exploring agent, our proposed method trains K independent exploring agents. The exploring agents do not communicate affect each other directly, however they indirectly affect each other through the intrinsic rewards. Each agent uses its own history or replay memory to train its own version of fo intrinsic motivation model). For any agent, the total intrinsic reward used for agent training is formed by combining all trained reward functions of all the agents.
3.2 CONCURRENT SHARED OBJECTS
commonly used primitive in concurrent or distributed computing concurrent shared objects. is basically the user interface of concurrent data structures. A concurrent shared objects is composed of several methods which all processes can call. The behavior of shared object is defined by its sequential execution traces. For example, concurrent Set object has three methods, add(v). find(v) and remove(v). The behavior of this concurrent object is defined by the sequential executions of an ordinary (non-concurrent) Set data type. Also, property called "linearizability" is applied to guarantee consistent usage of the concurrent object. Lincarizability basically says that for any concurrent execution traces of the concurrent object, can find sequential execution trace which yields exactly the same results and the sequential execution trace preserves the real time ordering of the concurrent execution trace. In other words, all the methods of the concurrent objects can be viewed as executed atomically. In this paper, we assume all concurrent objects are linearizable.
DIVIDE AND EXPLORE
We design a learning mechanism named Exxpl-- (D&E) to realize the idea mentioned above. An overview of D&EE is illustrated in [ig[] The learning framework consists of indepen- dent agents {A), A2,..., A"), and each agent tries to explore a particular region in the state space. We let replay memory M denote the set of all the past experiences of the ith agent A' in the form (st, af, ri, 1. Let M = (M',.. M be the collection replay memories all agents. Note that in practice we usually don't need store all the past experiences in the replay memory, but some sufficient statistics of the experiences is enough.
Our algorithm first specifies learnalle intrinsic motivation model fo as reward function, and is the model's 's parameters. For each agent A', we keep track of distinct model f = fos, where function of the replay memory of agent A'. When agent A takes transition (sf, at, rř, ). for every agent A1, our training architecture computes intrinsic reward p/ based on the transition and the agent's fJ, as following
0¹ ((M*)
P.. ( (s) at
3

---

Under review as a conference paper at ICLR 2022
RL update
Intrinsic Motivation
Update
Agent
Rollout
State
RL update
Intrinsic Motivation
Update
Agent 1
Rollout
R
RL update
Intrinsic
Update
Agent
Rollout
Motivation n
An Overview Divide and Explore. Agents are exploring different regions of state space automat- ically, and intrinsic motivations are shared across all the agents for updating.
Figure I:
Intuitively, F/ measures the transition's novelty for agent AP, If ff is large, means that the state 8c+1 is rarely visited and not explored fully by Agent A'. The intrinsic motivation model fo can be selected with any kind of count-based, curiosity-driver and memory-based exploration methods (e.g. Pathak et al. 2017): Burda et al. (2018)) according to the specified task.
4.1
DIVIDE AND EXPLORE REWARD
Here we define the D&E intrinsic reward for agent learning as below:
R wo Σ piu woff
(2)
where (n 1) W. 1.
According to the definition, the D&E reward aggregates other agents' intrinsic rewards and the agent's own intrinsic reward, which encourages the agent to explore the state space where all agents have rarely explored. This mechanism enables full use of all agents' experience and gets rid of wasting time fully explored state space as improve exploration efficiency. The weights wo and w, determine the exploration preference between agent separability and self exploration. If w. zero, the algorithm turns to be conventional intrinsic-driven exploration, while if we is zero, the algorithm only considers separability and the agent may converge to local state space where other agents haven't 't visited and stop exploring further. In practice, we set W. Wa to encourage agents to explore separately while keeping self exploration.
With the D& intrinsic reward, the total reward defined as follows:
F +
where rf is the task reward provided by the environment, and a is the weight of D& intrinsic reward. Base on the definition above, we are able turn sparse reward task into dense one, and apply conventional model free reinforcement learning algorithms such as PPO (Schulman et al. 2017) or IMPALA (Espeholt al. 2018) to train the agent policy.
4.2 AUTOMATIC REWARD WEIGHT DECAY FOR EXPLOITATION
While the main goal of Divide-and- -ndpll-orr- is encourage efficient exploration, also consider the trade-off between exploration and exploitation. In conventional model free reinforcement learn- ing, the trade-off is usually controlled in the agent roll out process via e-greedy (Mnih et al. 2015)

---

Under review as a conference paper at ICLR 2022
or the action probability distribution (Mnih al. 2016 Because in D& intrinsic reward to encourage agent to explore, we reduce the intrinsic reward weight to make agent pay more attention to the trajectories with high extrinsic reward, especially when the agent has trained a lot and has probability of obtaining some rewards. The decay of intrinsic reward weight can be formulated as de og, and de is decay factor. Compared to linear decay, we design adaptive decay method as follows:
1
af d,00 max
N
R2), .00
in which og > 0 is the initial value of ai, d; is the decay factor of agent A' at time t, N is the number of A historical episodes and R2 is the return of episodes. We consider more about newly is a hyperparameter, which determines the decay
generated episodes by introducing Φ E (0.1) rate, and in general, we let U = Σラー RR- According to Eq of has negative correlation with the sum of historical rewards and is adjusted automatically during training, which matches the goal of tra
4.3 FULLY ASYNCHRONOUS UPDATES WITH CONCURRENT DATA STRUCTURES
In order to speed up computing and scale approach large number of exploring agents, non-blocking concurrent computing primitives in our implementation. In our algorithm, the reward functions f fø
2-... n shared across all agents. Only agent A' is allowed to modify 0i, but all agents can read 0², We can store 0 = (0), 0², ...00) in shared concurrent object O.
In the terminology of concurrent data structures, we define O as concurrent object that allows two operations, read(i) which returns copy of 6' and update(i, v) which overwrite current Øt into value A lot of concurrent data structures could be used implement this concurrent object, such as read-write locks (locking), transactional memory or software transactional memory (lock-free). In our work, we use software transactional memory since non-blocking, making our algorithm fully asynchronous.
Our approach summarized in Algorithm 1. All the agents are trained asynchronously in a dis- tributed training framework, and the shared object O is read every L steps. Besides, there is special attribute of D&E that all agents will be convergence as long as only one agent accomplishes the task. With the shared intrinsic motivation and learning mechanism, the failure experience of other agents transferred to the succeeding agent as bad cases to avoid repetition of wrong exploration, which demonstrates the superiority of this algorithm.
EXPERIMENTS
In this section, evaluate D&EE on two challenging reinforcement learning research platforms MiniGrid (Chevalier- Boisverrt et al. 2018 and VizDoom (Kempka et al., 2016), and both of them have hard exploration environments with sparse rewards, which commonly used as benchmarks. For all our experiments. we compare our approach against various state-of-the-arl baselines for hard exploration including Count (Bellemare et al. 2016). ICM (Pathak al. (017). RND (Burda al.,
2018).
AMIGo (Campero et al. 2000). RIDE RRaileanu & Rocktäschel 2020). RAPID (Zha et al.
2021
and AGAC (Flet-Berliac al. 202I
5.1
MINIGRID EXPERIMENTS
The MiniGrid environments are set of sereduralll-ggneeran 2D grid worlds with sparse reward. Following AGAC -Flei-Berliac et al. 2021)'s setting, we choose three types of tasks for experiments: MultiRoom(MR) KeyCorrider (KC) and ObstructeddMaze((MMM) In the MultiRoom tasks, there are multiple rooms connected one- by- with doors. The agent has to go through these rooms to reach goal. In the KeyCorrider tasks, several rooms nested together and the mission of the agent is to find a key so as to open a locked room and pick up an object. For the last, the ObstructedMaze tasks are updated versions of KeyCorrider and most challenging where
Environment
5

---

Under review as a conference paper at ICLR 2022
Algorithm Divide and Explore
Input: Time horizon T, decay frequency K, reward model update frequency reward weight
w and oko
2: Initialize n agent policies { n² ...., ) and the shared object contains reward model param-
eters O, agent replay memory {M' M .... M"). }. For each agent policy, initialize local copy of reward models 0' 00" .......""
3: for iteration=1,2... until convergence do for all agents 1 2.2... ... do asynchronously
4:
5:
Roll out 7" timesteps with n² in the environment for every L steps do
6:
7:
Update local reward model parameters @ O.raad(j)) end for
8:
9:
Compute total rewards using Eq based on locally stored 0 6 ,.. ...00" 0".
10:
Update x¹ with RL algorithm like PPO
11:
Update agent replay memory M
12:
Compute new model parameters based on the replay memory 0 = 0 (M')
13:
Update shared object by calling shared method Ouupdate (")
end for
15:
for every K steps do
Update reward weight a with Eq 17: end for
18:
end for
19: Return the optimal policy '
the key is hidden in boxes and the door is obstructed by balls. For all tasks, we make use of partial observations, which means the observation space 7x7x3 tensor indicating the information of partially-observabbe grid cells, and there are actions for executing.
D&E with Count-based Exploration Due the simplicity of 2D grid environment, we count the states in the form of tabular as exploration bonus in the MiniGrid tasks, which has been used in previous works, e.g. Count (Bellemare al. (2166). RIDE (Raileanu & Rocktäschel] (020) and AGAC (Flet-Berliac et al.) 2021). Therefore, We combine count-based exploration with D&EE for experiments. In detail, we define the intrinsic reward by:
P/ 1//
Σ I(s 1(s= 1(+1) +1 SEMS
(5)
where I(.) is indicator function. We also introduce an inner-episodic count mask multiplier by replacing original reward weight of with:
af + 1( 1 1(s s;) 0)a²
(6)
SEC:
where Cs is the trajectory in current episode. With this mask, if state has been visited in the trajectory, the intrinsic reward will be masked, which improve the exploration efficiency.
Training We use PPO (Schulman et al. (2017)) standard model free RL algorithm to train all the agents. For fair comparison, the architecture of actor and critic network we use is the same AGAC's network in (FFet-Berlian fully connected layer for feature processing. More concretely, each convolutional block contains a 32-filter 3x3 convolution with stride size 2, and the hidden size of fully connected layer is 512. The number of agents in D&E is set to 3 as default for all tasks. More hyperparameters can be seen in Appendix A.
al. (2021), which consists of 3 convolutional blocks and
Results We train D&E with various MiniGrid tasks and compare the results with baselines in both exploration efficiency and performance. For exploration efficiency, illustrated the learning

---

Under review as a conference paper at ICLR 2022
MGNC
AMF
Count
D6E
RAPID
RID
AND
M
Min Crdd-
1.0
1.0
1.0
Nvrraee Rehum
0.8
Nverage Retum
0.8
Nverage Retum
0.8
0.5
0.8
08
04
014
04
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.3
1.5
2.0
2
Number of Timestaps
167
Number of Timestepe
167
Number of Timestaps
107
1.0
1.0
1.0
Awerage Rorum
Rorum
0.5
Aveeage Raeum
116
Awergge Rotum
0.5
01
0.4
0.4
04
0.2
0.2
022
0.3
0.
06
-0.2
-02
-02
0.0
0.2
014
06
0.8
1.0
0A1
0.2
044
24
1.0
0.2
022
044
08
0.5
1.
Number of Timestops
Number of Timesteps
1el
Number of Timestaps
161
MinkSrd Orennuteciizzee 10.10
MinkSind Obetructedittze
MinGind OestructedWazz 20
Average Rotur
Average Ratur
Average Rotum
0.4
0.4
0.4
02
0.2
022
0.0
00
-02
-0.2
-02
Timestapo
Numbbr Timostaps
Timestapo
Figure 2: Performance of D&DE and baselines in MiniGrid. We plot the best agent's performance in D&E.
Task
KC-S4R3
KC-S5R3
MR-N10S10
MR-N12S10
OM-2DIhb
OM-1Q
OM-2Q
D&EE
0.91
0.90
0.60
0.60
0.16
0.39
0.09
AGAC
0.93
0.90
0.52
0.51
0.04
0.09
0.20
RAPID
0.90
0.00
0.54
0.55
0.00
0.00
0.00
RIDE
0.00
0.00
0.00
0.00
0.00
0.00
0.00
AMIGo
0.00
0.00
0.00
0.00
0.00
0.00
0.00
RND
0.00
0.00
0.00
0.00
0.03
0.00
0.00
ICM
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Count
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Table 1: Final performance of D&EE and basclines on MiniGrid environments. All the methods are trained in 50M total timesteps.
curves of D& and basclines in less than 50M timesteps in Fig Because of the training con- currency in all D&E's agents, we record the best agent's performance for D&E with the agent's timesteps by default. In all the tasks, D&E achieves state-of-the-arr results and the performance boosts with much less timesteps compared to other methods especially AGAC and RAPID. Only in the ObstructedMaaze-2 task, the performance of AGAC is comparable with D&
For performance only, we report the final performance of all methods in Table T All baselines are trained in 50M timesteps. Due to the multi-agent learning mechanism, the total number of timesteps used D&F directly proportional to the number of agents. Therefore, we only record the performance of D& 16.7M timesteps for fair comparison. As we can see in the table, D& performs best in most scenarios while several approaches such as RND, ICM and Count totally failed in all the listed tasks.
For better understanding the exploration efficiency, we also plot the state visitation heatmaps of the challenging MolliRoom-N12SSiA task with 1M timesteps in Fig The policies are trained in singleton environment. While D&E finds a good trajectory to the last room with few wrong actions, AGAC spends more time in the journey, and RAPID is stuck in the sixth room.
7

---

Under review as a conference paper at ICLR 2022
DBE
RAPID
AGAC
AMiGo
RIDE
RND
ICM
Count
Figure 3: State visitation heatmaps in the MultiRoom-N12-SH4 task. All the comparison methods only train 1M timesteps.
Timesteps
1M
2M
3M
4M
5M
6M
D&E
0.23 0.5
0.96.1 0.1
0.97.L 0.02
0.97.L
0.01
0.97.L 0.001
0.97. 0.001
AGAC
0.12.L 0.5
0.84.1 0.1
0.95. 0.1
0.97.1 0.01
0.97.L 0.001
0.97. 0.001
RIDE
0.
0.
S 0.
0.
0.
0.95 0.001
ICM
0.
S
0.95 0.001
Rapid
< 0.
0.
S. 0.
0.
S 0.
0.
AMIGo
S 0.
S 0.
S 0.
S 0.
S 0.
0.
RND
0.
0.
S 0.
0.
0.
0.
Count
0.
S 0.
0.
0.
W 0.
Table 2: Performance of D&E and baselines on the VizDoom environment.
5.2 VIZDOOM EXPERIMENTS
Environment The VizDoom environments are a series of Doom-based 3D scenarios for rein- forcement learning. We choose the My MyWayHome task following AGAC (Flet-Berliac et al. 2021)'s setting. The MyWayHome task consists of multiple rooms and corridors, which is a 3D naviga- tion task that the agent explores rooms find the object. Different from MiniGrid, the state is an ego-centric visual observation with a size of 42x42x4, and the number of action is 5.
D&E with Furiosity-Driven Exploration In order to demonstrate the generality of D&&E and compare with baselines, we get rid of using count-based reward and combine D&E with the curiosity-driven exploration method ICM (Pathak al. 2017). Every agent has its independent ICM which calculates intrinsic reward based on transition (st, az 81+1). For each agent, the ICM is trained together with its policy and shared across all the agents. We set the learnable reward model f' mentioned in Eq as the ICM model, following ICM's definition with normalization.
Training and Results Following MiniGrid experiments, train D& in VizDoom with minor changes. We use the same network architecture of actor and critic with different state input in VizDoom, but replace count-based exploration with ICM using the original implementation. The hyperparameters are reported in Appendix A. We list the comparison results in Table 2 and D&E outperforms other baselines in any selected timesteps. Interestingly, while ICM is slow solve the task, D& learns fast and achieves high performance. Even when we compare D& at 2M timesteps (6M timesteps in total agents) with ICM 6M timesteps, the performance D&EE is still better.
5.3 ABLATION STUDY
In the ablation study, we aim to answer following questions: 1) Is D&E capable to guide agents to explore different regions of state space separately? 2) Compared to single-agent setting, is D&E's
8

---

Under review as a conference paper at ICLR 2022
learning mechanism key reason to improve exploration efficiency? 3) How the number of agents influence the performance?
For question 1, train D&E the KeyCoriid:S6R33 task with singleton environment, and the number agents is 3. The exploration process different agents shown in Fig in which state visitation count in historical trajectories is accumulated and visualized in the form of heatmap. As we can see in the figure, is clear that three agents are exploring different regions of the environment separately, which validates the learning mechanism of D&E.
0
agent
5-
10
15
100
200
300
400
500
Figure 4: Exploration processes of three different agents in singleton environment.
64
68
TLACOM
Figure 5: The performance of D& with different number of agents.
For question and 3, we train D&E on the KeyCoriidoS6R3 task with various number of agents in singleton environment. The left figure in Fig shows the best agent's learning curve under dif- ferent agent number setting, which confirms that the number of agent brings benefits to the training efficiency of the algorithm. In fact, if the number of agents one, the method is a conventional count-based method. On the other hand, as shown in the right figure in Fig we also plot the num- ber of training timesteps required when the return episode reaches 0.8 and 0.9 for the first time. While the more agents, the less time algorithm needs to be trained, it shows diminishing marginal effect. Thus, the number of agents should be chosen according to the task.
6 CONCLUSIONS
We proposed brand new algorithm named Beside-and-ExploreODDe ration problem in reinforcement learning. Inspired by concurrent exploring agents, and successfully guides each agent exploring different regions of state space with shared intrinsic motivations while keeping exploring the boundary, which produces state- of-the-art results. More generally, D&E can be seen as a framework, which can combine existing single-agent exploration methods, e.g. Count, ICM, RIDE and
handle the hard explo- D& trains multiple
improve learning efficiency. In the future work, we' like to extend this idea to large scale setting and test it on general RL bench- marks like atari games (Mnih et al. 2015) and MuJoCo tasks (ooorov al. 2012). And will also explore the application of D&E on 3D navigation in robotics (Zhelo et al.
2018 or playtesting
coverage in games (Gordillo et al. 2021

---

Under review as a conference paper at ICLR 2022
REFERENCES
Adria Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.066 06038, 2020.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances neural information pro- cessing systems, 29:1471-1479, 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: /810. 12894, 2018.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros Large-scale study of curiosity-driven learning. In International Conference on Learning Repre- sentations, 2019.
Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. arXiv preprint trXiv:2006.12122 2020.
Maxime Chevalier-Boisvert Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https: httb:::///githtee com/ www./owoo..cooc..oooc.eeo.oao..e www.maaa.mmaaaeeemmmm mww.mmgccbmmeeeeeeeeee minigrid. 2018.
Leonardo De Moura and Nikolaj Bjørner. Z3: An efficient smt solver. In International conference on Tools and Algorithms for the Construction and Analysis of Systems, pp. 337-340. Springer, 2008.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc' aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks. Advances in neural information processing systems, 25:1223 1231, 2012.
Edward I. Deci, John Nezlek, and Louise Sheinman. Characteristics of the rewarder and intrinsic motivation of the rewardee. Journal of personality and social psychology, 40(1):1, 1981.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dumning, et Impalat Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International Conference Machine Learning, pp. 1407-1416. PMIR, 2018.
Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. Seed rl: Scal- able and efficient deep-rl with accelerated central inference. In International Conference Learning Representations, 2019.
Yannis Flet-Berliac, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist. Adversari-
ally guided actor-critic. arXiv preprint wrXi::2002.04366, 2021.
Camilo Gordillo, Joakim Bergdahl, Konrad Tollmar, and Linus Gisslén. Improving playtesting coverage via curiosity driven reinforcement learning agents. arXiv preprint 1xXii:2103.137990 2021.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. In International Conference Learn- ing Representations, 2018.
Michat Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaskowski. Viz- doom: A doom-based research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEEE, 2016.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47), 2020.
Sergey Levine. Unsupervised reinforcement learning. In Proceedings of the 19th International Conference Autonomous Agents and MultiAgent Systems, pp. 5-6, 2020.
10

---

Under review as a conference paper at ICLR 2022
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning, nature, 518(7540):529- 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928 1937. PMLR, 2016.
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, et al. Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv: 507.04296, 2015.
OpenAl. Openai five.
2018.
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721-2730, PMLR, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-278 PMLR, 2017.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for
procedurally- environments. In International Conference on Learning Representations,
2020. URL URL http:htttt///opeeeeeeeee net.://weeeeeeeeeeeeeeeeeeeeeeee feret//oormmmidrrree d=reiiidrrkk---o wwwet//oormmmmmarrggg-g TJRFFF
TJRFPR
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE
Transactions on Autonomous Mental Development, 2(3):230- 247, 2010.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-488, 2016.
Satinder Singh, Andrew G Barto, and Nuttapong Chentanez. Intrinsically motivated reinforcement learning. Technical report, MASSACHUSETTS UNIV AMHERST DEPT OF COMPUTER SCI- ENCE, 2005.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Oriol Vinyals, Igor Babuschkin, Wojeiech Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)350-334, 2019.
Daochen Zha, Wenye Ma, Lei Yuan, Xia Hu, and Ji Liu. Rank the episodes: A simple approach for environments. arXiv preprint a:X/v:2/01.08//2, 2021.
exploration in procedurally-geneeeeee
Oleksii Zhelo, Jingwei Zhang, Lei Tai, Ming Liu, and Wolfram Burgard. Curiosity- explo- ration for mapless navigation with deep reinforcement learning. arXiv preprint 1804 2018.
11

---

Under review as a conference paper at ICLR 2022
A HYPERPARAMETERS
The hyperparameters of D&&E used in both MiniGrid and VizDoom environments are listed in Table For Count, RND, ICM, RIDE, we conduct experiments based on the authors' implementation while the policy networks are modified to be the same as D&E. And for other algorithms, we use the authors' implementations with default hyperparameters, e.g. AMIGO RAPII and and AGAC
Parameter
Value
Time horizon T Number of epochs
2048
4
Number of minibatches
8
Learning rate Final learning rate
3e-4
3e-5
Learning rate anneal schedule Learning rate anneal steps
linear
1c4 (3e4 in VizDoom)
Discount y
0.99
GAE parameter
0.95
Gradient clip threshold
1.0 (5.0 VizDoom)
Number of agents
3
Decay frequency
Reward model update frequency L
2048
Reward weight W,
0.465
Reward weight
0.07
Reward weight
0.1
Threshold U
42 (30 VizDoom)
Decay rate &
0.98
Table 3: Hyperparameters used in D&:E
B ADDITIONAL EXPERIMENTS
B. HARDER EXPLORATION TASK
There are many harder exploration tasks in MiniGrid that have not been well resolved in previous studies, for example, Keyiirrd-KeCooridosSS perspectives. Firstly, because the agent can only observe small range of 7 x 7, large map will con- fuse corridors and rooms, making it difficult for the agent to know about its own location through ob- servation. Secondly, the larger environment makes agent's 's trajectory longer and the rewards become more sparse, and this reinforces agent's dependence on intrinsie rewards during training. Thirdly, the larger environment makes the connection between rooms more complex, and agent often needs to go through several rooms to find the key and the object. However, D&E achieves amazing per- formance as shown in Fig which shows that D&E does enhance ability to solve problems rather than simply improve efficiency.
The difficulty of this task comes from three
B.2 EXPERIMENT DETAILS
To the results mentioned in Fig we plot the best performance in several independent runs of baseline algorithms, illustrate the benefits are mainly from the particular intrinsic motivation. As shown in Fig 7 compared with independent runs, the interact between different agents and intrinsic reward calculated from all agents' replay buffer indeed accelcrate the exploration efficiency.
12

---

Under review as a conference paper at ICLR 2022
MinkSnid KeyComdort
0A
Aneraca Rat.m
02
Ind
Figure 6: The performance D& with different decay method.
AMiic
RAPID
KeyConncor54R33v00
M
1.00
1.00
1100
Pverage Relum
Retum
0.50
0.00
0.60
Avtrage
0.00
0.00
0.00
0.0
0.5
1.0
200
0.0
0.5
1.5
2.0
Number of Timeslaps
N 1.rbba ber Timestaps
te?
Number Timestene Mutifoom
MultRoom
MutRoom
1.00
1.00
1.00
Averag Retum
Ritum
0.50
Averaga
0.00
0.80
3.00
0.00
0.00
0.0
0.2
0.4
08
0.5
1.0
0.0
0.2
0.4
0.
0.5
1.0
3.0
022
0.6
088
1.0
Number of Timedlaps
167
Nur ber Timestaps
le7
Number Timestege
d-O.ssuutbaaaas--oooo CwvD
1.00
1.00
1.00
Avwrage Relum
0.50
0.00
0.50
Avarago
3.25
3.00
000
0.00
0
2
3
3
Number of Timesteps
167
Namber of Timesteps
107
Number Timesteps
Figure 7: Performance of D&E and baselines in MiniGrid. We plot the best agent's performance in D&E and best performance in several independent runs of baseline algorithms.
B.3 ABLATION EXPERIMENTS FOR DECAY METHOD
As supplement, we also compare the effects of different intrinsic reward weight decay methods. We compare the conventional linear decay and adaptive decay in the MiniGrid- KonGGnddkKyCCorrddoSoR-3-- task. The intrinsic reward weight in the former calculated as:
af max
U 0} au
while in the latter it is calculated as Eq The results are shown in the left figure of Fig which validates the benefits of the adaptive decay method. Considering the historical episodes' rewards, agent make trade-off between exploration and exploitation better, and reduce exploration when has found good trajectories and obtained high extrinsic rewards, which is similar with human decision. We plot the curve of decay factor d,, and there three phases in the training process.
13

---

Under review as a conference paper at ICLR 2022
At the beginning, agent rarely find trajectory to complete the task and the decay factor de is large which means that agent should pay more attention to intrinsic reward and explore as much possible. When train timesteps is larger than 2M, agent has increasingly high probability of completing tasks, and de decreases fast. After 10M training, agent's policy converges and does not need explore.
Maxtinal 40
1.0
05000 4boo of ssnract Doward Nooghh
0.0
014
02
Broar Jrn0
0.2
0.7
1.20
1.70
1,5
Tireeeapee
Timeerppe
Figure The performance of D&E with different decay methods.
14


Reviews:
[{"review_id": "xGt5hY-4nAG", "reviewer": "Reviewer_BfC2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The idea of parallelizing agents for exploration is interesting.\n* The approach can solve many tasks on MiniGrid, which are challenging due to their procedural nature and sparse rewards.\n* The approach can solve the VizDoom MyWayHome task, which many baselines (Rapid, AMIGo, RND, Count) cannot.\n\nWeaknesses:\n* The method is very similar to prior work. See “Efficient Exploration via State Marginal Matching” (Lee et al. 2019) section 2.3, “Better SMM with Mixtures of Policies.” D&E is like SMM Equation 3 terms (b) and (d). D&E can also be seen as “Diversity Is All You Need” (Eysenbach et al. 2018), where each D&E agent is a DIAYN skill. Additionally, if we set $w_o=w_s=1/n$ in Equation 2 (weighting current agent and other agents equally), D&E is equivalent to a general intrinsic reward. I’d expect at least an ablation with different weights of $w_o$ and $w_s$.\n* The results are not apples-to-apples comparisons. Figures 2 and 5 and Tables 1 and 2 should show the total number of steps across all agents for each method. Since D&E uses 3 agents, its performance is effectively reported as being 3 times more sample efficient than it actually is.\n* Using 3 agents in parallel is not very scaled. For example, IMPALA is frequently run with 40+ actors. If the motivation for D&E is parallelism, then I’d expect more agents to be used.\n\nOther thoughts and suggestions:\n* Section 4.2: Versions of learning reward weights (as described in Equation 4) have been done before. See “On Multi-objective Policy Optimization as a Tool for Reinforcement Learning” (Abdolmaleki et al. 2021) for example. I also don’t understand the described reasoning of matching the “goal of tradeoff” – what does this concretely refer to?\n* The count mask defined in Equation 6 should be ablated and/or used in baselines. \n* It’s unclear whether the D&E heatmap in Figure 3 shows just the best of the 3 agents or the average. I’m guessing it’s just the best agent since the normalization looks different (the D&E heatmap is lighter than the others). If this is the case, it would be nice to include heatmaps for the other agents as well.\n* Section 5.2: It’s odd to use a count-based reward on MiniGrid and switch ICM on VizDoom. Pick one to use consistently across these settings or show both formulations everywhere.\n* Section 5.3: The experimental questions aren’t that interesting. As a rule of thumb, yes/no questions aren’t great for these.\n* Figure 4: These heatmaps are interesting and shed a lot more light on the method. I’d be curious to see these heatmaps for MultiRoom (the environment in Figure 3) because it’s less clear to me what each agent would cover in this more sequential environment.", "summary_of_the_paper": "The paper proposes looking at intrinsic exploration from a multi-agent perspective. The proposed method (D&E) rewards an agent for visiting unseen states and for visiting states that are unseen by other agents. The motivation is that if there are multiple parts of the state space, agents can explore these in parallel, thus speeding up the wall-clock exploration time. Experiments look at D&E performance on several MiniGrid tasks as well as VizDoom.", "summary_of_the_review": "This is an interesting setting, but the method lacks novelty (see first weakness). Additionally, the results do not paint an accurate picture of D&E compared with other baselines (see second weakness). Perhaps this project could look more at reconciling options/skill learning work with intrinsic motivation and do more evaluation and comparison of baselines in these areas.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jP8LFiMKME", "reviewer": "Reviewer_cJEK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strengths of this work is in its ability to naturally harness distributed computational infrastructure to solve computationally intractable exploration problems.  To the extent that this work addresses the underlying problem it would be impactful and significant.\n\nThe biggest weakness of this work is that it is difficult to tell if the results are from the particular intrinsic motivation, or if they are a side effect of using parallelism to run independent training runs. The experiments show best of the population of the agents in the proposed method, introducing a selection bias.  Thus we could expect it to outperform other methods even if the intrinsic motivation was ineffective.  To correct for this  the average agent of your method could be report, or the baselines could be allowed to take the max of take the max of n independent runs.  \n\nOn a similar issue, the correction for the number of samples in Table 1 is good, but it could be that max of 3 short runs is generically better than 1 long run in this setting, if initialization matters significantly.   A more apples-to-apples comparison would run the duplicates like mentioned above.\n\nThe second biggest concern is that, even if the intrinsic motivation is necessary, the proposed mechanism of effect is increasing the diversity of the exploration.  If this is the case, then the method should be compared to a naive application of a diversity bonus like such as:\n\"Diversity is All You Need: Learning Skills without a Reward Function\" Eysenbach, B. et al.\n\"The Emergence of Individuality in Multi-Agent Reinforcement Learning\"  Jiang J. et al.\n\nMy third concern is that the system could generally get stuck. Concretely, the claim in the last sentence of section 4 saying that an agent would avoid other agents' failures cuts both ways, in that it would also avoid other agent's successes.  This could end up with two agents which can each only solve half of environments and do not want to explore to solve the rest of them because the other agent is more successful in those cases.\n\n\nMinor: \n* The decay rate in 4 looks like it would sometimes increase over time if the agent did not receive any intrinsic reward.  This may cause oscillation, and is a bit counterintuitive.  If this is intentional it should be justified and flagged for readers.\n* in equation 4 why is the constant 1/U rather than U?\n* In equation 5 and 6 you should define I\n* At the start of the last paragraph on page 8 the sentence structure around the phrases \"on one hand, we use... on the other hand, we replace...\" is unclear, as I assumed from the sentence structure that the se ideas would be conflicting in some way.\n* It seems like the studies mentioned in section 5.3 \"Ablation Study\" are not actually ablations, in that they do not remove parts of the methods, but they visualize the method and check its sensitivity to changes in it's parameters.  These are good studies to have, but I would not consider them ablations.\n* In Figure 5 the key is not in numerical order, which makes the figure harder to read.\n* in the last paragraph of section 4 the phrase \"is considered convergence\" seems to be a typo.", "summary_of_the_paper": "This paper proposes a distributed algorithm for exploration in reinforcement learning, based off of the principle of \"divide and conquer\".  This is implemented as a multi-agent system where each agent is run on it's own node in parallel and given a custom intrinsic reward to motivate exploration and diversity.  These intrinsic rewards are built out of standard intrinsic reward methods for exploration, but are modified so each agent not only gets reward from it's own intrinsic motivation function but also gets reward by how their actions score with respect to the other agent's intrinsic motivation functions.  This motivates each agent to specialize to find the strategies the other agents' intrinsic motivation functions would see as novel.", "summary_of_the_review": "I will weakly recommend rejection, as it is unclear the extent to which the effect of this method is from the parallelism v.s. the intrinsic  motivation, and it is unclear how this approach would compare to the standard diversity promoting intrinsic motivation methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_5CjDXTvDnt", "reviewer": "Reviewer_VtVT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n* The setting of concurrent multi-agent exploration for single agent tasks is worthwhile and understudied\n* Thorough evaluation with respect to existing single agent exploration methods\n\n## Weaknesses\n* Seemingly missing references/discussion regarding some extremely relevant related work. Most closely related is [1] as it addresses the same setting of multiple concurrent agents solving a single-agent task by a divide-and-conquer exploration approach. Moreover, [2] introduces the idea of multiple agents each maintaining their own \"novelty detection\" function and combining these functions to generate intrinsic rewards that prevent redundant exploration of similar spaces.\n* It's unclear that the D&E reward will always result in the desired behavior. It seems as though it requires agents to redundantly explore the same regions or else $\\sum_{j \\neq i} \\tilde{r}^{i,j}$ will always be non-zero.\n* The inner-episodic count mask seems like it would not have a scalable analogue in more complex domains.\n* The method referred to as \"Count\" in the experiments is a psuedo-count method which derives an approximate visit count from a learned density model. A worthwhile baseline (for only the MiniGrid setting) would be to train with the same \"real\" count-based intrinsic rewards without concurrent exploration and the D&E reward. This would eliminate the confounding factor of the effectiveness of the individual intrinsic reward mechanism and highlight how important the contributions of this paper are.\n\n## Questions\n* How are timesteps being counted for your method? I assume it's the sum of steps across all concurrent processes?\n* Why not plot the training curve for VizDoom rather than include a table?\n\n\n### References\n[1] Dimakopoulou, Maria, and Benjamin Van Roy. \"Coordinated exploration in concurrent reinforcement learning.\" International Conference on Machine Learning. PMLR, 2018.\n\n[2] Iqbal, Shariq, and Fei Sha. \"Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning.\" arXiv preprint arXiv:1905.12127 (2019).", "summary_of_the_paper": "This paper presents an approach to enable parallelizable \"divide-and-conquer\" style exploration in sparse reward RL tasks, where each concurrent process maintains its own intrinsic reward function but they are combined in a way that minimizes redundant exploration of the state space across processes.", "summary_of_the_review": "This paper presents interesting ideas; however, they are extremely similar to prior work that is not discussed or compared to in the paper. It is difficult to judge the significance of this work without comparisons to (or at the very least discussion of) this prior work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-bNwqYcmpXx", "reviewer": "Reviewer_xEFb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n- This paper is clearly written. The central ideas around the novel multi-agent intrinsic reward and concurrent implementation details are clearly communicated. The authors also highlight where they introduce additional hyperparameters.\n- The experimental setting consists of standard exploration benchmarks—MiniGrid and VizDoom.\n- A comprehensive set of intrinsic reward baselines used in procedurally-generated environments is included in the experiments.\n- The method is simple and the results are strong in comparison to the intrinsic reward baselines.\n\nWeaknesses\n\n- The D&E method is only benchmarked for the case when the intrinsic reward is a simple count-based reward. Therefore, the experiments contain no direct ablation of the key multi-agent intrinsic reward that is the crux of D&E. This is because the Count method introduced by Bellemere et al, 2016 is not the same as simply applying a count-based exploration bonus. Including an ablation using only the simple count-based intrinsic reward would make the results more conclusive.\n- Related to the above point, including results for D&E combined with at least one of the non-count-based intrinsic motivation methods would further strengthen the results in support of D&E's multi-agent intrinsic reward.\n- Including information about how the additional hyperparameters introduced by D&E were tuned and how sensitive the method is to these parameters would be useful. This is especially important as D&E introduces several hyperparameters: number of agents, decay frequency $K$, reward model update frequency $L$, reward weights $w_0$, $w_s$, $\\alpha_0$, threshold $U$, and decay rate $\\phi$.\n- The authors mention that they used the default hyperparameters for the baseline methods, but as some of the methods were not originally benchmarked on VizDoom (e.g. AMIGo and RAPID), it seems that these VizDoom baselines may be improperly tuned.\n- Including additional seeds for Figure 4 and averaging over seeds for Figure 3 would add confidence that these results are not cherrypicked.", "summary_of_the_paper": "This paper introduces a method for performing multi-agent exploration by supplementing any intrinsic reward method by providing each agent an additional reward that is the sum of intrinsic rewards of all other agents were they to experience the agent's transition. This additional weighted reward term allows the agent to balance exploring states that other agents have rarely encountered alongside those that the agent iself has rarely encountered. The experiments show that this divide-and-conquer strategy for multi-agent exploration results in state-of-the-art results on MiniGrid and VizDoom.", "summary_of_the_review": "This paper clearly presents a simple multi-agent intrinsic reward method for encouraging agents to not only explore regions that they have rarely encountered, but also those that the other agents have rarely encountered. Their results are promising, but lack proper baselines to get the full story about the effectiveness of their method. If the authors address the weaknesses pointed out above, I will enthusiastically raise my score to an 8.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a strategy for multiple learning agents to explore a large RL problem's state space, via the divide and conqeuer principle. It prescribes a design for each agent's reward function, which when optimized enables the agents to 'carve out' and cover different parts of the state space yielding efficient exploratory behavior. The argument for efficacy of the proposed method is purely experimental, with numerical benchmarking on complex simulated environments.

The reviewers have raised several concerns that persist even after receiving detailed responses from the author(s). These include the lack of discussion about comparisons with seemingly closely related and applicable work, the perception that the comparisons of this method with others are not fair ("not apples to apples"), and the assessment that the ablation studies and investigation of the sensitivity to hyperparameters may not be comprehensive to make a compelling argument. Thus, keeping in mind the unanimous impression of the reviewers, I am of the view that while the paper contributes an interesting principle, more work is needed to argue for its acceptance in a clear way.
Decision: Reject