RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
Authors: Roberta Raileanu
Tim Rocktäschel
Venue:  (2020)
Keywords: reinforcement learning
exploration
curiosity

Abstract: Exploration in sparse reward environments remains one of the key challenges of model-free reinforcement learning. Instead of solely relying on extrinsic rewards provided by the environment, many state-of-the-art methods use intrinsic rewards to encourage exploration. However, we show that existing methods fall short in procedurally-generated environments where an agent is unlikely to visit a state more than once. We propose a novel type of intrinsic reward which encourages the agent to take actions that lead to significant changes in its learned state representation. We evaluate our method on multiple challenging procedurally-generated tasks in MiniGrid, as well as on tasks with high-dimensional observations used in prior work. Our experiments demonstrate that this approach is more sample efficient than existing exploration methods, particularly for procedurally-generated MiniGrid environments. Furthermore, we analyze the learned behavior as well as the intrinsic reward received by our agent. In contrast to previous approaches, our intrinsic reward does not diminish during the course of training and it rewards the agent substantially more for interacting with objects that it can control.

Published as a conference paper at ICLR 2020
RIDE: REWARDING IMPACT-DRIVEN EXPLORATION FOR PROCEDURALLY-GENERATE ENVIRONMENTS
Roberta Raileanu"
Tim Rocktäschel
Facebook AI Research
Facebook AI Research University College London rockt fb. com
New York University
ra ellennuucc. cs nyu. edu
ABSTRACT
Exploration in sparse reward environments remains one of the key challenges of model-free reinforcement learning. Instead of solely relying on extrinsic rewards provided by the environment, many state-of-the-ar methods use intrinsic rewards to encourage exploration. However, we show that existing methods fall short in procedurally-generaaee environments where an agent is unlikely to visit a state more than once. We propose a novel type of intrinsic reward which encourages the agent to take actions that lead to significant changes in its learned state represen- tation. We evaluate our method multiple challenging tasks MiniGrid, as well as on tasks with nigh-dimensional observations used in prior work. Our experiments demonstrate that this approach is more sample ef- ficient than existing exploration methods, particularly for vrocedurally-geneaaeee MiniGrid environments. Furthermore, we analyze the learned behavior well the intrinsic reward received by our agent. In contrast to previous approaches, our intrinsic reward does not diminish during the course of training and rewards the agent substantially more for interacting with objects that it can control.
seccedurally-generaate
INTRODUCTION
Deep reinforcement learning (RL) is one of the most popular frameworks for developing agents that solve wide range of complex tasks (Mnih et al., 2016; Silver et al., 2016; 2017). RL agents learn to act in new environments through trial and error, in an attempt to maximize their cumulative reward. However, many environments of interest, particularly those closer to real-world problems, do not provide a steady stream of rewards for agents to learn from. In such settings, agents require many episodes come across any reward, often rendering standard RL methods inapplicable.
Inspired by human learning, the use of intrinsic motivation has been proposed to encourage agents to learn about their environments even when extrinsic feedback is rarely provided (Schmidhuber, 1991b; 2010; Oudeyer et al., 2007; Oudeyer & Kaplan, 2009). This type of exploration bonus emboldens the agent to visit new states (Bellemare et al.. 2016; Burda et al., 2019b; Ecoffet al., 2019) to improve its knowledge and forward prediction of the world dynamics (Pathak et al., 2017: Burda al., 2019a), and can be highly effective for learning in hard exploration games such as Montezuma's Revenge (Mnih et al., 2016). However, most hard exploration environments used in previous work have either limited state space or an easy way to measure similarity between states (Ecoffet et al., 2019) and generally use the same "singleton" environment for training and evaluation (Mnih et al., 2016; Burda et al., 2019a). Deep RL agents trained in this way are prone to overfitting to specific environment and often struggle generalize even slightly different settings (Rajeswaran et al., 2017; Zhang et al., 2018a;b). As a first step towards addressing this problem, a number of mnnceuurally-genrratee environments have been recently released, for example DeepMind Lab (Beattic et al., 2016), Sokoban (Racanière et al., 2017), Malmö (Johnson et al., 2016), CraftAssist (Jemnite al., 2019), Sonic (Nichol et al., 2018), CoinRun (Cobbe et al., 2019), Obstacle Tower (Juliani et al., 2019), or Capture the Flag (Jaderberg et al., 2019).
In this paper, we investigate exploration in procedurally generrted sparse-reward environments. Throughout the paper, we will refer to the general problem that needs to be solved as the task
Work done during an intersship Facebook AI Research.

---

Published as a conference paper at ICLR 2020
embed
Cg
5t
embed
forward
Φ1+1
Gr
Lino
RIDE
inverse
Rour
Lsw
8t+1
embed
Ø1+1
Figure I: RIDE rewards the agent for actions that have an impact on the state representation (RIDE), which is learned using both forward (L)w) and inverse dynamics (Linv) model.
(e.g. find goal inside maze) and to the particular instantiation of this task the environment (e.g. maze layout, colors, textures, locations of the objects, environment dynamics etc.). The environment be singleton procedurally geeerraaedddee Singleton environments are those in which the agent has to solve the same task in the same environment in every episode, i.e., the environment does not change between episodes. popular example of hard exploration environment that falls into that category is Montezuma's Revenge. In sereedurally-geeerrte environments, the agent needs to solve the same task, but in every episode the environment is constructed differently (e.g. resulting in different maze layout), making it unlikely for an agent to ever visit the same state twice. Thus, agents in such environments have learn policies that generalize well across a very large state space. We demonstrate that current exploration methods fall short in such environments as they (i) make strong assumptions about the environment (deterministic or resettable to previous states) (Ecoffet et al., 2019; Aytar al., 2018), (ii) make strong assumptions about the state space (small number of different states or easy to determine two states are similar) (Ecoffet et al., 2019; Burda al., 2019b; Bellemare al., 2016; Ostrovski et al., 2017; Machado et al., 2018a), or (iii) provide intrinsic rewards that can diminish quickly during training (Pathak et al., 2017; Burda et al., 2019a).
To overcome these limitations, propose Rewarding Impact-Driven Exploration (RIDE), novel intrinsic reward for exploration in RL that encourages the agent to take actions which result in impactful changes its representation of the environment state (see Figure for illustration). We compare against state-of-the-art intrinsic reward methods on singleton environments with high- dimensional observations (i.e. visual inputs), as well on hard-exploration tasks in procedurally- generated grid-world environments. Our experiments show that RIDE outperforms kate-of-the-an exploration methods, particularly in procedurally- gelllayeeeeer environments. Furthermore, we present a qualitative analysis demonstrating that RIDE. in contrast to prior work, does not suffer from diminishing intrinsic rewards during training and encourages agents substantially more interact with objects that they can control (relative to other state-action pairs).
2 RELATED WORK
The problem of exploration in reinforcement learning has been extensively studied. Exploration methods encourage RL agents to visit novel states in various ways, for example by rewarding sur- prise (Schmidhuber. 1991b;a; 2010; 2006; Achiam & Sastry, 2017), information gain (Little & Som- mer, 2013; Still & Precup. 2012; Houthooft al., 2016), curiosity (Pathak et al., 2017; Burda al., 2019b), empowerment (Klyubin et al., 2005; Rezende & Mohamed, 2015; Gregor et al., 2017). di- versity (Eysenbach et al., 2019), feature control (Jaderberg et al., 2017: Dilokthanakul al., 2019), or decision states (Goyal et al., 2019; Modhe et al., 2019). Another class of exploration methods apply the Thompson sampling heurisite (Osband et al., 2016; Ostrovski et al., 2017: O'Donoghue et al., 2018; Tang et al., 2017). Osband et al. (2016) use family of randomized Q-functions trained on bootstrapped data to select actions, while Fortunato et al. (2018) add noise in parameter space to encourage exploration. Here, we focus intrinsic motivation methods, which widely-used and have proven effective for various hard-exploration tasks (Mnih et al., 2016; Pathak et al., 2017; Bellemare et al., 2016; Burda al., 2019b).
2

---

Published as a conference paper at ICLR 2020
Intrinsic motivation can be useful in guiding the exploration of RL agents, particularly in environ- ments where the extrinsic feedback is sparse or missing altogether (Oudeyer et al., 2007: 2008; Oudeyer & Kaplan, 2009; Schmidhuber, 1991b; 2010). The most popular and effective kinds of intrinsic motivation can be split into two broad classes: count-based methods that encourage the agent visit novel states and curiosity-based methods that encourage the agent to learn about the environment dynamics.
Count-Based Exploration. Strehl & Littman (2008) proposed the use of state visitation counts as an exploration bonus in tabular settings. More recently, such methods were extended to high- dimensional state spaces (Bellemare et al., 2016; Ostrovski et al., 2017; Martin al., 2017: Tang et al., 2017; Machado et al., 2018 Bellemare et al. (2016) use a Context-Tree Switching (CTS) density model to derive state pseudo-count, while Ostrovski et al. (2017) use PixeICNN as a state density estimator. Burda et al. (2019b) employ the prediction error of a random network as explo- ration bonus with the aim of rewarding novel states more than previously seen ones. However, one can expect count-based exploration methods to be less effective in procedurally-generaae ments with sparse reward. In these settings, the agent is likely to characterize two states as being different even when they only differ by features that are irrelevant for the task (e.g. the texture of the walls). If the agent considers most states to be "novel", the feedback signal will not be distinctive or varied enough to guide the agent.
environ-
Curiosity-Driver Exploration. Curiosity-based bonuses encourage the agent to explore the en- vironment to learn about its dynamics. Curiosity can be formulated as the error or uncertainty in predicting the consequences of the agent's actions (Stadie et al., 2015; Pathak et al., 2017; Burda al., 2019b). For example, Pathak et al. (2017) learn a latent representation of the state and design an intrinsic reward based on the error of predicting the next state in the learned latent space. While we use a similar mechanism for learning state embeddings, our exploration bonus is very different and builds upon the difference between the latent representations of two consecutive states. As we will see in the following sections, one problem with their approach is that the intrinsic reward can vanish during training, leaving the agent with no incentive to further explore the environment and reducing its feedback to extrinsic reward only.
Generalization in Deep RL. Most of the existing exploration methods that have achieved impres- sive results on difficult tasks (Ecoffet et al., 2019; Pathak et al., 2017; Burda et al., 2019b; Bellemare et al., 2016; Choi et al., 2019; Aytar et al., 2018), have been trained and tested on the same environ- ment and thus do not generalize to new instances. Several recent papers (Rajeswaran et al., 2017; Zhang et al., 2018a;b; Machado et al., 201 8h; Foley et al., 2018) demonstrate that deep RI. is sus- ceptible to severe overfitting. As a result, a number of benchmarks have been recently released for testing generalization in RL (Beattie et al., 2016; Cobbe et al., 2019; Packer al., 2018; Justesen et al., 2018; Leike et al., 2017; Nichol et al., 2018; Juliani et al., 2019). Here, we make another step towards developing exploration methods that can generalize to unseen scenarios by evaluating them on serredurally-generate
environments. We opted for MiniGrid 2018) because it fast to run, provides a standard set of tasks with varied difficulty levels, focuses on single-agent, and does not use visual inputs, thereby allowing us to better isolate the exploration problem.
Seevaaler-Booiverr et al.,
More closely related to our work are the papers of Marino et al. (2019) and Zhang et al. (2019). Marino et al. (2019) use a reward that encourages changing the values of the non-proprioceptive features for training low-level policies on locomotion tasks. Their work assumes that the agent has access to a decomposition of the observation state into internal and external parts, an assumption which may not hold in many cases and may not be trivial to obtain even if it exists. Zhang et al. (2019) use the difference between the successor features of consecutive states as intrinsic reward. In this framework, state is characterized through the features of all its successor states. While both of these papers use fixed (i.e. not learned) state representations to define the intrinsic reward, we use forward and inverse dynamics models to learn a state representation constrained only capture elements in the environment that can be influenced by the agent. Lesort et al. (2018) emphasize the benefits of using a learned state representation for control as opposed to a fixed one (which may not contain information relevant for acting in the environment). In the case of Zhang et al. (2019), constructing a temporally extended state representation for aiding exploration is not trivial. Such a feature space may add extra noise to the intrinsic reward due to the uncertainty of future states. This is particularly problematic when the environment is highly stochastic or the agent often encounters novel states (as it is the case in procedurally geneeeerrlllyggeeerrteeed environments).
3

---

Published as a conference paper at ICLR 2020
3 BACKGROUND: CURIOSITY-DRIVER EXPLORATION
We use the standard formalism of a single agent Markov Decision Process (MDP) defined by of states S, a set of actions A, and a transition function T S A + P(S) providing the probability distribution of the next state given a current state and action. The agent chooses actions by sampling from a stochastic policy π S - P(A), and receives reward r : S time step. The agent's goal is to learn policy which maximizes its discounted expected return R. - Px-E 2
+ R at every
where ry is the sum of the intrinsic and extrinsic reward received by the agent at time i, 7 E [0, 1] is the discount factor, and the expectation is taken with respect to both the policy and the environment. Here, we consider the case of episodic RL in which the agent maximizes the reward received within a finite time horizon.
In this paper consider that, along with the extrinsic reward rf. the agent also receives some intrinsic reward rí. which can be computed for any (st, ae, St+1) tuple. Consequently, the agent tries to maximize the weighted sum of the intrinsic and extrinsic reward: Tr = r + wirri where war a hyperparameter to weight the importance of both rewards.
We built upon the work of Pathak al. (2017) who note that some parts of the observation may have no influence on the agent's state. Thus, Pathak et al. propose learning a state representation that disregards those parts of the observation and instead only models (i) the elements that the agent can control, as well as (ii) those that can affect the agent, even the agent cannot have an effect on them. Concretely, Pathak al. learn state representations d(s) femb(s; Oemb) of a state s using an inverse and a forward dynamics model (see Figure 1). The forward dynamics model neural network parametrized by frr that takes as inputs (s{)) and at, predicts the next state representation: ((8+11) = (tu, ae; Ofw:), and it is trained to minimize (0 fw: Oemb) = [l0(s++1) (((+1)) The inverse dynamics model is also a neural network parameterized by predicts the agent's action: â = fime(ór, Φ141; 0mne), and it is trained to minimize Line (f Bemb) Cross ProosEnntopyya a2) when the action space is discrete. Pathak et al.'s 's curiosity-based intrinsic reward is proportional to the squared Euclidean distance between the actual embedding of the next state ((8++1) 1 and the one predicted by the forward model ((1+11).
dine that takes as inputs ((s)) and predicts
4 IMPACT-DRIVER EXPLORATION
Our main contribution is a novel intrinsic reward based on the change in the state representation produced by the agent's action. The proposed method encourages the agent try out actions that have significant impact on the environment. We demonstrate that this approach can promote effective exploration strategies when the feedback from the environment is sparse.
We train a forward and an inverse dynamics model to learn a latent state representation d(s) as proposed by Pathak al. (2017). However, instead of using the Euclidean distance between the predicted next state representation and the actual next state representation as intrinsic reward (Rour in Figure 1), we define impact-driven reward as the Euclidean distance between consecutive state representations (RIDE in Figure 1). Compared to curiosity- driven exploration, impact-driven ex- ploration rewards the agent for very different state-actions, leading to distinct agent behaviors which we analyze in Section 6.1.1.
Stanton & Clune (2018) categorize exploration into: across-training and intra-life and argue they are complementary. Popular methods such as count-based exploration (Bellemare et al., 2016) encour- age agents to visit novel states in relation to all prior training episodes (i.e. across-training novelty), but they do not consider whether an agent visits novel states within some episode (i.e. intra-life novelty). As we will see, RIDE combines both types of exploration.
Formally, RIDE is computed as the L2-norm [[0(s++1) @(st) of the difference in the learned state representation between consecutive states. However, to ensure that the agent does not go back and forth between sequence of states (with a large difference in their embeddings) in order to gain intrinsic reward, we discount RIDE by episodic state visitation counts. Concretely, we divide the impact-driven reward by NN J. where (op() is the number of times that state has been visited during the current episode, which is initialized to 1 in the beginning of the episode. In high-
4

---

Published as a conference paper at ICLR 2020
dimensional regimes, one episodic pseudo counts instead (Bellemare et al., 2016; Ostrovski et al., 2017). Thus, the overall intrinsic reward provided by RIDE is calculated as:
RIDE a.) = r 0.)) = [1((5+1) - ((st)l2
/Nep(st+1)
where ((8(+1) and (({)) are the learned representations of consecutive states, resulting from the agent transitioning state 8141 after taking action az in state st. The state is projected into latent space using a neural network with parameters femb-
The overall optimization problem that is solved for training the agent is
min
[w (0x)) + fw (0s Oemb) + Winn One embb]]
0ino Oy aor000mm Oomb
where 0x are the parameters of the policy and value network (a, ~ (=));())) )), and Was Wine and W scalars that weigh the relative importance of the reinforcement learning (RL) loss to that of the inverse and forward dynamics losses which are used for learning the intrinsic reward signal. Note that never update the parameters of the inverse (0mn)), forward (w)w) or embedding networks (0ems) using the signal from the intrinsic or extrinsic reward (i.e. the RL loss); we only use these learned state embeddings for constructing the exploration bonus and never as part of the agent's policy (Figure I highlights that the policy learns its own internal representation of the state vi, which only used for control and never for computing the intrinsic reward). Otherwise, the agent can artificially maximize its intrinsic reward by constructing state representations with large distances among themselves, without grounding them in environment observations.
Note that there is no incentive for the learned state representations to encode features of the envi- ronment that cannot be influenced by the agent's actions. Thus, our agent will not receive rewards for reaching states that are inherently unpredictable, making exploration robust with respect to dis- tractor objects or other inconsequential sources of variation in the environment. As we will later show, RIDE is robust to the errors in the prediction of its forward model (such as the one proposed in Pathak et al. (2017)), gets attracted local sources of entropy in the environment. Furthermore, the difference of consecutive state representations is unlikely to go to zero during learning as they are representations of actual states visited by the agent and constrained by the forward and inverse model. This is in contrast to Pathak et al. (2017) and Burda et al. (2019b) where the intrinsic reward goes to zero as soon as the forward model becomes sufficiently accurate or the agent's policy only explores well known parts of the state space.
noisy-TV problem in which agent, that is rewarded for
5 EXPERIMENTS
We evaluate RIDE on procedurally-generatee environments from MiniGrid, well as two existing singleton environments with high-dimensional observations used in prior work, and compare it against both standard RL and three commonly used intrinsic re- ward methods for exploration. For all our experiments, we show the mean and standard deviation of the average return across dif- ferent seeds for each model. The average return is computed as the rolling mean over the past 100 episodes.
5.1 ENVIRONMENTS
The first set of environments are amcedurally-generaae grid- worlds in MiniGrid Schevalier-Boisver et al., 2018). We consider three types of hard exploration tasks: MultiRoomNXSY, KeyCorri- dorS3R3, and ObstructedMaze2Dil..
Figure 2: Rendering of
procedurally-geenrated en- vironment from MiniGrid's MultiooomN12S10 task.
In MiniGrid, the world is a partially observable grid of size N x N. Each tile in the grid contains at most one of the following objects:
wall, door, key, ball, box and goal. The agent can take of seven actions: turn left right, move forward, pick up or drop an object, toggle or done. More details about the MiniGrid environment and tasks be found in A.3.
5

---

Published as a conference paper at ICLR 2020
MultiRoom-N la)
MutiRRom--Noisy11-N/-55
KeyCorridor-S2-4
MultiRoom--N1--a (d)
10
Average Rtturn
0.6
RIDE
RND
0.2
IMPALA
0.00
0.25
0.50
0.75
1.00
1e7
1n7
107
168
MuttiRoom-NT- (e)
MultiRoom N10-54 54 (f)
Obstructec Mze 2Dth (g)
MultiRoom-NLa N12-S10 (h)
1.0
Aversge Aversge Iteum
0.6
07
0.0
0.00
0.25
0.50
0.75
1.00
Number Frames
167
Number Frames 167
Number Fromes 167
Number Fromes 1e8
Figure 3:
Performance of RIDE, Count, RND, ICM and IMPALA on a variety of hard exploration problems in MiniGrid. Note RIDE is the only one that can solve the hardest tasks.
For the sole purpose of comparing in a fair way to the curiosity-driven exploration work by Pathak al. (2017), we ran one-off experiment on their Mario (singleton) environment (Kauten, 2018). We train model with and without extrinsic reward on the first level of the game.
The last (singleton) environment we evaluate on ViaDoom (Kempka et al., 2016). Details about the environment can be found in A.4.
5.2 BASELINES
For all our experiments, we use IMPALA (Espeholt et al., 2018) following the implementation of Küttler et al. (2019) as the base RL algorithm, and RMSProp (Tieleman & Hinton, 2012) for op- timization. All models use the same basic RI. algorithm and network architecture for the policy and value functions (see Appendix A.2 and Appendix A.I for details regarding the hyperparameters and network architectures), differing only in how intrinsic rewards are defined. In our experiments we compare with the following baselines: Count: Count-Bacee Exploration by Bellemare et al. (2016) which uses state visitation counts to give higher rewards for new or rarely seen states. RND: Random Network Distillation Exploration by Burda al. (2019b) which uses the prediction error of random network as exploration bonus with the aim of rewarding novel states more than previ- ously encountered ones. ICM: Intrinsic Curiosity Module by Pathak al. (2017) (see Section 3). IMPALA: Standard RL approach by Espeholt al. (2018) that uses only extrinsic reward and en- courages random exploration by entropy regularization the policy.
RESULTS AND DISCUSSION
We present the results of RIDE in comparison to popular exploration methods, well an analysis of the learned policies and properties of the intrinsic reward generated by different methods.
6.1 MINIGRID
Figure summarizes our results on various hard MiniGrid tasks. Note that the standard RI. approach IMPALA (purple) is not able to learn in any of the environments since the extrinsic reward is too sparse. Furthermore, our results reveal that RIDE is more sample efficient compared to all the other exploration methods across all MiniGrid tasks considered here. While other exploration bonuses seem effective on easier tasks and are able to learn optimal policies where IMPALA fails, the gap between approach and the others is increasing with the difficulty of the task. Furthermore, RIDE manages to solve some very challenging tasks on which the other methods fail to get any reward even after training 100M frames (Figure 3).

---

Published as a conference paper at ICLR 2020
RND
ICM
RIDE
0.20
0,018
0012
0.004
0.000
JP
0.020
0018
0.008
0.000
0.020
0.018
0012
0.008
0.004
Figure 4: Intrinsic reward heatmaps for RND, ICM, and RIDE (from left to right) for opening doors (green), moving forward (blue), or turning left or right (red) on a random environment from the MultiRoomNNS4 task. A is the agent's starting position, G is the goal position and D are doors that have to be opened on the way.
Open Door
Turn Left Right
Move Forward
Model
Mean
Std
Mean
Std
Mean
Std
RIDE
0.0490
0.0019
0.0071
0.0034
0.0181
0,0116
RND 0.0032
0.0018
0.0031
0.0028
0.0026
0.0017
ICM
0.0055
0.0003
0.0052
0.0003
0.0056
0.0003
Table 1: Mean intrinsic reward per action over 100 episodes random maze in MultiRoomN7S4.
In addition to existing MiniGrid tasks, we also tested the model's ability to deal with stochasticity in the environment adding "noisy TV" in the MiniGridN7S4 task, resulting in the Mini- GirdN7S4NoisyTV task (left-center plot in the top row of Figure 3). The noisy TV is implemented ball that changes its color a randomly picked one whenever the agent takes particular action. As expected, the performance of ICM drops as the agent becomes attracted to the ball while obtain- ing intrinsic rewarded for not being able predict the next color. The Count model also needs more time to train, likely caused by the increasing number of rare and novel states (due to the changing color of the ball).
We include results for ablations to our model in Appendix A.5, highlighting the importance of combining impact-driven exploration with episodic state visitation discounting.
6.1.1 ANALYSIS THE INTRINSIC REWARD
To better understand the effectiveness of different exploration methods, we investigate the intrinsic reward an agent receives for certain trajectories in the environment.
Figure 4 shows heatmap of the intrinsic reward received by RND, ICM, and RIDE on a sam- pled environment after having been trained on poocedurally-generatee environments from the Mul- tiRoomN7S4 task. While all three methods can solve this task, the intrinsic rewards received are different. Specifically, the RIDE agent is rewarded in a much more structured manner for opening doors, entering new rooms and turning at decision points. Table I provides quantitative numbers for this phenomenon. We record the intrinsic rewards received for each type of action, averaged over 100 episodes. We found that RIDE is putting more emphasis actions interacting with the door than for moving forward or turning left or right, while the other methods reward actions more uniformly.

---

Published as a conference paper at ICLR 2020
Figure 12 and Table 3 in A.6.2 show similar pattern for the intrinsic rewards for agents trained on the MultiRoom 12000 task, while Figure 13 and Table in A.6.3 contain the equivalent analysis for agents trained on ObstrrateddMaze2Dll. As emphasized there, RIDE is rewarding the agent more for interactions with objects as opposed to actions for moving around in the maze, a characteristic which is not prevalent in the other models.
Mult/Room-N12-510
Figure shows the mean intrinsic reward of all models while training on the MaltiRoomNI2SI0 task. While the ICM,
0.010
RIDE
Average Intrinnic Reward
COUNT
RND, and Count intrinsic reward converges to very low val-
0.008
RND
ues quite early in the training process, the RIDE bonus keeps
changing and has a higher value even after training on 100M
0.006
frames. Hence, RIDE constantly encourages the agent to take actions that change the local environment. In contrast, Count, RND, and Curiosity may not consider certain states to be "novel" or "surprising" after longer periods of training as they have seen similar states in the past or learned to almost perfectly predict the next state in a subset of the environment states. Consequently, their intrinsic rewards diminish during training and the agent struggles to distinguish between actions that lead to novel or surprising states from those that do not, thereby getting trapped in some parts of the state space (see Figure 12).
0.004
0.002
0.00
0.25 0.50 0.75 1.00
Number of Frames 1eB
Figure 5: Mean intrinsic re- ward for models trained on Multi- RoomN12S10.
6.1.2 SINGLETON VERSUS PROCEDURALLY-
ENVIRONMENTS
It is important to understand and quantify how much harder it to train existing deep RL exploration methods on tasks in rnvcduualll-ggenerate environments compared to singleton environment.
Fixed ObstructedMasee2Dh
To investigate this dependency, trained the models on singleton environment of the the ObstructedMazee2Dil task so that the beginning of every episode, the agent is spawned in exactly the same maze with all objects located in the same positions. In this setting, we see that Count, RND, and IM- PALA are also able to solve the task (see Figure 6 and com- pare with the center-right plot in the bottom row of Fig- ure 3 for senoouurallygeneratee environments of the same task). As expected, this emphasizes that training an agent in rrreddurally-generate
1.0
Average Retur
0.5
COUNT
0.0
RND
ILM
Number Frames 1e7
environments creates significant chal- lenges over training on a singleton environment for the same task. Moreover, highlights the importance of training a variety of environments to avoid overfitting to the idiosyn- crasies of a particular environment.
Figure 6: Training on singleton instance of ObstructedMazee2Dll..
6.1.3 No EXTRINSIC REWARD
To analyze the way different methods explore environments without depending on the chance of run- ning into extrinsic reward (which can dramatically change the agent's policy), we analyze agents that are trained without any extrinsic reward on both singleton and mnocedurally-generatte environments.
The top row of Figure 7 shows state visitation heatmaps for all the models in singleton environment on the MultiRoomN NISS task, after training all of them for 50M frames with intrinsic reward only. The agents allowed to take 200 steps in every episode. The figure indicates that all models have effective exploration strategics when trained on a singleton maze, the 10th, 9th and 6th rooms are reached by RIDE, Count/RND, and ICM, respectively. The Random policy fully explores the first room but does not get to the second room within the time limit.
When trained procedurally generraed mazes, existing models are exploring much less efficiently as can be seen in the bottom row of Figure 7. Here, Count, RND, and ICM only make it to the 4th, 3rd and 2nd rooms respectively within episode, while RIDE is able explore all rooms. This
8

---

Published as a conference paper at ICLR 2020
Count
RND
ICM
Random
RIDE
7.5
3.0
0.0
9.0
6.0
4.5
3.0
1.5
Figure 7: State visitation heatmaps for Count, RND, ICM, Random, and RIDE models (from left to right) trained for 50m frames without any extrinsic reward on singleton maze (top row) and on pooceuralll--eeneaaa mazes (bottom row) in MultiRoomN1OS:
Mario Intrinsic (a)
Mario intrinsic and Extrinsic (b)
VitDoom (c)
800
800
600
800
0.75
400
400
Average
200
200
0.50
RINF
-200
RND
ICM
-20
0.00
-400
IMPALA
-400
-0.25
0.0
0.4
0.8
Number Frames [1e6]
Number of Frames [3e6]
Number Frames (167)
Figure 8: Performance Mario with intrinsic reward only (a), with intrinsic and extrinsic reward (b), and VizDoom (c). Note that IMPALA trained with extrinsic reward only in cases.
further supports that RIDE learns state representation that allows generalization across different mazes and not as distracted by less important details that change from one procedurally-genee environment to another.
6.2 MARIO AND VIZDOOM
In order to compare Pathak al. (2017), evaluate RIDE the first level of the Mario envi- ronment. Our results (see Figure 8 and b) suggest that this environment may not be as challenging as previously believed, given that all the methods evaluated here, including vanilla IMPALA, can learn similarly good policies after training only Im frames even without any intrinsic reward (left figure). Note that we are able reproduce the results mentioned in the original ICM paper (Pathak et al.. 2017). However, when training with both intrinsic and extrinsic reward (center figure), the curiosity- exploration bonus (ICM) hurts learning, converging later and to a lower value than the other methods evaluated here.
For VizDoom (see Figure c) we observe that RIDE performs as well as ICM, while all the other baselines fail to learn effective policies given the same amount of training. Note that our ICM implementation can reproduce the results in the original paper on this task. achieving a 100% success rate after training approximately 60m frames (Pathak et al., 2017).

---

Published as a conference paper at ICLR 2020
7 CONCLUSION AND FUTURE WORK
In this work, we propose Rewarding Impact-Drivee Exploration (RIDE), an intrinsic reward bonus that encourages agents to explore actions that substantially change the state of the environment, as measured in a learned latent space. RIDE has a number of desirable properties: it attracts agents states where they can affect the environment, it provides signal agents even after training for a long time, and it is conceptually simple as well as compatible with other intrinsic or extrinsic rewards and any deep RL algorithm.
Our approach is particularly effective in procedurally-generaae sparse-reward environments where significantly outperforms IMPALA (Espeholt et al., 2018), well as some of the most popu- lar exploration methods such as Count (Bellemare et al., 2016), RND (Burda et al., 2019b), and ICM (Pathak et al., 2017). Furthermore, RIDE explores sereeddurally-gneerrae environments more efficiently than other exploration methods.
However, there are still many ways to improve upon RIDE. For example, one can make use of sym- bolic information to measure or characterize the agent's impact, consider longer-term effects of the agent's actions, or promote diversity among the kinds of changes the agent makes to the environ- ment. Another interesting avenue for future research is to develop algorithms that can distinguish between desirable and undesirable types of impact the agent can have in the environment, thus con- straining the agent act safely and avoid distractions (Le. actions that lead to large changes in the environment but that are not useful for given task). The different kinds of impact might corre- spond to distinctive skills lowelevel policies that hierarchical controller could use learn more complex policies or better exploration strategies.
ACKNOWLEDGMENTS
We would like to thank Heinrich Küttler, Edward Grefenstette, Nantas Nardelli, Jakob Foerster, Kyunghyun Cho, Arthur Szlam, Rob Fergus, Victor Zhong and Léon Bottou for insightful discus- sions and valuable feedback on this work.
REFERENCES
Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. CoRR, abs/1703.01132, 2017. URL
www//orgsss....oooo.... at//ab///11000 1703. ...00030000 01 732..
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems, pp. 2930-2941, 2018.
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Mareus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, Julian Schritwieser,, Keith Ander- son, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab, CoRR, abs/1612.03801, 2016. URL http: //txivi org/ abs 1612 03801.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1606:0:666..05..0 2016.
Yuri Burda, Harrison Edwards, Deepak Pathak, Amos J. Storkey, Trevor Darrell, and Alexei A. Efros. Lcrge-scale study of curiosity- driiven learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019a. URL https: /ppenreviiew.. net./ fort///ormm?? wtt.rrmmmmm..d.dww..a.......... JNWD
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random net- work distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA. USA, May 6-9, 2019, 2019b. URL https: http::/:///peeevvee...oo net forum? did= H1 Ymm.5
10

---

Published as a conference paper at ICLR 2020
Maxime Chevalier- Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. h /ttpttgttt/..////////////...b..t com/ wwtp//wwwwicaa.aca/taca..tccacae hetl:/yymmmaaci maymmmmiiigrdd, 2018.
Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and exploration in reinforcement learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL https://////c. www.oeveew........... net. forum? d
Honglak Lee. Contingency-
Ajó Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In 4th International Conference Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico May 2-4, 2016. Conference Track Proceedings, 2016. URL http:// /tt//aaiiivvv. org/ als 1511. 07289.
Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, ICMI, 2019, 9-15 June 2019, Long Beach, California, USA, pp. 1282-1289, 2019. URL httptht:pp//oeeeeetttttt
ml press...s................ /v97
cobbel 9a html.
Nat Dilokthanakul, Christos Kaplanis, Nick Pawlowski, and Murray Shanahan. Feature control as intrinsic motivation for hierarchical reinforcement learning. IEEE transactions neural net- works and learning systems, 2019.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for explorattn problems. arXiv preprint arXiv:1901.109% 2019.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Jain Dunning, Shane Legg, and Koray Kavukcuoglu. IM- PALA: scalable distributed deep-rl with importance weighted actor-leamer architectures. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stock- holmsmässan, Stockholm, Sweden, July 10-15, 2018, pp. 1406-1415, 2018. URL http:
/ //proceedings . mwm.m.rrreee www...press/vv00 www.eepppphltttceeeeeeeeeeeeeeee html.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. In 7th International Conference Learn- ing Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL https: / ///openeeview.netee ntwpeeewweeeeeeeeeeeeeeeeeeeeeee www.:/sttoooddddeeeeee.e
John Foley, Emma Tosch, Kaleigh Clary, and David Jensen. Toybox: Better atari environments for testing reinforcement learning agents. CoRR, abs/1812.02555 2018. URL attt:://arxia
org/abs /112. 02850.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Os- band, Alex Graves, Volodymyr Mnih, Rémi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In 6th International Confer- ence on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL https:// /ttp://///penrvie..ommomm net. forum? ry=yHWACK
Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Hugo Larochelle, Matthew Botvinick, Yoshua Bengio, and Sergey Levine. Infobot: Transfer and exploration via the formation bottleneck. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL https: id=rJg8yhAqKm.
forum?
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings, 2017. URL https: Skc-Fo4Yg.
http::://opeeeeiieeeeeee http//pppnre.e..eeeeeee net /oormmtumm Pid=
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1 1109-1111 117,, 2016.
11

---

Published as a conference paper at ICLR 2020
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czamecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24- 26, 2017, Conference Track Proceedings, 2017. URL https:/ / 1d=SJ6yPD5xg.
/pppoppeeevviieee net/ forum?
Max Jaderberg, Wojeiech M Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari Morcos, Avraham Ruderman, et al. Human- level performance in 3d multiplayer games with population-based reinforcement learning, Sci- ence, 364(644):33)98-66666 2019.
Yacine Jemiie,, Kavya Srinet, Jonathan Gray, and Arthur Szlam. Craftassist instruction pars- ing: Semantic parsing for a minecraft assistant. CoRR, abs/1905.01978, 2019. URL http:
//aaxxiv.org/aaeeeee abs 1905. 01978.
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artifi- cial intelligence experimentation. In Proceedings of the Twenty- Coonty-Fifth International Joint Confer- ence on Artificial Intelligence, IJCAI 2016, New York, NY. USA, 9-15 July 2016, pp. 4246-4247, 2016. URL http:://///. / /www. jcai .org/ Awst / 643.
Arthur Juliani, Ahmed Khalifa, Fincent-Pierre Berges, Jonathan Harper, Ervin Teng, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: A generalization challenge in vision, control, and planning. In Proceedings of the Twenty-Fighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 16,16, 2019, pp. 2684-269) 2019. doi: 01.244663/jcai.22199//3 URL https: org.iorr///ooooooooooooooooooooo 0 24 01244433/jjooo 2019/373.
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv preprint arXiv: 10729, 2018.
Christian Kauten. Super Mario Bros for OpenAl Gym. GitHub, 2018. URL UtLhttps:////thh com/Kautenjja//ee mwppa//o------------------------
Michal Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaskowski. Viz- doom: A doom-based research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEEE, 2016.
Alexander Klyubin, Daniel Polani, and Chrystopher Nehaniv. All else being equal empow- ered. In European Conference on Artificial Life, pp. 744-753. Springer, 2005.
Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. TorchBeast: A PyTorch Platform for Dis- tributed RL. arXiv preprint arXiv:1910.03550 2019. URL https:: / com/ sicebokkreearrchhtocchbeee
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq. Lau- rent Orseau, and Shane Legg. safety gridworlds. arXiv preprint arXiv: 1.09813, 2017.
Timothée Lesort, Natalia Díaz Rodríguez, Jean-François
Goudou, and David Filliat. State represen- tation learning for control: An overview. Neural Nenworks, 108:379-392 2018. doi: 10.1016/j. neunet.2018.07.00 URL https: /ttp::::////ooooo/////./
doi. org/10 1016/ neunet 2018. 10008.00.006....
Daniel Ying-Jeh Little and Friedrich Tobias Sommer. Learning and exploration in action-perception loops. Frontiers neural circuits, 7:37, 2013.
Marlos C Machado, Mare G Bellemare, and Michael Bowling. Count-based exploration with the successor representation. arXiv preprint arXiv: 1807. 1622, 2018a
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 201 8b.
12

---

Published as a conference paper at ICLR 2020
Kenneth Marino, Abhinav Gupta, Rob Fergus, and Arthur Szlam. Hierarchical RL using an ensemble of proprioceptive periodic policies. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL https: / /ppenrevieee net. forum? dd= x20cFQ.
Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Marcus Hutter. Count-based explo- ration in feature space for reinforcement learning. In Proceedings of the Twenty-Sixth Inter- national Joins Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pp. 2471-2478, 2017. doi: (0.24963/jccai22117744 URL https: httpss://do// /to.//o.ooooo .rg/ 10. 24963/ j ja 201 7/344.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
Nirbhay Modhe, Prithvijit Chattopadhyay, Mohit Sharma, Abhishek Das, Devi Parikh, Dhruv Ba- tra, and Ramakrishna Vedantam. Unsupervised discovery of decision states for transfer in rein- forcement learning. CoRR, abs/1907.10580, 2019. URL http: org/ abs ab919977. 1907. 10580.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprint arXiv: SEE.:v880400770 2018
Brendan O' 'Donoghue, Ian Osband, Rémi Munos, and Volodymyr Mnih. The uncertainty bellman equation and exploration. In Proceedings of the 35th International Conference Machine Learn- ing. 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018. pp. 3836-3845, 2018. URL http:::/ppocceeeinnggg/o//////// ml.r
http://///ddddoooodeeeee a..mml..
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp. 4026-4034, 2016.
Georg Ostrovski, Mare Bellemare, Aäron van den Oord, and Rémi Munos. Count-based ex- ploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning-Volume Volume 70, pp.
JMLR. org, 2017.
Pierre- Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? typology of computa- tional approaches. Frontiers in neurorobotics, 1:6, 2009.
Pierre-Yve Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions on evolutionary computation, 11(2):265-286. 2007.
Bherre-Yees Oudeyer, Frederic Kaplan, et al. How can we define intrinsic motivation. In Proc. of the 8th Conf. on Epigenetic Robotics, volume 5, pp. 29-31, 2008
Charles Packer, Katelyn Gao, Jemej Kos, Philipp Krähenbühl. Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv: /810. 2018.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Sébastien Racanière, Theophane Weber, David P. Reichert, Lars Buesing. Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomênech Badia, Oriol Vinyals, Nico- las Heess, Yujia Li, Razvan Pascanu, Peter W. Battaglia, Demis Hassabis. David Silver, and Daan Wierstra. learning. In Advances in Neural Information Processing Systems 30: Annual Con- ference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 5690-5701, 2017. URL http:/ http://appes...em nirr..nnpp..... cttppperr 7152-imagin wwwiiiiggnnniioo augmented agents- for deep- reinforcement wwwporreeeeet-eernnnnn
imaginnation-augeentt agents for deep reinforcement
Aravind Rajeswaran. Kendall Lowrey, Emanuel V Todorov, and Sham M Kakade. Towards gen- cralization and simplicity in continuous control. In Advances in Neural Information Processing Systems, pp. 6550-6561, 2017.
13

---

Published as a conference paper at ICLR 2020
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pp. 1530-1538, 2015. URL http: htttt:::/ppcceedinngg. .mm.eeddinnnmm.r........ .press// v37/rezender 5. html.
Jürgen Schmidhuber.
Curious model-building control systems. In Proc. international joint confer- ence on neural networks, pp. 1458-11 1458-1463, 19
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model- building neu- ral controllers. In Proc. of the international conference simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991b.
Jürgen Schmidhuber. Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts. Connection Science, 18(2):173-18) 187, 2006.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions on Autonomous Mental Development, 2(3):230-247, 2010.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676)3354, 2017.
Bradly Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXivv 2015.
Christopher Stanton and Jeff Clune. Deep curiosity search: Intra-life exploration can improve perfor- mance on challenging deep reinforcement learning problems. arXiv preprint arXiv: 0001.80000.0 2018.
Susanne Still and Doina Precup. An informattion-thcoretee approach to curiosity- driven reinforce- ment learning. Theory in Biosciences, 131(3)119-118,, 2012.
Alexander L Strehl and Michael L Littman. An analysis of model- interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-1333) 2008
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Chen, Yan Duan, John Schul- man, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pp. 2753- 2762, 2017.
T Tieleman and G Hinton. Rmsprop: Divide the gradient by running average of its recent mag- nitude. coursera: Neural networks for machine learning. Tech. Rep., Technical report, pp. 31, 2012.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint
079377 201 8a.
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study overfitting in deep reinforcement learning. arXiv preprint 3rXiv:1804.0630 2018b.
Jingwei Zhang, Niklas Wetzel, Nicolai Dorka, Joschka Boedecker, and Wolfram Burgard. Sched- uled intrinsic drive: A hierarchical take on intrinsically motivated exploration. arXiv preprint arXiv:1903.074000 2019.
14

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 NETWORK ARCHITECTURES
All our models use the same network architecture for the policy and value networks. The input is passed through a sequence of three (for MiniGrid) or four (for the environments used by Pathak al. (2017)) convolutional layers with 32 filters each, kernel size of 3x3, stride of and padding of 1. An exponential linear unit (ELU; (Clevert et al. (2016))) is used after each convolution layer. The output of the last convolution layer is fed into a LSTM with 256 units. Two separate fully connected layers are used to predict the value function and the action from the LSTM feature representation.
For the singleton environments used in prior work, the agents are trained using visual inputs that are pre-pre pr-ccoseesed similarly to Mnih et al. (2016). The RGB images are converted into gray-scale and resized to 42 42. The input given to both the policy and the state representation networks consists of the current frame concatenated with the previous three frames. In order to reduce overfitting, during training, we use action repeat four. At inference time, we sample the policy without any action repeats.
A.2 HYPERPARAMETERS
We ran grid searches over the learning rate E [0.0001, 0.0005, 0.000]. batch size E [8, [8] and unroll length E [20, 40, 100, 200]. The best values for all models can be found Table 2. The learning rate is linearly annealed to 0 in all experiments.
Parameter
Value
Learning Rate
0.0001
Batch
Unroll Length
100
Discount
0.99
RMSProp Momentum
0,0
RMSProp
0.01
Clip Gradient Norm e
40.0
Table Hyperparameters common to all experiments.
We also ran grid searches over the intrinsic reward coefficient E [1.0, 0.5, 0.1, 0.05, 0.01, 0.005, 0.001] and the entropy coefficient [0.01, 0.005, 0.001, 0.0005, 0.0001, 0.00005] for all the models on all environments. The best intrinsic reward coefficient was 0.1 for ICM and RND, and 0.005 for Count on all environments. The best entropy coefficient was 0.0001 for ICM, RND, and Count on all environments. For RIDE, we used an intrinsic reward coefficient of 0.1 and entropy coefficient of 0.0005 for MultiRoomN7S4, MultiRoomNoisyTVM. and 0.5, 0.001 for MultiRoomN7S8, Maze2Dlh. The ablations use the same hyperparameters as RIDE. In all experiments presented here, the best values found for each model.
MultiRoomN10S4, KeyCoridorS3R3 MultiRoomMNOS/ 10, MultRoommN12S100 and Obstructed-
A.3 MINIGRID ENVIRONMENT
In MiniGrid, the world is partially observable grid of size NxN. Each tile in the grid contains exactly zero or one objects. The possible object types are wall, door, key, ball, box and goal.
Each object in MiniGrid has an associated discrete color, which can be one of red, green, blue, purple, yellow or grey. By default, walls are always grey and goal squares are always green. Rewards sparse for all MiniGrid environments.
There are seven actions in MiniGrid: turn left, turn right, move forward, pick up an object, drop an object, toggle and done. The agent use the turn left and turn right action to rotate and face one of 4 possible directions (north, south, east, west). The move forward action makes the agent move from its current tile onto the tile in the direction it currently facing, provided there nothing
15

---

Published as a conference paper at ICLR 2020
that tile, or that the tile contains open door. The agent open doors if they are right in front of it by using the toggle action.
Observations in MiniGrid are partial and egocentric. By default, the agent sees square of 7x7 tiles in the direction it is facing. These include the tile the agent is standing on. The agent cannot see tensor of shape 7x7x3. However,
through walls closed doors. The observations are provided note that these are not RGB images. Each tile is encoded using 3 integer values: one describing the type of object contained in the cell, one describing its color, and flag indicating whether doors are open or closed. This compact encoding was chosen for space efficiency and to enable faster training. For all tasks, the agent gets an egocentric view of its surroundings, consisting of 3x3 pixels. neural network parameterized as a CNN is used to process the visual observation.
The MulriRoomNXXSY environment consists of rooms, with size at most Y, connected in random orientations. The agent is placed in the first room and must navigate to a green goal square in the most distant room from the agent. The agent receives an egocentric view of its surrounding, consisting of 3x3 pixels. The task increases in difficulty with X and Y. Episodes finish with positive reward when the agent reaches the green goal square. Otherwise, episodes are terminated with zero reward after maximum of 20xN steps.
In the KeyCorridorS3R3 environment, the agent has to pick up an object which is behind a locked door. The key is hidden in another room, and the agent has to explore the environment to find it. Episodes finish with a positive reward when the agent picks up the ball behind the locked door or after maximum of 270 steps.
In the ObstructedMaze2Dlh environment, the agent has to pick up a box which is placed in a corner of 3x3 maze. The doors are locked, the keys are hidden in boxes and the doors are obstructed by balls. Episodes finish with a positive reward when the agent picks up the ball behind the locked door or after maximum of 576 steps.
In the DynamicOsstacles environment, the agent has to navigate to a fixed goal location while avoiding moving obstacles. In our experiments, the agent is randomly initialized location in the grid. If the agent collides with an obstacles, it receives a penalty of -1 and the episode ends.
A.4 VIZDOOM ENVIRONMENT
We consider the Doom 3D navigation task where the action space of the agent consists of four discrete actions: move forward, move left, move right and no-action. Our testing setup in all the experiments is the (Brockman et al., 2016). Episodes are terminated either when the agent finds the vest or if the agent exceeds a maximum of 2100 time steps. The map consists and the agent is tasked to reach some fixed goal location from its spawning location. The agent is always spawned in Room-13 which is 270 steps away from the goal under an optimal policy. A long sequence of actions is required to reach the goals from these rooms, making this setting hard exploration problem. The agent is only provided a sparse terminal reward of +1 if it finds the vest and 0 otherwise. While this environment has sparse reward, it is not procedurally-generee agent finds itself in exactly the same environment in each episode and does not need to generalize to different environment instantiations. This environment is identical the "sparse" setting used in Pathak et al. (2017).
environments which is available as part of OpenAl Gym
rooms connected by corridors
so the
A.5 ABLATIONS
In this section, we aim to better understand the effect of using episodic discounting as part of the intrinsic reward, well that of using entropy regularization as part of the IMPALA loss.
Figure compares the performance of our model on different MiniGrid tasks with that of three abla- tions. The first one only uses episodic state counts as exploration bonus without multiplying it by the impact-driven intrinsic reward COnlyEpisodicCountt exploration bonus without multiplying it by the episodic state count tenm (NoEpisodieCounn) while the third one is the NoEpisodic model without the entropy regularization term in the IMPALA loss (NoEntoppNofiioddiCCmm
the second one only uses the impact-driven
16

---

Published as a conference paper at ICLR 2020
OnlyEpisodicCounts does not solve any of the tasks. NoEnrropNoopissdiicCouutss either converges to a suboptimal policy or completely fails. In contrast, NoEpisodicCounti can solve the easier tasks but it requires more interactions than RIDE and fails to learn on the hardest domain. During training, NofEpssadiCComti can get stuck cycling between two states (with a large distance in the embedding states) but due entropy regularization, it sometimes escape such local optima (unlike OoEaropsNoEiisodiCoom sparse, NoEpioodic Counts is insufficient while RIDE still succeeds, indicating the effectiveness of augmenting the impact-driven intrinsic reward with the episodic count term.
if it finds extrinsic reward. However, when the reward is too
MultiRoom-NoisyTV--N7555
MultiRoom--N0--5
Average Return
Average Return
04
02
0.2
00
0.0
-0.2
2.0
0.0
0.5
10
1.5
Number of Frames
l07
Number Frames
107
ObstructedMaaze-DDlh
MultiRoom-N12-510
0.6
1.0
Average Retum
0.8
0.4
RIDE
0.6
0.3
NoEpisooiicoouts
Average
OnlyEptooiccoontt
0.4
0.2
0.2
0.0
0.0
-0.1
0.8
Number of Frames
1n7
Number Frames
105
Figure 9: Comparison between the performance of RIDE and three ablations: OntyEpisodiccoouts, NoEpisodicCounts, and NoEntnopyNa&PiisctcCComs
Figure 10 shows the average number states visited during an episode Multiooom 12S 10, measured at different training stages for full RIDE model and the NoEpisodicCount ablation. While the NoEpisodicCounts ablation always visits a low number of different states each episode (S 10), RIDE visits increasing number of states throughout training (converging to 100 for an optimal policy). Hence, can be inferred that CoFFpsodiiCountt revisits some of the states. This claim can be further verified by visualizing the agents' behaviors. After training, NoEpiisoicCounts goes back and forth between two states, while RIDE visits each state once on its path to the goal. Consistent with our intuition, discounting the intrinsic reward by the episodic state-count term does help to avoid this failure mode.
A.6 ANALYSIS
A. STATE VISITATION IN 12S10
In this section, we analyze the behavior learned by the agents. Figure 11 shows the state visitation heatmaps for all models trained on 100m frames of MultiRoomN122I reward. Note that while our model has already reached the goal the farthest room the maze, Count has explored about half of the maze, while RND and ICM still in the first two rooms.
which has very sparse
17

---

Published as a conference paper at ICLR 2020
140
RIDE
120
No Episodic Counts
Number Visitod States
100
20
1m
5m
1 100
15m
20m
Number Training Frames
Figure 10: Average number of states visited during an episode of MultioommN12S10, measured different training stages for our full RIDE model (blue) and the NolEpisodiCCouns ablation (or- ange).
Figure 11: State visitation heatmaps for Count, RND, ICM, and RIDE (from left to right) trained for 100m frames on MultioommN12S10 12S10.
Figure 12: Intrinsic reward heatmaps for RND, ICM, and RIDE (from left to right) on Multi-
RoomN12S10.
A.6.2 INTRINSIC REWARD IN MULTIROOMN 12520
Figure 12 shows a heatmap of the intrinsic reward received by RIDE, RND, and ICM agents trained on the procedurally -enerratd MultiRoomN12S11 environment. Table 3 shows the corresponding intrinsic rewards received for each type of action, averaged over 100 episodes, for the trained models. This environment is very challenging since the chance of randomly stumbling upon extrinsic reward extremely low. Thus, we see that while the intrinsic reward provided by RIDE is still effective at exploring the maze and finding extrinsic reward, the exploration bonuses used by RND and ICM are less useful, leading to agents that do not go beyond the second room, even after training on 100m frames.
18

---

Published as a conference paper at ICLR 2020
Open Door
Turn Left / Right
Move Forward
Model
Mean
Std
Mean
Std
Mean
Std
RIDE
0.0116
0.0011
0.0042
0.0020
0.0032
0.0016
RND
0.0041
0,0016
0,0035
0,0013
0.0034
0.0012
ICM
0.0082
0.0003
0.0074
0.0005
0.0086
0.0002
Table 3: Mean intrinsic reward per action computed over 100 episodes on a random map from MultiRooMN12100.
0.0030
0.0006
0.0025
0.0005
0.0020
0.0004
0.0015
0.0003
0.0010
0.0002
0.0005
0,0001
0.0000
0,0000
0.0030
0.0006
0.0025
0.0005
0.0020
0.0004
0.0015
0,0003
0.0010
0.0002
0.0005
0.0001
0.0000
0.0000
0.0030
0.0006
0.0025
36C
0.0005
0.0020
0.0004
0.0015
0.0003
0.0010
0.0002
0.0005
0.0001
0.0000
0.0000
Figure 13: Intrinsic reward heatmaps for RND (left) and RIDE (right) for interacting with objects (i.e. open doors, pick up/ / drop keys or balls) (green), moving forward (blue), or turning left or right (red) on random map from Dbs hidden inside boxes (that need to be opened in order to see their colors), D are colored doors that
the agent's starting position, K are the keys can only be opened by keys with the same color, and order to win the game. After passing through the door the agent also needs to drop the key in order to be able to pick up the ball since it can only hold one object at a time.
the ball that the agent needs to pick up in
A.6.3 INTRINSIC REWARD OBSTRUCTEDMAZEZDDII
In order to understand how various interactions with objects are rewarded by the different exploration methods, also looked at the intrinsic reward the ObstructedMaze2Dllhh environment which contains multiple objects However, the rooms are connected by locked doors and the keys for unlocking the doors are hidden inside boxes. The agent does not know in which room the ball located and it needs the color of the key to match that of the door in order open it. Moreover, the agent cannot hold more than one object so needs to drop one in order to pick up another.
Figure 13 and Table 4 indicate that RIDE rewards the agent significantly for interacting with various objects (e.g. opening the box, picking up the key, opening the door, dropping the key, picking up the ball) relative other actions such moving forward turning left and right. In contrast, RND again rewards all actions much more uniformly and often times, within an episode, rewards the interactions with objects less than the ones for moving around inside the maze.
19

---

Published as a conference paper at ICLR 2020
Open Door
Pick Ball
Pick Key
Drop Key
Other
Model
Mean
Std
Mean
Std
Mean
Std
Mean
Std
Mean
Std
RIDE
0.0005
0.0002
0.0004
0.0001
0.0004
0.00001
0.0004
0.00007
0.0003
0.00001
RND
0.0034
0.0015
0.0027
0.0006
0.0026
0.0060
0.0030
0.0010
0.0025
0.0006
Table 4: Mean intrinsic reward per action computed over 100 episodes on random map from ObstructedMazc2DII..
SymamicOsstacleds
SynamicOlstccleeee
0:31
0.0
Mi0 FART
6.00
ran
122
2.22
-223
D
-350
-835
13
10
62
Figure 14: Performance DynamicObstacles with varying degrees of difficulty.
Randomized Wall Colors
0.8
Average Return
RIDE
COUNT
0.4
RX
0.2
IMPALA
-0.2
Number of Frames
Figure 15: Performance on a version of the MiniGridRoomN7S4 in which the colors of the walls and goals randomly picked from set of 4 colors at the beginning of each episode.
A.7 DYNAMIC OBSTACLES ENVIRONMENT
One potential limitation of RIDE that it may be drawn to take actions that significantly change the environment, even when those actions are undesirable. In order test the limits RIDE, we ran experiments on the DynamicObstacles environment in MiniGrid. As seen in Figure 14, RIDE learns solve the task of avoiding the moving obstacles in the environment, even if chasing them provides large intrinsic rewards. Hence, RIDE is still able to learn effectively in certain scenarios which high-impact actions are detrimental to solving the task.
A.8 GENERALIZATION TO UNSEEN COLORS
In order to test generalization to unseen colors, we also ran experiments on version of Multi- RoomN7S4 in which the colors of the walls and the goal change at each episode. The models are trained on set of 4 colors and tested on a held-out set of colors. As seen in Figure 15 and Table 5, RIDE and Count learn to solve this task and can generalize to unseen colors at test time without any extra fine-tuning RND and ICM perform slightly worse the test environments, and only one out of five seeds of ICM converges to the optimal policy on the train environments. The best seed for each model used evaluate the test
20

---

Published as a conference paper at ICLR 2020
Test Return
Model
Mean
Std
RIDE
0.77
0.02
Count
0.77
0.02
RND
0.76
0.11
ICM
0.73
0.03
IMPALA
0.00
0.00
Table 5: Average return over 100 episodes on a version of MuliRoomNN7S4 in which the colors of the walls and goals change with each episode. The models were trained until convergence on set of 4 colors and tested on held-out set of 2 colors.
A.9 OTHER PRACTICAL INSIGHTS
While developing this work, we also experimented with few other variations of RIDE that did not work. First, we tried to use observations instead learned state embeddings for computing the RIDE reward, but this was not able to solve any of the tasks. Using common state representation for both the policy and the embeddings also proved to be ineffective.
21


Reviews:
[{"review_id": "r1l45lqTKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the problem of intrinsically motivating in DRL. In particular, it focuses on exploration of procedurally generated environments where many states are novel compared to training experiences. It offers an intrinsic reward based on large movement in a state embedding space where this state embedding representation is co-trained on the same data already collected for learning. The paper claims to overcome shortcomings of specific past approaches (e.g. count-based / curiosity).\n\nThe need for intrinsic motivation in exploration is well motivated, and the approach for training a state embedding is anchored in multiple past works. The use of movement in this state embedding as an intrinsic reward is importantly novel and valuable. The problematic propensity for RL researchers to train on the test environments or design agents that are confused by proverbial noisy TVs and/or sacrifice extrinsic rewards in favor of intrinsic rewards is satisfyingly discussed and addressed through detailed experiments.\n\nThis reviewer moves to accept the paper for its contributions to intrinsically motivated exploration with thorough discussion of how the technique addresses shortcomings of past methods. This reviewer is thankful that the authors do not overinterpret the MiniGrid results and that they provide intuition for why the state embedding functions capture what we want them to capture. The fact that this approach makes joint use of the whole (s,a,r,s') tuple feels significant, as does the fact that this approach does not require any changes to the policy network (e.g. presuming that features useful for computing intrinsic rewards are also going to be useful for directly acting to optimize extrinsic rewards).\n \nQuestion:\n- In partially observable environments that require agents to wait for something, should a RIDE-motivated agent consider changes in its own internal clocks (part of the recurrent state) impactful moves? If an environment might require a recurrent / history-aware action policy, should RIDE also be made history aware? Might a history-aware RIDE reward sufficiently motivate a stateless/reactive policy?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1xw9yhZcH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThis paper proposes a Rewarding Impact-Driven Exploration (RIDE), which is an intrinsic exploration bonus for procedurally-generated environments. RIDE is built upon the ICM architecture (Pathak et al. 2017), which learns a state feature representation by minimizing the L2 distance between the actual next state feature and the predicted next state feature while minimizing the cross-entropy loss between the true action and the estimated action from the consecutive state features. Finally, RIDE's intrinsic reward bonus is computed by L2 norm of the difference between the current state feature and the next state feature, divided by the square root of the visitation count of the next state within the episode. Experimental results show that RIDE outperforms the existing exploration methods in the procedurally-generated environments (MiniGrd), and is competitive in singleton environments.\n\n\nComments and questions:\n- In reinforcement learning, the agent should explore the experiment due to uncertainty. If everything in the environment is certain to the agent, then it does not have to explore and just exploiting the past experience would be the best. My major concern about the paper is 'impact-driven' reward bonus may not account for the uncertainty. Constantly encouraging the states that have a high impact would not always good, and it may interfere to converge to an optimal policy.\n- It seems that RIDE assumes that 'high-impact' states are always good, thus rewarded. It could be true on the conducted MiniGrid domains, but this assumption may not hold in general. Could 'impact-driven' exploration be realistic and be applied to more general problems?\n- Similarly, in the problems where high-impact states have to be avoided, can RIDE still work effectively? For example, how about 'Dynamic-Obstacles' domains implemented in MiniGrid? In this task, RIDE may promote to chase obstacles that have to be avoided, interfering with learning optimal policy. It would be great to show the effectiveness of RIDE in such environments.\n- In MiniGrid problems, if the colors of walls and the goal are changed at every episode, does RIDE work well?\n- In Figure 4, why the intrinsic reward heatmaps are drawn only on the straight paths?\n- Minor: In the last sentence of Section 3, \"the current state and the next state predicted by the forward model\" -> \"the actual next state and the next state predicted by the forward model\"\n\n\n---\nafter rebuttal:\n\nThank the authors for clarifying my questions and concerns. Most of my concerns are addressed, and I raise my score accordingly.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkxHBiPAKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new intrinsic reward method for model-free reinforcement learning agents in environments with sparse reward. The method, Impact-Driven Exploration, learns a state representation of the environment separate from the agent to be trained, based on a combied forward and inverse dynamics loss. The agent is then separately trained with a reward encouraging sequences of actions that maximally change the learned state.\n\nLike other latent state transition models (Pathak et al. 2017), RIDE learns a state representation based on a combined forward and inverse dynamics loss. However, Pathak et al. rewards the agent for taking actions that lead to large difference between the actual next state and the predicted next state. RIDE instead rewards the agent for taking actions that lead to a large difference between the actual next state and the current state. However, because rewarding one-step state differences may cause an agent to loop between two maximally-different states, the RIDE loss term is augmented with a state visitation count term, which decreases intrinsic reward for a state based on the number of times that state has been visited in the current episode.\n\nThe experiments compare RIDE to a selection of other intrinsic reward methods in the MiniGrid, Mario, and VizDoom environments. RIDE provides improved performance on a number of tasks, and solves challenging versions of the MiniGrid tasks that are not solved by other algorithms.\n\nDecision: Weak Accept.\n\nThe main weakness of the paper seems to be a limitation in novelty.\nPrevious papers such as (Pathak et al. 2017) have trained RL policies using an implicit reward based on learned latent states. Previous papers such as (Marino et al. 2019) have used difference between subsequent states as an implicit reward for training an RL policy. It is not a large leap to combine these two ideas by training with difference between subsequent learned states. However, this paper seems to be the first to do so.\n\nStrengths:\nThe experiments section is very thorough, and the visualizations of state counts and intrinsic reward returns are insightful.\nThe results appear to be state of the art for RL agents on the larger MiniGridWorld tasks.\nThe paper is clearly-written and easy to follow.\nThe Mario environment result discussed in section 6.2 is interesting in its own right, and provides some insight into previous work.\n\nDespite the limited novelty of the IDE reward term, the experiments and analysis provide insight into the behavior of trained agents and the results seem to improve on existing methods.\nOverall, the paper seems like a worthwhile contribution.\n\nNotes:\nIn section 2 paragraph 4, \"sintrinsic\" should be \"intrinsic\".\nIn section 3, at \"minimizes its discounted expected return,\" seems like it should be \"maximizes\".\nThe explanation of IMPALA (Espeholt et al., 2018) should occur before the references to IMPALA on page 5.\nLabels for the axes in figures 4 and 6 would be helpful for readability.\n\nThe motivation for augmenting the RIDE reward with an episodic count term is that the IDE loss alone would cause an agent to loop between two maximally different states.\nIt would be interesting to know whether this suspected behavior actually occurs in practice, and how much the episodic count term changes this behavior.\nIt is surprising that in the ablation in section A.5, removing the state count term does not lead to the expected behavior of looping between two states, but instead the agent converges to the same behavior as without the state count term.\n\nAlso, in Figure 9, was the OnlyEpisodicCounts ablation model subjected to the same grid search described in A.2, or was it trained with the same intrinsic reward coefficient as the other models?\nBased on the values in Table 4, it seems like replacing the L2 term with 1 without changing the reward coefficient would multiply the intrinsic reward by a large value.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the problem of exploration in deep reinforcement learning in procedurally-generated environments, where the same state is rarely encountered twice. The authors show that existing methods do not perform well in these settings and propose an approach based on intrinsic reward bonus to address this problem. More specifically, they combine two existing ideas for training RL policies: 1) using implicit reward based on latent state representations (Pathak et al. 2017) and 2) using implicit rewards based on difference between subsequent states (Marino et al. 2019).

Most concerns of the reviewers have been addressed in the rebuttals. Given that it builds so closely on existing ideas, the main weakness of this work seems to be the novelty. The strength of this paper resides in the extensive experiments and analysis that highlight the shortcomings of current techniques and provide insight into the behaviour of trained agents, in addition to proposing a strategy which improves upon existing methods.

The reviewers all agree that the paper should be accepted. I therefore recommend acceptance.
Decision: Accept (Poster)